Introduction
This document describes the architecture of the KISTI-NCSA Science Gateway (KNSG) Framework. To fully describe this architecture, this document also provides information about external technologies that were used to build the framework (e.g. Eclipse).
KNSG provides an extensible open source software platform for building HPC software applications that are managed by PTPFlow. The KNSG frameworks provides an easy to use graphical user interface for setting up and launching HPC workflows and views to manage the execution and retrieve results. The initial framework builds upon the Eclipse Rich Client Platform (Eclipse RCP), PTPFlow and Bard.
Purpose
The purpose of this document is to outline the major components of the KNSG Framework architecture for developers who are interested in using and extending the framework for their software applications. All software extension points are documented with the eAIRS CFD workflow use case used as an example concrete implementation.
Scope
The result of version 1.0 is an initial framework for building new HPC applications with some semantic capabilities and an easy to use graphical user interface for setting up, launching and monitoring HPC workflows.
Definitions, Acronyms and Abbreviations
See the Glossary for definitions, acronyms and abbreviations used in this document.
Overview
This document should contain all of the required information for developing new KNSG applications. Where external technologies are used, there will be links to documentation for them. The Use Case section indentifies the initial goal of the project, which is to provide users with an application for setting up, launching and monitoring the eAIRS CFD workflow.
Architectural Goals and Constraints
Development Tools and Resources
Eclipse RCP/Subversion
The Eclipse RCP platform provides an plugin-based architecture for building new applications. Eclipse provides core application functionality (such as drag and drop, window management, extension points and extension capabilities, etc) that allows users to build professional-looking application, with native look-and-feel, on multiple platforms and allowing them to focus on their value-add instead of building the entire application framework from scratch.
Graphical Editing Framework
The graphical editing framework is a toolkit for graphing and editing models. You can learn more about GEF here and here.
C/C++ Development Plugin
The PTPFlow library used included in KNSG uses a small amount of functionality from the C++ Development Plugin provided by Eclipse since PTPFlow was initially intended to be just views and perspectives withing the Eclipse IDE for C+. The C+ IDE was chosen primarily for it's size (it is the smallest Ecli with an easier to use and pse download). This also illustrates the power of KNSG's plugin-based architecture. KNSG version 1.0 puts an easier to use GUI on top of PTPFlow making it easier for novice users to run HPC Workflows while still providing the full capabilities of PTPFlow for advanced users. For more information about PTPFlow, go here.
Java
JavaSE 1.6
The KNSG framework is built against JavaSE 1.6; however, it should also build with JavaSE 1.5 since no JavaSE 1.6 language specific features are being used.
Initial Use Case: eAIRS CFD
The initial use case is the eAIRS RCP application which will be built using the KNSG Application Framework. This version of eAIRS and KNSG will focus on providing an easy to use graphical user interface for setting up the eAIRS CFD Workflow and launching it with PTPFlow. After launching the workflow, PTPFlow's monitoring facilities will be used to monitor the execution of the workflow and then the framework will provide the ability to retrieve workflow results.
Data Catalog
Repositories
Interfaces and Base Class
public IDataset retrieveDataset(DatasetId id); public List<DatasetProperties> listAllDatasetProperties(); public boolean hasDataset(DatasetId id); public boolean deleteDataset(DatasetProperties properties); public DatasetProperties getDatasetProperties(DatasetId id); public boolean isDisabled(); public void setDisabled(boolean disabled); public boolean isWritable();
public DatasetId importDataset(DatasetProperties properties, List<URI> files); public void saveDatasetProperties(DatasetProperties properties);
protected boolean disabled = false;
public boolean isWritable() {
return true;
}
Local Repository
This represents a concrete implementation of a repository on the local machine. It provides methods for managing data on the local machine.
// Base directory where the local repository is located protected File baseDirectory; // Directory containing the datasets protected File datasetDirectory; // Directory containing the dataset properties protected File propertiesDirectory;
Dataset
Dataset Interface
public String getFriendlyName(); public void setFriendlyName(String friendlyName); public DatasetId getDatasetId(); public void setDatasetId(DatasetId datasetId);
DatasetProperties
private String name; private String description; private DatasetId datasetId; private List<IMetadata> properties = new LinkedList<IMetadata>();
DatasetBean
This class represents a generic file dataset in KNSG. Datasets can be tagged as inputs for different workflows so that the analysis pages can use this information for adding the correct input to workflows.
private DatasetId datasetId; private String friendlyName; private File data;
Metadata
Metadata is loosely defined as data that describes data. In the KNSG Framework, any new metadata types should implement the IMetadata interface so the system is aware of it and it is stored properly. The first section below defines the basics of the IMetadata interface and the section that follows the interface definition contains two concrete examples, TagBean and AnnotationBean for tagging and commenting datasets.
IMetadata Interface
public static final String EXT_PT = "edu.illinois.ncsa.knsg.metadata"; public String getLabel(); public String getValue();
Metadata Implementations
Add a tag for a dataset using a TagBean. There is no limit to the number of tags a dataset could have (e.g. eAIRS-Input, result, etc).
private String tag;
// metadata type
public String getLabel() {
return "tag";
}
public String getValue() {
return tag;
}
The AnnotationBean allows users to add comments to a dataset. The annotation consists of a title, comment and date the comment was made.
private String title; private String annotation; private Date date = new Date();
Scenario
A Scenario is the container object for all things that are in a users scenario including including a list of datasets, the workflows that have been executed, and the RMI Service that the scenario uses. The scenario tracks all parts that make up each scenario and the framework is designed to support multiple scenarios. The scenario inherits from the IUserFacing interface, as do all objects in the KNSG Framework that are saved and restored as beans. Here we will only show the Scenario class, see the repository for the other classes.
private List<WorkflowBuilderModel> workflows; private List<IDataset> datasets; private ServiceInfo rmiService;
Managers
Managers provide access to objects in the KNSG framework and manage them. For example, the ScenarioManager allows users to build multiple scenarios simultaneously so that many what-if type analyses can be executed. Managers in the KNSG Framework should extend the BaseManager class, which provides basic services such as saving the state of the managers members and providing change listeners so objects are aware of changes to members of the Manager.
protected File stateDirectory; // The type of members this manager manages protected String type; protected List<BaseMember> members; protected IMemberChangeListenerSupport listeners = new IMemberChangeListenerSupport(); protected IMemberChangedListener myPassThroughListener = new MyIMemberChangeListener(); protected Object lock = new Object(); protected long lastSave = System.currentTimeMillis(); // BaseMember contains the change listener so adding and removing members is slightly different protected void addMember(BaseMember member); protected void removeMember(IMember member); public BaseMember[] getAllMembers();
Scenario Manager
The ScenarioManager manages each scenario that is created.
private static ScenarioManager instance;
private ScenarioManager() {
type = "scenario";
}
public Scenario getSelectedScenario();
public void addScenario(Scenario scenario);
public void removeScenario(Scenario scenario);
protected BaseMember createMember(File file);
public void removeDatasetFromAllScenarios(DatasetProperties properties);
public void removeDatasetFromAllScenarios(IDataset dataset);
The type tells KNSG where to store the state information for this manager and the objects that it manages. For example, this manager's state will be stored in user.home/NCSA/KNSG/scenario. So, if user.home was /home/foo, this would result in the state directory being /home/foo/NCSA/KNSG/scenario