Purpose
The Workflow Broker (formerly "Ensemble Broker") is a GSI web service designed to manage large computational jobs on production hosts; its features include
- Highly expressive description logic which allows for
- Structuring workflow in two independent layers:
- Top level (broker-centric): a directed acyclical graph of executable nodes;
- Individual node payloads: scripts to be run on the production resource, usually inside an Elf container;
- Configuration of scheduling and execution properties on a node-by-node basis;
- Structuring workflow in two independent layers:
- Integration with scheduling systems, such as MOAB;
- (Potentially recursive) Parameterized expansion of nodes;
- Staging, submission and monitoring of jobs;
- Cancellation and restart of nodes or entire workflows.
The broker is designed to be extensible in terms of
- Type of node payload (currently ELF/Ogrescript and C-Shell are available);
- Type of launching and monitoring protocols used (currently pre-WS GRAM, GSI-SSH to the native batch system, and local submission to a MOAB peer are available).