You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 81 Next »

Purpose

The Workflow Broker (formerly "Ensemble Broker") is a GSI web service designed to manage complex sequences of large computational jobs on production hosts.

Features

  • Expressive (XML) description logic which allows for
    • Structuring workflow in two independent layers:
      • Top level (broker-centric): a directed acyclical graph of executable nodes;
      • Individual node payloads: scripts to be run on the production resource, usually inside an Elf container;
    • Configuration of scheduling and execution properties on a node-by-node basis;
  • Integration with scheduling systems, such as MOAB;
  • (Potentially recursive) service-side parameterized expansion of nodes (done lazily, as the nodes become ready to run);
  • Fully asynchronous staging, submission and monitoring of jobs;
  • State management through a persistent object-relational layer (Hibernate);
  • Cancellation and restart of nodes or entire workflows.

The broker is designed to be extensible with respect to

  • Type of node payload (currently ELF/Ogrescript and C-Shell are available);
  • Type of launching and monitoring protocols used (currently pre-WS GRAM, GSI-SSH to the native batch system, and local submission to a MOAB peer are available).

These extensions usually only involve implementing a special module type.

Comparison with other systems

Mode of Usage

In order to run through the broker, a Workflow Description must be created and submitted (refer to the Siege pages for a walk-through of this process). This description in turn will contain one or more scripted payloads, which can run from trivial to very complex, depending on the nature of the work to be accomplished through a single job launched onto the host; however, no special changes or recompilation of the underlying computational codes is necessary.

Siege also allows the user to inspect the state of workflows which have been or are being processed and to view all the events published in connection with them; this is particularly useful for debugging purposes. Once again this can be done entirely asynchronously (i.e., one can disconnect and then, upon reconnecting, retrieve a historical view of the state of the workflow).

Interactions

This service sends and receives events via a JMS message bus, and acts as a client to the Host Information Service in order to configure the payloads to run on specific resources. For its relative position in the larger service-infrastructure, see Services; a slightly more detailed look at the service topology is illustrated by this typical interaction diagram for a simple submission.

Design

  • No labels