Getting started with Open Cognitive Environment (OpenCE, former WMLCE)

Open Cognitive Environment

Welcome to the OpenCE project. The project contains everything that is needed to build conda packages for a collection of machine learning and deep learning frameworks. All packages created for a specific version of OpenCE have been designed to be installed within a single conda environment.

See Getting Started with Python Environment on HAL System for a detailed list of package versions in each environment.

Simple Example with TensorFlow

Interactive mode

Get a node for interactive use:

swrun -p gpux1

Once on the compute node, load PowerAI module using one of these:

module load opence
module load opence-v1.3.1

Copy the following code into file "mnist-demo.py":

import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(512, activation=tf.nn.relu),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)

Train on MNIST with keras API:

python ./mnist-demo.py

Batch mode

The same can be accomplished in batch mode using the following tf_sample.swb script:

wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/tf_sample.swb
sbatch tf_sample.swb
squeue

Visualization with TensorBoard

Interactive mode

Get a node for interactive use:

swrun -p gpux1

Once on the compute node, load PowerAI module using one of these:

module load opence
module load opence-v1.3.1

Download the code mnist-with-summaries.py to $HOME folder:

cd ~
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/mnist-with-summaries.py

Train on MNIST with TensorFlow summary:

python ./mnist-with-summaries.py

Batch mode

The same can be accomplished in batch mode using the following tfbd_sample.swb script:

wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/tfbd_sample.swb
sbatch tfbd_sample.swb
squeue

Start the TensorBorad session

After job completed the TensorFlow log files can be found in "~/tensorflow/mnist/logs", start the TensorBoard server on hal-ondemand, detail refers Getting started with HAL OnDemand.

Simple Example with Pytorch

Interactive mode

Get a node for interactive use:

swrun -p gpux1

Once on the compute node, load PowerAI module using one of these:

module load opence
module load opence-v1.3.1

Install samples for Pytorch:

pytorch-install-samples ~/pytorch-samples
cd ~/pytorch-samples

Train on MNIST with Pytorch:

python ./examples/mnist/main.py