Versions Compared

Old Version 5

changes.mady.by.user Dawei Mu

Saved on

compared with

New Version Current

changes.mady.by.user Yan Zhan

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Info
titleWMLCE has reached End-Of-Life and is now out of date.

See Getting started with Open Cognitive Environment (OpenCE, former WMLCE) for the latest software stack.


Table of Contents

IBM Watson Machine Learning Community Edition (WMLCE-1.7.0, WMLCE-1.6.2)

WMLCE The PowerAI is an enterprise software distribution that combines popular open-source deep learning frameworks, efficient AI development tools, and accelerated IBM® IBM Power Systems™ Systems servers. The PowerAI-1.6.0 includesIt includes the following frameworks:

.1 deep learning platform that provides a seamless path from research prototyping to production deployment Facebook
FrameworkVersionDescription
Caffe1.0Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research and by community contributors.
TensorFlowPytorch1.133.1TensorFlow Pytorch is an endopen-to-end open source platform for machine learningdeep learning platform that provides a seamless path from research prototyping to production deployment. It is developed by Google Facebook and by community contributors.Pytorch
TensorFlow2.1.0Pytorch TensorFlow is an end-to-end open-source platform for machine learning. It is developed by Google and by community contributors.

Load PowerAI Module

...

For complete WMLCE documentation, see https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Here we only show simple examples with system-specific instructions.

Simple Example with Caffe

Interactive mode

Get one compute node for interactive use:

Code Block
swrun -p gpux1

Once on the compute node, load PowerAI module using one of these:

Code Block
module load wmlce/1.6.2
module load wmlce/1.7.0

Install samples for Caffe:

Code Block
caffe-install-samples ~/caffe-samples
cd ~/caffe-samples

Download data for MNIST model:

Code Block
./data/mnist/get_mnist.sh

Convert data and create MNIST model:

Code Block
./examples/mnist/create_mnist.sh

Train LeNet on MNIST:

Code Block
./examples/mnist/train_lenet.sh

Batch mode

The same can be accomplished in batch mode using the following caffe_sample.swb script:

Code Block
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/caffe_sample.swb
swbatch caffe_sample.swb
squeue

Simple Example with Caffe2

Interactive mode

Get a node for interactive use:

Code Block
swrun -p gpux1

Once on the compute node, load PowerAI module using one of these:

Code Block
module load ibm/poweraiwmlce/1.6.2
module load wmlce/1.6.0.py2 # for python2 environment7.0

Install samples for Caffe2:

Code Block
caffe2-install-samples ~/caffe2-samples
cd ~/caffe2-samples

Download data with LMDB:

Code Block
python ./examples/lmdb_create_example.py --output_file lmdb

Train ResNet50 with Caffe2:

Code Block
python ./examples/resnet50_trainer.py --train_data ./lmdb

Batch mode

The same can be accomplished in batch mode using the following caffe2_sample.swb script:

Code Block
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/caffe2_sample.swb
sbatch caffe2_sample.swb
squeue

Simple Example with TensorFlow

Interactive mode

Get a node for interactive use:

Code Block
swrun -p gpux1

Once on the compute node, load PowerAI module using one of these:

Code Block
module load wmlce/1.6.2
module load ibm/poweraiwmlce/1.67.0.py3 # for python3 environment
module load ibm/powerai

Copy the following code into file "mnist-demo.py":

Code Block
import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(512, activation=tf.nn.relu),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              # python3 environment by default

Simple Example for Caffe

Initial an interactive job

Code Block
srun --partition=debug --pty --nodes=1 --ntasks-per-node=8 --gres=gpu:v100:1 -t 01:30:00 --wait=0 --export=ALL /bin/bash

Install samples for Caffe

metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)

Train on MNIST with keras API:

Code Block
python ./mnist-demo.py

Batch mode

The same can be accomplished in batch mode using the following tf_sample.swb script:

Code Block
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/tf_sample.swb
sbatch tf_sample.swb
squeue

Visualization with TensorBoard

Interactive mode

Get a node for interactive use:

Code Block
swrun -p gpux1

Once on the compute node, load PowerAI module using one of these:

Code Block
module load wmlce/1.6.2
module load wmlce/1.7.0

Download the code mnist-with-summaries.py to $HOME folder:

Code Block
cd ~
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/mnist-with-summaries.py

Train on MNIST with TensorFlow summary:

Code Block
python ./mnist-with-summaries.py

Batch mode

The same can be accomplished in batch mode using the following tfbd_sample.swb script:

Code Block
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/tfbd_sample.swb
sbatch tfbd_sample.swb
squeue

Start the TensorBorad session

After job completed the TensorFlow log files can be found in "~/tensorflow/mnist/logs", start the TensorBoard server on hal-ondemand, detail refers Getting started with HAL OnDemand.

Simple Example with Pytorch

Interactive mode

Get a node for interactive use:

Code Block
swrun -p gpux1

Once on the compute node, load PowerAI module using one of these:

Code Block
module load wmlce/1.6.2
module load wmlce/1.7.0

Install samples for Pytorch:

Code Block
pytorch
Code Block
caffe-install-samples ~/caffepytorch-samples
cd ~/caffepytorch-samples

Download data for MNIST modelTrain on MNIST with Pytorch:

Code Block
python ./dataexamples/mnist/get_mnist.sh

Convert data and create MNIST model

Code Block
./examples/mnist/create_mnist.sh

Training LeNet on MNIST

Code Block
./examples/mnist/train_lenet.sh

Simple Example for TensorFlow

main.py

Batch mode

The same can be accomplished in batch mode using the following pytorch_sample.swb script:

Code Block
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/pytorch_sample.swb
sbatch pytorch_sample.swb
squeue

Major Installed PowerAI Related Anaconda Modules

NameVersionDescription
caffe1.0Caffe is a deep learning framework made with expression, speed, and modularity in mind.
cudatoolkit10.2.89

The NVIDIA® CUDA® Toolkit provides a development environment for creating high-performance GPU-accelerated applications.

cudnn7.6.5+10.2

The NVIDIA CUDA® Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks.

nccl2.5.6The NVIDIA Collective Communications Library (NCCL) implements multi-GPU and multi-node collective communication primitives that are performance-optimized for NVIDIA GPUs.
opencv3.4.8OpenCV was designed for computational efficiency and with a strong focus on real-time applications.
pytorch1.3.1PyTorch enables fast, flexible experimentation and efficient production through a hybrid front-end, distributed training, and ecosystem of tools and libraries.
tensorboard2.1.0To make it easier to understand, debug, and optimize TensorFlow programs, we've included a suite of visualization tools called TensorBoard.
tensorflow-gpu2.1.0The core open-source library to help you develop and train ML models.

...