Table of Contents |
---|
Open Cognitive Environment
Welcome to the OpenCE project. The project contains everything that is needed to build conda packages for a collection of machine learning and deep learning frameworks. All packages created for a specific version of OpenCE have been designed to be installed within a single conda environment.
Environment | opence-v1.3.1 | opence-v1.2.2 | opence-v1.1.2 | opence-v1.0.0 |
---|---|---|---|---|
python | 3.8.0 | 3.8.0 | 3.8.12 | 3.8.12 |
cuda | 11.2.2 | 11.0.221 | 10.2.89 | 10.2.89 |
cudnn | 8.1.1 | 8.1.1 | 7.6.5 | 7.6.5 |
nccl | 2.8.3 | 2.7.8 | 2.7.8 | 2.7.8 |
openmpi | 4.1.1 | 3.1.3 | 3.1.3 | 3.1.3 |
apex | 0.1 | N/A | N/A | N/A |
hdf5 | 1.10.4 | 1.10.6 | 1.10.4 | 1.10.4 |
horovod | 0.21.3 | 0.21.0 | 0.21.0 | 0.19.5 |
ipython | 7.27.0 | 7.29.0 | 7.29.0 | 7.28.0 |
matplotlib | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.2 |
mpi4py | 3.1.1 | N/A | N/A | N/A |
numpy | 1.21.2 | 1.19.5 | 1.19.5 | 1.19.2 |
onnx | 1.7.0 | 1.6.0 | 1.6.0 | 1.6.0 |
opencv | 3.4.14 | 4.5.0 | 3.4.10 | 3.4.10 |
pandas | 1.3.2 | 1.3.4 | 1.3.4 | 1.2.4 |
pytorch | 1.8.1 | 1.7.1 | 1.7.1 | 1.6.0 |
scikit-learn | 0.24.2 | 1.0.1 | 1.0.1 | 1.0.1 |
scipy | 1.7.1 | 1.7.1 | 1.4.1 | 1.4.1 |
tensorflow | 2.5.1 | 2.4.1 | 2.4.1 | 2.3.1 |
tensorboard | 2.5.0 | 2.4.1 | 2.4.1 | 2.3.0 |
transformers | 4.4.2 | 2.1.1 | 4.12.2 | 4.12.2 |
Simple Example with TensorFlow
Interactive mode
Get a node for interactive use:
Code Block |
---|
swrun -p gpux1 |
Once on the compute node, load PowerAI module using one of these:
Code Block |
---|
module load opence
module load opence-v1.3.1 |
Copy the following code into file "mnist-demo.py":
Code Block |
---|
import tensorflow as tf
mnist = tf.keras.datasets.mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(512, activation=tf.nn.relu),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test) |
Train on MNIST with keras API:
Code Block |
---|
python ./mnist-demo.py |
Batch mode
The same can be accomplished in batch mode using the following tf_sample.swb script:
Code Block |
---|
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/tf_sample.swb
sbatch tf_sample.swb
squeue |
Visualization with TensorBoard
Interactive mode
Get a node for interactive use:
Code Block |
---|
swrun -p gpux1 |
Once on the compute node, load PowerAI module using one of these:
Code Block |
---|
module load wmlce/1.6.2
module load wmlce/1.7.0 |
Download the code mnist-with-summaries.py to $HOME folder:
Code Block |
---|
cd ~
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/mnist-with-summaries.py |
Train on MNIST with TensorFlow summary:
Code Block |
---|
python ./mnist-with-summaries.py |
Batch mode
The same can be accomplished in batch mode using the following tfbd_sample.swb script:
Code Block |
---|
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/tfbd_sample.swb
sbatch tfbd_sample.swb
squeue |
Start the TensorBorad session
After job completed the TensorFlow log files can be found in "~/tensorflow/mnist/logs", start the TensorBoard server on hal-ondemand, detail refers Getting started with HAL OnDemand.
Simple Example with Pytorch
Interactive mode
Get a node for interactive use:
Code Block |
---|
swrun -p gpux1 |
Once on the compute node, load PowerAI module using one of these:
Code Block |
---|
module load wmlce/1.6.2
module load wmlce/1.7.0 |
Install samples for Pytorch:
Code Block |
---|
pytorch-install-samples ~/pytorch-samples
cd ~/pytorch-samples |
Train on MNIST with Pytorch:
Code Block |
---|
python ./examples/mnist/main.py |