NCSA wiki will be offline Friday, Apr 19, 2024, from 1700 hours until Sun, Apr 21, 2024 in order to upgrade Confluence.
Open Cognitive Environment
Welcome to the OpenCE project. The project contains everything that is needed to build conda packages for a collection of machine learning and deep learning frameworks. All packages created for a specific version of OpenCE have been designed to be installed within a single conda environment.
| Environment | opence-v1.3.1 | opence-v1.2.2 | opence-v1.1.2 | opence-v1.0.0 |
|---|---|---|---|---|
| python | 3.8.0 | 3.8.0 | 3.8.12 | 3.8.12 |
| cuda | 11.2.2 | 11.0.221 | 10.2.89 | 10.2.89 |
| cudnn | 8.1.1 | 8.1.1 | 7.6.5 | 7.6.5 |
| nccl | 2.8.3 | 2.7.8 | 2.7.8 | 2.7.8 |
| openmpi | 4.1.1 | 3.1.3 | 3.1.3 | 3.1.3 |
| apex | 0.1 | N/A | N/A | N/A |
| hdf5 | 1.10.4 | 1.10.6 | 1.10.4 | 1.10.4 |
| horovod | 0.21.3 | 0.21.0 | 0.21.0 | 0.19.5 |
| ipython | 7.27.0 | 7.29.0 | 7.29.0 | 7.28.0 |
| matplotlib | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.2 |
| mpi4py | 3.1.1 | N/A | N/A | N/A |
| numpy | 1.21.2 | 1.19.5 | 1.19.5 | 1.19.2 |
| onnx | 1.7.0 | 1.6.0 | 1.6.0 | 1.6.0 |
| opencv | 3.4.14 | 4.5.0 | 3.4.10 | 3.4.10 |
| pandas | 1.3.2 | 1.3.4 | 1.3.4 | 1.2.4 |
| pytorch | 1.8.1 | 1.7.1 | 1.7.1 | 1.6.0 |
| scikit-learn | 0.24.2 | 1.0.1 | 1.0.1 | 1.0.1 |
| scipy | 1.7.1 | 1.7.1 | 1.4.1 | 1.4.1 |
| tensorflow | 2.5.1 | 2.4.1 | 2.4.1 | 2.3.1 |
| tensorboard | 2.5.0 | 2.4.1 | 2.4.1 | 2.3.0 |
| transformers | 4.4.2 | 2.1.1 | 4.12.2 | 4.12.2 |
Simple Example with TensorFlow
Interactive mode
Get a node for interactive use:
swrun -p gpux1
Once on the compute node, load PowerAI module using one of these:
module load opence module load opence-v1.3.1
Copy the following code into file "mnist-demo.py":
import tensorflow as tf
mnist = tf.keras.datasets.mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(512, activation=tf.nn.relu),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)
Train on MNIST with keras API:
python ./mnist-demo.py
Batch mode
The same can be accomplished in batch mode using the following tf_sample.swb script:
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/tf_sample.swb sbatch tf_sample.swb squeue
Visualization with TensorBoard
Interactive mode
Get a node for interactive use:
swrun -p gpux1
Once on the compute node, load PowerAI module using one of these:
module load opence module load opence-v1.3.1
Download the code mnist-with-summaries.py to $HOME folder:
cd ~ wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/mnist-with-summaries.py
Train on MNIST with TensorFlow summary:
python ./mnist-with-summaries.py
Batch mode
The same can be accomplished in batch mode using the following tfbd_sample.swb script:
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/tfbd_sample.swb sbatch tfbd_sample.swb squeue
Start the TensorBorad session
After job completed the TensorFlow log files can be found in "~/tensorflow/mnist/logs", start the TensorBoard server on hal-ondemand, detail refers Getting started with HAL OnDemand.
Simple Example with Pytorch
Interactive mode
Get a node for interactive use:
swrun -p gpux1
Once on the compute node, load PowerAI module using one of these:
module load opence module load opence-v1.3.1
Install samples for Pytorch:
pytorch-install-samples ~/pytorch-samples cd ~/pytorch-samples
Train on MNIST with Pytorch:
python ./examples/mnist/main.py