Background
Anaconda is a free and open-source distribution of the Python and R programming languages for scientific computing. The big difference between Conda and the pip package manager is how the package dependencies are managed, which is a significant challenge for Python data science and the reason Conda exists.
(see webpage https://www.anaconda.com for details)
Challenges
- How to install a specific package?
- Users can not and should not install packages in existing python environments such as opence-v1.5.1.
- Users need to create their own python environment to install their own packages.
- Users should search for all the available packages before installation.
- How to solve dependency conflict?
Existing Anaconda Environment
There are currently 6 Conda environments supported on the HAL system
| Environment Name | Location | Description |
|---|---|---|
| base | /opt/apps/anaconda3 | Default Conda env with basic python packages. |
deepspeed-v0.3.16 | /opt/miniconda3/envs/deepspeed-v0.3.16 | DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective. |
fastai-v0.1.18 | /opt/miniconda3/envs/fastai-v0.1.18 | fastai is a deep learning library that provides practitioners with high-level components that can quickly and easily provide state-of-the-art results in standard deep learning domains and provides researchers with low-level components that can be mixed and matched to build new approaches. |
| wmlce-v1.6.2 | /opt/anaconda3/envs/wmlce-v1.6.2 | Watson Machine Learning Community Edition is an IBM Cognitive Systems offering that is designed for the rapidly growing and quickly evolving AI category of deep learning. |
| wmlce-v1.7.0 | /opt/anaconda3/envs/wmlce-v1.7.0 | Watson Machine Learning Community Edition is an IBM Cognitive Systems offering that is designed for the rapidly growing and quickly evolving AI category of deep learning. |
| opence-v1.0.0 | /opt/miniconda3/envs/opence-v1.0.0 | Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
| opence-v1.1.2 | /opt/miniconda3/envs/opence-v1.1.2 | Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
| opence-v1.2.2 | /opt/miniconda3/envs/opence-v1.2.2 | Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
| opence-v1.3.1 | /opt/miniconda3/envs/opence-v1.3.1 | Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
| rapids | /opt/miniconda3/envs/rapids | The RAPIDS suite of software libraries, built on CUDA-X AI, gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. |
Create a New Env from Existing Environments
We recommend our users to create a new environment from one of our existing opence environment.
conda create --name=<new_env> --clone=opence-v1.5.1
The new Conda environment will be located within $HOME/.conda/envs/<new_env>, then users can search and/or install python packages via Conda
conda search r-tensorflow
Create Conda Environment from Scratch
Users can also create a new environment from scratch
conda create --name=<new_env_name>
Search Packages in All Default Channels
conda search openblas
Search Packages in a Specific Channel
conda search openblas -c conda-forge
openblas 0.3.13 h6ffa863_1 pkgs/main openblas 0.3.13 openmp_h25a920f_0 conda-forge openblas 0.3.13 pthreads_h92053e5_0 conda-forge
Note: If you want to use your own conda env in hal-ondemand, you need to install conda install ipykernel .