Anaconda Environment
Background
Anaconda Miniconda is a free and open-source distribution of the Python and R programming languages for scientific computing. The big difference between Conda and the pip package manager is in how the package dependencies are managed, which is a significant challenge for Python data science and the reason Conda exists.(see webpage https://www.anaconda.comfor details)
...
minimal installer for conda. It is a small, bootstrap version of Anaconda that includes only conda, Python, the packages they depend on, and a small number of other useful packages, including pip, zlib and a few others. Use the conda install
command to install 720+ additional conda packages from the Anaconda repository.
Challenges and Solutions
How to install a specific package?
Users can not and should not install packages in existing python environments.
Users need to create their own python environment to install their own packages.
Users could install a specific python package to avoid conflict or add functionality.
How to solve dependency conflict?
The conda has the capability to solve some dependency conflicts, but it cost time and is not guaranteed to work.
Users could create a python environment from scratch, then
Install the package which caused the conflict first.
Install other packages according to the conflicting package requirement.
How to use Customized Python Environment in HAL-OnDemand?
- Users need to install
conda install ipykernel
in order to select and/or switch their own conda environment in HAL-OnDemand.
Lists of Existing Anaconda Environments
There are currently 6 Conda environments supported on the HAL system
Environment Name | Location | Description |
---|
base | /opt/ |
apps/anaconda3 Conda conda env with basic python packages |
, python version 3.7.. |
jupyter | /opt/miniconda3/envs/jupyter | Default conda env with jupyter server. Used for hal-ondemand service only. |
deepspeed-v0.3.16 | /opt/miniconda3/envs/deepspeed-v0.3.16 | DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective. |
fastai-v0.1.18 | /opt/miniconda3/envs/fastai-v0.1.18 | fastai is a deep learning library that provides practitioners with high-level components that can quickly and easily provide state-of-the-art results in standard deep learning domains. |
wmlce-v1.6.2 | /opt/ |
appsanaconda3/.1-py2.7IBM WMLCE-v1.6.2 (former PowerAI) package, python version 2.7. | wmlce-v1.7.0 | /opt/apps/anaconda3/envs/wmlce-v1.7.0-py3.7 | IBM WMLCE-v1.7.0 (former PowerAI) package, python version 3.7. |
opence-v1.0.0 | opence-v1.1.2 | opence-v1.2.2 | opence-v1.3.1 | rapids | ...
.2 | Watson Machine Learning Community Edition is an IBM Cognitive Systems offering that is designed for the rapidly growing and quickly evolving AI category of deep learning. This is the only environment still have tensorflow-v1.15 |
wmlce-v1.7.0 | /opt/miniconda3/envs/wmlce-v1.7.0 | Watson Machine Learning Community Edition is an IBM Cognitive Systems offering that is designed for the rapidly growing and quickly evolving AI category of deep learning. This is the last version of WMLCE, the following module is opence. |
opence-v1.0.0 | /opt/miniconda3/envs/opence-v1.0.0 | Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
opence-v1.1.2 | /opt/miniconda3/envs/opence-v1.1.2 | Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
opence-v1.2.2 | /opt/miniconda3/envs/opence-v1.2.2 | Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
opence-v1.3.1 | /opt/miniconda3/envs/opence-v1.3.1 | Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
opence-v1.4.1 | /opt/miniconda3/envs/opence-v1.4.1 | Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
opence-v1.5.1
opence-v1.5.2 | /opt/miniconda3/envs/opence-v1.5.1
/opt/miniconda3/envs/opence-v1.5.2 | Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. This is the default python environment. Use opence-v1.5.1 as base, fixed openblas issue caused by openmp. |
theano-v1.0.4 | /opt/miniconda3/envs/theano-v1.0.4 | Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
rapids-v0.11.0 | /opt/miniconda3/envs/rapids | The RAPIDS suite of software libraries, built on CUDA-X AI, gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. |
Create a New Env from Existing Environments
We recommend our users to create a new environment from one of our existing powerai or wmlce opence environment.
Code Block |
---|
language | bash |
---|
title | Create a New Env from Existing Env |
---|
|
conda create --name=<newnew_env>env --clone=wmlceopence-v1.65.1-py3.6 |
The new Conda environment will be located within $HOME/.conda/envs/<newnew_env>env, then users can search and/or install python packages via Conda
Code Block |
---|
language | bash |
---|
title | Example: Search for a New Package |
---|
|
conda search r-tensorflow |
Code Block |
---|
language | bash |
---|
title | Example: Install a New Package |
---|
|
conda install r-tensorflow |
Create
...
Conda Environment from Scratch
Users can also create a new environment from scratch
Code Block |
---|
language | bash |
---|
title | Create a New Env from Existing Env |
---|
|
conda create --name=<new_env_name> |
Important Note about Install package with pip
...
Search Packages in All Default Channels
Code Block |
---|
conda search openblas |
Search Packages in a Specific Channel
Code Block |
---|
conda search openblas -c conda-forge |
Install a Package with a Specific Version and Build Code
Anchor |
---|
| install_package |
---|
| install_package |
---|
|
Install a package from the result of the search
Code Block |
---|
(new_env_name)[user_id@hal-login2 ~]# conda search openblas
...
openblas 0.3.12 pthreads_hca0ad1f_0 conda-forge
openblas 0.3.12 pthreads_hca0ad1f_1 conda-forge
openblas 0.3.13 h6ffa863_0 pkgs/main
openblas 0.3.13 h6ffa863_1 pkgs/main
openblas 0.3.13 openmp_h25a920f_0 conda-forge
openblas 0.3.13 pthreads_h92053e5_0 conda-forge
...
(new_env_name)[user_id@hal-login2 ~]# conda install openblas=0.3.13=openmp_h25a920f_0 |