Because no powerpc builds of ray are available on conda, we must build ray ourselves. However ray builds with gcc 8 (the current version available on HAL) have a bug which throws a segfault when trying to initialize the ray cluster. We must use at least gcc 9 then for the build, however because ray uses bazel which doesn't operate well with non-standard compilers, we must build a customized version of bazel with needed utilities statically linked. Then we can build ray using gcc 9 properly.
1. Create bazel build conda environment
We must first create a conda environment for the build environment for bazel. (I didn't record all needed dependencies, but they will be added upon discovery) for example, openjdk 11 might be needed, but I'm not 100% as I didn't try without it.
conda create python=3.8 -p ./bazel_venv conda activate ./bazel_venv
2. Build customized bazel with statically linked utilities
Download distribution version of bazel 4.2.1 (the version used in ray version 1.9.2 and current development), and apply patch to allow statically linking utilities (bazel-4.2.1-static-utils.patch)
wget https://github.com/bazelbuild/bazel/releases/download/4.2.1/bazel-4.2.1-dist.zip mkdir bazel-4.2.1-dist cd bazel-4.2.1-dist unzip ../bazel-4.2.1-dist.zip patch -i ../bazel-4.2.1-static-utils.patch bash compile.sh
Note: the patch command may not be correct for how I created the patch. This is meant as an approximate guide, so you may have to correct some things to get them to work.
3. Prepare ray build environment
We prepare another conda environment for building ray as we may need different requirements from building bazel.
conda create python=3.8 -p ./ray_build_venv cp ./bazel-4.2.1-dist/output/bazel ./ray_build_venv/bin
Note: I didn't note exactly what build dependencies I needed, they will be added later if found to be needed.
4. Build Ray
First, we activate gcc 9 and our ray build environment, then we can checkout ray and apply the needed patch for the stdc++fs library. (ray-fs.patch)
conda activate ./ray_build_venv git clone git@github.com:ray-project/ray cd ray git checkout ray-1.9.2 patch -i ../ray-fs.patch cd python python setup.py bdist_wheel
The finished wheel will be in the directory python/dist
Once completed, we can use pip to install the ray-1.9.2 wheel in a python environment of our choice, however we must ensure gcc 9 is properly available in our environment.