Because no powerpc builds of ray are available on conda, we must build ray ourselves. However ray builds with gcc 8 (the current version available on HAL) have a bug which throws a segfault when trying to initialize the ray cluster. We must use at least gcc 9 then for the build, however because ray uses bazel which doesn't operate well with non-standard compilers, we must build a customized version of bazel with needed utilities statically linked. Then we can build ray using gcc 9 properly.
1. Create bazel build conda environment
We must first create a conda environment for the build environment for bazel. (I didn't record all needed dependencies, but they will be added upon discovery) for example, openjdk 11 might be needed, but I'm not 100% as I didn't try without it.
conda create python=3.8 -p ./bazel_venv conda activate ./bazel_venv
2. Build customized bazel with statically linked utilities
Download distribution version of bazel 4.2.1 (the version used in ray version 1.9.2 and current development), and apply patch to allow statically linking utilities (attached to this page)
wget https://github.com/bazelbuild/bazel/releases/download/4.2.1/bazel-4.2.1-dist.zip mkdir bazel-4.2.1-dist cd bazel-4.2.1-dist unzip ../bazel-4.2.1-dist.zip patch -i ../bazel-4.2.1-static-utils.patch bash compile.sh
3. Prepare ray build environment
We prepare another conda environment