Because no powerpc builds of ray are available on conda, we must build ray ourselves. However ray builds with gcc 8 (the current version available on HAL) have a bug which throws a segfault when trying to initialize the ray cluster. We must use at least gcc 9 then for the build, however because ray uses bazel which doesn't operate well with non-standard compilers, we must build a customized version of bazel with needed utilities statically linked. Then we can build ray using gcc 9 properly.

1. Create bazel build conda environment

We must first create a conda environment for the build environment for bazel. (I didn't record all needed dependencies, but they will be added upon discovery) for example, openjdk 11 might be needed, but I'm not 100% as I didn't try without it.

conda create python=3.8 -p ./bazel_venv
conda activate ./bazel_venv

2. Build customized bazel with statically linked utilities

Download distribution version of bazel 4.2.1 (the version used in ray version 1.9.2 and current development), and apply patch to allow statically linking utilities (bazel-4.2.1-static-utils.patch)

wget https://github.com/bazelbuild/bazel/releases/download/4.2.1/bazel-4.2.1-dist.zip
mkdir bazel-4.2.1-dist
cd bazel-4.2.1-dist
unzip ../bazel-4.2.1-dist.zip

patch -i ../bazel-4.2.1-static-utils.patch

bash compile.sh

Note: the patch command may not be correct for how I created the patch. This is meant as an approximate guide, so you may have to correct some things to get them to work.

3. Prepare ray build environment

We prepare another conda environment for building ray as we may need different requirements from building bazel.

conda create python=3.8 -p ./ray_build_venv
cp ./bazel-4.2.1-dist/output/bazel ./ray_build_venv/bin

Note: I didn't note exactly what build dependencies I needed, they will be added later if found to be needed.

4. Build Ray

First, we activate gcc 9 and our ray build environment, then we can checkout ray and apply the needed patch for the stdc++fs library. (ray-fs.patch)

conda activate ./ray_build_venv

git clone git@github.com:ray-project/ray
cd ray
git checkout ray-1.9.2

patch -i ../ray-fs.patch

cd python
python setup.py bdist_wheel

The finished wheel will be in the directory python/dist 

Once completed, we can use pip to install the ray-1.9.2 wheel in a python environment of our choice, however we must ensure gcc 9 is properly available in our environment.