Installation on specific platforms
Installation on specific platforms¶
The following describes installation details for various systems and platforms that SmartSim may be used on.
Customizing environment variables¶
Various environment variables can be used to control the compilers and
dependencies for SmartSim. These are particularly important to set before the
smart build step to ensure that the Orchestrator and machine-learning
backends are compiled with the desired compilation environment.
The compilation environment that SmartSim is compiled with does not
necessarily have to be compatible with the SmartRedis library and the
simulation application that will be launched by SmartSim. To ensure
that this works as intended however, please be sure to set the
correct environment for the simulation using the
All of the following environment variables must be exported to ensure that
they are used throughout the entire build process. Additionally at runtime, the
environment in which the Orchestrator is launched must have the cuDNN and CUDA
Toolkit libraries findable by the link loader (e.g. available in the
LD_LIBRARY_PATH environment variable).
Unlike SmartRedis, we strongly encourage users to only use the GNU compiler chain to build the SmartSim dependencies. Notably, RedisAI has some coding conventions that prevent the use of Intel compiler chain. If a specific compiler should be used (e.g. the Cray Programming Environment wrappers), the following environment variables will control the C and C++ compilers:
CC: Path to the C compiler
CXX: Path the C++ compiler
GPU dependencies (non-root)¶
The Nvidia installation instructions for CUDA Toolkit and cuDNN tend to be tailored for users with root access. For those on HPC platforms where root access is rare, manually downloading and installing these dependencies as a user is possible.
wget https://developer.download.nvidia.com/compute/cuda/11.4.4/local_installers/cuda_11.4.4_470.82.01_linux.run chmod +x cuda_11.4.4_470.82.01_linux.run ./cuda_11.4.4_470.82.01_linux.run --toolkit --silent --toolkitpath=/path/to/install/location/
For cuDNN, follow Nvidia’s instructions, and copy the cuDNN libraries to the lib64 directory at the CUDA Toolkit location specified above.
HPE Cray supercomputers¶
On certain HPE Cray machines, the SmartSim dependencies have been installed system-wide though specific paths and names might vary (please contact the team if these instructions do not work).
module use -a /lus/scratch/smartsim/local/modulefiles module load cudatoolkit/11.4 cudnn git-lfs module unload PrgEnv-cray PrgEnv-intel PrgEnv-gcc module load PrgEnv-gnu module switch gcc/11.2.0 export CRAYPE_LINK_TYPE=dynamic
This should provide all the dependencies needed to build the GPU backends for the ML bakcends. Users can thus proceed with their preferred way of installing SmartSim either from PyPI or from source.
Cheyenne at NCAR¶
Since SmartSim does not currently support the Message Passing Toolkit (MPT), Cheyenne users of SmartSim will need to utilize OpenMPI.
The following module commands were utilized to run the examples:
$ module purge $ module load ncarenv/1.3 gnu/8.3.0 ncarcompilers/0.5.0 netcdf/4.7.4 openmpi/4.0.5
With this environment loaded, users will need to build and install both SmartSim and SmartRedis through pip. Usually we recommend users installing or loading miniconda and using the pip that comes with that installation.
$ pip install smartsim $ smart build --device cpu #(Since Cheyenne does not have GPUs)
To make the SmartRedis library (C, C++, Fortran clients), follow these steps with the same environment loaded.
# clone SmartRedis and build $ git clone https://github.com/SmartRedis.git smartredis $ cd smartredis $ make lib
Summit at OLCF¶
Since SmartSim does not have a built PowerPC build, the build steps for an IBM system are slightly different than other systems.
Luckily for us, a conda channel with all relevant packages is maintained as part of the OpenCE initiative. Users can follow these instructions to get a working SmartSim build with PyTorch and TensorFlow for GPU on Summit. Note that SmartSim and SmartRedis will be downloaded to the working directory from which these instructions are executed.
# setup Python and build environment export ENV_NAME=smartsim-0.4.2 git clone https://github.com/CrayLabs/SmartRedis.git smartredis git clone https://github.com/CrayLabs/SmartSim.git smartsim conda config --prepend channels https://ftp.osuosl.org/pub/open-ce/1.4.1/ conda create --name $ENV_NAME -y python=3.9 \ git-lfs \ cmake \ make \ cudnn=8.1.1_11.2 \ cudatoolkit=11.2.2 \ tensorflow=2.6.2 \ libtensorflow=2.6.2 \ pytorch=1.9.0 \ torchvision=0.10.0 conda activate $ENV_NAME export CC=$(which gcc) export CXX=$(which g++) export LDFLAGS="$LDFLAGS -pthread" export CUDNN_LIBRARY=/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/ export CUDNN_INCLUDE_DIR=/ccs/home/$USER/.conda/envs/$ENV_NAME/include/ module load cuda/11.4.2 export LD_LIBRARY_PATH=$CUDNN_LIBRARY:$LD_LIBRARY_PATH:/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/python3.9/site-packages/torch/lib module load gcc/9.3.0 module unload xalt # clone SmartRedis and build pushd smartredis make lib && pip install . popd # clone SmartSim and build pushd smartsim pip install . # install PyTorch and TensorFlow backend for the Orchestrator database. export Torch_DIR=/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/python3.9/site-packages/torch/share/cmake/Torch/ export CFLAGS="$CFLAGS -I/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/python3.9/site-packages/tensorflow/include" export SMARTSIM_REDISAI=1.2.5 export Tensorflow_BUILD_DIR=/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/python3.9/site-packages/tensorflow/ smart build --device=gpu --torch_dir $Torch_DIR --libtensorflow_dir $Tensorflow_BUILD_DIR -v # Show LD_LIBRARY_PATH for future reference echo "SmartSim installation is complete, LD_LIBRARY_PATH=$LD_LIBRARY_PATH"
When executing SmartSim, if you want to use the PyTorch and TensorFlow backends in the orchestrator, you will need to set up the same environment used at build time:
module load cuda/11.4.2 export CUDNN_LIBRARY=/ccs/home/$USER/.conda/envs/$ENV_NAME/lib/ export LD_LIBRARY_PATH=/ccs/home/$USER/.conda/envs/smartsim/lib/python3.8/site-packages/torch/lib/:$LD_LIBRARY_PATH:$CUDNN_LIBRARY module load gcc/9.3.0 module unload xalt
Certain HPE customer machines have a site installation of SmartSim. This means
that users can bypass the
smart build step that builds the ML backends and
the Redis binaries. Users on these platforms can install SmartSim from PyPI or
from source with the following steps replacing
SMARTSIM_VERSION with the desired entries.
module use -a /lus/scratch/smartsim/local/modulefiles module load cudatoolkit/11.4 cudnn smartsim-deps/COMPILER_VERSION/SMARTSIM_VERSION pip install smartsim[ml] smart build --only_python_packages --device gpu [--onnx]