Installation

The following will show how to install both SmartSim and SmartRedis

Quick Start

If you want to get started with SmartSim quickly, you can install it through pip. We highly recommend starting a fresh Python environment.

$ pip install smartsim
$ smart --device cpu

This will install SmartSim with PyTorch and TensorFlow backends for the Orchestrator database as well as the SmartRedis Python client.

Supported Versions

Supported System for Pre-built Wheels

Platform

CPU

GPU

Python Versions

MacOS

x86_64

not supported

3.7 - 3.9

Linux

x64_64

Nvidia

3.7 - 3.9

Note

Windows is not supported and there are currently no plans to support Windows.

SmartSim supports multiple machine learning libraries through the use of RedisAI. The following libraries are supported.

Supported ML Libraries

Library

Versions

Python Versions

Built By Default

PyTorch

1.7

3.7 - 3.9

Yes

Tensorflow / Keras

2.4.2

3.7 - 3.8

Yes

ONNX

1.6 - 1.7

3.7 - 3.9

No

TensorFlow 2.0 and Keras are supported through graph freezing.

ScikitLearn and Spark models are supported by SmartSim as well through the use of the ONNX runtime.


Prerequisites

The prerequisites to install SmartSim and SmartRedis are:

  • Python 3.7.x (or later) and pip

  • CMake 3.10.x (or later)

  • GCC > 5

  • GNU Make > 4.0

  • git-lfs

For most developer systems, many of these packages will already be installed.

Git LFS can be installed through conda install git-lfs && git lfs install

Be sure to reference the installation notes for helpful information regarding various system types before installation.

SmartSim

Activate a new virtual environment and install SmartSim from PyPi with the following command

pip install smartsim

If you would like SmartSim to also install Machine Learning libraries that can be used outside SmartSim to build SmartSim-compatible models, you can request their installation through the ml flag as follows:

pip install smartsim[ml]

At this point, SmartSim is installed and can be used for more basic features. If you want to use the machine learning features of SmartSim, you will need to install the ML backends in the section below.

Install ML Backends

Use the smart cli tool to install the machine learning backends that are built into the Orchestrator database. smart is installed during the pip installation of SmartSim and may only be available while your virtual environment is active.

To see all the installation options:

smart

To install the database ML backends for CPU, run

# run one of the following
smart --device cpu          # install PT and TF for cpu
smart --device cpu --onnx   # install all backends (PT, TF, ONNX) on gpu

By default, smart will install PyTorch and TensorFlow backends for use in SmartSim.

Note

If a re-build is needed for any reason, smart --clean will remove all of the previous installs for the ML backends and smart --clobber will remove all pre-built dependencies as well as the ML backends.

To install the database ML backends for GPU, set the following environment variables if CUDNN is not in your LD_LIBRARY_PATH or default loader locations.

  • CUDNN_INCLUDE_DIR - path to directory containing cudnn.h

  • CUDNN_LIBRARY - path to directory containing libcudnn.so

For example, for bash do

export CUDNN_LIBRARY=/lus/sonexion/spartee/cuda/lib64/
export CUDNN_INCLUDE_DIR=/lus/sonexion/spartee/cuda/include/
export LD_LIBRARY_PATH=$CUDNN_LIBRARY:$LD_LIBRARY_PATH
# run one of the following
smart --device gpu          # install PT and TF for gpu
smart --device gpu --onnx   # install all backends (PT, TF, ONNX) on gpu

Note

Currently, SmartSim is solely compatible with NVIDIA GPUs on Linux systems and CUDA >= 11 is required to build.

Note

If the smart tool is not found. Look for it in places like ~/.local/bin and other bin locations and add it to your $PATH


SmartRedis

There are implementations of the SmartRedis client in 4 languages: Python, C++, C and Fortran. The Python client is installed through pip and the compiled clients can be built as a static or shared library through cmake.

SmartRedis Python supports the same architectures for pre-built wheels that SmartSim does.

Supported Systems for Pre-built Wheels

Platform

Python Versions

MacOS

3.7 - 3.9

Linux

3.7 - 3.9

The Python client for SmartRedis is installed through pip as follows:

First, activate your Python virtual environment:

conda activate <env name>

Install SmartRedis Python client from PyPI:

pip install smartredis

Developers installing the Python client from PyPI can install the SmartRedis client with additional dependencies for testing and documentation with:

pip install smartredis[dev]
# or if using ZSH
pip install smartredis\[dev\]

Now, when inside your virtual environment, you should be able to import the Client from smartredis as follows

Python 3.7.7 (default, May  7 2020, 21:25:33)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from smartredis import Client
>>>

Build SmartRedis Library (C++, C, Fortran)

Clone the SmartRedis repository and checkout the most recent release:

git clone https://github.com/CrayLabs/SmartRedis.git --depth=1 --branch v0.2.0 smartredis

Note that the release tarball can also be used instead of cloning the git repository, but the preferred method is a repository clone.

To build the SmartRedis library for the C++, C, and Fortran, make sure to be in the top level directory of smartredis-0.2.0.

make lib

The SmartRedis library will be installed in smartredis-0.2.0/install/lib/ and the SmartRedis header files will be installed in smartredis-0.2.0/install/include/. The library installation can be used to easily include SmartRedis capabilities in C++, C, and Fortran applications. For example, the CMake instructions below illustrate how to compile a C or C++ application with SmartRedis.

project(Example)

cmake_minimum_required(VERSION 3.10)

set(CMAKE_CXX_STANDARD 17)

find_library(sr_lib smartredis
             PATHS path/to/smartredis/install/lib
             NO_DEFAULT_PATH REQUIRED
)

include_directories(SYSTEM
    /usr/local/include
    path/to/smartredis/install/include
)

# Build executables

add_executable(example
    example.cpp
)
target_link_libraries(example
    ${sr_lib}
)

Compiling a Fortran application with the SmartRedis library is very similar to the instructions above. The only difference is that the Fortran SmartRedis client source files currently need to be included in the compilation. An example CMake file is shown below for a Fortran application.

project(Example)

cmake_minimum_required(VERSION 3.10)

enable_language(Fortran)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_C_STANDARD 99)

set(ftn_client_src
    path/to/smartredis/src/fortran/fortran_c_interop.F90
    path/to/smartredis/src/fortran/dataset.F90
    path/to/smartredis/src/fortran/client.F90
)

find_library(sr_lib smartredis
             PATHS path/to/smartredis/install/lib
             NO_DEFAULT_PATH REQUIRED
)

include_directories(SYSTEM
    /usr/local/include
    path/to/smartredis/install/include
)

add_executable(example
    example.F90
        ${ftn_client_src}
)

target_link_libraries(example
    ${sr_lib}
)

From Source

This section will be geared towards contributors who want to install SmartSim and SmartRedis from source. If you are installing from source for other reasons, follow the steps below but use the distribution provided hosted on GitHub or PyPi.

Install SmartSim from Source

First, clone SmartSim.

git clone https://github.com/CrayLabs/SmartSim smartsim

And then install SmartSim with pip in editable mode. This way, SmartSim is installed in your virtual environment and available in PYTHONPATH, but the source remains at the site of the clone instead of in site-packages.

cd smartsim
pip install -e .[dev,ml]   # for bash users
pip install -e .\[dev,ml\] # for zsh users

Use the now installed smart cli to install the machine learning runtimes.

# run one of the following
smart -v --device cpu          # verbose install cpu
smart -v --device gpu          # verbose install gpu
smart -v --device gpu --onnx   # install all backends (PT, TF, ONNX) on gpu

Install SmartRedis from Source

Developer Makefile

SmartRedis has a Makefile that automates the build and install process. The Makefile is shown below, and in the following sections, the process for building and installing the SmartRedis clients from source will be described.

SmartRedis Makefile help

help                           - display this makefile's help information

Build
-------
deps                           - Make SmartRedis dependencies
lib                            - Build SmartRedis clients into a static library
test-deps                      - Make SmartRedis testing dependencies
test-deps-gpu                  - Make SmartRedis GPU testing dependencies
build-tests                    - build all tests (C, C++, Fortran)
build-test-cpp                 - build the C++ tests
build-test-c                   - build the C tests
build-test-fortran             - build the Fortran tests
build-examples                 - build all examples (serial, parallel)
build-example-serial           - buld serial examples
build-example-parallel         - build parallel examples (requires MPI)
clean-deps                     - remove third-party deps
clean                          - remove builds, pyc files, .gitignore rules
clobber                        - clean, remove deps, builds, (be careful)

Style
-------
style                          - Sort imports and format with black
check-style                    - check code style compliance
format                         - perform code style format
check-format                   - check code format compliance
sort-imports                   - apply import sort ordering
check-sort-imports             - check imports are sorted
check-lint                     - run static analysis checks

Documentation
-------
docs                           - generate project documentation
cov                            - generate html coverage report for Python client

Test
-------
test                           - Build and run all tests (C, C++, Fortran, Python)
test-verbose                   - Build and run all tests [verbosely]
test-c                         - Build and run all C tests
test-cpp                       - Build and run all C++ tests
test-py                        - run python tests
test-fortran                   - run fortran tests
testpy-cov                     - run python tests with coverage

Clone SmartRedis

First, clone the SmartRedis repo:

git clone https://github.com/CrayLabs/SmartRedis smartredis
cd smartredis

Building the Python Client from Source

After cloning the repository, the Python client can be installed from source with:

pip install .

If installing SmartRedis from source for development, it is recommended that the Python client be installed with the -e and [dev]:

pip install -e .[dev]
# or if using ZSH
pip install -e .\[dev\]

Now, when inside your virtual environment, you should be able to import the Client from smartredis as follows

Python 3.7.7 (default, May  7 2020, 21:25:33)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from smartredis import Client
>>>

Building SmartRedis Static Library from Source

Assuming the above steps have already been done, you are now ready to build SmartRedis as a static library.

A static library of the SmartRedis C++, C, and Fortran clients can be built with the command:

make lib

The SmartRedis library will be installed in smartredis/install/lib/ and the SmartRedis header files will be installed in smartredis/install/include/. The library installation can be used to easily include SmartRedis capabilities in C++, C, and Fortran applications. For example, the CMake instructions below illustrate how to compile a C or C++ application with SmartRedis.

project(Example)

cmake_minimum_required(VERSION 3.10)

set(CMAKE_CXX_STANDARD 17)

find_library(sr_lib smartredis
             PATHS path/to/smartredis/install/lib
             NO_DEFAULT_PATH REQUIRED
)

include_directories(SYSTEM
    /usr/local/include
    path/to/smartredis/install/include
)

# Build executables

add_executable(example
    example.cpp
)
target_link_libraries(example
    ${sr_lib}
)

Compiling a Fortran application with the SmartRedis library is very similar to the instructions above. The only difference is that the Fortran SmartRedis client source files currently need to be included in the compilation. An example CMake file is shown below for a Fortran application.

project(Example)

cmake_minimum_required(VERSION 3.10)

enable_language(Fortran)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_C_STANDARD 99)

set(ftn_client_src
    path/to/smartredis/src/fortran/fortran_c_interop.F90
    path/to/smartredis/src/fortran/dataset.F90
    path/to/smartredis/src/fortran/client.F90
)

find_library(sr_lib smartredis
             PATHS path/to/smartredis/install/lib
             NO_DEFAULT_PATH REQUIRED
)

include_directories(SYSTEM
    /usr/local/include
    path/to/smartredis/install/include
)

add_executable(example
    example.F90
        ${ftn_client_src}
)

target_link_libraries(example
    ${sr_lib}
)

Building the Documentation

Note

To build the full documentation, users need to install doxygen 1.9.1. For Mac OS users, doxygen can be installed through brew install doxygen

Users can optionally build documentation of SmartSim

git clone https://github.com/CrayLabs/SmartRedis.git
cd /smartsim      # top level smartsim dir
make docs

Once the documentation has successfully built, users can open the main documents page from doc/_build/html/index.html

Installation Notes for Specific System Types

The following describes installation details for various system types that SmartSim may be used on.

SmartSim on MacOS

We recommend users and contributors install brew for managing installed packages. For contributors, the following brew packages can be helpful

  • openmpi for building and running parallel SmartRedis examples

  • doxygen for building the documention

  • cmake for building SmartSim and SmartRedis from source

For Mac OS users, the version of make that comes with the Mac command line tools is often 3.81 which needs to be updated to install SmartSim. Users can brew install make to get make > 4.0 but brew will install it as gmake. An easy way around this is to do alias make=gmake.

SmartSim on Ubuntu or Linux Workstations

When building SmartSim for Linux systems where the user has root access, many of the needed packages can be installed through apt.

If you have a CUDA enabled GPU and want to run SmartSim on GPU on a Linux system you have root access to, you can install CUDA through apt with the cuda package.

In addition, cuDNN can be installed through the libcudnn and libcudnn-dev pacakges.

If you run into trouble compiling the machine learning runtimes on Ubuntu because of cuDNN, it might be due to this issue

SmartSim on Cray XC, CS, and EX

If on a Cray system, be sure to set the correct toolchain. SmartSim is tested on PrgEnv-GNU and PrgEnv-Cray modules.

Note

If on a Cray, please note that the intel and PGI compiler toolchains are currently not supported by SmartSim.

Before installing the machine learning runtimes with the smart cli tool, be sure to set the CRAYPE_LINK_TYPE to dynamic

export CRAYPE_LINK_TYPE=dynamic

Keep in mind, the libraries installed above need to be accessible by SmartSim at runtime. If using a networked file system (NFS), make sure to install these somewhere reachable from head, MOM, and compute nodes (network mounted).

CUDA and CUDNN on Cray

Usually cudatoolkit is available as a module and CUDA is installed in /usr/local/cuda. In this case, prior to installation run

module load cudatoolkit

If cuDNN libraries and includes are not installed as a module or otherwise, you can install them with conda.

# Install CUDA requirements
conda install cudatoolkit cudnn
export CUDNN_LIBRARY=/path/to/miniconda/pkgs/cudnn-x.x.x-cudax.x_x/lib
export CUDNN_INCLUDE_DIR=/path/to/miniconda/pkgs/cudnn-x.x.x-cudax.x_x/include

Be sure to get the cuDNN version that matches your CUDA installation The package names usually specify the versions.

SmartSim on Summit at OLCF

Since SmartSim does not have a built PowerPC build, the build steps for an IBM system are slightly different than other systems.

Luckily for us, Summit has an environment with many of the ML dependencies that SmartSim needs already built into it. Users can follow these instructions to get a working SmartSim build with PyTorch for GPU on Summit.

# setup Python and build environment
module load open-ce
conda create -p /ccs/home/<USERNAME>/.conda/envs/smartsim --clone open-ce-1.2.0-py38-0
conda activate smartsim
module load gcc/8.1.1
module load cuda/11.2.0
module unload xalt
export CC=$(which gcc)
export CXX=$(which g++)
export LDFLAGS="$LDFLAGS -pthread"
export CUDNN_LIBRARY=/sw/summit/cuda/11.2.0/lib64
export CUDNN_INCLUDE_DIR=/sw/summit/cuda/11.2.0/include/

# clone SmartRedis and build
git clone https://github.com/SmartRedis.git smartredis
pushd smartredis
make lib && pip install .
popd

# clone SmartSim and build
git clone https://github.com/SmartSim.git smartsim
pushd smartsim
pip install .

export Torch_DIR=/ccs/home/<USERNAME>/.conda/envs/smartsim/lib/python3.8/site-packages/torch/share/cmake/Torch/
export CFLAGS="$CFLAGS -I/ccs/home/<USERNAME>/.conda/envs/smarter/lib/python3.8/site-packages/tensorflow/include"
# install PyTorch backend for the Orchestrator database.
smart --device=gpu --torch_dir $Torch_DIR -v

SmartSim on Cheyenne at NCAR

Since SmartSim does not currently support the Message Passing Toolkit (MPT), Cheyenne users of SmartSim will need to utilize OpenMPI.

The following module commands were utilized to run the examples

$ module purge
$ module load ncarenv/1.3 gnu/8.3.0 ncarcompilers/0.5.0 netcdf/4.7.4 openmpi/4.0.5

With this environment loaded, users will need to build and install both SmartSim and SmartRedis through pip. Usually we recommend users installing or loading miniconda and using the pip that comes with that installation.

$ pip install smartsim
$ smart --device cpu  (Since Cheyenne does not have GPUs)

To make the SmartRedis library (C, C++, Fortran clients), follow these steps with the same environment loaded.

# clone SmartRedis and build
$ git clone https://github.com/SmartRedis.git smartredis
$ cd smartredis
$ make lib

Changing Redis installation and Configurations

If you want to use a pre-existing Redis installation, override the configuration, or install the dependencies for SmartSim manually, you can do so by specifying a SmartSim configuration file in a config.toml

This can be useful for
  • contributors who want to try bleeding edge features

  • users who want to edit Redis behavior (adding checkpoints)

  • sites who want to package SmartSim in a different manner.

  • sites who want to set intervals for scheduler communication

Override Pre-packaged Settings

The following fields can be overridden in the SmartSim configuration file

  • Redis installation location

  • Redis configuration file location

  • RedisAI module installation location

  • Test launcher/device/network interface

  • Log level default

  • Job manager interval default

Note

If you want to override the configuration, you need to supply values for each of these.

Install the SmartSim user configuration file. Usually this is in your $HOME directory under ~/.smartsim/config.toml, but for users of networked filesystems where the $HOME directory is not mounted, do the following

# set the configuration directory to somewhere accessible
# (e.g. /lustre) for HPC systems

echo "export SMARTSIM_HOME=/REPLACE/ME/.smartsim" >> ~/.bashrc
cd && source .bashrc
mkdir $SMARTSIM_HOME && cd $SMARTSIM_HOME
touch config.toml

The configuration file specifies user and developer settings for SmartSim as well as the location of third-party libraries. If you followed the instructions above, the third-party libraries will be installed in smartsim/third-party/.

You can copy paste, the below configuration file into the config.toml file you just created. The only part that should be changed is the path to where you installed SmartSim

[smartsim]
# number of seconds per job status update
# for jobs on WLM system (e.g. slurm, pbs, etc)
jm_interval = 15    # default
log_level = "info" # default

[redis]
# path to where "redis-server" and "redis-cli" binaries are located
bin = "/path/to/redis/src/"
config = "/path/to/redis.conf" # optional!

  [redis.modules]
  ai = "/path/to/RedisAI/install-cpu/"

[test]
launcher = "local" # default
interface = "ib0" # network interface to use in tests
device = "CPU" # device to use for tests (CPU/GPU)