Orchestrator#

The Orchestrator is an in-memory database that is launched prior to all other entities within an Experiment. The Orchestrator can be used to store and retrieve data during the course of an experiment and across multiple entities. In order to stream data into or receive data from the Orchestrator, one of the SmartSim clients (SmartRedis) has to be used within a Model.

Alternative text

Combined with the SmartRedis clients, the Orchestrator is capable of hosting and executing AI models written in Python on CPU or GPU. The Orchestrator supports models written with TensorFlow, Pytorch, TensorFlow-Lite, or models saved in an ONNX format (e.g. sci-kit learn).

Cluster Orchestrator#

The Orchestrator supports single node and distributed memory settings. This means that a single compute host can be used for the database or multiple by specifying db_nodes to be greater than 1.

Alternative text

With a clustered Orchestrator, multiple compute hosts memory can be used together to store data. As well, the CPU or GPU(s) where the Orchestrator is running can be used to execute the AI models, and Torchscript code on data stored within it.

Users do not need to know how the data is stored in a clustered configuration and can address the cluster with the SmartRedis clients like a single block of memory using simple put/get semantics in SmartRedis. SmartRedis will ensure that data is evenly distributed amongst all nodes in the cluster.

The cluster deployment is optimal for high data throughput scenarios such as online analysis, training and processing.

Colocated Orchestrator#

A colocated Orchestrator is a special type of Orchestrator that is deployed on the same compute hosts an a Model instance defined by the user. In this deployment, the database is not connected together in a cluster and each shard of the database is addressed individually by the processes running on that compute host.

Alternative text

This deployment is designed for highly performant online inference scenarios where a distributed process (likely MPI processes) are performing inference with data local to each process.

This method is deemed locality based inference since data is local to each process and the Orchestrator is deployed locally on each compute host where the distributed application is running.

To create a colocated model, first, create a Model instance and then call the Model.colocate_db_tcp or Model.colocate_db_uds function.

Model.colocate_db_tcp(port: int = 6379, ifname: t.Union[str, list[str]] = 'lo', db_cpus: int = 1, custom_pinning: t.Optional[t.Iterable[t.Union[int, t.Iterable[int]]]] = None, debug: bool = False, db_identifier: str = '', **kwargs: t.Any) None[source]

Colocate an Orchestrator instance with this Model over TCP/IP.

This method will initialize settings which add an unsharded database to this Model instance. Only this Model will be able to communicate with this colocated database by using the loopback TCP interface.

Extra parameters for the db can be passed through kwargs. This includes many performance, caching and inference settings.

ex. kwargs = {
    maxclients: 100000,
    threads_per_queue: 1,
    inter_op_threads: 1,
    intra_op_threads: 1,
    server_threads: 2 # keydb only
}

Generally these don’t need to be changed.

Parameters:
  • port (int, optional) – port to use for orchestrator database, defaults to 6379

  • ifname (str | list[str], optional) – interface to use for orchestrator, defaults to “lo”

  • db_cpus (int, optional) – number of cpus to use for orchestrator, defaults to 1

  • custom_pinning (iterable of ints or iterable of ints, optional) – CPUs to pin the orchestrator to. Passing an empty iterable disables pinning

  • debug (bool, optional) – launch Model with extra debug information about the colocated db

  • kwargs (dict, optional) – additional keyword arguments to pass to the orchestrator database

Model.colocate_db_uds(unix_socket: str = '/tmp/redis.socket', socket_permissions: int = 755, db_cpus: int = 1, custom_pinning: Iterable[int | Iterable[int]] | None = None, debug: bool = False, db_identifier: str = '', **kwargs: Any) None[source]

Colocate an Orchestrator instance with this Model over UDS.

This method will initialize settings which add an unsharded database to this Model instance. Only this Model will be able to communicate with this colocated database by using Unix Domain sockets.

Extra parameters for the db can be passed through kwargs. This includes many performance, caching and inference settings.

example_kwargs = {
    "maxclients": 100000,
    "threads_per_queue": 1,
    "inter_op_threads": 1,
    "intra_op_threads": 1,
    "server_threads": 2 # keydb only
}

Generally these don’t need to be changed.

Parameters:
  • unix_socket (str, optional) – path to where the socket file will be created

  • socket_permissions (int, optional) – permissions for the socketfile

  • db_cpus (int, optional) – number of cpus to use for orchestrator, defaults to 1

  • custom_pinning (iterable of ints or iterable of ints, optional) – CPUs to pin the orchestrator to. Passing an empty iterable disables pinning

  • debug (bool, optional) – launch Model with extra debug information about the colocated db

  • kwargs (dict, optional) – additional keyword arguments to pass to the orchestrator database

Here is an example of creating a simple model that is colocated with an Orchestrator deployment using Unix Domain Sockets

from smartsim import Experiment
exp = Experiment("colo-test", launcher="auto")

colo_settings = exp.create_run_settings(exe="./some_mpi_app")

colo_model = exp.create_model("colocated_model", colo_settings)
colo_model.colocate_db_uds(
        db_cpus=1,              # cpus given to the database on each node
        debug=False             # include debug information (will be slower)
        ifname=network_interface # specify network interface(s) to use (i.e. "ib0" or ["ib0", "lo"])
)
exp.start(colo_model)

By default, SmartSim will pin the database to the first _N_ CPUs according to db_cpus. By specifying the optional argument custom_pinning, an alternative pinning can be specified by sending in a list of CPU ids (e.g [0,2,range(5,8)]). For optimal performance, most users will want to also modify the RunSettings for the model to pin their application to cores not occupied by the database.

Warning

Pinning is not supported on MacOS X. Setting custom_pinning to anything other than None will raise a warning and the input will be ignored.

Note

Pinning _only_ affects the co-located deployment because both the application and the database are sharing the same compute node. For the clustered deployment, a shard occupies the entirety of the node.

Redis#

The Orchestrator is built on Redis. Largely, the job of the Orchestrator is to create a Python reference to a Redis deployment so that users can launch, monitor and stop a Redis deployment on workstations and HPC systems.

Redis was chosen for the Orchestrator because it resides in-memory, can be distributed on-node as well as across nodes, and provides low latency data access to many clients in parallel. The Redis ecosystem was a primary driver as the Redis module system provides APIs for languages, libraries, and techniques used in Data Science. In particular, the Orchestrator relies on RedisAI to provide access to Machine Learning runtimes.

At its core, Redis is a key-value store. This means that put/get semantics are used to send messages to and from the database. SmartRedis clients use a specific hashing algorithm, CRC16, to ensure that data is evenly distributed amongst all database nodes. Notably, a user is not required to know where (which database node) data or Datasets (see Dataset API) are stored as the SmartRedis clients will infer their location for the user.

KeyDB#

KeyDB is a multi-threaded fork of Redis that can be swapped in as the database for the Orchestrator in SmartSim. KeyDB can be swapped in for Redis by setting the REDIS_PATH environment variable to point to the keydb-server binary.

A full example of configuring KeyDB to run in SmartSim is shown below

# build KeyDB
# see https://github.com/EQ-Alpha/KeyDB

# get KeyDB configuration file
wget https://github.com/CrayLabs/SmartSim/blob/d3d252b611c9ce9d9429ba6eeb71c15471a78f08/smartsim/_core/config/keydb.conf

export REDIS_PATH=/path/to/keydb-server
export REDIS_CONF=/path/to/keydb.conf

# run smartsim workload

Multiple Orchestrator Example#

SmartSim offers functionality to automate the deployment of multiple databases, supporting workloads that require multiple Orchestrators for a Experiment. For instance, a workload may consist of a simulation with high inference performance demands (necessitating a co-located deployment), along with an analysis and visualization workflow connected to the simulation (requiring a standard orchestrator). In the following example, we simulate a simple version of this use case.

The example is comprised of two script files:

The Application Script Overview: In this example, the application script is a python file that contains instructions to complete computational tasks. Applications are not limited to Python and can also be written in C, C++ and Fortran. This script specifies creating a Python SmartRedis client for each standard orchestrator and a colocated orchestrator. We use the clients to request data from both standard databases, then transfer the data to the colocated database. The application file is launched by the experiment driver script through a Model stage.

The Application Script Contents:

  1. Connecting SmartRedis clients within the application to retrieve tensors from the standard databases to store in a colocated database. Details in section: Initialize the Clients.

The Experiment Driver Script Overview: The experiment driver script holds the stages of the workflow and manages their execution through the Experiment API. We initialize an Experiment at the beginning of the Python file and use the Experiment to iteratively create, configure and launch computational kernels on the system through the slurm launcher. In the driver script, we use the Experiment to create and launch a Model instance that runs the application.

The Experiment Driver Script Contents:

  1. Launching two standard Orchestrators with unique identifiers. Details in section: Launch Multiple Orchestrators.

  2. Launching the application script with a co-located database. Details in section: Initialize a Colocated Model.

  3. Connecting SmartRedis clients within the driver script to send tensors to standard Orchestrators for retrieval within the application. Details in section: Create Client Connections to Orchestrators.

Setup and run instructions can be found here

The Application Script#

Applications interact with the databases through a SmartRedis client. In this section, we write an application script to demonstrate how to connect SmartRedis clients in the context of multiple launched databases. Using the clients, we retrieve tensors from two databases launched in the driver script, then store the tensors in the colocated database.

Note

The Experiment must be started to use the Orchestrators within the application script. Otherwise, it will fail to connect. Find the instructions on how to launch here

To begin, import the necessary packages:

1from smartredis import ConfigOptions, Client
2from smartredis import *
3from smartredis.error import *

Initialize the Clients#

To establish a connection with each database, we need to initialize a new SmartRedis client for each Orchestrator.

Step 1: Initialize ConfigOptions#

Since we are launching multiple databases within the experiment, the SmartRedis ConfigOptions object is required when initializing a client in the application. We use the ConfigOptions.create_from_environment() function to create three instances of ConfigOptions, with one instance associated with each launched Orchestrator. Most importantly, to associate each launched Orchestrator to a ConfigOptions object, the create_from_environment() function requires specifying the unique database identifier argument named db_identifier.

For the single-sharded database:

1# Initialize a ConfigOptions object
2single_shard_config = ConfigOptions.create_from_environment("single_shard_db_identifier")

For the multi-sharded database:

1# Initialize a ConfigOptions object
2multi_shard_config = ConfigOptions.create_from_environment("multi_shard_db_identifier")

For the colocated database:

1# Initialize a ConfigOptions object
2colo_config = ConfigOptions.create_from_environment("colo_db_identifier")
Step 2: Initialize the Client Connections#

Now that we have three ConfigOptions objects, we have the tools necessary to initialize three SmartRedis clients and establish a connection with the three databases. We use the SmartRedis Client API to create the client instances by passing in the ConfigOptions objects and assigning a logger_name argument.

Single-sharded database:

1# Initialize a SmartRedis client for the single sharded database
2app_single_shard_client = Client(single_shard_config, logger_name="Model: single shard logger")

Multi-sharded database:

1# Initialize a SmartRedis client for the multi sharded database
2app_multi_shard_client = Client(multi_shard_config, logger_name="Model: multi shard logger")

Colocated database:

1# Initialize a SmartRedis client for the colocated database
2colo_client = Client(colo_config, logger_name="Model: colo logger")

Retrieve Data and Store Using SmartRedis Client Objects#

To confirm a successful connection to each database, we will retrieve the tensors that we plan to store in the python driver script. After retrieving, we store both tensors in the colocated database. The Client.get_tensor() method allows retrieval of a tensor. It requires the name of the tensor assigned when sent to the database via Client.put_tensor().

1# Retrieve the tensor placed in driver script using the associated client
2val1 = app_single_shard_client.get_tensor("tensor_1")
3val2 = app_multi_shard_client.get_tensor("tensor_2")
4
5# Print message to stdout using SmartRedis Client logger
6app_single_shard_client.log_data(LLInfo, f"The single sharded db tensor is: {val1}")
7app_multi_shard_client.log_data(LLInfo, f"The multi sharded db tensor is: {val2}")

Later, when you run the experiment driver script the following output will appear in tutorial_model.out located in getting-started-multidb/tutorial_model/:

Model: single shard logger@00-00-00:The single sharded db tensor is: [1 2 3 4]
Model: multi shard logger@00-00-00:The multi sharded db tensor is: [5 6 7 8]

This output showcases that we have established a connection with multiple Orchestrators.

Next, take the tensors retrieved from the standard deployment databases and store them in the colocated database using Client.put_tensor(name, data).

1# Place retrieved tensors in colocated database
2colo_client.put_tensor("tensor_1", val1)
3colo_client.put_tensor("tensor_2", val2)

Next, check if the tensors exist in the colocated database using Client.poll_tensor(). This function queries for data in the database. The function requires the tensor name (name), how many milliseconds to wait in between queries (poll_frequency_ms), and the total number of times to query (num_tries):

1# Check that tensors are in colocated database
2colo_val1 = colo_client.poll_tensor("tensor_1", 10, 10)
3colo_val2 = colo_client.poll_tensor("tensor_2", 10, 10)
4# Print message to stdout using SmartRedis Client logger
5colo_client.log_data(LLInfo, f"The colocated db has tensor_1: {colo_val1}")
6colo_client.log_data(LLInfo, f"The colocated db has tensor_2: {colo_val2}")

The output will be as follows:

Model: colo logger@00-00-00:The colocated db has tensor_1: True
Model: colo logger@00-00-00:The colocated db has tensor_2: True

The Experiment Driver Script#

To run the previous application, we must define workflow stages within a workload. Defining workflow stages requires the utilization of functions associated with the Experiment object. The Experiment object is intended to be instantiated once and utilized throughout the workflow runtime. In this example, we instantiate an Experiment object with the name getting-started-multidb. We setup the SmartSim logger to output information from the Experiment.

 1import numpy as np
 2from smartredis import Client
 3from smartsim import Experiment
 4from smartsim.log import get_logger
 5import sys
 6
 7exe_ex = sys.executable
 8logger = get_logger("Multidb Experiment Log")
 9# Initialize the Experiment
10exp = Experiment("getting-started-multidb", launcher="auto")

Launch Multiple Orchestrators#

In the context of this Experiment, it’s essential to create and launch the databases as a preliminary step before any other components since the application script requests tensors from the launched databases.

We aim to showcase the multi-database automation capabilities of SmartSim, so we create two databases in the workflow: a single-sharded database and a multi-sharded database.

Step 1: Initialize Orchestrators#

To create an database, utilize the Experiment.create_database() function. The function requires specifying a unique database identifier argument named db_identifier to launch multiple databases. This step is necessary to connect to databases outside of the driver script. We will use the db_identifier names we specified in the application script.

For the single-sharded database:

1# Initialize a single sharded database
2single_shard_db = exp.create_database(port=6379, db_nodes=1, interface="ib0", db_identifier="single_shard_db_identifier")
3exp.generate(single_shard_db, overwrite=True)

For the multi-sharded database:

1# Initialize a multi sharded database
2multi_shard_db = exp.create_database(port=6380, db_nodes=3, interface="ib0", db_identifier="multi_shard_db_identifier")
3exp.generate(multi_shard_db, overwrite=True)

Note

Calling exp.generate() will create two subfolders (one for each Orchestrator created in the previous step) whose names are based on the db_identifier of that Orchestrator. In this example, the Experiment folder is named getting-started-multidb/. Within this folder, two Orchestrator subfolders will be created, namely single_shard_db_identifier/ and multi_shard_db_identifier/.

Step 2: Start Databases#

Next, to launch the databases, pass the database instances to Experiment.start().

1# Launch the single and multi sharded database
2exp.start(single_shard_db, multi_shard_db, summary=True)

The Experiment.start() function launches the Orchestrators for use within the workflow. In other words, the function deploys the databases on the allocated compute resources.

Note

By setting summary=True, SmartSim will print a summary of the experiment before it is launched. After printing the experiment summary, the experiment is paused for 10 seconds giving the user time to briefly scan the summary contents. If we set summary=False, then the experiment would be launched immediately with no summary.

Create Client Connections to Orchestrators#

The SmartRedis Client object contains functions that manipulate, send, and receive data within the database. Each database has a single, dedicated SmartRedis Client. Begin by initializing a SmartRedis Client object per launched database.

To create a designated SmartRedis Client, you need to specify the address of the target running database. You can easily retrieve this address using the Orchestrator.get_address() function.

For the single-sharded database:

1# Initialize SmartRedis client for single sharded database
2driver_client_single_shard = Client(cluster=False, address=single_shard_db.get_address()[0], logger_name="Single shard db logger")

For the multi-sharded database:

1# Initialize SmartRedis client for multi sharded database
2driver_client_multi_shard = Client(cluster=True, address=multi_shard_db.get_address()[0], logger_name="Multi shard db logger")

Store Data Using Clients#

In the application script, we retrieved two NumPy tensors. To support the apps functionality, we will create two NumPy arrays in the python driver script and send them to the a database. To accomplish this, we use the Client.put_tensor() function with the respective database client instances.

For the single-sharded database:

1# Create NumPy array
2array_1 = np.array([1, 2, 3, 4])
3# Use single shard db SmartRedis client to place tensor in single sharded db
4driver_client_single_shard.put_tensor("tensor_1", array_1)

For the multi-sharded database:

1# Create NumPy array
2array_2 = np.array([5, 6, 7, 8])
3# Use single shard db SmartRedis client to place tensor in multi sharded db
4driver_client_multi_shard.put_tensor("tensor_2", array_2)

Lets check to make sure the database tensors do not exist in the incorrect databases:

1# Check that tensors are in correct databases
2check_single_shard_db_tensor_incorrect = driver_client_single_shard.key_exists("tensor_2")
3check_multi_shard_db_tensor_incorrect = driver_client_multi_shard.key_exists("tensor_1")
4logger.info(f"The multi shard array key exists in the incorrect database: {check_single_shard_db_tensor_incorrect}")
5logger.info(f"The single shard array key exists in the incorrect database: {check_multi_shard_db_tensor_incorrect}")

When you run the experiment, the following output will appear:

00:00:00 system.host.com SmartSim[#####] INFO The multi shard array key exists in the incorrect database: False
00:00:00 system.host.com SmartSim[#####] INFO The single shard array key exists in the incorrect database: False

Initialize a Colocated Model#

In the next stage of the experiment, we launch the application script with a co-located database by configuring and creating a SmartSim colocated Model.

Step 1: Configure#

You can specify the run settings of a model. In this experiment, we invoke the Python interpreter to run the python script defined in section: The Application Script. To configure this into a Model, we use the Experiment.create_run_settings() function. The function returns a RunSettings object. When initializing the RunSettings object, we specify the path to the application file, application_script.py, for exe_args, and the run command for exe.

1# Initialize a RunSettings object
2model_settings = exp.create_run_settings(exe=exe_ex, exe_args="./path/to/application_script.py")

Note

You will have to change the exe_args argument to the path of the application script on your machine to run the example.

With the RunSettings instance, configure the the distribution of computational tasks (RunSettings.set_nodes()) and the number of instances the script is execute on each node (RunSettings.set_tasks_per_node()). In this example, we specify to SmartSim that we intend to execute the script once on a single node.

1# Configure RunSettings object
2model_settings.set_nodes(1)
3model_settings.set_tasks_per_node(1)
Step 2: Initialize#

Next, create a Model instance using the Experiment.create_model(). Pass the model_settings object as an argument to the create_model() function and assign to the variable model.

1# Initialize a SmartSim Model
2model = exp.create_model("colo_model", model_settings)
Step 2: Colocate#

To colocate the model, use the Model.colocate_db_uds() function to Colocate an Orchestrator instance with this Model over a Unix domain socket connection.

1# Colocate the Model
2model.colocate_db_tcp(db_identifier="colo_db_identifier")

This method will initialize settings which add an unsharded database to this Model instance. Only this Model will be able to communicate with this colocated database by using the loopback TCP interface.

Step 3: Start#

Next, launch the colocated model instance using the Experiment.start() function.

1# Launch the colocated Model
2exp.start(model, block=True, summary=True)

Note

We set block=True, so that Experiment.start() waits until the last Model has finished before returning: it will act like a job monitor, letting us know if processes run, complete, or fail.

Cleanup Experiment#

Finally, use the Experiment.stop() function to stop the database instances. Print the workflow summary with Experiment.summary().

1# Tear down the single and multi sharded databases
2exp.stop(single_shard_db, multi_shard_db)
3# Print the Experiment summary
4logger.info(exp.summary())

When you run the experiment, the following output will appear:

00:00:00 system.host.com SmartSim[#####]INFO
|    | Name                         | Entity-Type   | JobID       | RunID   | Time    | Status    | Returncode   |
|----|------------------------------|---------------|-------------|---------|---------|-----------|--------------|
| 0  | colo_model                   | Model         | 1556529.5   | 0       | 1.7437  | Completed | 0            |
| 1  | single_shard_db_identifier_0 | DBNode        | 1556529.3   | 0       | 68.8732 | Cancelled | 0            |
| 2  | multi_shard_db_identifier_0  | DBNode        | 1556529.4+2 | 0       | 45.5139 | Cancelled | 0            |

How to Run the Example#

Below are the steps to run the experiment. Find the experiment source code and application source code below in the respective subsections.

Note

The example assumes that you have already installed and built SmartSim and SmartRedis. Please refer to Section Basic Installation for further details. For simplicity, we assume that you are running on a SLURM-based HPC-platform. Refer to the steps below for more details.

Step 1Setup your directory tree

Your directory tree should look similar to below:

SmartSim/
SmartRedis/
Multi-db-example/
  application_script.py
  experiment_script.py

You can find the application and experiment source code in subsections below.

Step 2Install and Build SmartSim

This example assumes you have installed SmartSim and SmartRedis in your Python environment. We also assume that you have built SmartSim with the necessary modules for the machine you are running on.

Step 3Change the exe_args file path

When configuring the colocated model in experiment_script.py, we pass the file path of application_script.py to the exe_args argument on line 33 in experiment_script.py. Edit this argument to the file path of your application_script.py

Step 4Run the Experiment

Finally, run the experiment with python experiment_script.py.

Application Source Code#

 1from smartredis import ConfigOptions, Client
 2from smartredis import *
 3from smartredis.error import *
 4
 5# Initialize a ConfigOptions object
 6single_shard_config = ConfigOptions.create_from_environment("single_shard_db_identifier")
 7# Initialize a SmartRedis client for the single sharded database
 8app_single_shard_client = Client(single_shard_config, logger_name="Model: single shard logger")
 9
10# Initialize a ConfigOptions object
11multi_shard_config = ConfigOptions.create_from_environment("multi_shard_db_identifier")
12# Initialize a SmartRedis client for the multi sharded database
13app_multi_shard_client = Client(multi_shard_config, logger_name="Model: multi shard logger")
14
15# Initialize a ConfigOptions object
16colo_config = ConfigOptions.create_from_environment("colo_db_identifier")
17# Initialize a SmartRedis client for the colocated database
18colo_client = Client(colo_config, logger_name="Model: colo logger")
19
20# Retrieve the tensor placed in driver script using the associated client
21val1 = app_single_shard_client.get_tensor("tensor_1")
22val2 = app_multi_shard_client.get_tensor("tensor_2")
23
24# Print message to stdout using SmartRedis Client logger
25app_single_shard_client.log_data(LLInfo, f"The single sharded db tensor is: {val1}")
26app_multi_shard_client.log_data(LLInfo, f"The multi sharded db tensor is: {val2}")
27
28# Place retrieved tensors in colocated database
29colo_client.put_tensor("tensor_1", val1)
30colo_client.put_tensor("tensor_2", val2)
31
32# Check that tensors are in colocated database
33colo_val1 = colo_client.poll_tensor("tensor_1", 10, 10)
34colo_val2 = colo_client.poll_tensor("tensor_2", 10, 10)
35# Print message to stdout using SmartRedis Client logger
36colo_client.log_data(LLInfo, f"The colocated db has tensor_1: {colo_val1}")
37colo_client.log_data(LLInfo, f"The colocated db has tensor_2: {colo_val2}")

Experiment Source Code#

 1import numpy as np
 2from smartredis import Client
 3from smartsim import Experiment
 4from smartsim.log import get_logger
 5import sys
 6
 7exe_ex = sys.executable
 8logger = get_logger("Multidb Experiment Log")
 9# Initialize the Experiment
10exp = Experiment("getting-started-multidb", launcher="auto")
11
12# Initialize a single sharded database
13single_shard_db = exp.create_database(port=6379, db_nodes=1, interface="ib0", db_identifier="single_shard_db_identifier")
14exp.generate(single_shard_db, overwrite=True)
15
16# Initialize a multi sharded database
17multi_shard_db = exp.create_database(port=6380, db_nodes=3, interface="ib0", db_identifier="multi_shard_db_identifier")
18exp.generate(multi_shard_db, overwrite=True)
19
20# Launch the single and multi sharded database
21exp.start(single_shard_db, multi_shard_db, summary=True)
22
23# Initialize SmartRedis client for single sharded database
24driver_client_single_shard = Client(cluster=False, address=single_shard_db.get_address()[0], logger_name="Single shard db logger")
25# Initialize SmartRedis client for multi sharded database
26driver_client_multi_shard = Client(cluster=True, address=multi_shard_db.get_address()[0], logger_name="Multi shard db logger")
27
28# Create NumPy array
29array_1 = np.array([1, 2, 3, 4])
30# Use single shard db SmartRedis client to place tensor in single sharded db
31driver_client_single_shard.put_tensor("tensor_1", array_1)
32
33# Create NumPy array
34array_2 = np.array([5, 6, 7, 8])
35# Use single shard db SmartRedis client to place tensor in multi sharded db
36driver_client_multi_shard.put_tensor("tensor_2", array_2)
37
38# Check that tensors are in correct databases
39check_single_shard_db_tensor_incorrect = driver_client_single_shard.key_exists("tensor_2")
40check_multi_shard_db_tensor_incorrect = driver_client_multi_shard.key_exists("tensor_1")
41logger.info(f"The multi shard array key exists in the incorrect database: {check_single_shard_db_tensor_incorrect}")
42logger.info(f"The single shard array key exists in the incorrect database: {check_multi_shard_db_tensor_incorrect}")
43
44# Initialize a RunSettings object
45model_settings = exp.create_run_settings(exe=exe_ex, exe_args="./path/to/application_script.py")
46# Configure RunSettings object
47model_settings.set_nodes(1)
48model_settings.set_tasks_per_node(1)
49# Initialize a SmartSim Model
50model = exp.create_model("colo_model", model_settings)
51# Colocate the Model
52model.colocate_db_tcp(db_identifier="colo_db_identifier")
53# Launch the colocated Model
54exp.start(model, block=True, summary=True)
55
56# Tear down the single and multi sharded databases
57exp.stop(single_shard_db, multi_shard_db)
58# Print the Experiment summary
59logger.info(exp.summary())