Skip to content

Commit 83d082e

Browse files
giltirngiltirn
committed
Documentation:
 Reworked run section, now the text is designed for a generic computer Broke out Summit specific run information into a separate section Added description of URS file usage Added details on how to run on Spock
1 parent 97c6ecd commit 83d082e

1 file changed

Lines changed: 113 additions & 22 deletions

File tree

‎sphinx/source/install_usage/run_chimbuko.rst‎

Lines changed: 113 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -48,22 +48,82 @@ A number of variables in **chimbuko_config.sh** are marked :code:`<------------
4848

4949
A full list of variables along with their description is provided in the `Appendix Section <../appendix/appendix_usage.html#chimbuko-config>`_, and more guidance is also provided in the template script.
5050

51-
Next, in the run script, export the config script as follows:
51+
Next, in the run script, export the config script location and also source its contents into the script environment as follows:
5252

5353
.. code:: bash
5454
5555
export CHIMBUKO_CONFIG=chimbuko_config.sh
56+
source ${CHIMBUKO_CONFIG}
5657
5758
---------------------------
5859

5960
In order to avoid having the Chimbuko services interfere with the running of the application, we typically run the services on a dedicated node. It is also necessary to place the ranks of the AD module on the same node as the corresponding rank of the application to avoid having to pipe the traces over the network. Unfortunately the means of setting this up will vary from system to system depending on the job scheduler (e.g. using a **hostfile** with :code:`mpirun`).
6061

61-
In this section we will concentrate on the Summit supercomputer, where the process is made more difficult by the restrictions on sharing resources between resource sets which forces us to dedicate cores to the AD instances. To achieve the job placement the first step is to generate **explicit resource files** (ERF) for the head node, AD, and main programs. For convenience we provide a script `here <https://github.com/CODARcode/PerformanceAnalysis/blob/ckelly_develop/scripts/summit/gen_erf_summit.sh>`_ to generate the ERF files. It generates three ERF files (main.erf, ad.erf, services.erf) which are later used as input ERF files to instantiate Chimbuko services using **jsrun** command.
62-
This can be achieved by running the following script
62+
We launch the services (provenance database, the visualization module, and the parameter server) using the **run_services.sh** script provided in the PerformanceAnalysis install path. The location of this script is contained in the **chimbuko_services** variable set by *chimbuko_config.sh*. By default this location is inferred automatically but it can be set manually in the *chimbuko_config.sh*. A description of commands used in the services script is provided in `Appendix <../appendix/appendix_usage.html#launch-services>`_.
63+
64+
The services are run as a background task such that they continue to run throughout the job. The service script writes out several useful files in the *chimbuko/vars* subdirectory, including a file containing the full run command for the AD module, *chimbuko/vars/chimbuko_ad_cmdline.var*, assuming a basic (single application) workflow. We will use the existence of this file in a wait condition to ensure the services are ready before launching the online AD module and application instances:
65+
66+
.. code:: bash
67+
68+
<LAUNCH ON HEAD NODE> ${chimbuko_services} &
69+
while [ ! -f chimbuko/vars/chimbuko_ad_cmdline.var ]; do sleep 1; done
70+
71+
Here :code:`<LAUNCH ON HEAD NODE>` is the appropriate command to launch a process on the head node of the job.
72+
73+
-----------------------------
74+
75+
We next launch the AD modules:
76+
77+
.. code:: bash
78+
79+
ad_cmd=$(cat chimbuko/vars/chimbuko_ad_cmdline.var)
80+
eval "<LAUNCH N RANKS OF AD ON BODY NODES> ${ad_cmd} &"
81+
82+
Where :code:`<LAUNCH N RANKS OF AD ON BODY NODES>` is the appropriate command to launch *N* ranks of the online AD module on the nodes other than the head node, where *N* is the same as the number of application ranks. Note that this command must ensure that AD rank *i* is launched on the same physical node as application rank *i*
83+
84+
For more complicated workflows the AD will need to be invoked differently. To aid the user the services script writes a second file, **chimbuko/vars/chimbuko_ad_opts.var**, which contains just the initial command line options for the AD. Examples of various setups can be found among the :ref:`benchmark applications <benchmark_suite>`.
85+
86+
-----------------------------
87+
88+
Finally, the application is instantiated using the following command:
89+
90+
.. code:: bash
91+
92+
<LAUNCH N RANKS OF APP ON BODY NODES> ${TAU_EXEC} ${EXE} ${EXE_CMDS}
93+
94+
Where :code:`<LAUNCH N RANKS OF APP ON BODY NODES>` is the appropriate command to launch *N* ranks of the application on the nodes other than the head node, **${TAU_EXEC}** is defined in chimbuko config file as described above, **${EXE}** is the full path to the application's executable and **${EXE_CMDS}** specifies all input parameters that are required by the application executable.
95+
96+
------------------------------
97+
98+
Chimbuko can be run to perform offline analysis of the application by changing configuration for Tau's ADIOS plugin as `described here <../appendix/appendix_usage.html#offline-analysis>`_.
99+
100+
------------------------------
101+
102+
Running on Summit
103+
^^^^^^^^^^^^^^^^^
104+
105+
In this section we provide specifics on launching on the Summit machine.
106+
107+
The following *chimbuko_config.sh* setup provides optimal network performance for the Chimbuko services:
108+
109+
- **service_node_iface** : ib0
110+
- **provdb_engine** : verbs
111+
- **provdb_domain** : mlx5_0
112+
113+
Summit job components are started using IBM's custom *jsrun* command, which supports two methods for completely specifying resource sets that we can use to perform the placement of the services and the AD and application ranks. Note that *jsrun* does not allow different resource sets to share the same hardware resources, hence we are forced to dedicate cores to the AD instances.
114+
115+
ERF files
116+
"""""""""
117+
118+
**(WARNING: As of 12/8/21 this feature is broken and the user should use the secondary URS method documented below until a fix is made available)**
119+
120+
The default method for completely specifying resource sets is **explicit resource files** (ERF) which are supplied using the following command: :code:`jsrun --erf_input=${erf_file}`. The file format is documented `here <https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=SSWRJV_10.1.0/jsm/jsrun.html>`_.
121+
122+
For convenience we provide a script `here <https://github.com/CODARcode/PerformanceAnalysis/blob/ckelly_develop/scripts/summit/gen_erf_summit.sh>`_ to generate the ERF files, which is executed as follows:
63123

64124
.. code:: bash
65125
66-
./gen_erf.pl ${n_nodes_total} ${n_mpi_ranks_per_node} ${n_cores_per_rank_main} ${n_gpus_per_rank_main} ${ncores_per_host_ad}
126+
./gen_erf_summit.pl ${n_nodes_total} ${n_mpi_ranks_per_node} ${n_cores_per_rank_main} ${n_gpus_per_rank_main} ${ncores_per_host_ad}
67127
68128
where
69129

@@ -72,45 +132,76 @@ where
72132
- **${n_cores_per_rank_main}** and **${n_gpus_per_rank_main}** specify the number of cores and GPUs, respectively, given to each rank of the application.
73133
- **${ncores_per_host_ad}** is the number of cores dedicated to the Chimbuko AD modules (must be a multiple of 2), with the application running on the remaining cores. Note that the total number of cores allocated per node must not exceed 42.
74134

75-
More details on ERF can be found `here <https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=SSWRJV_10.1.0/jsm/jsrun.html>`_.
135+
The script writes out three files: *services.erf*, *ad.erf* and *main.erf*. This allows us to fully specify the various :code:`<LAUNCH ...>` commands from the previous section:
136+
137+
which can be used as follows:
138+
139+
.. code:: bash
140+
141+
<LAUNCH ON HEAD NODE> = jsrun --erf_input=services.erf
142+
<LAUNCH N RANKS OF AD ON BODY NODES> = jsrun --erf_input=ad.erf
143+
<LAUNCH N RANKS OF APP ON BODY NODES> = jsrun --erf_input=main.erf
76144
77-
----------------------------
145+
URS files
146+
"""""""""
78147

79-
In the next step the Chimbuko services are launched by running the **run_services.sh** script using **jsrun** command as following:
148+
The *jsrun* command also supports specifying resource sets using :code:`jsrun --use_resource=${urf_file}` or :code:`jsrun -U ${urf_file}` where documentation of the format of these "URS" files can be found `here <https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=SSWRJV_10.1.0/jsm/jsrun.html>`. Note that, unlike the ERF files, the URS files do not allow specification of resource sets at the level of hardware threads, only at the level of cores.
149+
150+
For convenience we provide a script `here <https://github.com/CODARcode/PerformanceAnalysis/blob/ckelly_develop/scripts/summit/gen_urs_summit.pl>`_ to generate the URS files, which is executed as follows:
80151

81152
.. code:: bash
82153
83-
jsrun --erf_input=services.erf ${chimbuko_services} &
84-
while [ ! -f chimbuko/vars/chimbuko_ad_cmdline.var ]; do sleep 1; done
154+
./gen_urs_summit.pl ${n_nodes_total} ${n_mpi_ranks_per_node} ${n_cores_per_rank_main} ${n_gpus_per_rank_main}
155+
156+
where
85157

86-
Here **--erf_input=services.erf** launches the services using the the services ERF generated in the previous step. **${chimbuko_services}** is the path to a script which specifies commands to launch the provenance database, the visualization module, and the parameter server. This variable is set by the configuration script. A description of commands used in the services script is provided in `Appendix <../appendix/appendix_usage.html#launch-services>`_.
158+
- **${n_nodes_total}** is the total number of nodes used, including the one node that is dedicated to run the services.
159+
- **${n_mpi_ranks_per_node}** is the number of MPI ranks of the application (and AD) that will run on each node (must be a multiple of 2).
160+
- **${n_cores_per_rank_main}** and **${n_gpus_per_rank_main}** specify the number of cores and GPUs, respectively, given to each rank of the application.
161+
162+
Note that 1 core is assigned per rank of the AD, and so :code:`${n_mpi_ranks_per_node} * (${n_cores_per_rank_main} + 1)` should not exceed 42, the number of cores per node.
87163

88-
The while loop after the **jsrun** command is used to wait until the services have progressed to a stage at which connection is possible. At this point the script generates a command file (**chimbuko/vars/chimbuko_ad_cmdline.var**) which provides the to launch the Chimbuko's anomaly detection driver program, assuming a basic (single component) workflow. This can be invoked as follows:
164+
The script writes out three files: *services.urs*, *ad.urs* and *main.urs*. This allows us to fully specify the various :code:`<LAUNCH ...>` commands from the previous section:
165+
166+
which can be used as follows:
89167

90168
.. code:: bash
91169
92-
ad_cmd=$(cat chimbuko/vars/chimbuko_ad_cmdline.var)
93-
eval "jsrun --erf_input=ad.erf -e prepended ${ad_cmd} &"
170+
<LAUNCH ON HEAD NODE> = jsrun -U services.urs
171+
<LAUNCH N RANKS OF AD ON BODY NODES> = jsrun -U ad.urs
172+
<LAUNCH N RANKS OF APP ON BODY NODES> = jsrun -U main.urs
94173
95-
Here the **ad.erf** file is used as **--erf_input** for the **jsrun** command.
96174
97-
For more complicated workflows the AD will need to be invoked differently. To aid the user we write a second file, **chimbuko/vars/chimbuko_ad_opts.var**, which contains just the initial command line options for the AD. Examples of various setups can be found among the :ref:`benchmark applications <benchmark_suite>`.
175+
Running on Spock
176+
^^^^^^^^^^^^^^^^
98177

99-
-----------------------------
178+
In this section we provide specifics on launching on the Spock machine.
179+
180+
Spock uses the *slurm* job management system. To control the explicit placement of the ranks we will use the :code:`--nodelist` (:code:`-w`) slurm option to specify the nodes associated with a resource set, the :code:`--nodes` (:code:`-N`) option to specify the number of nodes and the :code:`--overlap` option to allow the AD and application resource sets to coexist on the same node. These options are documented `here <https://slurm.schedmd.com/srun.html>`_.
100181

101-
Finally, the application instantiated using the following command:
182+
The :code:`--nodelist` option requires the range of full hostnames of the nodes to be provided. In order to simplify the generation of this list we provide a script `here <https://github.com/CODARcode/PerformanceAnalysis/blob/ckelly_develop/scripts/spock/get_nodes.pl>`_ that parses the **SLURM_JOB_NODELIST** environment variable and generates the nodelist for the services and application. To use:
102183

103184
.. code:: bash
104185
105-
jsrun --erf_input=main.erf -e prepended ${TAU_EXEC} ${EXE} ${EXE_CMDS}
186+
service_node=$(./get_nodes.pl HEAD)
187+
body_nodelist=$(./get_nodes.pl BODY)
106188
107-
Here we use the third ERF (**main.erf**) which was generated in the previous step. **${TAU_EXEC}** is defined in chimbuko config file as described in previous step. **${EXE}** is the full path to the application's executable. **${EXE_CMDS}** specifies all input parameters that are required by the application executable.
189+
We can now set the various :code:`<LAUNCH ..>` commands in the section above:
108190

109-
------------------------------
191+
.. code:: bash
110192
111-
Chimbuko can be run to perform offline analysis of the application by changing configuration for Tau's ADIOS plugin as `described here <../appendix/appendix_usage.html#offline-analysis>`_.
193+
<LAUNCH ON HEAD NODE> = srun -n 1 -c 64 --threads-per-core=1 -N 1-1 --ntasks-per-node=1 -w ${service_node}
194+
<LAUNCH N RANKS OF AD ON BODY NODES> = srun -n ${N} -c 1 -N ${bodynodes}-${bodynodes} --ntasks-per-node=${n_mpi_ranks_per_node} -w ${body_nodelist} --overlap
195+
<LAUNCH N RANKS OF APP ON BODY NODES> = srun -n ${N} -c ${ncores_per_rank_main} -N ${bodynodes}-${bodynodes} --ntasks-per-node=${n_mpi_ranks_per_node} -w ${body_nodelist} --gpus-per-task=${n_gpus_per_rank_main} --gpu-bind=closest --overlap
112196
113-
------------------------------
197+
Where
198+
199+
- **${n_nodes_total}** is the total number of nodes used, including the one node that is dedicated to run the services.
200+
- **${bodynodes}** is the number of nodes dedicated to the application and AD ranks (i.e. :code:`n_nodes_total-1`)
201+
- **${n_mpi_ranks_per_node}** is the number of MPI ranks of the application (and AD) that will run on each node (must be a multiple of 2).
202+
- **${n_cores_per_rank_main}** and **${n_gpus_per_rank_main}** specify the number of cores and GPUs, respectively, given to each rank of the application.
203+
204+
Note that we have assigned 1 core to each rank of the AD, and so :code:`${n_mpi_ranks_per_node} * (${n_cores_per_rank_main} + 1)` should not exceed 64, the number of available cores.
114205

115206
Scaling to large job sizes
116207
^^^^^^^^^^^^^^^^^^^^^^^^^^

0 commit comments

Comments
 (0)