Skip to content

Commit 0cc046b

Browse files
giltirngiltirn
committed
Rewrote installation section to describe building using spack. Manual installation details are moved to the appendix.
1 parent 36ec29d commit 0cc046b

2 files changed

Lines changed: 71 additions & 71 deletions

File tree

‎sphinx/source/appendix/appendix_install.rst‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,3 +31,20 @@ GoogleTest on Summit
3131
cd ${GTEST_BUILD_DIR}
3232
CC=gcc CXX=g++ cmake -DCMAKE_INSTALL_PREFIX=${GTEST_INSTALL_DIR} ${GTEST_SOURCE_DIR}/googletest-release-1.10.0
3333
make install
34+
35+
.. _manual_installation_of_chimbuko:
36+
37+
Manual installation of Chimbuko
38+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
39+
40+
Instructions on how to install the Chimbuko visualization can be found on GitHub `here <https://github.com/CODARcode/ChimbukoVisualizationII/>`. To build the AD module:
41+
42+
.. code:: bash
43+
44+
cd ${AD_SOURCE_DIR}
45+
./autogen.sh
46+
cd ${AD_BUILD_DIR}
47+
${AD_SOURCE_DIR}/configure --with-adios2=${ADIOS2_INSTALL_DIR} --with-network=ZMQ --with-perf-metric --prefix=${AD_INSTALL_DIR}
48+
make install
49+
50+
Here **--with-perf-metric** is an optional flag that enables generation of performance information of Chimbuko. The ZMQ network is the only one presently supported; an MPI network is maintained for legacy purposes and is not recommended for use. Assuming Sonata is installed and the mochi-sonata spack module loaded, the configure script will automatically detect the installation and will build the provenance database.

‎sphinx/source/install_usage/install.rst‎

Lines changed: 54 additions & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -2,124 +2,107 @@
22
Installation
33
************
44

5-
For x86 systems we provide pre-built docker images users can quickly start with their own TAU instrumented applications (See `Chimbuko docker <https://codarcode.github.io/Chimbuko/installation/docker.html>`_). Otherwise, first, download (or clone) **Chimbuko** AD module.
5+
For x86 systems we provide pre-built docker images users can quickly start with their own TAU instrumented applications (See `Chimbuko docker <https://codarcode.github.io/Chimbuko/installation/docker.html>`_). Otherwise, we recommend that Chimbuko be installed via the `Spack package manager <https://spack.io/>`_. Below we provide instructions for installing Chimbuko on a typical Ubuntu desktop and also on the Summit computer. Some details on installing Chimbuko in absence of Spack can be found in the :ref:`Appendix <manual_installation_of_chimbuko>`.
66

7-
.. code:: bash
8-
9-
git clone https://github.com/CODARcode/PerformanceAnalysis.git
10-
11-
Then follow the build instructions below.
7+
In all cases, the first step is to download and install Spack following the instructions `here <https://github.com/spack/spack>`_ . Note that installing Spack requires Python.
128

13-
The AD module requires ADIOS2_ for communicating between TAU and Chimbuko, ZeroMQ_ for communication between the AD components, and Sonata_ for the provenance database. Sonata is not required if the provenance database will not be used. GoogleTest_ is required to build the unit and integration tests.
9+
We require Spack repositories for Chimbuko and for the Mochi stack:
1410

15-
Note that in the documentation below, we will use bash-style variables such as ${AD_SOURCE_DIR} and ${ADIOS2_INSTALL_DIR} to indicate directories that the user must specify.
11+
.. code:: bash
1612
17-
Ubuntu 18.04
18-
~~~~~~~~~~~~
13+
git clone https://github.com/mochi-hpc/mochi-spack-packages.git
14+
spack repo add mochi-spack-packages
15+
git clone https://github.com/CODARcode/PerformanceAnalysis.git
16+
spack repo add PerformanceAnalysis/spack/repo/chimbuko
17+
18+
Then follow the build instructions below.
1919

20-
To install ADIOS2_, please check `it's website <https://adios2.readthedocs.io/en/latest/setting_up/setting_up.html>`_. The online analysis requires ADIOS2 to be built with MPI (ADIOS2_USE_MPI) the SST engine (ADIOS2_USE_SST).
20+
Basic installation
21+
~~~~~~~~~~~~~~~~~~
2122

22-
For ZeroMQ_,
23+
A basic installation of Chimbuko can be achieved very easily:
2324

2425
.. code:: bash
2526
26-
apt-get install libzmq3-dev
27+
spack install chimbuko^py-setuptools-scm+toml
2728
28-
Sonata_ is most conveniently installed using Spack_. For detailed instructions on the usage and installation of Spack please check `it's website <https://spack.readthedocs.io/en/latest/>`_, and details of the installation of Sonata can be found `here <https://xgitlab.cels.anl.gov/sds/sonata>`_. We recommend building Sonata as follows:
29+
Note that the dependency on :code:`py-setuptools-scm+toml` resolves a dependency conflict likely resulting from a bug in Spack's current dependency resolution.
2930

30-
.. code:: bash
31-
32-
git clone https://xgitlab.cels.anl.gov/sds/sds-repo.git
33-
spack repo add sds-repo
34-
spack install mochi-sonata@master ^libfabric fabrics=tcp,rxm
31+
A Dockerfile (instructions for building a Docker image) that installs Chimbuko on top of a basic Ubuntu 18.04 image following the above steps can be found `here <https://github.com/CODARcode/PerformanceAnalysis/blob/master/docker/ubuntu18.04/openmpi4.0.4/Dockerfile.chimbuko.spack>`_ .
3532

36-
If libfabric is pre-installed on the user's system it should be indicated to Spack via the packages.yaml (cf `here <https://spack-tutorial.readthedocs.io/en/latest/tutorial_configuration.html>`_). In order to build and run with Sonata it is necessary to ensure the **mochi-sonata** spack package is loaded:
33+
Once installed, the unit and integration tests can be run as:
3734

3835
.. code:: bash
3936
40-
spack load mochi-sonata
37+
cd $(spack location -i chimbuko-performance-analysis)/test
38+
./run_all.sh
4139
42-
To build test cases, users need to install gtest. See Instructions to installs gtest `here <../appendix/appendix_install.html>`_.
40+
A note on libfabric providers
41+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4342

44-
Finally, to build the AD module
43+
The Mercury library used for the provenance database requires a libfabric provider that supports the **FI_EP_RDM** endpoint. By default spack installs libfabric with the **sockets**, **tcp** and **udp** providers, of which only **sockets** supports this endpoint. However **sockets** is being deprecated as its performance is not as good as other dedicated providers. We recommend installing the **rxm** utility provider alongside **tcp** for most purposes, by appending the spack spec with :code:`^libfabric fabrics=sockets,tcp,rxm`.
4544

46-
.. code:: bash
45+
For network hardware supporting the Linux Verbs API (such as Infiniband) the **verbs** provider (with **rxm**) may provide better performance. This can be added to the spec as, for example, :code:`^libfabric fabrics=sockets,tcp,rxm,verbs`.
4746

48-
cd ${AD_SOURCE_DIR}
49-
./autogen.sh
50-
cd ${AD_BUILD_DIR}
51-
${AD_SOURCE_DIR}/configure --with-adios2=${ADIOS2_INSTALL_DIR} --with-network=ZMQ --with-perf-metric --prefix=${AD_INSTALL_DIR}
52-
make install
47+
Details of how to choose the libfabrics provider used by Mercury can be found `here <>`_. For further information consider the `Mercury documentation <https://mercury-hpc.github.io/documentation/#network-abstraction-layer>`_ .
5348

54-
Here **--with-perf-metric** is an optional flag that enables generation of performance information of Chimbuko. The ZMQ network is the only one presently supported; an MPI network is maintained for legacy purposes and is not recommended for use. Assuming Sonata is installed and the mochi-sonata spack module loaded, the configure script will automatically detect the installation and will build the provenance database.
49+
Integrating with system-installed MPI
50+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5551

56-
Once installed the unit and integration tests can be run as:
52+
Chimbuko requires an installation of MPI. While Spack can install MPI automatically as a dependency of Chimbuko, in most cases it is desirable to utilize the system installation. Instructions on configuring Spack to use external dependencies can be found `here <https://spack.readthedocs.io/en/latest/build_settings.html#external-packages>`_ . The simplest approach in general is to edit (create) a **packages.yaml** in one of Spack's search paths, e.g. :code:`~/.spack/packages.yaml`, with the following content:
5753

58-
.. code:: bash
59-
60-
cd ${AD_INSTALL_DIR}/test
61-
./run_all.sh
54+
.. code:: yaml
6255
56+
packages:
57+
openmpi:
58+
buildable: false
59+
externals:
60+
- spec: openmpi@4.0.4
61+
prefix: /opt/openmpi4.0.4
6362
63+
Modified as necessary to point to your installation.
6464

6565
Summit
6666
~~~~~~
6767

68-
Prior to building anything the user should ensure to load the required modules:
68+
While the above instructions are sufficient for building Chimbuko on Summit, it is advantageous to take advantage of the pre-existing modules for many of the dependencies. For convenience we provide a Spack **environment** which can be used to install in a self-contained environment Chimbuko using various system libraries. To install, first download the Chimbuko and Mochi repositories:
6969

7070
.. code:: bash
7171
72-
source ${AD_SOURCE_DIR}/env.summit.sh
73-
74-
..
75-
We provide :download:`an installation script<files/install_adios2.sh>` for ADIOS2_,
76-
if the latest version is not availale on Summit.
77-
78-
A build of ADIOS2 that has been tested as compatible with Chimbuko can be found `here <../appendix/appendix_install.html#ADIOS2-build>`_.
79-
80-
The process for building Sonata_ is somewhat more complicated on Summit. The Mochi developers recommend using a Spack environment that can be setup as follows:
72+
git clone https://github.com/mochi-hpc/mochi-spack-packages.git
73+
git clone https://github.com/CODARcode/PerformanceAnalysis.git
8174
82-
.. code:: bash
75+
Copy the file :code:`spack/environments/summit.yaml` from the PerformanceAnalysis git repository to a convenient location and edit the paths in the :code:`repos` section to point to the paths at which you downloaded the repositories:
8376

84-
cd ${SDS_REPO_INSTALL_DIR}
85-
git clone https://xgitlab.cels.anl.gov/sds/sds-repo.git
86-
cp ${AD_SOURCE_DIR}/scripts/summit/summit_spack.yaml .
77+
.. code:: yaml
8778
88-
The user must then edit the summit_spack.yaml, replacing line 22 (below "repos:") with the path to the sds-repo directory created by cloning sds-repo.git (i.e. ${SDS_REPO_INSTALL_DIR}/sds-repo)
79+
repos:
80+
- /autofs/nccs-svm1_home1/ckelly/install/mochi-spack-packages
81+
- /autofs/nccs-svm1_home1/ckelly/src/AD/PerformanceAnalysis/spack/repo/chimbuko
8982
90-
Then:
83+
This environment uses the :code:`gcc/9.1.0` and :code:`cuda/11.1.0` modules, which must be loaded prior to installation and running:
9184

9285
.. code:: bash
9386
94-
spack env create mochi summit_spack.yaml
95-
spack env activate mochi
96-
spack install
87+
module load gcc/9.1.0 cuda/11.2.0
9788
98-
The user must henceforth ensure to activate the **mochi** environment prior to building and using Chimbuko as follows:
89+
Then simply create a new environment and install:
9990

10091
.. code:: bash
10192
102-
spack env activate mochi
103-
spack load mochi-sonata
104-
105-
GoogleTest_ is not installed by default on Summit, hence we must install it as described `here <../appendix/appendix_install.html#googletest-on-summit>`_.
93+
spack env create my_chimbuko_env summit.yaml
94+
spack env activate my_chimbuko_env
95+
spack install
10696
107-
To build the AD:
97+
Once installed, simply
10898

10999
.. code:: bash
110100
111-
cd ${AD_SOURCE_DIR}
112-
./autogen.sh
113-
cd ${AD_BUILD_DIR}
114-
115-
CXXFLAGS_IN="-I${GTEST_INSTALL_DIR}/include"
116-
LDFLAGS_IN="-L${GTEST_INSTALL_DIR}/lib64"
117-
118-
CC=mpicc CXX=mpicxx LDFLAGS=${LDFLAGS_IN} CXXFLAGS=${CXXFLAGS_IN} \
119-
/path/to/ad/source/configure --with-adios2=${ADIOS2_INSTALL_DIR} --with-network=ZMQ --with-perf-metric --prefix=${AD_INSTALL_DIR}
120-
make install
121-
101+
spack env activate my_chimbuko_env
102+
spack load tau chimbuko-performance-analysis chimbuko-visualization2
122103
104+
after loading the modules above.
105+
123106
.. _ADIOS2: https://github.com/ornladios/ADIOS2
124107
.. _ZeroMQ: https://zeromq.org/
125108
.. _CURL: https://curl.haxx.se/

0 commit comments

Comments
 (0)