updated scipts to correctly run all the executables and install necessary tool programs

Author: gregbolet

Dockerfile

@@ -9,12 +9,17 @@ SHELL ["/bin/bash", "-c"]
 # Update packages, install build dependencies, and the specified version of CMake
 RUN apt-get update && \
     apt-get upgrade -y && \
-    apt-get install -y wget make git gfortran libomp-18-dev libboost-all-dev clang-18 clang-tools-18 && \
+    apt-get install -y wget make git gfortran libomp-18-dev libboost-all-dev clang-18 clang-tools-18 unzip && \
     wget https://github.com/Kitware/CMake/releases/download/v3.28.0/cmake-3.28.0-linux-x86_64.sh && \
     chmod +x cmake-3.28.0-linux-x86_64.sh && \
     ./cmake-3.28.0-linux-x86_64.sh --skip-license --prefix=/usr/local && \
     rm cmake-3.28.0-linux-x86_64.sh
 
+# setup alternative names for the compiler
+RUN update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-18 100 && \
+    update-alternatives --install /usr/bin/clang clang /usr/bin/clang-18 100 && \
+    update-alternatives --install /usr/bin/llvm-cxxfilt llvm-cxxfilt /usr/bin/llvm-cxxfilt-18 100
+
 
 # Set the working directory
 WORKDIR /gpu-flopbench

README.md

@@ -46,12 +46,13 @@ For checking the build process, we include the following Github Action in the `.
 This sets up a Docker container with all the necessary programs and code to build all the HecBench codes with our CMakeLists approach.
 We also provide a `Dockerfile` so you can set up and run our code on your own system with a GPU.
 
-[![Build ALL CUDA/OMP Codes](https://github.com/gregbolet/HeCBench-roofline/actions/workflows/buildAllCodesGithubAction.yml/badge.svg)](https://github.com/gregbolet/HeCBench-roofline/actions/workflows/buildAllCodesGithubAction.yml)
+[![Build ALL CUDA/OMP Codes](https://github.com/gregbolet/gpu-flopbench/actions/workflows/buildAllCodesGithubAction.yml/badge.svg)](https://github.com/gregbolet/gpu-flopbench/actions/workflows/buildAllCodesGithubAction.yml)
 
 ## Building
 
 Execute the following command to get the Makefile generated and to start the build process.
 This will automatically `make` all the programs, **you'll NEED to edit the `runBuild.sh` script to properly set any compilers/options for the codes to build**.
+NOTE: If you're running this from a Docker container generated from our Dockerfile, it should work out-of-the-box.
 By default, we have everything building with `clang++` and `clang`, this should mostly work out-of-the-box but some include paths may need to be set/overriden. (SEE BELOW)
 ```
 source ./runBuild.sh
@@ -79,10 +80,11 @@ Here's a list of other common build issues that might help if you're encounterin
 We used Python3 (v3.11.11) for executing our Python scripts.
 The `requirements.txt` file contains all the necessary packages and their versions that should be installed prior to using any of our Python scripts.
 It is strongly advised to set up a new Conda environment to not mess up the base Python installation on your system.
+NOTE: This is already done for you if you're using the supplied Dockerfile.
 
 ```
-conda create --name "hecbench-roofline" python=3.11.11
-conda activate hecbench-roofline
+conda create --name "gpu-flopbench" python=3.11.11
+conda activate gpu-flopbench
 pip install -r ./requirements.txt
 ```
 
@@ -91,12 +93,12 @@ pip install -r ./requirements.txt
 Once all the codes are built, we can start the data collection process. We have our own script called `gatherData.py` which can be invoked to gather the roofline benchmarking data of each of the built programs.
 
 ```
-LD_LIBRARY_PATH=/usr/lib/llvm-18/lib:$LD_LIBRARY_PATH DATAPATH=/home/gbolet/hecbench-roofline/src/prna-cuda/data_tables python3 ./gatherData.py --outfile=roofline-data.csv 2>&1 | tee runlog.txt
+LD_LIBRARY_PATH=/usr/lib/llvm-18/lib:$LD_LIBRARY_PATH DATAPATH=$PWD/src/prna-cuda/data_tables python3 ./gatherData.py --outfile=roofline-data.csv 2>&1 | tee runlog.txt
 ```
 
 This will automatically invoke each of the built executables, using `ncu` (NVIDIA Nsight Compute) to profile each of the kernels in the executable. Some of the codes require files to be downloaded proir, this script takes care of the downloading process and makes sure that all the requested files are in place.
 The `DATAPATH` environment variable is only needed by `frna-cuda` and `prna-cuda`, so if you're not running those, you can drop it.
-The `LD_LIBRARY_PATH` environment variable is for all the OMP codes, this is the path to the `libomptarget.so` library. One some machines CMake isn't adding the path, so we just manually add it. We should probably `rpath` this in in the future, but for now this is fine.
+The `LD_LIBRARY_PATH` environment variable is for all the OMP codes, this is the path to the `libomptarget.so` library. On some machines CMake isn't adding the path, so we just manually add it. We should probably `rpath` this in in the future, but for now this is fine.
 
 The internal workflow at a high level looks like the following:
 1. Download rodinia dataset (skip if requested with `--skipRodiniaDownload`).

gatherData.py

@@ -69,7 +69,8 @@ def has_rodinia_datasets():
 def download_rodinia_and_extract():
     print('Downloading Rodinia Data...')
 
-    command = f'wget http://www.cs.virginia.edu/~skadron/lava/Rodinia/Packages/rodinia_3.1.tar.bz2 && tar -xf ./rodinia_3.1.tar.bz2 rodinia_3.1/data && mv ./rodinia_3.1/data {SRC_DIR}/data'
+    # http://www.cs.virginia.edu/~skadron/lava/rodinia/Packages/rodinia_3.1.tar.bz2 
+    command = f'wget http://www.cs.virginia.edu/~skadron/lava/rodinia/Packages/rodinia_3.1.tar.bz2 && tar -xf ./rodinia_3.1.tar.bz2 rodinia_3.1/data && mv ./rodinia_3.1/data {SRC_DIR}/data'
     print('executing command', command)
     result = subprocess.run(command, cwd=DOWNLOAD_DIR, shell=True)
 
@@ -80,8 +81,12 @@ def download_rodinia_and_extract():
 
     return
 
-def run_setup_command_for_file(targetFile, command, targetDir=DOWNLOAD_DIR):
+def run_setup_command_for_file(targetFile, command, targetDir=None):
+    if targetDir is None:
+        targetDir = DOWNLOAD_DIR
     if not os.path.isfile(targetFile):
+        print('running command: ', command)
+        print('targetDir: ', targetDir)
         result = subprocess.run(args=command, cwd=targetDir, shell=True)
         assert result.returncode == 0
     return
@@ -166,7 +171,7 @@ def download_files_for_some_targets(targets):
                 result = subprocess.run(command, cwd=DOWNLOAD_DIR, shell=True)
                 assert result.returncode == 0
 
-        elif basename == 'opticalFlow-':
+        elif 'opticalFlow-' in basename:
             if not os.path.isfile(f'{srcDir}/frame10.ppm'):
                 command = f'wget --no-check-certificate https://github.com/NVIDIA/cuda-samples/raw/c94ff366aed18c797b8a85dfaac7817b0228b420/Samples/5_Domain_Specific/HSOpticalFlow/data/frame10.ppm && mv ./frame10.ppm {srcDir}/'
                 result = subprocess.run(command, cwd=DOWNLOAD_DIR, shell=True)
@@ -176,13 +181,15 @@ def download_files_for_some_targets(targets):
                 result = subprocess.run(command, cwd=DOWNLOAD_DIR, shell=True)
                 assert result.returncode == 0
 
-        elif basename == 'tsne-':
+        elif 'tsne-' in basename:
             if not os.path.isfile(f'{srcDir}/data/points.txt'):
                 command = f'wget --no-check-certificate https://raw.githubusercontent.com/oneapi-src/Velocity-Bench/refs/heads/main/tsne/data/points.txt && mv ./points.txt {srcDir}/data/points.txt'
                 result = subprocess.run(command, cwd=DOWNLOAD_DIR, shell=True)
                 assert result.returncode == 0
 
-        elif basename == 'tpacf-':
+        elif 'tpacf-' in basename:
+            result = subprocess.run(f'mkdir -p ./data/small', cwd=srcDir, shell=True)
+            assert result.returncode == 0
             # this only downloads the two files necessary for the beginning of the program to hit the kernels for profiling
             if not os.path.isfile(f'{srcDir}/data/small/Datapnts.1'):
                 command = f'wget --no-check-certificate https://raw.githubusercontent.com/fengzhangcs/CoRunBench/refs/heads/master/parboil/datasets/tpacf/small/input/Datapnts.1 && mv ./Datapnts.1 {srcDir}/data/small/Datapnts.1'
@@ -246,7 +253,7 @@ def download_files_for_some_targets(targets):
 
         elif basename == 'mcpr-cuda':
             if not os.path.isfile(f'{srcDir}/alphas.csv'):
-                command = f'bunzip2 alphas.csv.bz2'
+                command = f'wget --no-check-certificate https://github.com/zjin-lcf/HeCBench/raw/refs/heads/master/src/mcpr-cuda/alphas.csv.bz2 && bunzip2 alphas.csv.bz2'
                 result = subprocess.run(command, cwd=srcDir, shell=True)
                 assert result.returncode == 0
 
@@ -271,8 +278,10 @@ def download_files_for_some_targets(targets):
         # need to have python2 installed for this to work
         elif basename == 'heat2d-cuda':
             if not os.path.isfile(f'{srcDir}/data.txt'):
-                command = f'python2 mkinit.py 4096 8192 data.txt'
-                result = subprocess.run(command, cwd=srcDir, shell=True)
+                command = f'python3 mkinit.py 4096 8192 data.txt'
+                result = subprocess.run(command, cwd=srcDir, shell=True, capture_output=True)
+                print(result.stdout)
+                print(result.stderr)
                 assert result.returncode == 0
 
         elif (basename == 'frna-cuda') or (basename == 'prna-cuda'):
@@ -337,6 +346,7 @@ def get_runnable_targets():
 
             # check we have the source code too
             assert os.path.isdir(execSrcDir)
+            assert execSrcDir != ''
 
             execDict = {'basename':basename, 
                         'exe':entry, 
@@ -534,7 +544,7 @@ def search_and_extract_file(inFile):
             print('Extracted zip archive:', filename)
 
         # now let's check that the file exists
-        assert os.path.exists(inFile)
+        assert os.path.exists(inFile), f'Could not find input file: {inFile} after extracting archives!'
 
     return
 
@@ -551,7 +561,7 @@ def check_and_unzip_input_files(targets:list):
         if len(inputFiles) > 0:
             for inFile in inputFiles:
                 # if the word `output` is in the filename, we skip checking for it
-                if not ('output' in inFile.lower()):
+                if not ('output' in inFile.lower() or 'results' in inFile.lower()):
                     print(f'looking for input file: {srcDir}/{inFile}')
                     search_and_extract_file(f'{srcDir}/{inFile}')
 

src/heat2d-cuda/mkinit.py

@@ -12,7 +12,7 @@ def main(argv=None):
     try:
         try:
             opts, args = getopt.getopt(argv[1:], "h", ["help"])
-        except getopt.GetoptError, msg:
+        except getopt.GetoptError as msg:
             raise Usage(msg)
 
         for o, a in opts:
@@ -29,7 +29,7 @@ def main(argv=None):
         if len(args) != 3:
             raise Usage(" Usage: %s [OPTIONS] LX LY FILE_NAME" % argv[0])
 
-    except Usage, err:
+    except Usage as err:
         print(err.msg)
         print(" for help use --help")
         return 2