Skip to content

Make tutorial generic for master branch #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
svalat wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Make tutorial generic for master branch #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
db7bc87
master: update the notebook to make it generic for master branch
svalat Mar 19, 2026
06b91af
git: add git ignore
svalat Mar 19, 2026
3009ad2
pip: add python_requirements.txt to install all the python required f…
svalat Mar 19, 2026
3605e8c
restore the ipython configuration in Readme.ipynb
svalat Mar 23, 2026
f8e41ca
strip: add script to strip the notebook before commit (manually for now)
svalat Mar 23, 2026
5d01732
restore the ipython configuration in other notebooks
svalat Mar 23, 2026
da38e3a
replace a path to use relative instead of the os.environ as for the o…
svalat Mar 23, 2026
289c3f2
strip only files from git + fix a quoting issue
svalat Mar 23, 2026
542e8de
remove some still staying refercences to Jureca
svalat Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# cmake generated files
CMakeCache.txt
CMakeFiles
build
Makefile
cmake_install.cmake
idefix
idefix.*.log
generated
git-state.txt
cmake_packages

# data generated by the run
*.vtk
*.dmp

# jupyter tmp files
.ipynb_checkpoints

# python local venv
.venv

7 changes: 3 additions & 4 deletions AdvancedSetup/solution1/read_problem.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,8 @@
"%matplotlib widget\n",
"import os\n",
"import sys\n",
"# These path are specific to this Jureca tutorial\n",
"user=os.getenv(\"USER\")\n",
"sys.path.append(\"/p/project1/training2437/tera_day2/\"+user+\"/idefix/idefix.src/\")\n",
"# Possibly adapt the idefix path depending on where you cloned the sources\n",
"sys.path.append(\"../../../idefix.src\")\n",
"from pytools.vtk_io import readVTK\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np"
Expand Down Expand Up @@ -76,7 +75,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.14"
"version": "3.12.3"
}
},
"nbformat": 4,
Expand Down
139 changes: 35 additions & 104 deletions Readme.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,60 +9,9 @@
"<!-- tocstop -->\n",
"<a id=\"about\"></a>\n",
"# About this tutorial\n",
"This tutorial is provided as a github repository and is mirrored on Jureca for easier access. It is part of the [Toward Exascale-Ready Astrophysics workshop](https://indico3-jsc.fz-juelich.de/event/169/) and has been prepared by Geoffroy Lesur (geoffroy.lesur@univ-grenoble-alpes.fr)\n",
"This tutorial is provided as a github repository. It has been prepared by Geoffroy Lesur (geoffroy.lesur@univ-grenoble-alpes.fr).\n",
"\n",
"In this tutorial, you will learn how to use idefix on various architectures. Here, we will do everything through a Jupyter notebook opened on Jureca on the `dc-gpu` partition, but you can also do most of the tutorial on the CPU of your laptop.\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "5e87953c-007b-4506-8183-62227f5fad5d",
"metadata": {},
"source": [
"# 1- Deployment on Jureca\n",
"\n",
"Log in to https://jupyter-jsc.fz-juelich.de/\n",
"\n",
"Open a lab environment with\n",
"\n",
"- Lab Config:\n",
" - System: JURECA\n",
" - Project: training2437\n",
" - Partition: dc-gpu\n",
" - Reservation: tera_day2\n",
"- Resources (opens once dc-gpu is selected)\n",
" - Nodes: 1\n",
" - GPUs: 4\n",
" - Runtime: 90\n",
"Kernels and extensions: keep defaults\n",
"\n",
"\n",
"First open a new console on your Jupyter notebook. We then clone the idefix Github repository. Since we don't have direct access to the internet, we use a small script to copy the sources and the tutorial from a shared directory:\n",
"\n",
"```shell\n",
"source /p/project1/training2437/tera_day2/idefix/deploy.sh\n",
"```\n",
"\n",
"This will put everything into `/p/project1/training2437/$USER/tera_day2/idefix`\n",
"\n",
"The deploy script already set up the environement (module and environement variable). If you loose connection and need a new console, you can reload the environement:\n",
"\n",
"```shell\n",
"source /p/project1/training2437/tera_day2/idefix/env.sh\n",
"```\n",
"\n",
"You should now open this notebook on your Jupyter instance on Jureca, in\n",
"\n",
"`/p/project1/training2437/$USER/tera_day2/idefix/tutorial/Readme.ipynb`\n"
]
},
{
"cell_type": "markdown",
"id": "75842bfd-0baa-4953-9657-278840d02ec8",
"metadata": {},
"source": [
"In Jupyter, we need to load Idefix environement:"
"In this tutorial, you will learn how to use idefix on various architectures. Here, we will do everything through a Jupyter notebook. Except for some GPU specific parts you can also do most of the tutorial on the CPU of your laptop."
]
},
{
Expand All @@ -76,9 +25,8 @@
"%matplotlib widget\n",
"import os\n",
"import sys\n",
"# These path are specific to this Jureca tutorial\n",
"user=os.getenv(\"USER\")\n",
"sys.path.append(\"/p/project1/training2437/\"+user+\"/tera_day2/idefix/idefix.src/\")\n",
"# Possibly adapt the idefix path depending on where you cloned the sources\n",
"sys.path.append(\"../idefix.src\")\n",
"from pytools.vtk_io import readVTK\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np"
Expand All @@ -89,24 +37,17 @@
"id": "83936685-a050-428e-ac85-93f3c1b973c8",
"metadata": {},
"source": [
"and we're good to go!\n",
"\n",
"## Optionnal: Deploy on your machine\n",
"## Deploy on your machine\n",
"\n",
"<details> \n",
"<summary>Click here to deploy Idefix on your laptop.</summary>\n",
"<br>\n",
"Optionnally, you can play around with this tutorial on your laptop/machine. In this case you can clone this tutorial and idefix source code on your machine, so that you can directly use these source files and test what you are doing. In the directory of your choice (this requires an internet access):\n",
"You can play around with this tutorial on your laptop/machine. In this case you can clone this tutorial and idefix source code on your machine, so that you can directly use these source files and test what you are doing. In the directory of your choice (this requires an internet access):\n",
"\n",
"```shell\n",
"git clone --recurse-submodules https://github.com/idefix-code/idefix.git idefix.src\n",
"export IDEFIX_DIR=$PWD/idefix.src\n",
"git clone https://github.com/idefix-code/tutorial.git\n",
"git clone --branch master https://github.com/idefix-code/tutorial.git\n",
"cd tutorial\n",
"git checkout Jureca\n",
"```\n",
"The last line allows you to reach the dedicated tutorial for Jureca.\n",
"\n",
"For conveniance, we set the `IDEFIX_DIR` environment variable to the absolute path of the root directory of idefix (as above). \n",
"\n",
"If you intend to use the python script provided in this tutorial, best is to deploy a python environement with everything already set up. \n",
Expand All @@ -117,9 +58,7 @@
"python3 -m venv ./env\n",
"source env/bin/activate\n",
"pip install -r python_requirements.txt\n",
"```\n",
"\n",
"</details>\n"
"```"
]
},
{
Expand Down Expand Up @@ -156,9 +95,12 @@
"<a id=\"running\"></a>\n",
"## Run an example\n",
"\n",
"launch the executable\n",
"Launch the executable depending on your running location with either :\n",
"\n",
"```shell\n",
"# On your laptop\n",
"./idefix\n",
"# On a slurm based cluster\n",
"srun -n 1 ./idefix\n",
"```\n",
"\n",
Expand Down Expand Up @@ -190,20 +132,14 @@
"make -j 8\n",
"```\n",
"\n",
"if your build is successful, you can now try to launch idefix with automatic domain decomposition. On a Jureca node (using Slurm):\n",
"\n",
"```shell\n",
"srun -n 4 ./idefix\n",
"```\n",
"\n",
"<details>\n",
"<summary>(optional) On your laptop:</summary>\n",
"if your build is successful, you can now try to launch idefix with automatic domain decomposition. Depending on your run location it can be either :\n",
"\n",
"```shell\n",
"# on your latop\n",
"mpirun -np 4 ./idefix\n",
"```\n",
"</details>\n",
"\n"
"# on a slurm cluster\n",
"srun -n 4 ./idefix\n",
"```\n"
]
},
{
Expand All @@ -214,7 +150,7 @@
"# 3- Configuration, compilation, run on GPUs\n",
"## First tests\n",
"\n",
"For this first test, we are going to run a simple Orszag-Tang test problem on a single GPU. First cd to the right directory\n",
"For this first test, we are going to run a simple Orszag-Tang test problem on a single GPU. First `cd` to the right directory\n",
"\n",
"```shell\n",
"cd $IDEFIX_DIR/test/MHD/OrszagTang\n",
Expand All @@ -240,9 +176,12 @@
"<a id=\"running\"></a>\n",
"### Running the code on GPUs\n",
"\n",
"You then simply launch the executable using srun:\n",
"You then simply launch the executable depending on your running location with either :\n",
"\n",
"```shell\n",
"# Run locally\n",
"./idefix\n",
"# Run in a slurm cluster.\n",
"srun -n 1 ./idefix\n",
"```\n",
"\n",
Expand All @@ -251,7 +190,7 @@
"<a id=\"mpi\"></a>\n",
"### Multi-GPUs runs\n",
"\n",
"Idefix can run on multiple GPUs (it's been tested on +4000 GPUs simultaneously). This requires an MPI installation compatible with Cuda (e.g. GPU-aware OpenMPI). If you have loaded the environement in [Getting Started](README.md), you should be able to compile a GPU version of Idefix with parallelisation support.\n",
"Idefix can run on multiple GPUs (it's been tested on +4000 GPUs simultaneously). This requires an MPI installation compatible with Cuda (e.g. GPU-aware OpenMPI). If you have loaded the required environement into your cluster about your network and GPU stack, you should be able to compile a GPU version of Idefix with parallelisation support.\n",
"\n",
"You should first configure the code with CMake adding `-DIdefix_MPI=ON` to the command line and compile. \n",
"\n",
Expand All @@ -263,6 +202,7 @@
"If the compilation succeeds, then you can run a multi-GPU simulation with 4 gpus:\n",
"\n",
"```shell\n",
"# On a slurm cluster\n",
"srun -n 4 ./idefix\n",
"```\n",
"\n",
Expand Down Expand Up @@ -328,12 +268,11 @@
"\n",
"### Configure the code, build and run it\n",
"\n",
"Follow the instruction in the [Getting Started](../GettingStarted/README.md#compile-an-example) section. Note that you can run your setup on a CPU or an GPU, choosing the right configuration with `Cmake`.\n",
"Follow the instruction in the [Getting Started](#compilation) section. Note that you can run your setup on a CPU or an GPU, choosing the right configuration with `Cmake`.\n",
"\n",
"### Check the outputs\n",
"\n",
"To visualize the flow, you may use Paraview or visit to open the files generated by Idefix. On Jureca, you can open the notebook `read_problem.ipynb` in `SimpleSetup/problem1` and execute it to visualize the result directly.\n",
"\n"
"To visualize the flow, you may use Paraview or visit to open the files generated by Idefix. You can also open the notebook `read_problem.ipynb` in `SimpleSetup/problem1` and execute it to visualize the result directly."
]
},
{
Expand All @@ -344,7 +283,7 @@
"outputs": [],
"source": [
"# Load the last VTK file produced by Idefix\n",
"V=readVTK(\"/p/project1/training2437/\"+user+\"/tera_day2/idefix/tutorial/SimpleSetup/problem1/data.0005.vtk\")"
"V=readVTK(\"./SimpleSetup/problem1/data.0005.vtk\")"
]
},
{
Expand All @@ -358,7 +297,7 @@
"for field in V.data.keys():\n",
" plt.figure(figsize=(10,4))\n",
" plt.pcolormesh(V.x,V.y,V.data[field][:,:,0].T)\n",
" plt.title(field+ \" @ t=%f\"%V.t)\n",
" plt.title(field+ \" @ t=%f\"%V.t[0])\n",
" plt.colorbar()\n",
" plt.gca().set_aspect('equal')"
]
Expand All @@ -383,7 +322,7 @@
"source": [
"plt.figure(figsize=(10,4))\n",
"plt.pcolormesh(V.x,V.y,wz[:,:,0].T)\n",
"plt.title(r\"$\\omega_z$ @ t=%f\"%V.t)\n",
"plt.title(r\"$\\omega_z$ @ t=%f\"%V.t[0])\n",
"plt.colorbar()\n",
"plt.gca().set_aspect('equal')"
]
Expand All @@ -397,7 +336,7 @@
"\n",
"Try to compile the code for a single GPU (without MPI). The initial resolution is $256\\times 64$. Edit `idefix.ini` and try to increase the resolution to $512\\times 128$ and $1024\\times 256$ and look at the final performances. You should observe a strong dependence on the problem size. This is because for GPU computing to be efficient, you need to keep busy the 1000s of compute core of your GPU. This typically means that you need at least about 1 million cells on modern GPUs for Idefix to reach its full efficiency.\n",
"\n",
"These are the typical performances measured on Jureca on a single Nvidia A100 GPU:\n",
"These are the typical performances measured on a single Nvidia A100 GPU:\n",
"- $256\\times 64$: 6.1e7 cell/s\n",
"- $512\\times 128$: 2.2e8 cell/s\n",
"- $1024\\times 256$: 5.9e8 cell/s\n",
Expand Down Expand Up @@ -506,7 +445,7 @@
"source": [
"# Read output file after 3 orbits\n",
"\n",
"V=readVTK(\"/p/project1/training2437/\"+user+\"/tera_day2/idefix/tutorial/AdvancedSetup/problem1/data.0003.vtk\")\n",
"V=readVTK(\"./AdvancedSetup/problem1/data.0003.vtk\")\n",
"# define cartesian coordinates from radius (xl) and azimuth (yl) in the VTK file\n",
"x=V.xl[:,None]*np.cos(V.yl[None,:])\n",
"y=V.xl[:,None]*np.sin(V.yl[None,:])\n",
Expand Down Expand Up @@ -546,7 +485,7 @@
"source": [
"# Read output file after 3 orbits\n",
"\n",
"V=readVTK(\"/p/project1/training2437/\"+user+\"/tera_day2/idefix/tutorial/AdvancedSetup/problem1/data.0003.vtk\")\n",
"V=readVTK(\"./AdvancedSetup/problem1/data.0003.vtk\")\n",
"# define cartesian coordinates from radius (xl) and azimuth (yl) in the VTK file\n",
"x=V.xl[:,None]*np.cos(V.yl[None,:])\n",
"y=V.xl[:,None]*np.sin(V.yl[None,:])\n",
Expand Down Expand Up @@ -646,7 +585,7 @@
"## Problem1: a CPU segmentation fault\n",
"\n",
"### Base run\n",
"The first problem is a simple 1D shock tube problem. This can be compiled and run on your laptop or on Jureca.\n",
"The first problem is a simple 1D shock tube problem. This can be compiled and run on your laptop or on a cluster.\n",
"\n",
"```shell\n",
"cd $IDEFIX_ROOT/tutorial/Debugging/problem1\n",
Expand Down Expand Up @@ -850,7 +789,7 @@
"\n",
"## Problem 4: a low performance bug.\n",
"\n",
"Let's move to problem 4, which is again a planet-disk interraction problem. This can be compiled and run *on your laptop* or on the Jureca cluster, but let's focus for now on the GPU version on the Jureca cluster (you can try to do the exercise on your laptop). First go to the right directory\n",
"Let's move to problem 4, which is again a planet-disk interraction problem. This can be compiled and run *on your laptop* or on the on a cluster, but let's focus for now on the GPU version on the a cluster (you can try to do the exercise on your laptop). First go to the right directory\n",
"\n",
"```shell\n",
"cd $IDEFIX_ROOT/tutorial/Debugging/problem4\n",
Expand Down Expand Up @@ -897,14 +836,6 @@
"</p>\n",
"</details>\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a09fb8a4-9723-4671-babb-44b51dd98274",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
Expand All @@ -923,7 +854,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.14"
"version": "3.12.3"
}
},
"nbformat": 4,
Expand Down
19 changes: 13 additions & 6 deletions SimpleSetup/problem1/read_problem.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,8 @@
"%matplotlib widget\n",
"import os\n",
"import sys\n",
"# These path are specific to this Jureca tutorial\n",
"user=os.getenv(\"USER\")\n",
"sys.path.append(\"/p/project1/training2437/tera_day2/\"+user+\"/idefix/idefix.src/\")\n",
"# Possibly adapt the idefix path depending on where you cloned the sources\n",
"sys.path.append(\"../../../idefix.src\")\n",
"from pytools.vtk_io import readVTK\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np"
Expand Down Expand Up @@ -42,7 +41,7 @@
"for field in V.data.keys():\n",
" plt.figure(figsize=(10,4))\n",
" plt.pcolormesh(V.x,V.y,V.data[field][:,:,0].T)\n",
" plt.title(field+ \" @ t=%f\"%V.t)\n",
" plt.title(field+ \" @ t=%f\"%V.t[0])\n",
" plt.colorbar()\n",
" plt.gca().set_aspect('equal')"
]
Expand All @@ -67,10 +66,18 @@
"source": [
"plt.figure(figsize=(10,4))\n",
"plt.pcolormesh(V.x,V.y,wz[:,:,0].T)\n",
"plt.title(r\"$\\omega_z$ @ t=%f\"%V.t)\n",
"plt.title(r\"$\\omega_z$ @ t=%f\"%V.t[0])\n",
"plt.colorbar()\n",
"plt.gca().set_aspect('equal')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "eb4a74e5-b88b-4be3-bcb9-a7c7d9086e74",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
Expand All @@ -89,7 +96,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.14"
"version": "3.12.3"
}
},
"nbformat": 4,
Expand Down
Loading