Skip to content

Commit 983e0ab

Browse files
authored
Document how to merge all Grid subjobs for a run by hand (#1599)
* Document how to merge all Grid subjobs for a run by hand * Update DevelopersTips.md
1 parent de2741e commit 983e0ab

1 file changed

Lines changed: 39 additions & 0 deletions

File tree

doc/DevelopersTips.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -394,3 +394,42 @@ query to see for a given run the number of objects per detector:
394394
```
395395
select ccdb.metadata -> '1337188343' as detector, count(distinct path), SUM(COUNT(distinct path)) from ccdb, ccdb_paths where ccdb_paths.pathid = ccdb.pathid AND ccdb.metadata -> '1048595860' = '529439' group by detector;
396396
```
397+
398+
### Merge and upload QC results for all subjobs of a grid job
399+
400+
Please keep in mind that the file pattern in Grid could have changed since this was written.
401+
```
402+
#!/usr/bin/env bash
403+
set -e
404+
set -x
405+
set -u
406+
407+
# we get the list of all QC.root files for o2_ctf and o2_rawtf directories
408+
alien_find /alice/data/2022/LHC22m/523821/apass2_cpu 'o2_*/QC.root' > qc.list
409+
# we add alien:// prefix, so ROOT knows to look for them in alien
410+
sed -i -e 's/^/alien:\/\//' qc.list
411+
# we split the big list into smallers ones of -l lines, so we can parallelize the processing
412+
# one can play with the -l parameter
413+
split -d -l 25 qc.list qc_list_
414+
415+
# for each split file run the merger executable
416+
for QC_LIST in qc_list_*
417+
do
418+
o2-qc-file-merger --enable-alien --input-files-list "${QC_LIST}" --output-file "merged_${QC_LIST}.root" &
419+
done
420+
421+
# wait for the jobs started in the loop
422+
wait $(jobs -p)
423+
424+
# we merge the files of the first "stage"
425+
o2-qc-file-merger --input-files merged_* --output-file QC_fullrun.root
426+
427+
# we take the first QC config file we find and use it to perform the remote-batch QC
428+
CONFIG=$(alien_find /alice/data/2022/LHC22m/523821/apass2_cpu QC_production.json | head -n 1)
429+
if [ -n "$CONFIG" ]
430+
then
431+
alien_cp "$CONFIG" file://QC_production.json
432+
# we override activity values, as QC_production.json might have only placeholders
433+
o2-qc --remote-batch QC_fullrun.root --config "json://QC_production.json" -b --override-values "qc.config.Activity.number=523897;qc.config.Activity.passName=apass2;qc.config.Activity.periodName=LHC22m"
434+
fi
435+
```

0 commit comments

Comments
 (0)