Skip to content

Commit 9d46036

Browse files
franzpoeschelax3l
franzpoeschel
and
ax3l
authored
Wheels: ADIOS2 -> v2.9.0, blosc2 (#1387)
* Update blosc -> blosc2, ADIOS2 -> v2.9.0-rc1 ADIOS2 should be replace by v2.9.0 finally * Fix Blosc2 CMake params * Guard against CMAKE_OSX_ARCHITECTURES * For now, deactivate the pthread patch in ADIOS2 * Test small patch for blosc2 CMake * Add -fPIC flag * MacOS compatibility * -DADIOS2_USE_Blosc2=ON * Build atl and ffs manually to avoid CMake import error * Revert "Build atl and ffs manually to avoid CMake import error" This reverts commit c5469a3. * RC1 -> full release * Don't build ADIOS1 * -DADIOS2_USE_MHS=OFF * Use -DADIOS2_INSTALL_GENERATE_CONFIG=OFF workaround See ornladios/ADIOS2#3348 (comment) * Do not disable find calls * Revert "Do not disable find calls" This reverts commit 7065b89. * Try setting ADIOS2_PATH * Keep ADIOS1 + c-blosc1 * Undo `build.yml` Changes * ADIOS2: no c-blosc1 pthread issue * HDF5: 1.14.1-2 * macOS CI: Now has 3 cores * C-Blosc2 Updates - CMake options - Windows - build stamp * Windows Updates * HDF5 Patches * C-Blosc 2.9.3 + CMake Patch * ARM: H5Detect * Windows: Avoid CXX20 in C-Blosc2 * macOS arm64 HDF5: H5detect cross-compile * HDF5 arm64: cross-compiling H5detect * [Patch] ADIOS2 c-blosc2 windows.h std::min * [Patch] C-Blosc2: External Zlib * [Hack] Windows: Ignore Ext. ZLIB for c-blosc2 Just build again and close eyes. * ppc64le: Skip C-Blosc & ADIOS1 Due to time limits on Travis-CI. * Unix: Keep HDF5 1.12.2 for Now - issues with cross-compile cannot quickly be solved for macOS arm64/aarch64 - disable HDF5 support in ADIOS2 for now * [Patch] C-Blosc2 PUBLIC Linkage Zlib * Use Blosc2Config.cmake Package * Cleanup and `cat` ADIOS2 Config File * [Patch] [Hack] CMake --trace-expand * Cleanup: cat adios2-config.cmake * [Travis] PPC64le: `travis_wait 30` Takes longer now w/o output to the terminal than the default wait time of 10min. https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received * [ADIOS2] CMake --trace-expand * Undo Trace Expand Hacks * [Travis] PPC64le: `travis_wait 45` * ADIOS2: w/ SST * Blosc2: CMake Config PR Merged * macOS (x86): 10.15 -> 11.0 The macOS 10.15 build images on GH Actions are gone, so we remove this EOL platform for good now. * Windows: Skip Building Blosc1 Co-authored-by: Franz Pöschel <franz.poeschel@gmail.com> * Python: 3.8+ Skip older Python releases, 3.7 is EOL since last month. * [Patch] ADIOS2: Remove FindBlosc2.cmake * C-Blosc 2.10.1 * Win: ADIOS 2.9.1 * ADIOS2_Blosc2_PREFER_SHARED=OFF ornladios/ADIOS2#3715 (comment) * cibuildwheel 2.12.1 -> 2.14.1 For `pp310-manylinux_aarch64` --------- Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
1 parent 25dec08 commit 9d46036

4 files changed

Lines changed: 150 additions & 104 deletions

File tree

‎.github/workflows/build.yml‎

Lines changed: 21 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -40,10 +40,10 @@ jobs:
4040
CMAKE_GENERATOR: "Visual Studio 16 2019"
4141
CMAKE_GENERATOR_PLATFORM: "Win32"
4242

43-
- os: macos-10.15
43+
- os: macos-11
4444
arch: "x86_64"
4545
env:
46-
MACOSX_DEPLOYMENT_TARGET: 10.15
46+
MACOSX_DEPLOYMENT_TARGET: 11.0
4747

4848
# Apple Silicon M1/arm64/aarch64 builds:
4949
# https://cibuildwheel.readthedocs.io/en/stable/faq/#apple-silicon
@@ -79,55 +79,45 @@ jobs:
7979
- uses: actions/setup-python@v1
8080
name: Install Python
8181
with:
82-
python-version: '3.7'
82+
python-version: '3.8'
8383

8484
- name: Install cibuildwheel
8585
run: |
8686
python -m pip install --upgrade pip setuptools wheel
87-
python -m pip install cibuildwheel==2.12.1
88-
89-
# # 0.15.0: CMake: Fix Python Install Directory
90-
# # https://github.com/openPMD/openPMD-api/pull/1393
91-
# - name: Download Patch 1/2
92-
# uses: suisei-cn/actions-download-file@v1
93-
# id: patch1
94-
# with:
95-
# url: "https://github.com/ax3l/openPMD-api/commit/b622cc5ea770f866c1e373185a9e389c04bdb54c.patch"
96-
# target: src/.patch/
97-
#
98-
# # 0.15.0.post2 bump
99-
# - name: Download Patch 2/2
100-
# uses: suisei-cn/actions-download-file@v1
101-
# id: setupversion
102-
# with:
103-
# url: "https://gist.githubusercontent.com/ax3l/5e83edefe4b05cf6aa2a971649285fe0/raw/023cbd69e06715e5191ece741bb1de56560a9f96/0001-Bump-setup.py.patch"
104-
# target: src/.patch/
105-
106-
# - name: Apply Patches
107-
# run: |
108-
# python -m pip install "patch==1.*"
109-
# cd src
110-
# python -m patch .patch/b622cc5ea770f866c1e373185a9e389c04bdb54c.patch
111-
# python -m patch .patch/0001-Bump-setup.py.patch
87+
python -m pip install cibuildwheel==2.14.1
88+
89+
# Hack: cmake --trace-expand
90+
#- name: Download Patch 2/2
91+
# uses: suisei-cn/actions-download-file@v1
92+
# id: setupversion
93+
# with:
94+
# url: "https://gist.githubusercontent.com/ax3l/9b15c5bc73d7f3c1c66784187a016474/raw/e31dd54498c9c8fc465a7cbb6f900f768b9d0c0b/0001-Hack-setup.py-CMake-trace-expand.patch"
95+
# target: src/.patch/
96+
97+
#- name: Apply Patches
98+
# run: |
99+
# python -m pip install "patch==1.*"
100+
# cd src
101+
# python -m patch .patch/0001-Hack-setup.py-CMake-trace-expand.patch
112102

113103
- name: Build wheel
114104
env:
115105
# (1) Disable PyPy (manylinux image: yum repo issues)
116106
# https://github.com/pypa/manylinux/issues/899
117107
CIBW_SKIP: "pp*-manylinux*"
118108
CIBW_ARCHS: "${{ matrix.arch }}"
119-
CIBW_PROJECT_REQUIRES_PYTHON: ">=3.7"
109+
CIBW_PROJECT_REQUIRES_PYTHON: ">=3.8"
120110
# Install dependencies
121111
CIBW_BEFORE_BUILD_LINUX: bash -x .github/library_builders.sh
122112
CIBW_BEFORE_BUILD_MACOS: bash -x .github/library_builders.sh
123113
CIBW_BEFORE_BUILD_WINDOWS: 'cmd /E:ON /V:ON /C .github\library_builders.bat'
124114
# for the openPMD-api build, CMake shall search for
125115
# static dependencies of HDF5 and ADIOS1 (see setup.py)
126116
CIBW_ENVIRONMENT: HDF5_USE_STATIC_LIBRARIES='ON' ZLIB_USE_STATIC_LIBS='ON' ADIOS_USE_STATIC_LIBS='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON'
127-
CIBW_ENVIRONMENT_WINDOWS: HDF5_USE_STATIC_LIBRARIES='ON' ZLIB_USE_STATIC_LIBS='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON' CMAKE_PREFIX_PATH='C:/Program Files (x86)/ADIOS2;C:/Program Files (x86)/blosc;C:/Program Files (x86)/HDF5;C:/Program Files (x86)/ZFP;C:/Program Files (x86)/zlib'
117+
CIBW_ENVIRONMENT_WINDOWS: HDF5_USE_STATIC_LIBRARIES='ON' ZLIB_USE_STATIC_LIBS='ON' openPMD_CMAKE_openPMD_USE_HDF5='ON' openPMD_CMAKE_openPMD_USE_ADIOS2='ON' CMAKE_PREFIX_PATH='C:/Program Files (x86)/ADIOS2;C:/Program Files (x86)/blosc;C:/Program Files (x86)/blosc2;C:/Program Files (x86)/HDF5;C:/Program Files (x86)/ZFP;C:/Program Files (x86)/zlib'
128118
# C++17 support in macOS 10.13+ (partial) and 10.14+ (std::visit) and 10.15+ (std::filesystem::path)
129119
# https://cibuildwheel.readthedocs.io/en/stable/cpp_standards/#macos-and-deployment-target-versions
130-
# arm64 Python interpreters are built with 11.0
120+
# arm64/aarch64 (M1/M2) requires 11.0+
131121
MACOSX_DEPLOYMENT_TARGET: "${{ matrix.env.MACOSX_DEPLOYMENT_TARGET }}"
132122
CMAKE_OSX_ARCHITECTURES: "${{ matrix.env.CMAKE_OSX_ARCHITECTURES }}"
133123
# Windows cross-compilation

‎.travis.yml‎

Lines changed: 8 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ env:
1010
global:
1111
- OPENPMD_GIT_REF="0.15.1"
1212

13-
- CIBW_PROJECT_REQUIRES_PYTHON=">=3.7"
13+
- CIBW_PROJECT_REQUIRES_PYTHON=">=3.8"
1414
# Install dependencies on Linux and OSX
1515
- CIBW_BEFORE_BUILD="bash -x .github/library_builders.sh"
1616
# for the openPMD-api build, CMake shall search for
@@ -28,17 +28,17 @@ jobs:
2828
arch: arm64
2929
dist: focal
3030
env:
31-
- CIBW_BUILD="cp37-manylinux_aarch64 cp38-manylinux_aarch64"
31+
- CIBW_BUILD="cp38-manylinux_aarch64 cp39-manylinux_aarch64"
3232
- services: docker
3333
arch: arm64
3434
dist: focal
3535
env:
36-
- CIBW_BUILD="cp39-manylinux_aarch64 cp310-manylinux_aarch64"
36+
- CIBW_BUILD="cp310-manylinux_aarch64 cp311-manylinux_aarch64"
3737
- services: docker
3838
arch: arm64
3939
dist: focal
4040
env:
41-
- CIBW_BUILD="cp311-manylinux_aarch64 pp37-manylinux_aarch64"
41+
- CIBW_BUILD="pp310-manylinux_aarch64"
4242
- services: docker
4343
arch: arm64
4444
dist: focal
@@ -49,24 +49,14 @@ jobs:
4949
arch: arm64
5050
dist: focal
5151
env:
52-
- CIBW_BUILD="cp37-musllinux_aarch64 cp38-musllinux_aarch64"
52+
- CIBW_BUILD="cp38-musllinux_aarch64 cp39-musllinux_aarch64"
5353
- services: docker
5454
arch: arm64
5555
dist: focal
5656
env:
57-
- CIBW_BUILD="cp39-musllinux_aarch64 cp310-musllinux_aarch64"
58-
- services: docker
59-
arch: arm64
60-
dist: focal
61-
env:
62-
- CIBW_BUILD="cp311-musllinux_aarch64"
57+
- CIBW_BUILD="cp310-musllinux_aarch64 cp311-musllinux_aarch64"
6358

6459
# perform a linux PPC64LE build
65-
- services: docker
66-
arch: ppc64le
67-
dist: focal
68-
env:
69-
- CIBW_BUILD="cp37-manylinux_ppc64le"
7060
- services: docker
7161
arch: ppc64le
7262
dist: focal
@@ -87,11 +77,6 @@ jobs:
8777
dist: focal
8878
env:
8979
- CIBW_BUILD="cp311-manylinux_ppc64le"
90-
- services: docker
91-
arch: ppc64le
92-
dist: focal
93-
env:
94-
- CIBW_BUILD="cp37-musllinux_ppc64le"
9580
- services: docker
9681
arch: ppc64le
9782
dist: focal
@@ -131,7 +116,7 @@ install:
131116
- git clone --branch ${OPENPMD_GIT_REF} --depth 1 https://github.com/openPMD/openPMD-api.git src
132117
- cp library_builders.sh src/.github/
133118
- python -m pip install --upgrade pip setuptools wheel
134-
- python -m pip install cibuildwheel==2.12.1
119+
- python -m pip install cibuildwheel==2.14.1
135120
# twine & cryptography: see
136121
# https://github.com/scikit-build/cmake-python-distributions/blob/4730aeee240917303f293dffc89a8d8d5a4787c4/requirements-deploy.txt
137122
# https://github.com/pyca/cryptography/issues/6086
@@ -155,7 +140,7 @@ install:
155140

156141
script:
157142
- cd src
158-
- python3 -m cibuildwheel --output-dir ../wheelhouse
143+
- travis_wait 45 python3 -m cibuildwheel --output-dir ../wheelhouse
159144
- cd ..
160145

161146
deploy:

‎library_builders.bat‎

Lines changed: 45 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -16,25 +16,22 @@ exit /b 0
1616

1717
:build_adios2
1818
if exist adios2-stamp exit /b 0
19-
curl -sLo adios2-2.7.1.zip ^
20-
https://github.com/ornladios/ADIOS2/archive/v2.7.1.zip
21-
powershell Expand-Archive adios2-2.7.1.zip -DestinationPath dep-adios2
19+
curl -sLo adios2-2.9.1.zip ^
20+
https://github.com/ornladios/ADIOS2/archive/v2.9.1.zip
21+
powershell Expand-Archive adios2-2.9.1.zip -DestinationPath dep-adios2
2222

23-
:: Patch Blosc Compression w/ ADIOS 2.7.1
24-
curl -sLo adios2-blosc.patch ^
25-
https://patch-diff.githubusercontent.com/raw/ornladios/ADIOS2/pull/2746.patch
26-
python -m patch -p 1 -d dep-adios2/ADIOS2-2.7.1 adios2-blosc.patch
27-
28-
cmake -S dep-adios2/ADIOS2-2.7.1 -B build-adios2 ^
23+
cmake -S dep-adios2/ADIOS2-2.9.1 -B build-adios2 ^
2924
-DCMAKE_BUILD_TYPE=Release ^
3025
-DBUILD_SHARED_LIBS=OFF ^
3126
-DBUILD_TESTING=OFF ^
3227
-DADIOS2_USE_MPI=OFF ^
3328
-DADIOS2_BUILD_EXAMPLES=OFF ^
34-
-DADIOS2_USE_Blosc=ON ^
29+
-DADIOS2_Blosc2_PREFER_SHARED=OFF ^
30+
-DADIOS2_USE_Blosc2=ON ^
3531
-DADIOS2_USE_BZip2=OFF ^
3632
-DADIOS2_USE_Fortran=OFF ^
3733
-DADIOS2_USE_HDF5=OFF ^
34+
-DADIOS2_USE_MHS=OFF ^
3835
-DADIOS2_USE_PNG=OFF ^
3936
-DADIOS2_USE_Profiling=OFF ^
4037
-DADIOS2_USE_Python=OFF ^
@@ -88,14 +85,47 @@ exit /b 0
8885
if errorlevel 1 exit 1
8986
exit /b 0
9087

88+
:build_blosc2
89+
if exist blosc2-stamp exit /b 0
90+
91+
curl -sLo blosc2-2.10.1.zip ^
92+
https://github.com/Blosc/c-blosc2/archive/refs/tags/v2.10.1.zip
93+
powershell Expand-Archive blosc2-2.10.1.zip -DestinationPath dep-blosc2
94+
95+
cmake -S dep-blosc2/c-blosc2-2.10.1 -B build-blosc2 ^
96+
-DCMAKE_BUILD_TYPE=Release ^
97+
-DBUILD_SHARED=OFF ^
98+
-DBUILD_STATIC=ON ^
99+
-DBUILD_BENCHMARKS=OFF ^
100+
-DBUILD_EXAMPLES=OFF ^
101+
-DBUILD_FUZZERS=OFF ^
102+
-DBUILD_PLUGINS=OFF ^
103+
-DBUILD_TESTS=OFF
104+
:: -DPREFER_EXTERNAL_ZLIB=ON
105+
:: -DZLIB_USE_STATIC_LIBS=ON
106+
if errorlevel 1 exit 1
107+
108+
cmake --build build-blosc2 --parallel %CPU_COUNT%
109+
if errorlevel 1 exit 1
110+
111+
cmake --build build-blosc2 --target install --config Release
112+
if errorlevel 1 exit 1
113+
114+
rmdir /s /q build-blosc2
115+
if errorlevel 1 exit 1
116+
117+
break > blosc2-stamp
118+
if errorlevel 1 exit 1
119+
exit /b 0
120+
91121
:build_hdf5
92122
if exist hdf5-stamp exit /b 0
93123

94-
curl -sLo hdf5-1.12.2.zip ^
95-
https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.2/src/hdf5-1.12.2.zip
96-
powershell Expand-Archive hdf5-1.12.2.zip -DestinationPath dep-hdf5
124+
curl -sLo hdf5-1.14.1-2.zip ^
125+
https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.14/hdf5-1.14.1/src/hdf5-1.14.1-2.zip
126+
powershell Expand-Archive hdf5-1.14.1-2.zip -DestinationPath dep-hdf5
97127

98-
cmake -S dep-hdf5/hdf5-1.12.2 -B build-hdf5 ^
128+
cmake -S dep-hdf5/hdf5-1.14.1-2 -B build-hdf5 ^
99129
-DCMAKE_BUILD_TYPE=Release ^
100130
-DCMAKE_VERBOSE_MAKEFILE=ON ^
101131
-DBUILD_SHARED_LIBS=OFF ^
@@ -193,7 +223,7 @@ call :install_buildessentials
193223
call :build_zlib
194224
:: build_bzip2
195225
:: build_szip
196-
call :build_blosc
197226
call :build_zfp
227+
call :build_blosc2
198228
call :build_hdf5
199229
call :build_adios2

0 commit comments

Comments
 (0)