Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
ac939fc
feat(cpp/client, py/client, csharp/client): DH-22472, DH-22473, DH-22…
kosak Apr 27, 2026
10b078c
touch a file to trigger a build
kosak May 1, 2026
ddc91a1
Revert "touch a file to trigger a build"
kosak May 1, 2026
be99f3a
rename step
kosak May 1, 2026
11c44b0
let's see what it feels like when names are shorter
kosak May 6, 2026
b7e400e
let's see how this looks now
kosak May 6, 2026
fc8cc8a
lots more renames. Bring it on!!
kosak May 6, 2026
25c7a01
try unified
kosak May 6, 2026
c2c3334
Let's watch this crash and burn
kosak May 6, 2026
300521b
ok this is going to be UGLY
kosak May 6, 2026
1b42285
let's try this again -- I can't see what is what
kosak May 6, 2026
daf12d8
oops can't have two "ifs"
kosak May 6, 2026
310aa4b
unify windows versions ok?
kosak May 6, 2026
804fb3b
not dead yet
kosak May 6, 2026
c5e765c
if this doesn't work, we will hardcode
kosak May 6, 2026
2b083d6
HACK JOB 666
kosak May 6, 2026
f9f2b4b
zamboni time 12456666666
kosak May 6, 2026
d3f29a2
tv and ps3 and xbox
kosak May 6, 2026
5177b2c
try this hell
kosak May 6, 2026
7528fdb
no no trhis
kosak May 6, 2026
dd1f5e9
$20,000 / night
kosak May 6, 2026
9a3ac71
one femto-step at a time
kosak May 6, 2026
9fddbc3
what a hellish mess
kosak May 6, 2026
3958815
gakkk
kosak May 6, 2026
e27b132
blah
kosak May 6, 2026
3df64a0
super blah
kosak May 6, 2026
c3c2de0
ok confirm this works
kosak May 6, 2026
34911a7
godspeed
kosak May 6, 2026
2d3e761
zamboni time 2
kosak May 6, 2026
37b5b8e
why am I sad
kosak May 6, 2026
af37011
i'll be honest this is pissing me off
kosak May 6, 2026
a23011a
thi s too
kosak May 6, 2026
eba194f
might as well go all the way
kosak May 6, 2026
c1e61f0
the final ultimatum
kosak May 6, 2026
1a07d28
now try this version
kosak May 6, 2026
41c05aa
arg one more painful change
kosak May 6, 2026
fe922c8
This pain, no name
kosak May 6, 2026
dbf5d2a
NO
kosak May 6, 2026
98bb00d
REALLY NO
kosak May 6, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
321 changes: 321 additions & 0 deletions .github/workflows/clients-cpp-python.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,321 @@
name: "C++/Python"
on:
pull_request:
branches: [ 'main', 'rc/v*' ]
push:
branches: [ 'main', 'check/**', 'release/v*' ]
permissions:
contents: read

#===============================================================================
# Background: in this workflow there are three different kinds of caching going
# on:
#
# 1. The outputs are Github Build artifacts, attached to every action run.
# These artifacts can be downloaded from e.g.
# https://github.com/deephaven/deephaven-core/actions/runs/17416308139
# but the specific URL depends on the build.
# We use this to upload:
# - The C++ install directory
# - The .whl file for the static Python client
# - The .whl file for the ticking Python client
#
# 2. The C++ install directory is cached via actions/cache, and keyed
# by the hash of the cpp-client and proto directories. This allows us
# to skip recompilation if these files don't change. This is an optional
# optimization that might not be very significant.
#
# 3. Github Packages, which is a NuGet-compatible repo accessible from
# https://github.com/orgs/deephaven/packages?repo_name=deephaven-core
# and controlled via the NuGet API. We use this package repository to hold
# the binaries for the C++ dependencies e build (grpc, arrow, etc.). It is
# managed automatically for us by cmake/vcpkg/nuget. This optimization is
# important because building the C++ dependencies takes significant time
# (45 minutes maybe).
#===============================================================================

env:
GH_PACKAGES_USERNAME: ${{ github.repository_owner }}
GH_PACKAGES_FEED: https://nuget.pkg.github.com/${{ github.repository_owner }}/index.json
DHINSTALL: ${{ github.workspace }}/dhinstall

jobs:
#=============================================================================
# Identify version of Deephaven Core
#=============================================================================
version_job:
uses: ./.github/workflows/dhc-version.yml

#=============================================================================
# Build the C++ Client
#=============================================================================
cpp_job:
name: "Build C++ (or use cache)"
permissions:
contents: read
packages: write
strategy:
matrix:
include:
- os: ubuntu-24.04
target_triplet: x64-linux-dynamic-release
- os: windows-2025-vs2026
target_triplet: x64-windows-release
fail-fast: false
runs-on: ${{ matrix.os }}
defaults:
run:
# This gets you Git Bash on Windows
shell: bash

steps:
- name: Check out this repo
uses: actions/checkout@v6

# Try to fetch the whole installation directory (the final product of the
# build) from the Github Cache. The key to this cache is derived
# from the hash of the contents of the files in the cpp-client and proto
# directories. If this fetch is successful, we skip over all the build
# work and just copy this cached install directory to the output artifact
# for this run.

- name: Try to restore cached installdir
id: cache-cpp-install
uses: actions/cache@v5
with:
path: ${{ env.DHINSTALL }}
key: ${{ runner.os }}-cpp-client-${{ hashFiles('cpp-client/**', 'proto/**') }}

# Make sure we check out the same version of the vcpkg tool that we
# have configured as our registry baseline. This (we suppose) will help
# improve the problem we have with our package hash drift.

- name: Extract vcpkg baseline
id: vcpkg-baseline
run: |
baseline=$(jq -r '.["default-registry"].baseline' ./cpp-client/deephaven/vcpkg-configuration.json)
echo "baseline=$baseline"
echo "baseline=$baseline" >> "$GITHUB_OUTPUT"

# [Note: skip this step if cache-cpp-install was successful]
# Check out vcpkg

- name: Check out vcpkg
id: checkout-vcpkg
if: steps.cache-cpp-install.outputs.cache-hit != 'true'
uses: actions/checkout@v6
with:
repository: microsoft/vcpkg
ref: ${{ steps.vcpkg-baseline.outputs.baseline }}
path: vcpkg

# [Note: skip this step if cache-cpp-install was successful]
# Bootstrap vcpkg

- name: Bootstrap vcpkg
if: steps.cache-cpp-install.outputs.cache-hit != 'true'
run: "$GITHUB_WORKSPACE/vcpkg/bootstrap-vcpkg.sh"

# [Note: Linux only]
# [Note: skip this step if cache-cpp-install was successful]
# Get mono so that we can run nuget. This is because the nuget.exe that
# ships with vcpkg is an old .NET Framework binary (Windows format), and
# only mono knows how to run such a thing on Linux. Some day the
# maintainers will fix this so it runs with the cross-platform "dotnet".

- name: Install mono (Linux)
if: runner.os == 'Linux' && steps.cache-cpp-install.outputs.cache-hit != 'true'
run: sudo apt install -y mono-complete

# [Note: skip this step if cache-cpp-install was successful]
# Adapted from
# https://learn.microsoft.com/en-us/vcpkg/consume/binary-caching-github-packages?pivots=linux-runner
# Configure nuget to know how to download (or upload) each cached
# package that we need (or create)

- name: "Configure nuget (Linux)"
if: runner.os == 'Linux' && steps.cache-cpp-install.outputs.cache-hit != 'true'
run: |
vcpkg_exe=${{ github.workspace }}/vcpkg/vcpkg
nuget_dotnet=$($vcpkg_exe fetch nuget | tail -n 1)
mono $nuget_dotnet \
sources add \
-Source "${{ env.GH_PACKAGES_FEED }}" \
-StorePasswordInClearText \
-Name GitHubPackages \
-UserName "${{ env.GH_PACKAGES_USERNAME }}" \
-Password "${{ secrets.GITHUB_TOKEN }}"
mono $nuget_dotnet \
setapikey "${{ secrets.GITHUB_TOKEN }}" \
-Source "${{ env.GH_PACKAGES_FEED }}"

- name: "Configure nuget (Windows)"
if: runner.os == 'Windows' && steps.cache-cpp-install.outputs.cache-hit != 'true'
shell: pwsh
run: |
$nuget_exe = & ./vcpkg/vcpkg.exe fetch nuget
& $nuget_exe sources add `
-Source "${{ env.GH_PACKAGES_FEED }}" `
-StorePasswordInClearText `
-Name GitHubPackages `
-UserName "${{ env.GH_PACKAGES_USERNAME }}" `
-Password "${{ secrets.GITHUB_TOKEN }}"
& $nuget_exe setapikey "${{ secrets.GITHUB_TOKEN }}" -Source "${{ env.GH_PACKAGES_FEED }}"

# [Note: skip this step if cache-cpp-install was successful]
# Build packages and configure cmake. Check nupkg cache for existing
# packages, otherwise build and upload.
# 1. Invoke vcpkg to:
# a) Try to fetch each of the dependent libraries from the nuget cache
# b) If not found, build them and insert them in the nuget cache
# 2. Configure cmake to prepare for the build/install step, which comes next

- name: "Build/cache dependencies"
if: steps.cache-cpp-install.outputs.cache-hit != 'true'
env:
VCPKG_BINARY_SOURCES: "clear;nuget,${{ env.GH_PACKAGES_FEED }},readwrite"
run: cmake -S cpp-client/deephaven -B cpp-client/deephaven/build --toolchain "$GITHUB_WORKSPACE/vcpkg/scripts/buildsystems/vcpkg.cmake" -DCMAKE_INSTALL_PREFIX="$DHINSTALL" -DVCPKG_TARGET_TRIPLET="${{ matrix.target_triplet }}" -DVCPKG_OVERLAY_TRIPLETS=cpp-client/deephaven/custom-triplets

# [Note: skip this step if cache-cpp-install was successful]
# Build our executables and install them in the configured install
# directory {{ env.DHINSTALL }}

- name: Build and install
if: steps.cache-cpp-install.outputs.cache-hit != 'true'
run: cmake --build cpp-client/deephaven/build --config RelWithDebInfo --target install --parallel

# Upload results as build artifact. Note: if cache-cpp-install was
# successful, then none of the intervening steps were executed. In that
# case, this effectively just copies the cache to the build artifact.

- name: Upload install dir
id: upload-install-directory
uses: actions/upload-artifact@v7
with:
name: cpp-client-install-${{ matrix.os }}
path: ${{ env.DHINSTALL }}

#=============================================================================
# Build the Python Static Client (not OS dependent)
#=============================================================================

python_static_job:
name: Python static
runs-on: ubuntu-24.04
needs: [version_job]
defaults:
run:
shell: bash

steps:
- name: Check out this repo
uses: actions/checkout@v6

- name: Install Python
uses: actions/setup-python@v6
with:
python-version: '3.10'

- name: Install the 'wheel' package
run: pip3 install wheel

- name: Install requirements-dev.txt
run: |
cd ./py/client
pip3 install -r requirements-dev.txt

- name: Run setup.py
env:
DEEPHAVEN_VERSION: ${{ needs.version_job.outputs.dhc_version }}
run: |
cd ./py/client
python setup.py bdist_wheel

# Upload static .whl file as build artifact, to be used directly or by # dependent clients like Python ticking
- name: Upload .whl file
uses: actions/upload-artifact@v7
with:
name: py-static-wheel
path: ./py/client/dist/*.whl

#=============================================================================
# Build the Python Ticking Client (OS dependent)
#=============================================================================

python_ticking_job:
strategy:
fail-fast: false
matrix:
os: [ubuntu-24.04, windows-2025-vs2026]
name: Python ticking
runs-on: ${{ matrix.os }}
needs: [cpp_job, python_static_job, version_job]
defaults:
run:
shell: bash

steps:
- name: Check out this repo
uses: actions/checkout@v6

# Get the C++ installation as a build artifact
- name: Fetch C++ installation
uses: actions/download-artifact@v8
with:
name: cpp-client-install-${{ matrix.os }}
path: ${{ env.DHINSTALL }}

# Get the Python static installation as a build artifact
- name: Fetch Python static installation
uses: actions/download-artifact@v8
with:
name: py-static-wheel
path: ./py/client/dist/

- name: Install Python
uses: actions/setup-python@v6
with:
python-version: '3.10'

- name: Install the 'wheel' and 'cython' packages
run: pip3 install wheel cython

- name: Install requirements-dev.txt
run: |
cd ./py/client
pip3 install -r requirements-dev.txt

- name: Install static client
run: pip3 install --force --no-deps ./py/client/dist/*.whl

# Pinned to a specific SHA to avoid future surprises
- name: set up PATH for cl.exe
if: runner.os == 'Windows'
uses: ilammy/msvc-dev-cmd@0b201ec74fa43914dc39ae48a89fd1d8cb592756

- name: Build our Cython code (Linux)
if: runner.os == 'Linux'
env:
DEEPHAVEN_VERSION: ${{ needs.version_job.outputs.dhc_version }}
CPPFLAGS: -I${{ env.DHINSTALL }}/include
LDFLAGS: -L${{ env.DHINSTALL }}/lib
run: |
cd ./py/client-ticking
python setup.py build_ext -i
python setup.py bdist_wheel

- name: Build our Cython code (Windows)
if: runner.os == 'Windows'
shell: cmd
env:
DEEPHAVEN_VERSION: ${{ needs.version_job.outputs.dhc_version }}
run: |
cd .\py\client-ticking
python setup.py build_ext -i
python setup.py bdist_wheel

- name: Upload ticking .whl file as build artifact
uses: actions/upload-artifact@v7
with:
name: py-ticking-wheel-${{ matrix.os }}
path: ./py/client-ticking/dist/*.whl
58 changes: 58 additions & 0 deletions .github/workflows/clients-csharp.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name: "C# Client"
on:
pull_request:
branches: [ 'main', 'rc/v*' ]
push:
branches: [ 'main', 'check/**', 'release/v*' ]
permissions:
packages: write

env:
GH_PACKAGES_USERNAME: ${{ github.repository_owner }}
GH_PACKAGES_FEED: https://nuget.pkg.github.com/${{ github.repository_owner }}/index.json
DHINSTALL: ${{ github.workspace }}/dhinstall

jobs:
#=============================================================================
# Identify version of Deephaven Core
#=============================================================================
version_job:
uses: ./.github/workflows/dhc-version.yml

#=============================================================================
# Build the .NET Client (cross-platform, but we use the Ubuntu runner)
#=============================================================================
dotnet_job:
name: Build .NET client
runs-on: ubuntu-24.04
needs: version_job
defaults:
run:
shell: bash

steps:
- name: Check out this repo
uses: actions/checkout@v6

- name: Setup dotnet
uses: actions/setup-dotnet@v5
with:
dotnet-version: '10.0.x'

- name: Build Deephaven Core C# Client
run: |
cd csharp/client/Dh_NetClient
dotnet build -c Release

- name: Create unsigned NuGet package
env:
DHC_VERSION: ${{ needs.version_job.outputs.dhc_version }}
run: |
cd csharp/client/Dh_NetClient
dotnet pack --no-build -c Release /p:Platform="Any CPU" /p:PackageVersion=$DHC_VERSION

- name: Upload unsigned NuGet package as build artifact
uses: actions/upload-artifact@v7
with:
name: dotnet-client
path: ./csharp/client/Dh_NetClient/bin/Release/*.nupkg
Loading
Loading