Exit JVM and dump heap on OutOfMemoryError for server containers

Add -XX:+ExitOnOutOfMemoryError, -XX:+HeapDumpOnOutOfMemoryError, and
-XX:HeapDumpPath=/dump to the long-running server JVMs (data, api,
submit, sched, db, rest). On OOM the JVM now writes a heap dump to
/dump and exits cleanly so K8s recreates the container, instead of
limping along in undefined state.

Motivated by the 2026-05-04 / 2026-05-06 prod data-pod wedges. Both
were precipitated by Java heap space errors which corrupted the
JVM-wide static InactivityMonitor Timer (a TimerTask hit OOM mid-run
and the TimerThread silently terminated). With these flags, the JVM
aborts on the first OOM before the InactivityMonitor TimerThread can
be touched, eliminating the OOM-driven path to the failover-wedge
condition entirely. The companion JmsFailoverWatchdog (in PR #1681)
remains as defense-in-depth for non-OOM wedge causes.

Targeted scope:
- 5 swarm-style server Dockerfiles (data, api, submit, sched, db) —
  add the three flags right after -XX:MaxRAMPercentage=80.
- 2 Quarkus rest Dockerfiles (Dockerfile.jvm, Dockerfile.legacy-jar) —
  append to JAVA_OPTS.

Skipped intentionally:
- Dockerfile-batch-dev (per-job SLURM lifecycle, no /dump volume).
- Dockerfile-clientgen-dev / docker/build/admin (build/utility tools).
- Dockerfile.native and Dockerfile.native-micro (Quarkus native image;
  -XX: flags don't apply to GraalVM native builds).

Companion change required in vcell-fluxcd: add a /dump emptyDir volume
mount to each affected Deployment so the JVM has somewhere to write
the dump. Without that mount the dump silently fails and the JVM still
exits — log signal still works, just no postmortem artifact.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: jcschaff

docker/build/Dockerfile-api-dev

@@ -84,6 +84,9 @@ EXPOSE 8000
 ENTRYPOINT java \
     -Xdebug -agentlib:jdwp=transport=dt_socket,address=*:8000,server=y,suspend=n \
     -XX:MaxRAMPercentage=80 \
+    -XX:+ExitOnOutOfMemoryError \
+    -XX:+HeapDumpOnOutOfMemoryError \
+    -XX:HeapDumpPath=/dump \
 #    -XX:+PrintFlagsFinal -XshowSettings:vm \
 	-Dvcell.softwareVersion="${softwareVersion}" \
 	-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager \

docker/build/Dockerfile-data-dev

@@ -76,6 +76,9 @@ EXPOSE 8000
 ENTRYPOINT java \
     -Xdebug -agentlib:jdwp=transport=dt_socket,address=*:8000,server=y,suspend=n \
 	-XX:MaxRAMPercentage=80 \
+	-XX:+ExitOnOutOfMemoryError \
+	-XX:+HeapDumpOnOutOfMemoryError \
+	-XX:HeapDumpPath=/dump \
 #	-XX:+PrintFlagsFinal -XshowSettings:vm \
     -Djava.awt.headless=true \
 	-Dvcell.softwareVersion="${softwareVersion}" \

docker/build/Dockerfile-db-dev

@@ -56,6 +56,9 @@ EXPOSE 8000
 ENTRYPOINT java \
     -Xdebug -agentlib:jdwp=transport=dt_socket,address=*:8000,server=y,suspend=n \
 	-XX:MaxRAMPercentage=80 \
+	-XX:+ExitOnOutOfMemoryError \
+	-XX:+HeapDumpOnOutOfMemoryError \
+	-XX:HeapDumpPath=/dump \
 #	-XX:+PrintFlagsFinal -XshowSettings:vm \
     -Djava.awt.headless=true \
 	-Dvcell.softwareVersion="${softwareVersion}" \

docker/build/Dockerfile-sched-dev

@@ -87,6 +87,9 @@ VOLUME /htclogs
 ENTRYPOINT java \
     -Xdebug -agentlib:jdwp=transport=dt_socket,address=*:8000,server=y,suspend=n \
 	-XX:MaxRAMPercentage=80 \
+	-XX:+ExitOnOutOfMemoryError \
+	-XX:+HeapDumpOnOutOfMemoryError \
+	-XX:HeapDumpPath=/dump \
 #	-XX:+PrintFlagsFinal -XshowSettings:vm \
     -Djava.awt.headless=true \
 	-Dvcell.softwareVersion="${softwareVersion}" \

docker/build/Dockerfile-submit-dev

@@ -117,6 +117,9 @@ EXPOSE 8000
 ENTRYPOINT java \
     -Xdebug -agentlib:jdwp=transport=dt_socket,address=*:8000,server=y,suspend=n \
 	-XX:MaxRAMPercentage=80 \
+	-XX:+ExitOnOutOfMemoryError \
+	-XX:+HeapDumpOnOutOfMemoryError \
+	-XX:HeapDumpPath=/dump \
 #	-XX:+PrintFlagsFinal -XshowSettings:vm \
     -Djava.awt.headless=true \
 	-Dvcell.softwareVersion="${softwareVersion}" \

vcell-rest/src/main/docker/Dockerfile.jvm

@@ -95,7 +95,10 @@ ENV JAVA_OPTS="\
 -Dquarkus.http.host=0.0.0.0 \
 -Djava.util.logging.manager=org.jboss.logmanager.LogManager \
 -Dvcellapi.privateKey.file=/run/secrets/jwt-secret/apiprivkey \
--Dvcellapi.publicKey.file=/run/secrets/jwt-secret/apipubkey"
+-Dvcellapi.publicKey.file=/run/secrets/jwt-secret/apipubkey \
+-XX:+ExitOnOutOfMemoryError \
+-XX:+HeapDumpOnOutOfMemoryError \
+-XX:HeapDumpPath=/dump"
 ENV JAVA_APP_JAR="/deployments/quarkus-run.jar"
 
 ENTRYPOINT [ "/opt/jboss/container/java/run/run-java.sh" ]

vcell-rest/src/main/docker/Dockerfile.legacy-jar

@@ -87,7 +87,7 @@ COPY target/*-runner.jar /deployments/quarkus-run.jar
 
 EXPOSE 8080
 USER 185
-ENV JAVA_OPTS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -D"
+ENV JAVA_OPTS="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dump -D"
 ENV JAVA_APP_JAR="/deployments/quarkus-run.jar"
 
 ENTRYPOINT [ "/opt/jboss/container/java/run/run-java.sh" ]