try-runtime-cli Crashes Due to OOM (Out-of-Memory) & Long Execution Time – Resolved with 32GB RAM

[TomatoAres]

Description

We encounter critical resource and performance issues when running try-runtime-cli with our target runtimes, which improve drastically only after upgrading system memory. Key details are as follows:

1. Memory Exhaustion & OOM Termination: With the initial 16GB physical memory configuration, the system rapidly runs out of memory during try-runtime execution, causing severe system lag. Eventually, the try-runtime process is consistently killed by the operating system due to an Out-of-Memory (OOM) error.
2. Extreme Execution Time (16GB RAM Scenario): In the rare cases where the process avoids OOM and completes successfully with 16GB RAM, execution takes approximately 3 hours—far exceeding acceptable workflow limits.
3. Resolution with Increased RAM: After upgrading the system’s physical memory to 32GB (keeping other configurations unchanged), the issue is resolved: try-runtime completes execution quickly (no specific long delays observed) and no OOM crashes occur.

Environment Configuration

Component	Initial Configuration (Problematic)	Updated Configuration (Working)
try-runtime-cli Version	0.8.0	0.8.0
Target Runtime(s)	- bifrost-polkadot (snapshot: ~3.1GB) - bifrost-paseo	- bifrost-polkadot (snapshot: ~3.1GB) - bifrost-paseo
Operating System	Ubuntu 24.04	Ubuntu 24.04
System Resources	- CPU: 8 cores - Physical Memory: 16GB - Swap Space: 16~32GB (enabled)	- CPU: 8 cores - Physical Memory: 32GB - Swap Space: 16~32GB (enabled)

Additional Context

This behavior suggests the try-runtime process (especially when handling the ~3.1GB bifrost-polkadot snapshot and Bifrost-specific runtimes) has a higher minimum memory requirement than initially anticipated. While upgrading to 32GB RAM resolves the issue for us, we have questions for further optimization:

1. Is the 32GB RAM requirement expected for try-runtime-cli 0.8.0 when processing large snapshots (e.g., 3.1GB) for Bifrost runtimes?
2. Are there CLI flags, runtime configurations, or snapshot optimization techniques (e.g., pruning) that could reduce the memory footprint, allowing reliable execution with 16GB RAM?
3. Could the high memory usage be linked to runtime-specific logic in bifrost-polkadot/bifrost-paseo (e.g., unoptimized state handling during try-runtime validation) rather than just try-runtime-cli itself?

Steps to Reproduce

1. Reproduce OOM/Long Execution (16GB RAM):
   • Use try-runtime-cli 0.8.0 to run validation against bifrost-polkadot (with 3.1GB snapshot) on a system with 8 cores, 16GB RAM, and 1632GB swap.
   • Monitor memory usage: Observe rapid memory depletion, system lag, and eventual OOM killing. If successful, execution takes ~3 hours.
2. Verify Resolution (32GB RAM):
   • Upgrade the same system’s physical memory to 32GB.
   • Re-run the same try-runtime command: Execution completes quickly with no OOM errors.