hw: per-bank PLATFORM_MEMORY_OFFSET — fix U250 vx_busy hang at boot

[hwirys]

Why this matters: the U250 path on master cannot boot

On Vortex master, building for Alveo U250 appears to succeed (Vitis produces an xclbin), but the kernel never starts on real silicon. ap_start is asserted, vx_busy goes high, and stays high forever. The CTL register reads back 0x1 indefinitely; the host eventually times out. This is the failure pattern reported in #262, #263, #278.

Vortex 2.3 added the U250 build infrastructure (platform recognition, xrt::bo allocation, connectivity.sp plumbing) but did not address the BAR-mismatch between Vortex's compile-time absolute addresses and where XRT actually places memory. The xclbin is produced; it just doesn't execute.

This PR is the smallest patch that makes a U250 boot. It is a logical prerequisite for any other U250 work.

---

Root cause

Vortex hard-codes some compile-time absolute byte addresses:

Symbol	Value (XLEN=64)	Where
STARTUP_ADDR	0x1_8000_0000	Where the kernel binary's text/data lives, fetched by Vortex on ap_start
STACK_BASE_ADDR	0x1_FFFF_0000	Per-thread stack top

XRT, on the other hand, allocates each xrt::bo at a virtual address chosen by the platform. On U250, bank 0 lands at 0x40_0000_0000 (256 GiB into the device address space). All AXI traffic from Vortex therefore goes to a region the platform's AXI fabric does not decode → vx_busy = 1 forever.

The pre-existing PLATFORM_MEMORY_OFFSET macro is conceptually the right escape hatch — it adds a constant offset to every outgoing AXI address — but on master it is a single global offset for all banks, while real Xilinx XRT places each bank at a different virtual address. So a single offset cannot cover U200/U250 four-channel deployments either.

---

Fix

Three small files:

hw/rtl/afu/xrt/vortex_afu.vh (+18 / −0)

Introduce per-bank macros. Each defaults to the legacy global PLATFORM_MEMORY_OFFSET so existing single-channel platforms are unchanged:

`ifndef PLATFORM_MEMORY_OFFSET_0
`define PLATFORM_MEMORY_OFFSET_0 `PLATFORM_MEMORY_OFFSET
`endif
// ... _1, _2, _3 likewise

hw/rtl/afu/xrt/VX_afu_wrap.sv (+14 / −2)

Build a 4-entry array from those macros and add the bank-i offset to each m_axi_mem_<i> outgoing AW/AR address (replaces the single global add):

wire [C_M_AXI_MEM_ADDR_WIDTH-1:0] platform_memory_offsets [4];
assign platform_memory_offsets[0] = `PLATFORM_MEMORY_OFFSET_0;
// ... _1, _2, _3
for (genvar i = 0; i < C_M_AXI_MEM_NUM_BANKS; ++i) begin : g_addressing
    assign m_axi_mem_awaddr_a[i] = m_axi_mem_awaddr_u[i] + platform_memory_offsets[i];
    assign m_axi_mem_araddr_a[i] = m_axi_mem_araddr_u[i] + platform_memory_offsets[i];
end

hw/syn/xilinx/xrt/platforms.mk (+6 / −2)

Switch the U250 entry to single-channel with the offset hard-coded so the build is deployable out of the box:

else ifneq ($(findstring xilinx_u250,$(XSA)),)
  CONFIGS += -DPLATFORM_MEMORY_NUM_BANKS=1 -DPLATFORM_MEMORY_ADDR_WIDTH=34
  VPP_FLAGS += --connectivity.sp vortex_afu_1.m_axi_mem_0:DDR[0]
  CONFIGS += -DPLATFORM_MEMORY_OFFSET_0=40'h4000000000

(0x40_0000_0000 is the empirical XRT base for DDR[0] on the
xilinx_u250_gen3x16_xdma_4_1_202210_1 shell. Multi-bank — full 64 GB across 4 DDR4 channels — needs the AFU to learn each bo's actual XRT VA at runtime, since each xrt::bo is placed by the runtime allocator. That follow-up will be submitted as a separate PR with a DCR-based runtime path; this PR is intentionally minimal and out-of-the-box deployable.)

---

Verification on real Alveo U250

Built with this patch (default DSP FPU, NUM_CORES=2 NUM_WARPS=4 XLEN=64, 200 MHz) on real U250 silicon — XRT 2.19.194, shell xilinx_u250_gen3x16_xdma_4_1_202210_1. The PLP load (xdma_shell_4_1) is required once per cold boot as documented in the README.

Test category	Without this PR	With this PR
Any kernel start	hangs at ap_start, vx_busy=1 forever	boots ✅
vecadd (n = 16…16384)	n/a (hang)	all sizes PASS
dogfood Test0..Test20 (iadd, imul, idiv, fadd, fsub, fmul, fmadd, fnmadd, fdiv, fsqrt, ftoi, itof, fclamp, iclamp, …)	n/a	PASS
regression: sgemm, dotproduct, demo, dropout, conv3, io_addr, fence, diverge	n/a	PASS
OpenCL: saxpy, vecadd, sgemm, sgemm2, sgemm3, stencil, sfilter, spmv, psort, oclprintf	n/a	PASS
single-rank MPI: mpi_vecadd, mpi_dotproduct, mpi_diverge, mpi_put_dotproduct	n/a	PASS

That spans integer arithmetic, single-precision FP, vector reductions, matrix multiply, convolution, ML-style ops (dropout), and the full OpenCL/MPI execution stacks. Across all of those there is no observed regression introduced by the patch.

(dogfood Test21-trig produces FP64 NaNs because the default DSP FPU is FP32-only — that's a pre-existing FPU-implementation issue, not addressed here.)

Build numbers: total xclbin link 2 h 30 m, WNS = +0.057 ns @ 200 MHz (positive, no impl-strategy tuning), area added by this patch ≈ 4 × 36-bit adders. Negligible.

---

Backward compatibility

• PLATFORM_MEMORY_OFFSET_<i> defaults to PLATFORM_MEMORY_OFFSET, so any platform that didn't define per-bank offsets sees the same effective offset on every bank as before. HBM platforms (U280/U55C/U50, all PLATFORM_MERGED_MEMORY_INTERFACE), VCK5000 single-channel, and Zynq UltraScale+ are byte-for-byte identical.
• The U250 platforms.mk entry changes from NUM_BANKS=4 (no connectivity) to NUM_BANKS=1 (DDR[0] + offset). The previous configuration produced an xclbin that didn't actually run, so this is a strict improvement; any user wishing to revisit four-channel deployment needs the runtime VA plumbing of the follow-up PR anyway.

---

Files touched

• hw/rtl/afu/xrt/vortex_afu.vh — +18 / −0
• hw/rtl/afu/xrt/VX_afu_wrap.sv — +14 / −2
• hw/syn/xilinx/xrt/platforms.mk — +6 / −2

References: closes (or substantially mitigates) #262, #263, #278.

[tinebp]

Great investigation and catch!
Your proposed solution has a hardcoded 4-bank assumption. Can you generalize the fix in the RTL code?

[hwirys]

Thank you so much! I'll update general version ASAP

[hwirys]

Thanks for the review! Pushed 19bfcdf65 which drops the 4-bank assumption.

Changes

• hw/rtl/afu/xrt/VX_afu_wrap.sv: platform_memory_offsets array now sized by C_M_AXI_MEM_NUM_BANKS, per-bank assignment via a generate-for loop using the single PLATFORM_MEMORY_OFFSET macro.
• hw/rtl/afu/xrt/vortex_afu.vh: dropped the numbered PLATFORM_MEMORY_OFFSET_0..3 macros; only PLATFORM_MEMORY_OFFSET remains.
• hw/syn/xilinx/xrt/platforms.mk: U250 uses the unified PLATFORM_MEMORY_OFFSET=40'h4000000000.

Net: +13 / -29 lines.

For platforms that need a different VA per bank (e.g. U250 4-bank DDR4), a DCR-based runtime push mechanism is on master and is intentionally a separate follow-up — this PR stays scoped to single-bank end-to-end fix + RTL generality.

Verification on real Alveo U250 (xilinx_u250_gen3x16_xdma_4_1)

Three independent xclbin builds at NUM_CORES=1, KERNEL_FREQ=150 MHz, identical RTL, varying PLATFORM_MEMORY_NUM_BANKS:

Config	Result
NUM_BANKS=1	19/19 PASS + 1 known-fail
NUM_BANKS=2	19/19 PASS + 1 known-fail
NUM_BANKS=4	19/19 PASS + 1 known-fail

Test coverage (each build):

• Regression (CUDA-style): vecadd, sgemm, sgemm2, dotproduct, diverge, demo, conv3, fence, io_addr, dropout, printf, basic, arith, mstress
• OpenCL: saxpy, vecadd, sgemm, sfilter, oclprintf

The known-fail is dogfood Test21-trig (NaN from sinf — pre-existing DSP-FPU FP64 limitation, fixable with -DFPU_FPNEW, unrelated to bank addressing).

NUM_BANKS=4 IPC is consistently higher than NUM_BANKS=1 across memory-bound workloads (e.g. vecadd 0.339 vs 0.308, dotproduct 0.367 vs 0.332), confirming the generalized per-bank addressing distributes traffic correctly.