This repository contains a set of patches that enable external GPUs on Arm platforms with non-standard PCIe integrations. It is based on the Ampere Altra Linux patches, with additional fixes to support a wider range of systems and kernel versions.
All AMD cards supported by the amdgpu driver should work fine.
Testing has been done on an RK3588-based SBC with an AMD Radeon RX 580 connected via a cheap M.2 to PCIe x4 adapter.
See the nvidia-wip branch. It contains patches for both the open and proprietary NVIDIA kernel modules, along with experimental nouveau patches. Also check out the open-gpu-kernel-modules pull request.
On RK3588 running EDK2 built from the latest source + patched NVIDIA modules, no kernel changes are necessary. U-Boot/DT boot may require 0004-PCI-dwrockchip-Disable-root-port-BARs.patch with DT changes to correct the PCIe MMIO regions.
Most Arm SoCs suffer from the same PCIe memory access and caching issues, which the current patches try to address. Additionally, some SoCs may also:
- lack sufficient contiguous MMIO space to accomodate the large VRAM BAR of GPUs (e.g. RX 580 needs over 256 MB, while the older RK3399 only has 30 MB).
- have a 32-bit (or less...) DMA constraint (might be an issue for > 4 GB RAM with some drivers).
- have a strict 32-bit access requirement for MMIO (i.e. even a 64-bit aligned access will not work).
These latter limitations can make the hardware totally unsuitable for GPU usage.
If your device is not supported upstream, then you'll need to use the BSP provided by the vendor.
~/linux$ git am -3 ../arm-pcie-gpu-patches/linux/<VERSION>/*.patchNote
The patches apply to kernel 6.1 and later (verified up to 6.17).
Use the latest <VERSION> that is less than or equal to your kernel version. E.g. for kernel 6.6, use the 6.1 patch set.
Make sure that the relevant DRM GPU drivers and workarounds provided here are enabled in the kernel config.
All workarounds default to Y (as intended for RK3588), but some platforms may not require everything. For instance:
CONFIG_ARM64_FORCE_PCIE_MMIO_DEVICE_MAPPINGSimpacts performance and should be disabled if unaligned accesses to PCIe MMIO work correctly.CONFIG_DRM_FORCE_DMA_WRITE_COMBINED_MAPPINGSmay cause issues on cache-coherent PCIe (can disable for Altra).
GPU devices require non-free firmware. E.g. for AMD GPUs on Debian, install firmware-amd-graphics.
It is also possible to get UEFI GOP support on these systems. A fully fledged UEFI implementation is required, i.e. EDK2-based.
GPUs bundle an x64 UEFI driver in the OpROM, which can be executed through an emulator.
AMD's x64 OpROM driver assumes cache-coherent PCIe, which may not be the case on your system leading to a corrupted display image.
Instead, you need to use the AArch64 drivers.
The drivers above (along with the emulator) need to be loaded by the firmware somehow. They can be:
-
included as part of the firmware build process (if the firmware is open). See:
- https://github.com/edk2-porting/edk2-rk3588/tree/824e6c12168a51cf72470fb43da60642e835198f/edk2-rockchip-non-osi/Drivers/AMD/Gop
- https://github.com/edk2-porting/edk2-rk3588/blob/824e6c12168a51cf72470fb43da60642e835198f/edk2-rockchip/Silicon/Rockchip/Rockchip.dsc.inc#L638-L640
- https://github.com/edk2-porting/edk2-rk3588/blob/824e6c12168a51cf72470fb43da60642e835198f/edk2-rockchip/Silicon/Rockchip/FvMainModules.fdf.inc#L131-L133
Note: if you're not getting any display output and the card has a dual BIOS switch, try toggling it. The GOP driver will not load if the VBIOS is modded with a mismatched signature.
Support may vary based on the card.
The x64 OpROM driver in an RTX 3050 has been verified to run correctly under the emulator (with a small caveat that the driver does not support MMIO translation for the framebuffer).