NVIDIA GPU support

[hrushirajg23]

@mariobalanica I would like to know how you dealt with this issue, or is this nvidia specific issue? Would you help me here, I would be thankful
vicharak@nvidia:~$ lspci
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0000:01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3080 Lite Hash Rate] (rev a1)
0000:01:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
0003:30:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3588 (rev 01)
0003:31:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5765DL NVMe SSD Controller (DRAM-less) (rev 01)

The bus number 01 shows up the nvidia GPU .
Below it shows the pcie only as gen1 single lane

rk-pcie fe150000.pcie: PCIe Link up, LTSSM is 0x30011
[ 15.717803] rk-pcie fe150000.pcie: PCIe Gen.1 x1 link up
[ 15.717916] rk-pcie fe150000.pcie: PCI host bridge to bus 0000:00

pci 0000:01:00.0: BAR 1: assigned [mem 0x900000000-0x90fffffff 64bit pref]
[ 15.749922] pci 0000:01:00.0: BAR 3: assigned [mem 0x910000000-0x911ffffff 64bit pref]
[ 15.749956] pci 0003:31:00.0: [10ec:5765] type 00 class 0x010802
[ 15.749963] pci 0000:01:00.0: BAR 0: no space for [mem size 0x01000000]
[ 15.749968] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x01000000]
[ 15.749973] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00080000 pref]
[ 15.749977] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00080000 pref]
[ 15.749982] pci 0000:01:00.1: BAR 0: no space for [mem size 0x00004000]
[ 15.749987] pci 0000:01:00.1: BAR 0: failed to assign [mem size 0x00004000]
[ 15.749991] pci 0000:01:00.0: BAR 5: assigned [io 0x200000-0x20007f]

ranges:

rk-pcie fe150000.pcie: host bridge /pcie@fe150000 ranges:
[ 15.368417] rk-pcie fe150000.pcie: IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
[ 15.368440] rk-pcie fe150000.pcie: MEM 0x00f0200000..0x00f0ffffff -> 0x00f0200000
[ 15.368453] rk-pcie fe150000.pcie: MEM 0x0900000000..0x093fffffff -> 0x0900000000
[ 15.368492] rk-pcie fe150000.pcie: iATU unroll: enabled
[ 15.368498] rk-pcie fe150000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G

rk-pcie fe150000.pcie: PCIe Link up, LTSSM is 0x30011
[ 15.717803] rk-pcie fe150000.pcie: PCIe Gen.1 x1 link up
[ 15.717916] rk-pcie fe150000.pcie: PCI host bridge to bus 0000:00

Here is my dtb node:

pcie3x4: pcie@fe150000 {
compatible = "rockchip,rk3588-pcie", "snps,dw-pcie";
#address-cells = <3>;
#size-cells = <2>;
bus-range = <0x00 0x0f>;
clocks = <&cru ACLK_PCIE_4L_MSTR>, <&cru ACLK_PCIE_4L_SLV>,
<&cru ACLK_PCIE_4L_DBI>, <&cru PCLK_PCIE_4L>,
<&cru CLK_PCIE_AUX0>, <&cru CLK_PCIE4L_PIPE>;
clock-names = "aclk_mst", "aclk_slv",
"aclk_dbi", "pclk",
"aux", "pipe";
device_type = "pci";
interrupts = <GIC_SPI 263 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 262 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 261 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 260 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 259 IRQ_TYPE_LEVEL_HIGH>;
interrupt-names = "sys", "pmc", "msg", "legacy", "err";
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 7>;
interrupt-map = <0 0 0 1 &pcie3x4_intc 0>,
<0 0 0 2 &pcie3x4_intc 1>,
<0 0 0 3 &pcie3x4_intc 2>,
<0 0 0 4 &pcie3x4_intc 3>;
linux,pci-domain = <0>;
num-ib-windows = <16>;
num-ob-windows = <16>;
num-viewport = <8>;
max-link-speed = <3>;
msi-map = <0x0000 &its1 0x0000 0x1000>;
num-lanes = <4>;
phys = <&pcie30phy>;
phy-names = "pcie-phy";
power-domains = <&power RK3588_PD_PCIE>;
ranges = <0x81000000 0x0 0xf0100000 0x0 0xf0100000 0x0 0x100000
0x82000000 0x0 0xf0200000 0x0 0xf0200000 0x0 0xe00000
0xc3000000 0x9 0x00000000 0x9 0x00000000 0x0 0x40000000>;
reg = <0x0 0xfe150000 0x0 0x10000>,
<0xa 0x40000000 0x0 0x400000>,
<0x0 0xf0000000 0x0 0x100000>;
reg-names = "pcie-apb", "pcie-dbi", "config";
resets = <&cru SRST_PCIE0_POWER_UP>, <&cru SRST_P_PCIE0>;
reset-names = "pcie", "periph";
rockchip,pipe-grf = <&php_grf>;
status = "disabled";

	pcie3x4_intc: legacy-interrupt-controller {
		interrupt-controller;
		#address-cells = <0>;
		#interrupt-cells = <1>;
		interrupt-parent = <&gic>;
		interrupts = <GIC_SPI 260 IRQ_TYPE_EDGE_RISING>;
	};
};

Then I tried increasing the non-prefetchable memory range from 14mb to 16mb. still I have BAR issues

@@ -556,7 +556,7 @@ pcie3x4: pcie@fe150000 {
phy-names = "pcie-phy";
power-domains = <&power RK3588_PD_PCIE>;
ranges = <0x81000000 0x0 0xf0100000 0x0 0xf0100000 0x0 0x100000

•                     0x82000000 0x0 0xf0200000 0x0 0xf0200000 0x0 0xe00000
  

•                     0x82000000 0x0 0xf0200000 0x0 0xf0200000 0x0 0x01000000
                      0xc3000000 0x9 0x00000000 0x9 0x00000000 0x0 0x40000000>;
            reg = <0x0 0xfe150000 0x0 0x10000>,
                  <0xa 0x40000000 0x0 0x400000>,
  

.

[mariobalanica]

Have you applied the patches here?

[geerlingguy]

@hrushirajg23 - I don't think anyone's demonstrated Nvidia GPUs working on any Rockchip boards yet; there are still driver quirks there that cause additional issues.

[hrushirajg23]

@mariobalanica yes I have applied the patches. Does your patch solve the BAR issues ?

[mariobalanica]

BAR 0 is 32-bit non-prefetchable and BAR 6 is 32-bit prefetchable. Currently, they both have to fit into the 14 MB 32-bit non-prefetchable window.

For BAR 0, it first tries to assign 16 MB, which is obviously impossible in this configuration. Then it also attempts 16 KB and fails. (what's this BAR for?)

Then I'm not quite sure why BAR 6 also fails to assign 512 KB as that's well below 14 MB.

I do have a potential solution to support larger 32-bit BARs using translation onto the 64-bit MMIO window, but I need to do some testing on my hardware first.

[hrushirajg23]

I have a question. If BAR6 is prefetchable , why does it need to fit into the non-prefetchable window ?

ranges = <0x81000000 0x0 0xf0100000 0x0 0xf0100000 0x0 0x100000 /* i/o 1mb (BAR 5)/
0x82000000 0x0 0xf0200000 0x0 0xf0200000 0x0 0xe00000 / non-prefetchable 14mb (BAR 0) /
0xc3000000 0x9 0x00000000 0x9 0x00000000 0x0 0x40000000>; / prefetchable 1GB (BAR 1,3,6)*/

is this right ?

I tried changing the non-prefetchable range size from 14mb to 16mb, yet doesn't work.

Regarding the translation thing , go on , great move.

[mariobalanica]

Okay, I forgot that both MMIO windows are marked as 32-bit and that the 64-bit one is already translated (CPU 0x900000000 -> PCIe 0x40000000).

ranges = <0x01000000 0x0 0xf0100000 0x0 0xf0100000 0x0 0x00100000>,
  	     <0x02000000 0x0 0xf0200000 0x0 0xf0200000 0x0 0x00e00000>,
	     <0x03000000 0x0 0x40000000 0x9 0x00000000 0x0 0x40000000>;

https://github.com/torvalds/linux/blob/v6.18-rc1/arch/arm64/boot/dts/rockchip/rk3588-extra.dtsi#L376-L378

So, what happens if you also mark the larger window as non-prefetchable?

<0x02000000 0x0 0x40000000 0x9 0x00000000 0x0 0x40000000>

In our case the prefetchability no longer matters since we have to map everything MMIO as Device memory anyway.

[akshar001]

@geerlingguy Yes, you're right. We'll keep trying until we hit a hardware dead-end with Rockchip. However, I think QCS6490 chipset that we're using for our next sbc, might perform better in situations like this, since Qualcomm has stronger PCIe hardware and DMA capabilities. Let’s see what we can do.

However, @mariobalanica you've spent years and years towards making a rockchip better eco-system, if we solve this issue, you'll be the one who'll be the credited the most, thanks for doing the hard work so far.

[hrushirajg23]

@mariobalanica you're right about making the non-prefetchable window larger , I had tried it earlier with making it into 16mb, and now as per your suggestion modifying the third range for non-prefetching.
It solved the BAR6 issue, but BAR0 issue still remains.

This is my patch , the node didn't have the iommu property , so I added it.

pci 0000:01:00.0: BAR 3: assigned [mem 0x910000000-0x911ffffff 64bit pref]
[ 15.745101] pci 0003:30:00.0: [1d87:3588] type 01 class 0x060400
[ 15.745117] pci 0000:01:00.0: BAR 0: no space for [mem size 0x01000000]
[ 15.745122] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x01000000]
[ 15.745127] pci 0000:01:00.0: BAR 6: assigned [mem 0x912000000-0x91207ffff pref]
[ 15.745132] pci 0000:01:00.1: BAR 0: no space for [mem size 0x00004000]
[ 15.745136] pci 0000:01:00.1: BAR 0: failed to assign [mem size 0x00004000]
[ 15.745141] pci 0000:01:00.0: BAR 5: assigned [io 0x200000-0x20007f]

[ 15.719190] pci 0000:01:00.0: [10de:2216] type 00 class 0x030000
[ 15.719257] pci 0000:01:00.0: reg 0x10: [mem 0x900000000-0x900ffffff]
[ 15.719312] pci 0000:01:00.0: reg 0x14: [mem 0x900000000-0x90fffffff 64bit pref]
[ 15.719366] pci 0000:01:00.0: reg 0x1c: [mem 0x900000000-0x901ffffff 64bit pref]
[ 15.719398] pci 0000:01:00.0: reg 0x24: [io 0x0000-0x007f]
[ 15.719430] pci 0000:01:00.0: reg 0x30: [mem 0x900000000-0x90007ffff pref]

[ 15.363248] rk-pcie fe150000.pcie: host bridge /pcie@fe150000 ranges:
[ 15.363274] rk-pcie fe150000.pcie: IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
[ 15.363294] rk-pcie fe150000.pcie: MEM 0x00f0200000..0x00f11fffff -> 0x00f0200000
[ 15.363309] rk-pcie fe150000.pcie: MEM 0x0900000000..0x093fffffff -> 0x0000000000

[hrushirajg23]

if BAR0 and 6 both are both 32-bit non-prefetchable why does only BAR0 fail ?

[mariobalanica]

you're right about making the non-prefetchable window larger , I had tried it earlier with making it into 16mb

Note that the whole MMIO window is 16 MB starting at 0xf0000000, so you can't simply increase the size of the non-prefetchable region given that it starts at 0xf0200000. 1 MB is reserved by the config space (see the config entry in reg) and another 1 MB is taken up by I/O space.

Also, 0xc2000000 for the 64-bit region means relocatable, prefetchable, 32-bit memory, which is bogus. See: https://elinux.org/Device_Tree_Usage#PCI_Address_Translation

In this case, the 14 MB available for 32-bit non-prefetchable memory are hopeless. I don't think having two ranges for the same memory type can even work. PCI bridges only cover a single range for each type: 32-bit non-prefetchable, 64-bit prefetchable and I/O (see Type 1 config header).

So, we should just drop the 14 MB region and have only 32-bit non-prefetchable space covered by the 64-bit window with translation:

ranges = <0x01000000 0x0 0xf0100000 0x0 0xf0100000 0x0 0x00100000>,
	 <0x02000000 0x0 0x40000000 0x9 0x00000000 0x0 0x40000000>;

Or, if some code really wants 64-bit prefetchable memory (which I doubt):

ranges = <0x01000000 0x0 0xf0100000 0x0 0xf0100000 0x0 0x00100000>,
	 <0x02000000 0x0 0x40000000 0x9 0x00000000 0x0 0x02000000>,
	 <0x43000000 0x9 0x02000000 0x9 0x02000000 0x0 0x3E000000>;

[mariobalanica]

And, for reference, what driver are you trying to bring-up here?

I believe nouveau should work with minimal or no changes as it also uses the TTM allocator, which was the main issue for DMA coherency, as it has no knowledge of device limitations. No clue about out-of-tree drivers.

Apart from that, the alignment-related patches are global and will make PCIe memory appear sane to all kernel/user-space software. There aren't any amdgpu specific fixes required.

[hrushirajg23]

I used the first one having only 32bit non-prefetchable, not getting the BAR issue, I hope this is working.

[ 5.737405] pci 0000:01:00.0: BAR 1: assigned [mem 0x900000000-0x90fffffff 64bit pref]
[ 5.737433] pci 0000:01:00.0: BAR 3: assigned [mem 0x910000000-0x911ffffff 64bit pref]
[ 5.737461] pci 0000:01:00.0: BAR 0: assigned [mem 0x918000000-0x918ffffff]
[ 5.737473] pci 0000:01:00.0: BAR 6: assigned [mem 0x919000000-0x91907ffff pref]
[ 5.737478] pci 0000:01:00.1: BAR 0: assigned [mem 0x919080000-0x919083fff]
[ 5.737489] pci 0000:01:00.0: BAR 5: assigned [io 0x200000-0x20007f]

[ 5.354739] rk-pcie fe150000.pcie: IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
[ 5.354755] rk-pcie fe150000.pcie: MEM 0x0900000000..0x093fffffff -> 0x0040000000
[ 5.354791] rk-pcie fe150000.pcie: iATU unroll: enabled
[ 5.354797] rk-pcie fe150000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G

For this one I tried nouveau, else earlier I was trying nvidia driver. I will test nvidia's also.

[ 12.564075] nouveau 0000:01:00.0: enabling device (0000 -> 0003)
[ 12.564125] nouveau 0000:01:00.0: NVIDIA GA102 (b72000a1)
[ 12.720959] nouveau 0000:01:00.0: bios: version 94.02.71.40.83
[ 12.730331] nouveau 0000:01:00.0: bios: M0203E type 0a
[ 12.730350] nouveau 0000:01:00.0: fb: 10240 MiB of unknown memory type
[ 12.751717] nouveau 0000:01:00.0: DRM: VRAM: 10240 MiB
[ 12.751719] nouveau 0000:01:00.0: DRM: GART: 536870912 MiB
[ 12.751723] nouveau 0000:01:00.0: DRM: BIT table 'A' not found
[ 12.751725] nouveau 0000:01:00.0: DRM: BIT table 'L' not found
[ 12.751728] nouveau 0000:01:00.0: DRM: TMDS table version 2.0
[ 12.751730] nouveau 0000:01:00.0: DRM: DCB version 4.1
[ 12.751733] nouveau 0000:01:00.0: DRM: DCB outp 00: 04800f86 04600020
[ 12.751737] nouveau 0000:01:00.0: DRM: DCB outp 01: 04000f82 00020020
[ 12.751739] nouveau 0000:01:00.0: DRM: DCB outp 02: 04811f76 04600010
[ 12.751741] nouveau 0000:01:00.0: DRM: DCB outp 03: 04011f72 00020010
[ 12.751743] nouveau 0000:01:00.0: DRM: DCB outp 04: 02822f66 04600020
[ 12.751744] nouveau 0000:01:00.0: DRM: DCB outp 05: 02022f62 00020020
[ 12.751746] nouveau 0000:01:00.0: DRM: DCB outp 07: 02033f52 00020010
[ 12.751749] nouveau 0000:01:00.0: DRM: DCB conn 00: 02000046
[ 12.751751] nouveau 0000:01:00.0: DRM: DCB conn 01: 01000146
[ 12.751752] nouveau 0000:01:00.0: DRM: DCB conn 02: 00020246
[ 12.751754] nouveau 0000:01:00.0: DRM: DCB conn 03: 00010361
[ 12.752265] nouveau 0000:01:00.0: DRM: MM: using COPY for buffer copies
[ 13.017887] nouveau 0000:01:00.0: DRM: allocated 1024x768 fb: 0x200000, bo 00000000d0811c1d
[ 15.274770] nouveau 0000:01:00.0: DRM: core notifier timeout

then I ran kmscube
I get

Using modifier ffffffffffffff
Modifiers failed!
Using modifier ffffffffffffff
Modifiers failed!
Rendered 0 frames in 2.093405 sec (0.000000 fps)
Rendered 1 frames in 2.067469 sec (0.483683 fps)
Rendered 2 frames in 4.134849 sec (0.483694 fps)

dmesg logs for kmscube
[ 19.544288] nouveau 0000:01:00.0: [drm] fb0: nouveaudrmfb frame buffer device
[ 19.573048] [drm] Initialized nouveau 1.3.1 20120801 for 0000:01:00.0 on minor 3
[ 25.421310] nouveau 0000:01:00.0: DRM: core notifier timeout
[ 27.787771] nouveau 0000:01:00.0: DRM: core notifier timeout
[ 29.788421] nouveau 0000:01:00.0: DRM: wndw-0: timeout
[ 33.685126] Modules linked in: nouveau drm_ttm_helper ttm nfnetlink
[ 33.687443] pc : nvkm_gr_units+0x8/0x30 [nouveau]
[ 33.688037] lr : nouveau_abi16_ioctl_getparam+0x60/0x1c0 [nouveau]
[ 33.695638] nvkm_gr_units+0x8/0x30 [nouveau]
[ 33.696869] nouveau_drm_ioctl+0x6c/0xe0 [nouveau]
[ 82.929923] nouveau 0000:01:00.0: DRM: wndw-0: timeout

[hrushirajg23]

[mariobalanica]

What happens if you do the following changes:

• return true here: https://github.com/torvalds/linux/blob/v6.17/drivers/gpu/drm/nouveau/nouveau_drv.h#L321
• Comment out this check: https://github.com/torvalds/linux/blob/v6.17/drivers/gpu/drm/nouveau/nouveau_bo.c#L238 (set nvbo->force_coherent = true; unconditionally)
• Perhaps also comment out this check: https://github.com/torvalds/linux/blob/v6.17/drivers/gpu/drm/nouveau/nouveau_bo.c#L1291 (set reg->bus.caching = ttm_uncached; unconditionally)