Linux: [Questions] Raspberry Pi 4 upstreaming

@pelwell any ideas?

@P33M wrote/copied the GIC dts.

1. Maintenance interrupt is standard, I assume. The GIC is unmodified IP. Property is missing.
2. interrupt-affinity follows the IRQ definitions - first cell corresponds to cpu0, next cpu1 etc. Property is missing. Also, we should delete the a53 PMU compatible parameter and just leave the a72 parameter.
3. I believe the property is needed - the arm stubs don't do any timer setup except for setting the counter frequency and zeroing CNTVOFF_EL2.
   https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S

3a. Removing the always-on property has side-effects. On an idle system with the property removed, I get 500 IRQs per second in vmstat with equal numbers of arch timer interrupts across all 4 cores. With the property added, I get ~110 IRQs per second in vmstat with differing numbers of timer interrupts. It seems like CPUs aren't dropping into a tickless idle state - which would have power consumption implications.

As a first pass, fixes for 1 and 2 could also be taken downstream:

https://github.com/raspberrypi/linux/pull/3104

Thanks a lot

More questions:

1. Is it correct that "brcm,bcm2835-system-timer" @7e003000 isn't available for BCM2711 anymore?
2. Is the timer setup done for ARMv7 and ARMv8?
3. Is the timer setup about clk frequency and CNTVOFF_EL2 done for all ARM cores?
4. Is it correct that the ARM cores never enter any deep sleep modes, which disable the timer comparators?

Sorry, for be annoying but there are only few days left before my holiday. I need the answers before i can send out the next version of my patch series.

The system timer register at 0x7e003000 still exists and is used by the firmware, I think it's 0xfe003000 ARM side. Start.elf and the bootloader initialise it to run at 1MHz from the OSC at the start of day.
Sorry, I don't know enough about the ARM side setup to comment on the deep sleep modes or CNTVOFF_EL2

Okay, this answers question 1 and makes the movement of timer@7e003000 into bcm2835-common.dtsi wrong.

Regarding the armstub7.S i couldn't find something related to the timer setup, so i assume this only done in armstub8.S

@timg236 Since we have a GIC now, it would be to necessary to define the 4 interrupts accordingly.

Is it possible to publish those definitions?

@pelwell @P33M Is it possible to get the GIC interrupts for the BCM2711 system timer?

You should be able to work it out by inter/extrapolation from the others.

Okay, i will take the following:

interrupts = <GIC_SPI 64 IRQ_TYPE_LEVEL_HIGH>,
             <GIC_SPI 65 IRQ_TYPE_LEVEL_HIGH>,
             <GIC_SPI 66 IRQ_TYPE_LEVEL_HIGH>,
             <GIC_SPI 67 IRQ_TYPE_LEVEL_HIGH>;

Yes - that looks correct.

Thanks

Regarding upstreaming there are more questions:
1) Are the genet clocks (called genet125 and genet250 in vcgencmd) accessible for bcm2835-cprman? What's their ctl and div register? How is the tcnt_mux?
2) Is there a interrupt connected to the Ethernet PHY?
3) How long is the register range of the thermal block?

1. The Genet clocks are from generic cprman divisors.

CM_GENET_125CTL = 0x7e101210
CM_GENET_125DIV = 0x7e101214
CM_GENET_250CTL = 0x7e1011e8
CM_GENET_250DIV = 0x7e1011ec

tcntmux has genet_250 as 45, genet_125 as 50.

1. No :( MDIO polling is used for link state.
2. DTS has the thermal registers in:

        thermal: thermal@7d5d2200 {
            compatible = "brcm,avs-tmon-bcm2838";
            reg = <0x7d5d2200 0x2c>;
            interrupts = <GIC_SPI 137 IRQ_TYPE_LEVEL_HIGH>;
            interrupt-names = "tmon";
            clocks = <&clocks BCM2835_CLOCK_TSENS>;
            #thermal-sensor-cells = <0>;
            status = "okay";
        };

?
Or are you asking about readout field bitwidths?

According to point 3: the problem is that i suggested register range in the DTS file comes from me and based on the assumption that the BCM2711 has a AVS TMON block. AFAIR the original register range was 4. According to the bit defines the BCM2711 seems to be not compatible to any existing bcm thermal driver. So would like to use the real register range based on the hardware. At least reading out the register indicate a bigger range than 4.

The temperature reg being used is actually part of AVS_RO and not AVS_TMON. The block has a whole heap of registers to do with ring oscillators/core voltage monitoring but the only one to do with temperature is at 0x7d5d2200.

It looks like there is a AVS_TMON block at 0x7d5d1500 with trips/interrupt registers in the right places - what happens if you point the brcmstb_thermal driver at that address?

Thanks, i'll give it a try.

Unfortunately this doesn't work. The brcmstb_thermal driver complains about invalid readouts. An attempt to read the registers via /dev/mem result at 0xfd5d1500 results in kernel faults.

@P33M Is this correct ?

       /* GENET clocks (only available for BCM2711) */
       [BCM2711_CLOCK_GENET250]        = REGISTER_PER_CLK(
               SOC_BCM2711,
               .name = "genet250",
               .ctl_reg = CM_GENET250CTL,
               .div_reg = CM_GENET250DIV,
               .int_bits = 4,
               .frac_bits = 8,
               .tcnt_mux = 45),

The documentation refers to 4 bit integer and 8 bit fractional divisor widths, so I'd say it is.

Thanks

Some more questions:

1. According to the bcmgenet binding there is a optional third interrupt for Wake-on-LAN. In case this is available on BCM2711, what is the interrupt number?
2. Is the following assumption correct: BCM2711_CLOCK_GENET250 = normal operation clock, BCM2711_CLOCK_GENET125 = Wake-on-LAN clock?
3. There is a mismatch in bcm2838.dtsi at the ethernet phy between genet-phy@0 and the reg value 0x1.
   I assume reg value is the correct one.
4. Who provides the MAC address for the genet driver?
5. Which BCM2711 clock can i pass as EEE clock to the genet driver?

Edit:
Answers:

1. BCM2711_CLOCK_GENET125 is the ref clock for RGMII. Not sure which clock is suitable for Wake-on-LAN. Unfortunately we need to provide it, even we don't use Wake-on-LAN.
2. reg value 0x1 is correct
3. The bootloader uses alias ethernet0
4. According to the driver the clock must have 27 MHz.

I sent out V3 of the initial RPi 4 support today and there is at least one request for comments.

bcm2711.dtsi

soc {
        ranges = <0x7e000000  0x0 0xfe000000  0x01800000>,
             <0x7c000000  0x0 0xfc000000  0x02000000>,
             <0x40000000  0x0 0xff800000  0x00800000>;

Might be nice to get some comments about

@pelwell gentle ping

Lots of things might be nice, but are they necessary? It's as well documented as all other chips:

    soc {
        ranges = <0x7e000000 0x3f000000 0x1000000>,
             <0x40000000 0x40000000 0x00001000>;
        dma-ranges = <0xc0000000 0x00000000 0x3f000000>;

etc.

Try this:

        ranges = <0x7e000000  0x0 0xfe000000  0x01800000>,  // Common BCM283x peripherals
             <0x7c000000  0x0 0xfc000000  0x02000000>,  // BCM2838-specific peripherals
             <0x40000000  0x0 0xff800000  0x00800000>;  // ARM-local peripherals

Some more questions:

1. According to the bcmgenet binding there is a optional third interrupt for Wake-on-LAN. In case this is available on BCM2711, what is the interrupt number?

2. Is the following assumption correct: BCM2711_CLOCK_GENET250 = normal operation clock, BCM2711_CLOCK_GENET125 = Wake-on-LAN clock?

3. There is a mismatch in bcm2838.dtsi at the ethernet phy between genet-phy@0 and the reg value 0x1.
   I assume reg value is the correct one.

4. Who provides the MAC address for the genet driver?

5. Which BCM2711 clock can i pass as EEE clock to the genet driver?

@P33M gentle ping

Wake on LAN won’t work on Pi4 model because the interrupt from the phy is not connected.

The MAC address comes from OTP and is passed via command line and:or device tree.

Others might have to comment on the EEE or 125 MHz clock.

@timg236 Thanks for your fast reply. I need to define two separate things the BCM2711 and the Raspberry Pi 4. Since the genet binding provides 3 interrupts i assumed the third was available on the SoC / GIC400, because the PHY interrupt is defined in the PHY node.

Now the initial RPi 4 DTS has been applied, we are working on LAN support. So any help about the clock assignment is appreciated.

I was asking about MAC address, because i noticed that the passing via DT didn't worked with the cleaned-up version of genet node.

I'm not surprised it isn't working - what happened to the aliases? You are supposed to have an alias called ethernet0 referring to the MAC node for the primary interface. This is a standard Linux pattern, and it is what the firmware uses to locate where to write the MAC address.

Sorry, my bad. Thanks for pointing out.

During conversation with Florian, he said that the phy mode provided by rpi4 dts doesn't seem to match the hardware.

All possible options from the YAML binding:

RX and TX delays are added by the MAC when required:

• rgmii

RGMII with internal RX and TX delays provided by the PHY, the MAC should not add the RX or TX delays in this case:

• rgmii-id

RGMII with internal RX delay provided by the PHY, the MAC should not add an RX delay in this case:

• rgmii-rxid

RGMII with internal TX delay provided by the PHY, the MAC should not add an TX delay in this case:

• rgmii-txid

According to Florian's explanation 360c8e98883f9cd075564be8a7fc25ac0785dee4 is only a workaround to avoid calling bcm54xx_config_clock_delay and to preserve the default delays.

I dumped the following register values before bcm54xx_config_clock_delay tries to change the values:
SHDWSEL_MISC: 0x71E7
SHD_CLK_CTL: 0x0100

Based on this settings the PHY mode would be: rgmii-rxid

I'm not sure this is correct.

The better approach would be to add phy mode support for the missing types (rgmii-id & rgmii-rxid) to the GENET driver and configure the right mode via devicetree.

Would be nice to get a confirmation about the right PHY mode.

I'm not very familiar with the Linux bcm genet driver but checked the firmware (*) and it doesn't do any configuration of GENET or the PHY. The 0x0100 for CLK_CTL lines up with the default value in the datasheet.

• The bootloader network boot doesn't configure the CLK_CTL either. If network boot was used then MAC DMA setup is stopped and cleared before starting Linux but that's about it.

I have a question regarding the ring oscilator block which came up during thermal upstreaming:

What is the register offset and size in hex of the ring oscillator (AVS RO) block?

AVS_RO_REGISTERS_0: 0x7d5d2200-0x7d5d22e3

This looks strange too me due all the other blocks are aligned to 0x1000. What is at 0x7d5d2000?

There is a top level block that contains a number of sub-blocks (AVS_RO_REGISTERS_0 is one).
AVS_MONITOR 0x7d5d2000 - 0x7d5d2eff

Since pcie support will land soon, now the focus is on USB support. The commit b9b1e6f49e64fa1cd4807f07bb8e9c70f896de0a doesn't have any explanation.

Why is this quirk the right way to gain LPM support?
What is the source that says VIA VL805 support LPM?

Why is this quirk the right way to gain LPM support?

Do you know of a better way?

What is the source that says VIA VL805 support LPM?

The documentation we have from VIA, and observation of the bus showing it entering the LP states.

I'm not against the funtional change, but we need a good explantion in the commit log. The fact that we need to use a quirk let me think there is some kind of issue with the VL805. I don't know anything about PCIe but it looks like the LPM advertisement of this chip isn't properly implemented.

@P33M Was this quirk required because the PCIe capabilities had 0us for the RestoreTime and PowerOnTime? If so, is this quirk no longer required if running the current 0137ab firmware?

I think it was because U1/U2 exit latencies were 0us. With 0137ab I see that they are set to 4 and 231us respectively. Reverting to test...

Yes - sudo lsusb -v | & grep ExitLat shows non-zero values for me.

Ah. With the commit reverted the root hub descriptor doesn't have the exit latencies populated. It requires setting the quirk for xhci_create_usb3_bos_desc() to populate the fields from the HC params, so LPM is disabled by default on all host controllers that don't match the big table in xhci-pci.c:xhci-pci-quirks().

Okay, based on the last answer i consider the commit b9b1e6f49e64fa1cd4807f07bb8e9c70f896de0a as upstreaming candidate.

New questions regarding GPIO:

1. Which GPIOs of the BCM2711 are connected to the SPI Flash?
2. Are the signals HDMI0_HPD_N and HDMI1_HPD_N GPIOs of the BCM2711?
3. Are the following BCM2711 GPIOs connected to a specific function?
4. GPIO27, GPIO28, GPIO29, GPIO46, GPIO47, GPIO48, GPIO49, GPIO50, GPIO51, GPIO52, GPIO53

gentle ping @pelwell

1. GPIOs 40-43 are MISO, MOSI, CLK and CE_N. However, 40&41 are shared with the audio PWM, so don't access the flash with the audio amplifier enabled. It would be better to treat this a ROM that the firmware can update.
2. Yes - BCM2711 has dedicated HTPLG pins for HDMI0 and HDMI1.
   3.
   GPIO27 is on the 40-pin header, so no.
   GPIO28&39 drive the Ethernet PHY MDIO interface, i.e. rgmii_mdio_gpio28 (raspi-gpio reports RGMII_MDIO and RGMIO_MDC).
   GPIO46-51 are RGMII_RX, and GPIO52-57 are RGMII_TX. These are switched by a higher-level pin mux bit set by the firmware, hence why raspi-gpio doesn't see anything.

Thanks. Oops, i wasn't aware that the BCM2711 supports more GPIOs.

So there are more questions (hoping you are allowed to answer):

1. What are the register ranges for GPIO54-57 (BCM2711)?
2. What are the exact signal names (e.g. RGMII_RXD0 ...) for GPIO46-57?

What are the register ranges for GPIO54-57 (BCM2711)?

The same - they fill in some of the unused bits in the BCM2835 registers, with the obvious offsets.

What are the exact signal names (e.g. RGMII_RXD0 ...) for GPIO46-57?

46: RGMII_RXCLK
47: RGMII_RXCTL
48: RGMII_RXD0
49: RGMII_RXD1
50: RGMII_RXD2
51: RGMII_RXD3

52: RGMII_TXCLK
53: RGMII_TXCTL
54: RGMII_TXD0
55: RGMII_TXD1
56: RGMII_TXD2
57: RGMII_TXD3

Hold on. GPIO48–GPIO53 was previously used to access the on-board microSD card reader (using AltFn 0 for SD0_xxx). If these GPIOs are now used fo RGMII, where is EMMC now?

The Raspberry Pi 4 has 3 SD card interfaces (sdhost, sdhci and emmc2). The new emmc2 which is currently used for SD cards is hardwired and not accessible via GPIO.

Ah, right, that's configured by the new undocumented GPIO register at 0x7e2000d0… But I never thought that there are also dedicated pins for the SD card now. Anyway, I believe it's also possible to map the sdhci interface to those pins using the above-mentioned register. Let me try out a few things with that register.

Edit: Yes, it seems the register at 0x7e2000d0 (can I provisionally call it GP_SDCARD until we learn the official name?) controls how those dedicated SD card pins are routed:

• bit 0: connect the pins to the Arasan MMC interface if set
• bit 1: connect the pins to the new EMMC2 interface if set

Okay, but that doesn't actually mean the EMMC2 is connected to GPIO48-53.

No, surely not.

FTR the firmware prints out +DEDICATED_SD_PINS at boot to indicate this feature.

I've got a question about the use of I²C for the on-board PMIC (XR77004). The firmare on a RPi4 prints out +PMIC_XR77004 and +SMPS_I2C. RPi 3B and RPi 3B+ also had +PMIC_XR77004, but then +SMPS_GPIO.

The built-in pin configuration says the following for RPi 3B+:

• /pins_3bplus/pin_defines/pin_define@SMPS_SDA { type="internal"; number=<46> }
• /pins_3bplus/pin_defines/pin_define@SMPS_SCL { type="internal"; number=<47> }

On the RPi 4B, both SMPS_SDA and SMPS_SCL are defined as type="absent". This makes some sense, since pin 46 is now RGMII_RXCLK, and pin 47 is now RGMII_RXCTL. Does it mean, there are also dedicated pins for the I²C bus with the PMIC? If yes, which I²C block is attached to it?

Never mind. By tracing hardware register accesses on the VPU, it appears that there is one more (undocumented) I²C block at 0x7e205e00, and the firmware communicates to a device with address 0x1d. This address is used by the XR77004. Since the I²C traffic does not appear on any other accessible pin, I think it's safe to assume the PMIC is on its own dedicated pins.

I have some questions regarding Maxime's KMS patch series (don't want to bother the review of this very long series, so here):

1. Is the 108MHz-clock accessible by bcm2835-cprman?
2. What is it parent?
3. How is the tcnt_mux?
4. Can the clock act as a gate?

108 is from PLLD PER which is unfortunate because PLLD PER is actually 750 MHz so not an integer divider.

It's a CPRMAN clock CTL 0x7e101200 DIV 0x7e101204
tcnt 48 = clk_108

I have a feeling that it's required for PCIe and should be treated as an always-on clock