I came across a pretty exotic bug on IMX8MP which I intend to eventually fix, but so far I managed to more or less isolate and bandaid it. Posting it here just in case someone tripped over it already, or may trip in future. And because I'm fairly certain it's kernel-related. Please feel free to refer me to the mailing list if this should be raised at mainline instead.
The setup is pretty simple, I use UART2 as a boot console with 115200 baud (default for Olimex and EVK). Unlike the boards mentioned I do not have a pull-up on UART RX and it goes through a 74LVC1T45DW buffer/level shifter IC.
The board tends to lock up time to time at random intervals (like in several hours of uptime), but one clue for what's happening was when I tried to see what's in the output during this event via a USB-UART adapter on the port, it didn't crash for two days straight. I started looking in the driver code for anything out ordinary and found this gem of working around hardware issues:
https://github.com/Freescale/linux-fslc/blob/6.12.x+fslc/drivers/tty/serial/imx.c#L810
I managed to force the lockup condition mentioned in the comment and here's what I'm getting:
- case 1: Kernel is operating the console as per
console=ttymxc1,115200 passed from the bootloader. When I trigger the flooding, the kernel stops kernelling, the processes freeze although the heartbeat led does seem to stay blinking at times (maybe it's timer based?)
- case 2: Kernel is told not to operate the console with
console=null. When I trigger the flooding, the driver code manages to detect it and counteract (as seen by dmesg messages), the kernel stays intact. I tried both keeping a getty over the device as well as consume data from it directly with screen.
So it would seem there's serious issue within the kernel console code specifically. My hypothesis is that the unconnected floating RX pin catches some radio noise big enough to fire the level shifter and it can just coincide with the conditon for flooding randomly. Of course I should and will pull the pin up or something to make sure I'm not getting input garbage in first place, but this seems to be pretty dangerous behaviour overall.
This seems to be either the same or a similar bug too: https://community.toradex.com/t/torizon-os-crash-with-rx-flooding-on-uart/27843/9
If nobody minds I'll use the ticket to track the investigation, I'll try doing kdump/kexec magic or some other way to actually get what the kernel outputs before locking up if anything when I have time.
I came across a pretty exotic bug on IMX8MP which I intend to eventually fix, but so far I managed to more or less isolate and bandaid it. Posting it here just in case someone tripped over it already, or may trip in future. And because I'm fairly certain it's kernel-related. Please feel free to refer me to the mailing list if this should be raised at mainline instead.
The setup is pretty simple, I use UART2 as a boot console with 115200 baud (default for Olimex and EVK). Unlike the boards mentioned I do not have a pull-up on UART RX and it goes through a 74LVC1T45DW buffer/level shifter IC.
The board tends to lock up time to time at random intervals (like in several hours of uptime), but one clue for what's happening was when I tried to see what's in the output during this event via a USB-UART adapter on the port, it didn't crash for two days straight. I started looking in the driver code for anything out ordinary and found this gem of working around hardware issues:
https://github.com/Freescale/linux-fslc/blob/6.12.x+fslc/drivers/tty/serial/imx.c#L810
I managed to force the lockup condition mentioned in the comment and here's what I'm getting:
console=ttymxc1,115200passed from the bootloader. When I trigger the flooding, the kernel stops kernelling, the processes freeze although the heartbeat led does seem to stay blinking at times (maybe it's timer based?)console=null. When I trigger the flooding, the driver code manages to detect it and counteract (as seen by dmesg messages), the kernel stays intact. I tried both keeping a getty over the device as well as consume data from it directly withscreen.So it would seem there's serious issue within the kernel console code specifically. My hypothesis is that the unconnected floating RX pin catches some radio noise big enough to fire the level shifter and it can just coincide with the conditon for flooding randomly. Of course I should and will pull the pin up or something to make sure I'm not getting input garbage in first place, but this seems to be pretty dangerous behaviour overall.
This seems to be either the same or a similar bug too: https://community.toradex.com/t/torizon-os-crash-with-rx-flooding-on-uart/27843/9
If nobody minds I'll use the ticket to track the investigation, I'll try doing
kdump/kexecmagic or some other way to actually get what the kernel outputs before locking up if anything when I have time.