Skip to content

DEVLOG.md

Latest commit

 

History

History
709 lines (597 loc) · 44.6 KB

DEVLOG.md

File metadata and controls

709 lines (597 loc) · 44.6 KB
Raw

z3s Mainline Linux Port, Developer Log

Project: boot mainline Linux on Samsung Galaxy S20 Ultra 5G (SM-G988B / z3s / Exynos 990) Started: 2026-07-22. Device dedicated to this project. Phone rooted (Magisk), bootloader UNLOCKED, vbmeta orange.

1. Device dumps (2026-07-22), 01_dumps/

  • All 29 partitions dd'd via rooted adb, MD5-verified vs live partitions.
  • super (8.8GB) split via custom Python LP parser (05_tools/lpunpack.py) → 02_extracted/super_extracted/{system,vendor,product,odm}.img
  • LP metadata quirks found: geometry magic gDla@4096, backup@8192, header LP0A@12288 (NOT 8192).
  • vendor.img extracted fully to 02_extracted/vendor_fs/ via WSL debugfs rdump: 109 firmware blobs (WiFi BCM4375 bcmdhd, BT hcd, abox audio DSP, cs35l40 amps, camera ISP, fingerprint a96t365, NPU tflite), 829.so HALs, TrustZone TAs, VINTF manifest.
  • efs/sec_efs contain IMEI, never share. Backup exists for signal recovery.

2. Hardware identity (from dumps + live device)

  • SoC Exynos 990 = internal "exynos9830". Stock kernel 4.19.87 (Samsung downstream, clang 8).
  • 12GB RAM. System RAM map (live /proc/iomem): 0x80000000+0x3ab00000 | 0xc1200000+0x1ee00000 | 0xe1900000+0x1e700000 | 0x8_80000000+0x2_80000000
  • Panel: Samsung s6e3hab "hubble3_a3_s0" 1440x3200 120Hz DSI (driver source in Samsung kernel: drivers/video/fbdev/exynos/panel/s6e3hab/). lcdtype=8462596.
  • Touch: sec_ts (I2C). PMIC: s2mps19+s2mps22. Charger/MUIC: max77705. GPU: Mali-G77 MP11.
  • WiFi/BT: BCM4375 (brcmfmac-compatible, firmware in vendor_fs/firmware).
  • UFS: samsung,exynos-ufs. Modem: Shannon (exynos,modem_if), radio.img dumped.
  • Board revision = 23 = 0x17 (androidboot.revision=23). dtbo_idx = 8 → stock overlay #8 (of 10) is OUR board's config (02_extracted/dts/dtbo_8_*.dts).

3. Samsung boot image format, verified against known-working magisk_patched boot.img

Header v2, page 2048. Samsung layout quirks (DIFFER from AOSP struct):

  • recovery_dtbo_size@1632, recovery_dtbo_offset@1636 (u64 UNALIGNED), header_size=1660@1644, dtb_size@1648, dtb_addr@1652 (u64), dtb_addr MUST be 0x10000000 (0x0 → boot fail).
  • kernel_addr 0x10008000, ramdisk_addr 0x11000000, tags 0x10000100.
  • os_version 0x16000191, name "SRPSI19B023KU".
  • The dtb section is NOT a raw FDT, it's a DTBO table (magic 0xD7B7AB1E big-endian): 8-field header (32B: magic,total,hdrsz,entry_size,count,entries_off,page,reserved) + 32B entries {size,offset,id,rev,custom[4]}.
  • Entry custom fields = board rev window: custom[0]=hw_rev_start, custom[1]=hw_rev_end. Stock boot dtb entry: 0x00-0xff (matches all). Stock dtbo partition: 10 entries with windows 0x04, 0x05, 0x06, 0x10, 0x11, 0x12-13, 0x14, 0x15, 0x16-0x17 (=our board), 0x18-0xff.

4. S-LK bootloader boot checks (learned by failing, in order)

  1. dtb table header check fail badmagic → dtb section must be DTBO table (fix: pack_dtbo.py)
  2. not found 0x17 load fail → dtb_addr was 0x0 (fix: 0x10000000) AND/OR entry rev window must include 0x17
  3. no node __symbols__ in main, failed to perform fixups → base dtb needs dtc -@ (kbuild: DTC_FLAGS_exynos990-z3s += -@)
  4. couldn't find "pinctrl_0" symbol → Samsung's dtbo overlays reference DOWNSTREAM labels; they can never apply onto a mainline tree. Fix: replace dtbo partition with no-op overlay (valid table, 1 entry rev-window 0x00-0xff, trivial /plugin/ overlay adding marker property).
  5. dt entry: not found ref 0x17 → fully empty dtbo also rejected: S-LK REQUIRES ≥1 matching entry.
  6. → boots to frozen Samsung logo = bootloader accepted image, kernel took over.

5. Mainline kernel state (7.2-rc4, cloned to WSL /root/linux)

  • Upstream has: exynos990.dtsi (419 lines: CPUs, PSCI, GIC, MCT, WDT, chipid, cmu-top/peric0/1/peris/hsi0, pinctrl x6, usbdrd phy+dwc3, pmu/reboot, timer), clk-exynos990.c, ufs-exynos.c, pinctrl support.
  • Device trees upstream: x1s (S20), x1slte, c1s (Note20), r8s (S20 FE), thin, boot-stage only.
  • NO UART/serial nodes in mainline dtsi (patches pending on LKML; debug UART likely 0x108c0000, earlycon=exynos4210,, noted for later, user prefers no-jig debugging for now).
  • Our tree: arch/arm64/boot/dts/exynos/exynos990-z3s.dts includes x1s-common (simplefb@0xf1000000 1440x3200, gpio-keys gpa0-3/gpa0-4/gpa2-4, usbdrd okay, dummy regulator).
  • Config: defconfig + FB_SIMPLE, SERIAL_SAMSUNG(+CONSOLE), DWC3_EXYNOS, USB configfs serial gadget, DEVTMPFS, INITRD. CONFIG_FRAMEBUFFER_CONSOLE was already y.
  • Kernel Image 51MB (stock 39MB), fits 59MB boot partition.

6. Live-device intel (rooted stock boot)

  • bootargs goldmine: s3cfb.bootloaderfb=0xf1000000 (confirms simplefb address), console=ram, sec_debug_next=0x1000000@0x91200000 (Samsung RAM console = our pstore site), lcdtype=8462596, dram 12G, sec_atd.tty=/dev/ttySAC0.
  • GPIO keys identical to x1s (vol- gpa0-4, vol+ gpa0-3, power gpa2-4).

7. Current status / debugging strategy (no-UART)

  • Symptom: kernel takes over, frozen Samsung logo, no USB enumeration, no screen text. FRAMEBUFFER_CONSOLE is enabled, so silence suggests very early death (before/at fbcon).
  • Plan A (this build): ramoops pstore @ 0x91200000 (16MB, S-LK-preserved across warm reset): CONFIG_PSTORE_CONSOLE/PMSG/RAM=y, COMPRESS=n. dts reserved-memory + ramoops node added. cmdline += ignore_loglevel loglevel=8 (S-LK appends loglevel=4, ignore_loglevel wins).
  • Read-back loop: mainline hang → Vol-+Power (WARM reset, preserves RAM) → download mode → Odin STOCK_RESTORE.tar → Android → dd /dev/mem @0x91200000 (root) → analyze log.
  • Also still in play: simplefb screen console (should print if kernel reaches fbcon), USB ACM serial gadget (VID 0x04e8 PID 0x6860 "SAMSUNG Mobile USB CDC Composite") once initramfs runs.
  • Flash path while rooted: dd boot image straight to /dev/block/by-name/boot (no Odin needed until a boot fails). STOCK_RESTORE.tar = stock boot+dtbo via Odin for recovery.
  • LESSON (hit on 2026-07-22): STOCK_RESTORE.tar restores stock dtbo too. Any dd-flash of boot afterwards MUST be paired with re-flashing the no-op dtbo (dd or DTBO_ONLY.tar), else S-LK applies Samsung overlays again -> 'pinctrl_0 symbol' failure. Rule: mainline boot.img + no-op dtbo ALWAYS go together. (Hit AGAIN same day after STOCK_RESTORE -> dd boot-only -> pinctrl_0. Now enforced by 05_tools/flash_z3s.sh which always dd-flashes BOTH partitions in one shot.)

8. Toolchain inventory (05_tools/)

9. Blind-debug campaign (2026-07-22 evening)

Goal: find where mainline dies without UART. Results in order:

  • pstore/ramoops build: console records never appeared in stock /sys/fs/pstore after cycle => death is EARLY (before pstore driver init).
  • last_kmsg after cycle contained PREVIOUS Android session's log => RAM survives warm resets AND Odin flashes. Read-back channel proven viable.
  • EFI stub found + disabled (stock kernel has plain b entry; ours had MZ header). Text_offset 0x0 vs stock 0x80000 (mainline standard, probably fine).
  • Framebuffer stripes: inconclusive, s6e3hab is DSI command-mode, panel GRAM holds logo regardless of DRAM writes.
  • 90s self-reboot initramfs test: no reboot => never reached initramfs (OR syscon-reboot broken).
  • DRAM boot-stage markers @ 0x91A00000 (magic 0x5A35C0DE + stage bitmask in init/main.c: stage0 start_kernel, 1 setup_arch done, 2 rest_init, 3 kernel_init), flashed, markers presumed in RAM, READING them is the blocker.
  • Read-back attempts: /dev/mem DOESN'T EXIST on stock (DEVKMEM off). /proc/kcore absent. insmod: SELinux enforcing, setenforce 0 denied even for magisk:s0, init_module denied (EPERM) for every reachable context; /data/adb inaccessible from adb-su context. Magisk ramdisk has NO init.rc (magiskinit) so rc-injection into /init.rc does nothing.
  • Built FULL stock kernel from source (gcc): needed -Wno-error patches, exynos-devfreq.h int fix, mm.h extern-inline fix, vmscan.c inline fix, bcmdhd clang-flag fixes, lib/lockref.o -O1 (gcc9 ICE), python2 for RKP_CFP. Module.symvers obtained. ramdump.ko builds but insmod still ENOEXEC.
  • Built-in dump driver (late_initcall + kthread writing /data/local/tmp/z3s_markers.txt): compiled INTO stock build => gcc-built stock kernel HANGS AT LOGO TOO (2026-07-22 ~23:35). Samsung 4.19 tree is clang-only in practice (or RKP_CFP instrumentation matters).
  • Boot images inventory (04_build/): STOCK_RESTORE.tar (= magisk boot+stock dtbo, from dumps), STOCK_RAMDUMP.tar (stock kernel + insmod init.rc - DOES NOTHING, magisk ramdisk has no init.rc), MAGISK_RESTORE.tar (from user's original magisk_patched.tar), boot_stock_ramdump.img / boot_stockdump_builtin.img (dd-flashable variants), boot_z3s_mainline.tar (mainline kernel + noop dtbo; MUST stay paired).

10. BREAKTHROUGH: lk3rd (2026-07-22 ~23:40)

Discovered via postmarketOS Exynos990 wiki: the stock Samsung bootloader CANNOT boot mainline, it expects Samsung-specific DT overlay nodes that don't exist in mainline trees. The entire community uses lk3rd (github.com/exynos990-mainline/lk3rd) instead.

lk3rd is a secondary bootloader:

  • Flashed to BOOT partition via Odin (S-LK loads it as a 'kernel')
  • lk3rd relocates itself, initializes DECON/UFS/USB cleanly
  • Presents a fastboot interface (Pixel-style)
  • fastboot oem enable-mainline-quirks = one-time setup
  • fastboot boot boot.img = boot from RAM without flashing (instant test iteration!)

The community recipe (standard AOSP mkbootimg v2): mkbootimg --kernel Image.gz --dtb exynos990-z3s.dtb --ramdisk initramfs --header 2 Key differences from our Samsung approach:

  • Image.gz (gzip-compressed) instead of raw Image
  • RAW.dtb instead of DTBO-table-wrapped
  • Standard AOSP header (dtb_addr = base+0x01f00000, not Samsung's 0x10000000)
  • NO dtbo partition needed (no overlays = no pinctrl_0/symbols errors)

Files ready (04_build/):

  • lk3rd-z3s.tar (Odin flash to BOOT → installs lk3rd bootloader)
  • boot_lk3rd.img (16MB: Image.gz 15M + initramfs 1.2M + raw dtb 64K)

11. lk3rd campaign results (2026-07-23)

  • lk3rd 2.4-hotfix flashed to BOOT via Odin. Boots to FastBoot UI. fastboot commands work (after installing Google USB driver with lk3rd PID 0x4D00 via pnputil).
  • fastboot oem enable-mainline-quirks works.
  • BLOCKER: fastboot boot fails for ALL images incl. lk3rd's own ("Boot image magic not found" mainline path / "boot failure!" android path). fastboot flash also corrupts (bricked dev lk3rd to Samsung-logo hang). Systematic download-path corruption.
  • Theories: (A) cache coherency USB-DMA vs CPU (no arch_invalidate_cache_range in lk3rd download path), (B) USB3 hw corruption on z3s. z3s USB also drops/reconnects in lk3rd mode.
  • Built lk3rd from dev branch (3 compile patches: start.S index->idx SVE conflict, fastboot_cmd.c VLA init + case label, arch/ops.h include for cache fix). Packaged with kdrag0n mkbootimg + z3s rules.mk args. Added arch_invalidate_cache_range before memcpy in mainline_boot_fb_boot + boot_fb_boot. Result: build UNRELIABLE (boots sometimes, Samsung- logo hang others), and fastboot boot STILL failed with "boot failure!" when it did boot. Cache theory NOT definitively disproven (fix placement/build quality uncertain).
  • usbipd-win installed (UsbdK filter conflict, attach failed). WSL has fastboot 8.1.0.
  • Handoff written: HANDOFF.md, full state + next steps for the continuing agent.

12. Immediate next actions for continuing agent

  1. Reflash release lk3rd if needed (04_build/lk3rd-z3s.tar via Odin), dev build is untrusted.
  2. Determine root cause of fastboot download corruption (cache vs USB3), build lk3rd with -l logging + UART, or fix download-path cache invalidate placement properly.
  3. Consider bypass: TWRP-based kernel loading or lk3rd patched to boot from a dedicated partition (avoid flaky fastboot entirely).
  4. UART jig (619k) would resolve all debugging instantly, lk3rd UART is toggleable.
  5. When fastboot boot works: fastboot boot 04_build/boot_lk3rd.img -> mainline runs.
  • lpunpack.py (super splitter), extract_dtbo.py (DTBO table parser), extract_boot.py (boot.img parser), pack_dtbo.py (DTBO table writer, hw_rev 0x00-0xff), mkbootimg_z3s.py (Samsung-layout boot packer), fix_makefile.py, setup_config.sh, init (initramfs: kmsg markers, USB ACM gadget, heartbeat, shells).
  • busybox 1.36.1 static aarch64 built in WSL (/root/busybox-1.36.1).
  • Kernel build: WSL2 Ubuntu, gcc-aarch64 9.4, nice -n 19 make -j2 (user's PC must stay usable!)
  • WSL kills background processes on session close; Git Bash mangles /paths (use MSYS_NO_PATHCONV=1 or //double-slash for adb).

SESSION 3, 2026-07-23, "THE OS IS ALIVE"

Ended the session with Debian 12 booting from the microSD: root@z3s:~# on the phone's own display, PID 1 = systemd, systemctl is-system-running = running, 58 GB free.

1. UFS deep-dive (six real bugs fixed, still not linking)

Backported the 6.14 UFS WIP onto stable 6.12 and drove it from "doesn't probe" to "host is perfect, device is silent":

  • 0x9514 / PA_DBG_CLK_PERIOD, mainline issues this as a DME_SET; on exynos990 (embedded UFS) the downstream writes it as a UNIPRO register instead. Gave exynos990 its own uic_attr with pa_dbg_clk_period_off = 0 so config_unipro() skips the rejected write.
  • PHY calibration incomplete, added the 8 missing PHY_PMA_COMN writes (regs 0x00a, 0x00d, 0x00e, 0x011, 0x02c, 0x041, 0x013, 0x048) that configure the common block before the 0x043 cal-trigger toggle.
  • PHY lane offset was 4x too large (real bug): downstream stride is 0x800 bytes, but mainline applies PHY_APB_ADDR (<<2) on top, so the index-space value must be 0x200 (gs101 uses 0x200 too). At 0x800 every lane-1 register was written to the wrong address.
  • Added wait_for_cal (gs101-style poll of TRSV reg 0x338 bit 3 = LN0_MON_RX_CAL_DONE), the mainline equivalent of downstream's PHY_EMB_CAL_WAIT.
  • Removed automotive-only hooks (exynosauto_ufs_post_hce_enable enables Virtual-Host / Multi-Host, which a phone SoC does not have) and added the UniPro connection establishment (N_DEVICEID, N_DEVICEID_VALID, T_PEERDEVICEID, T_CONNECTIONSTATE).
  • Fixed CLKS_NR_TOP in clk-exynos990.c: it was CLK_GOUT_CMU_APM_BUS + 1 (=137) but clock IDs run to 191, so every clock above 137 was silently unavailable. Now VRA_BUS + 1. (This later mattered, the MMC clocks are IDs 169/170.)

Verified-good and NOT worth re-checking: PLL locked (0x1e bit5), RX cal done on both lanes (reg338=0x9d), refclk ungated (HCI_CLKSTOP_CTRL=0), device reset deasserted (HCI_GPIO_OUT=1), pinmux applied (gpf2-0/gpf2-1 func2, confirmed in the compiled DTB), PMU isolation offset 0x724 is correct, mclk 166562500 Hz is correct (real cmu_top output is 533 MHz and 166562500 x 3.2 = 533000000 exactly), and UFS VCC verified driven (gpio-600 regulator-ufs-vcc out hi). A cold boot (full VCC power-cycle) did not help. Conclusion: UFS is parked, it would need hardware-level probing.

2. microSD, the pivot that worked

Added the SD controller from the downstream ground truth: mmc@132e0000, compatible samsung,exynos7-dw-mshc (mainline dw_mmc-exynos; downstream's samsung,exynos-dw-mshc has no mainline match), IRQ GIC_SPI 313, clocks CLK_GOUT_CMU_HSI1_BUS (biu) + CLK_GOUT_CMU_HSI1_MMC_CARD (ciu), pins sd2_clk/cmd/bus1/bus4 (already in exynos990-pinctrl.dtsi), cd-gpios = <&gpa1 5 GPIO_ACTIVE_LOW>, fifo-depth 0x40, samsung,dw-mshc-ciu-div = <3>. Driver configs were already =y.

Result: controller probed, "Got CD GPIO", card detected, but every command timed out (Busy; trying anyway -> Timeout sending command), data line stuck low = no I/O pull-up voltage. Proof it was power, not the controller: the DT had no vmmc/vqmmc-supply, so dw_mmc already proceeded assuming power and still failed.

3. The real blocker: the S2MPS19 PMIC, reached only via a coprocessor

The SD rails are PMIC LDOs with no GPIO enable, a regulator-fixed DT stub is a pure lie to the framework and cannot work (it only works for UFS VCC, which genuinely is GPIO-gated on gpg1-0). The CPU cannot reach the PMIC directly:

AP -> AP2APM mailbox @0x15900000 (SPI 41) -> APM Cortex-M running ACPM firmware
   -> SPEEDY bus -> S2MPS19

speedy@15940000 is a red herring (disabled; ACPM firmware owns that bus).

Mainline already had this entire stack for Google Tensor gs101, PMIC command encoding bit-identical, shmem structs byte-identical, same APM SRAM base 0x2039000, and drivers/mfd/sec-acpm.c even hardcodes acpm_chan_id=2, speedy_channel=0, exactly matching the S20. But it landed in 6.15-7.1, after our 6.12 fork.

Tried migrating to mainline 7.2 first (/root/linux, which already has upstreamed exynos990 + z3s): it booted but panicked, Undefined instruction in sched_init_debug, executing ASCII data (0x43474244). GCC 9.4 miscompiling 7.2 code. So: backport instead.

Backport into 6.12 needed only two fixes: copy include/linux/firmware/samsung/*.h and include/linux/mailbox/exynos-message.h, and strip the newer __counted_by_ptr() annotation.

exynos990 deltas written:

  • drivers/mailbox/exynos-mailbox.c: per-SoC match data. exynos990 doorbell genuinely differs, write BIT(chan_id) << 16 to INTGR0 @0x08, mask INTMR0 @0x10 (gs101: BIT(chan_id) to INTGR1 @0x40, INTMR0 @0x28).
  • drivers/firmware/samsung/exynos-acpm.c: samsung,exynos990-acpm-ipc with initdata_base = 0x7000 (gs101 uses 0xa000) + NULL-guard on acpm_clk_dev_name (no CMU_APM driver exists).
  • DT: apm_sram: sram@2039000 (0x38000), ap2apm_mailbox: mailbox@15900000 (SPI 41, pclk = fixed-clock stub since there's no CMU_APM), and firmware { acpm_ipc: power-management {...} }.

Safety-first smoke test (read-only), writing rails the ACPM firmware owns can hard-brick a boot, so reads were proven first:

ACPMTEST: OK chipid=0x00
ACPMTEST: LDO2(vqmmc)=0x00 LDO15(vmmc)=0x2e

LDO15=0x2e proved it was a real read: enable bits off, vsel 0x2e = 1.8V + 46x25mV = 2.95V, exactly the documented vmmc voltage. Both rails confirmed disabled.

Then wrote 0xC0 | 0x2e (= 0xEE, enable + 2.95V) to reg 0x54 (vmmc/LDO15) and reg 0x47 (vqmmc/LDO2), and dropped sd-uhs-* to avoid the 1.8V signalling switch:

mmc0: new high speed SDXC card at address aaaa
mmcblk0: mmc0:aaaa SN64G 59.5 GiB

This is S2MPS19-over-ACPM on Exynos 990, not previously supported in mainline Linux.

4. Debian on the card

Formatted the whole device ext4 (label z3sroot; TWRP's fdisk wouldn't take piped input, and a partition table isn't needed for a single-filesystem OS disk), extracted the 364 MB Debian 12 rootfs, and configured it offline: fstab, hostname, autologin root on both ttyGS0 and tty0, cleared root password.

Rewrote the initramfs init to switch_root into the card, keeping the USB gadget setup early and a fallback shell if the handoff fails.

Gotcha that cost one flash cycle: [ -e /newroot/sbin/init ] returns false on Debian, /sbin/init is a symlink to the absolute path /lib/systemd/systemd, which resolves against the initramfs (where it doesn't exist). Fixed by testing for /newroot/lib/systemd/systemd instead. (A chroot /newroot /bin/bash test confirmed the rootfs itself was fine before reflashing.)

Result: root@z3s:~#, systemd as PID 1, /dev/mmcblk0 mounted rw on /, 58 GB free, 8 cores, systemctl is-system-running = running.

SESSION 3 (cont.), USB networking + internet + SSH

Made the OS actually usable, networking over the USB cable.

Composite USB gadget (RNDIS + ACM)

Kernel already had all the ethernet-gadget functions (RNDIS/ECM/NCM/EEM all =y), so no kernel rebuild, only the initramfs gadget setup changed. New init builds a composite gadget:

  • rndis.usb0 (networking) + acm.usb0 (serial console) in one config
  • IAD device class (bDeviceClass=0xEF/0x02/0x01) so Windows binds the composite
  • MS OS descriptors (os_desc/use=1, b_vendor_code=0xcd, qw_sign=MSFT100, RNDIS compatible_id) → Windows auto-loads the RNDIS driver ("Remote NDIS Compatible Device")
  • On Windows this appears as a network adapter and a COM port (serial survived).

Addressing / Windows ICS

Phone side (systemd oneshot z3s-usbnet.service → /usr/local/sbin/z3s-usbnet.sh): usb0 = <lan-ip>/24, default via <lan-ip>, DNS <lan-ip> + 1.1.1.1/8.8.8.8. These match Windows Internet Connection Sharing, which forces the shared adapter to <lan-ip> and NATs to the internet. User enables ICS via the internet adapter's Properties → Sharing tab → share to the RNDIS adapter. Result: phone pings 1.1.1.1 (17ms) and resolves DNS. The phone is online through the PC over USB.

THE CLOCK GOTCHA (cost a debug cycle)

apt update failed:

E: Release file ... is not valid yet (invalid for another 74d ...)

The phone has no working RTC, so it boots with a wrong (past) date and apt rejects the repo signature. Fix: set the clock (date -u -s "<real UTC>", pulled from the PC), then apt works. This recurs every reboot until a persistent time source is set up (systemd-timesyncd isn't in the minimal rootfs; installing openssh-server pulled in dbus, so timesyncd is now installable, TODO).

SSH

apt install openssh-server (ran in the background on the phone, logged to /tmp/apt.log, to avoid jamming the interactive serial). Set root:z3s, PermitRootLogin yes, enabled+started ssh. Verified: ss -tlnp shows sshd on 0.0.0.0:22, Test-NetConnection... 22 = True.

Log in from the PC: ssh root@<lan-ip> (password z3s). Now a real terminal.

Serial console note

Composite gadget re-enumerated the ACM as a new COM port (was COM40 → now COM41). Long foreground commands (apt) jam the interactive serial, run them backgrounded to a logfile and poll, or just use SSH now. Ctrl-C (send chr(3)) unjams a stuck serial shell.

TODO next

  • persistent time (install systemd-timesyncd / enable NTP, or bring up the S2MPS19 RTC)
  • passwd a real root password
  • from here: it's a normal Debian box, build toward a UI, then touch (sec_ts) + keyboard.

2026-07-26 - GPU fault isolation and stable touch desktop

  • Rebuilt the kernel with G77 r0p1 matching; the issue mask changed but sustained faults remained.
  • Tested upstream Mesa 26.3-devel from an isolated /opt prefix without replacing system Mesa. Surfaceless EGL was clean, but full KWin still produced recurring faults and scheduler failures.
  • Proved plasmashell/taskbar was not the source: faults continued while plasmashell was stopped.
  • Made QPainter the greetd default through /usr/local/bin/plasma-wayland-stable; responsiveness improved and the accelerated-KWin fault burst stopped.
  • Added window rules for reachable title bars and Meta+F11 fullscreen toggling.
  • Replaced QML Plastik with native Breeze, set ButtonSize=4, BorderSize=VeryLarge, and halved animation duration for touch use and lower repaint latency.
  • Profiled abundant free RAM, nearly idle CPUs, and no I/O pressure. Remaining sluggishness is predominantly 1440x3200 software-composition latency.
  • Full details: SESSION5_FINDINGS.md.

Automatic UI boot - final resolution

After repeated cold-boot failures, the full greetd/Plasma session chain was audited. Plasma 5 delegated processes to user-systemd while greetd ended or replaced the VT-owning login. KWin and plasmashell therefore remained alive without a reliably displayable session; process-only watchdogs produced false positives. Secondary symptoms included duplicate logins, stale Xauthority, ksmserver crashes, KSplash timeouts, and DRM permission errors.

Replaced greetd with native /etc/systemd/system/z3s-plasma.service: a single systemd/PAM z3s login bound directly to tty1, starting /usr/local/bin/plasma-wayland-direct. z3s-plasma is enabled, greetd is masked, the old ensure-desktop watchdog is disabled, and the default target is graphical. A clean reboot reached the complete Wayland workspace in 16.5 seconds with no failed user units or DRM presentation errors; the phone screen was physically confirmed showing Plasma.

Persistent clock, FIXED (verified)

apt install systemd-timesyncd fake-hwclock, both enabled at boot. timesyncd VERIFIED syncing: "Contacted time server... 0.debian.pool.ntp.org / Initial clock synchronization to 2026-07-23 UTC" (NTP reaches the pool through the ICS NAT). Boot order: fake-hwclock restores last-saved time -> networking (z3s-usbnet) -> timesyncd syncs from NTP. Reboots no longer break apt. Note: dbus-daemon is not running by default (minimal rootfs) so timedatectl reports "Failed to connect to bus", cosmetic; timesyncd is a standalone NTP client and works regardless.

2026-07-26, BCM4375 onboard Wi-Fi investigation

The Galaxy S20 Ultra's onboard Wi-Fi is PCIe, not SDIO. The stock Samsung DTBO enables pcie_0 (pcie@133B0000) as an endpoint named WIFI, with:

  • compatible = "samsung,brcm-wlan"
  • WLAN enable on gpb0-4
  • WLAN host-wake on gpa3-7
  • one PCIe lane, Gen1, no MSI, no SYSMMU/cache coherency
  • Exynos9830/990 PCIe ELBI at 0x133b0000, PHY at 0x133e0000

The live kernel (6.12.0-rc5-next-20241104-g75138397b8f8-dirty, build #21) currently has no PCI devices and no WLAN interface. Its config proves why:

# CONFIG_PCI_EXYNOS is not set
CONFIG_BRCMFMAC=m
CONFIG_BRCMFMAC_SDIO=y
# CONFIG_BRCMFMAC_PCIE is not set

Merely installing firmware or enabling NetworkManager cannot fix this. The upstream drivers/pci/controller/dwc/pci-exynos.c in /home/<user>/z3s-build/linux990 supports Exynos5440, not the Exynos9830/990 PCIe PHY/root complex. Samsung's 4.19 source in 03_kernel_src/main contains the missing vendor implementation:

  • pcie-exynos-host-v0.c
  • pcie-exynos9830-v0-cal.c
  • related Exynos PCIe common/debug code

Those pieces must be carefully ported to the active 6.12 DesignWare PCI API and integrated with the active Exynos990 clock/pinctrl/PMU support. Do not replace the working 6.12 kernel with the old Android kernel.

The extracted stock firmware is available under 02_extracted/vendor_fs/firmware:

  • bcmdhd_sta.bin_b1
  • bcmdhd_clm.blob
  • multiple board-specific nvram.txt_*_b1 files

After PCIe controller 0 enumerates the BCM4375, enable CONFIG_BRCMFMAC_PCIE, determine the correct Murata/Semco NVRAM variant, and install firmware under brcmfmac-compatible names. Keep NetworkManager from managing usb0, because USB networking is the recovery/SSH path.

2026-07-27, BCM4375 Wi-Fi SOLVED (full WPA2 + internet)

Result: Wi-Fi fully works. Associates, completes the WPA2 4-way handshake, gets a DHCP lease (<lan-ip>), routes to the internet, and the user browsed the web on it. First full WPA2 connection by this SoC's BCM4375 on mainline.

The prior diagnosis was wrong, which is why nothing had stuck. By session 6/7 the PCIe controller + brcmfmac PCIe port already worked: the endpoint enumerated, firmware booted (BCM4375/5... 18.41.117), IOCTLs/association succeeded, and scans returned 30-38 BSSes. The remaining failure, WPA 4-way "timeout", AP deauth reason 15, rx_packets=0, was read as a broken RX / interrupt / async-event path. It wasn't. Interrupts (IRQ 52) and the firmware event path (scan results) worked fine the whole time.

Actual root cause: mainline brcmfmac never advertises host capabilities. The vendor Samsung driver (bcmdhd_101_12) writes a host_cap word to the PCIe shared area in dhd_set_host_cap() for shared-protocol rev ≥ 6. This firmware is PCIE shared version 7. Without that advertisement, rev-7 firmware never DMAs received data frames into the host RX buffers / never completes them on the D2H RX ring, so EAPOL-Key M1 never reaches wpa_supplicant and the 4-way can't even begin. dcmd/IOCTL responses still worked because those are polled, and scan results still worked because those come via the firmware event path, masking the data-RX gap.

Fix (in brcmf_pcie_init_share_ram_info(), after reading ring_info_addr):

if (shared->version >= 6) {
        u32 host_cap = shared->version |   /* active_ipc_version (=7) */
                       0x00000400 |        /* HOSTCAP_H2D_ENABLE_HOSTRDY */
                       0x00001000 |        /* HOSTCAP_DS_NO_OOB_DW       */
                       0x00020000 |        /* HOSTCAP_EXTENDED_TRAP_DATA */
                       0x00800000;         /* HOSTCAP_UR_FW_NO_TRAP      */
        brcmf_pcie_write_tcm32(devinfo, sharedram_addr + 84, host_cap);
}
  • Value 0x00821407 for version 7. Bits mirror exactly what mainline brcmfmac implements (HOSTRDY, disable-DS, ext-trap, no-UR-trap); HWA/IDMA/IFRM/DAR/HSCB/ H2D_PHASE deliberately omitted.
  • Offset sharedram_addr + 84 = the host_cap field. Verified byte-for-byte against the vendor pciedev_shared_t (bcmpcie.h): flags2@80, host_cap@84. Cross-checks every mainline offset #define (max_rxbufs@34, rx_dataoffset@36, h2d_mb@40, d2h_mb@44, ring_info@48).
  • Written early in probe, before brcmf_pcie_hostready() signals the FW, matching the vendor ordering.

How it was proven without the Wi-Fi password. In WPA2-PSK the AP sends M1 to any station that completes 802.11 association, regardless of key. So a deliberately-wrong PSK exercises the exact RX-data path with zero real credential:

module RX pkts EAPOL M1 outcome
known-good fd1df1e1 (no host_cap) 0 never arrives wpa self-deauths (reason 3), no handshake
patched 854e3bd1 (host_cap) 6 WPA: RX message 1 of 4-Way Handshake sends M2, AP rejects wrong key (reason 15), normal

Then the user connected with the real key → wpa_state=COMPLETED, CCMP, <lan-ip>, pings + browser working.

Build/deploy (all on the Cortex Raspberry Pi, native arm64 builder now, not WSL):

  • Tree: ~/z3s-build/linux990 (ported, matches phone kernel 6.12.0-rc5-next-20241104-g75138397b8f8-dirty). Source backup pcie.c.prehostcap.bak.
  • nice -n 19 make -j3 M=drivers/net/wireless/broadcom/brcm80211/brcmfmac modules
  • Module sha256 854e3bd15005fed845b4f6850f3e773d447588fc4788dbb3e135d0cf194613a6, vermagic matched exactly. Installed to /lib/modules/<krel>/.../brcmfmac.ko, depmod -a, survives reboot. Known-good kept on phone as brcmfmac.ko.knowngood-fd1df1e1 for instant rollback.
  • No new boot image flashed, the fix is a rootfs module swap only.

Access note: the phone is now reached through Cortex as an IPv6 jump host (ssh -J-style ProxyCommand), because its USB link now lands on Cortex. Full command in WIFI_BRINGUP.md.

Minor, non-blocking: brcmf_p2p_create_p2pdev: timeout still logs on probe (P2P-dev iface only, cosmetic); samsung,z3s.bin/.txcap_blob -2 are benign board-variant probes before the generic firmware loads.

See WIFI_BRINGUP.md for the complete writeup, offsets, artifact hashes, verification, and rollback.

2026-07-27 (later) CPU DVFS and cpufreq: all 8 cores scaling

The phone felt sluggish because the port had no CPU DVFS: the clusters ran at a fixed low boot clock, the scheduler treated every core as equal capacity, and the two Exynos M5 prime cores were offlined by a mitigation to avoid stalls. Two things fixed it.

First, a scheduler capacity model: added capacity-dmips-mhz to the CPU nodes so the scheduler steers latency-sensitive work to the fast cores. That alone gave a 3x single-thread improvement.

Then real CPU DVFS. The ACPM DVFS channel (channel 5, cluster ids 2/3/4) was blocked through several layers: mainline refused the interrupt-mode channel, used the gs101 timestamp in cmd[3], and, the key one, deadlocked in acpm_wait_for_queue_slots for the single-slot (qlen==1) DVFS channel so the command was never sent. Fixing those made the frequency track set_rate (A55 442 MHz to 2.0 GHz, 4.4x). A single-slot INTSR1 completion path then made set_rate return cleanly. On top of that sits a small exynos990-cpufreq driver (per-cluster policies, schedutil). Finally the M5 prime cores were re-enabled by retiring the offline mitigation; with DVFS driving their frequency and capacity modeled, the old stalls do not recur.

Final: all 8 cores online, three cpufreq policies under schedutil, capacities 242/975/1024, M5 at 2.73 GHz. Full detail, the debugging narrative, and the patch stack are in CPU_PERF_DVFS.md.

2026-07-28, Audio (ABOX Calliope DSP) and power-off

Audio: the DSP runs its real firmware (secure SysMMU bypassed)

The hardest subsystem. Sound needs the ABOX Calliope DSP (dual Cortex-A32) booted and running its firmware. The wall was the secure ABOX SysMMU: the DSP fetches its DRAM firmware by IOVA 0x80000000, which the SysMMU must translate to the physical firmware region 0xf8000000. The stock driver programs that; we never could, because the SysMMU is secure write-locked from a non-secure kernel (reads work, CTRL=0x4 bypass; any write to 0x18e50000 wedges the SoC; the DTZPC unlock SMC is accepted but does not unlock). So the DSP fetched garbage from 0x80000000 and executed chaos the instant it left SRAM, the real cause of every earlier "the DSP won't run" symptom.

The fix was to not program the SysMMU but bypass it. It sits in pass-through, so if the DSP's own MMU maps VA 0x80000000 to 0xf8000000 (instead of identity), the bypass SysMMU passes 0xf8000000 straight through to the real firmware, no secure write. We inject a routine into the DSP's page-table builder (at DSP SRAM 0x24ac, run while its MMU is still off) that adds 0x78000000 to every 1 MB firmware section (and walks L2 tables). The DSP then fixes its own page tables and runs real firmware. Confirmed: caught executing inside the injected code, and its behaviour flipped from inert to a real boot.

Current wall: the DSP boots EL1/SYS (CPSR 0x600001df), runs its CPU-setup, then faults to exception vector 0x14 and livelocks before its C-runtime/main. 0x14 is the AArch32 Hyp-trap slot. Bisecting the CPU-setup (a spin-probe that halts the DSP at a chosen instruction and reads its PC) pinned the fault to a SCTLR write (the MMU-enable at SRAM 0x34bc, or cache-enable 0x34e0); ruled out the CPUACTLR access and the secure-GIC read. An EL1 SCTLR write trapping to 0x14 is the HCR_EL2.TVM signature: the DSP's MMU-enable traps to EL2 because the secure/EL2 state the stock secure boot sets up is missing (we release the CA32 cores from a non-secure kernel). The fix needs secure-world (EL3/EL2) access we do not have from a non-secure kernel (HCR is a CP15 sysreg of the DSP core, not an SFR, so the EL3 SMC_CMD_REG SFR path cannot reach it). Open. Full writeup, the exact fault, the open problem, and the bring-up module in AUDIO_DSP.md.

Power-off (partial): PS_HOLD works, but the APM re-powers the AP

Shutdown hung on a black screen forever. Cause: pm_power_off is PSCI SYSTEM_OFF, which hangs S-Boot (reboot uses a different PSCI path and works). Real power-off on exynos9830 is driving PS_HOLD low: PMU PS_HOLD_CONTROL = 0x030C, clear bit 8 (not the old S5P 0x330C which reproduces the hang). Added a syscon-poweroff DT node (CONFIG_POWER_RESET_SYSCON_POWEROFF=y) at SYS_OFF_PRIO_HIGH so it beats the broken PSCI handler, plus an ACPM STOP (enter-WFI) step before it. Result: clean power-off, power really is cut, but the phone auto-reboots ~20s later even unplugged, the APM firmware re-powers the AP and the ACPM STOP does not suppress it. Partial. Detail in POWEROFF.md.

Audio: past the EL2 wall, the front-end pumps, the amps power up (2026-07-29)

The EL2 wall above is crossed: the DSP now boots all the way and runs the full firmware (the MMU-enable no longer traps, via a secure ABOX-GIC init over an EL3 SMC). With the graph driven correctly (tonegen=0, front-end fed by a real virtual-PCM stream) the whole front-end pumps real audio: VPCMOUT_DAI -> PCMOUT -> BD_SRCCOM0 -> BD_MIXER all carry real byte counts. Two more cracks: (1) the RDMA3 update-trigger: __rdma_primcom_source_update (firmware VA 0x8000bd40) will not consume the mixer output unless the word at DSP 0x81004530 is 0x1111; forcing it makes RDMA3 process the mixed audio (found by capstone disassembly). (2) the CS35L41 speaker amps power up and are clocked, once the amp init writes GLOBAL_EN (reg 0x2014) last, after PLL/clock/format/enables; that also proves the ABOX UAIF2 I2S bit clock is live. New wall: SIFS0/RDMA3/UAIF2 are DSP-managed back-end DAIs; SIFS0 still serializes zeros because the DSP will not run the back-end hw_params -> prepare -> trigger lifecycle for register/IPC pokes. The bring-up-module approach is exhausted; the real path is the mainline exynos ABOX ASoC driver. Detail in AUDIO_DSP.md.

Display: FHD 1080x2400 @ 120Hz live reprogram (2026-07-29, in progress)

The panel is up at the bootloader's WQHD 1440x3200 but not at 120Hz. The s6e3hab "hubble3" panel physically cannot do 120Hz at WQHD (its VRR table lists 120HS only at 1080x2400 and 720x1600), so 120Hz means dropping to FHD, a full mode-set. There is no exynos display driver in mainline (the framebuffer is a bootloader-lit DECON presented by simpledrm), so the mode-set is done by a module poking the live DECON_F (0x19050000), the two DSC encoders (0x19054000/0x19055000), and the DSIM (0x190C0000), plus sending the panel's FHD@120 DCS sequence over the DSIM command FIFO. All of it was reverse-engineered from the downstream dpu30/cal_9830 and panel drivers and is captured byte-for-byte in the doc: panel mode-switch DCS, the 128-byte FHD PPS, DECON size/OUTFIFO/blender registers, DSC CONTROL3 and PPS, the active window (WIN5), the shadow-update trigger, and the D-PHY PLL read. The register-poke approach is proven end to end (the size/DSC registers take and the system stays alive); v2 with the DSC encoder reprogram renders a partial FHD frame then fades. Next is the panel 120HS fine timing (vfp_hs/ltps), then moving the whole switch into the bootloader so Linux boots clean at FHD@120. Detail in DISPLAY.md.

Power-off: root cause found, it is the PMIC WTSR watchdog (2026-07-29)

The ~20s re-power above is diagnosed and it is not the APM and not the power button. Controlled proof: systemctl poweroff over SSH with nothing touched still re-powers at ~20s. The cause is the main PMIC (S2MPS19) WTSR (Watchdog Timer Software Reset, RTC register 0x01 bit 6) and SMPL (bit 7), which S-Boot leaves enabled. A normal Samsung phone disables them in the PMIC RTC driver's .shutdown(); mainline rtc-s5m.c has no such disable, and on this port no PMIC/RTC driver is loaded at all (the S2MPS19 is fully ACPM-managed, /sys/class/rtc is empty), so nothing ever disables WTSR and its watchdog times out ~20s after PS_HOLD drops. The fix is one register: clear bits 6+7 of RTC reg 0x01 over ACPM (PMIC channel 2, speedy 0, RTC sub-address 0x02) in acpm_power_off_notify() before dropping PS_HOLD. Detail in POWEROFF.md.

Power-off SOLVED: WTSR/SMPL cleared over ACPM, phone stays off (2026-07-30)

The fix from the entry above is implemented, flashed, and verified. Shutdown now works end to end from any path (systemctl poweroff, the Plasma GUI shutdown, poweroff), because they all funnel into the kernel sys-off chain where the handler lives.

Validation before touching the poweroff path: a read-only module (acpm-rtc-probe.c, one-shot, self-unloading) dumped the S2MPS19 RTC block over ACPM (PMIC channel 2, type = sub-block i2c address 0x02, speedy channel 0). RTC reg 0x01 read back 0xe3, WTSR_EN (bit 6) and SMPL_EN (bit 7) both armed by S-Boot, exactly as diagnosed. The same register read via the PMIC type (0x01) returned 0x10, proving the ACPM type field selects the sub-block (PMIC 0x01, RTC 0x02, DEBUG 0x0F). SUB_SMPL (CAPSEL reg 0x03 bit 3) already read 0.

Implementation: s2mps19_disable_wtsr_smpl() in exynos-acpm.c, called at the top of acpm_power_off_notify() BEFORE acpm_enter_wfi() sends the APM STOP, because the APM must still be alive to service the PMIC channel. It reads RTC reg 0x01, and if bits 6/7 are set writes the register back with them cleared, then reads back and logs the result with dev_emerg. Full patched driver: kernel/drivers/acpm-pmic/exynos-acpm.c.

Flashed as kernel #22 via the uniLoader pipeline and tested over SSH on battery: systemctl poweroff, screen black, and the phone stayed off (previously it re-powered at ~20 s every time). Boot back up with the power button is clean. One caveat: plugging in USB power while off can boot the phone via charger detection, which is normal PMIC behaviour, not the WTSR bug.

Also note the shutdown-time dev_emerg lines are usually lost from the journal because PS_HOLD cuts power before journald flushes; the staying-off behaviour is the proof.

Power key UX working; real suspend blocked by BCM4375 D3 (2026-07-30)

Phone-style power-button handling is now installed and proven:

  • short press locks Plasma and turns the OLED off
  • next short press turns the OLED on at the password screen
  • hold for four seconds opens Plasma's shutdown/restart prompt
  • releasing a long hold does not queue a lock action

Implementation: root systemd service z3s-powerkeyd reads KEY_POWER directly from the gpio-keys event device (gpa2-4). logind ignores the key. PowerDevil's profile button actions (previously action 16, logout prompt) and its global PowerDown/PowerOff shortcuts are disabled so there is only one owner. Plasma calls use the real Wayland session bus and environment. kscreen-doctor --dpms off must be fire-and-forget: it stays blocked while the OLED is off; waiting 15 seconds queued the wake press and produced a delayed relock. The final nonblocking implementation has no delayed lock after password unlock. Files and installer are in rootfs/; full detail in POWERKEY.md.

Real kernel suspend was diagnosed rather than exposed. Kernel #23 enabled PM_DEBUG and PM_SLEEP_DEBUG. pm_test freezer passed. With console suspension disabled and VT2 active, the complete device callback trace showed UFS, USB/DWC3, GPU, audio, I2C, MMC, and regulators suspend cleanly. BCM4375 is the first failure:

brcmf_pcie_pm_enter_D3: Timeout on response for entering D3 substate -> pci suspend returns -EIO after about two seconds -> PM core aborts. The rev-7 firmware never ACKs HOST_D3_INFORM. A non-fatal-timeout module experiment was rolled back; brcmfmac live reprobe hit an Oops in dev_addr_mod/brcmf_net_attach, and replacing the module blocked a boot until TWRP restored the original. Suspend stays masked; the working feature is safe soft sleep (lock + OLED DPMS off, CPUs naturally idle).

Two recovery lessons from the same session: ramoops at the S-LK sec_debug address 0x91200000 kills boot during early UFS setup, and a stale uniLoader blob/dtb caused the bad DT to be repacked twice. Always rebuild/copy/decompile-verify the embedded DTB before flashing. Both bad flashes were recovered from the deploy script's on-device TWRP backup; no partition damage.

Pocket development workstation foundation (2026-07-30)

Created DAILY_DRIVER.md and completed the first userspace-only phase without a kernel flash or recovery-path change. The Debian arm64 installation now has GCC/Clang, CMake/Ninja/Meson, GDB/strace/shellcheck, Python venv, Git/Git LFS/ GitHub CLI, tmux, Kitty, Neovim, ripgrep/fd/fzf/bat and system-monitoring tools. Native GCC/Clang programs and Python venv passed smoke tests.

Installed pinned NVM v0.40.3 plus Node v22.23.2 arm64. Pi Coding Agent 0.83.0 was installed with npm lifecycle scripts disabled, project trust set to ask, install telemetry disabled and global phone-specific safety instructions. Provider auth is intentionally not copied or published. Official VS Code 1.131.0 arm64 was package/architecture verified before install, then given a user Wayland launcher and narrow-screen editor defaults.

Added the secret-free rootfs/plasma/z3s-health-report utility. Weekly UFS fstrim was already active. smartd was disabled because the UFS SCSI LUNs do not expose SMART-enabled devices and it always exited with a false failed-unit warning. The only current thermal zone is the battery gauge; SoC thermal-zone support remains a kernel daily-driver task.

Development account and remote-tool integration

Pi provider/model configuration was transferred over SSH with credential values never printed; all auth/model/settings files are user-owned mode 0600. An authenticated prompt passed on the phone. Git identity and GitHub CLI auth were imported without storing identity or tokens in this repository.

A phone-only ED25519 key now provides tested SSH access to the external ARM64 build host. The desktop has a tested ssh linuxphone alias that logs in as the normal z3s user; root recovery access remains unchanged. The public project was cloned into ~/Projects with owner permission verified.

VS Code now has C/C++, Python/Pylance/debugpy, CMake Tools, Remote SSH, ShellCheck, YAML and TOML support. Underlying phone-to-build-host SSH is proven; the first user-visible VS Code Remote SSH session remains a UI test.