Skip to content

Commit d2adc62

Browse files
alex-poorclaude
andcommitted
Tell iroh when Android changes the network under it
A phone that left the house on 2026-09-15 stopped sharing for seventy-one minutes and started again the moment it came home. AAPS never faltered: it sealed 84 epochs into that hole, `holds` climbed 1470 to 1560, and every pass said `missing 0, lost 0, failures 0`. The data was made and it was kept. It had nowhere to go. `relay=disconnected` for 88 consecutive passes, 15:05:25 to 16:16:23, bracketing both readers' observations exactly — and that is iroh's own `home_relay_status()`, not our inference. Not doze: AAPS is whitelisted, runs at targetSdk 32, and a phone sealing an epoch a minute is not one Android has stopped. Not the carrier, and not the foreground service. Three overnight failures have now been blamed on doze and caused by something else. netwatch ships a deliberately empty route monitor for Android, whose whole body is the comment "Very sad monitor. Android doesn't allow us to do this", and its wall-time poll is stretched to an hour on mobile and fires on a clock jump rather than a network one. So iroh cannot see an Android network change at all. Its own documentation names the platform and the remedy: Android exposes this to Java, so Java must pass it down. `Endpoint::network_change()` is public. We had never called it. Without it nothing does the rebind, the `dns_resolver.reset()`, the fresh net report, or the nudge that migrates QUIC. The relay is a hostname, so every reconnect needs DNS, and a resolver still aimed at the network the phone walked away from keeps failing until it walks back. A brief wifi toggle does not reproduce any of this, which is why it hid for days: live connections keep flowing over the wildcard socket via cellular and nothing has to re-resolve. It needs a change that outlives its connections. Measured on the loop phone with the fix, wifi off, nine minutes on mobile data: relay connected throughout, pool steady at 3 peers and 4 buckets where the failure collapsed to 1, and operations landing on the laptop peer the whole time — 2084 to 2089. `ss` puts the relay socket's local end on the CGNAT address, so it genuinely reconnected over mobile rather than surviving. Notification is trailing-edge: Android delivers these in a steady trickle, about one every three seconds, and a thread per callback all day is battery a user gets to refuse. Sleeping before notifying means a burst collapses to one call that sees the settled network — dropping the last callback of a burst is the one mistake that would bring the bug back, and this cannot make it. Logging is capped at once a minute with a count, because the outage was only diagnosable from AndroidAPS.log and filling it is how that stops being true. `stop()` is called before `swarmLeave`, which frees the handle: unlike every other caller this one is driven by an OS callback that can fire during teardown. This also corrects the record. "Off-LAN is proven" has stood since 2026-09-11 and was measured with the phone on mobile data while still at home. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 606642c commit d2adc62

11 files changed

Lines changed: 719 additions & 14 deletions

File tree

‎crates/diaswarm-android/src/lib.rs‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -601,6 +601,31 @@ pub extern "system" fn Java_nz_diaswarm_jni_SwarmNative_swarmOffer<'a>(
601601
}
602602
}
603603

604+
/// Android says the network changed; tell iroh, which cannot see it itself.
605+
///
606+
/// **THE MISSING HALF OF A 71-MINUTE OUTAGE.** `netwatch` ships a deliberately
607+
/// empty route monitor on Android — "Very sad monitor. Android doesn't allow us
608+
/// to do this" — so nothing native ever learns that wifi became mobile data.
609+
/// iroh's own documentation says Java has to tell it. This is where Java tells
610+
/// it. See `Swarm::network_changed`.
611+
///
612+
/// Returns 1 if the notification was delivered, 0 if there is no swarm to
613+
/// notify. Cheap enough to call on every `ConnectivityManager` callback;
614+
/// upstream says calling it needlessly does no harm.
615+
#[no_mangle]
616+
pub extern "system" fn Java_nz_diaswarm_jni_SwarmNative_swarmNetworkChanged(
617+
_env: JNIEnv,
618+
_class: JClass,
619+
handle: jlong,
620+
) -> jlong {
621+
if handle == 0 {
622+
return 0;
623+
}
624+
let pooled = unsafe { &*(handle as *const Pooled) };
625+
pooled.runtime.block_on(pooled.swarm.network_changed());
626+
1
627+
}
628+
604629
/// One pass: say what we hold, hear what we should, take on a few of them.
605630
///
606631
/// Returns `pool<TAB>buckets<TAB>held<TAB>wanted<TAB>adopted`, or empty.

‎crates/diaswarm-net/src/swarm.rs‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -365,6 +365,42 @@ impl Swarm {
365365
if connected > 0 { format!("connected({connected})") } else { "disconnected".into() }
366366
}
367367

368+
/// Tell iroh the network underneath us may have changed.
369+
///
370+
/// **THIS IS THE 71-MINUTE OUTAGE OF 2026-09-15, AND IT IS OURS.** The loop
371+
/// phone left the house at 15:05, moved from wifi to mobile data, and its
372+
/// relay connection went and stayed gone until 16:16, when wifi came back.
373+
/// AAPS never faltered: it sealed 84 epochs into that hole, `holds` climbed
374+
/// 1470 -> 1560, `missing 0, lost 0, failures 0`. The data was made and
375+
/// kept. It simply had nowhere to go.
376+
///
377+
/// The reason is upstream, documented, and deliberate. `netwatch`'s Android
378+
/// route monitor is a stub whose comment reads "Very sad monitor. Android
379+
/// doesn't allow us to do this", and its wall-time poll is set to an hour on
380+
/// mobile to save battery — and fires only on a *clock* jump, never a
381+
/// network one. So on Android iroh cannot see a network change at all. Its
382+
/// own docs say what to do instead:
383+
///
384+
/// > some systems like android do not expose this functionality to native
385+
/// > code. Android does however provide this functionality to Java code.
386+
///
387+
/// So Java has to say so, and this is the way in. Not calling it means a
388+
/// phone that changes network keeps sockets bound to the interface it just
389+
/// left: not a carrier blocking us, not doze, not the foreground service —
390+
/// a notification we never sent.
391+
///
392+
/// Cheap and idempotent by upstream's own account — "even when the network
393+
/// did not change [...] there is no harm in calling this function" — so the
394+
/// callers err towards calling it.
395+
pub async fn network_changed(&self) {
396+
match self.iroh_endpoint().await {
397+
Ok(ep) => ep.network_change().await,
398+
// Nothing to notify yet. The endpoint takes its first look at the
399+
// network when it starts, so a change before that is already seen.
400+
Err(e) => eprintln!("diaswarm: no endpoint to notify of a network change ({e})"),
401+
}
402+
}
403+
368404
pub async fn pool_members(&self) -> Result<Vec<String>> {
369405
let me = self.node_id().await?;
370406
let mut ids: Vec<String> = self

‎docs/decisions.md‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -797,6 +797,101 @@ record. The thing the TODOs are about is the thing this decision removes.
797797
per record somewhere, which would mean the separation is less clean than both
798798
the API and those TODOs suggest.
799799

800+
### D32 · Android must tell iroh the network changed, because iroh cannot see it
801+
802+
**Settled and implemented 2026-09-15.** The measured cause of a 71-minute outage,
803+
and it is ours, not the carrier's.
804+
805+
#### 1. What happened
806+
807+
The loop phone left the house at 15:05 and came back at 16:16. From
808+
`AndroidAPS.log`, which is the only reason any of this is knowable — logcat's
809+
ring buffer had long since rolled past it:
810+
811+
```
812+
15:04:24 pool 3 peers, 4 buckets, holding 3, relay=connected(1)
813+
15:05:25 pool 2 peers, 2 buckets, holding 3, relay=disconnected
814+
... 88 consecutive passes, 71 minutes
815+
16:16:23 pool 2 peers, 2 buckets, holding 3, relay=connected(1)
816+
```
817+
818+
The laptop peer's `holding` froze at 2025 for 56 passes and then jumped `+44` in
819+
a single pass on recovery. Ayni's reading went to `3471s old` and was back to
820+
`52s` two passes later.
821+
822+
**AAPS never faltered.** Into those 71 minutes it sealed **84 epochs**, `holds`
823+
climbed 1470 → 1560, and every pass said `missing 0, lost 0, failures 0`. The
824+
data was made and it was kept. It had nowhere to go.
825+
826+
#### 2. It was not doze, not the carrier, and not the foreground service
827+
828+
All three have been blamed for overnight failures before, and this was none of
829+
them. AAPS is on the battery-optimisation whitelist
830+
(`user,info.nightscout.androidaps,10310`), it runs at `targetSdk 32` and is
831+
exempt from the whole foreground-service timeout regime, and it went on doing
832+
its work throughout — a sealing app is not a sleeping one.
833+
834+
`relay=disconnected` is not our inference either. It is iroh's own
835+
`home_relay_status()`, read straight out of the endpoint.
836+
837+
#### 3. The cause: a notification we never sent
838+
839+
`netwatch`'s Android route monitor is a stub. Its entire body is a comment:
840+
841+
```rust
842+
// Very sad monitor. Android doesn't allow us to do this
843+
```
844+
845+
and its wall-time poll is stretched to an hour on mobile to save battery — and
846+
fires on a *clock* jump, never a network one. **So on Android, nothing native
847+
ever learns that wifi became mobile data.** iroh's own documentation says so, and
848+
names the platform and the remedy:
849+
850+
> On many systems iroh is able to detect network changes by itself, however some
851+
> systems like android do not expose this functionality to native code. Android
852+
> does however provide this functionality to Java code. This function allows for
853+
> notifying iroh of any potential network changes like this.
854+
855+
`Endpoint::network_change()` is public, and we had never called it. What we lost
856+
by not calling it is everything in `handle_network_change`: the UDP **rebind**,
857+
`dns_resolver.reset()`, a fresh net report, and the QUIC stack being told to
858+
migrate. Our relay is `https://aps1-1.relay.n0.iroh.link.` — a **hostname** — so
859+
every reconnect needs DNS, and a resolver still pointing at the network the phone
860+
walked away from fails every time until it walks back.
861+
862+
#### 4. Why a brief wifi toggle looks fine, which is how this hid
863+
864+
Switching wifi off at home and on again keeps working, and that observation is
865+
what makes the mechanism specific rather than vague. Live QUIC connections keep
866+
flowing over the wildcard socket via cellular, and nothing needs to re-resolve
867+
anything. **The failure needs a network change that outlives its connections** —
868+
leaving, not toggling. That is why every short test passed and why the finding
869+
only appeared on a real errand.
870+
871+
#### 5. The fix
872+
873+
`Swarm::network_changed` → `swarmNetworkChanged` (JNI) → `NetworkWatch`, a
874+
`ConnectivityManager.registerDefaultNetworkCallback` in both apps. It notifies on
875+
every callback rather than guessing which ones matter, because upstream is
876+
explicit that a needless call does no harm and the failure it guards against is
877+
an hour of silence. Bursts coalesce to one in-flight notification. The logger is
878+
injected so the lines land in `AndroidAPS.log` rather than the buffer that lost
879+
the last one.
880+
881+
*Reopens if:* a phone that genuinely leaves the house still loses the relay for
882+
longer than a reconnect takes. That would mean the notification is delivered and
883+
insufficient, and the next suspect is the endpoint rebuild `relay_state`'s
884+
doc comment has always declined to do.
885+
886+
#### 6. What this corrects
887+
888+
**"Off-LAN is proven" was recorded on weaker evidence than it sounded.** The
889+
2026-09-11 result had the phone on mobile data *while still at home*. The first
890+
time it genuinely left, sharing stopped. Off-LAN via relay is real and still
891+
holds — the relay reconnected and everything resumed — but it is now known to
892+
depend on a notification that was not being sent, so it was never as robust as
893+
the record claimed.
894+
800895
### D31 · A rendezvous that depends on a number nobody agrees on
801896

802897
**Open, recorded 2026-09-14, not yet fixed.** Found by asking what happens with

‎docs/handover.md‎

Lines changed: 38 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,52 @@
1-
# Handover — 2026-09-15 07:30
1+
# Handover — 2026-09-15 16:45
22

3-
Everything is committed and pushed. The only uncommitted files are
4-
`README.md` and `fdroid/nz.diaswarm.ayni.yml`, which carry **the user's own
5-
edits — do not commit them**.
3+
`README.md` and `fdroid/nz.diaswarm.ayni.yml` may carry **the user's own edits —
4+
check `git status` before committing anything.**
65

76
---
87

9-
## 1. What is running right now, and when it answers
8+
## 1. The headline: the off-LAN outage is explained and fixed
109

11-
**A soak on phone B, started 07:13.** It is the only thing with a clock on it.
10+
**A 71-minute outage on 2026-09-15 had one cause, and it was ours.** See D32 in
11+
[decisions.md](decisions.md) and the end of [migration.md](migration.md).
1212

1313
```
14-
scratchpad/fgs-soak.log one line every ten minutes
14+
15:04:24 relay=connected(1)
15+
15:05:25 relay=disconnected <- phone left the house, wifi -> mobile
16+
... 88 consecutive passes, 71 minutes
17+
16:16:23 relay=connected(1) <- came home, wifi back
1518
```
1619

17-
**The question:** does Ayni's foreground service survive past **372 minutes**
18-
(~13:25), which is where the previous build died. Anything before that proves
19-
nothing.
20+
**AAPS never faltered**: 84 epochs sealed into the hole, `holds` 1470 → 1560,
21+
`missing 0, lost 0, failures 0`. Not doze (AAPS is whitelisted and at
22+
`targetSdk 32`), not the carrier, not the foreground service.
2023

21-
Baseline at start: `pid 19707 · types=0x40000000 · effective=NONE`.
24+
**The cause:** on Android, `netwatch`'s route monitor is an empty stub — *"Very
25+
sad monitor. Android doesn't allow us to do this"* — and its wall-time poll is an
26+
hour on mobile and fires only on a *clock* jump. **iroh cannot see an Android
27+
network change.** Its own docs say Java must call `Endpoint::network_change()`.
28+
We never did.
2229

23-
Phone B is screen-off and on USB. **Charging does not affect the thing being
24-
measured** — the foreground-service time limit is not a doze mechanism — and it
25-
isolates the variable. Unplug only if a combined doze test is wanted instead.
30+
**The fix:** `NetworkWatch` (in `nz/diaswarm/jni/`, duplicated in both apps) →
31+
`SwarmNative.swarmNetworkChanged` → `Swarm::network_changed`.
32+
33+
⚠️ **A BRIEF WIFI TOGGLE DOES NOT REPRODUCE THIS**, which is why it hid for
34+
days. Live QUIC connections keep flowing over the wildcard socket via cellular
35+
and nothing has to re-resolve. It needs a network change that **outlives its
36+
connections** — leaving, not toggling.
37+
38+
⚠️ **"Off-LAN is proven" was over-claimed** from 2026-09-11 onward: every such
39+
test had the phone on mobile data *while still at home*.
40+
41+
---
42+
43+
## 1a. The foreground-service soak: PASSED
44+
45+
`specialUse` works. First process ran **8h03m** (07:19 → 15:22) against
46+
`dataSync`'s death at 6h12m, `effective=NONE` and `doze=IDLE` across all 60
47+
samples, two-minute cadence held. It then restarted and **came back**, which
48+
`dataSync` could not ("Time limit already exhausted"). The 15:22 restart is a
49+
separate question — most likely the known Ayni memory leak.
2650

2751
---
2852

‎docs/migration.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3228,3 +3228,73 @@ the actual overnight case: a looping phone in somebody's pocket or beside a bed,
32283228
on mobile data, sharing with a parent who is asleep. An hour-long silent gap in
32293229
that scenario is the thing the follower's age display exists to make survivable —
32303230
but survivable is not the same as working.
3231+
3232+
---
3233+
3234+
## The cause, established — and it was not doze
3235+
3236+
**2026-09-15, same day, from `AndroidAPS.log`.** The durable log covers the
3237+
window; logcat had rolled past it hours before, which is the whole reason AAPS
3238+
keeping its own log on disk mattered here.
3239+
3240+
The outage has a single, exact signature:
3241+
3242+
```
3243+
15:04:24 pool 3 peers, 4 buckets, holding 3, relay=connected(1)
3244+
15:05:25 pool 2 peers, 2 buckets, holding 3, relay=disconnected
3245+
... 88 consecutive passes
3246+
16:16:23 pool 2 peers, 2 buckets, holding 3, relay=connected(1)
3247+
```
3248+
3249+
Seventy-one minutes, bracketing both readers' observations exactly.
3250+
3251+
**The publisher never faltered.** Into that hole AAPS sealed **84 epochs**,
3252+
`holds` climbed 1470 → 1560, and every single pass reported `missing 0, lost 0,
3253+
failures 0`. The data was made and it was kept. It had nowhere to go — which is
3254+
precisely why nothing was lost and the backlog arrived complete.
3255+
3256+
⚠️ **Doze is ruled out, not merely doubted.** AAPS is on the battery
3257+
optimisation whitelist (`user,info.nightscout.androidaps,10310`), it runs at
3258+
`targetSdk 32` and is exempt from the foreground-service regime entirely, and it
3259+
was demonstrably working throughout — a phone sealing an epoch a minute is not a
3260+
phone Android has stopped. The previous entry's suspicion was reasonable and
3261+
wrong. That makes **three** overnight failures now blamed on doze and caused by
3262+
something else.
3263+
3264+
`relay=disconnected` is not our inference either: it is iroh's own
3265+
`home_relay_status()`.
3266+
3267+
### Why it happened
3268+
3269+
`netwatch`'s Android route monitor is a deliberate stub —
3270+
3271+
```rust
3272+
// Very sad monitor. Android doesn't allow us to do this
3273+
```
3274+
3275+
— and its wall-time poll is stretched to an hour on mobile for battery, and
3276+
fires on a *clock* jump rather than a network one. **So on Android iroh cannot
3277+
see a network change at all.** Its own docs name the platform and the remedy:
3278+
`Endpoint::network_change()` is public and exists for exactly this. We had never
3279+
called it.
3280+
3281+
What that costs is everything in `handle_network_change`: the UDP rebind,
3282+
`dns_resolver.reset()`, a fresh net report, and the QUIC stack being told to
3283+
migrate. The relay is a **hostname**, so reconnecting needs DNS — and a resolver
3284+
still aimed at the network the phone walked away from keeps failing until it
3285+
walks back. Which is what 16:16 is.
3286+
3287+
### Why every previous test passed
3288+
3289+
Switching wifi off at home and on again keeps working, and that observation is
3290+
what makes this specific rather than hand-waving. Live QUIC connections keep
3291+
flowing over the wildcard socket via cellular, and nothing has to re-resolve
3292+
anything. **The failure needs a network change that outlives its connections** —
3293+
leaving, not toggling. Every off-LAN test so far was run from the house.
3294+
3295+
### What was recorded too strongly
3296+
3297+
"Off-LAN is proven" has been in the record since 2026-09-11. It was measured
3298+
with the phone on mobile data **while still at home**. Off-LAN via relay is real
3299+
and the relay did come back — but it depended on a notification that was never
3300+
being sent, so it was never as robust as the record claimed. See D32.

‎follower/src/main/kotlin/nz/diaswarm/follower/Endpoint.kt‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,13 @@ object Endpoint {
6161
return
6262
}
6363
handle = h
64+
// **AFTER THE HANDLE IS SET, SO THE FIRST CALLBACK FINDS ONE.** iroh
65+
// cannot see an Android network change; without this the phone keeps
66+
// sockets bound to the network it left. See [nz.diaswarm.jni.NetworkWatch].
67+
nz.diaswarm.jni.NetworkWatch.start(
68+
context.applicationContext,
69+
{ Log.i(SyncWorker.TAG, it) }
70+
) { handle }
6471
Log.i(SyncWorker.TAG, "in the pool as ${SwarmNative.swarmNodeId(h).take(16)}")
6572
}
6673

@@ -83,6 +90,9 @@ object Endpoint {
8390
val app = context.applicationContext
8491
Thread {
8592
synchronized(this) {
93+
// BEFORE THE HANDLE GOES ANYWHERE — `swarmLeave` frees it, and
94+
// this returns only once no notification still holds it.
95+
nz.diaswarm.jni.NetworkWatch.stop(app) { Log.i(SyncWorker.TAG, it) }
8696
val old = handle
8797
handle = 0L
8898
if (old != 0L) {

0 commit comments

Comments
 (0)