Move to another node when one cannot be reached, and only then

A node that is down was a dead end: the tunnel reported a failure and the user
had to pick another node themselves, on a screen that did not say which of their
nodes was working.

The policy is deliberately one rule. **Only unreachability moves a connection.**
A Portal answers a rejected authentication frame with silence rather than a
close — on purpose, so that failure is not an oracle for active probing — which
is the same shape on the wire as a Portal that is down. A client that failed over
on it would spend the whole node list on one mistyped character, take several
seconds doing it, and then report the last node's failure: the wrong node, the
wrong message, and a user sent to check the wrong thing. So a refusal ends the
attempt where it is, and a refused pin is a refusal rather than an outage
because every other node will answer the same way to the same wrong pin.

Health is a memory with the memory fading. Consecutive failures are counted, a
success clears them, a refusal is a separate axis rather than a bigger failure
count, and anything older than ten minutes stops counting — without decay the
first list a user builds is the last one they can use, because every node that
ever failed stays condemned and the ranking freezes. It is not persisted, for
the same reason: what it measures is the device's network, and health carried
across a restart would describe the coffee shop's Wi-Fi in the user's kitchen.

Ranking puts health first and latency second, in that order. A node that answers
a probe in 12 ms and then carries nothing is worse than one that answers in
200 ms and works, and sorting on latency would keep the fast broken one at the
top forever. An untried node sorts between healthy and degraded, because the
alternative is a list where a node can never earn its place back, and a refusing
node is sorted last rather than removed — five nodes offering four, with no way
to see why, is worse than five with one at the bottom. The user's own order is
the final tiebreak, and automatic selection never overrides a node they picked:
they chose it for a reason this client cannot see.

`nodes.*` joins the coverage gate for the reason `routing.*` did. A failover
that moved on the wrong kind of failure sends a user's traffic through a Portal
they did not pick, and reports the wrong node's error while doing it.

Writing the device case taught something worth keeping. `TunnelController.state`
is process-wide and outlives the test that set it, so "wait until it is not
Connecting" is true before the service has run and "wait until it is Failed" is
true because the previous test failed. Both were written and both passed
vacuously. The wait is now on the health itself, which `@Before` clears, so it
waits for this attempt and no other.

Author: AlbertoDeckow

app/build.gradle.kts

@@ -247,6 +247,11 @@ kover {
                     // worst direction: traffic leaves the device somewhere the
                     // user did not ask it to.
                     "eu.nodepass.somewhere.routing.*",
+                    // Which node carries the traffic, for the same reason
+                    // again: a failover that moved on the wrong kind of failure
+                    // sends a user's traffic through a Portal they did not pick
+                    // and reports the wrong node's error while doing it.
+                    "eu.nodepass.somewhere.nodes.*",
                 )
             }
             excludes {

app/src/androidTest/kotlin/eu/nodepass/somewhere/vpn/FailoverOnDeviceTest.kt

@@ -0,0 +1,117 @@
+// SPDX-License-Identifier: GPL-3.0-only
+// Copyright (C) 2026 The Somewhere Authors
+
+package eu.nodepass.somewhere.vpn
+
+import androidx.test.ext.junit.runners.AndroidJUnit4
+import eu.nodepass.somewhere.SomewhereApplication
+import eu.nodepass.somewhere.data.NodeStore
+import eu.nodepass.somewhere.nodes.Attempt
+import eu.nodepass.somewhere.nodes.Health
+import eu.nodepass.somewhere.protocol.DecodeResult
+import eu.nodepass.somewhere.protocol.url.NowhereUrl
+import org.junit.After
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertTrue
+import org.junit.Before
+import org.junit.Test
+import org.junit.runner.RunWith
+
+/**
+ * Failing over on a device, where the node list and the service are real.
+ *
+ * The policy itself is unit-tested and needs no device — what needs one is the
+ * wiring: that the service records what happened to the node it was given, and
+ * that an unreachable node leads to another being tried rather than to a
+ * failure the user has to act on.
+ *
+ * **A node that cannot be resolved is the reachable-looking failure to stage.**
+ * A refused connection needs something listening to refuse it; a name that does
+ * not exist fails the same way from every network, on every device, in under a
+ * second, and reaches exactly the code path a Portal that is down reaches.
+ */
+@RunWith(AndroidJUnit4::class)
+class FailoverOnDeviceTest {
+    private val application get() = TunnelHarness.context.applicationContext as SomewhereApplication
+
+    private fun url(host: String): NowhereUrl =
+        when (val parsed = NowhereUrl.parse("nowhere://key@$host:443?up=tcp&down=tcp")) {
+            is DecodeResult.Ok -> parsed.value
+            is DecodeResult.Invalid -> throw AssertionError("test URL does not parse: ${parsed.reason.detail}")
+        }
+
+    private var saved: List<NodeStore.Entry> = emptyList()
+
+    @Before
+    fun rememberTheUsersNodes() {
+        E2eEnvironment.requireConsent(TunnelHarness.context)
+        saved = application.nodes.nodes.value
+        application.nodeHealth.clear()
+    }
+
+    @After
+    fun restore() {
+        application.nodeHealth.clear()
+        runCatching { TunnelHarness.stop() }
+    }
+
+    /**
+     * Waits for the service to have an opinion about [node].
+     *
+     * On the health rather than on the tunnel state, and that is the whole
+     * lesson of writing this. `state` is process-wide and survives the test
+     * that set it, so "wait until it is not Connecting" is true before the
+     * service has run, and "wait until it is Failed" is true because the
+     * *previous* test failed. Health is cleared in `@Before`, so waiting for it
+     * to stop being Untried waits for this attempt and no other.
+     */
+    private fun awaitOutcome(node: NowhereUrl) =
+        TunnelHarness.await("the service to record an outcome for ${node.host}") {
+            application.nodeHealth.health(node.toUrl()) != Health.Untried
+        }
+
+    @Test
+    fun anUnresolvableNodeIsRecordedAsUnreachableRatherThanRefused() {
+        // The distinction the whole policy rests on. Unreachable moves to
+        // another node; refused does not, because a Portal answers a bad key
+        // with silence and failing over on that would walk the entire list on
+        // one mistyped character.
+        val node = url("no-such-host.invalid")
+        SomewhereVpnService.start(TunnelHarness.context, node)
+        awaitOutcome(node)
+
+        val health = application.nodeHealth.health(node.toUrl())
+        assertTrue(
+            "an unresolvable node should be degraded, not refusing; it is $health",
+            health is Health.Degraded,
+        )
+    }
+
+    @Test
+    fun aSucceedingNodeClearsWhatWentBeforeIt() {
+        // Recorded through the same object the service writes to, so a
+        // regression in either half shows up here.
+        val node = url("no-such-host.invalid")
+        application.nodeHealth.record(node.toUrl(), Attempt.Unreachable)
+        application.nodeHealth.record(node.toUrl(), Attempt.Succeeded)
+        assertEquals(Health.Healthy, application.nodeHealth.health(node.toUrl()))
+    }
+
+    @Test
+    fun theServiceRecordsHealthAgainstTheNodeItWasActuallyGiven() {
+        // Keyed on the URL the service was started with, which is what the
+        // node list holds. A key derived any other way — the display name, the
+        // host alone — would collide between two nodes on one Portal and
+        // condemn both when one failed.
+        val first = url("no-such-host-one.invalid")
+        val second = url("no-such-host-two.invalid")
+        SomewhereVpnService.start(TunnelHarness.context, first)
+        awaitOutcome(first)
+        assertTrue(application.nodeHealth.health(first.toUrl()) is Health.Degraded)
+        assertEquals(
+            "an untouched node must not be marked by another node's failure",
+            Health.Untried,
+            application.nodeHealth.health(second.toUrl()),
+        )
+    }
+}

app/src/main/kotlin/eu/nodepass/somewhere/SomewhereApplication.kt

@@ -9,6 +9,7 @@ import eu.nodepass.somewhere.apps.PackageManagerApps
 import eu.nodepass.somewhere.data.NodeRepository
 import eu.nodepass.somewhere.data.NodeStore
 import eu.nodepass.somewhere.data.SubscriptionStore
+import eu.nodepass.somewhere.nodes.NodeHealth
 import eu.nodepass.somewhere.routing.RoutingPreferences
 import eu.nodepass.somewhere.routing.RuleStore
 import eu.nodepass.somewhere.subscription.SubscriptionFetcher
@@ -64,4 +65,15 @@ class SomewhereApplication : Application() {
     val rules: RuleStore by lazy { RuleStore(File(filesDir, "routing/rules.txt")) }
 
     val routing: RoutingPreferences by lazy { RoutingPreferences(File(filesDir, "routing/settings.txt")) }
+
+    /**
+     * How nodes have behaved, this run.
+     *
+     * Deliberately not persisted. What it measures is the device's network, and
+     * that changes when the device moves — health carried across a restart
+     * would describe the coffee shop's Wi-Fi in the user's kitchen. Latency
+     * from a probe is persisted by the repository, because a measurement is a
+     * measurement; a verdict is not.
+     */
+    val nodeHealth: NodeHealth by lazy { NodeHealth() }
 }

app/src/main/kotlin/eu/nodepass/somewhere/nodes/Failover.kt

@@ -0,0 +1,64 @@
+// SPDX-License-Identifier: GPL-3.0-only
+// Copyright (C) 2026 The Somewhere Authors
+
+package eu.nodepass.somewhere.nodes
+
+/**
+ * Whether to try another node, and which.
+ *
+ * Separate from [NodeHealth] because the two answer different questions and
+ * only one of them is allowed to be clever. Health is a measurement; this is a
+ * policy, and the policy is deliberately small.
+ *
+ * ## The rule that matters
+ *
+ * **Only unreachability moves a connection.** Everything else — a refused
+ * authentication, a Portal that answered and said no, a node the user
+ * configured wrongly — stays where it is and reports what happened. See
+ * [Attempt] for why: a Portal answers a bad key with silence, so a client that
+ * treated silence as unreachability would walk the whole list on one typo and
+ * then blame the last node.
+ *
+ * ## The other rule
+ *
+ * **A node is tried once per attempt.** Without that, two nodes that each fail
+ * over to the other spin until something else notices. The excluded set is the
+ * caller's, so an attempt that spans a reconnect can carry it or drop it
+ * deliberately rather than by accident.
+ */
+object Failover {
+    /** Whether [attempt] is the kind of failure another node could fix. */
+    fun shouldMove(attempt: Attempt): Boolean = attempt is Attempt.Unreachable
+
+    /**
+     * The next node to try, or null when there is none worth trying.
+     *
+     * @param candidates every node the user has, in their own order.
+     * @param tried nodes this attempt has already used, including the one that
+     *   just failed.
+     */
+    fun next(
+        candidates: List<String>,
+        health: NodeHealth,
+        tried: Set<String>,
+        latencyMillis: (String) -> Int? = { null },
+    ): String? =
+        health
+            .ranked(candidates, latencyMillis)
+            .firstOrNull { it !in tried && health.health(it).isUsable }
+
+    /**
+     * The node to start on when nobody has chosen one.
+     *
+     * **Not used when the user has chosen.** An automatic selection that
+     * overrode a person's pick would be a defect wearing a feature's clothes:
+     * they chose that node for a reason this client cannot see — a region, a
+     * subscription's quota, a Portal they administer — and the ranking here
+     * knows none of it.
+     */
+    fun preferred(
+        candidates: List<String>,
+        health: NodeHealth,
+        latencyMillis: (String) -> Int? = { null },
+    ): String? = health.ranked(candidates, latencyMillis).firstOrNull { health.health(it).isUsable }
+}