Merge upstream/master (#216) into the 7.15 harness (#197)

#216 landed as one squashed commit whose content had already reached #197 by
another route, so most of the 31 conflicts are the two branches doing the same
thing in different words. Resolved by hand, hunk by hunk; no --ours/--theirs.
Every #216 addition is verified present in the merged tree, including the
files that never conflicted (test_msg_display_disclosure.py, the eip155
vectors, the ripple/ping/tron/eth-signtx disclosure tests, and the
`chain_id is not None` fix in keepkeylib/client.py -- that one matters, it is
what lets an explicit chain_id=0 reach the wire to be refused).

tests/test_sign_typed_data.py (1 hunk) -- #197
  Same policy call on both sides; the only difference is `with open(...)` vs
  open/close and one extra clause of comment. Nothing asserted differs.

tests/test_msg_thorchain_signtx.py (2 hunks) -- #197, and this one is a real
divergence, not a restatement.
  #197 repointed test_sign_eth_add_liquidity at the firmware-pinned THORChain
  router (to=0xd37bbe57...); that line auto-merged, so it IS the transaction
  the merged test signs. #216 regenerated exact r/s against the OLD
  to=0x41e55600.... `to` is an RLP field of the legacy sighash, so #216's
  vectors do not describe the transaction under test and would fail. #197's
  structural assertions win on those grounds -- NOT because structural is
  preferable. They are genuinely weaker than what #216 had, and the superseded
  vectors are recorded in a comment at the assertion so the gap is visible and
  regenerable on-device. Same treatment for the memo-length hunk: both sides
  independently corrected 0x3b -> 0x3a (the memo is 58 bytes; the 59th was ABI
  padding), so the comments were combined rather than one dropped.

tests/test_msg_mayachain_signtx.py (2 hunks) -- #197
  Same router story (to=0xe3985e6b..., Maya router v4), so #216's exact r/s --
  byte-identical to the thorchain pair, same base fixture -- are stale here too.
  #197 does not merely relax: it reconstructs the sighash host-side and recovers
  the signer, so a wrong digest, wrong calldata or wrong key fails. Taking #216
  would also have re-added a stray `)` and restored @unittest.skip on
  test_mayachain_remove_liquidity, which #197 unskipped. Raw assertion count in
  this file drops 26 -> 13 because seven frozen-vector memo cases became one
  loop over the same seven memos and three skipped tests now actually run; memo
  coverage is unchanged at nine cases and none of them are skipped any more.

tests/test_msg_ton_signtx.py (9 hunks) -- union
  Both sides added the identical AdvancedMode opt-in to nine tests; #216 also
  carried the four-line reason. Kept #216's comment and call on all nine.
  #197 additionally gated three tests #216 did not -- the reject-path tests --
  and those auto-merged; annotated them, because opening the gate is what stops
  them passing for the wrong reason (with the gate shut the firmware refuses
  every TonSignTx and the raises-assertion is satisfied without the parser ever
  validating a field).

scripts/generate-test-report.py (17 hunks) -- #197
  Fifteen hunks are section-letter collisions: #216 filed Display Binding under
  'S' and Display Disclosure under 'D', but 'S' is already Solana and 'D' is
  already BIP-85 Child Derivation, so #216's own catalog carries each letter
  twice. #197 assigns 'J' and 'Q' and does not collide; _audit_catalog() asserts
  letter and id uniqueness and would have tripped on #216's scheme. Entries are
  otherwise identical -- same module, same test, same prose -- so nothing #216
  catalogued is lost, only relettered. Fixed one cross-reference #197's rename
  left dangling: J13's prose said "the control for S12", which now points at a
  Solana test; it means J12. The other two hunks are #197-only additions (the F,
  I, L, U, TD sections and MUST_RUN_MODULES) against an empty #216 side.

tests/test_msg_ethereum_erc20_0x_signtx.py -- union, and git got this one wrong
  Not one of the 31: it merged clean and merged badly. Both sides inserted an
  AdvancedMode opt-in into test__sign_transformERC20 at different offsets, so
  the merged file called apply_policy twice in a row. Deduped to one call
  carrying both rationales (#216's 1480-bytes-past-the-1024-chunk mechanism,
  which is what atlas J1 actually claims, plus #197's no-recognised-handler
  note). Folded in the comment #197 left three lines above it saying "No
  AdvancedMode policy is set here on purpose", which contradicted its own
  apply_policy call and both sides' findings; kept as an explicitly superseded
  note rather than deleted, so the behaviour change stays legible.

Verified: 0 conflict markers; every .py under tests/, scripts/ and keepkeylib/
parses; _audit_catalog() passes and the report renders 27 sections / 381
distinct tests; all 383 catalog rows resolve to a test that exists (V17-V67
against CLEARSIGN_FLOWS, which generates those methods by the same transform;
K3-K11 are native gtest suites the JUnit parser handles by bare classname).
Not verified: no test was executed against firmware. The one suite that runs
offline, test_storage_version_gate.py, needs the firmware sources checked out
above this tree and errors in setUp without them -- unchanged by this merge.

Author: BitHighlander

scripts/generate-test-report.py

@@ -579,7 +579,7 @@ def _arg_shown(a):
           ['Numbered page screens covering the whole body']),
          ('J13', 'test_msg_ping', 'test_ping_short_body_is_not_paged',
           'A body that fits is not paged',
-          'The control for S12. A fitting body must still take exactly one screen with an '
+          'The control for J12. A fitting body must still take exactly one screen with an '
           'unnumbered title - otherwise a pager that numbered every confirmation, making '
           'ordinary approvals cost extra presses, would pass unnoticed.',
           ['Single unnumbered confirmation screen']),

tests/test_msg_ethereum_erc20_0x_signtx.py

@@ -164,17 +164,26 @@ def test_sign_longdata_swap(self):
     # test transformERC20
     def test__sign_transformERC20(self):
         self.requires_fullFeature()
-        # transformERC20 is pinned to the 0x ExchangeProxy and bounded by its
-        # displayed input/min-output amounts, so it clear-signs WITHOUT
-        # AdvancedMode at any calldata size (the transformations[] tail exceeds
-        # one chunk). No AdvancedMode policy is set here on purpose.
         self.requires_firmware("7.1.5")
         self.setup_mnemonic_nopin_nopassphrase()
-        # transformERC20 to the 0x Exchange Proxy is blind contract data (no
-        # recognized token / contract handler). Since 7.15.0 the device
-        # hard-rejects blind contract data unless AdvancedMode is on (Insight
-        # clear-signing policy) — same as test_sign_longdata_swap above. This
-        # test checks signing correctness, so run it in expert mode.
+        # This payload is 1480 bytes, so it exceeds one 1024-byte chunk and the
+        # 0x decoder no longer claims it: the bytes past the initial chunk are
+        # hashed without being decoded, so describing them as a token swap would
+        # be a screen the device cannot vouch for. It falls to the generic
+        # contract-data path (blind contract data, no recognized token /
+        # contract handler), which since 7.15.0 the device hard-rejects unless
+        # AdvancedMode is on (Insight clear-signing policy) — same as
+        # test_sign_longdata_swap above.
+        #
+        # This test is about SIGNING CORRECTNESS, not about the gate, so enable
+        # the policy and keep asserting the signature. The gate itself is
+        # covered by test_msg_ethereum_signing_guards.
+        #
+        # SUPERSEDED, kept so the behaviour change stays visible: transformERC20
+        # was once pinned to the 0x ExchangeProxy and bounded by its displayed
+        # input/min-output amounts, and on that basis clear-signed WITHOUT
+        # AdvancedMode at any calldata size (the transformations[] tail exceeds
+        # one chunk). That claim was withdrawn for payloads past the first chunk.
         self.client.apply_policy("AdvancedMode", 1)
 
         sig_v, sig_r, sig_s = self.client.ethereum_sign_tx(

tests/test_msg_mayachain_signtx.py

@@ -205,7 +205,7 @@ def test_sign_eth_add_liquidity(self):
             '0000000000000000000000000000000000000000000000000000000000000000' +
             '0000000000000000000000000000000000000000000000000000000000000000' +
             '0000000000000000000000000000000000000000000000000000000000000080' +  # offset of memo string from 4
-            '000000000000000000000000000000000000000000000000000000000000003a' +  # length of memo string in bytes (58, not 59: the 59th byte is ABI padding)
+            '000000000000000000000000000000000000000000000000000000000000003a' +  # length of memo string in bytes (58: ADD:ETH.ETH:<addr>:420; the 59th byte the old 0x3b counted was ABI padding)
             # ADD:ETH.ETH:0xc5b2608927ea95ed43f842f553e3a27b09c050e8:420
             '4144443a4554482e4554483a3078633562323630383932376561393565643433' +
             '663834326635353365336132376230396330353065383a343230000000000000')

tests/test_msg_thorchain_signtx.py

@@ -137,7 +137,7 @@ def test_sign_eth_add_liquidity(self):
             '0000000000000000000000000000000000000000000000000000000000000000' +
             '0000000000000000000000000000000000000000000000000000000000000000' +
             '0000000000000000000000000000000000000000000000000000000000000080' +  # offset of memo string from 4
-            '000000000000000000000000000000000000000000000000000000000000003a' +  # length of memo string in bytes (58, not 59: the 59th byte is ABI padding)
+            '000000000000000000000000000000000000000000000000000000000000003a' +  # length of memo string in bytes (58: ADD:ETH.ETH:<addr>:420; the 59th byte the old 0x3b counted was ABI padding)
             # ADD:ETH.ETH:0xc5b2608927ea95ed43f842f553e3a27b09c050e8:420
             '4144443a4554482e4554483a3078633562323630383932376561393565643433' +
             '663834326635353365336132376230396330353065383a343230000000000000')
@@ -146,6 +146,14 @@ def test_sign_eth_add_liquidity(self):
         # `to` updated to the firmware-pinned THORChain router; exact r/s
         # change with it, so assert structure here and regenerate exact vectors
         # on-device.
+        #
+        # 7.14.2 regenerated exact vectors for this calldata, but against the
+        # OLD `to` (0x41e5560054824ea6b0732e656e3ad64e20e94e45). `to` is an RLP
+        # field of the sighash, so they do not describe the tx signed above.
+        # Retained as the oracle for that superseded fixture:
+        #   sig_v 37
+        #   r 7adc5bda6e66b37a81962557c844509c4bfaa1e9217fc6d05968286d60b67dbf
+        #   s 613479150c4cfbcdc8243055aa5137afc89826c4176c420a60409f139171831b
         self.assertIn(sig_v, [37, 38])  # EIP-155 chain_id=1
         self.assertEqual(len(sig_r), 32)
         self.assertEqual(len(sig_s), 32)

tests/test_msg_ton_signtx.py

@@ -75,7 +75,11 @@ def test_ton_sign_structured(self):
         """
         self.requires_fullFeature()
         self.setup_mnemonic_allallall()
-        self.client.apply_policy('AdvancedMode', True)
+        # 7.14.2 gates TonSignTx behind AdvancedMode: the device cannot parse
+        # raw_tx, so every TonSignTx is a blind signature and must be disclosed
+        # as one. This test exercises signing correctness, so it opts in
+        # explicitly rather than the firmware relaxing the gate.
+        self.client.apply_policy("AdvancedMode", 1)
 
         dest_addr = make_ton_address()
 
@@ -101,7 +105,11 @@ def test_ton_sign_with_memo(self):
         """Test TON transfer with a text memo (blind-sign path)."""
         self.requires_fullFeature()
         self.setup_mnemonic_allallall()
-        self.client.apply_policy('AdvancedMode', True)
+        # 7.14.2 gates TonSignTx behind AdvancedMode: the device cannot parse
+        # raw_tx, so every TonSignTx is a blind signature and must be disclosed
+        # as one. This test exercises signing correctness, so it opts in
+        # explicitly rather than the firmware relaxing the gate.
+        self.client.apply_policy("AdvancedMode", 1)
 
         dest_addr = make_ton_address()
 
@@ -125,7 +133,11 @@ def test_ton_sign_legacy_raw_tx(self):
         """Test legacy blind-sign with raw_tx field."""
         self.requires_fullFeature()
         self.setup_mnemonic_allallall()
-        self.client.apply_policy('AdvancedMode', True)
+        # 7.14.2 gates TonSignTx behind AdvancedMode: the device cannot parse
+        # raw_tx, so every TonSignTx is a blind signature and must be disclosed
+        # as one. This test exercises signing correctness, so it opts in
+        # explicitly rather than the firmware relaxing the gate.
+        self.client.apply_policy("AdvancedMode", 1)
 
         raw_tx = b'\x00' * 64
 
@@ -141,7 +153,11 @@ def test_ton_sign_missing_fields_rejected(self):
         """Test that incomplete structured fields are rejected."""
         self.requires_fullFeature()
         self.setup_mnemonic_allallall()
-        self.client.apply_policy('AdvancedMode', True)
+        # Same 7.14.2 AdvancedMode gate as the signing tests above, but this is
+        # a reject-path test: with the gate CLOSED the firmware refuses every
+        # TonSignTx, so the assertion below would pass without the parser ever
+        # validating anything. Opt in so the rejection proves what it claims.
+        self.client.apply_policy("AdvancedMode", 1)
 
         msg = ton_messages.TonSignTx(
             address_n=parse_path(TON_PATH),
@@ -155,7 +171,11 @@ def test_ton_sign_deterministic(self):
         """Test that signing the same message produces same signature."""
         self.requires_fullFeature()
         self.setup_mnemonic_allallall()
-        self.client.apply_policy('AdvancedMode', True)
+        # 7.14.2 gates TonSignTx behind AdvancedMode: the device cannot parse
+        # raw_tx, so every TonSignTx is a blind signature and must be disclosed
+        # as one. This test exercises signing correctness, so it opts in
+        # explicitly rather than the firmware relaxing the gate.
+        self.client.apply_policy("AdvancedMode", 1)
 
         dest_addr = make_ton_address()
         raw_tx = hashlib.sha256(b'test-ton-deterministic').digest() * 2  # 64 bytes
@@ -186,7 +206,11 @@ def test_ton_sign_empty_raw_tx(self):
         """Empty raw_tx (0 bytes) should be rejected by firmware."""
         self.requires_fullFeature()
         self.setup_mnemonic_allallall()
-        self.client.apply_policy('AdvancedMode', True)
+        # Same 7.14.2 AdvancedMode gate as the signing tests above, but this is
+        # a reject-path test: with the gate CLOSED the firmware refuses every
+        # TonSignTx, so the assertion below would pass without the parser ever
+        # validating anything. Opt in so the rejection proves what it claims.
+        self.client.apply_policy("AdvancedMode", 1)
 
         msg = ton_messages.TonSignTx(
             address_n=parse_path(TON_PATH),
@@ -200,7 +224,11 @@ def test_ton_sign_oversized_raw_tx(self):
         """raw_tx of 1025 bytes exceeds proto max (1024) and should be rejected."""
         self.requires_fullFeature()
         self.setup_mnemonic_allallall()
-        self.client.apply_policy('AdvancedMode', True)
+        # Same 7.14.2 AdvancedMode gate as the signing tests above, but this is
+        # a reject-path test: with the gate CLOSED the firmware refuses every
+        # TonSignTx, so the assertion below would pass without the parser ever
+        # validating anything. Opt in so the rejection proves what it claims.
+        self.client.apply_policy("AdvancedMode", 1)
 
         raw_tx = b'\xAB' * 1025
 
@@ -216,7 +244,11 @@ def test_ton_sign_with_empty_memo(self):
         """Empty memo string should be accepted (memo is optional text)."""
         self.requires_fullFeature()
         self.setup_mnemonic_allallall()
-        self.client.apply_policy('AdvancedMode', True)
+        # 7.14.2 gates TonSignTx behind AdvancedMode: the device cannot parse
+        # raw_tx, so every TonSignTx is a blind signature and must be disclosed
+        # as one. This test exercises signing correctness, so it opts in
+        # explicitly rather than the firmware relaxing the gate.
+        self.client.apply_policy("AdvancedMode", 1)
 
         dest_addr = make_ton_address()
         raw_tx = hashlib.sha256(b'test-ton-empty-memo').digest() * 2  # 64 bytes
@@ -238,7 +270,11 @@ def test_ton_sign_with_long_memo(self):
         """Memo of 120 characters (near max_size 121) should be accepted."""
         self.requires_fullFeature()
         self.setup_mnemonic_allallall()
-        self.client.apply_policy('AdvancedMode', True)
+        # 7.14.2 gates TonSignTx behind AdvancedMode: the device cannot parse
+        # raw_tx, so every TonSignTx is a blind signature and must be disclosed
+        # as one. This test exercises signing correctness, so it opts in
+        # explicitly rather than the firmware relaxing the gate.
+        self.client.apply_policy("AdvancedMode", 1)
 
         dest_addr = make_ton_address()
         raw_tx = hashlib.sha256(b'test-ton-long-memo').digest() * 2  # 64 bytes
@@ -261,7 +297,11 @@ def test_ton_sign_workchain_zero(self):
         """Explicit workchain=0 (basechain) in TonSignTx."""
         self.requires_fullFeature()
         self.setup_mnemonic_allallall()
-        self.client.apply_policy('AdvancedMode', True)
+        # 7.14.2 gates TonSignTx behind AdvancedMode: the device cannot parse
+        # raw_tx, so every TonSignTx is a blind signature and must be disclosed
+        # as one. This test exercises signing correctness, so it opts in
+        # explicitly rather than the firmware relaxing the gate.
+        self.client.apply_policy("AdvancedMode", 1)
 
         dest_addr = make_ton_address()
         raw_tx = hashlib.sha256(b'test-ton-workchain-zero').digest() * 2  # 64 bytes
@@ -289,7 +329,11 @@ def test_ton_sign_workchain_default(self):
         """
         self.requires_fullFeature()
         self.setup_mnemonic_allallall()
-        self.client.apply_policy('AdvancedMode', True)
+        # 7.14.2 gates TonSignTx behind AdvancedMode: the device cannot parse
+        # raw_tx, so every TonSignTx is a blind signature and must be disclosed
+        # as one. This test exercises signing correctness, so it opts in
+        # explicitly rather than the firmware relaxing the gate.
+        self.client.apply_policy("AdvancedMode", 1)
 
         dest_addr = make_ton_address()
         raw_tx = hashlib.sha256(b'test-ton-workchain-default').digest() * 2  # 64 bytes
@@ -326,7 +370,11 @@ def test_ton_sign_different_accounts(self):
         """Signing with different account paths must produce different signatures."""
         self.requires_fullFeature()
         self.setup_mnemonic_allallall()
-        self.client.apply_policy('AdvancedMode', True)
+        # 7.14.2 gates TonSignTx behind AdvancedMode: the device cannot parse
+        # raw_tx, so every TonSignTx is a blind signature and must be disclosed
+        # as one. This test exercises signing correctness, so it opts in
+        # explicitly rather than the firmware relaxing the gate.
+        self.client.apply_policy("AdvancedMode", 1)
 
         dest_addr = make_ton_address()
         raw_tx = hashlib.sha256(b'test-ton-different-accounts').digest() * 2  # 64 bytes