Skip to content

Support save version 67.2 (DE build 101.103.x) - #147

Open
gwaihirf22 wants to merge 1 commit into
happyleavesaoc:masterfrom
gwaihirf22:support-save-version-67-2
Open

gwaihirf22 wants to merge 1 commit into
happyleavesaoc:masterfrom
gwaihirf22:support-save-version-67-2

Conversation

@gwaihirf22

Copy link
Copy Markdown

Every recorded game at save version 67.2 and 68.0 (DE build 101.103.x) fails to parse. Two fields moved, and both are gated at >= 67.2 here.

1. Each player record gained a trailing de_string

parse_de reads a player record to its end, then expects the next player's dlc_id where a string marker now sits — so it fails on the first of eight players:

RuntimeError: invalid mgz file: expected 8 to 8, found 1
    (parsing) -> de -> de -> players

Three record shapes confirm it. A human record ends at +107, an AI one at +207, an empty slot at +239 — and there is an empty de_string at each of those offsets. Skipping it puts every following record's ai_type exactly where it belongs (the next one landing on PromiDE, the AI's name, rather than mid-field).

2. The AI count is no longer the first field of the metadata section

parse_metadata reads its first four bytes as the AI count. In 67.2 two words were inserted ahead of it and the first is a timestamp, so the count is always non-zero and the forward scan for the end of the AI data always runs:

with an AI    7f 91 8c 6a | 00 00 00 00 | 01 00 00 00   then "Attacking..."
no opponent   17 b6 88 6a | 00 00 00 00 | 00 00 00 00   then zeros
              ^ timestamp    ^ reserved   ^ the real count

Where AI data exists the scan lands correctly by construction — the section ends in a zero pad and the following struct opens with 24 skipped bytes that are themselves zero. Where a game has no AI opponent there is no AI data at all, the scan matches the first unrelated zero run and overshoots (offset 6007 in my sample, where the metadata struct is at 1807), and everything after reads as garbage: game_speed=0.0, owner_id=0, num_players=0. The empty player list is what finally raises, in parse_players, a long way from the cause.

That is why the affected files are single-player games — no opponent, no AI section, nothing to find.

Measurements

Thresholds are bracketed rather than assumed. Reading the first three words of the metadata section, one file per save version:

save version first word second third reading
26.32 – 62.0 0 0 0 first word is the AI count
63.0 – 66.6 1 (data) 0 first word is the AI count
67.2 1772083683 0 1 first word is a timestamp
68.0 1787429369 0 1 first word is a timestamp

Against a local archive of 340+ recorded games across fifteen save versions:

save version before after
26.32 – 66.6 (265 files) baseline identical
67.2 (9 files) 0 9
68.0 (42 files) 0 41

The one remaining 68.0 file is rec.aoe2record, the recording the game writes while a match is in progress. It is truncated by definition and refusing it is correct.

Your test suite on my machine: 1 failed → 0 failed, and the 15 errors (HD, UserPatch15, DE scenario, model) are present identically with and without this patch.

What this does not fix

The same two fields are added to the construct parser and are correct there, but 67.2 changed something inside initial as well — the construct header still runs off the end of the stream on these files, failing in initial or scenario_header. I could not measure that change and did not want to guess at a layout I hadn't verified, so the two new recs join the existing skip list in test_files_fast.

Happy to split this into two commits, drop the construct changes, or supply more sample files — the archive here spans 26.32 to 68.0, and I have the no-opponent case at several versions.

Test data

de-68.0.aoe2record (a game against an AI) and de-68.0-no-opponent.aoe2record (the case that fails differently). Both are small — 0.8 MB and 0.7 MB.

Two fields moved in 67.2 and both break the fast header parser outright;
every 67.2 and 68.0 file fails.

1. Each player record in `parse_de` gained a trailing `de_string`. Without
   it the parser reads a record, then expects the next player's `dlc_id`
   where a string marker now sits, and fails on the first of eight
   players. Confirmed against three record shapes: a human record ends at
   +107, an AI one at +207, an empty slot at +239, and there is an empty
   `de_string` at each. Skipping it puts every following record's
   `ai_type` exactly where it belongs.

2. `parse_metadata` reads its first four bytes as the AI count. In 67.2
   two words were inserted ahead of that field and the first is a
   timestamp, so the count is always non-zero and the forward scan for the
   end of the AI data always runs. Where AI data exists the scan happens
   to land correctly; where a game has NO AI opponent it matches the first
   unrelated zero run and overshoots, and everything after reads as
   garbage - game_speed 0.0, owner_id 0, num_players 0. An empty player
   list is what finally raises, in `parse_players`, a long way from the
   cause.

       with an AI    7f 91 8c 6a | 00 00 00 00 | 01 00 00 00  then "Attacking..."
       no opponent   17 b6 88 6a | 00 00 00 00 | 00 00 00 00  then zeros
                     ^ timestamp    ^ reserved   ^ the real count

Thresholds are bracketed by measurement rather than assumed: at 66.6 and
below the first word IS the count, at 67.2 and above it is a timestamp.

Measured over a local archive of 340+ recorded games spanning fifteen save
versions: 67.2 goes from 0/9 files to 9/9 and 68.0 from 0/42 to 41/42,
with every version from 26.32 to 66.6 byte-for-byte identical with and
without the patch. The one remaining 68.0 file is the recording the game
writes while a match is in progress, which is truncated and correctly
refused.

The same two fields are added to the construct parser, where they are
also correct - but 67.2 changed something inside `initial` as well, so the
construct header still runs off the end of the stream on these files. I
could not measure that change and did not want to guess at it, so the two
new recs are added to the existing skip list in test_files_fast.

Test recs are a 68.0 game against an AI and a 68.0 game with no opponent,
the latter being the case that fails differently.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant