Support save version 67.2 (DE build 101.103.x) - #147
Open
gwaihirf22 wants to merge 1 commit into
Open
gwaihirf22 wants to merge 1 commit into
gwaihirf22 wants to merge 1 commit into
Conversation
Two fields moved in 67.2 and both break the fast header parser outright;
every 67.2 and 68.0 file fails.
1. Each player record in `parse_de` gained a trailing `de_string`. Without
it the parser reads a record, then expects the next player's `dlc_id`
where a string marker now sits, and fails on the first of eight
players. Confirmed against three record shapes: a human record ends at
+107, an AI one at +207, an empty slot at +239, and there is an empty
`de_string` at each. Skipping it puts every following record's
`ai_type` exactly where it belongs.
2. `parse_metadata` reads its first four bytes as the AI count. In 67.2
two words were inserted ahead of that field and the first is a
timestamp, so the count is always non-zero and the forward scan for the
end of the AI data always runs. Where AI data exists the scan happens
to land correctly; where a game has NO AI opponent it matches the first
unrelated zero run and overshoots, and everything after reads as
garbage - game_speed 0.0, owner_id 0, num_players 0. An empty player
list is what finally raises, in `parse_players`, a long way from the
cause.
with an AI 7f 91 8c 6a | 00 00 00 00 | 01 00 00 00 then "Attacking..."
no opponent 17 b6 88 6a | 00 00 00 00 | 00 00 00 00 then zeros
^ timestamp ^ reserved ^ the real count
Thresholds are bracketed by measurement rather than assumed: at 66.6 and
below the first word IS the count, at 67.2 and above it is a timestamp.
Measured over a local archive of 340+ recorded games spanning fifteen save
versions: 67.2 goes from 0/9 files to 9/9 and 68.0 from 0/42 to 41/42,
with every version from 26.32 to 66.6 byte-for-byte identical with and
without the patch. The one remaining 68.0 file is the recording the game
writes while a match is in progress, which is truncated and correctly
refused.
The same two fields are added to the construct parser, where they are
also correct - but 67.2 changed something inside `initial` as well, so the
construct header still runs off the end of the stream on these files. I
could not measure that change and did not want to guess at it, so the two
new recs are added to the existing skip list in test_files_fast.
Test recs are a 68.0 game against an AI and a 68.0 game with no opponent,
the latter being the case that fails differently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every recorded game at save version 67.2 and 68.0 (DE build
101.103.x) fails to parse. Two fields moved, and both are gated at>= 67.2here.1. Each player record gained a trailing
de_stringparse_dereads a player record to its end, then expects the next player'sdlc_idwhere a string marker now sits — so it fails on the first of eight players:Three record shapes confirm it. A human record ends at +107, an AI one at +207, an empty slot at +239 — and there is an empty
de_stringat each of those offsets. Skipping it puts every following record'sai_typeexactly where it belongs (the next one landing onPromiDE, the AI's name, rather than mid-field).2. The AI count is no longer the first field of the metadata section
parse_metadatareads its first four bytes as the AI count. In 67.2 two words were inserted ahead of it and the first is a timestamp, so the count is always non-zero and the forward scan for the end of the AI data always runs:Where AI data exists the scan lands correctly by construction — the section ends in a zero pad and the following struct opens with 24 skipped bytes that are themselves zero. Where a game has no AI opponent there is no AI data at all, the scan matches the first unrelated zero run and overshoots (offset 6007 in my sample, where the metadata struct is at 1807), and everything after reads as garbage:
game_speed=0.0,owner_id=0,num_players=0. The empty player list is what finally raises, inparse_players, a long way from the cause.That is why the affected files are single-player games — no opponent, no AI section, nothing to find.
Measurements
Thresholds are bracketed rather than assumed. Reading the first three words of the metadata section, one file per save version:
Against a local archive of 340+ recorded games across fifteen save versions:
The one remaining 68.0 file is
rec.aoe2record, the recording the game writes while a match is in progress. It is truncated by definition and refusing it is correct.Your test suite on my machine: 1 failed → 0 failed, and the 15 errors (HD, UserPatch15, DE scenario, model) are present identically with and without this patch.
What this does not fix
The same two fields are added to the construct parser and are correct there, but 67.2 changed something inside
initialas well — the construct header still runs off the end of the stream on these files, failing ininitialorscenario_header. I could not measure that change and did not want to guess at a layout I hadn't verified, so the two new recs join the existing skip list intest_files_fast.Happy to split this into two commits, drop the construct changes, or supply more sample files — the archive here spans 26.32 to 68.0, and I have the no-opponent case at several versions.
Test data
de-68.0.aoe2record(a game against an AI) andde-68.0-no-opponent.aoe2record(the case that fails differently). Both are small — 0.8 MB and 0.7 MB.