Skip to content

Commit 2d5a118

Browse files
sp00ktoberclaude
andauthored
feat(mhp-formats): MH Portable file formats behind one API (#5)
* feat(mhp-formats): package skeleton, GPL-3.0-or-later, dependencies Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): one PAC container for both games Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): DATA.BIN container and its cipher DataBin reads and writes the decrypted archive byte for byte (block table, exact-size records, table fill); decrypt/encrypt run mhef's DataCipher over it in memory, imported only when called. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): MHFU stage file and its HITS collision Stage splits st*.pac into its entries by role over Pac, including the all-empty placeholder stages. Environment decodes kind, fog colour and fog range; Hits decodes the broadphase grid, the stored cell lists and the 56-byte triangles, and writes them back byte-exact. build_hits and its helpers make a chunk from triangles; verify and grid_check report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): extract a game's ISO, and the mhp-formats CLI extract() copies the image's files, identifies the game from UMD_DATA.BIN, and splits the decrypted DATA.BIN into data_files/ under the names and contents the old extractor wrote. `mhp-formats extract ISO OUT` runs it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(mhp-formats): Pac reads a table of only empty entries; stages use it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): the GE vertex type and a byte-exact vertex buffer codec Layout, through mode, morph targets and normalisation as PPSSPP decodes them; colour pack/unpack for 5650/5551/4444/8888. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): GE display lists and the draws they make Commands as data, a byte-exact list reader, and a walker that resolves ORIGIN/OFFSETADDR/BASE and advances VADDR/IADDR after each PRIM as the GE does; triangle extraction per primitive with the FFACE winding rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): stripifier that hands out strips of a requested length Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): skeletons of both games One codec for the 0xC0000000 and 0x80000000 blobs: a parameter section, then a section per bone, 0x10C bytes with a zero tail (MHFU) or 0x5C with an 8-byte name (MHP3rd); the magic is independent of the layout. Round-trips all 4031 MHFU and 2144 MHP3rd skeletons byte-exact, including the three-parameter headers and kind-2/3 bones the old parsers stopped at. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): animation packs of both games The pack is N streams of slot tables, the first table at the header's end; clips follow back to back, and slots sharing a clip share the object. Both games use it; MHFU's clips have u32 tagged sections, MHP3rd's u16 headers and a f32 loop start the old parser dropped. MHFU's lobby and in-game packs are one encoding with 2, 4, 6 or 10 streams. Round-trips all 120 MHFU and 115 MHP3rd packs byte-exact. Keyframes stay raw s16; quantize/dequantize is the one float view. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): PSP texture swizzle, both directions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): TMH texture banks, byte-exact, decode and same-size encode Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): export the GE layer, skeletons and animations Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(mhp-formats): one colour codec for textures and vertices Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build(mhp-formats): mhef from the MHFU-ModKit fork, which runs on Python 3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(mhp-formats): PMO models of both games on the GE layer One Pmo for MHFU's 1.0 and a p3rd subclass for 102: meshes, groups, material remap, bone palette and materials, and the geometry region as per-group blocks (display list, Vertices, indices) laid out through one emitter. Byte-exact on every shipped PMO of both games, the MHP3rd companion geometry files included. Views for triangles, positions, materials and palette influences; edits: build, clear, clear_prims, budget, pack, and an in-place layout for resident PACs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(mhp-formats): PMO lists go out through DisplayList Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(mhp-formats): one quantiser and one GE address state; detect() The vertex quantiser sits next to its inverse in psp.vtype, the PMO codec uses the GE walker's own ORIGIN/OFFSETADDR/BASE state, and detect() picks a format from each one's sniff. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(mhp-formats): the 49 monster PACs rebuild from their decoded entries Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(mhp-formats): README with the model codec; mhff attribution on every derived file Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(mhp-formats): a signalling NaN round-trips or is refused, as the platform allows Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 576df88 commit 2d5a118

49 files changed

Lines changed: 6824 additions & 1 deletion

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎LICENSES/GPL-3.0-or-later.txt‎

Lines changed: 232 additions & 0 deletions
Large diffs are not rendered by default.

‎REUSE.toml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,9 @@ SPDX-License-Identifier = "MIT"
1111
path = "ppsspp/patches/**"
1212
SPDX-FileCopyrightText = ["2026 sp00ktober", "PPSSPP Project"]
1313
SPDX-License-Identifier = "GPL-2.0-or-later"
14+
15+
# the file-format library contains code derived from mhff; its sources also carry SPDX headers
16+
[[annotations]]
17+
path = "packages/mhp-formats/**"
18+
SPDX-FileCopyrightText = "2026 sp00ktober"
19+
SPDX-License-Identifier = "GPL-3.0-or-later"

‎packages/mhp-formats/README.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# mhp-formats
2+
3+
The file formats of Monster Hunter Freedom Unite and Monster Hunter Portable 3rd (PSP). Every
4+
format is a dataclass that reads with `from_bytes` and writes back with `to_bytes`, byte for byte
5+
on every file the games ship.
6+
7+
```python
8+
from mhp_formats import Pac, Pmo, Tmh, detect
9+
10+
pac = Pac.from_bytes(data) # a big monster: skeleton, model, textures, ...
11+
model = Pmo.from_bytes(pac.entries[1])
12+
positions = model.positions(0) # group 0, scaled to model units
13+
rgba = Tmh.from_bytes(pac.entries[2]).images[0].decode()
14+
assert pac.to_bytes() == data
15+
detect(pac.entries[0]) # -> Skeleton
16+
```
17+
18+
| Module | Format |
19+
|---|---|
20+
| `pac` | The container both games pack everything into |
21+
| `pmo` | Models, MHFU's 1.0 and MHP3rd's 102: meshes, materials, bone palettes, the geometry, and edits laid out through the one GE emitter |
22+
| `tmh` | Texture banks: every pixel and palette format the games use, DXT1 included |
23+
| `skeleton` | Skeletons of both games |
24+
| `anim`, `fu.anim`, `p3rd.anim` | Animation packs; only the clip encoding differs between the games |
25+
| `fu.stage` | MHFU stages: terrain, textures, environment and the collision grid |
26+
| `databin`, `iso` | DATA.BIN and getting a game's files out of its ISO |
27+
| `detect` | Which format a blob is |
28+
| `psp` | What the PSP itself defines: GE display lists, vertex types, colours, swizzle, strips |
29+
30+
Where the games differ, the difference lives in `mhp_formats.fu` (MHFU) or `mhp_formats.p3rd`
31+
(MHP3rd).
32+
33+
## Extracting a game
34+
35+
```bash
36+
uv run mhp-formats extract game.iso extracted/ # in this repository; elsewhere, the [iso] extra
37+
```
38+
39+
`extracted/data_files/file_NNNNN.bin` is the game's engine file NNNNN + 1. Only MHFU EU
40+
(ULES-01213) and MHP3rd (ULJM-05800) are recognised.
41+
42+
## Licence
43+
44+
GPL-3.0-or-later. The texture, vertex, display-list and model codecs are derived in part from
45+
[mhff](https://github.com/svanheulen/mhff) by Seth VanHeulen; DATA.BIN decryption uses his
46+
[mhef](https://github.com/svanheulen/mhef), through [a fork](https://github.com/MHFU-ModKit/mhef)
47+
that runs on Python 3.
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
[project]
2+
name = "mhp-formats"
3+
version = "0.1.0"
4+
description = "Monster Hunter Portable (PSP) file formats: archives, models, textures, skeletons, animations, stages."
5+
readme = "README.md"
6+
requires-python = ">=3.11"
7+
license = "GPL-3.0-or-later"
8+
dependencies = ["construct>=2.10.70", "construct-typing>=0.8"]
9+
classifiers = ["Typing :: Typed"]
10+
11+
[project.optional-dependencies]
12+
# mhef is not on PyPI, and upstream (archived) no longer runs on Python 3; the fork does
13+
iso = [
14+
"pycdlib>=1.14",
15+
"mhef @ git+https://github.com/MHFU-ModKit/mhef@0d0ba4802d4dfcf4f4960a8ebb7c69a8bf48780f",
16+
]
17+
18+
[project.scripts]
19+
mhp-formats = "mhp_formats.cli:main"
20+
21+
[build-system]
22+
requires = ["uv_build>=0.11,<0.12"]
23+
build-backend = "uv_build"
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# SPDX-License-Identifier: GPL-3.0-or-later
2+
# SPDX-FileCopyrightText: 2026 sp00ktober
3+
"""Monster Hunter Portable file formats (MHFU and MHP3rd), each one a dataclass that reads with
4+
`from_bytes` and writes back with `to_bytes`, byte for byte."""
5+
6+
from ._base import Format, FormatError
7+
from .anim import AnimPack, Channel, Clip, Keyframe, Track
8+
from .detect import detect
9+
from .pac import Pac
10+
from .pmo import Block, BoneSlot, Group, Material, Mesh, Pmo
11+
from .skeleton import Bone, Skeleton
12+
from .tmh import Tmh, TmhImage
13+
14+
__all__ = [
15+
"AnimPack",
16+
"Block",
17+
"Bone",
18+
"BoneSlot",
19+
"Channel",
20+
"Clip",
21+
"Format",
22+
"FormatError",
23+
"Group",
24+
"Keyframe",
25+
"Material",
26+
"Mesh",
27+
"Pac",
28+
"Pmo",
29+
"Skeleton",
30+
"Tmh",
31+
"TmhImage",
32+
"Track",
33+
"detect",
34+
]
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# SPDX-License-Identifier: GPL-3.0-or-later
2+
# SPDX-FileCopyrightText: 2026 sp00ktober
3+
"""The contract every format in the package keeps."""
4+
5+
from typing import Protocol, Self
6+
7+
8+
class FormatError(ValueError):
9+
"""Bytes that are not the format they were read as."""
10+
11+
12+
class Format(Protocol):
13+
"""A file format: `from_bytes(data).to_bytes() == data` for every file the games ship."""
14+
15+
@staticmethod
16+
def sniff(data: bytes) -> bool:
17+
"""A cheap look at the magic: True does not promise `from_bytes` succeeds."""
18+
...
19+
20+
@classmethod
21+
def from_bytes(cls, data: bytes) -> Self: ...
22+
23+
def to_bytes(self) -> bytes: ...
Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
# SPDX-License-Identifier: GPL-3.0-or-later
2+
# SPDX-FileCopyrightText: 2026 sp00ktober
3+
"""Animation packs of both games: streams of slot tables, each slot a clip or empty, each clip one
4+
keyframe track per bone. Only the clip encoding differs between the games; it lives in
5+
`mhp_formats.fu.anim` and `mhp_formats.p3rd.anim`.
6+
7+
MHFU splits one rig across its streams (a big monster's body, head and tail play the same slot
8+
together); MHP3rd's streams are separate clip sets over the whole rig.
9+
"""
10+
11+
import math
12+
import struct
13+
from collections.abc import Callable
14+
from dataclasses import dataclass, field
15+
from functools import partial
16+
from itertools import chain
17+
from typing import Literal, NamedTuple, Self
18+
19+
from construct import Array, Int32ul
20+
21+
from ._base import FormatError
22+
23+
EMPTY = 0xFFFFFFFF
24+
"""A slot without a clip."""
25+
26+
Kind = Literal["rot", "loc", "scl"]
27+
28+
CHANNEL_BITS: dict[int, tuple[Kind, int]] = {
29+
0x008: ("rot", 0),
30+
0x010: ("rot", 1),
31+
0x020: ("rot", 2),
32+
0x040: ("loc", 0),
33+
0x080: ("loc", 1),
34+
0x100: ("loc", 2),
35+
0x200: ("scl", 0),
36+
0x400: ("scl", 1),
37+
0x800: ("scl", 2),
38+
}
39+
"""Channel bit -> (kind, axis). Bits 0x1/0x2/0x4 also occur, with values near 16; their meaning
40+
is not known. MHFU's engine crashes on the scale bits."""
41+
42+
_STEP: dict[Kind, float] = {"rot": math.pi / 8192, "loc": 1 / 16, "scl": 1 / 256}
43+
"""What one raw unit is worth: 4096 = 90 degrees, 16 = 1.0, 256 = 1.0."""
44+
45+
_MAX_HEADER = 0x800
46+
47+
48+
class Keyframe(NamedTuple):
49+
"""Raw s16 words; `value` goes through `dequantize`."""
50+
51+
value: int
52+
frame: int
53+
ease_in: int = 0
54+
ease_out: int = 0
55+
56+
57+
@dataclass
58+
class Channel:
59+
"""One animated component of a bone."""
60+
61+
bit: int
62+
keyframes: list[Keyframe] = field(default_factory=list)
63+
64+
65+
@dataclass
66+
class Track:
67+
"""The channels of one bone."""
68+
69+
channels: list[Channel] = field(default_factory=list)
70+
71+
@property
72+
def mask(self) -> int:
73+
mask = 0
74+
for channel in self.channels:
75+
mask |= channel.bit
76+
return mask
77+
78+
79+
@dataclass
80+
class Clip:
81+
tracks: list[Track] = field(default_factory=list)
82+
loop: int = 0
83+
loop_start: float = 0.0
84+
85+
86+
Stream = list[Clip | None]
87+
88+
89+
def channel_kind(bit: int) -> tuple[Kind, int] | None:
90+
"""`(kind, axis)` of a channel bit, None for the bits nobody has named."""
91+
return CHANNEL_BITS.get(bit)
92+
93+
94+
def dequantize(kind: Kind, raw: int) -> float:
95+
"""Rotation comes out in radians."""
96+
return raw * _STEP[kind]
97+
98+
99+
def quantize(kind: Kind, value: float) -> int:
100+
"""Rounds to nearest and clamps to s16."""
101+
return max(-0x8000, min(0x7FFF, round(value / _STEP[kind])))
102+
103+
104+
# Keyframes and the section headers around them are packed with `struct`, not construct: a pack
105+
# holds up to ~200k keyframes, and construct parses one 8-byte keyframe in ~4 us against ~0.1 us.
106+
_KEY = struct.Struct("<4h")
107+
_to_keyframe = partial(tuple.__new__, Keyframe)
108+
109+
110+
def read_keyframes(view: memoryview) -> list[Keyframe]:
111+
return list(map(_to_keyframe, _KEY.iter_unpack(view)))
112+
113+
114+
def write_keyframes(keyframes: list[Keyframe]) -> bytes:
115+
try:
116+
return struct.pack(f"<{4 * len(keyframes)}h", *chain.from_iterable(keyframes))
117+
except struct.error as e:
118+
raise ValueError(f"a keyframe does not fit s16: {e}") from None
119+
120+
121+
@dataclass
122+
class AnimPack:
123+
"""Header: N x (u32 slot count, u32 table offset), u32 0, u32 end of the tables; then the
124+
tables, then the clips back to back in the order the slots first use them. Slots holding the
125+
same `Clip` object share one copy in the file."""
126+
127+
streams: list[Stream] = field(default_factory=list)
128+
tail: bytes = b""
129+
"""Bytes after the last clip (a standalone pack pads to 0x800)."""
130+
131+
@classmethod
132+
def _read_clip(cls, data: memoryview, off: int) -> tuple[Clip, int]:
133+
"""The clip at `off` and where it ends."""
134+
raise NotImplementedError
135+
136+
@staticmethod
137+
def _write_clip(clip: Clip, out: bytearray) -> None:
138+
raise NotImplementedError
139+
140+
@classmethod
141+
def from_bytes(cls, data: bytes) -> Self:
142+
layout = _layout(data)
143+
if layout is None:
144+
raise FormatError("not an animation pack: the header does not describe its tables")
145+
tables, cursor = layout
146+
view = memoryview(data)
147+
clips: dict[int, Clip] = {}
148+
streams: list[Stream] = []
149+
for table in tables:
150+
stream: Stream = []
151+
for off in table:
152+
clip = clips.get(off)
153+
if clip is None and off != EMPTY:
154+
if off != cursor:
155+
raise FormatError(f"clip at {off:#x}, expected the next one at {cursor:#x}")
156+
try:
157+
clip, cursor = cls._read_clip(view, off)
158+
except struct.error as e:
159+
raise FormatError(f"clip at {off:#x} runs past the data") from e
160+
clips[off] = clip
161+
stream.append(clip)
162+
streams.append(stream)
163+
return cls(streams, data[cursor:])
164+
165+
def to_bytes(self) -> bytes:
166+
if not self.streams:
167+
raise ValueError("an animation pack has at least one stream")
168+
header = []
169+
run = 8 * (len(self.streams) + 1)
170+
for stream in self.streams:
171+
header += [len(stream), run]
172+
run += 4 * len(stream)
173+
header += [0, run]
174+
out = bytearray(run)
175+
placed: dict[int, int] = {}
176+
for stream in self.streams:
177+
for clip in stream:
178+
if clip is not None and id(clip) not in placed:
179+
placed[id(clip)] = len(out)
180+
self._write_clip(clip, out)
181+
for stream in self.streams:
182+
header += [EMPTY if clip is None else placed[id(clip)] for clip in stream]
183+
out[:run] = Array(len(header), Int32ul).build(header)
184+
return bytes(out) + self.tail
185+
186+
187+
def sniff_pack(data: bytes, is_clip: Callable[[int], bool]) -> bool:
188+
"""A pack header whose first clip starts with a word `is_clip` accepts."""
189+
layout = _layout(data)
190+
if layout is None:
191+
return False
192+
tables, start = layout
193+
if all(off == EMPTY for table in tables for off in table):
194+
return True
195+
return len(data) >= start + 4 and is_clip(int.from_bytes(data[start : start + 4], "little"))
196+
197+
198+
def _layout(data: bytes) -> tuple[list[list[int]], int] | None:
199+
"""The slot tables and where the clips start, or None if the header does not chain."""
200+
if len(data) < 16:
201+
return None
202+
hsize = int.from_bytes(data[4:8], "little")
203+
if not 16 <= hsize <= min(len(data), _MAX_HEADER) or hsize % 8:
204+
return None
205+
words = Array(hsize // 4, Int32ul).parse(data)
206+
run = hsize
207+
for i in range(0, hsize // 4 - 2, 2):
208+
if words[i + 1] != run:
209+
return None
210+
run += 4 * words[i]
211+
if words[-2] != 0 or words[-1] != run or run > len(data):
212+
return None
213+
flat = Array((run - hsize) // 4, Int32ul).parse(data[hsize:run])
214+
tables = []
215+
for i in range(0, hsize // 4 - 2, 2):
216+
start = (words[i + 1] - hsize) // 4
217+
tables.append(list(flat[start : start + words[i]]))
218+
return tables, run
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# SPDX-License-Identifier: GPL-3.0-or-later
2+
# SPDX-FileCopyrightText: 2026 sp00ktober
3+
"""`mhp-formats`, the command line."""
4+
5+
import argparse
6+
import sys
7+
from collections.abc import Sequence
8+
from pathlib import Path
9+
10+
from . import iso
11+
from ._base import FormatError
12+
13+
14+
def main(argv: Sequence[str] | None = None) -> int:
15+
parser = argparse.ArgumentParser(prog="mhp-formats", description=__doc__)
16+
commands = parser.add_subparsers(required=True, metavar="COMMAND")
17+
extract = commands.add_parser(
18+
"extract",
19+
help="copy a game's files out of its ISO, DATA.BIN decrypted and split",
20+
description="Copy the ISO's files into OUT, and DATA.BIN decrypted into OUT/data_files, "
21+
"one file per engine file id: file_NNNNN is id NNNNN + 1.",
22+
)
23+
extract.add_argument("iso", type=Path, help="the game's UMD image")
24+
extract.add_argument("out", type=Path, help="directory to write into")
25+
extract.set_defaults(run=_extract)
26+
args = parser.parse_args(argv)
27+
try:
28+
args.run(args)
29+
except (FormatError, ImportError, OSError) as e:
30+
parser.exit(1, f"mhp-formats: {e}\n")
31+
return 0
32+
33+
34+
def _extract(args: argparse.Namespace) -> None:
35+
game = iso.extract(args.iso, args.out, lambda step: print(step, file=sys.stderr))
36+
print(f"{game.name} ({game.value}) extracted to {args.out}")

0 commit comments

Comments
 (0)