chore: add memory index and corpus gap analysis documentation

Author: FlavioCFOliveira

.claude/agent-memory/image-metadata-auditor/MEMORY.md

@@ -0,0 +1,4 @@
+# Memory Index
+
+- [Test corpus sources](reference_test_corpus.md) — 12 public repos/sources; confirmed file lists, download URLs, and edge-case coverage per source
+- [Corpus gap analysis](project_corpus_gaps.md) — Identified gaps resolved in download.sh: progressive JPEG, XMP sidecar, multi-page TIFF, BigTIFF endianness, exotic RAW formats, MakerNote variants, XMP-only

.claude/agent-memory/image-metadata-auditor/project_corpus_gaps.md

@@ -0,0 +1,107 @@
+---
+name: Corpus gap analysis
+description: Gaps identified in testdata/corpus as of 2026-04-03 audit, and how each was resolved in download.sh
+type: project
+---
+
+# Test Corpus Gap Analysis (2026-04-03)
+
+## Corpus state before this audit
+1,726 files across jpeg/tiff/png/heif/webp/raw. No xmp/ directory.
+
+## Gaps identified and resolutions
+
+### 1. Progressive JPEG — RESOLVED
+**Gap**: No progressive JPEG test files in corpus.
+**Resolution**: SOURCE 9 (sindresorhus/is-progressive) added. Provides both progressive and baseline variants with MIT license.
+**Files**: kitten-progressive.jpg, progressive.jpg, baseline.jpg, kitten.jpg, curious-exif.jpg
+**Destination**: corpus/jpeg/progressive/
+
+### 2. XMP-only JPEG — RESOLVED
+**Gap**: No JPEG with XMP but no EXIF was in corpus (needed to test XMP-first parsing path).
+**Resolution**: SOURCE 8 (exiftool) XMP.jpg (XMP-only, 10KB). SOURCE 1 (ianare) jpg/xmp/no_exif.jpg.
+**Files**: corpus/jpeg/exiftool/XMP.jpg, corpus/jpeg/exif-samples/no_exif.jpg
+
+### 3. EXIF+IPTC+XMP combined — RESOLVED
+**Gap**: No single image confirmed to carry all three metadata standards simultaneously.
+**Resolution**: SOURCE 8 (exiftool) ExifTool.jpg (confirmed EXIF+IPTC+XMP, 26KB). MWG.jpg also confirmed.
+**Files**: corpus/jpeg/exiftool/ExifTool.jpg, corpus/jpeg/exiftool/MWG.jpg
+
+### 4. No-metadata JPEG — RESOLVED
+**Gap**: Unknown.jpg from ExifTool test suite was not in corpus.
+**Resolution**: SOURCE 8 now includes Unknown.jpg (7.2 KB, no recognized metadata).
+**Files**: corpus/jpeg/exiftool/Unknown.jpg
+
+### 5. Extended/multi-packet XMP — RESOLVED
+**Gap**: No file testing the case where XMP is split across multiple APP1 segments.
+**Resolution**: SOURCE 8 ExtendedXMP.jpg (1.4KB, multi-packet XMP).
+**Files**: corpus/jpeg/exiftool/ExtendedXMP.jpg
+
+### 6. BigTIFF big-endian (Motorola byte order) — CONFIRMED PRESENT
+**Status**: BigTIFFMotorola.tif and BigTIFFMotorolaLongStrips.tif already in corpus via SOURCE 2.
+**Files**: corpus/tiff/metadata-extractor/BigTIFFMotorola.tif
+
+### 7. Multi-page TIFF — CONFIRMED PRESENT
+**Status**: multipage.tif already in corpus via SOURCE 2.
+**Files**: corpus/tiff/metadata-extractor/multipage.tif
+
+### 8. Fuji RAF — RESOLVED
+**Gap**: No RAF files in corpus (copy_ext in old download.sh did not include raf extension).
+**Resolution**: SOURCE 2 now copies raf extension; SOURCE 8 includes FujiFilm.raf.
+**Files**: corpus/raw/metadata-extractor/*.raf, corpus/raw/exiftool/FujiFilm.raf
+
+### 9. Pentax PEF — RESOLVED
+**Gap**: PEF format not collected.
+**Resolution**: SOURCE 2 now copies pef extension.
+**Files**: corpus/raw/metadata-extractor/Pentax K-1 Mark II.pef
+
+### 10. Sigma X3F — RESOLVED
+**Gap**: X3F format not collected.
+**Resolution**: SOURCE 2 now copies x3f extension. SOURCE 8 includes Sigma.x3f.
+**Files**: corpus/raw/metadata-extractor/*.x3f, corpus/raw/exiftool/Sigma.x3f
+
+### 11. Minolta MRW — RESOLVED
+**Gap**: MRW format not collected.
+**Resolution**: SOURCE 8 includes Minolta.mrw.
+**Files**: corpus/raw/exiftool/Minolta.mrw
+
+### 12. Canon CRW (legacy) — RESOLVED
+**Gap**: CRW (pre-CR2 Canon format) not collected.
+**Resolution**: SOURCE 2 now copies crw extension. SOURCE 8 includes CanonRaw.crw.
+**Files**: corpus/raw/exiftool/CanonRaw.crw
+
+### 13. Phase One IIQ — RESOLVED
+**Gap**: No medium-format RAW coverage.
+**Resolution**: SOURCE 8 includes PhaseOne.iiq.
+**Files**: corpus/raw/exiftool/PhaseOne.iiq
+
+### 14. XMP sidecar files — RESOLVED
+**Gap**: No .xmp sidecar files were collected (only embedded XMP).
+**Resolution**: xmp/ corpus directory created. SOURCE 2 collects xmp/ directory. SOURCE 6 collects BlueSquare.xmp, StaffPhotographer-Example.xmp. SOURCE 8 collects PLUS.xmp.
+**Files**: corpus/xmp/
+
+### 15. PNG with EXIF / PNG with no metadata / PNG with tEXt — RESOLVED
+**Gap**: No explicit minimal PNG test cases.
+**Resolution**: SOURCE 6 (Exiv2) adds 1343_exif.png, 1343_comment.png, 1343_empty.png.
+**Files**: corpus/png/exiv2/*.png
+
+### 16. MakerNote variants (Olympus 5 variants, Pentax 3 variants) — RESOLVED
+**Gap**: MakerNote variant testing was limited.
+**Resolution**: SOURCE 4 (libexif/libexif test/testdata) adds 9 explicit MakerNote variant JPEGs.
+**Files**: corpus/jpeg/libexif-makernotes/
+
+### 17. IPTC backward compatibility (older standard versions) — RESOLVED
+**Gap**: Only Std2024.1 and Std2021.1 were downloaded; Core1.3 failed (404).
+**Resolution**: Download now attempts Std2023.2, Std2019.1, Std2017.1, Std2016, Std2014 in addition to Core1.3.
+**Note**: Core1.3 URL may still 404 — verify manually.
+
+### 18. TIFF structural variants (planar, tiled, multi-page, compressed) — RESOLVED
+**Gap**: Existing TIFF corpus was mostly camera-metadata focused, not TIFF-structure focused.
+**Resolution**: SOURCE 10 (tlnagy/exampletiffs) adds mri.tif (multi-page), compression variants, planar, tiled.
+**Files**: corpus/tiff/exampletiffs/
+
+## Remaining confirmed gaps
+- **Hasselblad 3FR/FFF**: No source found with freely licensed files
+- **Phase One RAW beyond IIQ**: Limited to ExifTool's single sample
+- **Unicode in JPEG metadata body**: 46_UnicodeEncodeError.jpg from ianare exists but tests encoder error, not embedded UTF-8 characters in field values — a hand-crafted test file would be more precise
+- **Images with very large metadata blocks (>64KB XMP)**: Not confirmed present in any source; would need synthetic generation

.claude/agent-memory/image-metadata-auditor/reference_test_corpus.md

@@ -0,0 +1,130 @@
+---
+name: Test corpus sources
+description: 12 public repos/sources for image metadata test images; confirmed file lists, download URLs, and edge-case coverage per source as of 2026-04-03
+type: reference
+---
+
+# Image Metadata Test Corpus Sources
+
+All sources are incorporated in testdata/download.sh.
+
+## SOURCE 1: ianare/exif-samples
+**URL**: https://github.com/ianare/exif-samples
+**License**: Not explicitly stated (public test images)
+**Key content**:
+- jpg/tests/: 13 edge-case JPEGs (type errors, GPS IFD, Unicode errors, zero-length strings, memory errors, IndexError)
+- jpg/invalid/: 7 malformed JPEG files (image00971.jpg through image02206.jpg)
+- jpg/gps/: 9 Nikon DSCN GPS-tagged JPEGs
+- jpg/orientation/: 16 orientation-tagged JPEGs (landscape_1-8, portrait_1-8)
+- jpg/xmp/: no_exif.jpg (XMP-only, no EXIF), BlueSquare.jpg
+- tiff/: 8 TIFF files including Crémieux11.tiff (Unicode filename test)
+- heic/, heic/mobile/: iPhone 13 Pro Max, Nokia 8.3 5G HEIC/HEIF
+
+## SOURCE 2: drewnoakes/metadata-extractor-images
+**URL**: https://github.com/drewnoakes/metadata-extractor-images
+**License**: Varies per file (community contributed)
+**Key content** (CONFIRMED via API):
+- tif/BigTIFF/BigTIFF.tif: BigTIFF Intel/LE
+- tif/BigTIFF/BigTIFFMotorola.tif: BigTIFF Motorola/BE (explicit big-endian test)
+- tif/BigTIFF/BigTIFFSubIFD4.tif, BigTIFFSubIFD8.tif: SubIFD traversal tests
+- tif/multipage.tif: multi-page TIFF (1.5 MB)
+- tif/ImageTestSuite/: ~45 hash-named TIFF edge cases
+- png/PngSuite/: ~175 official PNG test suite files
+- png/photoshop-8x12-rgb24-all-metadata.png: EXIF+IPTC+XMP in PNG
+- png/sampleWithExifData.png: EXIF in PNG
+- png/photoshop-8x12-rgb24-interlaced.png, rgba32-interlaced.png: interlaced PNG
+- png/invalid-iCCP-missing-adler32-checksum.png: corrupt PNG chunk
+- webp/: 15 WebP files covering VP8, VP8L, VP8X, lossless, lossy, animated
+- raf/: FujiFilm EX-2, FinePix S5500, X-E1
+- x3f/: Sigma DP1 Quattro, DP2 Quattro
+- pef/: Pentax K-1 Mark II
+- xmp/: 4 sidecar XMP files (digiKam, jPhotoTagger, ExifTool 9.74, APhotoManager)
+
+## SOURCE 3: libexif/libexif-testsuite
+**URL**: https://github.com/libexif/libexif-testsuite
+**Images/**: 4 files: canon-eos-m6mark2.jpg, canon-powershot-g2.jpg, nikon-z6.jpg, samsung-s9.jpg
+
+## SOURCE 4: libexif/libexif test/testdata
+**URL**: https://github.com/libexif/libexif (test/testdata path)
+**Key content**: 9 MakerNote variant JPEGs with .parsed golden files
+- canon_makernote_variant_1.jpg
+- fuji_makernote_variant_1.jpg
+- olympus_makernote_variant_2.jpg through _5.jpg (4 variants)
+- pentax_makernote_variant_2.jpg through _4.jpg (3 variants)
+**Download pattern**: https://raw.githubusercontent.com/libexif/libexif/master/test/testdata/{filename}
+
+## SOURCE 5: IPTC reference images (iptc.org)
+**URL**: https://iptc.org/standards/photo-metadata/reference-images/
+**License**: Official IPTC standard reference images (freely distributable for testing)
+**Available versions** (CONFIRMED via web):
+- Std2024.1: https://iptc.org/std/photometadata/examples/IPTC-PhotometadataRef-Std2024.1.jpg
+- Std2023.2: https://iptc.org/std/photometadata/examples/IPTC-PhotometadataRef-Std2023.2.jpg
+- Std2021.1: https://www.iptc.org/std/photometadata/examples/IPTC-PhotometadataRef-Std2021.1.jpg
+- Std2019.1: https://www.iptc.org/std/photometadata/examples/IPTC-PhotometadataRef-Std2019.1.jpg
+- Std2017.1: https://www.iptc.org/std/photometadata/examples/IPTC-PhotometadataRef-Std2017.1.jpg
+- Core1.3: https://www.iptc.org/std/photometadata/examples/IPTC-PhotometadataRef-Core1.3.jpg
+- Std2016: https://iptc.org/std/photometadata/examples/IPTC-PhotometadataRef-Std2016_large.jpg
+- Std2014: https://iptc.org/std/photometadata/examples/IPTC-PhotometadataRef-Std2014_large.jpg
+**Note**: Core1.3 failed to download in prior runs (404 possible). Each image has every field populated with its own field name as value.
+
+## SOURCE 6: Exiv2/exiv2 test/data
+**URL**: https://github.com/Exiv2/exiv2 (test/data path)
+**Key content** (CONFIRMED via API):
+- HEIC: Stonehenge.heic, IMG_3578.heic, 2021-02-13-1929.heic
+- PNG: 1343_comment.png (tEXt chunk), 1343_exif.png (EXIF in PNG), 1343_empty.png (no metadata)
+- XMP sidecars: BlueSquare.xmp, StaffPhotographer-Example.xmp
+- TIFF: Reagan.tiff, ReaganLargeTiff.tiff
+- Multiple *_poc files: fuzzing/security inputs
+- exiv2-fujifilm-finepix-s2pro.jpg: Fuji MakerNote
+- exiv2-sigma-d10.jpg: Sigma MakerNote
+- Canon-R6-pruned.CR3: Canon CR3
+
+## SOURCE 7: adobe/XMP-Toolkit-SDK (samples/testfiles)
+**URL**: https://github.com/adobe/XMP-Toolkit-SDK
+**Key content**: XMP-heavy JPEGs and TIFFs from Adobe's reference XMP implementation
+**Note**: Only 6 JPEGs and 2 TIFFs at time of audit — small but authoritative
+
+## SOURCE 8: exiftool/exiftool (t/images)
+**URL**: https://github.com/exiftool/exiftool (t/images path)
+**License**: Artistic License (Perl standard)
+**Key files** (all CONFIRMED via API, sizes in bytes):
+- ExifTool.jpg (26,106): EXIF + IPTC + XMP combined
+- MWG.jpg (2,255): Metadata Working Group compliance
+- GPS.jpg (2,133): GPS IFD with coordinates
+- IPTC.jpg (9,851): IPTC-focused JPEG
+- XMP.jpg (10,314): XMP-only JPEG (key gap filler — no EXIF)
+- ExtendedXMP.jpg (1,380): multi-packet extended XMP
+- Unknown.jpg (7,199): JPEG with no recognized metadata
+- ExifTool.tif (4,864): TIFF with EXIF+IPTC+XMP
+- GeoTiff.tif (2,660): GeoTIFF with projection metadata
+- BigTIFF.btf (384): BigTIFF format (minimal)
+- FujiFilm.raf (38,452): Fuji RAF raw
+- PhaseOne.iiq (3,364): Phase One IIQ medium format
+- CanonRaw.crw (not sized): Canon CRW legacy format
+- Minolta.mrw (2,596): Minolta MRW raw
+- Sigma.x3f (1,464): Sigma X3F raw
+- PLUS.xmp (4,763): PLUS licensing XMP sidecar
+
+## SOURCE 9: sindresorhus/is-progressive
+**URL**: https://github.com/sindresorhus/is-progressive
+**License**: MIT (confirmed — license file present in repo)
+**Key files** (CONFIRMED via API):
+- kitten-progressive.jpg: progressive JPEG encoding
+- progressive.jpg: progressive JPEG encoding
+- baseline.jpg: baseline JPEG encoding
+- kitten.jpg: baseline JPEG encoding
+- curious-exif.jpg: EXIF data in baseline JPEG
+**Download base**: https://raw.githubusercontent.com/sindresorhus/is-progressive/main/fixture/
+
+## SOURCE 10: tlnagy/exampletiffs
+**URL**: https://github.com/tlnagy/exampletiffs
+**Key files** (CONFIRMED via API, sizes):
+- mri.tif (230,578): multi-page TIFF (MRI slices)
+- shapes_tiled_multi.tif (37,604): tiled multi-page TIFF
+- shapes_lzw.tif (11,530): LZW compressed TIFF
+- shapes_deflate.tif: deflate compressed TIFF
+- shapes_uncompressed.tif: uncompressed TIFF
+- shapes_lzw_planar.tif: planar storage TIFF
+- shapes_lzw_palette.tif: palette TIFF
+- 4D-series.ome.tif: OME-TIFF with XML in ImageDescription
+**Download base**: https://raw.githubusercontent.com/tlnagy/exampletiffs/master/

.claude/agent-memory/xmp-metadata-expert/MEMORY.md

@@ -0,0 +1,3 @@
+# Memory Index
+
+- [XMP/IPTC Spec Audit — 2026-04-03](project_xmp_iptc_audit.md) — XMP 7/10, IPTC 7.5/10; specific gaps identified in struct serialization, array-of-structs parsing, IRB write, charset handling

.claude/agent-memory/xmp-metadata-expert/project_xmp_iptc_audit.md

@@ -0,0 +1,61 @@
+---
+name: XMP/IPTC Spec Audit — 2026-04-03
+description: Detailed spec compliance audit of xmp/ and iptc/ packages; scores and specific defects documented
+type: project
+---
+
+Audit conducted against XMP ISO 16684-1 and IPTC IIM 4.2.
+
+## XMP Score: 7/10
+
+### Confirmed correct
+- Packet scanning: `<?xpacket begin=` + `<?xpacket end=` markers; no premature `?>` termination
+- BOM embedded as attribute value in `begin=` field
+- `id="W5M0MpCehiHzreSzNTczkc9d"` magic ID present
+- `rdf:Alt/Seq/Bag` with `rdf:li` accumulation
+- `xml:lang` on `rdf:li` stored as `lang|value`; x-default stored without prefix
+- `rdf:parseType="Resource"` → struct handling
+- `rdf:resource` shorthand attribute
+- Inline attribute shorthand on `rdf:Description`
+- All core namespaces correct (xmp, xmpRights, xmpMM, dc, photoshop, exif, tiff, aux, Iptc4xmpCore, Iptc4xmpExt)
+- `xml.EscapeText` used for all character data in write
+- 2 KB whitespace padding + `<?xpacket end="w"?>`
+- Deterministic output via sort
+
+### Defects / Gaps
+1. **No UTF-16 packet support** — XMP §7.2 allows UTF-16/UTF-32; `bytes.Index` on UTF-8 literals fails
+2. **`begin=` attribute value not validated** — accepts malformed PIs
+3. **Array-of-structs not parsed** — `rdf:li` containing nested `rdf:Description` (e.g., xmpMM:History) silently dropped
+4. **`xml:lang` not namespace-qualified** in rdf:li scanning (minor)
+5. **Struct fields cross-namespace key attribution** — stored under field NS not parent NS
+6. **Struct properties not serialized correctly** — `propLocal.fieldLocal` emitted as dotted XML element name (invalid)
+7. **`collectionType` incomplete** — xmpMM:History/Ingredients should be Seq, not Bag
+8. **Missing type-support namespaces**: stEvt, stRef, stDim, stVer, xmpBJ, xmpTPg, xmpDM, xmpG, xmpGImg
+9. **`rdf:about` not validated** — spec requires consistent subject URI across all rdf:Description blocks
+
+## IPTC Score: 7.5/10
+
+### Confirmed correct
+- Tag marker 0x1C scanning
+- 5-byte header: marker + record + dataset + 2-byte size
+- Standard 2-byte length (big-endian)
+- Extended length: `0x80 | nBytes` in high byte; subsequent nBytes carry actual length
+- IRB: 8BIM marker, resource ID, Pascal string name + even-padding, 4-byte size, data even-padding
+- Resource ID 0x0404 for IPTC-NAA
+- UTF-8 declaration: ESC % G (0x1B 0x25 0x47) in 1:90
+- ISO-8859-1 fallback via golang.org/x/text charmap
+- UTF-8 declaration re-emitted on encode when originally present
+- Dataset registry: good coverage of photographic datasets (5, 25, 55, 60, 80, 105, 116, 120, etc.)
+
+### Defects / Gaps
+1. **Destructive IRB write** — buildIRB emits only 0x0404; all other resource blocks (ICC, thumbnail, XMP, etc.) discarded
+2. **Multi-segment APP13 not supported** — large IPTC can span multiple APP13 segments; only first parsed
+3. **No auto UTF-8 declaration on write** — SetCaption/SetCopyright with non-ASCII values written without 1:90 dataset injection
+4. **`isUTF8Declaration` len==3 too strict** — some tools append NUL padding; HasPrefix would be more robust
+5. **No ISO 2022 charset diversity** — non-UTF-8, non-Latin-1 declared charsets (e.g., Shift-JIS) produce garbled output
+6. **Malformed data silently truncates** — break on integrity failure drops all subsequent datasets without error
+7. **Missing IIM datasets**: 2:22, 2:26-2:31, 2:35-2:38, 2:42, 2:45-2:47, 2:75 (agency workflow fields)
+8. **Missing convenience accessors**: ObjectName (2:05), DateCreated (2:55), Headline (2:105), City/ProvinceState/CountryName
+
+**Why:** Research-only audit for library gap assessment.
+**How to apply:** Use these findings to prioritize remediation work; P0 is struct serialization (XMP) and destructive IRB write (IPTC).