fix: preserve XLSX workbook fidelity

Author: rastagan-git

.agents/skills/curate-aroma-data/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: curate-aroma-data
-description: Use AromaNexus to validate, normalize, enrich, and export flavor, odor, aroma, and chemical-sensory datasets with source-level provenance. Use for XLSX, CSV, or TSV workbooks containing CAS numbers, compound names, calculated retention indices, sensory descriptors, odor thresholds, or olfactory-receptor evidence; for selecting among NIST, PubChem, Pyrfume, M2OR, MFFI, and the permission-gated ChemicalBook compatibility connector; and for verifying analysis-ready outputs. Do not use to bypass access controls, relicense source data, or make biomedical or machine-learning claims unsupported by the exported evidence.
+description: Use AromaNexus to validate, normalize, enrich, and export flavor, odor, aroma, and chemical-sensory datasets with source-level provenance. Use for XLSX, CSV, or TSV files containing CAS numbers, compound names, calculated retention indices, sensory descriptors, odor thresholds, or olfactory-receptor evidence; for selecting among NIST, PubChem, Pyrfume, M2OR, MFFI, and the permission-gated ChemicalBook compatibility connector; and for verifying analysis-ready outputs. Do not use to bypass access controls, relicense source data, or make biomedical or machine-learning claims unsupported by the exported evidence.
 ---
 
 # Curate Aroma Data
@@ -10,8 +10,9 @@ Build traceable compound tables through the repository's `aromanexus` CLI. Keep
 ## Workflow
 
 1. Inspect the input without modifying it.
-   - Run `python .agents/skills/curate-aroma-data/scripts/inspect_workbook.py INPUT` from the repository root.
-   - Confirm the row count, exact column names, identifier quality, duplicates, and formula-like cells.
+   - For XLSX, run `python .agents/skills/curate-aroma-data/scripts/inspect_workbook.py INPUT --sheet "SHEET"` from the repository root. Omit `--sheet` only when the first worksheet is the intended target.
+   - For CSV or TSV, run the inspector without `--sheet`; flat files have no worksheets.
+   - Confirm the worksheet order and exact target when applicable, row count, exact column names, identifier quality, duplicates, source formulas, workbook properties, per-sheet content digests, styles, dimensions, and reported features.
    - Identify section labels, headers, totals, and other structural rows before provider calls. Define an explicit dataset-specific skip rule; do not assume that text such as `C6` is globally non-chemical.
 2. Choose the smallest provider set that supplies the requested fields.
    - Read [references/provider-matrix.md](references/provider-matrix.md) before any network or browser operation.
@@ -20,8 +21,9 @@ Build traceable compound tables through the repository's `aromanexus` CLI. Keep
    - Use Pyrfume only for explicitly selected archives after reviewing each manifest note.
    - Use M2OR only when receptor bioassay evidence is relevant; label species and assay scope.
 3. Preview the operation.
-   - State the input, new output path, selected provider, expected columns, skip patterns, approximate request count, cache behavior, and material access caveats.
-   - Write a sibling output by default. Do not pass `--force` or overwrite the input unless the user explicitly requests that exact replacement.
+   - State the input, selected worksheet for XLSX, new output path, selected provider, expected columns, skip patterns, approximate request count, cache behavior, and material access caveats.
+   - Write a sibling output by default. Never reuse the input path as the output path; `--force` is only for a separate existing destination.
+   - Keep XLSX input and output when worksheet formulas, formatting, or other workbook content must survive; CSV/TSV output is a flat export.
 4. Run one focused command.
    - Identity and odor metadata: `aromanexus pubchem INPUT --identifier-column "CAS Number"`
    - Name lookup with dataset-specific structural rows: `aromanexus pubchem INPUT --identifier-column "Name" --skip-pattern '^C\d+$'`
@@ -30,9 +32,11 @@ Build traceable compound tables through the repository's `aromanexus` CLI. Keep
    - Curated descriptors: `aromanexus pyrfume INPUT --archives aromadb,superscent`
    - Receptor evidence: `aromanexus m2or INPUT --cas-column "CAS Number"`
    - Source inventory: `aromanexus sources`
+   - For XLSX only, append `--sheet "SHEET"` to any table command when the target is not the first worksheet. Never pass `--sheet` for CSV or TSV.
 5. Verify the result.
-   - Re-run the inspection script on the output.
+   - Re-run the inspection script on the same worksheet for XLSX, or without `--sheet` for CSV/TSV.
    - Confirm identical row order and row count, expected new fields, typed status counts, source URL, retrieval time, version, and license/access fields.
+   - For XLSX output, also compare worksheet order and names, per-sheet content digests, workbook properties, non-target-sheet content, untargeted source formulas and cached values, styles, dimensions, and reported workbook features.
    - Treat `PubChem Status` as provider execution state, not proof of a uniquely resolved CAS. Check `PubChem CAS Resolution`, candidate count, and `Resolved CAS` separately.
    - Leave `multiple` and `missing` CAS resolutions unresolved; retain all candidates and route only the affected rows to a targeted fallback source or manual review.
    - Treat `http_error`, `network_error`, `parse_error`, `missing_data`, `data_error`, `partial`, `blocked`, and `skipped` separately from `not_found`.
@@ -46,5 +50,7 @@ Build traceable compound tables through the repository's `aromanexus` CLI. Keep
 - Do not run ChemicalBook automation unless the user confirms documented permission; its current robots policy excludes the legacy routes.
 - Do not describe the toolkit as AI-powered. Say that structured exports can support downstream statistics, cheminformatics, or machine-learning experiments.
 - Do not bundle or republish downloaded Pyrfume or M2OR data in the repository.
-- Preserve remote text as literal spreadsheet cells to prevent formula execution.
+- Preserve newly fetched remote text as literal spreadsheet cells to prevent formula execution. Preserve legitimate source formulas outside cells explicitly targeted by an output field.
+- Preserve merged cells outside the selected tabular rectangle; stop before provider calls when a merge intersects that rectangle.
+- Stop before provider calls if XLSX preflight reports a known unsafe feature or any OOXML package part that the in-memory trial write would discard.
 - Never select the first PubChem CAS candidate merely because the provider status is `ok`.

.agents/skills/curate-aroma-data/references/output-schema.md

@@ -58,3 +58,9 @@ After every run, confirm:
 5. Remote strings beginning with `=`, `+`, `-`, or `@` are stored as literal text.
 6. Partial outputs are reported separately if a run is interrupted.
 7. `multiple`, `missing`, `not_evaluated`, and `skipped` PubChem CAS resolutions never contain an automatic `Resolved CAS`.
+8. XLSX worksheet order and names match the input, and supported non-target worksheet content and features are unchanged.
+9. Source formulas and cached results outside explicitly targeted output cells, plus styles, dimensions, freeze panes, filters, tables, data validation, conditional formatting, and workbook properties, remain present where applicable.
+10. Merged cells outside the selected tabular rectangle remain present; a merge intersecting that rectangle is rejected before provider access.
+11. The target worksheet used for post-run inspection is the same exact worksheet selected for enrichment, and per-sheet content digests are compared.
+
+For XLSX input, the default target is the first worksheet in workbook order. Pass `--sheet "Name"` to both the command and inspection helper when another worksheet is intended. Never pass `--sheet` for CSV or TSV; flat output cannot satisfy workbook-level preservation checks. XLSX preflight rejects known unsafe features and any OOXML package part dropped by its in-memory trial write. Excel's optional calculation chain may be removed and rebuilt by spreadsheet software.

.agents/skills/curate-aroma-data/scripts/inspect_workbook.py

@@ -3,24 +3,106 @@
 from __future__ import annotations
 
 import argparse
+import hashlib
 import json
 import sys
 from pathlib import Path
 
+from openpyxl import load_workbook
+
 REPO_ROOT = Path(__file__).resolve().parents[4]
 if str(REPO_ROOT) not in sys.path:
     sys.path.insert(0, str(REPO_ROOT))
 
-from aromanexus.excel_io import FORMULA_PREFIXES, read_table  # noqa: E402
+from aromanexus.excel_io import (  # noqa: E402
+    FORMULA_PREFIXES,
+    read_table,
+    workbook_sheet_names,
+)
 from aromanexus.identifiers import is_valid_cas, normalize_cas  # noqa: E402
 
 
-def inspect(path: Path, cas_column: str | None = None) -> dict[str, object]:
-    frame = read_table(path)
+def _stable_cell_value(value: object) -> str | None:
+    if value is None:
+        return None
+    return value.isoformat() if hasattr(value, "isoformat") else str(value)
+
+
+def _sheet_summary(worksheet, cached_worksheet) -> dict[str, object]:
+    digest = hashlib.sha256()
+    formula_cache_digest = hashlib.sha256()
+    formula_cells = 0
+    cached_formula_results = 0
+    styled_cells = 0
+    nonempty_cells = 0
+    for row in worksheet.iter_rows():
+        for cell in row:
+            if cell.has_style:
+                styled_cells += 1
+            if cell.value is None:
+                continue
+            nonempty_cells += 1
+            formula_cells += int(cell.data_type == "f")
+            value = _stable_cell_value(cell.value)
+            record = json.dumps(
+                [cell.coordinate, cell.data_type, value],
+                ensure_ascii=False,
+                separators=(",", ":"),
+            )
+            digest.update(record.encode("utf-8"))
+            digest.update(b"\n")
+            if cell.data_type == "f":
+                cached_cell = cached_worksheet[cell.coordinate]
+                cached_value = _stable_cell_value(cached_cell.value)
+                cached_formula_results += int(cached_value is not None)
+                cached_record = json.dumps(
+                    [cell.coordinate, cached_cell.data_type, cached_value],
+                    ensure_ascii=False,
+                    separators=(",", ":"),
+                )
+                formula_cache_digest.update(cached_record.encode("utf-8"))
+                formula_cache_digest.update(b"\n")
+    return {
+        "max_row": worksheet.max_row,
+        "max_column": worksheet.max_column,
+        "nonempty_cells": nonempty_cells,
+        "formula_cells": formula_cells,
+        "cached_formula_results": cached_formula_results,
+        "formula_cache_sha256": formula_cache_digest.hexdigest(),
+        "styled_cells": styled_cells,
+        "content_sha256": digest.hexdigest(),
+        "row_heights": {
+            str(index): dimension.height
+            for index, dimension in worksheet.row_dimensions.items()
+            if dimension.height is not None
+        },
+        "column_widths": {
+            name: dimension.width
+            for name, dimension in worksheet.column_dimensions.items()
+            if dimension.width is not None
+        },
+        "freeze_panes": str(worksheet.freeze_panes or ""),
+        "auto_filter": str(worksheet.auto_filter.ref or ""),
+        "merged_ranges": [str(item) for item in worksheet.merged_cells.ranges],
+        "tables": list(worksheet.tables),
+        "data_validations": len(worksheet.data_validations.dataValidation),
+        "conditional_formatting_ranges": len(worksheet.conditional_formatting),
+    }
+
+
+def inspect(
+    path: Path,
+    cas_column: str | None = None,
+    sheet_name: str | None = None,
+) -> dict[str, object]:
+    frame = read_table(path, sheet_name=sheet_name)
     detected_cas = cas_column or next(
         (name for name in ("CAS Number", "CAS", "cas", "cas_number") if name in frame.columns),
         None,
     )
+    if detected_cas is not None and detected_cas not in frame.columns:
+        available = ", ".join(map(str, frame.columns))
+        raise ValueError(f"CAS column {detected_cas!r} was not found. Available: {available}")
     formula_like = 0
     for column in frame.columns:
         formula_like += int(
@@ -39,6 +121,42 @@ def inspect(path: Path, cas_column: str | None = None) -> dict[str, object]:
             for column in frame.columns
         },
     }
+    if path.suffix.lower() == ".xlsx":
+        sheets = workbook_sheet_names(path)
+        selected_sheet = sheet_name or sheets[0]
+        workbook = load_workbook(
+            path,
+            read_only=False,
+            data_only=False,
+            keep_links=True,
+            rich_text=True,
+        )
+        cached_workbook = load_workbook(
+            path,
+            read_only=False,
+            data_only=True,
+            keep_links=True,
+            rich_text=True,
+        )
+        try:
+            sheet_summaries = {
+                worksheet.title: _sheet_summary(
+                    worksheet,
+                    cached_workbook[worksheet.title],
+                )
+                for worksheet in workbook.worksheets
+            }
+            report["workbook"] = {
+                "title": workbook.properties.title or "",
+                "creator": workbook.properties.creator or "",
+                "sheets": sheets,
+                "selected_sheet": selected_sheet,
+                "sheet_summaries": sheet_summaries,
+                "selected_sheet_summary": sheet_summaries[selected_sheet],
+            }
+        finally:
+            workbook.close()
+            cached_workbook.close()
     if detected_cas:
         normalized = frame[detected_cas].map(normalize_cas)
         valid_mask = normalized.map(is_valid_cas)
@@ -61,8 +179,19 @@ def main() -> int:
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("input", type=Path)
     parser.add_argument("--cas-column")
+    parser.add_argument("--sheet", help="XLSX worksheet to inspect (defaults to the first)")
     args = parser.parse_args()
-    print(json.dumps(inspect(args.input, args.cas_column), ensure_ascii=False, indent=2))
+    try:
+        print(
+            json.dumps(
+                inspect(args.input, args.cas_column, args.sheet),
+                ensure_ascii=False,
+                indent=2,
+            )
+        )
+    except (FileNotFoundError, ValueError) as exc:
+        print(f"Error: {exc}", file=sys.stderr)
+        return 2
     return 0