Skip to content

Commit 604db44

Browse files
committed
feat(references): add support for remote datasets
1 parent 9a4fc8f commit 604db44

17 files changed

Lines changed: 1617 additions & 328 deletions

.claude/harness/architecture.md

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# JavaScript Content API Library — Architecture
2+
3+
Reference description of the project, its layout, the core modules, the request flow, and the CaaS storage model it reads from. Referenced from the project `CLAUDE.md`.
4+
5+
## Project overview
6+
7+
**fsxa-api** (the JavaScript Content API Library, a.k.a. Content API) is a published npm library — not an application. It reads content from the FirstSpirit **CaaS** (Content-as-a-Service) and from the **Navigation Service**, and maps the raw CaaS JSON into a stable, consumer-facing shape for PWAs and other frontends. It ships as CommonJS + ES5 bundles plus type declarations, and is consumed by Crownpeak PWA templates and by customer projects.
8+
9+
Because it is a library, its exported surface is the product. See "Public API surface" below.
10+
11+
## Layout
12+
13+
| Path | Description |
14+
|------|-------------|
15+
| `src/modules/` | The core: both API implementations, the mapper, the query builder, the event stream, logging |
16+
| `src/integrations/` | Adapters for hosting the proxy backend — the Express router and the framework-agnostic wrapper |
17+
| `src/types.ts` | Both sides of every mapping: `CaaSApi_*` (raw CaaS shapes) and the mapped consumer shapes |
18+
| `src/enums.ts` | Error message enums, content mode, proxy routes, HTTP status |
19+
| `src/routes.ts` | Proxy route paths and request body interfaces — the contract between proxy and remote side |
20+
| `src/testutils/` | Fixture factories used by unit tests (`createPageRef`, `createDataset`, …) |
21+
| `src/helpers/`, `src/utils.ts` | Small pure helpers (locale discovery, navigation-map pruning, regex validation, rich-text link merging) |
22+
| `integrationtests/` | Tests against a real CaaS tenant — see the testing guidelines |
23+
| `proxy/` | npm workspace publishing the proxy-only entry point |
24+
| `dev/` | Local scratch harness (`npm run dev`) for hitting a real CaaS by hand |
25+
| `docs/superpowers/plans/` | Implementation plans written by the `writing-plans` skill |
26+
27+
## The two API implementations
28+
29+
Both implement the same `FSXAApi` interface (`src/types.ts`) and expose `fetchElement`, `fetchByFilter`, `fetchNavigation`, `fetchProjectProperties`:
30+
31+
- **`FSXARemoteApi`** (`src/modules/FSXARemoteApi.ts`) — talks to CaaS and the Navigation Service directly. It holds the API key, so it only ever runs server-side. It owns URL construction (`buildCaaSUrl`, `buildNavigationServiceUrl`), the trust check for reference URLs (`isTrustedReferenceUrl`), and the config: `apikey`, `caasURL`, `navigationServiceURL`, `tenantID`, `projectID`, `contentMode`, `remotes` (deprecated), `maxReferenceDepth`, `customMapper`, `navigationItemFilter`, `caasItemFilter`.
32+
- **`FSXAProxyApi`** (`src/modules/FSXAProxyApi.ts`) — same interface, but forwards each call as an HTTP POST to a backend that hosts an `FSXARemoteApi`. It carries no secrets and is the client-side implementation.
33+
34+
`FSXAApiSingleton` holds one process-wide instance. The proxy backend is built with `src/integrations/express.ts` (`getExpressRouter`) or, for other frameworks, `useEndpointIntegrationWrapper` in `src/integrations/endpointIntegrationWrapper.ts`. The routes and body shapes both sides agree on live in `src/routes.ts`.
35+
36+
**Consequence for any change to a fetch method:** a new parameter has to be threaded through `FSXAApi` (the interface), `FSXARemoteApi` (the implementation), `FSXAProxyApi` (serialize into the request body), `src/routes.ts` (the body interface), and the integration adapters (deserialize and validate). Changing only the remote side silently leaves proxy-mode consumers behind.
37+
38+
## CaaSMapper and reference resolution
39+
40+
`src/modules/CaaSMapper.ts` is the heart of the library and the file most work touches. It converts raw CaaS documents (`CaaSApi_PageRef`, `CaaSApi_Dataset`, `CaaSApi_Media`, …) into mapped items, walking the FirstSpirit document model: `PageRef``Page``Body``Section`, each carrying `formData` / `metaFormData` of typed input components (`CMS_INPUT_*`, `FS_REFERENCE`, `FS_DATASET`, `FS_INDEX`, `FS_CATALOG`, `CMS_INPUT_IMAGEMAP`, `Content2Section`).
41+
42+
Reference resolution is **two-phase**, and understanding this is a prerequisite for editing the mapper:
43+
44+
1. **Register.** While mapping, a reference is not fetched. `registerReferenceFromUrl` / `registerReferencedItem` record the referenced id together with the path in the output object where the resolved item must later be placed, and mapping returns a placeholder string. One id can be registered at many paths.
45+
2. **Resolve.** `resolveAllReferences` walks the registered groups and calls `resolveReferencesForGroup` per group, which chunks ids (`REFERENCED_ITEMS_CHUNK_SIZE = 30`) and fetches them via `fetchByFilter`.
46+
3. **Denormalize.** `MappingUtils.denormalizeResolvedReferences` writes each fetched item into every path registered for it — or, in normalized mode, the caller receives `items` plus a flat `referenceMap` instead.
47+
48+
Grouping is keyed by **(projectId, locale)**`buildGroupKey` — because one CaaS filter query carries exactly one locale and one collection. Each disjoint pair therefore costs at least one request. `unifyId` namespaces ids as `projectId#uuid.locale` so items from different projects cannot collide in the cache or the reference map.
49+
50+
Which project and locale a reference belongs to is derived from the reference's own CaaS document URL by `deriveReferenceTarget`, using `src/modules/ReferenceUrlParser.ts`. `FSXARemoteApi.isTrustedReferenceUrl` gates this: a reference URL is only followed when its origin and tenant match the configured `caasURL`/`tenantID`. The `remotes` configuration is deprecated and no longer consulted for resolution — see the README section "Resolving references across projects".
51+
52+
Two guards bound the recursion: `maxReferenceDepth` (default `DEFAULT_MAX_REFERENCE_DEPTH = 2`) and `_processedItems`, which prevents re-fetching an id already handled.
53+
54+
**Sharp edge:** `setLocaleFromCaasItem` mutates `this.locale` per mapped item, and `unifyId` falls back to `this.locale`. For a reference that carries no URL, the resulting group key therefore depends on which item was mapped last. Do not rely on `this.locale` being stable across a `mapFilterResponse` call.
55+
56+
## Request flow (remote mode)
57+
58+
1. The consumer calls `fetchElement` / `fetchByFilter` / `fetchNavigation` / `fetchProjectProperties`.
59+
2. `FSXARemoteApi` builds the URL (`buildCaaSUrl`) and, for filters, translates the query via `QueryBuilder` into the CaaS filter syntax; the requested locale becomes `locale.language` + `locale.country` filters.
60+
3. The response is handed to `CaaSMapper`, which maps documents and registers references (phase 1).
61+
4. References are resolved per group and denormalized into the result (phases 2 and 3).
62+
5. Optional hooks run: `customMapper` per data entry, `caasItemFilter` on mapped items, `navigationItemFilter` on navigation items.
63+
64+
`CaaSEventStream` (lazily loaded via `CaaSEventStreamLazy` so `better-sse` stays out of browser bundles) is a separate path: it streams CaaS change events instead of fetching documents.
65+
66+
## CaaS storage model
67+
68+
A CaaS document URL is `baseURL / tenantID / collectionID / documentID`, for example:
69+
70+
```
71+
https://enterprise-caas-api.e-spirit.cloud/enterprise-prod/3bb083df-446f-4cdd-af7e-514886c4dc20.preview.content/21f3109e-63b2-47e8-9728-5680c2decb02.en_US
72+
```
73+
74+
- `collectionID` = `<project uuid>.<content mode>.content`
75+
- `documentID` = `<document uuid>.<locale>`
76+
77+
Content lives in `preview.content` / `release.content`; binaries in `preview.files` / `release.files`. The **content mode never comes from content data** — it comes only from this library's configuration. Preserve that when touching URL construction or reference parsing.
78+
79+
## Public API surface
80+
81+
Everything re-exported from `src/index.ts` is published: the modules listed there, all of `src/enums.ts`, all of `src/types.ts`, the helpers, the exceptions, `ROUTES`, and the integration wrappers. A rename or a narrowed type in `src/types.ts` is a breaking change for consumers even if nothing inside `src/` notices.
82+
83+
`src/modules/index.ts` re-exports in a **deliberate order** to break a circular-dependency cycle. Do not reorder those export lines.
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# JavaScript Content API Library — Coding Guidelines
2+
3+
Project-specific coding rules that complement the project overview in `CLAUDE.md`. Apply at write time, not only at review time.
4+
5+
## Code style & quality
6+
7+
- **Language**: TypeScript, `strict: true`, compiled to ES5 / CommonJS (`tsconfig.json`). No decorators in new code despite `experimentalDecorators` being on.
8+
- **Formatting**: Prettier with `semi: false`, `singleQuote: true` (config lives in `package.json`). Committed code uses the Prettier 2 trailing-comma default — see the formatting section in `CLAUDE.md` before running Prettier at all.
9+
- **Node**: `.nvmrc` pins 24.12.0; CI runs 24.x. `engines` still claims `>=14`, so do not use syntax or APIs newer than the declared floor in shipped code without raising `engines` deliberately.
10+
- **No linter**: there is no ESLint and no `lint` script. Type errors are the only automated style signal, so run `npx tsc --noEmit` yourself — `npm test` will not surface a type error in a file no test imports.
11+
- **Dependencies**: this is a library consumed in browsers. Prefer none. A new runtime dependency lands in every consumer's bundle and needs an explicit justification; heavy or server-only packages belong in `optionalDependencies` and must be lazily imported like `better-sse` is in `CaaSEventStreamLazy.ts`.
12+
13+
## Never break the published surface
14+
15+
**Rule:** Treat every symbol re-exported from `src/index.ts` — including every type in `src/types.ts` — as a public contract. Additive changes only: new optional fields, new optional parameters, widened return types. When behavior must change or a name must go, deprecate instead of removing.
16+
17+
This library is published to npm and consumed by PWA templates and customer projects that upgrade on their own schedule. A removed field or a narrowed type breaks builds in repositories you cannot see or fix. Removing something is also a release decision, not just a code decision: `release-it` reads Conventional Commits, so a `BREAKING CHANGE:` footer ships a major version.
18+
19+
**Smell:** renaming a field in a `CaaSApi_*` or mapped type "because nothing in `src/` uses the old name" — `src/` is not the consumer.
20+
21+
### How to deprecate
22+
23+
1. Keep the old symbol working, and keep it exported.
24+
2. Add a `@deprecated` JSDoc tag naming the replacement and the README section that explains the migration.
25+
3. Document the change in `README.md` under a stable heading (update notices link to those anchors).
26+
4. Carry the deprecated path for at least one release before proposing removal — and propose it, do not do it unasked.
27+
28+
If a behavior change is unavoidable even while the old API keeps compiling (a value now derived from a different source, a config field now ignored), that is still a behavior change for existing applications: document it explicitly as such, not just in the changelog.
29+
30+
## Content data must never decide where a request goes
31+
32+
**Rule:** No value read from a CaaS document may determine the host, tenant, or content mode of an outbound request that carries the API key. Host and tenant come from configuration; a URL found in content may only be *checked against* configuration, never used in its place.
33+
34+
`FSXARemoteApi` holds the CaaS API key and runs server-side. A URL inside a document is editor-controlled data. If it selected the target host, a crafted document would make the server send its API key to an attacker's endpoint — a textbook SSRF with credential leak. `isTrustedReferenceUrl` exists exactly for this: origin and tenant must equal the configured `caasURL` origin and `tenantID`; only collection (project) and locale may differ.
35+
36+
**Smell:** `fetch(reference.url, { headers: { Authorization: apikey } })`, or passing a parsed `baseUrl` into `buildCaaSUrl` instead of comparing it to the configured one.
37+
38+
### Checklist when adding anything URL-derived
39+
40+
1. Parse with `ReferenceUrlParser` — do not hand-roll string splitting on CaaS URLs, and do not duplicate the layout knowledge it owns.
41+
2. Run the parse result through `isTrustedReferenceUrl` before acting on it.
42+
3. Use only the project id and locale from it. Content mode stays configuration-derived.
43+
4. On an untrusted or unparsable URL, log a warning and skip the reference. Do not fall back to "try it anyway".
44+
45+
## Mirror the change across parallel siblings
46+
47+
**Rule:** When you touch a method, branch, type, or route — whether fixing a bug, refactoring, or cleaning up — immediately scan the parallel siblings with the same pattern, test files included. If the same change on a sibling is structurally identical, include it in the same pass without asking, and flag it in the commit/PR message.
48+
49+
This codebase is built from mirrored pairs and families. Fixing one member ships the same bug under a different name in the others.
50+
51+
The sibling groups to check:
52+
53+
1. **`FSXARemoteApi``FSXAProxyApi`** — both implement `FSXAApi`. A changed parameter or behavior almost always needs the proxy counterpart, plus the body interface in `src/routes.ts`, plus the Express router in `src/integrations/express.ts` and `endpointIntegrationWrapper.ts`, plus `parameterValidation.ts`.
54+
2. **The `mapDataEntry` component branches**`FS_REFERENCE`, `FS_DATASET`, `FS_INDEX`, `FS_CATALOG`, `CMS_INPUT_IMAGEMAP` all register references; a fix to one registration site usually applies to the others.
55+
3. **The `map*` family**`mapPageRef`, `mapDataset`, `mapGCAPage`, `mapMedia*`, `mapProjectProperties`.
56+
4. **Raw ↔ mapped types** — a new field on a `CaaSApi_*` type usually needs its counterpart on the mapped type, and vice versa.
57+
5. **preview ↔ release** content modes.
58+
59+
**Scope:** the sweep covers every file changed on the current branch, committed and uncommitted alike. Do not leave a half-applied edit in the working tree.
60+
61+
A structurally identical mirror change is bounded — apply it inline. A mirror that needs real per-sibling design judgment is separate scope — name it and leave it, rather than silently widening the diff.
62+
63+
## Register-then-resolve: never fetch during mapping
64+
65+
**Rule:** Mapping code (`mapDataEntry`, `map*`) must not issue a fetch for a referenced item. Register the reference and return the placeholder; let `resolveAllReferences` do the I/O.
66+
67+
The two-phase design is what makes reference loading batched: one request per (projectId, locale) group of up to 30 ids instead of one request per reference. A single `await api.fetchElement(...)` inside a mapping branch turns a page with 50 images into 50 sequential round-trips, and it bypasses both `_processedItems` dedup and the `maxReferenceDepth` guard.
68+
69+
**Smell:** an `await this.api.fetch…` inside a `case 'FS_…'` branch, or building a result object that already contains a resolved item rather than the id returned by `registerReferencedItem`.
70+
71+
### Checklist when adding a new referencing component type
72+
73+
1. Add the branch to `mapDataEntry` and register via `registerReferenceFromUrl` with the reference's own CaaS URL, so project and locale are derived, not guessed.
74+
2. Return the value `registerReferencedItem` gives you — and handle `null` (untrusted or unresolvable), which means the reference is dropped.
75+
3. Confirm the new id ends up in the right group: assert on the group key and on the resulting `fetchByFilter` calls, not just on the mapped output.
76+
4. Add the raw shape to `src/types.ts` and a factory to `src/testutils/`.
77+
78+
## Comments must add information not derivable from the code
79+
80+
**Rule:** Default to writing no comments. A comment is only justified when it adds information not derivable from the code under it — a hidden constraint, a workaround for a specific bug, a non-obvious invariant, behavior that would surprise a reader. If unsure whether a comment adds information: delete it.
81+
82+
This applies at write time and at all times after — write no useless comment in the first place, and remove any useless comment you come across, even one you just wrote. Comment hygiene is never deferred to a cleanup pass. It applies to test code exactly as to production code.
83+
84+
Never write comments that:
85+
86+
1. Restate what the code does — a well-named identifier, signature, or return type already says it.
87+
2. Reference the current task, ticket, or caller ("added for CAAS-123", "used by the proxy") — that belongs in the commit message and the PR description.
88+
3. Mark removed code (`// removed foo`).
89+
4. Explain a parameter whose meaning is obvious from its type and name.
90+
91+
**Judge against the code, not a snippet:** evaluate a comment by reading it together with the full declaration and body it sits on, never from a `git diff` hunk or a `grep` match in isolation.
92+
93+
Conversely, keep comments that encode a decision the code cannot show. This repo has several that earn their place: why `src/modules/index.ts` fixes its export order (circular dependency), why `CaaSEventStream` is loaded lazily (browser bundle size), why a reference URL is only trusted after an origin check. Documentation of the public API is different in kind — JSDoc on exported symbols is consumer-facing and welcome.
94+
95+
## Errors: reuse the enums, stay actionable
96+
97+
**Rule:** User-facing failure messages belong in `FSXAApiErrors` / `CaaSMapperErrors` / `QueryBuilderErrors`, not inline in a `throw`. Before adding a case, check whether one already covers it.
98+
99+
Consumers assert on these strings and match on them in their own error handling, which makes the enum values part of the public surface — editing an existing message is a breaking change for someone. Add a new member instead.
100+
101+
Choose deliberately between throwing and warning. A misconfiguration the consumer must fix (missing API key, invalid locale) throws at construction or call time. Bad *content* — an unparsable reference URL, a broken reference, a component the mapper does not know — must not take down a page render: log through `this._logger` / `this.logger` and degrade, as the mapper already does for dropped references. Never use bare `console.log`; the `Logger` respects the configured `logLevel`.

0 commit comments

Comments
 (0)