feat(catalog): reload catalog when a new manifest is fetched (#18280)

[esrevi]

Proposed changes

• Add query catalogsRevisions to compare current revision with backend catalog revision,
• On new revision, query catalog and update ingestion catalog

Related issues

• closes feat(xtm-integration): load catalog when new manifest is fetched #18280

How to test this PR

Checklist

• I consider the submitted work as finished
• I tested the code for its functionality
• I wrote test cases for the relevant use cases (coverage and e2e)
• I added/updated the relevant documentation (either on GitHub or on Notion)
• Where necessary, I refactored code to improve the overall quality

Further comments

[codecov]

Codecov Report

❌ Patch coverage is 66.14699% with 304 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.85%. Comparing base (45db6c9) to head (27808eb).
⚠️ Report is 168 commits behind head on issue/16059-a-runtime-behavior.

Files with missing lines	Patch %	Lines
...ql/src/modules/catalog/sync/catalog-sync-domain.ts	63.63%	68 Missing ⚠️
...-graphql/src/modules/catalog/catalog-repository.ts	17.72%	65 Missing ⚠️
...ncti-graphql/src/modules/catalog/catalog-domain.ts	70.27%	33 Missing ⚠️
...odules/catalog/sync/catalog-sync-source-gateway.ts	83.78%	24 Missing ⚠️
...raphql/src/modules/catalog/catalog-logo-storage.ts	71.25%	23 Missing ⚠️
...cti-graphql/src/modules/catalog/catalog-manager.ts	0.00%	23 Missing ⚠️
...tform/opencti-graphql/src/manager/managerModule.ts	11.11%	8 Missing ⚠️
...orm/opencti-graphql/src/modules/catalog/catalog.ts	38.46%	8 Missing ⚠️
...c/private/components/integrations/Integrations.tsx	0.00%	7 Missing ⚠️
...aphql/src/modules/catalog/catalog-version-utils.ts	80.00%	7 Missing ⚠️
... and 14 more

❗ There is a different number of reports uploaded between BASE (45db6c9) and HEAD (27808eb). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (45db6c9)	HEAD (27808eb)
opencti-client-python	3	0
opencti-graphql	3	2

Additional details and impacted files

@@                         Coverage Diff                         @@
##           issue/16059-a-runtime-behavior   #18308       +/-   ##
===================================================================
- Coverage                           34.65%   20.85%   -13.80%     
===================================================================
  Files                                3400     3335       -65     
  Lines                              139050   128239    -10811     
  Branches                            37713    37924      +211     
===================================================================
- Hits                                48181    26740    -21441     
- Misses                              90869   101499    +10630     

Flag	Coverage Δ
opencti-client-python	?
opencti-front	11.24% <81.33%> (+0.06%)	⬆️
opencti-graphql	36.58% <64.76%> (-33.68%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Adds catalog revision polling so ingestion catalogs refresh when backend manifests change.

Changes:

• Adds the catalogsRevisions GraphQL endpoint and backend support.
• Adds frontend polling, visibility handling, and catalog refetching.
• Adds backend and frontend polling tests.

Critical findings: the polling request can produce an unhandled rejection, and one test asserts a console message the implementation never emits.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Summary
opencti-platform/opencti-graphql/tests/03-integration/10-modules/catalog/catalog-resolver-test.ts	Tests the revisions query payload.
opencti-platform/opencti-graphql/src/modules/catalog/catalog.graphql	Defines the revision type and query.
opencti-platform/opencti-graphql/src/modules/catalog/catalog-types.ts	Adds backend revision typing.
opencti-platform/opencti-graphql/src/modules/catalog/catalog-resolver.ts	Resolves catalog revisions.
opencti-platform/opencti-graphql/src/modules/catalog/catalog-repository.ts	Loads lightweight revision data.
opencti-platform/opencti-graphql/src/modules/catalog/catalog-domain.ts	Maps revision responses.
opencti-platform/opencti-graphql/src/generated/graphql.ts	Updates generated backend GraphQL types.
opencti-platform/opencti-front/src/schema/relay.schema.graphql	Updates frontend schema definitions.
opencti-platform/opencti-front/src/private/components/integrations/Integrations.tsx	Exposes catalog refetching.
opencti-platform/opencti-front/src/private/components/integrations/catalog/IngestionConnectorsCatalog.tsx	Defines the revision query.
opencti-platform/opencti-front/src/private/components/integrations/catalog/hooks/useCatalogPolling.ts	Implements visibility-aware polling; request failures can become unhandled rejections.
opencti-platform/opencti-front/src/private/components/integrations/catalog/hooks/useCatalogPolling.test.ts	Tests polling behavior; includes an assertion for an un emitted console message.
opencti-platform/opencti-front/src/private/components/integrations/catalog/catalog-constants.ts	Defines the polling interval.
opencti-platform/opencti-front/src/private/components/integrations/available/IntegrationsAvailable.tsx	Enables polling on the catalog page.

Suppressed comments (3)

opencti-platform/opencti-front/src/private/components/integrations/catalog/hooks/useCatalogPolling.ts:105

• If the catalog page is mounted while the browser tab is already hidden, checkCatalogRevisions returns here before scheduling a timer, but wasPausedRef is never set. When the tab becomes visible, the visibility handler therefore does nothing and polling never starts until the page is remounted.

    void checkCatalogRevisions();

opencti-platform/opencti-front/src/private/components/integrations/catalog/hooks/useCatalogPolling.ts:85

• refetchCatalogs only calls loadCatalogs and returns void, so this await completes before the catalog query has succeeded. Updating the baseline immediately means a failed full-catalog refresh consumes the revision and later polls will not retry it, leaving stale contracts until another manifest revision or a page reload; advance the baseline only after a successful refresh.

        await onCatalogRevisionsChanged();
        baselineRef.current = nextBaseline;

opencti-platform/opencti-front/src/private/components/integrations/catalog/hooks/useCatalogPolling.ts:72

• This polling call uses the shared fetchQuery wrapper without a network-only policy. Relay's default fetch policy can satisfy repeated executions from the store, so after the first response this timer can keep comparing the same cached revisions and never detect a backend manifest update; expose/pass a network-only policy for this query.

        const result = await fetchQuery<IngestionConnectorsCatalogRevisionsQuery>(
          ingestionConnectorsCatalogRevisionsQuery,
          {},
        ).toPromise();

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

opencti-platform/opencti-front/src/private/components/integrations/Integrations.tsx:94

• refetchCatalogs is recreated on every IntegrationsDataProvider render, while useCatalogPolling uses onCatalogRevisionsChanged as an effect dependency. Any provider update consequently tears down and restarts the poller, immediately issuing another revisions request and resetting the 60-second cadence. Memoize this callback (or make the hook keep a stable callback ref) so unrelated provider updates do not trigger extra polling.
  opencti-platform/opencti-front/src/private/components/integrations/catalog/hooks/useCatalogPolling.ts:90
• The callback used here is refetchCatalogs, which only starts loadCatalogs and returns void; awaiting it does not wait for the full catalog request to succeed. The baseline is therefore advanced immediately, so a failed catalog refresh is treated as handled and will not be retried while the manifest revision remains unchanged. Have the callback resolve only after the catalog refresh succeeds, or update the baseline from the successful catalog data instead.
  opencti-platform/opencti-front/src/private/components/integrations/catalog/hooks/useCatalogPolling.ts:110
• When the page is mounted while the document is already hidden, the initial check returns at line 64 without setting wasPausedRef. The later visibilitychange handler therefore sees wasPausedRef.current === false and never starts polling after the tab becomes visible. Initialize the paused state from document.hidden (and only start the initial check when visible) so background-opened catalog pages resume correctly.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 3 comments.

Suppressed comments (5)

Previously missed (3) — in code that hasn't changed since the last review.

opencti-platform/opencti-front/src/private/components/integrations/catalog/hooks/useCatalogPolling.ts:77

• Cleanup only stops future timers; an in-flight fetchQuery can still resolve after the user leaves the Available tab. In that case this code will invoke onCatalogRevisionsChanged, and the provider will reload catalogs even though polling has been disabled by unmount. Check isUnmountedRef.current after the await before processing the response.
  opencti-platform/opencti-front/src/private/components/integrations/catalog/hooks/useCatalogPolling.ts:91
• refetchCatalogs only calls loadCatalogs, which is a fire-and-forget query-loader action and is typed as () => void. The baseline is therefore advanced before the full catalog request completes; if that request fails, polling sees the new revision as already handled and never retries, leaving the old catalog displayed. Return or await a refresh completion signal before updating the baseline, or keep the old baseline until a successful catalog refresh is confirmed.
  opencti-platform/opencti-front/src/private/components/integrations/catalog/hooks/useCatalogPolling.ts:107
• If this hook mounts while document.hidden is already true, the initial check returns at line 64 before setting wasPausedRef. When the tab becomes visible, this condition is false, so no revision check or timer is started and the page will never detect later catalog changes. Treat an initial hidden mount as paused (or also trigger when the baseline is still unset).

opencti-platform/opencti-front/src/private/components/integrations/Integrations.tsx:94

• refetchCatalogs returns immediately after scheduling loadCatalogs; it does not report whether the store-and-network catalog request succeeded. The polling hook awaits this callback and then records nextBaseline as handled, so a transient full-catalog refresh failure can leave stale contracts displayed while suppressing retries until another revision appears. Propagate a completion signal and advance the baseline only after the catalog refresh succeeds.

  const refetchCatalogs = () => {
    if (isConnectorReader) {
      loadCatalogs({}, { fetchPolicy: 'store-and-network' });
    }
  };

opencti-platform/opencti-front/src/private/components/integrations/Integrations.tsx:94

• refetchCatalogs is recreated on every IntegrationsDataProvider render, while useCatalogPolling includes it in the effect dependencies and performs an immediate check whenever that effect starts. Any provider update therefore tears down the 60-second timer and issues an extra revision request, including after a catalog refresh, which can create duplicate network traffic. Memoize this handler so the polling effect is stable.

  const refetchCatalogs = () => {
    if (isConnectorReader) {
      loadCatalogs({}, { fetchPolicy: 'store-and-network' });
    }
  };

[Copilot]

🟡 Changes recommended

The new backend integration test queries/asserts id instead of the schema field catalog_id, which will fail at runtime until corrected.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

opencti-platform/opencti-graphql/tests/03-integration/10-modules/catalog/catalog-resolver-test.ts:224

• These assertions still expect id, but the catalogsRevisions payload contains catalog_id (and revision is nullable in the schema).

      expect(Object.keys(revisionEntry).sort()).toEqual(['id', 'revision']);
      expect(revisionEntry.id).toEqual(expect.any(String));
      expect(revisionEntry.revision).toEqual(expect.any(String));
    }

• Files reviewed: 14/15 changed files
• Comments generated: 1
• Review effort level: Lite

[esrevi]

closed and replaced by this PR #18332