Skip to content

feat: add data export management API - #5792

Open
tgmendes wants to merge 12 commits into
mainfrom
feat/otel-data-forwarding-api
Open

feat: add data export management API#5792
tgmendes wants to merge 12 commits into
mainfrom
feat/otel-data-forwarding-api

Conversation

@tgmendes

@tgmendes tgmendes commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Builds on the database invariant from #5859.

Summary

  • Add the project-scoped dataExports Goa management service, SQLc repository, generated Go/TypeScript SDK surfaces, and generic listDestinations, createDestination, updateDestination, and deleteDestination operations alongside route CRUD.
  • Represent each destination as a generic envelope with a destination_type discriminator and transport-specific configuration. OTEL configuration remains typed under otel; handlers dispatch CRUD by type and aggregate per-table list results in stable order so future SIEM and S3 destinations can live in their own tables without flattening transport-specific fields into one model.
  • Require org:read for reads and org:admin for mutations while retaining the selected-project header and project-pinned persistence. Every repository query is pinned by both organization and project.
  • Keep one non-deleted route configuration per project and data source with nullable typed destination slots. This change implements otel_destination_id; future destination types can add sibling slots on the same route row.
  • Store OTEL headers as one encrypted, write-only map. Responses and audit snapshots expose only header names and whether a value exists; omitted update values preserve the existing secret by case-insensitive header name.
  • Store and validate the destination-level sensitive_data egress policy, defaulting missing database values to exclude. This PR does not implement payload sanitization or define the eventual field-removal contract.
  • Serialize destination secret merges, deletion checks, and route lifecycle changes with row locks, and record destination and route audit events atomically with each mutation.

Motivation

Migration PR #5859 establishes one non-deleted route row per project and data source while retaining nullable typed destination slots. This PR supplies the organization-authorized application boundary needed to manage that schema safely without making the management API OTEL-specific. Transport-specific configuration and persistence remain isolated, allowing SIEM, S3, and other destination types to use separate tables while sharing one stable destination and routing API.

@tgmendes
tgmendes requested review from a team as code owners August 27, 2026 10:06
@tgmendes tgmendes added enhancement New feature or request go Pull requests that update go code javascript Pull requests that update javascript code labels Aug 27, 2026
@changeset-bot

changeset-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: a76aa6c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
server Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Running ultrareview automatically — 'Adds a new tenant-scoped data export API with authorization, write-only encrypted secrets, row-lock concurrency, and audit events across 80 files and generated SDKs — a bug in project scoping, secret handling, or route lifecycle could leak data or credentials across tenants.'. I'll post findings when complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultrareview completed in 15m 37s

All reported issues were addressed across 80 files

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread server/internal/dataexports/destinations.go
Comment thread server/internal/dataexports/destinations.go
Comment thread server/internal/urn/data_export_route.go Outdated
Comment thread server/internal/audit/oteldestinations.go Outdated
Comment thread server/design/dataexports/design.go Outdated
Comment thread server/internal/dataexports/destinations.go Outdated
Comment thread server/internal/dataexports/destinations.go Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 24 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread server/internal/dataexports/destinations_test.go
@tgmendes
tgmendes force-pushed the feat/otel-data-forwarding-api branch from aae73d2 to a414ea0 Compare August 27, 2026 15:54
@tgmendes
tgmendes changed the base branch from main to mig/data-exports-fanout August 27, 2026 15:54
@blacksmith-sh

This comment has been minimized.

Base automatically changed from mig/data-exports-fanout to main August 28, 2026 10:00
@tgmendes
tgmendes force-pushed the feat/otel-data-forwarding-api branch 2 times, most recently from 9043ebb to deb74bb Compare September 1, 2026 09:15
@blacksmith-sh

This comment has been minimized.

@tgmendes

tgmendes commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Flaky ClickHouse startup; rerun passed.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 54 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread server/design/dataexports/design.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request go Pull requests that update go code javascript Pull requests that update javascript code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant