Packages backend

[fredex42]

What's changed?

Adds a new backend feature to support collections that do not belong to any front at all - i.e., packages.

The intention is to support Feast's upcoming "Guardian Selections" feature - here referred to more prosaically as "recipe packages". As discussed with @davidfurey et. al., it's been written in such a way as to be easily extendable for a future port of Story Packages.

Implementation notes

This PR implements a number of endpoints under /packages inspired by the way the Editions endpoints work - see ./doc/Packages%20Endpoints.md. The only frontend changes are Typescript definitions for the core API data types.

Packages live in the Postgres database - the core class has been renamed to FaciaDB from EditionsDB since it no longer just serves Editions (and Editions no longer exist as a product).

How to test

You can exercise the package endpoints locally or in CODE, once it is deployed; authentication is handled by logging in with the browser then copying your guAuth-assum cookie into the request.

You need to make sure you have the "Edit fronts packages" permission enabled in Permissions tool for your account otherwise your requests will be rejected.

You can use any REST client of your choice, though the detailed runthrough shows curl

Drop down to see a full test protocol

Preconditions

1. Start the app locally (or use a deployed environment).
2. Ensure you are authenticated for Editions API access (the endpoints use EditEditionsAuthAction).
3. Set a base URL and auth cookie for your session.

export BASE_URL="https://fronts.local.dev-gutools.co.uk"
export AUTH_COOKIE='your_auth_cookie_name=your_cookie_value'

4. Generate IDs for this run (keeps the run idempotent):

export PREFILLED_PACKAGE_ID="$(uuidgen | tr '[:upper:]' '[:lower:]')"
export CREATED_PACKAGE_ID="$(uuidgen | tr '[:upper:]' '[:lower:]')"

5. Use this timestamp value in request bodies:

export NOW_MS="$(date +%s%3N)"

0) Prefill a package fixture and cards

Create a baseline package (acts as fixture). Use POST /packages.

{
  "id": "${PREFILLED_PACKAGE_ID}",
  "name": "Prefilled package",
  "isHidden": false,
  "webMetadata": {
    "source": "manual-protocol"
  },
  "feastMetadata": {
    "targetedRegions": ["uk"],
    "excludedRegions": ["us"]
  },
  "prefill": "recipes",
  "createdOn": ${NOW_MS},
  "createdBy": "Manual Tester",
  "createdEmail": "manual.tester@guardian.co.uk"
}

Expected:

• HTTP 201 Created
• Empty body

Add cards by overwriting the package with PUT /packages/:id (this is the supported endpoint-level way to set package items):

{
  "id": "${PREFILLED_PACKAGE_ID}",
  "name": "Prefilled package",
  "isHidden": false,
  "webMetadata": {
    "source": "manual-protocol"
  },
  "feastMetadata": {
    "targetedRegions": ["uk"],
    "excludedRegions": ["us"]
  },
  "prefill": "recipes",
  "createdOn": ${NOW_MS},
  "createdBy": "Manual Tester",
  "createdEmail": "manual.tester@guardian.co.uk",
  "updatedOn": ${NOW_MS},
  "updatedBy": "Manual Tester",
  "updatedEmail": "manual.tester@guardian.co.uk",
  "items": [
    {
      "id": "recipe-10001",
      "cardType": "recipe",
      "addedOn": ${NOW_MS},
      "metadata": {"slot": 0}
    },
    {
      "id": "recipe-10002",
      "cardType": "recipe",
      "addedOn": ${NOW_MS},
      "metadata": {"slot": 1}
    },
    {
      "id": "recipe-10003",
      "cardType": "recipe",
      "addedOn": ${NOW_MS},
      "metadata": {"slot": 2}
    }
  ]
}

Expected:

• HTTP 200 OK
• JSON object with matching package id, name, and items length 3

1) Create a new package with POST

POST /packages

{
  "id": "${CREATED_PACKAGE_ID}",
  "name": "Created package",
  "isHidden": false,
  "webMetadata": {
    "source": "manual-create"
  },
  "feastMetadata": {
    "targetedRegions": ["gb"],
    "excludedRegions": ["us"]
  },
  "prefill": "recipes",
  "createdOn": ${NOW_MS},
  "createdBy": "Manual Tester",
  "createdEmail": "manual.tester@guardian.co.uk"
}

Expected:

• HTTP 201 Created
• Empty body

2) Get the prefilled package by ID

GET /packages/${PREFILLED_PACKAGE_ID}

Expected:

• HTTP 200 OK
• JSON object includes:
  • id == PREFILLED_PACKAGE_ID
  • name == "Prefilled package"
  • items array exists and contains 3 elements

3) List available packages

GET /packages

Expected:

• HTTP 200 OK
• JSON object:
  • status == "ok"
  • packages is an array containing at least both package IDs from this protocol

4) Verify sort order params and title search

4a. Sort by created time

GET /packages?order=created

Expected:

• HTTP 200 OK
• packages array returned, newest first by created timestamp

4b. Sort by updated time

GET /packages?order=updated

Expected:

• HTTP 200 OK
• packages array returned, newest first by updated timestamp

4c. Search by title/name

GET /packages?order=title&title=Created

Expected:

• HTTP 200 OK
• packages array where entries match name search (contains Created package)

5) Overwrite an existing package with PUT

PUT /packages/${PREFILLED_PACKAGE_ID}

{
  "id": "${PREFILLED_PACKAGE_ID}",
  "name": "Prefilled package overwritten",
  "isHidden": false,
  "webMetadata": {
    "headline": "Updated by overwrite"
  },
  "feastMetadata": {
    "targetedRegions": ["au", "uk"],
    "excludedRegions": ["us"]
  },
  "prefill": "recipes",
  "createdOn": ${NOW_MS},
  "createdBy": "Manual Tester",
  "createdEmail": "manual.tester@guardian.co.uk",
  "updatedOn": ${NOW_MS},
  "updatedBy": "Manual Tester",
  "updatedEmail": "manual.tester@guardian.co.uk",
  "items": [
    {
      "id": "recipe-20001",
      "cardType": "recipe",
      "addedOn": ${NOW_MS},
      "metadata": {"slot": 0}
    },
    {
      "id": "recipe-20002",
      "cardType": "recipe",
      "addedOn": ${NOW_MS},
      "metadata": {"slot": 1}
    }
  ]
}

Expected:

• HTTP 200 OK
• JSON object:
  • name == "Prefilled package overwritten"
  • items contains exactly the two overwritten cards

6) Update package name only with PATCH

PATCH /packages/${PREFILLED_PACKAGE_ID}/name

Body is plain text (not JSON):

Renamed with patch

Expected:

• HTTP 204 No Content
• Follow-up GET /packages/:id shows name == "Renamed with patch"

7) Update feast-metadata regions only with PATCH

PATCH /packages/${PREFILLED_PACKAGE_ID}/update-regions

{
  "targetedRegions": ["eu", "uk"],
  "excludedRegions": ["us"]
}

Expected:

• HTTP 204 No Content
• Follow-up GET /packages/:id shows feastMetadata.targetedRegions == ["eu", "uk"]

8) Update hidden flag only with PUT is-hidden

PUT /packages/${PREFILLED_PACKAGE_ID}/is-hidden/true

Expected:

• HTTP 204 No Content
• Follow-up GET /packages/:id shows isHidden == true

9) Update metadata only with PUT metadata

PUT /packages/${PREFILLED_PACKAGE_ID}/metadata

Use a Feast metadata body (this endpoint accepts PackageMetadata, either Feast or Web shape):

{
  "targetedRegions": ["eu"],
  "excludedRegions": ["us"]
}

Expected:

• HTTP 204 No Content
• Follow-up GET /packages/:id confirms metadata changed and package name remains unchanged

Example curl pattern

Use this shape for each call (replace method/path/body):

curl -i \
  -X <METHOD> \
  -H "Content-Type: application/json" \
  -H "Cookie: ${AUTH_COOKIE}" \
  "${BASE_URL}<PATH>" \
  --data '<JSON BODY>'

For the name patch endpoint, send plain text body and Content-Type: text/plain; charset=utf-8.

Checklist

General

• 🤖 Relevant tests added
• ✅ CI checks / tests run locally
• 🔍 Checked on CODE

Client

• 🚫 No obvious console errors on the client (i.e. React dev mode errors) n/a
• 🎛️ No regressions with existing user interactions (i.e. all existing buttons, inputs etc. work) n/a
• 📷 Screenshots / GIFs of relevant UI changes included n/a

[Copilot]

🟡 Changes recommended

Critical compilation, migration, routing, and data-handling issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds a PostgreSQL-backed /packages backend for standalone recipe and story packages, including APIs, persistence, client types, documentation, and tests.

Changes:

• Adds package models, database queries, schema evolution, controllers, and integration tests.
• Renames EditionsDB to FaciaDB.
• Adds TypeScript API types, documentation, and package routes.

File summaries

File	Reviewed changes and final findings
test/tools/FaciaApiTest.scala	Reformats test helper calls.
test/services/editions/publishing/PublishingTest.scala	Updates database mock references.
test/services/editions/db/PackageDBTest.scala	Adds package persistence tests.
test/services/editions/db/EditionsDBTest.scala	Updates renamed database references.
test/services/editions/db/EditionsDBEvolutionsTest.scala	Updates evolution fixture references.
test/fixtures/FaciaDBService.scala	Adds the renamed database fixture.
test/fixtures/FaciaDBEvolutions.scala	Renames the evolution fixture.
test/controllers/PackageControllerHttpIntegrationTest.scala	Adds package endpoint integration tests.
fronts-client/src/types/FaciaApi.ts	Adds package API types. Moderate, 3 votes: new types are omitted from the export block.
docs/Packages API.md	Documents package APIs. Nit, 2 votes: use its instead of it's at lines 9, 24, and 40.
conf/routes	Adds package routes. Moderate, 3 votes: the content PATCH action has no route.
conf/logback.xml	Changes logging configuration. Moderate, 1 vote: structured logging is lost. Moderate, 2 votes: rolling file logging is disabled.
conf/evolutions/default/14.sql	Adds package tables. Critical, 1 vote: evolution markers use an unsupported format.
app/services/editions/publishing/Publishing.scala	Uses FaciaDB.
app/services/editions/publishing/events/PublishEventsListener.scala	Uses FaciaDB.
app/services/editions/db/PackageQueries.scala	Implements package queries. Critical, 2 votes: malformed interpolated SQL will not compile. Moderate, 3 votes: title ordering uses the wrong column. Moderate, 3 votes: overwrites do not update card_type. Moderate, 3 votes: patch insertion does not reindex existing cards. Moderate, 2 votes: the non-strict timestamp filter excludes the boundary. Nit, 2 votes: identifer is misspelled. Nit, 1 vote: documentation says 410 while the handler returns 409.
app/services/editions/db/IssueQueries.scala	Updates renamed database references.
app/services/editions/db/FaciaDB.scala	Renames and extends the database facade.
app/services/editions/db/CollectionsQueries.scala	Updates renamed database references.
app/model/packages/PackageMetadata.scala	Defines package metadata. Critical, 3 votes: optional schemas with ignored unknown fields make web metadata parse as Feast metadata.
app/model/packages/PackageCardType.scala	Defines card types. Critical, 1 vote: the toString declaration requires override.
app/model/packages/PackageCardRow.scala	Defines persisted package cards.
app/model/packages/PackageCard.scala	Defines package card models.
app/model/packages/Package.scala	Defines the package model.
app/model/packages/MetadataHelpers.scala	Provides metadata conversion helpers.
app/model/packages/client/UpdateRegionsRequest.scala	Defines region update requests.
app/model/packages/client/PatchContentRequest.scala	Defines content patch operations.
app/model/packages/client/CreatePackageRequest.scala	Defines package creation requests.
app/model/packages/client/ClientPackageHeader.scala	Defines package list responses.
app/model/packages/client/ClientPackageCard.scala	Converts package cards for API use.
app/model/packages/client/ClientPackage.scala	Defines package API payloads.
app/model/forms/GetPackagesFilter.scala	Defines package filtering.
app/controllers/V2App.scala	Updates the database dependency type.
app/controllers/PackageController.scala	Implements package actions. Critical, 2 votes: remove operations are discarded, so content PATCH cannot delete cards. Moderate, 3 votes: invalid IDs broaden the query to all packages. Moderate, 3 votes: full=true performs up to 500 extra queries. Moderate, 1 vote: internal exception details are exposed. Moderate, 1 vote: list errors expose exception messages. Moderate, 1 vote: create errors expose exception messages. Moderate, 1 vote: date and limit parsing and validation occur outside safe handling. Critical, 1 vote: logger.logger does not compile. Moderate, 2 votes: valid date-only input cannot be parsed as OffsetDateTime. Moderate, 2 votes: malformed or negative limits are not safely handled. Moderate, 1 vote: region updates mutate story packages unexpectedly. Moderate, 1 vote: metadata updates do not return 404 for unknown packages. Moderate, 1 vote: name updates return 500 instead of 404 for unknown packages.
app/controllers/EditionsController.scala	Updates the database dependency type.
app/Components.scala	Wires package services and controllers.

Review details

Suppressed comments (11)

app/controllers/PackageController.scala:141

• This internal exception message is returned to the client, potentially exposing SQL/schema or other implementation details. Keep the exception in logs and return a stable generic detail to callers.

        "detail" -> JsString(
          err.getMessage
        ) // TODO - tighten this up when we are done testing

app/controllers/PackageController.scala:124

• The list error path independently returns err.getMessage, so database and serialization failures can expose internal details even when genericErrorHandler is corrected. Return a fixed error detail here as well.

              "status" -> JsString("error"),
              "detail" -> JsString(
                err.getMessage
              ) // TODO - tighten this up when we are done testing

app/controllers/PackageController.scala:208

• The create error path also sends err.getMessage in a 500 response, which can expose internal database or application details. Return a stable generic detail and retain the exception only in the server log.

            "status" -> JsString("error"),
            "detail" -> JsString(
              err.getMessage
            ) // TODO - tighten this up when we are done testing

app/controllers/PackageController.scala:66

• The date and limit query values are parsed before the try block, so malformed input such as limit=abc or an invalid date escapes the action instead of returning a 400; negative limits are also not rejected before reaching SQL. Parse these values safely and validate their range before querying.

    val maybeDate =
      req.getQueryString("date").map(OffsetDateTime.parse(_, dateFormatter))
    val strictDate = req.getQueryString("strict").isDefined
    val maybeTitleSearch = req.getQueryString("title")
    val limit = req.getQueryString("limit").map(_.toInt).getOrElse(200)

app/controllers/PackageController.scala:334

• The documented behavior says region updates are a no-op for Story Packages, but when feastMetadata is absent this fallback creates a new FeastPackageMetadata and writes it to feast_metadata. A story package will therefore be mutated and change type; detect the non-Feast case and return without updating (or reject it).

          newMeta = maybeUpdate.getOrElse(
            FeastPackageMetadata(
              excludedRegions = req.body.excludedRegions,
              targetedRegions = req.body.targetedRegions
            )
          ),

app/controllers/PackageController.scala:243

• updatePackageMeta returns the affected-row count, but it is ignored here, so a PUT /packages/:id/metadata for an unknown package returns 204 instead of the 404 used by the other package update handlers. Check the count and return NotFound when it is zero.

        db.updatePackageMeta(id, req.body, req.user.username, req.user.email)
        NoContent

app/controllers/PackageController.scala:286

• For a nonexistent package, updatePackageName updates zero rows, then the assertion fails and the controller converts it into a 500 response. The other package update endpoints return 404 for an unknown ID, so check the update count and map this case to NotFound.

      db.updatePackageName(
        id,
        decoder.decode(req.body.asByteBuffer).toString,
        req.user.username,
        req.user.email

app/services/editions/db/PackageQueries.scala:133

• The API documentation says this handler returns 410 Conflict, but the controller's psqlErrorHandler returns HTTP 409 for conflicts. Update the documentation to avoid giving clients the wrong status code.

    * controller catches this with `psqlErrorHandler` and returns a 410 Conflict

conf/logback.xml:22

• This switches stdout from the Logstash encoder to a plain text encoder even though the application emits structured markers (for example via StructuredLogger). Any production log ingestion expecting JSON fields will lose them. Keep the structured encoder for production or make the format environment-specific.

        <encoder>
            <pattern>%date [%thread] %-5level %logger{36} - %msg%n%xException{3}</pattern>
        </encoder>
        <!-- <encoder class="net.logstash.logback.encoder.LogstashEncoder" />-->

docs/Packages API.md:24

• Correct the typo in the documentation: Editiorial should be Editorial.

deliberately broad in order that MRR and Editiorial can use them however they want.

docs/Packages API.md:40

• targetting is misspelled; use targeting.

Packages and effectively a no-op for Story Packages, since they do not have the relevant targetting fields

• Files reviewed: 36/36 changed files
• Comments generated: 19
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[davidfurey]

This seems like valid feedback from copilot

[github-actions]

Deploy build 9973 of cms-fronts::facia-tool to CODE

All deployment options

• Deploy build 9973 of cms-fronts::facia-tool to CODE
• Deploy parts of build 9973 to CODE by previewing it first
• What's on CODE right now?

---

From guardian/actions-riff-raff.

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate defects remain in package operations, SQL queries, error handling, metadata handling, and client types.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (8)

Previously missed (4) — in code that hasn't changed since the last review.

app/controllers/PackageController.scala:263

• updatePackageMeta returns the affected-row count, but this action discards it and always returns 204. A valid UUID for a missing package is therefore reported as a successful update; check the count and return 404, as the hidden-state and region endpoints do.
  app/services/editions/db/PackageQueries.scala:128
• For a nonexistent package, the UPDATE affects zero rows and the subsequent fetch is empty, so this assertion throws. PackageController.updateName catches that Throwable and returns 500 instead of 404; return an explicit not-found result (or an affected-row/Option result) for this case.
  app/services/editions/db/PackageQueries.scala:509
• The documented and tested query parameter is order=title, but this parser only recognizes name. order=title therefore silently falls back to created_on rather than sorting by package name; accept the title alias or change the public contract.
  docs/Packages API.md:24
• Typo: Editiorial should be Editorial.

app/controllers/PackageController.scala:463

• The result of updatePackageContent is the affected-row count, but it is discarded here. An empty patch for a nonexistent package therefore returns 200/status: ok instead of 404; use the count to handle the missing package consistently with the other mutation endpoints.

        db.updatePackageContent(
          packageId,
          adds,
          removes,
          req.user.username,
          req.user.email
        )

app/model/packages/PackageMetadata.scala:53

• Because every Feast metadata field is optional and Play JSON ignores unknown fields, a Web-shaped payload such as { "headline": "..." } successfully reads as an empty FeastPackageMetadata. Consequently PUT /packages/:id/metadata always selects feast_metadata for Web payloads, so WebPackageMetadata is never reachable. Add an explicit discriminator or strict shape detection before selecting the subtype.

    override def reads(json: JsValue): JsResult[PackageMetadata] = {
      FeastPackageMetadata.format.reads(json) orElse WebPackageMetadata.format
        .reads(json)

app/services/editions/db/PackageQueries.scala:348

• atIndex is client-controlled, but ListBuffer.insert requires an index between zero and the current list length. An out-of-range PATCH therefore throws IndexOutOfBoundsException and is returned as a generic 500 instead of a validation error; reject invalid indices before reindexing.

    packageCards.foreach(card =>
      intermediateIndices.insert(card.index, (card.pageCode, card.index))
    )

fronts-client/src/types/FaciaApi.ts:171

• Recipe cards omit metadata from the TypeScript interface, although the backend ClientPackageCard accepts and returns it and the package protocol includes metadata on recipe items. TypeScript consumers cannot type a valid recipe card carrying metadata; add the optional field to this variant.

• Files reviewed: 39/40 changed files
• Comments generated: 7
• Review effort level: Lite

[davidfurey]

Why is this called pageCode when it is only a pageCode for articles?

[fredex42]

to follow the naming convention used elsewhere. We can change it to something more generic, but I started off trying to keep the models as close as possible

[davidfurey]

Is GetPackagesFilter used?

[fredex42]

I think you're right, i'll zap it if unused

[fredex42]

⚡

[davidfurey]

Would it make more sense to have a single JSONB metadata, and a separate field to indicate the package type?

[davidfurey]

What is prefill?

[fredex42]

prefill is copied over from the existing Editions model. In an article container I believe it's a CAPI query to programatically fill the container and in Feast it would be used as a search to prefill the container. I can remove it if you'd rather as it doesn't have a wired in use at the moment

[davidfurey]

Fair enough, I know this as backfill

[davidfurey]

Perhaps case s if s == PSQLState.UNIQUE_VIOLATION.getState => to avoid a magic number?

[davidfurey]

Instead of accessing getQueryString you could make the capabilities of this endpoint obvious in the routes file by passing the query string values as parameter values to listPackages (https://www.playframework.com/documentation/3.0.x/ScalaRouting#Call-to-the-Action-generator-method)
e.g. listPackages(id: Option[String],...

[davidfurey]

You've used this enough to make it a case class with a json formatter

[davidfurey]

It would be better if the code was structured so that the compiler was making that guarantee

[davidfurey]

Do you really want any package have a failure to cause the whole request to fail?