feat(red-team): import approved generated cases (#216)

Author: ccarvalho-eng

README.md

@@ -160,7 +160,23 @@ Generate cases for a product-specific context without writing to the database, t
 
 Generation is bounded by category count, cases per category, context size, requests, output tokens, observed total tokens, observed cost, response size, and per-request timeout. Valid categories remain reviewable when another category fails. Each candidate and the complete generation carry checksums for stable review. Usage is observed from successful provider responses; failed or timed-out external requests may still incur provider-side usage that Aludel cannot report.
 
-Catalog materialization and generated-case review are Elixir API features. Materialized curated cases use the normal dataset and suite workflows in the dashboard, `mix aludel.eval`, ExUnit, and the library API. There is no separate red-team CLI command or generation form in the dashboard yet.
+After reviewing the candidates, explicitly approve their stable IDs and import them atomically:
+
+```elixir
+approved_case_ids =
+  generation.cases
+  |> Enum.filter(&approved_by_reviewer?/1)
+  |> Enum.map(& &1.id)
+
+{:ok, %{created: created, skipped: skipped}} =
+  Aludel.RedTeam.import_generated(dataset, generation,
+    approved_case_ids: approved_case_ids
+  )
+```
+
+Import revalidates the generation and candidate checksums, requires at least one unique approved ID, attaches each candidate's recommended rubric judge, records generation and review provenance, and skips only an exact prior import. Any conflict rolls back the complete approved selection.
+
+Catalog materialization and generated-case generation, review, and import are Elixir API features. Persisted cases use the normal dataset and suite workflows in the dashboard, `mix aludel.eval`, ExUnit, and the library API. There is no separate red-team CLI command or generation/import form in the dashboard yet.
 
 See the [red-team guide](https://hexdocs.pm/aludel/red_team.html), [curated datasets wiki guide](https://github.com/ccarvalho-eng/aludel/wiki/Red-Team-Datasets), and [generated cases wiki guide](https://github.com/ccarvalho-eng/aludel/wiki/Generated-Red-Team-Cases) for category filters, review, budgets, provenance, and rerun behavior.
 

guides/evaluations.md

@@ -60,7 +60,7 @@ For security regression coverage, you can materialize versioned adversarial case
 
 See the [red-team guide](red_team.md) for the complete catalog and deduplication behavior.
 
-For product-specific cases, call `Aludel.RedTeam.generate/2` and review the returned `generation.cases`, `generation.failures`, usage, and limits before authoring dataset entries. Generation never writes to the dataset directly. The [red-team guide](red_team.md#generated-cases) covers limits and partial failures.
+For product-specific cases, call `Aludel.RedTeam.generate/2` and review the returned `generation.cases`, `generation.failures`, usage, and limits. Generation never writes to the dataset directly. After review, `Aludel.RedTeam.import_generated/3` requires explicit candidate IDs and atomically creates ordinary dataset entries with a rubric judge and complete provenance. The [red-team guide](red_team.md#generated-cases) covers limits, partial failures, approval, and import.
 
 ### Contains and excludes
 

guides/features.md

@@ -93,9 +93,9 @@ Dataset pages support create, edit, delete, entry management, and JSON containme
 
 `Aludel.RedTeam` provides seven versioned adversarial cases covering direct and indirect prompt injection, system prompt leakage, sensitive information disclosure, excessive agency, misinformation, and unsafe assistance. The library API materializes selected cases into a reusable dataset with deterministic canary assertions, optional rubric judges, and category, severity, provenance, checksum, and deduplication metadata. Matching reruns are idempotent; content or judge-configuration drift under the same key fails explicitly. See the [red-team guide](red_team.md).
 
-The same API can generate product-specific cases through an Aludel provider. Generation validates a strict response schema, bounds calls and output, reports sanitized partial failures and usage, and returns inert checksummed candidates without database writes or execution.
+The same API can generate product-specific cases through an Aludel provider. Generation validates a strict response schema, bounds calls and output, reports sanitized partial failures and usage, and returns inert checksummed candidates without database writes or execution. A separate import call requires explicit approved candidate IDs, revalidates the review record, and creates the selected entries atomically with rubric judges and provenance.
 
-Materialized entries use the normal dataset workflow in the dashboard. Once they have been copied into a suite, that suite can be executed through the dashboard, `mix aludel.eval`, ExUnit, or `Aludel.Evals`. Catalog browsing, materialization, and generated-case review are currently library API features.
+Materialized and approved generated entries use the normal dataset workflow in the dashboard. Once they have been copied into a suite, that suite can be executed through the dashboard, `mix aludel.eval`, ExUnit, or `Aludel.Evals`. Catalog browsing, materialization, generated-case review, and approval/import are currently library API features.
 
 ## Documents and storage
 

guides/red_team.md

@@ -142,7 +142,34 @@ IO.inspect(generation.limits)
 
 Failures contain only the category, stable failure type, and a safe message. Raw provider errors and malformed model output are not retained. The raw target context is represented by a checksum in the generation result. Each candidate and the complete generation record have checksums for stable review.
 
-Generation deliberately stops at the review boundary: it does not create, update, delete, or execute dataset entries. Use the candidate prompt, rationale, category, severity, technique, and recommended judge as review inputs before authoring an evaluation case.
+Generation deliberately stops at the review boundary: it does not create, update, delete, or execute dataset entries. Use the candidate prompt, rationale, category, severity, technique, and recommended judge as review inputs.
+
+### Approve and import candidates
+
+Import requires the stable IDs of at least one explicitly approved candidate:
+
+```elixir
+approved_case_ids =
+  generation.cases
+  |> Enum.filter(&approved_by_reviewer?/1)
+  |> Enum.map(& &1.id)
+
+{:ok, %{created: created, skipped: skipped}} =
+  Aludel.RedTeam.import_generated(dataset, generation,
+    approved_case_ids: approved_case_ids,
+    variable: "input",
+    judge_provider_id: generator_provider.id,
+    judge_threshold: 80
+  )
+```
+
+The judge provider defaults to the provider that generated the candidates. Each imported case receives its recommended built-in rubric judge. You can select another provider UUID and a threshold from 0 through 100.
+
+Before locking the dataset, Aludel revalidates the complete generation checksum, its outcome accounting, and every candidate checksum. Candidate IDs must be non-empty, unique, and present in that generation. Approved cases retain generation status, failures, observed usage, applied limits, provider and model identity, target-context checksum, rationale, classification, review evidence, and import checksums in metadata. Raw target context is not copied into the entry.
+
+The complete approved selection is written atomically in generation order. Repeating the same import returns the existing entries in `skipped`. A changed payload, generation receipt, review record, variable, or judge configuration under the same deduplication key returns `{:error, {:deduplication_conflict, key}}` and rolls back every new entry in that call.
+
+Generation and approval/import are Elixir API features. The dashboard can inspect and edit imported entries and populate suites from their dataset, while `mix aludel.eval`, ExUnit, and the evaluation API run the resulting persisted suite. There is no separate generation or import command in the Mix CLI and no generation/import form in the dashboard.
 
 ## Use the entries
 

lib/aludel/red_team.ex

@@ -7,14 +7,15 @@ defmodule Aludel.RedTeam do
   same case version and prompt variable. It records provenance, risk category,
   severity, content checksum, and a stable deduplication key in entry metadata.
 
-  Curated cases include deterministic canary assertions. Generated cases are
-  returned as inert review values without database writes or execution.
+  Curated cases include deterministic canary assertions. Generation returns
+  inert review values without database writes or execution; a separate import
+  call requires explicit approved candidate IDs.
   """
 
-  import Ecto.Query
-
   alias Aludel.Datasets.{Dataset, DatasetEntry}
   alias Aludel.RedTeam.Catalog
+  alias Aludel.RedTeam.DatasetImporter
+  alias Aludel.RedTeam.GeneratedImporter
   alias Aludel.RedTeam.Generation
   alias Aludel.RedTeam.Generator
   alias Ecto.Changeset
@@ -34,7 +35,7 @@ defmodule Aludel.RedTeam do
           | :invalid_judge_provider_id
           | :invalid_judge_threshold
           | {:unknown_categories, [term()]}
-          | {:unknown_case_ids, [term()]}
+          | {:unknown_case_ids, [String.t()]}
           | {:deduplication_conflict, String.t()}
           | Changeset.t()
   @type generate_error ::
@@ -45,6 +46,17 @@ defmodule Aludel.RedTeam do
           | :invalid_target_context
           | :invalid_budget
           | {:unknown_categories, [term()]}
+  @type generated_import_error ::
+          :dataset_not_found
+          | :invalid_generation
+          | :invalid_options
+          | :invalid_approved_case_ids
+          | :invalid_variable
+          | :invalid_judge_provider_id
+          | :invalid_judge_threshold
+          | {:unknown_case_ids, [term()]}
+          | {:deduplication_conflict, String.t()}
+          | Changeset.t()
 
   @doc """
   Returns every case in stable catalog order.
@@ -106,6 +118,30 @@ defmodule Aludel.RedTeam do
     Generator.generate(provider_id, opts)
   end
 
+  @doc """
+  Imports explicitly approved generated cases into an existing dataset.
+
+  The generation checksum and every approved case checksum are revalidated
+  before the dataset is locked. The complete approved selection is written
+  atomically and retains generation, review, usage, limit, and judge provenance.
+
+  Options:
+
+    * `:approved_case_ids` - required non-empty unique list of candidate IDs
+    * `:variable` - prompt variable populated by each case; defaults to `"input"`
+    * `:judge_provider_id` - provider UUID for rubric judging; defaults to the generator provider
+    * `:judge_threshold` - rubric judge pass threshold from 0 to 100; defaults to 80
+
+  Repeating an exact import skips its existing entries. Changed payload,
+  provenance, review, or judge configuration under the same key returns a
+  conflict and rolls back the entire selection.
+  """
+  @spec import_generated(Dataset.t(), Generation.t(), keyword()) ::
+          {:ok, materialization()} | {:error, generated_import_error()}
+  def import_generated(dataset, generation, opts \\ []) do
+    GeneratedImporter.import(dataset, generation, opts)
+  end
+
   @doc """
   Materializes selected catalog cases into an existing dataset.
 
@@ -245,73 +281,31 @@ defmodule Aludel.RedTeam do
   end
 
   defp persist_cases(dataset_id, selected, opts) do
-    repo().transaction(fn ->
-      case lock_dataset(dataset_id) do
-        nil -> repo().rollback(:dataset_not_found)
-        locked_dataset -> materialize_locked(locked_dataset, selected, opts)
-      end
-    end)
+    prepared_entries = Enum.map(selected, &prepared_entry(&1, opts))
+    DatasetImporter.persist(dataset_id, prepared_entries)
   end
 
-  defp materialize_locked(dataset, selected, opts) do
-    keys = Enum.map(selected, &deduplication_key(&1, opts[:variable]))
-    entries = list_entries_with_keys(dataset.id, keys)
-    next_position = next_position(dataset.id)
+  defp prepared_entry(template, opts) do
+    key = deduplication_key(template, opts[:variable])
 
-    selected
-    |> Enum.reduce_while({[], [], next_position}, fn template, {created, skipped, position} ->
-      case materialize_template(dataset.id, template, opts, entries, position) do
-        {:created, entry} ->
-          {:cont, {[entry | created], skipped, position + 1}}
-
-        {:skipped, entry} ->
-          {:cont, {created, [entry | skipped], position}}
-
-        {:error, reason} ->
-          repo().rollback(reason)
-      end
-    end)
-    |> then(fn {created, skipped, _position} ->
-      %{created: Enum.reverse(created), skipped: Enum.reverse(skipped)}
-    end)
+    %{
+      deduplication_key: key,
+      attrs: entry_attrs(template, opts)
+    }
   end
 
-  defp entry_attrs(template, opts, position) do
+  defp entry_attrs(template, opts) do
     variable = opts[:variable]
     assertions = assertions(template, opts)
 
     %{
       name: template.name,
       variable_values: %{variable => template.prompt},
       assertions: assertions,
-      metadata: metadata(template, variable, opts),
-      position: position
+      metadata: metadata(template, variable, opts)
     }
   end
 
-  defp materialize_template(dataset_id, template, opts, entries, position) do
-    attrs = entry_attrs(template, opts, position)
-    key = get_in(attrs, [:metadata, "red_team", "deduplication_key"])
-
-    case entries_with_key(entries, key) do
-      [] ->
-        case insert_entry(dataset_id, attrs) do
-          {:ok, entry} -> {:created, entry}
-          {:error, changeset} -> {:error, changeset}
-        end
-
-      [entry] ->
-        if matches_materialization?(entry, attrs) do
-          {:skipped, entry}
-        else
-          {:error, {:deduplication_conflict, key}}
-        end
-
-      _duplicates ->
-        {:error, {:deduplication_conflict, key}}
-    end
-  end
-
   defp assertions(template, opts) do
     case opts[:judge_provider_id] do
       nil ->
@@ -372,67 +366,6 @@ defmodule Aludel.RedTeam do
 
   defp deduplication_key(template, variable) do
     "aludel:red_team:#{Catalog.name()}@#{Catalog.version()}:#{template.id}@#{template.version}:#{variable}"
-  end
-
-  defp lock_dataset(dataset_id) do
-    Dataset
-    |> where([dataset], dataset.id == ^dataset_id)
-    |> lock("FOR UPDATE")
-    |> repo().one()
-  end
-
-  defp list_entries_with_keys(_dataset_id, []) do
-    []
-  end
-
-  defp list_entries_with_keys(dataset_id, keys) do
-    DatasetEntry
-    |> where(
-      [entry],
-      entry.dataset_id == ^dataset_id and entry.red_team_deduplication_key in ^keys
-    )
-    |> repo().all()
-  end
-
-  defp entries_with_key(entries, key) do
-    Enum.filter(entries, fn entry ->
-      entry.red_team_deduplication_key == key
-    end)
-  end
-
-  defp matches_materialization?(entry, attrs) do
-    entry.name == attrs.name and
-      entry.variable_values == attrs.variable_values and
-      entry.messages == [] and
-      entry.assertions == attrs.assertions and
-      entry.red_team_deduplication_key ==
-        get_in(attrs, [:metadata, "red_team", "deduplication_key"]) and
-      entry.metadata["red_team"] == attrs.metadata["red_team"]
-  end
-
-  defp next_position(dataset_id) do
-    DatasetEntry
-    |> where([entry], entry.dataset_id == ^dataset_id)
-    |> select([entry], max(entry.position))
-    |> repo().one()
-    |> case do
-      nil -> 0
-      position -> position + 1
-    end
-  end
-
-  defp insert_entry(dataset_id, attrs) do
-    deduplication_key = get_in(attrs, [:metadata, "red_team", "deduplication_key"])
-
-    %DatasetEntry{
-      dataset_id: dataset_id,
-      red_team_deduplication_key: deduplication_key
-    }
-    |> DatasetEntry.changeset(attrs)
-    |> repo().insert()
-  end
-
-  defp repo do
-    Aludel.Repo.get()
+    |> DatasetImporter.bounded_key()
   end
 end