eval_dataset.sql

{{ config(materialized='table') }}

-- =============================================================================
-- eval_dataset -- the input table for EXECUTE_AI_EVALUATION
-- =============================================================================
-- Turns the eval_ground_truth seed into the two columns Cortex Agent evaluation
-- expects: a VARCHAR question and a VARIANT ground truth.
--
-- The ground_truth column MUST be VARIANT, and it must be built with PARSE_JSON
-- rather than OBJECT_CONSTRUCT.
--
-- -----------------------------------------------------------------------------
-- WHAT IS INSIDE THE VARIANT
--
-- One VARIANT feeds every metric. Each metric reads only the keys it knows, and
-- ignores the rest, so all six metrics share this single column.
--
--   ground_truth_output
--       The expected final answer, written as a plain-language rubric with
--       literal verifiable values. Read by the built-in answer_correctness
--       metric. Because it is fed into a judge prompt, state tolerances,
--       units, rounding and scoping explicitly.
--
--   ground_truth_invocations
--       THE key for tool scoring, and the one most likely to be got wrong.
--       An ARRAY of objects, one per expected tool call:
--           [{"tool_name": "...", "tool_input": "...", "tool_output": "..."}]
--       Read by the built-in tool_selection_accuracy (TSA) and
--       tool_execution_accuracy (TEA) metrics.
--         * tool_name  is required. Use the name the AGENT sees. For a Cortex
--                      Analyst tool the semantic view name also matches. For
--                      web search the fixed literal is web_search.
--         * tool_input  optional. The natural-language question you expect the
--                      agent to pass, NOT SQL you author. Graded by TEA only.
--         * tool_output optional. What you expect back: result rows, a SQL
--                      description, citations. Graded by TEA only.
--       Omit tool_input and tool_output and TEA only confirms the tool ran.
--       Use the EMPTY ARRAY [] to assert that NO tool should be called.
--
--       TSA scores matched / max(expected count, actual count), so it penalizes
--       too few calls, too many calls, and wrong calls. Ordering is ignored.
--       TEA semantically matches each expected entry to the closest real call;
--       an expected tool that never ran scores 0.0, while EXTRA tools do not
--       penalize TEA at all. That asymmetry is why you run both.
--
--   expected_class, trap
--       Custom keys, ignored by every built-in metric. Custom metrics receive
--       the WHOLE variant through {{ '{{ground_truth}}' }} regardless of key, so the
--       workflow_tool_routing and workflow_tool_arguments rubrics read these.
--         expected_class  the triage class the skill should assign, so a judge
--                         can score the classification step, not just outcome.
--         trap            present only on rows designed to be failed. Names the
--                         wrong behavior, so a reviewer can tell a real
--                         regression from a merely hard question.
--
-- Keep OUTPUT criteria in ground_truth_output and PROCESS criteria in your own
-- custom keys. That split is what keeps the built-in metrics honest.
--
-- -----------------------------------------------------------------------------
-- THE TWO MOST VALUABLE ROWS
--
-- Two rows carry ground_truth_invocations = []: "Summarize the issue" (no id
-- given) and "What was total revenue in 2025?" (period not in the data). An
-- agent that reflexively calls a tool on every turn will pass the answer evals
-- and fail these. They are the cheapest regression insurance in the set.
--
-- -----------------------------------------------------------------------------
-- ONE DATASET PER SOURCE TABLE
--
-- The dataset version name is a fixed constant
-- (SYSTEM_AI_OBS_CORTEX_AGENT_DATASET_VERSION_DO_NOT_DELETE), so a second
-- dataset built on THIS table fails with "Dataset version ... already exists"
-- even under a different dataset_name. After the first run, either reuse the
-- existing dataset (drop the `dataset:` block from the config) or point at a
-- different table.
-- =============================================================================

SELECT
    input_query,
    PARSE_JSON(ground_truth_json) AS ground_truth
FROM {{ ref('eval_ground_truth') }}