Tutorial for autotrain

[parnianz]

No description provided.

[Copilot]

Pull request overview

Adds a new documentation tutorial page describing “AutoTrain”, a multi-agent AutoML workflow that combines a FastAPI frontend with a three-stage Flyte pipeline and agent-driven iterative experimentation, intended for the Union.ai/Flyte docs tutorials section.

Changes:

• Introduces a new tutorial page explaining the AutoTrain architecture (frontend + pipeline + 3 agents).
• Adds multiple {{< code ... >}} shortcode inclusions that pull referenced snippets from unionai-examples.
• Documents deployment, prerequisites (secrets), usage, and debugging steps for running the app/pipeline.

Comments suppressed due to low confidence (5)

content/tutorials/agents/auto_train/_index.md:17

• The page is included in both variants but this paragraph hard-codes “Union UI” and “Union project/domain”. Using {{< key product_name >}} keeps the wording accurate across Flyte vs Union renders.

AutoTrain automates that whole loop behind a simple web interface: a user submits a dataset link and a few fields, and a FastAPI app kicks off a three-stage Flyte pipeline. A **data agent** profiles and cleans the dataset, a **design agent** turns that profile into a concrete experiment plan and a starting `train.py`, and a **research agent** runs an iterative improve-and-evaluate loop with the Claude Agent SDK, committing progress to GitHub and opening a PR with the results. Every step is a task or trace visible in the Union UI. Architecturally, this is a hybrid app-task graph: the frontend and the pipeline's tasks are deployed to the same Union project/domain, and the app submits runs programmatically (`flyte.run.aio()`) on the user's behalf, then polls the cluster to report progress back to them.

content/tutorials/agents/auto_train/_index.md:166

• This line hard-codes “Union UI” even though the page is rendered for both Flyte and Union variants. Using the product_name key keeps the docs consistent across variants.

Every step of the loop - CLI setup, git clone, the baseline implementation, each change proposal, each training run, crash fixes, commits, PR creation, the final convergence check - is wrapped in `@flyte.trace`, so it shows up as a distinct traced action in the Union UI with its own inputs, outputs, and timing, all nested inside the single `run_research` task. Because a trace's identity is `(function name, input hash)`, every per-experiment call takes `exp_id` (and `run_training` an attempt counter) as an explicit input - otherwise a repeated call with the same arguments would replay a cached result instead of re-running.

content/tutorials/agents/auto_train/_index.md:263

• This section hard-codes “Union UI” / “Why Union?” even though the page is included in both +flyte and +union variants. Prefer the product_name key so the rendered docs match the active variant.

- **Union UI**: a live report on the research task - the running trace of what each experiment tried, a Performance tab charting the metric across experiments, and full logs for every stage.
- **GitHub**: an experiment branch with the full history of `train.py` revisions (one commit per improvement) and a PR summarizing the best result, ready to review or merge.
- **The status page**: the best metric value, a link to the PR, and - if the agent's own convergence check flagged the result as unsatisfactory - a plain-language explanation of what went wrong and what to try next.

## Why Union?

content/tutorials/agents/auto_train/_index.md:269

• Same variant consistency issue: “Union UI” is hard-coded on a page that renders for both variants.

**Observability into an unattended, multi-hour process.** The research loop can run for a long time with no one watching. Every internal step - installing the CLI, cloning the branch, implementing the baseline, each change proposal, each training run, crash fixes, commits, PR creation, the final convergence check - is wrapped in `@flyte.trace`, so it shows up as its own action in the Union UI with inputs, outputs, and timing, nested inside the single task container. If experiment 14 crashes, you're not left with a bare stack trace - you can see exactly what the previous 13 experiments tried and why.

content/tutorials/agents/auto_train/_index.md:277

• This concluding sentence hard-codes “Union” even though the page is rendered for both Flyte and Union variants. Using the product_name key avoids variant-specific branding in shared content.

AutoTrain is a specific example of a more general pattern: an LLM decides what to try next, and Union handles how that work actually runs - on the right compute, with the right secrets, with every step traceable, whether the loop takes five minutes or five hours.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ppiegaze]

@parnianz I reviewed this in detail. Nice tutorial, and the hybrid app-task graph it demonstrates is genuinely the most interesting thing in it. A few things need fixing before it can merge, starting with two that block it outright.

Blockers

1. The examples submodule points at an unmerged branch. This PR bumps unionai-examples to 032c3387, which only exists on docs/auto-train-fragments (unionai-examples#292, still open). That commit is what adds the {{docs-fragment ...}} markers; main doesn't have them. All 17 {{< code >}} shortcodes on the page depend on those markers, and a missing fragment calls errorf, which fails the Hugo build outright rather than warning.

So as it stands, the next routine make update-examples breaks the docs build for everyone. #292 needs to merge first, then re-point this submodule at the resulting main commit.

2. CI has never run on the current head. There are zero check runs on 5ba4b7ad. The only CI here is from July 30 against a commit three back, and it failed DCO: none of the three commits have a Signed-off-by line, and two also have a laptop hostname as the author email (parnianzargham@mac.lan). Fix with git rebase --signoff <base> and force-push. That'll also tell us whether the images-health failure from that run is fixed (it probably is; the image landed in a later commit).

Three things in the tutorial that won't work if a reader copies them

flyte create secret takes the value as an option, not a second argument. The page has:

flyte create secret --project flytesnacks --domain development internal-anthropic-api-key <key>

_create.py declares one positional (name); the value is --value. Two positionals makes click error with "Got unexpected extra argument." Should be ... internal-anthropic-api-key --value <key>. Same for the GitHub token. The upstream README at v2/tutorials/auto_train/ has the same error, worth fixing both.

The curl sends the wrong field name. It posts max_samples_raw=100, but the parameter is Form(None, alias="max_samples"), so the wire name is max_samples (the HTML template uses that too). As written the value is silently dropped and max_samples falls back to 0, meaning "all data" — so the "quick test" isn't quick. Your own source comment two lines above the fragment marker warns about exactly this. Should be -d "max_samples=100".

The documented setup leaves /run open to the internet. app.py defines WEBHOOK_API_KEY with HTTPBearer, but verify_token returns early when the key is empty, so auth is off by default. The tutorial's deploy command doesn't set it and the curl sends no Authorization header. Following the page as written gives you a publicly reachable endpoint that spawns multi-hour T4 GPU runs for anyone who finds the URL. The hybrid-graphs page this tutorial links to says "always secure webhooks with authentication" and "store API keys in Flyte secrets, not in code." Worth adding WEBHOOK_API_KEY to the prerequisites and the Authorization header to the curl. (Also, the source comment says union create secret --name automl-webhook-key, mixing the Union CLI into an otherwise-flyte tutorial.)

Screenshots

Two don't match what the text says:

• training_analysis.png — the caption describes the failure case ("a plain-language explanation of what went wrong and what to try next"), but the image shows the success state: "Training complete", ROC_AUC 0.908552, three buttons, no explanation anywhere. Either capture the unsatisfactory state or rewrite the caption.
• training_steps.png — the text says "the Task tab shows every sub-run", but the active tab in the shot is Summary; the nested list is the left-hand run tree. This one also shows internal infra: the union-oc-production-demo bucket with a user prefix, Cluster: oc-production, a pod and action ID, and Owned By: demo-EAGER_API_KEY-oc-production. No live secret, but that last string reads like a leaked credential to anyone skimming. Worth recapturing from a clean demo-tenant state or cropping the Input panel.

Also auto_train.png (the hero illustration) contradicts the page: it shows "F1 Score", "Model: XGBoost" and hyperparameters like lr 0.01 / batch 256, while the tutorial says the metric comes from roc_auc/macro_f1/rmse and the design agent emits a data-loading skeleton with no hyperparameters.

Smaller things

• Variant. Frontmatter is +flyte +union but "Union" is hardcoded in 8 places including the H1. Restricting the variant would be wrong — apps and hybrid-graphs both ship in the Flyte variant — so use {{% key product_name %}}. Important: in the H1 you need the percent form, not {{< >}}; the Check Build Determinism job rejects angle-bracket shortcodes in headings. Note three screenshots render "Union" in the image pixels, which no key can fix.
• Directory name. auto_train — all 12 existing agent tutorials are hyphenated. Rename to auto-train now; after publish it needs a redirect.
• preformance_tab.png is a typo for "performance", and it's in the committed path.
• Three fragment ranges don't show what the prose promises. The biggest is submit_run_endpoint: the start marker is inside the function body, so the rendered block omits @app.post("/run"), the whole signature, and the alias="max_samples" comment. Moving that marker up would make two of the problems above self-evident to a reader.
• Headings are Title Case; the sibling tutorials use sentence case.
• 28 instances of - used as sentence punctuation. Reads like a find/replace of em-dashes; the house style is to restructure into a comma, colon, or new sentence.

One editorial thought, take it or leave it: this overlaps a fair bit with autoresearch, parallelized-autoresearch-agent, and mle-bot, which are all "agent iterates on train.py, commits, opens a PR." What makes this one different is the FastAPI frontend submitting through flyte.run.aio(), which is currently buried under the AutoML narrative. Leading with that, and cross-linking the siblings, would help readers tell them apart.

---

— docsy · automated docs agent