Add Perplexity-enabled TerminalBench agent

[neubig]

Note

Superseded by OpenHands/research#81. The experiment-specific agent, matched configs, runtime builder, analyzer, provenance, and corrected report now live in the research repository rather than the production benchmarks package. This PR's agent/perplexity-cli-terminalbench branch is intentionally retained as historical run provenance.## Summary

• add a custom Harbor OpenHands SDK agent with a pinned, checksummed pplx binary
• forward PERPLEXITY_API_KEY only to the task-solving agent process
• safely handle leading-dash and punctuation-heavy bootstrap queries
• add a common verifier-ready OpenHands SDK base used identically by control and PPLX arms
• stage pinned uv 0.9.5 and managed Python 3.13.9 so verifier startup survives live installer failures

Validation

• uv run pytest -q tests -> 556 passed
• focused Ruff checks passed
• blocked-GitHub Bullseye simulation: upstream uv installation failed, but seeded uvx -p 3.13 -w pytest==8.4.1 completed with 1 passed
• matched smoke: control 31316591210 and PPLX 31316592925; all six verifiers reached pytest with zero fatal bootstrap failures

Corrected full TerminalBench evidence

The verifier-stable matched 89-task run used MiniMax M2.7 and SDK 43376f1868ffd702746080714a59c16d3f69ec12:

• control 31317411753: 40/89
• PPLX 31317413372: 50/89
• paired transitions: 14 treatment-only, 4 control-only, 36 both pass, 35 both fail
• difference: +10 tasks / +11.24 percentage points
• exact two-sided McNemar: p=0.0308837891
• verifier audit: 178/178 reward records, zero uvx/Python/read-only fatal signatures; 46 live uv-download failures still continued through tests using seeded tools

This replaces the earlier 34–38 run, which was confounded by asymmetric verifier failures. The corrected result remains one stochastic run and is heavily contaminated by benchmark lookup: 66/69 inspectable PPLX payloads contained the exact task slug and 67/69 contained TerminalBench/Harbor material. Treat it as integration/experiment evidence, not a general benchmark claim.

Companion evaluation change: https://github.com/OpenHands/evaluation/pull/601

Closes #767

This PR description update was generated by an AI agent (OpenHands) on behalf of neubig.

[neubig]

Superseded by OpenHands/research#81. I am leaving agent/perplexity-cli-terminalbench intact so the exact code used for the recorded workflows remains available.

This comment was created by an AI agent (OpenHands) on behalf of Graham Neubig.