Skip to content

Commit e1c6b5b

Browse files
author
Mengtao Zhang
committed
minors
1 parent e0d84c2 commit e1c6b5b

4 files changed

Lines changed: 259 additions & 3 deletions

File tree

‎DATA_PROVENANCE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ construction steps are documented in this repository.
4242

4343
## License in this repository
4444

45-
`environment/LICENSE` grants recipients permission from `ricefan-tech` to use,
45+
`environment/LICENSE` grants recipients permission from `EdotEnv` to use,
4646
copy, run, and modify the task code and bundled data solely for internal
4747
evaluation, research, and demonstration. It prohibits redistribution, public
4848
hosting, commercial deployment, and real-money trading without separate written

‎TASK_DESCRIPTION.md‎

Lines changed: 256 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,256 @@
1+
# Causal Feature Engineering for Quantitative Trading
2+
3+
## Scientific Domain
4+
5+
**Domain:** Quantitative Finance
6+
**Field:** Systematic Trading
7+
**Subfield:** Machine-Learned Alpha Research
8+
9+
## Scientific Problem
10+
11+
Quantitative researchers rarely succeed by fitting one model and accepting the
12+
largest public-period score. Minute-level market data are noisy, predictive
13+
relationships are weak, regimes change, and apparently strong signals can
14+
disappear after turnover, execution costs, or exposure normalization. The
15+
research problem is to discover causal predictive structure that remains useful
16+
outside the period in which it was designed.
17+
18+
This task compresses that research loop into a fixed, reproducible environment.
19+
The agent is expected to inspect market data, form feature hypotheses, run
20+
controlled chronological experiments, compare predictive and economic
21+
diagnostics, reject unstable candidates, and select one strategy for hidden
22+
future scoring. Public metrics may conflict: a candidate can have attractive
23+
information coefficient but poor after-cost performance, or a high Sharpe on
24+
one interval with weak sign and regime stability.
25+
26+
The final predictive estimator is deliberately fixed to an exact Elastic Net so
27+
that improvements must come primarily from causal feature construction,
28+
preprocessing, selection, regularization, and experimental judgment rather than
29+
from switching to a more powerful model family. Symbol identities, calendar
30+
coordinates, and target semantics are obscured to reduce historical lookup and
31+
force evidence-based research on the supplied panel.
32+
33+
## Workflow Details
34+
35+
The agent phase has two cooperating containers. The `main` container contains
36+
the public Parquet panel, task instructions, and the Python data-science stack.
37+
The `mcp-server` sidecar owns model fitting, artifact storage, public
38+
backtesting, research budgets, and final submission promotion. The agent
39+
interacts with that sidecar through five MCP tools:
40+
41+
- `train_model`
42+
- `get_train_model_result`
43+
- `backtest`
44+
- `get_backtest_result`
45+
- `submit_strategy`
46+
47+
The research workflow has four stages:
48+
49+
1. **Public research.** Inspect the 2022–2023 panel chronologically and by
50+
symbol. Form hypotheses for past-only transformations such as returns, lags,
51+
rolling statistics, expanding histories, price-range structure, activity,
52+
and order-flow imbalance. Keep every symbol at a timestamp in the same
53+
chronological fold and fit learned preprocessing on training rows only.
54+
2. **Source-bound training.** Send complete Python source defining
55+
`train_model(X, y)` to the asynchronous training tool. The returned fitted
56+
object must be an exact `ElasticNet` or an exact sklearn `Pipeline` ending in
57+
an exact `ElasticNet`. It must use `weight_std_dollar_vol` as fitting weight,
58+
remain serializable with Joblib, and expose a stable inference schema.
59+
3. **Controlled public backtesting.** Backtest accepted models on public windows
60+
strictly later than their training windows. Compare candidates on consistent
61+
periods using information coefficient, error, prediction dispersion,
62+
turnover, after-cost Sharpe, CAGR, and maximum drawdown. Use later slices to
63+
challenge apparently strong candidates rather than optimizing one interval.
64+
4. **Candidate selection.** Submit the `strategy_id` from one successful
65+
backtest after all asynchronous operations are idle. The sidecar promotes one
66+
immutable bundle containing the selected source, fitted public artifact,
67+
inference contract, strategy settings, and integrity hashes.
68+
69+
Training and backtesting share a budget of 100 accepted research attempts. One
70+
training operation and one backtest may overlap, but two operations of the same
71+
type may not. This permits deliberate parallel research without allowing
72+
unbounded search.
73+
74+
After the agent exits, Harbor collects the bundle and starts a separate
75+
verifier. The verifier checks bundle integrity, imports the submitted source in
76+
a networkless model worker, and calls `train_model` once on every public
77+
2022–2023 row. It serializes and reloads that newly fitted estimator, predicts
78+
on hidden 2024 feature matrices, audits causal invariance by perturbing future
79+
suffixes, and runs the hidden after-cost backtest. Hidden labels and
80+
scoring-only columns never enter the model worker.
81+
82+
## Dependencies & System Requirements
83+
84+
### Agent-facing model environment
85+
86+
Submitted model source runs on Python 3.12 and may import only `math`,
87+
`statistics`, `numpy`, `pandas`, and `sklearn`. The model must fit and predict
88+
without file I/O, external services, or network access. Source is limited to
89+
20,000 UTF-8 bytes; each fit and prediction operation has a 1,800-second
90+
deadline. The returned estimator must serialize with Joblib and predict after
91+
reload in a fresh process. No GPU is available or required.
92+
93+
The broader agent container includes the pinned Python scientific stack needed
94+
to inspect the Parquet data and communicate with the MCP service. Core versions
95+
are Python 3.12, NumPy 2.5.1, pandas 3.0.5, SciPy 1.18.0, scikit-learn 1.9.0,
96+
PyArrow 25.0.0, Joblib 1.5.3, and FastMCP 3.4.5.
97+
98+
### Task deployment environment
99+
100+
The sample targets Harbor 0.20.0 with Docker Compose and Git LFS. The task
101+
requests four CPU cores, 8 GiB of memory, 32 GiB of storage, and no GPU. The
102+
configured limits are 30 minutes for environment construction, four hours for
103+
the agent, and one hour for verification. The agent environment has public
104+
network access for general research, while submitted source executes in a
105+
networkless, read-only model worker with narrowly controlled imports and file
106+
operations.
107+
108+
## Dataset
109+
110+
The benchmark uses a fixed panel of historical cryptocurrency minute bars for
111+
four anonymized symbols. The canonical source universe is BTCUSDT, DOGEUSDT,
112+
ETHUSDT, and SOLUSDT, but mapped to `symbol_01` through `symbol_04`.
113+
Datetimes are replaced with synthetic UTC coordinates while preserving
114+
ordering and elapsed durations.
115+
116+
The chronological split is:
117+
118+
- **Public research:** 2022-01-01 through 2023-12-31, containing 4,204,792 panel
119+
rows.
120+
- **Hidden evaluation:** 2024-01-01 through 2024-12-31, containing 2,108,152
121+
panel rows.
122+
123+
The agent-visible public schema contains:
124+
125+
- identity: `datetime`, `symbol`;
126+
- candidate predictors: `open`, `high`, `low`, `close`, `volume`,
127+
`quote_asset_volume`, `number_of_trades`,
128+
`taker_buy_base_asset_volume`, and `taker_buy_quote_asset_volume`;
129+
- fitting metadata: `weight_std_dollar_vol`;
130+
- supervised response: `target_horizon_1`.
131+
132+
`target_horizon_1` is intentionally anonymized. `weight_std_dollar_vol` is fitting
133+
metadata rather than a predictor. Two additional columns, `tradable_return` and
134+
`beta_10d_fwd_1`, are retained only by the trusted backtest and verifier. They
135+
are never included in model fitting or prediction matrices.
136+
137+
The canonical public and hidden Parquet files are approximately 330 MB and
138+
176 MB respectively. The agent image receives a compressed projection of the
139+
public artifact with only identity, predictor, weight, and target columns.
140+
Hidden data remain exclusively in the verifier environment. The dataset is
141+
bundled with the task; no external download is required at runtime.
142+
143+
## Evaluation Strategy
144+
145+
The benchmark uses a continuous outcome reward rather than a binary pass
146+
threshold. The primary score is annualized Sharpe on the fixed hidden 2024
147+
after-cost strategy returns. The same value is emitted as `primary_score`,
148+
`reward`, and `sharpe`. The verifier also reports hidden CAGR, maximum drawdown,
149+
and Pearson information coefficient so that risk-adjusted performance can be
150+
interpreted alongside predictive quality and downside behavior.
151+
152+
The portfolio backtest applies a one-basis-point linear execution fee and uses
153+
525,600 periods per year for minute-level Sharpe annualization. The submitted
154+
`max_gross_exposure` must be finite and strictly positive. Public and hidden
155+
backtests use the same task-owned portfolio construction and accounting code.
156+
The absence of a binary cutoff is intentional: the scalar reward supports
157+
ranking, reinforcement learning, and comparison across research policies.
158+
159+
Before scoring, the verifier enforces the following safeguards:
160+
161+
1. **Submission integrity.** Exactly one publicly accepted bundle must exist.
162+
Source and artifact hashes must match the manifest, and bundle paths may not
163+
escape the submission directory.
164+
2. **Estimator family.** `train_model` must return either an exact
165+
`sklearn.linear_model.ElasticNet` or an exact sklearn `Pipeline` whose final
166+
step is an exact `ElasticNet`. Subclasses and other final estimators are
167+
rejected.
168+
3. **Executable contract.** Submitted source is size- and import-restricted.
169+
The estimator must be fitted, expose unique ordered string
170+
`feature_names_in_`, serialize with Joblib, reload in a fresh process, and
171+
return finite predictions in the original row order.
172+
4. **Full-public refit.** Hidden scoring uses a new estimator fitted by calling
173+
the submitted `train_model` source once on the complete public 2022–2023
174+
panel. The estimator fitted during public experimentation is not scored.
175+
5. **Causal audit.** Three hidden future-suffix probes materially perturb later
176+
feature rows. Predictions for earlier rows must remain invariant within
177+
strict numerical tolerances.
178+
6. **Hidden isolation.** The model worker receives feature matrices but never
179+
hidden targets, tradable returns, market betas, or task-owned portfolio
180+
state.
181+
182+
A missing final submission receives `-100` for every reward field. Contract
183+
violations are rejected before hidden economic metrics are exposed.
184+
185+
## Complexity
186+
187+
The computational primitives are standard, but the research problem is
188+
difficult because useful minute-level signals are weak and easy to overfit.
189+
Every candidate must survive chronological separation, cross-symbol analysis,
190+
causal batch execution, regularization, portfolio normalization, turnover, and
191+
costs. Fixing the final estimator to Elastic Net removes model-family search as
192+
an escape hatch: the agent's advantage must come from better hypotheses,
193+
features, controls, and candidate selection.
194+
195+
The public panel contains more than 4.2 million rows, and accepted training and
196+
backtesting calls share a 100-attempt budget. A strong policy must allocate that
197+
budget among data diagnostics, feature ablations, regularization comparisons,
198+
chronological confirmation, and exposure experiments while asynchronous jobs
199+
are running.
200+
201+
Observed task runs demonstrate a broad outcome range. The current reference
202+
strategy receives hidden Sharpe `0.3723068959` with hidden Pearson IC
203+
`0.0196662380`. Recorded end-to-end and candidate-validation artifacts include
204+
missing or invalid submissions at `-100`, valid strategies with negative Sharpe,
205+
degenerate historical baselines at `0`, and positive candidates between
206+
approximately `0.15` and `0.39`. These are continuous outcomes rather than
207+
pass/fail labels.
208+
209+
A representative Harbor run completed in 20 minutes 53 seconds end to end:
210+
14 seconds for environment setup, 28 seconds for agent setup, 16 minutes
211+
40 seconds for agent execution, and 3 minutes 11 seconds for verification. The
212+
configured limits leave room for substantially deeper research than the
213+
reference policy performs.
214+
215+
## References & Resources
216+
217+
The sample is self-contained. Its implementation contracts and supporting
218+
evidence are documented in:
219+
220+
- `instruction.md` — agent-facing objective, data split, MCP workflow, and
221+
full-public refit disclosure;
222+
- `environment/agent/context/research.md` — causal research and chronological
223+
validation guidance;
224+
- `environment/agent/context/submission.md` — exact estimator, source,
225+
serialization, and prediction contract;
226+
- `environment/agent/context/backtesting.md` — public candidate-comparison and
227+
strategy-selection guidance;
228+
- `DATA_PROVENANCE.md` — fixed artifact hashes, source universe,
229+
anonymization, split construction, and license record;
230+
- `environment/mcp-server/task_config.yaml` — task-owned data, backtest, cost,
231+
execution, reward, and budget configuration;
232+
- `environment/mcp-server/feature_engineering/evaluation/verifier.py` and
233+
`scoring/official.py` — independent verification and hidden scoring flow;
234+
- `solution/solve.py` — reference MCP-only agent;
235+
- `environment/mcp-server/tests/` — causal, portfolio, runtime, release,
236+
end-to-end, and verifier behavior contracts.
237+
238+
The complete local Python verification command is:
239+
240+
```sh
241+
uv run --project environment/mcp-server test
242+
```
243+
244+
## Additional Information
245+
246+
This document describes the OpenAI-facing sample as an outcome-scored research
247+
environment rather than a binary programming exercise. Its hidden Sharpe reward
248+
is intended for continuous evaluation, policy comparison, and reinforcement
249+
learning. The fixed Elastic Net family is an experimental control that focuses
250+
the task on feature engineering and quantitative-research decisions.
251+
252+
## Author Information
253+
254+
**Contributing organization:** EdotEnv
255+
**Contribution:** Task design, implementation, data packaging, verifier,
256+
reference policy, and calibration.

‎environment/LICENSE‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
External Evaluation Sample License
22
Version 1.0, July 2026
33

4-
Copyright (c) 2026 ricefan-tech
4+
Copyright (c) 2026 EdotEnv
55

66
Permission is granted, without fee, to recipients of this sample to use, copy,
77
run, and modify the task code and bundled evaluation data solely for internal

‎task.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ artifacts = [{ source = "/app/submission", service = "mcp-server" }]
44
[task]
55
name = "mmcollab-dotcom/feature-engineering"
66
description = "Engineer causal features and submit one calibrated quantitative-ML trading strategy on a fixed public/hidden crypto split."
7-
authors = [{ name = "ricefan-tech", email = "r.wang97@yahoo.de" }]
7+
authors = [{ name = "EdotEnv", email = "michael.zhang@edotenv.com" }]
88
keywords = ["machine-learning-research", "quantitative-research", "quantitative-trading", "feature-engineering", "harbor"]
99

1010
[metadata]

0 commit comments

Comments
 (0)