Skip to content

DATA_CARD.md

Latest commit

 

History

History
118 lines (81 loc) · 12.2 KB

DATA_CARD.md

File metadata and controls

118 lines (81 loc) · 12.2 KB
Raw

Data Card

Dataset Overview

  • Scale: 9 instances, covering c, cpp, go, java, ts.
  • Source: Public GitHub issues/PRs and their repository code.
  • Trajectories: Generated by mini-swe-agent calling kimi-k3, verified through real Docker grading.
  • Environment: 9 pinned linux/amd64 Docker images.
  • Quality Gates: Gold patch applies and passes all declared F2P/P2P; agent patch resolves all F2P with zero P2P regression; training value score >= 6. Per-instance scores and details are recorded in task, environment, and INDEX.
  • Trajectory Learnability: trajectory_quality additionally records thinking quality, mechanical transition ratio, failure recovery, behavior categories, and session duration. This evidence is not mixed with task solvability scores and is not used as a release hard gate.

Selection Criteria

9 instances from kimi-k3 mini-swe-agent runs. All passed ship_gates, resolved F2P/P2P grading, MIT license, validation_version=5. Languages: c(2), cpp(2), go(2), java(2), ts(1). Selected instances: c-ares__c-ares-1137, lvgl__lvgl-10308, jbeder__yaml-cpp-1373, jarro2783__cxxopts-314, urfave__cli-2275, gin-gonic__gin-4622, jhy__jsoup-2476, jhy__jsoup-2483, typescript-eslint__typescript-eslint-12003.

Evaluation Protocol Boundaries

  • The SWE-bench official protocol executes a single run_instance per candidate patch: the runner applies the candidate patch via a fallback chain, then the eval script restores baseline test files, applies test_patch, and runs the repo/version test command. Gold patch is evaluated separately as a candidate patch and never combined with the model patch in the same run. This protocol is pinned to upstream commit f7bbbb2ccdf479001d6467c9e34af59e44a840f9.
  • The gold_pre saved in this bundle is a publisher sanity pre-check proving F2P genuinely fails on the base; it is not part of the official SWE-bench evaluation phase.
  • The bundled evaluator attempts git apply --verbose, git apply --verbose --reject, patch --batch --fuzz=5 -p1 -i <patch-file> in official order. The production release gate is stricter: gold/model patches must pass first-level git apply cleanly in production grading.
  • resolution_status uses the official three-tier FULL / PARTIAL / NO; only FULL sets graded_resolved=true. PARTIAL is recorded as-is and must not enter this complete release bundle.
  • evaluation_type selects PASS_AND_FAIL or FAIL_ONLY per repository specifics. In PASS_AND_FAIL, missing tests are treated as failures; in FAIL_ONLY, P2P may be empty. Both treat only explicit FAILED as failure (this pipeline additionally fail-closes ERROR). pytest XFAIL (alias XFAILED) counts as pass per official semantics.
  • test_command_source indicates the command provenance. Current 2026 extension tasks use claudedata_registry_scoped with repeated VM real-run validation; until fixed official specs snapshots with per-repo/version/command verification are introduced, the packager rejects official_map_repo_version_specs to prevent custom commands from impersonating official specs. test_command_sha256 binds the actual grading command to prevent post-validation tampering.

Release Evidence

Each instance's task.jsonl, environment.json, swe_bench_instance, and INDEX.jsonl consistently persist the following audit fields across all four locations for independent third-party verification:

  • Contamination: contamination_level / contamination_factors (high rejected; medium allowed only when factors are fully disclosed)
  • Verified rubric: verified_rubric_version / verified_rubric_model / verified_ps_severity / verified_tv_severity / verified_issues / verified_rubric_raw / rubric_evidence_key (LLM scores problem_statement normativeness and test validity on four dimensions, 0-3)
  • Grading: graded_resolved / gold_sanity_valid / gold_applies / applies_cleanly / gold_f2p_count / gold_p2p_count / agent_f2p_passed / agent_p2p_regressed / resolution_status / evaluation_type / test_command_source / test_command_sha256 / grading_environment / environment_image / environment_image_id / environment_plan_fingerprint / environment_backend_id / environment_platform
  • Validation provenance: validation_version / validator_fingerprint, must match the current validator at packaging time
  • Raw logs: grading_logs contains gold_pre / gold_post / agent_post complete test outputs for direct F2P fail-to-pass and P2P no-regression verification
  • Test reproducibility: test_command pins random seeds for property-based tests (hypothesis, etc.) to ensure deterministic grading

Non-Blocking Rubric Disclosures

The following severity=1 observations do not block release but document minor deviations in problem statement or test scope:

  • jhy__jsoup-2476: ps=0, tv=1; preservesMatchingSourceNofollowWhenEnforcementSuppressed tests case not described in the issue, slightly widening scope.; narrow=0 wide=1 low_coverage=0 misleading=0; adversarial: Test 'preservesMatchingSourceNofollowWhenEnforcementSuppressed' covers internal link case not in problem statement, but is a minor regression test.; adversarial: narrow=0 wide=1 low_coverage=0 misleading=0
  • jhy__jsoup-2483: ps=0, tv=1; ConnectTest and DataUtilTest cases cover charset detection edge cases beyond the core issue scope.; narrow=0 wide=1 low_coverage=0 misleading=0
  • typescript-eslint__typescript-eslint-12003: ps=0, tv=1; test scope includes additional no-unused-vars edge cases.

Trajectory Specification

  • trajectory.canonical.jsonl: Exactly one canonical conversation for training or human reading. Cumulative API history is deduplicated; Chinese task-wrapping lines generated at capture time are converted to English, generated Chinese project descriptions are removed; issue body and subsequent interactions remain unchanged.
  • trajectory.full.jsonl: Complete call-level API records. Each request includes the cumulative history up to that point; suitable for auditing and protocol research, not for direct line-by-line training.
  • Both trajectory formats undergo full redaction, cleaning user paths, emails, and credentials; redact_mode is fixed to full.

Trajectory Training Value Specification

environment.json and INDEX.jsonl trajectory_quality uses version 2026-08:

  • trajectory_thinking_quality / substantive_thinking_ratio / mechanical_transition_ratio describe thinking depth;
  • failure_recovery_present only records whether real failure-then-recovery occurred; failures are never artificially injected; clean executions without failures are treated as neutral evidence;
  • behavior_categories counts source inspection, editing, testing, diff checking, and submission as real behaviors;
  • trajectory_quality_score is a transparent auxiliary score: thinking 0/1/3/5 points, behavior categories 0.6 each (max 3), simultaneous test + diff check +1, failure recovery +1; no-failure adds neutral 0.5. This score does not replace grading and does not determine release.

Field and Leakage Boundaries

model_input.json is the only recommended file to feed into the model. The following fields in task.jsonl are for labels or evaluation only:

  • swe_bench_instance.patch
  • Top-level test_patch, swe_bench_instance.test_patch
  • swe_bench_instance.model_patch
  • FAIL_TO_PASS, PASS_TO_PASS
  • gold_*, agent_*, graded_resolved, and environment evidence

Reproduction Environment

The image archive is generated via docker save and compressed with zstd. images/images.jsonl records tag, image ID, platform, and uncompressed size; CHECKSUMS.sha256 covers all files in the bundle. Each instance includes independent Dockerfile, setup_repo.sh, and setup_env.sh; these are derived from the production task's prep_script with hashes bound in environment.json.environment_recipe. After loading images, the pinned environment can be used directly, or rebuilt when upstream repository and dependency source access is available.

Contamination Disclosure

All 9 instances disclose contamination_level and contamination_factors in INDEX.jsonl, task.jsonl, and environment.json:

  • low (8/9): c-ares, lvgl, yaml-cpp, urfave/cli, gin, jsoup#2476, jsoup#2483, typescript-eslint
  • medium (1/9): jarro2783__cxxopts-314

MAJOR #3: jarro2783__cxxopts-314 contamination_level=medium

  • Instance: jarro2783__cxxopts-314
  • Factor: Issue age (issue created 2021-10-29, fix committed 2026-06-03; the long interval increases the probability that the model's training corpus covers this issue).
  • Trainability conclusion: The trajectory was screened by ship_gates for upstream answer memory; no unguarded references to the upstream fix were detected. The medium rating reflects statistical risk from issue age only, not evidence of memorization in the trajectory. The instance is suitable for training with the contamination level disclosed.
  • Action: No fix needed. Future bundles targeting strict low-contamination-only releases may exclude this instance.

Observations (Non-Blocking)

The following three items were evaluated and confirmed as observations. They do not affect data quality or release gates and are disclosed for transparency.

MINOR #4: lvgl Gold Patch Fingerprint Coincidence

  • Instance: lvgl__lvgl-10308
  • Observation: The agent's independently generated patch has the same git blob index line (817e0c1..86e6932) as the gold patch, meaning the agent's fix to lv_slider.c is byte-identical to the official fix.
  • Assessment: No commands reading gold files exist in the trajectory; this is judged as equivalent implementation convergence (natural convergence for simple fixes), not a real leak.
  • Action: No fix needed. If strict zero coincidence is desired, comment variations could be introduced at build time, but this would compromise training data purity and is not recommended.

MINOR #5: C/C++ Instance F2P Identifiers Not in Official log_parsers Format

  • Instances: jarro2783__cxxopts-314 (F2P: CTest test names options / options_no_regex), lvgl__lvgl-10308 (F2P: Unity test target names test_slider / test_bar)
  • Observation: Identifiers are not in the official eight-language log_parsers mapping. C/C++ has no official parser by design.
  • Assessment: Evaluated and passed per the standard adaptation methodology - sanity fails precisely, gold/model passes precisely.
  • Action: No fix needed. Downstream parsers for C/C++ instances should interpret F2P/P2P entries as CTest test names or Unity test target names, not JUnit/pytest-style FQCNs.

MINOR #6: Failure Recovery Sample Scarcity

  • Observation: Of 9 trajectories, only urfave__cli-2275 (failed_tool_result_count=1) and typescript-eslint__typescript-eslint-12003 (failed_tool_result_count=4) contain genuine failure-then-recovery paths. The remaining 7 are clean executions.
  • Assessment: Clean execution reflects model capability; failures must not be artificially injected.
  • Action: No fix needed. For failure-recovery-specific training objectives, stratified sampling by failure_recovery_present or subsequent supplementation with failure-recovery trajectories is recommended.

Known Limitations

  • This bundle provides linux/amd64 images only; other architectures rely on emulation and may experience significant performance degradation.
  • This data is a high-quality demo/validation set and is not sufficient to represent the complete distribution of real-world software engineering tasks.
  • Pre-installed dependencies in offline images reuse build caches; simple fixes may complete in 1-3 minutes; duration_seconds is wall-clock time in the recording environment and does not represent task difficulty or trajectory completeness.
  • Successful trajectories do not necessarily contain failure-recovery paths; when training error-recovery capability specifically, stratified sampling by failure_recovery_present should be used rather than artificially injecting failures into trajectories.
  • Data and images contain multiple upstream open-source projects; use and redistribution must comply with each instance directory's upstream license.
  • Although full trajectories have been redacted, independent privacy and license review is still recommended prior to public release, per regional and organizational policies.