README.md

vllm_gen_eval.py

Pass@n rollout.

python evaluation/vllm_gen_eval.py \
    --model_path ../ckpt/qwen-2.5-coder-7b-drpo-v4datal2-tau5-lambda0.1-log-n8-hf-model-676 \
    --data_path data/eval_v4_sub.jsonl \
    --output_dir evaluation_results/magicbench_v4/qwen-2.5-coder-7b-drpo-v4datal2-tau5-lambda0.1-log-n8-hf-model-676.json \
    --max_tokens 8192 \
    --prompt_style original \
    --tensor_parallel_size 4

subproblem.py

Prepare subproblems.

python evaluation/subproblem.py \
    --input data/eval_v4.jsonl \
    --output data/eval_v4_sub.jsonl \
    --max-ops 6

vllm_gen_subproblem.py

Subproblems rollout.

python evaluation/vllm_gen_subproblem.py \
    --model_path /path/to/model \
    --data_path data/eval_v4_sub.jsonl \
    --output_path evaluation_results/test_time_scaling/model.jsonl \
    --tensor_parallel_size 4 \
    --max_tokens 8192 \
    --prompt_style original 

python evaluation/vllm_gen_subproblem.py
--data_path data/eval_v4_new.jsonl
--output_path evaluation_results/eval_v4_new/Qwen2.5-Coder-7B-Instruct-2.jsonl
--tensor_parallel_size 4
--max_tokens 8192
--prompt_style openai

python evaluation/claude_gen_subproblem.py
--data_path data/kernelbench_level1_sub.jsonl
--output_path evaluation_results/kernelbench_l1/sonnet-5.jsonl
--max_tokens 8192
--prompt_style openai

subproblem_replace.py

Prepare runnable python code using generated subproblem triton codes.

python evaluation/subproblem_replace.py --input evaluation_results/test_time_scaling/model.jsonl 

eval_from_subproblem.py

Correctness and speedup evaluation for subproblem rollout.

python evaluation/eval_from_subproblem.py \
    --input evaluation_results/test_time_scaling/model_with_replacements.jsonl \
    --output evaluation_results/test_time_scaling/model_annotated.jsonl