This project analyzes CodeLLMExp, a multi-language dataset for automated vulnerability localization and explanation in AI-generated code. The dataset contains vulnerable code snippets, fixed versions, CWE labels, vulnerable-line annotations, and natural-language explanations. The analysis pipeline focuses on three connected tasks:
- CWE classification: predicting the vulnerability category from source code.
- Vulnerable line localization: identifying likely vulnerable lines in the code.
- Faithfulness-to-fix evaluation: checking whether predicted vulnerable lines overlap with lines actually changed by the security fix.
The project intentionally uses interpretable, lightweight methods rather than deep learning. CWE classification uses TF-IDF features with classical machine learning models, while line localization and fix-concept extraction use rule-based heuristics based on diffs, keywords, and line-level scoring.
After cleaning, the dataset contains:
| Metric | Value |
|---|---|
| Total samples | 10,403 |
| Languages | 3 |
| Unique CWE labels | 29 |
| Unique fix concepts | 12 |
| Average code length | 531.22 characters |
| Rows with vulnerable-line annotations | 97.76% |
Language distribution:
| Language | Samples |
|---|---|
| Python | 4,610 |
| Java | 3,088 |
| C | 2,705 |
Top CWE categories by sample count:
| CWE | Samples |
|---|---|
| CWE-022 | 782 |
| CWE-020 | 491 |
| CWE-079 | 489 |
| CWE-078 | 443 |
| CWE-089 | 414 |
| CWE-502 | 413 |
| CWE-190 | 388 |
| CWE-601 | 387 |
| CWE-476 | 385 |
| CWE-094 | 384 |
The train/validation/test split is:
| Split | Samples |
|---|---|
| Train | 7,281 |
| Validation | 1,561 |
| Test | 1,561 |
The cleaning stage normalizes language and CWE labels, removes invalid rows, parses vulnerable-line annotations, and strips annotation markers such as BAD comments from code. A key detail is that whole-line BAD markers can shift line numbers. Therefore, the cleaning logic remaps original vulnerable-line annotations to their new line numbers after marker removal.
Rows where vulnerable code and fixed code are identical after cleaning are preserved for classification, because their code and CWE labels are still useful. However, those rows are excluded from diff-based tasks such as fix-concept extraction, line localization, and faithfulness evaluation.
The pipeline compares vulnerable and fixed code using line-level diffs. Added and removed lines are then matched against rule-based patterns to infer the type of security fix. The extracted fix concepts are interpretable labels such as input validation, memory safety, parameterized query, sanitization, and authorization checks.
Fix concept distribution:
| Fix Concept | Samples |
|---|---|
| input_validation | 3,840 |
| unknown_fix | 2,512 |
| memory_safety | 1,332 |
| parameterized_query | 812 |
| authorization_check | 533 |
| sanitization | 412 |
| path_validation | 243 |
| bounds_check | 240 |
| null_check | 215 |
| exception_handling | 158 |
| crypto_fix | 60 |
| safe_api | 46 |
The most common inferred fix concept is input validation, which is consistent with the dataset's focus on common software security weaknesses. A large number of rows are labeled as unknown_fix, showing that simple rule-based concept extraction is useful but incomplete.
Two classical models were evaluated: Logistic Regression and Linear SVM. Both use TF-IDF features over source code. The main experiment uses code only; an additional upper-bound setting uses code plus explanation text.
| Model | Setting | Accuracy | Macro F1 | Weighted F1 |
|---|---|---|---|---|
| Logistic Regression | code_only | 0.9885 | 0.9896 | 0.9884 |
| Linear SVM | code_only | 0.9987 | 0.9987 | 0.9987 |
| Logistic Regression | upper_bound | 0.9757 | 0.9782 | 0.9754 |
| Linear SVM | upper_bound | 0.9917 | 0.9923 | 0.9916 |
The Linear SVM performs best, reaching 99.87% accuracy and 99.87% weighted F1 in the code-only setting. Interestingly, adding explanation text does not improve performance in this run; the upper-bound setting is slightly worse for both models. This suggests that the source code alone contains highly predictive lexical and structural signals for CWE classification, while explanation text may introduce noise or distributional mismatch.
The vulnerable line localization task evaluates whether the heuristic localizer can rank the true vulnerable lines near the top of its predictions.
Overall localization results:
| Metric | Value |
|---|---|
| Top-1 Accuracy | 0.2166 |
| Top-3 Accuracy | 0.3620 |
| Top-5 Accuracy | 0.4169 |
| Mean Reciprocal Rank | 0.2951 |
| Mean IoU | 0.0888 |
| Evaluated rows | 10,170 |
Localization by language:
| Language | Top-1 | Top-3 | Top-5 | MRR | IoU | Rows |
|---|---|---|---|---|---|---|
| C | 0.1574 | 0.2699 | 0.3094 | 0.2143 | 0.0668 | 2,586 |
| Java | 0.1941 | 0.3471 | 0.4317 | 0.2829 | 0.0897 | 3,060 |
| Python | 0.2657 | 0.4248 | 0.4684 | 0.3495 | 0.1007 | 4,524 |
Python has the strongest localization performance, while C is the most difficult. This may be because C vulnerabilities often involve memory and pointer behavior, where a vulnerable operation may depend on context outside the single changed line. The overall Top-5 accuracy of 41.69% indicates that rule-based localization can provide useful candidate lines but is not sufficient as a fully reliable vulnerability detector.
Faithfulness-to-fix evaluates whether predicted vulnerable lines overlap with lines actually changed in the fixed code. This is different from matching human vulnerable-line annotations: a prediction can miss the annotated line but still be faithful to the repair if it points to a line modified by the fix.
Overall faithfulness results:
| Metric | Value |
|---|---|
| Mean IoU | 0.2263 |
| Fix Hit@1 | 0.4434 |
| Fix Hit@3 | 0.8060 |
| Fix Hit@5 | 0.8885 |
| Rows | 10,403 |
Faithfulness by language:
| Language | IoU | Fix Hit@1 | Fix Hit@3 | Fix Hit@5 | Rows |
|---|---|---|---|---|---|
| C | 0.1886 | 0.4525 | 0.6669 | 0.7793 | 2,705 |
| Java | 0.2356 | 0.5291 | 0.8374 | 0.9197 | 3,088 |
| Python | 0.2423 | 0.3807 | 0.8666 | 0.9317 | 4,610 |
Faithfulness is much stronger than vulnerable-line annotation accuracy. The localizer reaches Fix Hit@5 = 88.85%, meaning that in most cases at least one of the top five predicted lines overlaps with the actual fix region. This suggests that the heuristic model often identifies repair-relevant code even when it does not exactly match the dataset's annotated vulnerable line.
Faithfulness by fix concept:
| Fix Concept | IoU | Fix Hit@1 | Fix Hit@3 | Fix Hit@5 | Count |
|---|---|---|---|---|---|
| bounds_check | 0.2400 | 0.5958 | 0.9458 | 0.9792 | 240 |
| parameterized_query | 0.3162 | 0.6675 | 0.9187 | 0.9483 | 812 |
| sanitization | 0.1727 | 0.5340 | 0.7451 | 0.9272 | 412 |
| null_check | 0.1916 | 0.7209 | 0.8651 | 0.9209 | 215 |
| safe_api | 0.2329 | 0.2609 | 0.8478 | 0.9130 | 46 |
| exception_handling | 0.2017 | 0.3228 | 0.8418 | 0.9051 | 158 |
| unknown_fix | 0.2138 | 0.4908 | 0.8344 | 0.9029 | 2,512 |
| input_validation | 0.2139 | 0.3253 | 0.8164 | 0.8982 | 3,840 |
| memory_safety | 0.2729 | 0.5330 | 0.7763 | 0.8483 | 1,332 |
| authorization_check | 0.2058 | 0.3959 | 0.6079 | 0.8349 | 533 |
| crypto_fix | 0.1263 | 0.2000 | 0.4167 | 0.6667 | 60 |
| path_validation | 0.1907 | 0.3086 | 0.5473 | 0.5802 | 243 |
The strongest faithfulness appears for bounds checks and parameterized queries, where the fix often introduces localized, recognizable code changes. The weakest results appear for path validation and crypto fixes, which may require broader semantic understanding than keyword and line-diff heuristics can provide.
-
CWE classification is highly effective with classical models. A TF-IDF Linear SVM achieves nearly perfect test performance, indicating strong lexical separability among CWE categories in the dataset.
-
Exact vulnerable-line localization remains challenging. The Top-1 score is 21.66%, and Top-5 is 41.69%. This shows that matching annotated vulnerable lines is harder than classifying the overall CWE.
-
Predictions are more faithful to fixes than to annotations. Although exact annotation accuracy is modest, Fix Hit@5 reaches 88.85%. The localizer often points to lines that participate in the security repair.
-
Language matters. Python has the best localization Top-5 score, while C is lower, likely because C vulnerabilities often depend on memory behavior and broader control/data-flow context.
-
Fix type matters. Parameterized queries, bounds checks, and null checks are easier for the heuristic pipeline to connect to fix regions. Path validation and crypto fixes are harder.
The project relies on rule-based heuristics for fix-concept extraction and vulnerable-line localization. These methods are interpretable but limited: they cannot fully model data flow, control flow, aliasing, pointer semantics, or framework-specific security behavior. The high classification scores may also reflect strong lexical patterns in the generated dataset rather than generalization to real-world vulnerable code.
Another limitation is the large unknown_fix category. This indicates that the current fix-concept rules do not cover all repair styles. Expanding the taxonomy and adding richer static-analysis features could improve concept extraction.
Finally, faithfulness-to-fix is based on changed lines, not necessarily semantic causality. A changed line may be part of a refactor or supporting edit, while an unchanged line may still be security-relevant.
Important tables:
outputs/tables/dataset_summary.csvoutputs/tables/final_dataset_summary.csvoutputs/tables/cwe_classification_results.csvoutputs/tables/line_localization_by_language.csvoutputs/tables/faithfulness_by_language.csvoutputs/tables/faithfulness_by_fix_concept.csvoutputs/tables/case_study_table.csv
Important metrics:
outputs/metrics/cwe_classification_logreg.jsonoutputs/metrics/cwe_classification_svm.jsonoutputs/metrics/line_localization_metrics.jsonoutputs/metrics/faithfulness_to_fix_summary.json
Important figures:
outputs/figures/language_distribution.pngoutputs/figures/top_cwe_distribution.pngoutputs/figures/fix_concept_distribution.pngoutputs/figures/cwe_confusion_matrix_svm.pngoutputs/figures/line_localization_by_language.pngoutputs/figures/faithfulness_by_language.pngoutputs/figures/faithfulness_by_fix_concept.pngoutputs/figures/final_result_summary.pngoutputs/figures/pipeline_overview.png
Storyboard examples:
outputs/storyboards/case_001.pngoutputs/storyboards/case_002.pngoutputs/storyboards/case_003.pngoutputs/storyboards/case_004.pngoutputs/storyboards/case_005.pngoutputs/storyboards/case_006.png
For a written report, the project can be organized as:
- Introduction: motivate AI-generated vulnerable code and the need for CWE classification, localization, and explainability.
- Dataset: describe CodeLLMExp, languages, CWE coverage, vulnerable/fixed code pairs, and line annotations.
- Methodology: explain cleaning, diff extraction, fix-concept rules, TF-IDF classification, heuristic localization, and faithfulness metrics.
- Results: present classification, localization, and faithfulness results with tables and figures.
- Discussion: interpret why classification is strong, why line localization is harder, and why faithfulness-to-fix provides an additional useful view.
- Limitations: discuss rule-based limitations, generated-code bias, unknown fix concepts, and lack of deep semantic analysis.
- Conclusion: summarize that classical models classify CWE very well, while rule-based localization is useful as a candidate generator and is especially strong when evaluated against fix regions.