GitHub Advisory Database Analyzer

This repository is a fork of the original project:
https://github.com/amjadnatouf/vulnerability_analyzer

It is maintained here for visibility on my GitHub profile as part of a joint bachelor’s thesis project by Dan Hjälmberg and Amjad Natouf.

---

About this project

This repository contains a tool developed to support the thesis:
“Curating a List of Open Source Projects for a Vulnerability Repair Benchmark”
(Mid Sweden University, B.Sc. in Computer Engineering, 2025)

The tool implements an automated pipeline for collecting vulnerability reports, identifying corresponding fix commits, and linking them to relevant test cases in open-source Python projects.

The work contributes toward building a benchmark dataset for automated vulnerability repair (AVR) - an important area in software engineering research driven by machine learning and large language models.

---

Thesis and dataset

Thesis (Mid Sweden University):
https://urn.kb.se/resolve?urn=urn:nbn:se:miun:diva-55483
http://miun.diva-portal.org/smash/record.jsf?pid=diva2:1996016

Dataset (Zenodo):

---

Note: This fork does not modify the original implementation. The canonical version of the code remains in the upstream repository.

GitHub Advisory Database Analyzer is an automated tool designed to collect, analyze, and process security advisories from the GitHub to construct a dataset of vulnerability-patch pairs, along with relevant metadata. This data set enables researchers and developers to mine advisory data, extract commit diffs, categorize vulnerabilities by CWE, and support Automated Vulnerability Repair (AVR) research and machine learning model training.

Features

• Download security advisories directly from GitHub.
• Extract metadata, repository details, and commit information.
• Identify security patches and map vulnerabilities to CWE categories.
• Store advisory data in a structured SQLite database.
• Generate reports and statistical insights from the dataset.
• Command-line interface (CLI) for user interaction.
• Export processed data as JSON for further research.
• Repository test detection and analysis.
• Benchmark candidate filtering and ranking.
• Comprehensive test case linking to vulnerability fixes.

Contributors

• Dan Hjälmberg
• Amjad Natouf

Installation

Prerequisites

• Python 3.8+ (Download Python)
• Git (Download Git)
• Virtual Environment (recommended for dependency management)

Project Dependencies

The project uses the following Python libraries:

Dependency	Purpose
requests	HTTP requests and API access
beautifulsoup4	HTML/XML parsing (web scraping)
tqdm	Progress bar display
sqlalchemy	Database ORM management
pandas	Data manipulation and CSV export
pydriller	Git repository and commit mining
python-dotenv	Environment variable handling
typing-extensions	Additional type hints support
pytest	Unit testing and assertion
pytest-cov	Code coverage analysis
unittest-mock	Mocking and testing utilities
pytest-fixture	Fixture management and setup
matplotlib	Data visualization and plotting
prettytable	Table formatting for CLI output

These dependencies are managed using requirements.txt while requirements-dev.txt contains additional dependencies for development and testing.

Installation Steps

1. Clone the repository:

git clone https://github.com/amjadnatouf/vulnerability_analyzer.git
cd vulnerability_analyzer

2. Set up a virtual environment: On macOS/Linux:

python -m venv venv
source venv/bin/activate

On Windows:

python -m venv venv
venv\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

4. Set environment variables (if necessary): Create a .env file at project root:

GITHUB_API_TOKEN=your_github_token_here

5. Install additional dependencies for development and testing:

pip install -r requirements-dev.txt

Usage

Running the Tool

To launch the tool, execute:

python main.py

This will open an interactive CLI where users can select various options.

Main Menu Options

Option	Description
1	Download vulnerability records from GitHub
2	Parse and save vulnerability records
3	Get repository metadata from GitHub
4	Parse and save repository metadata
5	Check for tests in repositories
6	Parse and save repository test metadata
7	Filter vulnerabilities and repositories
8	Sort vulnerabilities and repositories (opt)
9	Analyze repositories and commit fixes
10	Search for tests in fix commits
11	Filter benchmark candidates
12	Get CWE metadata for a vulnerability
13	Export analyzed vulnerabilities as datasets
14	Display database statistics
15	Settings
16	Show help menu
17	Extract vulnerability records from ZIP file
0	Exit the program

Data Collection Workflow

1. Download vulnerability records

To retrieve vulnerability records from GitHub:

1. Download vulnerability records from GitHub

This includes:

• Select a year and month from the prompts.
• Choose download options (all, sample, custom).

2. Parse local advisories

Parse locally stored GHSA advisory JSON files:

2. Parse and save vulnerability records

3. Get repository metadata

Fetch repository metadata using GitHub API:

3. Get repository metadata from GitHub

4. Parse repository metadata

Process downloaded repository metadata files:

4. Parse and save repository metadata

5. Check for tests in repositories

Analyze repositories for test-related directories and files:

5. Check for tests in repositories

6. Parse repository test metadata

Process repository test metadata:

6. Parse and save repository test metadata

Analysis Workflow

7. Filter vulnerabilities and repositories

Apply filtering criteria to select benchmark candidates:

7. Filter vulnerabilities and repositories

8. Sort vulnerabilities and repositories

Rank filtered candidates based on sorting criteria:

8. Sort vulnerabilities and repositories (opt)

9. Analyzing Vulnerabilities

To process vulnerabilities and analyze repository commits:

9. Analyze repositories and commit fixes

This includes:

• Cloning repositories
• Extracting commit diffs
• Identifying security patches
• Analyzing file modifications and method changes

10. Search for tests in fix commits

Identify test cases associated with vulnerability fixes:

10. Search for tests in fix commits

11. Filter benchmark candidates

Apply final filtering to benchmark candidates:

11. Filter benchmark candidates

12. Generating CWE metadata

To generate CWE metadata for vulnerabilities:

12. Get CWE metadata for a vulnerability

This includes generating CWE metadata for all vulnerabilities and saving CWE metadata to the database.

Data Export and Analysis

13. Exporting Data

To generate a structured dataset:

13. Export analyzed vulnerabilities as datasets

This creates CSV/JSON files in the analysis/ directory.

14. Viewing Statistics

To generate insights about analyzed vulnerabilities:

14. Display database statistics

This displays:

• Total vulnerabilities analyzed
• Number of vulnerabilities with fix commits
• Most common CWE categories
• Affected programming languages
• Top repositories with security advisories
• Repository and commit statistics
• Patch complexity metrics
• Test coverage analysis

15. Change Tool Settings

To configure the application settings:

15. Settings

16. Viewing Help Menu

To show the help menu:

16. Show help menu

17. Extract from ZIP file

Process vulnerability records from a downloaded ZIP file:

17. Extract vulnerability records from ZIP file

0. Exiting the Application

To exit the program:

0. Exit the program

Modules Overview

Core Modules

• main.py – Entry point for CLI interactions.
• core/database.py – Manages database operations.
• core/db_models.py – Defines SQLAlchemy models.

Downloader Modules

• downloader/advisory_downloader.py – Downloads security advisories from GitHub.
• downloader/repository_metadata_downloader.py – Fetches repository metadata from GitHub API.

Analysis Modules

• analysis/advisory_parser.py – Extracts advisory metadata.
• analysis/vulnerability_analyzer.py – Main coordinator for vulnerability analysis.
• analysis/commit_analyzer.py – Processes commit-level changes.
• analysis/file_analyzer.py – Analyzes modified files.
• analysis/cwe_analyzer.py – Maps vulnerabilities to CWE.
• analysis/repository_metadata_parser.py – Processes repository metadata.
• analysis/repository_test_checker.py – Analyzes repositories for test presence.
• analysis/test_locator.py – Links test cases to vulnerability fixes.
• analysis/statistics_generator.py – Computes and exports reports.
• analysis/dataset_generator.py – Generates research datasets.
• analysis/data_analyzer.py – SQL-based data extraction for analysis.

UI Modules

• ui/cli.py – Provides command-line functionality.
• ui/cli_statistics.py – Displays database statistics.

Notebooks Module

• notebooks/data_analysis.ipynb – Jupyter Notebook for data analysis and visualization.
• notebooks/setup.py - Utility script to set up the Jupyter Notebook environment.

Utility Modules

• utils/config.py – Stores application settings and configuration.
• utils/utils.py – Helper functions for processing advisories.
• utils/rate_limiter.py – GitHub API rate limiting management.
• utils/notebook_helpers.py – Functions for Jupyter notebook analysis.

Configuration

The application uses JSON configuration files for filtering and sorting:

• utils/filter_criteria.json – Define repository and vulnerability filtering criteria
• utils/sorting_criteria.json – Define sorting and ranking criteria for benchmark candidates
• utils/benchmark_candidates_filter_criteria.json – Final filtering criteria for benchmark candidates

Troubleshooting

Common Issues & Fixes

• Database Connection Issues: Ensure the database is running and the connection string is correct.
• Repository Retrieval Failures: Verify repository URLs and API keys.
• Commit Analysis Errors: Check commit hashes and file paths for accuracy.
• Statistics Generation Failures: Review database schema and statistics generation logic.
• General Errors: Consult the application logs for detailed error messages.

Issue: The tool fails to download advisories.

Fix: Ensure the advisory_downloader module has the correct API endpoint and authentication credentials

Issue: Database is not initializing.

Try resetting the database by deleting vulnDataset.db and reinitializing.

Issue: GitHub API rate limiting.

The tool includes automatic rate limiting. If you encounter persistent rate limiting, check your GitHub API token permissions and daily usage limits.