Request for Detailed Evaluation Metrics Documentation

[superlanc3]

First, thank you for creating this impressive benchmark for evaluating LLMs' ability to solve SQL issues. BIRD-CRITIC-1 addresses a critical real-world application need.

Issue Description

While reviewing the repository, I noticed limited documentation on how the various evaluation metrics (Soft EX, Soft EX + Parsing, Test Case, Query Execution Plan) are calculated in detail. This information would be valuable for researchers attempting to reproduce or build upon your work.

Requested Improvements

• Add technical documentation explaining how each evaluation metric is implemented
• Provide examples of successful vs. unsuccessful solutions for each metric type
• Consider adding an evaluation walkthrough for one or two example tasks

This additional documentation would greatly enhance the accessibility of your benchmark for the research community.

Thank you for your consideration.

[accpatrick]

Hi, @superlanc3 , thank you for your interest in our research. We are currently drafting a paper that will provide details about this work. Your 3 suggestions are impressive, we would improve our repository and will include demos after releasing full PostgreSQL set soon.

To obtain the detailed test case functions and solution SQLs, please send an email request to bird.bench25@gmail.com. We use this method to manage distribution and prevent potential data leakage through automated web crawlers. The requested files will be provided via email soon. Thanks.

Best,