This project demonstrates how to fine-tune GPT-2 on a small, handcrafted dataset to generate LaTeX code from natural language prompts. It’s a minimal but focused attempt to align an older language model with a specific instruction-following task — without requiring massive resources.
This project fine-tunes the gpt2 model to translate plain English prompts into LaTeX representations — enabling the generation of math expressions, equations, and symbols in LaTeX format.
We train the model using a handcrafted dataset of 50 English-to-LaTeX pairs, designed to expose the model to structured and well-scoped instructions. Through instruction-style prompting and fine-tuning, GPT-2 learns to produce syntactically accurate LaTeX outputs, even though it was never originally trained for this domain.
✅ The entire training process — from preprocessing and tokenization to fine-tuning and inference — is handled within a single Jupyter notebook.
✅ Built using Hugging Face’s transformers and datasets libraries for an end-to-end NLP pipeline.
🧪 Evaluation is conducted qualitatively using unseen prompts such as:
"write the LaTeX for x squared plus y squared equals z squared""generate LaTeX for limit of x as x approaches 0"
These are manually assessed to verify the correctness and formatting of the generated LaTeX code.
📊 The full pipeline includes:
- Fine-tuning GPT-2 on a domain-specific LaTeX dataset
- Prompt-based English-to-LaTeX generation
- Saving and loading the model for reuse
- Notebook-based training and result visualization
🎯 Goal: Demonstrate that even a small, instruction-tuned dataset can effectively repurpose a general language model like GPT-2 for structured code generation tasks — in this case, LaTeX math writing.
To get started with fine-tuning GPT-2 for English-to-LaTeX translation, follow the steps below:
git clone https://github.com/Daddy-Myth/fine-tuning-gpt2-for-latex-generation.git
cd fine-tuning-gpt2-for-latex-generation# Create a virtual environment
python -m venv venv
# Activate it
venv\Scripts\activateInstall the necessary Python packages using pip:
pip install -r requirements.txtRun the Jupyter notebook to start the training process:
jupyter notebookThen open Fine_tune_GPT2_LaTeX.ipynb and run through the cells sequentially to fine-tune and test the model.
The fine-tuning process is performed in a single Jupyter notebook and includes the following steps:
-
Preprocessing the Dataset
- Load and clean the handcrafted English-to-LaTeX pairs.
- Tokenize both inputs (plain English) and outputs (LaTeX code) using the GPT-2 tokenizer.
-
Model Preparation
- Load the
gpt2model from Hugging Face Transformers. - Resize the model embeddings to match the tokenizer's vocabulary (if extended).
- Configure the model for sequence-to-sequence style generation.
- Load the
-
Fine-Tuning
- Train the model using a small number of epochs due to dataset size.
- Track training loss with
tqdmprogress bars for easy monitoring. - Save the fine-tuned model locally for future inference.
-
Evaluation
- Prompt the fine-tuned model with unseen English queries.
- Generate corresponding LaTeX outputs.
- Manually verify correctness of generated LaTeX syntax and structure.
Once the model is fine-tuned, you can generate LaTeX code by prompting it with plain English inputs like:
Prompt:
write the LaTeX for x squared plus y squared equals z squared
Prompt:
generate LaTeX for limit of x as x approaches 0
🧠 Both of the above examples use the exact conversion prompt format the model was trained on:
conversion_text_sample = f'{CONVERSION_PROMPT}English: {text_sample}\n{CONVERSION_TOKEN}'This ensures the instruction-tuned model stays aligned and outputs the desired LaTeX syntax with high accuracy — reflecting the consistent input-output mapping learned during training.
Prompt:
g of x equals integral from 0 to 1 of x squared
When deviating from the expected format, the model struggles to generalize and may produce malformed or repetitive LaTeX code.
❗Tip: Always include the trained prefix (English:) and newline before the conversion token to get the best results.
💡 Important Notes:
-
The first two prompts (
write the LaTeX...,generate LaTeX...) were exactly taken from the training data. That’s why they produce clean and accurate LaTeX output. -
However, if the prompt format differs even slightly from what the model saw during training, the output can become inconsistent or contain unrelated tokens — especially random characters or hallucinated text at the end.
-
This behavior stems from the model trying to continue the format it learned (e.g., continuing a LaTeX block), and can be improved with more training data, better prompt conditioning, or output post-processing.
The fine-tuned model was tested on a range of English-to-LaTeX prompts to assess output quality and consistency.
| Prompt Type | Description | Output Quality |
|---|---|---|
| ✅ Seen Prompts | Exact prompts from the training dataset | 🟢 Accurate |
| Slightly reworded inputs similar to training examples | 🟡 Partial | |
| ❌ Unstructured Prompts | Prompts with unusual phrasing or missing keywords | 🔴 Inaccurate |
- ✅ Training Prompts yielded precise and clean LaTeX (e.g., correctly formatted
\frac,\int, etc.). - 🧠 The model generalizes decently when the phrasing is close to seen examples.
- ❗ Output sometimes ends with random trailing tokens (e.g., stray brackets or characters), especially for novel prompts.
- 🔁 No quantitative metric (like BLEU or ROUGE) was used — evaluation was qualitative and based on visual accuracy.
- Even correct prompts can sometimes result in outputs with noise at the end.
- This is likely due to:
- The model continuing a LaTeX-style completion.
- Lack of post-processing.
- Limited training data.
- 📚 Quick Start Guide to LLMs — the foundation and structure behind this project.