A decoder-only Transformer implemented from scratch in PyTorch for character-level name generation.
The project builds the core Transformer components directly using basic PyTorch operations and layers, without relying on prebuilt Transformer or multi-head attention modules.
The model learns names character by character and generates complete names from an initial prefix using greedy decoding and temperature-based sampling.
The goal of this project is to develop a practical understanding of the internal architecture of autoregressive Transformer language models.
The complete pipeline includes:
- Character-level tokenization
- Special-token handling
- Fixed-length sequence encoding
- Static sinusoidal positional encoding
- Causal attention masking
- Scaled dot-product attention
- Multi-head self-attention
- Position-wise feed-forward networks
- Pre-layer normalization
- Residual connections
- Stacked decoder blocks
- Autoregressive next-token prediction
- Gradient-clipped training
- Validation-loss tracking
- Greedy decoding
- Temperature sampling
- Generated-name analysis
Notebook:
transformer/decoder_only_transformer.ipynb
The model operates at the character level rather than using words or subword tokens.
The vocabulary contains:
- 26 lowercase English letters
<PAD><SOS><EOS>
Total vocabulary size:
29
Each name is converted into two aligned sequences.
<SOS> + name
name + <EOS>
Both sequences are padded with <PAD> tokens to a maximum length of:
20
For example, the encoded input representation of Alice is:
[1, 3, 14, 11, 5, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
The project uses:
data/Names.csv
The dataset contains an English-name column named:
Name
Names are:
- Converted to strings
- Stripped of surrounding whitespace
- Converted to lowercase
- Encoded at the character level
Number of dataset rows used by the notebook:
1,825,433
The dataset is split into training and validation subsets using a fixed random generator seed.
| Split | Samples |
|---|---|
| Training | 1,642,890 |
| Validation | 182,543 |
| Total | 1,825,433 |
The training DataLoader contains 6,418 batches, while the validation DataLoader contains 714 batches with the configured batch size.
Self-attention does not inherently represent token order, so static sinusoidal positional encodings are added to character embeddings.
The implementation uses the standard sine and cosine formulation across alternating embedding dimensions.
The positional encoding matrix has shape:
(20, 64)
corresponding to:
Maximum positions: 20
Embedding dimensions: 64
The positional encoding tensor is registered as a non-trainable PyTorch buffer and is added directly to token embeddings before the decoder blocks.
The notebook also visualizes the complete positional encoding matrix.
Because the model is autoregressive, each position must be prevented from accessing future characters.
A lower-triangular Boolean mask is constructed dynamically for each sequence:
Position 0 → can attend to position 0
Position 1 → can attend to positions 0–1
Position 2 → can attend to positions 0–2
...
Future attention scores are replaced with negative infinity before applying softmax.
This guarantees that next-character predictions depend only on the currently available prefix.
The attention mechanism is implemented directly from Query, Key, and Value tensors using:
The implementation performs:
- Query–Key similarity calculation
- Scaling by the square root of the head dimension
- Causal masking
- Softmax normalization
- Attention dropout
- Weighted aggregation of Value vectors
Both the attention output and attention-weight matrix are returned.
The custom multi-head self-attention layer contains separate linear projections for:
Query
Key
Value
Output
The projected representations are divided across multiple attention heads.
For this project:
d_model = 64
num_heads = 4
head_dimension = 16
Each head independently performs scaled dot-product attention.
The resulting head representations are concatenated and projected back to the original model dimension.
No prebuilt PyTorch MultiheadAttention module is used.
Each decoder block contains a two-layer feed-forward network:
64 → 256 → 64
The network applies:
- Linear projection
- ReLU activation
- Dropout
- Linear projection back to
d_model - Dropout
The same feed-forward transformation is independently applied to every sequence position.
Each Transformer decoder block uses a Pre-LN architecture.
The computation follows:
Input
│
├── LayerNorm
│
├── Causal Multi-Head Self-Attention
│
└── Residual Connection
│
├── LayerNorm
│
├── Feed-Forward Network
│
└── Residual Connection
│
Output
Layer normalization is therefore applied before each main sublayer rather than after the residual connection.
The complete model consists of:
Token IDs
↓
Character Embeddings
↓
Sinusoidal Positional Encoding
↓
Embedding Dropout
↓
Decoder Block × 3
↓
Final Layer Normalization
↓
Linear Language-Modeling Head
↓
Next-Character Logits
The language-modeling head produces a probability distribution over the complete vocabulary for every sequence position.
| Hyperparameter | Value |
|---|---|
| Decoder layers | 3 |
| Attention heads | 4 |
| Model dimension | 64 |
| Feed-forward dimension | 256 |
| Maximum sequence length | 20 |
| Dropout | 0.1 |
| Vocabulary size | 29 |
Total trainable parameters:
153,821
A forward-pass sanity check produces:
Input batch shape: [4, 20]
Target batch shape: [4, 20]
Logits shape: [4, 20, 29]
Attention maps: 3
The three attention maps correspond to the three stacked decoder layers.
The model is trained using next-character prediction.
| Hyperparameter | Value |
|---|---|
| Epochs | 3 |
| Batch size | 256 |
| Learning rate | 1e-3 |
| Optimizer | Adam |
| Validation split | 0.1 |
| Gradient clipping | 1.0 |
| Loss | Cross-Entropy |
| Device | CPU |
Padding tokens are excluded from the loss using:
nn.CrossEntropyLoss(
ignore_index=str_to_index["<PAD>"]
)Gradient norms are clipped before each optimizer update to improve training stability.
A fixed random seed of:
42
is used for Python, NumPy, PyTorch, and the train–validation split.
Training and validation loss decrease across all three epochs.
| Epoch | Train Loss | Validation Loss | Total Epoch Time |
|---|---|---|---|
| 1 | 1.994663 | 1.821821 | 1521.08 s |
| 2 | 1.887462 | 1.772410 | 1411.58 s |
| 3 | 1.859399 | 1.751050 | 1369.91 s |
Final results:
Final training loss: 1.859399
Final validation loss: 1.751050
The notebook also plots the training and validation loss curves across epochs.
Training was performed entirely on the CPU.
After training, the model generates names one character at a time.
Generation starts with:
<SOS> + prefix
At each step:
- The current sequence is passed through the Transformer.
- The logits corresponding to the next position are selected.
<PAD>and<SOS>are excluded from generation.- A next character is selected.
- The selected character is appended to the sequence.
- Generation continues until
<EOS>or the maximum length is reached.
The model supports two decoding strategies.
When:
temperature = 0
the model selects:
argmax(logits)at every generation step.
This produces deterministic output for the same prefix.
Examples from the trained model:
| Prefix | Greedy Output |
|---|---|
c |
charlee |
li |
lillianna |
mar |
marianna |
a |
alexandra |
Repeated greedy decoding produces the same name each time.
For positive temperature values, logits are scaled before applying softmax:
scaled_logits = logits / temperature
A character is then sampled from the resulting probability distribution.
The project compares:
temperature = 0.0
temperature = 0.5
temperature = 1.0
with five generations for each prefix.
Greedy — Temperature 0.0
charlee
Temperature 0.5
carolyn
christal
clevelle
carolee
carles
Temperature 1.0
coleen
cherian
corrise
caepon
corrinne
Greedy
lillianna
Temperature 0.5
lilyanna
linda
lillian
lilianna
linda
Temperature 1.0
lilianna
lilliana
lirelle
lisly
linda
Greedy
marianna
Temperature 0.5
marielle
marlene
marquel
mariana
margaret
Temperature 1.0
mare
maryloud
marika
marycus
martin
Greedy
alexandra
Temperature 0.5
alexandra
audriana
alisandra
alvin
aratha
Temperature 1.0
auber
anicia
armilo
amara
acey
The generated samples demonstrate the expected relationship between temperature and output diversity.
At:
temperature = 0
generation is deterministic because the highest-probability character is always selected.
At higher temperatures, sampling introduces alternative character choices and produces more diverse names.
For example, the prefix c produces only:
charlee
with greedy decoding, while both 0.5 and 1.0 produce five distinct outputs in the recorded experiment.
This illustrates the trade-off between high-probability deterministic generation and more diverse stochastic generation.
Generated names are also compared against the original dataset.
The analysis identifies:
- Generated names already present in the dataset
- Generated names not found in the dataset
- Existing dataset entries beginning with each tested prefix
Examples of generated names found in the dataset include:
charlee
carolyn
christal
carolee
coleen
corrinne
lillianna
lillian
linda
marianna
marielle
marlene
margaret
alexandra
audriana
amara
The model also produces names that are not found in the dataset, such as:
clevelle
cherian
corrise
caepon
lirelle
lisly
maryloud
marycus
aratha
auber
armilo
This demonstrates that sampling can produce both memorized or existing character patterns and novel combinations learned from the structure of names in the training data.
decoder-only-transformer-from-scratch/
│
├── data/
│ └── Names.csv
│
├── transformer/
│ └── decoder_only_transformer.ipynb
│
├── .gitignore
├── requirements.txt
└── README.md
- Python
- PyTorch
- NumPy
- Pandas
- Matplotlib
- Jupyter Notebook
Clone the repository:
git clone https://github.com/Hamidreza-Talei/decoder-only-transformer-from-scratch.git
cd decoder-only-transformer-from-scratchCreate a virtual environment:
python -m venv venvActivate it on Windows:
venv\Scripts\activateOn macOS or Linux:
source venv/bin/activateInstall the required packages:
pip install -r requirements.txtStart Jupyter Notebook:
jupyter notebookOpen:
transformer/decoder_only_transformer.ipynb
Run the notebook from beginning to end.
The notebook will:
- Load and preprocess the names dataset.
- Construct the character vocabulary.
- Encode names as input–target sequences.
- Visualize sinusoidal positional encodings.
- Create the train and validation datasets.
- Define causal scaled dot-product attention.
- Build multi-head self-attention.
- Construct the Pre-LN decoder blocks.
- Assemble the complete decoder-only Transformer.
- Verify model output dimensions.
- Train and validate the model.
- Plot the loss curves.
- Generate names autoregressively.
- Compare greedy and temperature-based decoding.
- Compare generated names with entries in the dataset.
The model is intentionally configured to run entirely on the CPU, although training the full dataset can take a significant amount of time.
The notebook uses a fixed seed:
42
for:
- Python random operations
- NumPy
- PyTorch
- Train–validation splitting
To reproduce the experiment:
- Install the dependencies.
- Restart the Jupyter kernel.
- Run all cells from the beginning.
- Avoid executing cells out of order.
Generation using positive temperatures is stochastic, although the notebook sets a PyTorch seed before the reported generation experiment.
This repository demonstrates:
- Transformer architecture
- Decoder-Only Transformers
- Autoregressive language modeling
- Character-level language modeling
- Token embeddings
- Sinusoidal positional encoding
- Causal masking
- Scaled dot-product attention
- Query, Key, and Value projections
- Multi-head self-attention
- Attention dropout
- Pre-layer normalization
- Residual connections
- Position-wise feed-forward networks
- Stacked decoder blocks
- Language-modeling heads
- Next-token prediction
- Cross-entropy loss
- Padding-mask handling
- Gradient clipping
- PyTorch datasets
- PyTorch DataLoaders
- Greedy decoding
- Temperature sampling
- Autoregressive generation
- Model evaluation
- Training-loss analysis
The primary goal of this project is to understand the mechanics of Transformer language models by implementing the architecture directly rather than relying on high-level Transformer abstractions.
The project follows the complete pipeline:
Raw Names
↓
Character Tokenization
↓
Input–Target Sequence Construction
↓
Token Embeddings
↓
Sinusoidal Positional Encoding
↓
Causal Multi-Head Self-Attention
↓
Pre-LN Decoder Blocks
↓
Next-Character Prediction
↓
Autoregressive Name Generation
It provides a compact, end-to-end implementation of the same core ideas that underlie much larger decoder-only language models.