A decoder-only transformer (GPT-style) implemented from scratch in PyTorch — no Hugging Face, no pre-built attention layers — and trained as a character-level language model on the Harry Potter novels until it generates passable wizard-flavored prose.
The goal was to understand the transformer architecture by building every piece myself: token and positional embeddings, scaled dot-product self-attention, multi-head attention, feed-forward blocks, residual connections, and layer normalization, assembled into a stack of decoder blocks with a language-model head.
A character-level tokenizer over the book corpus, a batched data pipeline with train/validation split, the full model (~6 layers, 6 heads, 384-dim embeddings, dropout regularization), a training loop with periodic loss estimation and checkpointing, and a sampling script for open-ended text generation. Training ran on an Intel GPU via the PyTorch XPU backend.
Python · PyTorch (built directly on nn.Module primitives) · Jupyter notebooks for experiments · Intel XPU acceleration.
【待填写】GitHub repo link and sample generated text.
Planned: switch from character-level to a BPE tokenizer, scale up the context window, and compare training dynamics against my MiniMind experiment runs to see how data quality and tokenization change the results.