1 / 17 UNLOCKED
Splash Normal
Nothing happened...
Micrograd Fire
Scalar autograd from scratch, no copy-paste.
Profiler Fix Steel
Profile benchmark runner — find and fix one memory inefficiency.
Makemore 1–3 Fire
Character-level language models: bigram, MLP, and beyond.
TransformerLens Psychic
Read TransformerLens source — understand activation patching.
Makemore 4–5 Fire
WaveNet architecture and becoming a backprop ninja.
Flamegraph Steel
py-spy flamegraph on a TransformerLens run — find where time goes.
Attention Fire
Implement single-head + multi-head attention in numpy, no PyTorch.
Cross-Entropy Psychic
Derive cross-entropy loss for next-token prediction by hand.
nanoGPT Build Fire
Implement nanoGPT yourself, begin training on Shakespeare.
Nanotron Read Steel
Read nanotron (Mistral) — notes on each parallelism choice.
nanoGPT Train Fire
Finish nanoGPT training, evaluate results.
Torchtitan Read Steel
Read torchtitan — same parallelism analysis as nanotron.
Break nanoGPT Fire
Break nanoGPT 3 different ways, diagnose from loss curves alone.
Hardware Specs Steel
Memorize H100 HBM bandwidth, VRAM, TFLOPS; Mac Studio unified memory.
Superposition Psychic
Read Anthropic's toy models of superposition paper.
Memory Budget Steel
Calculate 7B model full training memory budget from first principles.