LC: Two Sum, Contains Duplicate, Best Time to Buy and Sell Stock

`makemore` parts 1–3

Read TransformerLens source — understand activation patching implementation

LC: Valid Anagram, Group Anagrams, Valid Parentheses

`makemore` parts 4–5

`py-spy` flamegraph on a TransformerLens run — understand where time goes

LC: Maximum Subarray, Product of Array Except Self, 3Sum

Implement single-head + multi-head attention in numpy (no PyTorch)

Read: "Attention Is All You Need" — understand scaled dot-product attention, multi-head mechanism, positional encoding

Derive cross-entropy loss for next-token prediction by hand

LC: Container With Most Water, Longest Substring Without Repeating Characters

`nanoGPT` — implement yourself, begin training on Shakespeare

Read `nanotron` (Mistral) — write notes on why each parallelism choice was made

LC: Search in Rotated Sorted Array, Find Minimum in Rotated Sorted Array

`nanoGPT` — finish training, evaluate results

Read `torchtitan` — same exercise as nanotron

LC: Reverse Linked List, Merge Two Sorted Lists, Linked List Cycle

Break nanoGPT 3 different ways, diagnose from loss curves alone

Read: "Deep Residual Learning for Image Recognition" — skip connections, degradation problem, why depth works

LC: Invert Binary Tree, Maximum Depth of Binary Tree, Same Tree

Read: Anthropic's toy models of superposition paper

Calculate 7B model full training memory budget from first principles (weights + gradients + optimizer states + activations)

LC: Binary Tree Level Order Traversal, Validate BST, Subtree of Another Tree

Read: InstructGPT paper

Read: ZeRO paper sections 1–3 — understand memory savings arithmetic

LC: Number of Islands, Clone Graph, Course Schedule

Read: DPO paper

Read: FlashAttention paper introduction — kernel fusion, HBM roundtrip avoidance

LC: Pacific Atlantic Water Flow, Graph Valid Tree, Number of Connected Components

Understand GQA — Qwen3 uses 32 query / 8 KV heads

Read PyTorch DataLoader source — understand worker spawning model

LC: Climbing Stairs, Coin Change, Longest Increasing Subsequence

Understand RoPE positional encoding — how it differs from learned positions, why frontier models use it

Understand AllReduce, AllGather, ReduceScatter — what each does, when each is used

LC: Word Break, Combination Sum, House Robber

Understand sliding window attention — Qwen3 L0–L27 window=4096, full attention L28–L35

Mac Studio setup + benchmark MLX vs PyTorch MPS for SAE workload

LC: Merge Intervals, Insert Interval, Non-overlapping Intervals

Backpropagation from scratch in numpy — derive gradient updates manually

Read vllm scheduler (`vllm/core/scheduler.py`) — understand continuous batching

CF: Div 2 A/B — greedy + implementation (rating 800–1000)

SAE architecture deep dive: expansion factor, sparsity penalty choices, dead feature problem + mitigation

Read HuggingFace Trainer (`transformers/trainer.py`) — understand training loop abstraction

CF: Div 2 A/B — sorting + prefix sums (rating 800–1000)

Train small SAE on 2-layer transformer (before attempting Qwen3)

OTel + DCGM: collector config ingesting DCGM metrics

CF: Div 2 B — binary search + two pointers (rating 1000–1200)

SAE evaluation: activation maximization, ablation — how do you know a feature is interpretable?

OTel: custom spans — forward pass, backward pass, optimizer step, data loading

CF: Div 2 B — BFS/DFS on grids (rating 1000–1200)

Qwen3 8B architecture internals: GQA heads, SwiGLU, QK-Norm — why each design choice

OTel: dashboard — "why did my loss spike at step 3400?" answerable in under 2 minutes

CF: Div 2 B/C — basic DP (rating 1200–1400)

Research Q2: Does L34 suppression replicate on Gemma3 + Phi-4? (attribution scan on same fail cases)

CF: Div 2 C — constructive algorithms + math (rating 1200–1400)

Read: "Language Models are Few-Shot Learners" (GPT-3) — scaling laws, in-context learning, few-shot prompting

Read: "Densely Connected Convolutional Networks" (DenseNet) — feature reuse, hyperconnectivity via dense blocks

CF: Div 2 C — segment trees / BIT intro (rating 1300–1500)

Read: "An Image is Worth 16x16 Words" (ViT) — how transformers replaced CNNs, patch embedding, position encoding for vision

Read: "Highway Networks" — gating mechanisms for deep networks, precursor to residual connections

CF: Div 2 C — graphs: shortest paths, Dijkstra (rating 1400–1600)

Read: "BERT: Pre-training of Deep Bidirectional Transformers" — masked language modeling, bidirectional context, fine-tuning paradigm

Read: "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" — conditional computation, gating, expert routing

CF: Div 2 C/D — union-find / DSU (rating 1400–1600)

Read: "Scaling Laws for Neural Language Models" — compute-optimal training, power-law relationships, Chinchilla implications

Read: "Layer Normalization" — why LayerNorm over BatchNorm for transformers, Pre-LN vs Post-LN stability

CF: Div 2 D — number theory + modular arithmetic (rating 1500–1700)

Read: "Denoising Diffusion Probabilistic Models" — forward/reverse process, noise schedules, connection to score matching

Read: "LoRA: Low-Rank Adaptation of Large Language Models" — parameter-efficient fine-tuning, rank decomposition, when to use it

CF: Div 2 D — DP on trees (rating 1600–1800)

Read: "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" — retriever-reader architecture, when RAG beats fine-tuning

Read: "Switch Transformers: Scaling to Trillion Parameter Models" — simplified MoE routing, expert parallelism, load balancing

CF: Div 2 D — bitmask DP (rating 1600–1800)

System design practice: "Design a training observability system for a 1000-GPU cluster" — answer out loud, no notes

CF: Div 2 D/E — heavy-light decomposition / centroid decomposition (rating 1800–2000)

Mock interview: explain attention from first principles to a non-ML engineer

CF: Div 2 D/E — flows and matchings (rating 1800–2000)