Writing
Notes from building and studying models.
Research notes
- RL Teaches a Model When to Reason, Not How
Diffing an OLMo reasoning model across its final RL update, then causally steering the changed features.
- The Sparsity Knob That Did Nothing
Why L1 sparsity failed while training a crosscoder from scratch, and what worked instead.
- Adaptive LoRA Works for SFT. It Fails Under GRPO.
A controlled comparison of gradient-based rank allocation across supervised and reinforcement learning.
- Gradient-Based LoRA Rank Allocation Fails in GRPO
The experiment and debugging trail behind an unexpected negative result.
Building from scratch
- Splicing Vision Into a Frozen GPT
Joining a CLIP vision encoder and GPT-2 with one learned projection.
- How Do You Teach an LLM to See?
Building CLIP and learning why independently trained vector spaces cannot simply be wired together.
- From GPT Tokens to Image Tokens
Building a vision transformer by working out what a token means for an image.
Other work
- Personalizing LLMs for High-Stakes Decisions
Four lessons from using personalization where preferences and consequences can conflict.
New posts are available via RSS.