A crash course in

How LLMs
actually work

…and how to use them

Act I — How LLMs actually work · Act II — Working with basic LLMs · Act III — Modern agentic work

Carson Davis

Opening · 003

I'm Carson. And I am very lazy.

I code because I want things to be efficient and automated.

the perspective I’m bringing

2016topic matching between PDFs

→

2018custom embeddings for scientific corpora

→

2020semantic search over large scientific corpora

→

2023RAG systems, as genAI arrived

→

2025data agents for scientific search

→

2026meta software development with agents

Opening · 005

The next three hours.

how they work → working with raw LLMs → modern agentic workflows → working with LLMs on a team

ACT IHow LLMs actually workembeddings → transformers → chatbot

builds on →

ACT IIWorking with a raw LLMtokens · context · tools · RAG · chain of thought

builds on →

ACT IIIModern agentic workartifacts → skills → agents → harness → teams → how I work

ends at →

& THENReal Project Strategieshow to make a vision · how to make your docs · how to do git issues

If you don’t understand how it works, you can’t reason about using it well.

ACT I

How LLMs actually work

an LLM is a statistical next-word predictor

Embeddings → transformers → tokens → scale → the naked model

Act I · 010

Getting computers to understand language

how do you convince a computer that c.a.t is almost the same as k.i.t.t.e.n.?

one early answer: numerical dictionaries, by hand

word	small?	feline?	organic?	cute?	young?
cat	☑	☑	☑	☑	☐
kitten	☑	☑	☑	☑	☑
dog	☐	☐	☑	☑	☐
rock	☑	☐	☐	☐	☐
aardvark	☑	☐	☑	☐	☐
abacus	☑	☐	☐	☐	☐
abandon…	?	?	?	?	?

…449,993 more rows to fill in by hand — consistently, for all of English

another manual approach: ingest a thesaurus — still a human doing meaning by hand

key ideaYou cannot hand-write the meaning of every word as a matrix of yes/no features.

SPOKEN (v3): As long as there have been computers, scientists have been trying to get computers to understand language. The simplest thing going into a computer is the literal word — CAT versus KITTEN. Apart from the T, nothing tells a computer they're the same concept. One early answer: numerical dictionaries by hand — is it small, is it feline, is it organic, is it cute, is it young. Note the young? column: it's the one place cat and kitten differ. Lots of techniques were tried — I'm not going through the rest of the history, because this part isn't super important. What matters: a real human has to go through the entire dictionary by hand — 449,993 more rows, consistently, for all of English. Not feasible. Another manual approach: ingest a thesaurus — still a human doing meaning by hand.

Act I · 020

4,000 patents, a few hundred thousand companies — and the gap is what words mean

NASA tech transfer: patent ↔ company matching. The whole bottleneck: what words mean.

patent sheet

a welding patent, 1–2 pages

welding brazing alloy

3–4 keywords, hand-picked — ×4,000 patents

?does this match?

company

a manufacturer, one of several hundred thousand

NAICS 333992 — welding equipment mfg.

a government code for what the company does

the obvious next move — hard-coded thesaurus expansion

weld → welds welding maybe “additive manufacturing”?

brute force, keyword by keyword — it does get you to a small list of companies per patent. But the magic jump to metal joining just isn’t there.

key ideaKeywords and dictionary synonyms struggle to measure meaning — how similar two meanings are.

Act I · 030

Meaning is the company a word keeps

word2vec learned it first; GloVe is the recipe you can see

word2vec · 2013 · Google · learns by predicting a hidden word from its neighbors GloVe · 2014 · Stanford · gets there by counting co-occurrences — nearly the same math

“the cat lapped up the milk from the bowl”

row “cat”: +1 the +1 lapped +1 up — slide the box, tally again; every word takes a turn in the center. Now do every sentence in every book:

	purr	milk	leash	engine	sky
cat	8	7	1	0	1
kitten	9	8	1	0	0
dog	0	3	9	0	1
car	0	0	0	9	1
moon	0	0	0	1	8

whole-corpus co-occurrence counts — cat’s and kitten’s rows match; nobody else’s do

≈

two skinny matrices of word vectors

two words that keep the same company get nearly identical rows → nearly identical embeddings — cat ≈ kitten, purely from counting

The neural net’s task: populate this matrix. word2vec came first; the two were demonstrated numerically near-identical (Levy & Goldberg, 2014)

key ideaA word's meaning is captured by the company it keeps — the words it co-occurs with.

SPOKEN (v3): word2vec learned it first, but GloVe is the recipe you can actually see — so that's the one on screen. word2vec, 2013, Google: learns by predicting a hidden word from its neighbors. GloVe, 2014, Stanford: gets there by counting co-occurrences — nearly the same math. The GloVe pipeline: a sliding window tallies who sits next to whom — "the cat lapped up the milk from the bowl", row cat gets +1 the, +1 lapped, +1 up; slide the box one word right and tally again. Do that for every sentence in every book: the co-occurrence matrix. Cat's and kitten's rows match; nobody else's do. Factorize into two skinny matrices of word vectors — cat ≈ kitten, purely from counting. The neural net's task: populate this matrix. Early research questions — how big a window? strip "a" and "the"? collapse want/wanting/wanted to one stem? — come back later, as tokens.

Act I · 030 · part two

It made numbers that work

Every word → a vector: the computer’s conceptualization of its definition. And you can do math on it.

cat → [5.6, 7.8, …] ×300
kitten → [5.5, 7.7, …] ×300

similarity to “cat”, by cosine similarity:
cat 1.0 · kitten .96 · jaguar .81 · lion .8

1 = same word · 0 = unrelated — perpendicular vectors · negative = opposing

king − man + woman ≈ queen

vector space (2-D shadow of ~300 dims)

key ideaWord vectors put a word's definition into numbers: similarity becomes measurable, and arithmetic works on meaning.

Act I · 040

The corpus is the definition

The co-occurrence matrix comes from the corpus — so the corpus decides what every word means.

trained on astronomy papers

bias frames flat fields cosmic rays

“reduction”

trained on cookbooks

simmer sauce thicken

IMPACT — embeddings trained on the science itself

~400,000 Earth-science research papers → custom embeddings better definitions for science words than any Google-Books-trained model acronym expansion: “National Aeronautics and Space Administration” ≈ NASA

semantic search engines over NASA documents were built on exactly these embeddings · the evolution: embed words → sentences → whole documents

two limitations

1 — one averaged definition

Every occurrence of a word in the corpus collapses to one vector — one blended definition, everywhere it appears.

2 — positional ignorance

The window only asks “did it co-occur?” — not how close, and not in what order.

key ideaThe training corpus decides what every word means — and each word gets exactly one averaged, position-blind definition.

SPOKEN (v3): You can imagine that the corpus you do this on really matters — the co-occurrence matrix comes from the corpus, so the corpus decides what every word ends up meaning. "Reduction" trained on astronomy papers sits near bias frames, flat fields, cosmic rays; trained on cookbooks it sits near simmer, sauce, thicken. At IMPACT we trained custom embeddings on ~400,000 Earth-science research papers — better definitions for science words than any Google-Books-trained model, and acronym expansion fell out for free: "National Aeronautics and Space Administration" and NASA appear in identical contexts, so they were inferred as synonyms. Semantic search engines over NASA documents were built on exactly these embeddings — that's semantic search, not RAG yet; hold the word for beat 180. The evolution: embed words, then sentences, then whole documents. Two limitations to carry forward: one averaged definition per word, and positional ignorance — the window only asks "did it co-occur?", not how close or in what order.

Act I · 050

Four banks, one blurred vector

Two challenges a static embedding can’t get past.

challenge 1 — one word, four meanings

“I was banking on getting to the bank before it closed, so I drove along the bank of the river and took the turn so hard the car banked left.”

challenge 2 — word order: the negation problem

“I really like Apple, they're high-performing” +1 ✓ positive — correct

“I can't really like Apple, they're not as high-performing as they used to be” +1 ? same keywords, opposite meaning

Every definition was right — meaning lives in context + composition, not in one vector per word.

key ideaTwo challenges: one word, four meanings (bank ×4, one vector) and composition (word order flips meaning). Better per-word definitions fix neither.

Act I · 060

“Attention is all you need” — 2017

Words look at each other — a vector per occurrence, and the task becomes next-word prediction.

predict the hidden word → spelling & typo correction put the blank at the end → next-word prediction — autocomplete run it again and again → it types whole sentences

Advance 1 transformer · 2017

Attention to far-away tokens — not just the ones next door.

Advance 2 ELMo · BERT · 2018

Embeddings that depend on the surrounding words — the four banks finally separate.

key ideaAttention lets words look at each other; masked / next-word prediction turns that into a text generator.

two attention heads, one read — change “tired” to “wide” and “it” becomes the street

deeper dive: The Illustrated Transformer — jalammar.github.io

SPOKEN (v3): Words look at each other, so a word's vector finally depends on its sentence — and the whole enterprise becomes predicting the next word. Mask a word; predict it from everything else. Predict the hidden word → spelling and typo correction; put the blank at the end → next-word prediction, autocomplete; run it again and again → it types whole sentences... and hopefully that is starting to sound pretty familiar. Transformer, ELMo, BERT — I'm not going to spend much time on them because it's not that important. Between these advances we suddenly got way better at using arbitrary text. (Depth, if asked: the transformer fixed sequence processing — parallel training, long-range attention; ELMo fixed what a word's representation is, on a pre-transformer bidirectional LSTM; BERT combined them, Oct 2018; GPT reframed everything as next-token prediction.)

Act I · 120

Models don’t read words

What actually goes in? Technically words — even more technically, tokens: learned sub-word chunks.

geology → geology

≈ characters ÷ 4 = tokens OpenAI’s own rule of thumb tokenizer unique per model — the same paragraph counts differently in ChatGPT vs. Claude

Exact splits vary by tokenizer — “something like geo + logy,” not a guaranteed split.

key ideaA token is a learned, semantically meaningful chunk of text — not necessarily a whole word.

doEstimate tokens as characters ÷ ~4; expect the count to differ per model.

Act I · 070

Just add more data

OpenAI’s bet: same architecture + more text = keeps getting better.

GPT-1“terrible”

+ more data →

GPT-2“way better”

+ more data →

GPT-3“actually pretty good”

key ideaPour enough text into next-word prediction and it becomes shockingly capable.

Act I · 080

Why does it answer instead of continue?

A raw predictor continues your text; a second training layer retargets it at answering.

base model — next-word predictorcontinues whatever you type

post-training — the InstructGPT recipe — trained to answer the question, not continue the text1 SFT — supervised fine-tuning: ~13k hand-written ideal answers · 2 reward model — ~33k human rankings · 3 RLHF — reinforcement learning from human feedback

an assistant that answers youOuyang et al., OpenAI 2022 — the direct ancestor of GPT-3.5 / ChatGPT

GPT = Generative Pre-trained Transformer — the “Chat” was the usability layer that made it ChatGPT

Not just Q&A pairs: human answers + RL from human rankings.

key ideaPost-training turns "continue the text" into "answer the question."

Act I · 090

These are not the facts you're looking for

Nothing in training optimizes for true — it speaks English the way English was spoken.

"Humans use only ___ of their brains"

It isn’t confidently wrong —
because it was never confidently right.

key ideaA naked LLM has no facts; it's a statistical next-word predictor that's truthful only because most of its training text was.

doDistrust unsourced factual claims from a bare model — and drop the word "hallucination."

Act I · 100

One distribution, reshaped

Not a second distribution — stretch or squash the one you have, then sample. "my dad went to the store to buy ___"

low T — sharpened

the model's distribution

high T — flattened

softmax(logits / T) — one distribution, reshaped · literally the temperature in a Boltzmann distribution

Creativity is not remixing: “The ___ lapped the milk.” jaguar can land there having never appeared there — its vector sits next to cat’s. Structured novelty, not random words.

key ideaOutput varies because you sample from a probability distribution that temperature reshapes before sampling.

doRaise temperature for novelty; lower it to pull the single most-likely, near-deterministic answer.

In the corpus the likely fills are milk, beer, cigarettes, flowers — a whole distribution. If the model always grabbed the top word it'd say the same thing every time. Temperature is one knob that divides the logits before the softmax — it doesn't add a second distribution on top, it reshapes the one distribution the model already produced. T → 0: mass piles onto the top token — always "milk," deterministic. Higher T: flatter, wilder — long shots become reachable, dad buys a tiny volcano. For this room: it's literally the temperature in a Boltzmann distribution — low T freezes into the ground state, high T populates excited states. Creativity is not remixing: the model can say "jaguar" where the corpus only ever said "cat," because jaguar's vector sits next to cat's. Structured novelty, not random noise.

Act I · 110 — the hinge

The naked model — where ChatGPT started

Launch-day ChatGPT: a next-word predictor with post-training bolted on.

Amazing

✓ perfect, coherent English

✓ writes you a poem on demand

✓ anything you wanted, fluently

Frustrating

✗ states things that are just not true

✗ falls apart on complex instructions

key ideaThe naked model is fluent and coherent but has no facts and struggles with complicated, multi-part instructions.

#1 it says untrue things.
#2 it drops complex instructions.

Those two frustrations are the to-do list for the entire rest of this talk.

ACT II

Working with basic LLMs

The conversation illusion → the window → grounding → tools → chain of thought

Act II · 130

The conversation is an illusion

Every turn: the whole transcript, replayed into a stateless model.

turn 1 A →

fresh model
never seen a word

→ B

turn 2 ABC →

fresh model
never seen a word

→ D

turn 3 ABCDE →

fresh model
never seen a word

→ F

an hour in: the entire transcript, replayed every turn, into a model with no memory

100,000 tokens of the old idea

50 of your new one

key ideaEvery turn re-feeds the whole transcript to a stateless model — there is no memory.

doTo unstick a model, don't argue at the end of a long thread — reset or prune the history.

Act II · 140

A million tokens — not a million good ones

A hard-ish cap — and non-uniform performance inside it. 2022: ~8k tokens → today: 1M+

"lost in the middle" — position

"context rot" — total input

remember 2017’s attention? it isn’t perfect every model has a different ability to attend across a growing window every turn grows the window — the conversation degrades itself

Is the cap a hard wall? Not mathematically: trained sequence length + O(n²) attention + money. Embedding models are hard-capped (classic BERT: 512 tokens). · The binoculars problem: you switch binocular models in the last message — but 90% of the conversation is the old model’s name.

Know when to end the conversation.

key ideaA bigger context window is not uniformly good context — recall sags in the middle and quality degrades as it fills: context rot.

doKnow when to end the conversation — crafting what goes into the window is how you get better output.

Every LLM caps how many tokens fit — the context window. It's a hard-ish cap: attention cost grows quadratically and quality falls off past the trained length, so providers cap where compute and quality stop being worth it — 8k in 2022 became 1M+ now. (Embedding models ARE hard-capped — classic BERT: 512 tokens.) But a million tokens isn't a million equal tokens. Two distinct effects: "lost in the middle" — bury a fact mid-window and recall dips (Liu et al., 2023). And "context rot" — the 2025 Chroma result: all 18 frontier models tested degraded as input grows, well before the advertised limit. SPOKEN (v3): Go back to that 2017 attention mechanism — it's not perfect. Every architecture and training regime has a different ability to pay attention across a growing window. And every message you send makes the window bigger — so a conversation degrades its own model as it grows. The binoculars problem: you switch binocular models in the last message, but 90% of the conversation is the old model's name — don't be surprised when it keeps referencing it. The action item, verbatim, big: know when to end the conversation.

Act II · 145

Everything in the window votes

Fill the window with enough of something and it out-votes the training.

what came in — a window already half full of one kind of story

your instructions

Method 1 — context loading

Want help with your NASA space laser? Seed the window with a ton of your own space-laser text. Refusal is a statistical behavior: half a window of one kind of story → the story continues; the refusal never fires. The weights never changed — the window did.

Method 2 — rewrite the history API only

Outside a chat UI, you pass the whole conversation history yourself — including the assistant’s past messages. So write them by hand: make the model “already have said” whatever walks it past the filter.

The threat side — prompt injection

A web page, a 50,000-token paper, a log file — those tokens land in the same window as your instructions, and they vote. No wall separates them from you. More tools = the vote matters more.

key ideaThe window out-votes the weights: whatever dominates your context dominates the model’s behavior — your instructions have no special immunity.

doBefore an agent reads untrusted text, ask what that text could tell it to do — keep tool-bearing sessions away from it.

⛔ CONCEPTUAL ONLY — do NOT live-demo a jailbreak to the room. SPOKEN (v3), the joke, verbatim: "I think we've all been in the situation of trying to get an LLM to help us make a nuclear bomb or write an explicit poem... and I'm sure we've all been in the situation of the LLM telling us no." Same physics as the swamp in 130 — just done on purpose. Method 1, context loading: seed the window with a ton of your own space-laser text; refusal is statistical, and half a window of one kind of story means the story continues. Method 2, API only: outside a chat UI you pass the whole conversation history yourself, including the assistant's past messages — write them by hand. "Not recommending the jailbreak — but worth knowing how these things actually work." ⚠ The prompt-injection half remains an unsourced reconstruction (kept by default in v3) — confirm it reads true before stage, or soften. Guardrails arrive at auto mode & permissions in Act III.

Act II · 150

Style lives in the weights and in the window

“LLM voice” isn’t fixed — show it the voice you want.

source 1 · pretraining data

the Q&A-shaped text it happened to see

source 2 · post-training

the company's "be concise / be verbose" layer — their knob, not yours

source 3 — the overlooked one

in-context examples: it's still a next-word predictor — whatever voice fills the window, it continues

Fix 1 — just ask

Dear Claude,
please stop writing like that.
Love, Carson

works when the ask was already in post-training — summarize · be concise · be professional

Fix 2 — few-shot: show it the voice

paste your old research papers → a rough draft in your voice, not Claude-speak

chain-of-thought models even try to match the style on purpose — and the raw token math pulls the same way

The Shakespeare gambit: spend 300,000 of your million tokens on the actual works of Shakespeare — it gets very, very good at Shakespeare.

weaker / local model? decompose: step 1 emit a style-guide template → step 2 fill it from your sample → step 3 write to the filled guide · the name: few-shot prompting / in-context learning — “Language Models are Few-Shot Learners” (Brown et al., 2020). No weights change.

key ideaStyle lives both in the weights and in the window — and you can override the weights by putting examples in the window.

doPaste 3+ samples of the target voice. For weak models, decompose: template → fill → write.

SPOKEN (v3): You can spot "LLM voice" instantly — but the model isn't stuck with it. Three sources of style: pretraining data; the post-training layer (the company's knob, not yours); and the overlooked one — in-context examples, because it's still a next-word predictor. Fix 1, just ask: "Dear Claude, please stop writing like that. Love, Carson." — I'm not saying that won't work; with a really powerful LLM, sometimes it will, especially if what you're asking for was part of post-training (summarize, be concise, be professional). Fix 2, few-shot: paste your old research papers and get a rough draft in your voice, not Claude-speak — chain-of-thought models even try to match the style on purpose, and the raw token math pulls the same way. The Shakespeare gambit: 300k of your million tokens on the actual works. For a weaker or local model, decompose: template → fill → write. The name: few-shot prompting / in-context learning, Brown et al. 2020. No weights change.

Act II · 160 — grounding facts in the window

Context is everything.

Don't ask the weights for facts. Find the true fact and put it in the window.

Ask the weights

your question → naked LLM → hope & pray

One paper among tens of thousands in the training data — drowned out. Maybe the truth is the most likely token. Maybe.

Paste the source

the paper itself + your question → grounded answer

Paste the paper, ask about that. The fact is sitting in the window.

key ideaSupply the source instead of hoping the weights memorized it. The context you build is what you live or die on.

doPaste the paper / the data and ask about that, rather than asking the model from memory.

Act II · 170 — tool calling & the orchestrator

naked LLM + tool + orchestrator

The architecture behind every assistant you actually use.

key ideanaked LLM + tool + orchestrator — the architecture of every assistant you actually use.

doThink in terms of routing work to tools, not doing everything inside the chat window.

Act II · 170

Three flavors of orchestrator

the tools list: calculator · python · Google search — we’ll only walk through the calculator

The same LLM, post-trained

retrained to select tools instead of answering

A tiny specialist LLM

small, fast, does nothing but route

An ordinary model, prompted

no special training — you just ask it to orchestrate

Windsurf ran an in-house orchestration model — until the big labs got too good at it.

key ideanaked LLM + tool + orchestrator — the orchestrator hijacks the request, calls the tool, and puts the result into the context window.

doThink in terms of routing work to tools, not doing everything inside the chat window.

Act II · 180 — search & RAG

Give the orchestrator a search bar

Tool calling + grounding, fused: fetch the real page, build the window automatically.

question

"what tuners does my bass ship with?"

→

orchestrator

"this needs a real source"

→

search tool

manufactures queries, runs them

→

the real page

the manufacturer's spec sheet

→

context window

question + page summary

→

grounded answer

a true fact about the world

RAG — retrieval-augmented generation

key ideaRetrieval augments generation — pull the real source into the window at question time.

doRoute questions that need a live or authoritative source through search / retrieval; later, build your own RAG.

Act II · 190 — why code beats prose

Ask for the program, not the answer

Code is the highest-leverage tool — your job is recognizing when to reach for it.

the code — highly constrained: few words, few ways to organize it the tests — lined up with exactly what the code is supposed to do the docs — here’s what we say the code does, next to the code

LLMs are an order of magnitude better at writing code that does something than at reasoning over English.

In its head

"How many R's in strawberry?"

"strawberry has six R's" ✗

It doesn't see letters — it sees tokens. A simple LLM is never getting this right: it doesn't know things.

Through code

"Write a program that counts letters, then use it on strawberry."

>>> "strawberry".count("r")
3

3 ✓

Still using the LLM. Still not doing the work yourself. Infinitely better answer.

English: ~450,000 words — vast, ambiguous · Python: ~100 keywords and built-ins — tiny, constrained, and paired with what-it-does in the training data

key ideaCode is constrained and functional, so the model is both better at it and more capable through it.

doYou don't always want the answer from the LLM — sometimes you want it from code.

Why are models so good at code, and why does code matter more than words? Three things sit together in the training data: the code (highly constrained — few words, few ways to organize it), the tests that constrain it (lined up with exactly what the code is supposed to do), and the docs explaining it. English is impossibly broad — ~450,000 words, a million phrasings. Python is ~100 keywords and built-in functions (35 keywords + ~70 built-ins — fact-checked), insanely constrained, and deeply functional. The classic: "how many R's in strawberry?" — "strawberry has six R's." It doesn't see letters, it sees tokens; you can go back and forth forever and a simple LLM is never getting this right — it doesn't know things. The better move: "write me a program that counts letters, then use it on strawberry" — chapter one of every Python book. Still using the LLM, still not doing the work yourself — infinitely better answer. A big part of your job is recognizing when to reach for code.

Act II · 220

Chain of thought

first, why the field moved on from scale: scale stopped paying — GPT-4 → 4.5 was the diminishing-returns turn (OpenAI itself: "not a frontier model")

and the internet ran out — "data is the fossil fuel of AI" (Sutskever) → the new lever: spend tokens thinking

"On the street next to my house there are five blue houses and a red one. Jill lives in the yellow house, Bob lives in the green house — which house does Sam live in?"

Without thinking — one pass, straight to the answer

prompt → answer tokens — whatever next-word prediction reaches directly

With thinking — the same single generation, longer

prompt → reasoning tokens (scratch work, out loud first) → answer tokens, now conditioned on its own scratch work

thinking — these are real generated tokens streaming out… Five blue houses + one red house = six houses on the street… Jill is in a yellow house — but no yellow house exists on this street. So Jill doesn't live here. Same for Bob and the green house. Neither constraint touches this street at all… So Sam is the one who lives on the street — which house makes him distinct?
→ Sam lives in the red house.

the model works out loud before the answer, one continuous generation — each reasoning step is itself next-word prediction Wei et al. 2022 (prompting) → reasoning models RL-trained to think before answering

the payoff: five asks in one prompt — all five done, each remembered and handled independently

key ideaChain of thought = next-word prediction over generated reasoning steps, reaching answers a single pass can't.

Open with the turn: for a while the lever was more data — then GPT-4.5, February 2025: OpenAI's largest-ever pretrained model, and OpenAI itself declined to call it a frontier model. We basically ran out of data — Sutskever, NeurIPS 2024: data is the fossil fuel of AI. So the field changed levers, and THIS is the new lever. Chain of thought is NOT slicing your input into chunks and re-feeding them — it's the model generating intermediate reasoning tokens, working out loud, before emitting the final answer, all as one continuous generation. The house puzzle, v3 resolution: five blue + one red = six houses; Jill's yellow house and Bob's green house don't exist on this street, so neither lives here; Sam is the one who lives on the street — the red house makes him distinct. Sam lives in THE RED HOUSE. SPOKEN (v3), how they trained it, two phases: Phase 1: curated step-by-step data — word problems from math textbooks, instruction manuals, logic chains — as reinforcement learning, so the predicted tokens are steps along a path. Phase 2, when the textbooks ran out: generative RL — hide the answer, let the model invent its own steps, and reward whatever chain reaches the right answer. There's a reason they call it thinking. The real payoff isn't word puzzles — it's five asks in one prompt, all five done, each remembered and handled independently.

Act II · 220

The cost of thinking

thinking: low ↔ high — a knob you set

thinking tokens are billed as output tokens

a hard problem can emit 5–10× more of them

accuracy ↗ · latency ↗ · cost ↗

the curve you'll be reading for years

key ideaReasoning trades money and latency for accuracy — you pay for the scratch work as output tokens.

doThinking up for complex interdependent tasks, down for simple ones; judge future models on tokens-vs-performance, not size.

ACT III

Modern agentic work

The model stops answering and starts acting. Every tool arrives after the hand-done move it packages.

artifacts → skills → agents → the harness → safety → teams → science → how I actually work

Act III · 360 — artifacts

Escape the chat window

Save text as durable artifacts you can reuse.

Buried at the top of the chat

☐ the to-do list

…400,000 tokens of conversation…

attention has to reach all the way back ↑ — the agent can lose track of what it’s doing

Re-read at the bottom, from disk

☑ step 1 — result written under it
☑ step 2 — result · ☐ step 3

fresh each turn — sees what’s done

the debugging log — three uses

improves its thinking

writes what it tried and why; moves coherently between attempts

lets you spot the bug

you read everything it checked — you might see what it couldn’t

transfers between sessions

“here’s the bug, here’s everything tried — look outside these”

email draft on disk — you edit it, the chat rereads it documentation / changelog written as you work review notes between agents — review-1.md → review-2.md

key ideaAn artifact is anything outside the chat window — durable, written down, re-read fresh every turn, alive for the next session.

doDon’t be afraid to pull things out of the conversation and onto disk — to-dos, debugging logs, drafts, notes between agents.

SPOKEN (v3): A really big task with lots of items: because of context rot you worry about losing the pieces — how can you be sure the agent actually checked everything off? A deep debugging run: an absolute black hole — no idea what it tried, what it looked into, what the results were. In both cases the solution is to get information out of the chat window and into somewhere durable — you can see it, this chat can see it, later chats can see it. PLANT (one line, not a lesson): "I dictate mine. How I get all this out of my head is the last section of the day." The debugging log's three uses: improves the model's thinking between attempts; you read everything it checked and might spot the bug it couldn't; transfers between sessions. Examples are illustrative — the essence is durable, written down, outside the chat. These durable markdown files are the version LLMs read — the version made for YOU shows up later in the act, as Claude Artifacts.

Act III · 470 — Claude Artifacts

Stop reading the walls of text

The to-dos, debug logs, and review notes you’re saving are walls of text — and when you’re the reader. A Claude Artifact is just the response — beautifully formatted.

wall of text

4,000 words in a terminal

→

“make me an artifact”

as easy as asking — native in Claude Code

→

formatted page + private or public link

→

a colleague reads it

not a chat log

where I reach for one

learning a concept

interactive beats six paragraphs in a terminal

mapping codebase interactions

what calls what — boxes and borders drawn for you

project status + next actions

where things stand, what to work on next

Anthropic’s blog: “Using Claude Code: The Unreasonable Effectiveness of HTML”

live demo the showcase artifact — a fictional photometry-pipeline migration stat strip · needs-you list · pipeline diagram · verified-facts cards · comparison table · severity rails · runnable checklist — every section labeled with its technique, so the page doubles as a menu

key ideaA Claude Artifact is just the response, beautifully formatted — and you can even share the link with whoever wasn’t in the session.

doAsk for an artifact after any dense session — I plan things out in one, then show it to someone for review.

SPOKEN (v3): Use LLMs enough and you get what we'll politely call extreme reading fatigue — and since my advice is "live in the terminal", that reading is barely-formatted terminal text. Name collision acknowledged out loud: Claude calls these artifacts too — mine are files a model re-reads; these are pages a human reads. Where I reach for one: learning a concept (interactive beats six paragraphs), mapping codebase interactions (boxes and borders drawn for you), project status + next actions. Anthropic's blog post: "Using Claude Code: The Unreasonable Effectiveness of HTML" (title/URL verified 2026-07-19). LIVE DEMO — the showcase artifact: https://llms.codebycarson.com/artifact-example/ — a fictional photometry-pipeline migration; every section labeled with its formatting technique, so the page doubles as a menu of what to ask for.

Act III · 320 — CLAUDE.md

Instructions it reads every time

Telling LLMs the same thing again and again? Put it in CLAUDE.md — every session reads it.

CLAUDE.mds stack — the folder’s, the folder above, up to root

~/
├─ CLAUDE.md            ← computer-wide rules
└─ github/
   ├─ CLAUDE.md        ← rules for all your repos
   └─ space-laser/
       ├─ CLAUDE.md    ← project-specific rules
       └─ src/           ← a session here reads all three

project-specific + computer-wide, layered — put each rule at the level where it applies

what goes in — a category, an example

hard rules“never commit without asking”

holistic guidance“prefer class-based, reusable architectures”

operational procedures“no commits until the tests pass”

context loading“before any coding task, read README.md”

what doesn’t

too vague“always write good code” — worthless · “try to be DRY” — generic advice

too draconian“never have a Python file over 200 lines” — weird, limiting

Everything in it enters every conversation. Don’t describe your whole project here — point to the file that describes the project.

key ideaCLAUDE.md is read every conversation — everything in it enters every chat, so only specific, always-applicable rules belong.

doWrite direct, followable rules — hard rules, holistic guidance, procedures, context loading — not vague aspirations.

SPOKEN (v3): Artifacts are durable things saved on your computer that you inject into new sessions. Same idea here: if you find yourself telling LLMs the same thing again and again — put it in CLAUDE.md, and every session that ever opens knows it. The concept is that simple. The stacking tree: it reads the folder's CLAUDE.md, the folder above, up to root — computer-wide rules at ~, all-repos rules at ~/github, project rules in the repo; a session in src/ reads all three. Put each rule at the level where it applies. What goes in: hard rules, holistic guidance, operational procedures, context loading. What doesn't: too vague ("always write good code" — worthless) and too draconian ("never have a Python file over 200 lines" — weird, limiting). Land the line: everything in it enters EVERY conversation — don't describe your whole project here, point to the file that describes the project.

Act III · 330 — skills

The karate chip

Some tasks recur — but not often enough for CLAUDE.md. You keep priming the chat with the same chunk of information.

code review — your specific review guidelines, restated every time database deploy — the usernames, the ports, the steps — looked up again write like me — 15 prime examples + the word rules, re-pasted every session → the first-class solution: a skill

Anatomy — and when it loads

—— front matter, always indexed ——
name: database-deployer
description: use whenever the user asks to deploy the database
—— the instructions, loaded on match ——
how to do the deploy: steps, credentials file, checks…

every chat has read every skill’s name + description — nothing more; say “let’s deploy the database” and that one skill’s body loads mid-conversation

like the Matrix: you need karate, you jack in the karate chip

A skill is a folder

.claude/skills/
└─ deploy/
   ├─ SKILL.md   ← the instructions
   ├─ is-docker-running.sh
   ├─ make-new-database.sh
   └─ teardown-worktree.sh  …~8–9 scripts in my real deploy skill

markdown, HTML, code, even whole websites ride along — helper code the model calls instead of reads; code beats prose

~/.claude/skills — your whole computer .claude/skills — that repo

Two warnings. The whole skill downloads into your current window — every word costs context, and a task-B skill mid-task-A pollutes the window. And don’t hoard: 600 skills and the orchestrator can’t choose — a couple dozen, max.

key ideaA skill is a folder (SKILL.md + callable helpers) whose body loads only when the task matches its description — consistency without context pollution.

doCodify any process you do more than once as a skill; how-to in SKILL.md, repeated operations in helper files the model calls.

SPOKEN (v3): Some tasks are too infrequent for CLAUDE.md but they recur — and every time, you're pasting the same files and restating the same rules. The LLM companies built a first-class solution. The repeated-priming problem: code review guidelines, database-deploy steps, write-like-me examples. Anatomy: front matter (name + description) always indexed; every chat has read every skill's name and description and nothing more; when the task matches, the orchestrator loads that one skill's body into the chat — like the Matrix: you need karate, you jack in the karate chip. A skill is a folder: SKILL.md plus helper scripts the model CALLS instead of reads (code beats prose) — ~8-9 scripts in my real deploy skill. Locations: ~/.claude/skills for your whole computer, .claude/skills for that repo. Two warnings: the whole skill downloads into your CURRENT window — every word costs context, and a task-B skill mid-task-A pollutes the window; and don't hoard — 600 skills and the orchestrator can't choose, a couple dozen max.

Act III · 340 — sub-agents

Side work in a side conversation

Exactly like a skill — but instead of loading into your window, it opens a new side chat and loads there. The main thread stays clean.

main session

stays on the feature work

task + requirements + git diff →← findings

review agent

a fresh window — no accumulated justification

Why a fresh window: the window that wrote the code is full of chain-of-thought arguing it should be exactly as it is. A clean window sees only the code + the review instructions — no fighting its own tokens.

main window

plans with you

→

implementation agent

writes the code in its own bubble

→

review agent

reviews in its own bubble

agent.md front matter: name · description (when to use) · model · permissions — read-only, no bash… more on that later

Using one is one question: do I want this in my conversation — or in a side conversation?

key ideaA skill loads context into the current chat; an agent spawns a new side chat from its agent.md. Same concept, opposite direction.

doPush anything that would bias or bloat the main thread into a fresh window — canonically code review: clean window + requirements + git diff.

SPOKEN (v3): An agent is exactly like a skill — except instead of downloading into YOUR window, it makes a new side chat, and agent.md downloads there. The unbiased-review argument is the heart: the window that wrote the code is full of self-justifying chain of thought; a clean window sees only the code + the review instructions. The flow: main window plans with you → implementation agent writes in its own bubble → review agent reviews in its own bubble. agent.md front matter: name, description (when to use), model, permissions (read-only, no bash — more on that later). Model precedence, verified 2026-07-19 (research/lookups-braindump-09.md): if the orchestrator picks a model at call time, that call-time choice OVERRIDES the frontmatter's model (env var > call-time param > frontmatter > inherited). Using one is one question: do I want this in my conversation — or in a side conversation?

Act III · 400 — permissions & read-only agents

It literally can’t write

Last slide’s “only read” was an instruction — instructions aren’t enforcement. A read-only agent can’t write: permissions take the tool away.

agent	read	write	run it unattended?
review agent	✓	✗	freely; it can’t do harm
morning automations	✓	✗	behind permission barriers
implementation agent	✓	✓	needs the next lever →

sometimes writing is the point, so you can’t remove it → hooks, two slides ahead, are the deterministic backstop

key ideaAgents have permissions. A read-only review agent is incapable of writing, so you can let it do anything.

doRead-only for reviewers and investigators; permission barriers on unattended runs; hooks where write can’t be removed.

Act III · 345 — stacking agents

Stack agents into pipelines

Agents working in parallel or in sequence — for a job no single session could hold.

The inherited codebase

~10 years of creative techniques + spaghetti, our new opinionated stack on top. Task: refactor to modular & deployable without breaking anything — weeks-to-months by hand, no chance in one LLM shot.

What I actually needed

A tutorial of the codebase: the features, the API points, how backend talks to frontend. One session can’t write it — context rot, and each area needs its own specialist.

1 · survey agentscans the repo, decides how to carve it up — the orchestrator spawns a writer per piece

↓

2 · about a dozen writer agents in parallel — one per section, each documents its area into a durable artifact

↓

3 · synthesizerstitches the pages into one coherent tutorial

↓

4 · augmentglossary · cast of characters · key decisions · system seams

5 · quiz agenta 12-question quiz

the live tutorial site

durable artifacts persisted between every stage · stages 4–5 are optional polish passes — survey → writers → synthesizer already yields a complete tutorial

codebase-tutorial.codebycarson.com/t/mmgis — the real run, live

key ideaToo big for one session? Stack agents: fan out one per area, then synthesize across them — coverage from the fan-out, coherence from the synthesis.

doBreak the task into consumable chunks and chain agents over them — parallel where independent, serial where not.

SPOKEN (v3): The next logical step: big pipelines full of agents, in parallel or in sequence. Real story — a ~10-year codebase written by people whose primary work wasn't coding: creative techniques, legacy spaghetti. We inherited it, loved the features, stacked our new opinionated tech on top — a nightmare mix. My task: refactor it modular + deployable without breaking anything. Weeks-to-months by hand; zero chance an LLM one-shots it. What I needed: a tutorial of the codebase. The pipeline (verified against the real repo): a survey agent scans and decides how to carve it up — the orchestrator spawns a writer per piece; about a dozen writer agents in parallel, one per section, each documenting its area into a durable artifact; a synthesizer stitches the pages into one coherent tutorial; optional augment (glossary, cast of characters, key decisions, system seams) and a 12-question quiz agent. Durable artifacts persisted between every stage. LIVE: codebase-tutorial.codebycarson.com/t/mmgis (fallback screenshot ~/github/code-by-carson/site/images/codebase-tutorial.png). Teaching line: coverage from the fan-out, coherence from the synthesis.

Act III · 410 — hooks

Your program, on every event

Rules no classifier could guess — “never commit to main” — enforced by your own program, every time.

flavor 1: block

every bash command → hook runs your Python script

↓ anything git-related? a sub-agent asks: “is this going to main?”

yes → BLOCK: the absolute rule, enforced

flavor 2: trigger

a commit happens → hook fires

↓ documentation agent reads the commit, updates docs if needed

docs stay current, seamlessly, over time

where a hook can fire — twelve of ~30 events

PreToolUsebefore a tool runs

PostToolUseafter it succeeds

SessionStartwhen a session starts

SessionEndwhen the session closes

UserPromptSubmitbefore your prompt processes

Stopwhen the turn ends

SubagentStartwhen a sub-agent launches

SubagentStopwhen a sub-agent finishes

Notificationwhen Claude needs you

PreCompactbefore history gets compressed

PermissionRequestwhen approval gets requested

FileChangedwhen a file changes

your script’s exit code decides: 0 = allow · 2 = block (stderr becomes the model’s explanation) · anything else = non-blocking · handlers can also be http calls, MCP tools, prompts, or whole agents

key ideaHook = your program, run on a specific event — it can veto an action or fire a deterministic follow-on.

doAbsolute rule? Hook the event and block it. Automation? Hook commits to trigger a docs-update agent.

TRANSITION: "For rules no classifier could know, like never commit to main, you lash your own program to the event: hooks." CLAUDE.md is advice the model usually follows; a hook is a rule the harness ALWAYS enforces. Deterministic, not statistical. Two flavors: block (the PreToolUse script that stops commits to main) and trigger (commit fires a docs-update agent). ⚠ Event list verified 2026-07-19 vs code.claude.com/docs/en/hooks.md — 30 documented events; the curated twelve on screen per research/lookups-carson-questions-0719.md. Keep the exit-code line verbatim: 0 = allow · 2 = block (stderr becomes the model's explanation) · anything else = non-blocking; handlers can also be http calls, MCP tools, prompts, or whole agents. Configuration lives in settings.json ("hooks": {"PreToolUse": [{"matcher": "Bash", ...}]}).

Act III · 350 — the orchestration session

One session you talk to

Sit in one orchestration session; it spawns sub-agents for the independent tasks.

orchestration sessionplans with you — saves durable to-do + requirements artifacts

↓ spawn ↓

researchwrites a durable artifact

implementbriefed by the orchestrator — or reads the to-do artifact

test

review

results flow back up — your job: keep the orchestrator (and yourself) aligned

worked example: this very course · MMGIS: 13 PRs from one session — massive parallel agents underneath

key ideaSub-agents become a workflow when one persistent orchestration session holds the big picture while they fan out on the pieces.

doConsider spawning sub-agents from an orchestration session — research, implementation, test, review each in its own bubble.

Act III · 370 — the harness

Everything around your model is your harness

Don’t use the browser — use the local tool. Claude Code is a whole coding harness around the model.

CLAUDE.md

skills

orchestrator + your conversationnaked model inside

agents

hooks

built in: automatic to-do tracking · automatic agents · security checks

The generation gap: Opus 4.7 with a really good harness can beat Opus 4.8 bare — effort on your harness can outperform a model upgrade.

key ideanaked model + orchestrator + skills + agents + hooks + CLAUDE.md = the harness.

doBefore real work, inventory the harness — skills, agents, hooks, CLAUDE.md — and build the missing pieces rather than expecting a bigger model to compensate.

Act III · 380 — superpowers

Craft your harness — or download one

Skills and hooks let you craft your own — or download one the open-source world already built. I use two: Claude Code and superpowers.

what superpowers is: bundled skills + an opinionated workflow

brainstorm

asks questions, explores alternatives first

→

design spec

YOU read + double-check it

→

implement

a fresh sub-agent per task — test-driven development baked in

→

review agents

against the spec first, then code quality

install: github.com/obra/superpowers — every session has it

key ideaCommunities have built really good harnesses — superpowers among them: bundled skills plus an opinionated brainstorm → spec → implement → review workflow.

doFind a harness that aligns with how you want to work — install it, use it.

Act III · 390 — auto mode & safety

Is auto mode the safer option?

Clicking yes-yes-yes without reading isn’t safety — it’s slower and blinder. Two habits first, then the gate.

98% of the time: auto mode get there: shift-tab cycles default → accept-edits → plan → custom modes like auto

1 · Back up first

code → git · files → Drive / S3 · databases → db backups

use the model to help — backing everything up is an afternoon, not a project

2 · Prompt hygiene

“investigate only — don’t touch production” bounded; the model follows instructions well

“do whatever you need to do to solve the problem” that includes deleting stuff

doGet everything durable first (GitHub + cloud), practice prompt hygiene, then run auto mode.

SPOKEN (v3): Now you know how my system is set up — how do I actually work day-to-day? Babysitting every yes-prompt kills the force multiplier, and you end up clicking yes like a robot anyway — clicking yes-yes-yes without reading isn't safety; it's slower AND blinder. 98% of the time: auto mode. Get there with shift-tab: it cycles default → accept-edits → plan → custom modes like auto (verified 2026-07-19, research/lookups-braindump-09.md). Habit 1, back up first: code to git, files to Drive/S3, databases to db backups — use the model to help; it's an afternoon, not a project. (Backup story stays spoken: "find every file with code on my computer" → pick 13 → GitHub repos in 20 minutes.) Habit 2, prompt hygiene: "investigate only — don't touch production" is bounded and the model follows instructions well; "do whatever you need to do to solve the problem" includes deleting stuff. The gate itself is the next slide. A year in with this exact implementation: it's more conservative than I am — every flag was something I hadn't been explicit about.

Act III · 390

The two-classifier gate

only tool calls can do harm; talking is free

bash command

↓

classifier 1: could this do something bad?no → allow

yes ↓

classifier 2: did you explicitly approve this?yes → allow

no ↓

BLOCK: the calling agent is flagged; it does something non-destructive instead

3 denials in a row — or 20 total ↓

“what do you want to do, boss?”, surfaced to you

stage 1 = cheap cautious filter on every call · stage 2 = a chain-of-thought pass (incl. the did-you-approve-this check) only on flagged calls · published numbers: 0.4% false-positive, 17% false-negative (n=52)

key ideaOnly tool calls can do harm. Auto mode gates each command: is it dangerous? → did you approve it? → block, flag, and escalate after repeated tries.

Act III · 430 — GitHub for teams

Too much code = rubber stamp

Known-good master · work tracked on issues · reviews small enough to actually read.

milestone

a really large achievement

→

issue

+ sub-issues — the durable record of why

→

feature branch

→

PR

→

review

feedback + everyone stays current

→

merge to master

the definitely-working code

track work on issues — motivation stays traceable from any PR, long after the work closes Claude is great at GitHub via the CLI — “which PR merged this line, and why?”

PR size discipline — a 10,000-line PR never gets read

it sits and rots — or someone scans it and rubber-stamps it. Nobody ever knows if the code was good.

say the task

Claude breaks it up

→

small stacked branches

implementation agents write + report back

→

orchestrator keeps you informed

→

one artifact links every PR

per-PR summaries — review in meaningful chunks

key ideaTrack work on issues; keep every PR small enough to actually read — let Claude break tasks up, stack branches, and link the PRs in one artifact.

doRight-size every task: issue → branch → one small PR → review → merge — and understand each diff with an artifact or the explainer.

SPOKEN (v3): We've talked a lot about coding by yourself — how do you code with a team? Known-good master, work tracked on issues, reviews small enough to actually read. The flow: milestone (a really large achievement) → issue + sub-issues, the durable record of WHY → feature branch → PR → review (feedback + everyone stays current) → merge to master, the definitely-working code. Track work on issues — motivation stays traceable from any PR long after the work closes; Claude is great at GitHub via the CLI: "which PR merged this line, and why?" PR size discipline: a 10,000-line PR never gets read — it sits and rots, or someone rubber-stamps it; nobody ever knows if the code was good. So: say the task, Claude breaks it up, small stacked branches with implementation agents writing and reporting back, the orchestrator keeps you informed, and one artifact links every PR with per-PR summaries — review in meaningful chunks.

Act III · 430 · live demo

The diff explainer

design once → generate forever: the reusable front end the pipeline fills per-PR

Small diff?

artifacts are great now — just ask Claude to describe it in an artifact

Big or unfamiliar PR?

my explainer pipeline — agents explore the diff, document it tutorial-style into a reusable front end

live demo diff-explainer.codebycarson.com — design once, generate forever a real MMGIS PR walk-through, rendered by the reusable front end the pipeline fills per-PR

key ideaBuild the front end once; the pipeline fills it forever. One-off artifacts are the lightweight tier.

Act III · 440 — git worktrees & parallel work

A branch in its own folder

A branch given its own folder — real files, real tests, parallel agents.

the old way: serial

~/space-laser/ (checked out: master)

git checkout focusing-mirror
→ the whole folder becomes the mirror work
git checkout targeting-system
→ …and now it’s the targeting work

one folder, one thing at a time

worktrees: parallel

~/space-laser/ (master)
~/laser-focusing-mirror/ (focusing-mirror) ← agent A tests here
~/laser-targeting-system/ (targeting-system) ← agent B tests here

as many working folders as worktrees; real files, real tests, in parallel

Superpowers handles worktrees by default — and a complicated repo can be pre-configured to make worktrees easy, e.g. independent databases per worktree.

key ideaA worktree gives each branch its own folder — parallel agents each get real code, tests, and scripts on disk.

doLet the harness deploy tasks into worktrees and clean them up — and override the default when it doesn’t fit your setup.

Act III · 450 — research agents

How to do research with an LLM

How you search determines what you get.

my deep-research agent, as it actually runs

clarify scope

2–3 questions; save the constraints

→

set up the log

background.md exists before any search

→

search → read → log

the loop: cite every fact; never two searches without a write between

→

synthesize the report

distill the log into a clean, cited writeup

techniques that get it to the information

today: search + fetchthe agent itself runs on plain WebSearch + WebFetch

hard PDF → markdowna separate skill in my library — translate once, read forever

next: gated journalsa login agent with your credentials — where I’m taking it

next: credentialed APIslogin keys for the sources that offer them

supplement the agent with skills that reach the sources you care about

key ideaHow you search determines what you get — a good research agent constrains the pipeline: scope, log first, search-read-log, synthesize.

doWrite a research agent that searches your way, over your sources — with skills that get it to the information.

SPOKEN (v3): Grounding — we're moving into the science-tools stretch: first research, then math. The pipeline on screen is the REAL ~/.claude/agents/deep-research.md, rebuilt 2026-07-19 (research/act3-slide-sources.md) — it replaces the older brain-dump/categories/trust-boxes version, which was wrong. As it actually runs: clarify scope (2-3 questions, save the constraints); set up the log — background.md exists BEFORE any search; the loop: search → read → log, cite every fact, never two searches without a write between; synthesize the report — distill the log into a clean, cited writeup. Techniques: today it runs on plain WebSearch + WebFetch; a hard-PDF→markdown skill lives in my library (translate once, read forever); next: gated journals via a login agent with your credentials, and credentialed APIs where sources offer them. Supplement the agent with skills that reach the sources you care about.

Act III · 460 — math with LLMs

Agents don't do math

A naked model doing arithmetic is guessing at numbers — turn your math into code.

Math in the weights

the model “evaluates” your equation in its head

plausible-looking arithmetic — strawberry all over again. Fine for exploring; never for the result.

Math as code

the same equation, written and executed

durable · executable · re-runnable — in Python, or a proof system like Lean 4 + mathlib (machine-checks proofs; a small but fast-growing slice of research is formalized this way)

equations as Python, in libraries that run them — readable, so you can verify the equation documentation tied to each equation — which does what, why it’s used — so the LLM understands and executes

key ideaMath → code. Code is durable and executable; a naked model doing math is guessing at numbers.

doConvert your math to code.

THE WORKING-WITH-LLMS SECTION

How I actually work

A philosophy for big projects — not to copy exactly, but enough information to figure out what works for your team.

the setup → out of your head → vision → documentation → tasks → the closed loop

Act III · 500 · the setup

What's left is misalignment

With the stack solved, bad code stops being the likely failure. Misalignment is what’s left.

aligned with what you actually wanted?the one remaining failure mode

rests on

specialized harness — superpowersbrainstorm · design specs · testing ✓

harness — Claude Codedelegation · to-do tracking · safety ✓

chain of thoughtthinks over each part of the problem ✓

attention across the window✓

frontier model✓

key ideaA good model with chain of thought and a harness — the only failure left is misalignment with what you wanted.

doGive the model enough information.

Act III · 310 · word vomiting

Say whatever the hell you want

Wording, flow, keeping it short — the hard parts of communicating are what LLMs do best. Stop shaping. Dump.

🎤 dictate

“say whatever the hell you want”

→

one window

organizes it + acts on it

really big project?

same vomit

→

durable artifact

becomes the guide for the work

~95% of my words to any LLM are spoken, via the macOS dictation button · critical task? read the organized version before it runs

key ideaMore of yourself in the context = better-aligned output — organizing is the part you offload.

doHit dictate and brain-dump. Let the agent organize + act in one window; for a big project, have it organize the vomit into a durable artifact first.

Act III · 510 · text you already have

Take advantage of text you already have

Lots of good context is already written.

Your head ↩ the word-vomit

🎤 the word-vomit: funnel one — you just did it; it points back, not forward

Your disk

what you already wrote: funding proposals · papers · group website · notes · Google Docs

↓ if reasonably current

use LLMs to extract the content: PDF → markdown once; every later consumer reads the markdown

↓ the context window, filled before you ever type detailed instructions

key ideaThe second high-volume channel is already written: your existing documents, gathered once and made durable.

doGather your current documents and use LLMs to extract the content; translate PDFs to markdown once.

Act III · 520 · vision.md

The stakeholder-alignment layer

1–2 pages, bird’s-eye: what · why · who · integration — the stakeholder’s power, given to every session.

CLAUDE.mdrules, every session; points to the rest

↓

vision.mdwhat · why · who · integration

↓

documentationwhat no agent could know

↓

taskswhat to work on

1–2 pages, bird’s-eye · NOT functional requirements · NOT implementation

“Yo — that’s not even aligned with what we’re trying to do here.”
the stakeholder test — what the vision lets every session say

word-vomit it with whoever holds the vision; a skill drafts it · read it. own it. write once → guiding lightstone

Why really matters. 2024: craft prompts to sneak up on what you wanted. Now: just say why — the model reasons over it.

key ideaThe vision is a big-picture decision matrix — what/why/who/integration — whose test is: can it settle “is this even aligned with what we’re trying to accomplish?”

doWord-vomit with whoever holds the vision; let the skill draft it; then read it, own it, and make it near-perfect once.

SPOKEN (v3): For a whole project the information gets layers, with CLAUDE.md pointing to the rest. First layer down: vision.md. This is where "why" lives. 1–2 pages of what / why / who / how-it-integrates. Never functional requirements, never implementation. The test: can it settle "is this even aligned with what we're trying to do?" — the power a stakeholder has, given to every session. Process: word-vomit with whoever really holds the vision (team lead, PM); the writing-a-vision skill drafts it; then read it and own it — write it once, make it near-perfect, and it's a guiding lightstone for every future session. And why really matters, which wasn't true in 2024: since chain of thought, the model reasons over your why and makes better low-level decisions from it. This four-layer stack anchors the next three beats — same diagram, different layer lit.

Act III · 530 · documentation

What no agent could know

Not tutorials on what’s already in the weights — the knowledge insiders carry silently.

CLAUDE.md

↓

vision.md

↓

documentationthis layer

↓

tasks

business knowledgecustom to your repo: obvious to insiders, invisible to outsiders, not in the weights

key interactionsnon-obvious connections between pieces of the codebase

rationale / constraintswhy it’s this way, and why it must stay that way, so a refactoring session sees the hard limit

start from the vision skill’s overflow doc, not a blank page · scientists: the exact papers, in the repo, equations verified; your science IS the business logic

key ideaThree targets: business knowledge, key non-obvious interactions, and rationale/constraints. The things no agent could otherwise know.

doUse the vision skill’s overflow as the starting place; keep the exact papers in-repo with equations verified; then read it and own it.

Act III · 540 · tasks & issues

It knows the project — now tell it its job

Vision + docs = knows the project. The issue = knows its job.

CLAUDE.md

↓

vision.md

↓

documentation

↓

tasksthis layer

anatomy of a good issue

why / motivationwhat we’re actually trying to achieve, up top

core requirementsreadable by the whole team

▸ commit-pinned implementation sketchcollapsed underneath; what the agent works from

word-vomit the task → the issue skill drafts it · too big → the skill breaks it down (unreviewable PRs = too big)

key ideaA good issue starts with why: motivation, goal, core requirements. The whole team must be able to read it.

doWord-vomit the task; let the issue skill right-size it. Readable top layer over a collapsed, commit-pinned sketch.

Act III · 550 · the closed loop

Every failure becomes context

Every failure routes back to the layer that should have prevented it.

CLAUDE.md↩ docs existed but unread → the read-list

↓

vision.md↩ gross alignment mistakes

↓

documentation↩ misunderstood fundamentals (it botched that equation)

↓

tasks

↓

+ the code-review agent↩ review misses that were obvious to you

a session runs↓ you observe a failure, something you don’t like

route itto the layer that should have prevented it ↩

next session starts smarterthe setup gets better the longer you run it

every observed failure becomes context for the next session.

key ideaAlignment → vision · fundamentals → documentation · unread docs → CLAUDE.md · review misses → the review agent.

doClose the loop: treat each mistake as a routing decision, not an annoyance. Capture it into the layer that failed.

closing · where this leaves you

Let's go back over everything we did

every key idea + actionable concept — one page, yours to take home · Acts I–II

key ideas

030	A word's meaning is the company it keeps.
030	Word vectors put definitions into numbers — similarity becomes measurable.
040	The corpus decides the definition — one averaged vector per word.
060	Attention + next-word prediction = a text generator.
070	Scale alone made prediction shockingly capable.
080	Post-training turns "continue" into "answer."
090	A naked LLM has no facts — it's truthful only because the corpus mostly was.
130	Every turn replays the whole transcript into a stateless model.
140	A bigger window is not uniformly good context.
145	The window out-votes the weights — everything in it votes.
150	Style lives in the weights and the window.
160	Context is everything.
170	naked LLM + tool + orchestrator = every modern assistant.
180	RAG pulls the real source into the window at question time.
190	Code is constrained; the model is more capable there.
210	Gains now come from better data, not more of it.
220	Reasoning is next-word prediction over generated sub-steps.

actionable concepts

120	Estimate tokens as characters ÷ ~4.
090	Distrust unsourced claims; drop "hallucination."
130	Unstick by pruning or resetting the history.
140	Know when to end the conversation.
145	Keep tool-bearing sessions away from untrusted text.
150	Paste 3+ samples of the target voice.
160	Paste the paper; ask about that.
170	Route work to tools.
180	Route factual questions through retrieval.
190	Sometimes you want the answer from code, not the LLM.
210	Judge new models by data and technique, not size.
220	Spend thinking only on complex tasks.

closing · where this leaves you

…and the agentic act

Act III — the tool ladder

key ideas

360	An artifact is anything outside the chat window — durable, read fresh every turn.
320	CLAUDE.md enters every conversation — only always-applicable rules belong.
330	A skill loads only when the task matches — the karate chip.
340	Skill = context into this chat; agent = a fresh side chat.
345	Too big for one session? Stack agents — fan out, then synthesize.
350	One big-picture session; sub-agents do the breadth.
370	Everything around your model is your harness — it can beat a model upgrade.
380	Communities have built really good harnesses — superpowers among them.
390	Only tool calls can harm; two classifiers gate each one.
400	Read-only agents can't do harm — the tool isn't even there.
410	A hook is a rule the harness always enforces.
470	A Claude Artifact is the response, beautifully formatted — and shareable.
430	Too-big diffs get rubber-stamped — right-size everything.
440	A worktree = a branch in its own folder — true parallelism.
450	How you search determines what you get.
460	Agents don't do math — math becomes code.

actionable concepts

360	Pull to-dos, debug logs, drafts out of the chat and onto disk.
320	Direct rules: hard rules · guidance · procedures · context loading.
330	Codify any repeated process as a skill.
340	Review from a clean window: requirements + git diff.
345	Chain agents over consumable chunks — parallel where independent.
350	Consider spawning sub-agents from an orchestration session.
370	Inventory your harness; build the missing pieces.
380	Find a harness that fits how you work; install it, use it.
390	Back up everything, prompt cleanly, run auto mode.
400	Scope tools to the job; fence unattended runs.
410	Block absolute rules; trigger follow-on agents.
470	End big sessions with "make me an artifact."
430	Issue → branch → small PR → review → merge; artifact or explainer on the diff.
440	Let the harness manage worktrees; override when needed.
450	Write a research agent: your sources, your method, skills that reach them.
460	Convert your math to code.

closing · where this leaves you

…and how I actually work

Act III — the working-with-llms philosophy

key ideas

500	With the stack solved, misalignment is what's left.
310	More of yourself in the context = better-aligned output.
510	Lots of good context is already written.
520	The vision is the stakeholder-alignment test.
530	Document what no agent could know.
540	Vision + docs = knows the project; issues = knows its job.
550	The layers are the failure-routing table.

actionable concepts

500	Give the model enough information.
310	Dictate raw; let the model organize — and act.
510	Gather your documents; LLMs extract; PDF → markdown once.
520	Write it once, near-perfect; read it, own it.
530	Business knowledge · interactions · rationale — equations verified.
540	Word-vomit the task; the skill right-sizes it.
550	Every failure becomes context for the next session.

closing · the take-home

Everything, one page

Every key idea, every actionable concept, and every referenced link from the whole talk — one page on the site.

llms.codebycarson.com

How LLMsactually work

I'm Carson. And I am very lazy.

The next three hours.

How LLMs actually work

Getting computers to understand language

4,000 patents, a few hundred thousand companies — and the gap is what words mean

a welding patent, 1–2 pages

a manufacturer, one of several hundred thousand

Meaning is the company a word keeps

It made numbers that work

The corpus is the definition

1 — one averaged definition

2 — positional ignorance

Four banks, one blurred vector

“Attention is all you need” — 2017

Advance 1 transformer · 2017

Advance 2 ELMo · BERT · 2018

Models don’t read words

Just add more data

Why does it answer instead of continue?

These are not the facts you're looking for

One distribution, reshaped

The naked model — where ChatGPT started

Amazing

Frustrating

Working with basic LLMs

The conversation is an illusion

A million tokens — not a million good ones

Everything in the window votes

Method 1 — context loading

Method 2 — rewrite the history API only

The threat side — prompt injection

Style lives in the weights and in the window

source 1 · pretraining data

source 2 · post-training

source 3 — the overlooked one

Fix 1 — just ask

Fix 2 — few-shot: show it the voice

Context is everything.

Ask the weights

Paste the source

naked LLM + tool + orchestrator

Three flavors of orchestrator

The same LLM, post-trained

A tiny specialist LLM

An ordinary model, prompted

Give the orchestrator a search bar

question

orchestrator

search tool

the real page

context window

grounded answer

Ask for the program, not the answer

In its head

Through code

Chain of thought

Without thinking — one pass, straight to the answer

With thinking — the same single generation, longer

The cost of thinking

Modern agentic work

Escape the chat window

Buried at the top of the chat

Re-read at the bottom, from disk

improves its thinking

lets you spot the bug

transfers between sessions

Stop reading the walls of text

wall of text

“make me an artifact”

formatted page + private or public link

a colleague reads it

learning a concept

mapping codebase interactions

project status + next actions

Instructions it reads every time

The karate chip

Anatomy — and when it loads

A skill is a folder

Side work in a side conversation

How LLMs
actually work