Working with LLMs / Fact sheet

Draft — this material tracks a talk that hasn't shipped yet. Details may still shift before the final version.

The Fact Sheet — every key idea + every actionable, in order

The whole talk on one page. Each beat: Key idea (A) and, where there is one, Do (B) — so you can see the A-B, A-B, A-B rhythm all at once instead of scrolling the deck. Beats that only build the mental model have no "Do" — that's on purpose.

Act I — How LLMs actually work

010 · Hand-built definitional matrices Key idea: You can't hand-write the meaning of every word as a matrix of yes/no features — picking the dimensions and staying consistent across a whole language never scales.

020 · The NASA patent-matching problem Key idea: When the whole job is comparing meaning across two vocabularies, hand-built keywords and dictionary synonyms only get you part way — the missing piece is a real measure of semantic similarity.

030 · Embeddings (taught via GloVe) Key idea: A word's meaning is the company it keeps — count how often words co-occur, factorize that matrix, and similar-meaning words land on similar vectors. (Two recipes: word2vec 2013 predicts neighbors, GloVe 2014 counts then factorizes — secretly almost the same trick.) An embedding is a single pass: one vector, one definition per word.

040 · What embeddings bought us Key idea: Meaning becomes geometry — related things cluster, so you can search by meaning. But each word still gets one definition, and the corpus decides it (ICU = "I see you" on texts, "intensive care unit" on medical docs). That's semantic search — not RAG yet.

050 · The two walls Key idea: One static vector can't cross two walls — polysemy ("bank" four ways, one vector) and composition (word order: "not high-performing" flips the meaning even when every word's definition is right). Neither is fixable by better per-word definitions.

060 · Transformers → next-word prediction Key idea: Attention lets words look at each other, so a word gets a different vector in every sentence — breaking both walls. Contextual embeddings fixed polysemy (ELMo's LSTM first, BERT's transformer made it dominant); masked / next-word prediction turns it into a text generator.

070 · The scaling bet Key idea: Pour enough text into next-word prediction and it gets shockingly capable — GPT-1 terrible → GPT-2 better → GPT-3 good. Same idea, more scale.

080 · Post-training into a chatbot Key idea: Post-training turns "continue the text" into "answer the question" — the InstructGPT recipe: supervised fine-tuning on ~10k hand-written answers, then a reward model + RLHF. It's a knob the company controls.

090 · Statistical truth, not knowledge Key idea: A naked LLM has no facts — it's a statistical next-word predictor that's truthful only because most of its training text was. Not confidently wrong, because it was never confidently right. Do: Distrust unsourced factual claims from a bare model; drop the word "hallucination."

100 · Temperature & creativity Key idea: Output varies because you sample from a probability distribution that temperature reshapes; creativity is structured novelty the viewer values (it can say "jaguar" though the corpus only ever said "cat," because their vectors sit next to each other). Do: Raise temperature for novelty; lower it to pull the single most-likely, near-deterministic answer.

110 · The naked model (where ChatGPT started) Key idea: Fluent and coherent, but no facts and falls apart on complicated, multi-part instructions — the two frustrations the entire second half exists to fix.

Act II — Working with basic LLMs

120 · Tokens Key idea: A token is a learned, semantically-meaningful chunk of text — not necessarily a whole word ("geology → geo | logy"). Every model has its own tokenizer; it's all priced in tokens. Do: Estimate tokens as characters ÷ ~4; expect the count to differ per model.

130 · The conversation illusion Key idea: There's no memory — every turn replays the whole transcript (A → AB → ABC…) into a fresh, stateless model that has never seen a word before. Do: To unstick a model, don't argue at the end of a long thread — reset or prune the history, because 100k tokens of the old idea outweigh 50 of the new one.

140 · Context window & context rot Key idea: A bigger window is not uniformly good context — recall sags in the middle and as it fills ("context rot"). 8k → 1M+ tokens, but not a million tokens of equal quality. Do: Keep the window lean and relevant; don't dump everything in; put what matters where the model can attend to it.

150 · Style control & few-shot Key idea: Style lives in the weights (pretrain + post-train) and in the window (in-context examples) — you can override the weights by putting examples in the window. This is few-shot / in-context learning. Do: Paste 3+ samples of the target voice; for weaker models, decompose — emit a style-guide template, fill it from your sample, then write to it.

160 · Grounding facts in the window Key idea: Don't ask the weights for a fact — supply the source. Context is everything; what you engineer into the window is what you live or die on. Do: Paste the paper / the data and ask about that, rather than asking the model from memory.

170 · Tool calling & the orchestrator Key idea: naked LLM + tool + orchestrator — the orchestrator hijacks the request, calls the tool (a calculator for 14.7 ÷ 16.9), and folds the result back into the context. Do: Think in terms of routing work to tools, not doing everything inside the chat window.

180 · Search & RAG Key idea: Retrieval augments generation — the orchestrator fetches a real page and drops a summary into the window at question time. That's RAG. Do: Route questions that need a live/authoritative source through search; later, build your own RAG over your documents.

190 · Why code beats prose Key idea: English is vast and ambiguous; Python is tiny and constrained, with code sitting next to what-it-does in the training data — so models are better at code ("how many R's in strawberry?" → 3 ✗). Do: For anything code is good at (counting, math, parsing), ask the model to write and run the program, not to do it from memory.

200 · Model-shopping literacy Key idea: Parameters ≈ memory (a 70B model ≈ 140 GB native FP16, ~70 GB at 8-bit, ~40 GB at 4-bit); quantization trades precision for size with sub-linear quality loss; GPU memory beats spilling to RAM (bus speed). Do: When picking a local model, check size and quantization and read the published accuracy — a quantized 36B can beat a 120B crushed to the same footprint.

210 · The scaling era ends Key idea: More-data-forever broke (GPT-4 → GPT-4.5 disappointed); we ran out of internet ("peak data"), so gains now come from better data — filtered, synthetic, human — not just more of it. Do: Don't assume each new model is a giant leap from scale alone; weigh the data and technique, not just the size.

220 · Chain of thought & the cost of thinking Key idea: Chain of thought is the model writing out reasoning tokens before it answers (not chopping the input into chunks) — reaching answers a single pass can't. You pay for that scratch work as output tokens. Do: Turn thinking up for complex, interdependent tasks and down for simple ones; judge models on token-spend-vs-performance, not raw size.

Act III — Modern agentic work

310 · Word vomiting Key idea: The more of yourself you get into the context, the better aligned the output — and organizing and summarizing (the hard parts of communicating) are exactly what LLMs are best at. Do: Dump everything raw → let the LLM organize it into a durable artifact → implement from it. Skip the middle step on strong models; split it across two sessions on weak ones.

360 · Artifacts Key idea: A to-do list at the top of the chat competes with the whole transcript for attention; the same list re-read at the bottom is fresh every turn, shows what's done, and survives across sessions. Do: Word-vomit → task.md → a ~15-step checklist the model writes each result under and checks off. Same pattern for infrastructure logs and debugging logs.

320 · CLAUDE.md Key idea: CLAUDE.md is read every conversation — its power is its weakness: everything in it affects every chat, so it earns its place only if it's specific and always-applicable. Do: Write direct, followable rules (class-based architecture; commit after tests pass; read the README first; never commit private keys) — not too vague, not too draconian.

330 · Skills Key idea: A skill is a folder (SKILL.md + optional callable code) whose frontmatter carries a name and a when-to-use description; only name + description load until a task matches — so 15 skills don't drown every chat. The karate chip. Do: Codify any process you do more than once as a skill; put the how-to in SKILL.md and repeated operations in helper files the model calls rather than reads.

340 · Sub-agents Key idea: Skill vs. agent is one question — do I want this info in my current conversation, or in a side one? A skill loads context into the chat; an agent spawns a fresh side chat from its agent.md. Do: Use a fresh window for anything whose reasoning shouldn't bias the main thread — canonically code review: hand a clean agent the task, the requirements, and a git diff, and have it judge on set areas (bugs, logic, DRY-ness, unneeded functions).

350 · The orchestration session Key idea: Sub-agents become a workflow when one persistent orchestration session holds the big picture — the one you talk to — while sub-agents fan out on the pieces (MMGIS: 13 PRs from one session). Do: For any large project, make one big-picture session home base; delegate research / alternatives / review / implementation to sub-agents and spend your own effort keeping it aligned.

370 · The harness Key idea: naked model + orchestrator + skills + agents + hooks + CLAUDE.md = the harness. Useful work is a property of the whole system, not the raw model. Do: When planning real work, inventory your harness — what skills, agents, hooks, CLAUDE.md — and build the missing pieces instead of expecting a bigger model to compensate.

380 · Superpowers (a pre-built harness) Key idea: The patterns you just built by hand — artifacts, sub-agents, review — are already bundled. Superpowers turns a task into a design-spec artifact, tracks a to-do list through implementation, and runs a review agent that reconciles spec ↔ code. Do: Read and approve the design spec yourself before implementation — and override a bundled behavior when it doesn't fit your work (Carson replaced its worktree skill with his own).

390 · Auto mode & safety Key idea: Only tool calls can do harm. Auto mode gates every bash command with two classifiers — could this do something bad? if yes, did the user explicitly approve it? — blocking and flagging the agent otherwise, and escalating to you after ~3 denials in a row (or 20 total). Do: Back everything up first (GitHub + cloud/S3 — use the model to do it in an afternoon), practice prompt hygiene ("investigate, don't touch production"), and run auto mode.

400 · Permissions & read-only agents Key idea: Agents have permissions — give a review agent read-only access and it's incapable of writing, so the dangerous action isn't gated, it's absent. Do: Scope each agent's tools to its job (read-only for reviewers/investigators) and fence unattended runs behind permission barriers; when you can't remove write access, use hooks.

410 · Hooks Key idea: A hook is your program run deterministically on a specific event — CLAUDE.md is advice the model usually follows; a hook is a rule the harness always enforces. The pair to know: PreToolUse (can block) / PostToolUse (can react). Do: For an absolute rule, hook the event and block it (a PreToolUse script that stops commits to main); for automation, hook commits to trigger a docs-update agent.

430 · GitHub for teams Key idea: Master = definitely-working code; work happens in feature branches; quality collapses if a diff is too big to reason about — you'll rubber-stamp it. Right-size every task to a small, reviewable unit. Do: master + feature branches; keep a milestone / parent-issue reviewable (~≤20 issues; 70 means it's too big); each issue → one PR → review → merge; put a diff explainer on the PR.

440 · Git worktrees & parallel work Key idea: One checkout = one branch = one thing at a time. A git worktree is a branch in its own folder, so an agent can work many branches in parallel — each with real files to run and test. Do: Use worktrees to parallelize (in VS Code + terminal); let the harness deploy tasks into multiple worktrees and clean them up — and override the default when it doesn't fit (Carson's MMGIS independent-database workflow).

450 · Research agents Key idea: LLMs are extraordinary at googling, but naive "go find this" gives mediocre results — how you search determines what you get. A good research agent constrains the pipeline. Do: Write an agent that takes a brain-dump → categories → ~4 searches each → judge source trust → read → download + link in a search log; add a Playwright login-to-journal agent and a PDF→markdown skill for gated sources.

460 · Math with LLMs Key idea: Don't raw-dog math — a naked model doing arithmetic is just next-word prediction over numbers. Turn math into code, which is durable and executable. Do: Start in math to explore, then convert to code (Python — SymPy / NumPy / Julia — or a proof assistant like Lean); codify key equations in your project docs so every relevant session is grounded.

It all lives in one place. The guide, the workshop, and these sheets: llms.codebycarson.com/working-with-llms

The Links Sheet — every resource the talk points at

Every slide with a linkable resource carries a small link at its bottom — like the little "how to make a skill" link under the Skills slide that takes you to Claude's docs. This sheet collects all of them in one place, grouped by beat in the order they come up, so you can go back and dig into anything after the talk. One line each; all primary sources.

Act I — How LLMs actually work

030 · Embeddings (word2vec & GloVe) - word2vec (Mikolov et al., 2013) — predict-the-neighbors embeddings: https://arxiv.org/abs/1301.3781 - GloVe (Pennington, Socher & Manning, 2014) — count-then-factorize embeddings: https://nlp.stanford.edu/pubs/glove.pdf

060 · Transformers → next-word prediction - "Attention Is All You Need" (Vaswani et al., 2017) — the transformer: https://arxiv.org/abs/1706.03762 - ELMo (Peters et al., 2018) — contextual embeddings, done first with an LSTM: https://arxiv.org/abs/1802.05365 - BERT (Devlin et al., 2018) — the transformer that made contextual embeddings dominant: https://arxiv.org/abs/1810.04805

080 · Post-training into a chatbot - InstructGPT (Ouyang et al., 2022) — SFT + reward model + RLHF, the ChatGPT recipe: https://arxiv.org/abs/2203.02155

Act II — Working with basic LLMs

120 · Tokens - OpenAI tokenizer — see how text splits into tokens (~4 chars each): https://platform.openai.com/tokenizer

140 · Context window & context rot - "Context Rot" (Chroma Research, 2025) — every model degrades as input grows: https://www.trychroma.com/research/context-rot - "Lost in the Middle" (Liu et al., 2023) — facts buried mid-window are recalled worst: https://arxiv.org/abs/2307.03172

150 · Style control & few-shot - "Language Models are Few-Shot Learners" (Brown et al., GPT-3, 2020) — few-shot / in-context learning: https://arxiv.org/abs/2005.14165

210 · The scaling era ends - Ilya Sutskever, NeurIPS 2024 — "peak data," data as "the fossil fuel of AI": https://deepnewz.com/ai-modeling/ilya-sutskever-declares-pre-training-we-know-end-neurips-2024-citing-peak-data-18418711 - Model collapse (Shumailov et al., Nature 2024) — training on AI-generated data degrades models: https://www.nature.com/articles/s41586-024-07566-y

220 · Chain of thought - "Chain-of-Thought Prompting" (Wei et al., 2022) — reasoning tokens before the answer: https://arxiv.org/abs/2201.11903

Act III — Modern agentic work

320 · CLAUDE.md - Claude Code — Memory / CLAUDE.md: the standing-instructions files and their up-the-tree hierarchy: https://code.claude.com/docs/en/memory.md

330 · Skills — ("how to make a skill") - Claude Code — Agent Skills: where skills live, SKILL.md frontmatter, lazy loading: https://code.claude.com/docs/en/skills.md

340 · Sub-agents — ("how to make an agent") - Claude Code — Subagents: defining custom agents and how delegation / fresh-context handoff works: https://code.claude.com/docs/en/sub-agents.md

380 · Superpowers - Superpowers plugin (obra) — the brainstorm → plan → execute → review harness: https://github.com/obra/superpowers

390 · Auto mode & safety - Anthropic — "How we built Claude Code auto mode" (the two-stage safety classifier): https://www.anthropic.com/engineering/claude-code-auto-mode

410 · Hooks — ("how to make a hook") - Claude Code — Hooks reference: every event, settings.json format, exit-code / JSON blocking: https://code.claude.com/docs/en/hooks.md - Claude Code — Hooks guide (worked examples, incl. blocking protected files): https://code.claude.com/docs/en/hooks-guide.md

430 · GitHub for teams - GitHub — About Projects (boards/tables/fields, works on personal accounts): https://docs.github.com/en/issues/planning-and-tracking-with-projects/learning-about-projects/about-projects - GitHub — Sub-issues (native parent/child breakdown with rollup): https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/adding-sub-issues - GitHub — About milestones (dated buckets of issues toward a goal): https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/about-milestones

440 · Git worktrees - Git — worktrees (checking out multiple branches into parallel folders): https://git-scm.com/docs/git-worktree

460 · Math with LLMs - Lean + mathlib community — the proof assistant behind the AI-formalization results: https://leanprover-community.github.io/ - DeepMind — AlphaProof (IMO 2024 silver, proved in Lean): https://deepmind.google/blog/ai-solves-imo-problems-at-silver-medal-level/ - Practical "math → code" at the workbench: SymPy https://sympy.org · NumPy https://numpy.org · Julia/SciML https://sciml.ai

Everything in one place: the guide, the workshop, and both sheets live at llms.codebycarson.com — full how-to at /working-with-llms.