MoE from the brick up

A token is a row of 4,096 numbers

Text is chopped into tokens: whole words or pieces of words. "Credentialing" might become three tokens. Each token gets an ID from the vocabulary, and that ID is looked up in an embedding table, which returns a list of decimal numbers. From then on, that list is the token.

The length of that list is the model's hidden dimension, written d_model. Every token has the same width, at every layer, whether it is "the", a comma or "cardiology". A long word does not get a wider vector. It gets split into more tokens.

An expert is a brick with a fixed stud pattern

Every expert is a small feed-forward network. Its input and output must be exactly d_model wide so it snaps onto the rest of the model. Inside, it expands the token to a wider working space, filters it, and compresses it back. That inner width is d_ff, the intermediate size.

token (4,096) ─▶ gate proj  ─┐
token (4,096) ─▶ up proj    ─┴▶ SwiGLU filter (d_ff) ─▶ down proj ─▶ out (4,096)

Three matrices, so parameters per expert per layer = 3 × d_model × d_ff.

Coarse experts versus fine-grained experts

Mixtral uses a few big bricks. Newer models use many small ones, which lets the router mix specialists more precisely.

Check the math yourself

That is the whole trick of MoE. You store 397B parameters, but each token only touches about 17B of them. Memory scales with the total. Compute per token scales with the active count.

The router is a turntable, and you can bolt it in place

The router is a small linear layer. It scores every expert for the current token, turns the scores into probabilities with softmax, keeps the top k, and blends their outputs using those probabilities as weights.

logits  = router(token)          # one score per expert
probs   = softmax(logits)
w, idx  = topk(probs, k)         # keep the best k
out     = Σ w[i] · expert[idx[i]](token)  (+ shared expert)

Adding a specialist expert, in three phases

# Phase 1: freeze everything except the new expert
for p in model.parameters(): p.requires_grad = False
for p in moe.experts[NEW].parameters(): p.requires_grad = True
out = model(batch, force_expert_idx=NEW)   # router bypassed

# Phase 2: lock the new expert, unlock the router
for p in moe.experts[NEW].parameters(): p.requires_grad = False
for p in moe.router.parameters(): p.requires_grad = True
out = model(batch)                          # normal top-k routing

The cold-start trap

The router needs a new output row for the new expert. If that row starts random, the router may never select it, and an expert that is never selected sends no gradient back to tell the router it is useful. Common fixes: initialize the new row with a small positive bias, or add a temporary auxiliary loss that nudges domain tokens toward it.

Load balancing

Left alone, routers collapse onto a few favorite experts. Training adds a balancing pressure (an auxiliary loss, or in DeepSeek's case a per-expert bias adjustment) so tokens spread across experts. This matters for your specialist too: push too hard on balance and medical tokens get scattered away from it.

Quantization: same studs, coarser plastic

Quantization keeps the count of numbers identical and stores each one in fewer bits. A 397B model is still 397B parameters at 4-bit. What changes is memory and precision.

How 4-bit actually works

Four bits can only express 16 distinct values. To make that usable, weights are split into small blocks (often 64 or 128 values). Each block stores its own scale, so a block of tiny numbers and a block of large numbers both get the full 16 levels of resolution.

That 0.5 bit of overhead is why a "4-bit" 671B model weighs about 378 GB rather than 335 GB.

What gets quantized and what doesn't

Expert weights are the bulk of the model and the main target. Routers, norms and often the shared expert and embeddings are commonly kept at higher precision, because small errors there affect every token. A bad router decision sends a token to the wrong specialist entirely.

Hardware: capacity decides fit, bandwidth decides speed

Generating a token means streaming every active weight through the chip once. So for one user, tokens per second is capped by memory bandwidth divided by bytes read per token. Raw compute matters more for reading long prompts and serving many users at once.

Three ways to split a model across machines

Pipeline parallel: machine A holds layers 1 to 30, machine B holds 31 to 60. Simple, but machines wait on each other. Tensor parallel: every layer is sliced across all machines, which work together on each token. Fast, but chatty, so the link must be low latency. Expert parallel: different experts live on different machines and tokens are shipped to wherever their experts live. Native to MoE and the standard choice on H100 nodes.

Why RDMA over Thunderbolt mattered for Macs

Tensor parallel sends small messages between machines for every layer of every token. Over regular networking each hop took roughly 300 microseconds. macOS 26.2 added RDMA over Thunderbolt 5, which brought that under 50 microseconds and made tensor parallel across Mac Studios practical. Early reports had 4 × M3 Ultra running Qwen3 235B around 32 tok/s and Kimi K2 Thinking (1T) around 30 tok/s with exo. The new M5 Ultra has 50% more bandwidth than the M3 Ultra.

Size your build

Pick a model and precision. The calculator estimates weight memory, how many machines you need, and the bandwidth ceiling.

Assumptions: 90% of a Mac's unified memory usable for the model (macOS limits GPU memory by default; it can be raised), 90% of each H100 usable, plus 10% headroom for KV cache and runtime. H100 counts rounded up to whole 8-GPU nodes. Estimates, not guarantees.

Two reference builds for a 300B+ model

Recommended model: Qwen3.5-397B-A17B. It clears your 300B bar, uses the same 4,096 hidden width from this course, ships under Apache 2.0 for commercial use, and has an official FP8 release. Its hybrid attention keeps only 15 full-attention layers with 2 KV heads, so long-context memory stays small (roughly 30 KB per token at BF16, about 4 GB for 128K tokens).

Local: Mac Studio cluster

# Single machine, MLX
pip install mlx-lm
mlx_lm.server --model mlx-community/<Qwen3.5-397B-A17B 4-bit repo>

# Cluster: enable RDMA on each Mac from recovery (macOS 26.2+),
# cable every Mac to every other Mac over Thunderbolt 5,
# then run exo on each machine; it discovers peers and shards the model.

Pricing: the M5 Ultra Mac Studio starts at $5,499 with 96 GB. The 512 GB configurations ship in late October 2026; check Apple's configurator for current pricing.

Cloud: H100 nodes

# One 8×H100 node, SGLang, tensor + expert parallel
python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3.5-397B-A17B-FP8 \
  --tp 8 --ep 8 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder

Cost: H100s rent for roughly $2 to $12 per GPU-hour in 2026, with specialist clouds near the low end. One 8-GPU node running all month at $2.50 per GPU-hour is about $14,600.

Choosing between them

Mac Studios win on private, always-on inference for a small team, with no hourly bill, and keep sensitive data (such as PHI) on hardware you control. H100s win on throughput for many users, prompt-heavy workloads, and any fine-tuning. A practical split: serve locally on Macs, rent H100s by the hour when you train an adapter or specialist expert. Training needs several times the inference memory because of gradients and optimizer state.

Why training needs far more memory than inference

Inference holds the model and the KV cache. Training holds the model plus everything needed to change it: a precise master copy, a gradient for every weight, the optimizer's memory, and the saved activations from the forward pass.

Activations are the hidden cost. During the forward pass each layer's input is saved so the backward pass can use it. That memory grows with batch size × sequence length × layers. Inference discards each layer's output as soon as the next layer reads it.

Gradients, the backward pass and the optimizer

Training repeats one loop: forward pass, measure the loss, backward pass, update. Step through it below on a model with a single weight.

Gradient

For every weight, the gradient answers: if I nudge this weight up slightly, does the loss go up or down, and by how much? A gradient of −8 means increasing the weight lowers the loss, strongly. A real model has one gradient per weight, so gradients cost as much memory as the weights.

Backward pass

Most weights sit many layers away from the loss. The backward pass starts at the loss and walks back through the layers in reverse, using the chain rule. Each layer works out its share of the blame and passes the rest back to the layer before it. To do that, each layer needs the exact input it saw on the forward pass. That is why activations must be stored.

Optimizer state

Plain SGD moves each weight by learning rate × gradient and remembers nothing. Adam keeps two running averages per weight, for the whole run:

m, momentum: the recent average direction. It smooths noisy batches so weights don't zigzag. v, variance: the recent average of the squared gradient. Weights with big gradients take smaller steps, weights with tiny gradients take bigger ones.

Toggle the two optimizers above. SGD's step size follows the gradient. Adam takes roughly even steps. That per-weight memory is two FP32 numbers, 8 bytes, the largest item in the 16-byte budget.

LoRA and QLoRA: snap on thin bricks instead of rebuilding

LoRA freezes the original matrix W and trains two skinny matrices, B and A, whose product is the change. The adapter's output is added to the frozen output.

Rank r (16 here) sets the adapter's capacity. Common values are 8 to 64. B starts at zero, so at step 0 the adapter adds nothing and the model behaves exactly like the original. Only A and B get gradients and Adam state, so the 16-byte cost applies to about 1 to 2% of weights.

It works because the changes fine-tuning makes are mostly low-rank. Teaching a model denial patterns or your output format doesn't require rewriting every weight independently.

After training

Either merge B·A into W (no speed cost) or keep the adapter as a separate file of tens to a few hundred MB. Separate adapters let one base model serve many jobs: a denials adapter, a credentialing adapter, a payer-contract adapter, swapped per request.

QLoRA

Load the frozen base at 4-bit (Module 4) and keep A and B at full precision. It is the cheapest way to fine-tune a large model, with a small quality cost from the 4-bit base.

Rough figures for short sequences with gradient checkpointing. Long documents raise activation memory.

Other memory savers

Gradient checkpointing: store fewer activations and recompute them on the backward pass, about 30% more compute for much less memory. ZeRO or FSDP: split weights, gradients and optimizer state across GPUs instead of copying them to each one.

Worked example: a small fine-tuned denial triage model

A narrow, high-volume job is where a small fine-tuned model earns its keep. Here is one built for RCM.

One training example

{"messages": [
 {"role": "system", "content": "Classify the denial. Reply in JSON only."},
 {"role": "user", "content": "Payer: commercial PPO. CARC: CO-197. CPT: 72148. Billed: 1450.00. Paid: 0.00. Note: lumbar MRI, no auth on file."},
 {"role": "assistant", "content": "{\"root_cause\":\"missing_prior_auth\",\"category\":\"front_end\",\"appealable\":\"conditional\",\"next_action\":\"check payer retro-auth window, else adjust per contract\",\"owner\":\"auth_team\"}"}
]}

Train it

# On a Mac with MLX (data/ holds train.jsonl and valid.jsonl)
mlx_lm.lora --model <Qwen3.5-9B MLX repo> --train --data ./data \
  --fine-tune-type lora --batch-size 4 --iters 1500 \
  --adapter-path ./adapters/denials

# On a rented H100 with Hugging Face PEFT
LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
           target_modules="all-linear", task_type="CAUSAL_LM")
# load the base with BitsAndBytesConfig(load_in_4bit=True) for QLoRA

Benefits

Cheap: training costs tens of dollars, not thousands. Fast and private: it runs locally in milliseconds, so PHI never leaves your hardware. Consistent: it returns the same JSON shape every time, which makes downstream automation reliable. Your judgment, encoded: it learns how your team actually resolves denials, not generic advice. Stackable: one base, many adapters.

Drawbacks

Narrow: it is good at the one job and weak outside it. Weaker reasoning: a 9B model will miss multi-step logic a 397B model catches, so unusual cases need escalation. Goes stale: payer rules change and baked-in facts don't update, so retrain on a schedule. Copies your mistakes: inconsistent historical decisions become inconsistent model output. Data is the real cost: cleaning, de-identifying and labeling takes far longer than training.

An RCM agent team on a sub-$100K budget

You do not need to pretrain anything. Pretraining is the million-dollar part. You need one strong general model for reasoning, a few small fine-tuned models for high-volume narrow jobs, good tools, and humans approving anything that touches money.

The team

Large model: Qwen3.5-397B on one 512 GB Mac Studio at 4-bit, or a frontier API for the hardest cases. Small models: Qwen3.5-9B adapters on the same machine. A human approves submissions, write-offs and appeals.

Year-one budget (estimates)

Sequence

Weeks 1 to 4: pick three tasks, pull and de-identify historical data, build a 300-example graded test set. Months 2 to 3: run agents on the large model with RAG and no fine-tuning, and measure against the test set. Months 3 to 4: fine-tune small models only for tasks that are high-volume and where the large model is slow, costly or inconsistent. Ongoing: log every human correction, add it to the training data, retrain adapters quarterly.

Persistent memory: how agents remember between sessions

A model's weights are frozen once it is deployed, and its context window empties when a session ends. Everything an agent "remembers" next week has to be written somewhere and read back in. That store is persistent memory, and in 2026 it is a core design decision for any agent team.

The four kinds

Working memory is the context window: the prompt, documents and conversation the model sees right now. Episodic memory is a log of what happened: this claim, this payer, this appeal, this outcome. Semantic memory is distilled facts believed to be true now: "Payer X wants modifier 25 documentation attached." Procedural memory is know-how: playbooks, prompts and skills that tell an agent how to do a job. LoRA weights are the slowest layer, behavior baked into the model itself.

How it works in practice

Memories are stored in a database (often Postgres with a vector index, which Supabase provides), retrieved by similarity or by structured lookup, and inserted into the context window before the model answers. A background process consolidates: it turns many episodes into one semantic fact ("the last 12 appeals to Payer X won when we attached the op note"), and it prunes or corrects facts that went stale.

Memory for an RCM team

Episodic: every work item with its payer, codes, action taken and outcome. Semantic: payer rules by payer and state, client preferences, contract terms, each with a source and a date. Procedural: one playbook per task, for example "denial CO-197 at Payer X", that agents follow and propose edits to. Weights: the quarterly LoRA adapters from Module 11.

Reinforcement learning: training on outcomes, not examples

Supervised fine-tuning (Module 11) shows the model correct answers and asks it to imitate them. Reinforcement learning lets the model try, scores the attempt, and makes higher-scoring behavior more likely. It is how models learned to reason step by step, and it can now run on a single GPU for small models.

GRPO, the default RL method in 2026

For each prompt the model generates several answers. A reward function scores each one. Each answer's advantage is its score minus the group average. Training then raises the probability of above-average answers and lowers the rest. A penalty (the KL term) keeps the model close to where it started so it does not drift or forget its general skills. With LoRA, the frozen base model doubles as that reference, which keeps memory low. Libraries such as Unsloth and Hugging Face TRL run GRPO on a single 24 GB GPU for small models.

Verifiable rewards for RCM

RL only works if you can score answers automatically. RCM has more of these than most fields: is the output valid JSON; does the predicted denial category match how the claim was finally resolved; does the CARC-to-action mapping match your rules table; did the appeal the model recommended actually get overturned. That last one is a delayed reward, so you collect outcomes and train in batches.

Recursive self-improvement: where it stands and where it is going

Recursive self-improvement (RSI) means AI systems doing the work of building better AI systems, which then do that work even better. The full version, an AI that chooses its own research goals and builds its successor without humans, has not been demonstrated. Large parts of the loop are now automated.

Current state, September 2026

Anthropic reports that more than 80% of code merged into its codebase was written by Claude as of May 2026, up from low single digits before February 2025, and that its engineers merge about 8 times as much code per day as in 2024. On a fixed speed-optimization task, Claude went from about a 3× speedup in May 2025 to about 52× in April 2026; a skilled human reaches about 4× in four to eight hours. In an open-ended AI safety research problem, Claude agents closed 97% of a performance gap that two human researchers closed 23% of in a week, using about $18,000 of compute, though humans chose the problem and the scoring.

OpenAI said in February 2026 that early versions of GPT-5.3-Codex helped debug its own training and evaluation. Narrower self-improving systems, such as DeepMind's AlphaEvolve and the Darwin Gödel Machine, improve code or prompts against a fixed score they do not control. Both Anthropic and OpenAI say fully autonomous RSI is not happening yet. The remaining human role is choosing which problems matter and judging which results to trust.

Expected trajectory

METR measures the length of tasks AI can complete on its own; that length has been doubling about every four months. Anthropic projects that tasks taking a skilled person days could come into range in 2026, and tasks taking weeks in 2027. It describes three possible futures:

Even in the faster scenarios, bottlenecks move rather than disappear. Anthropic already finds human code review is its new constraint. For a business, the equivalent is approval, verification and payer response times, not drafting speed.

What this means for your RCM operation

You can run a bounded, safe version of the same loop today. Agents propose changes to their own playbooks and prompts. Each change is scored against a fixed test set you control (the 300 graded cases from Module 12). Only changes that score higher, and that a human approves, are kept. Corrections feed the next LoRA or GRPO round. That is recursive improvement of your agent team, with the scoring and the final say kept in human hands.

Quiz

Tap an answer to see the explanation. Your progress is saved in this browser.