Books
Books about software architecture, engineering, and the decisions behind systems that have to work in the real world.
Beyond the Model
A Systems Theory of Modern AI Quality
Beyond the Model is not a guide to choosing a model or a framework. It is a diagnostic method for finding the layer of the system that is actually limiting production quality, including the cases where the answer really is the model.
Your Model Got Better. Your Product Did Not.
The AI feature is not good enough, and someone proposes the obvious fix. Switch to a better model. The benchmark numbers go up. The product does not.
The model really was better, and that is what makes the failure strange. Modern models improve along a jagged frontier, sharply in some regions and hardly at all in others, and nothing in a benchmark score tells you in advance which regions those will be.
In production, that unevenness is absorbed by the machinery around the model. Routing, validation, a fallback for low confidence, an escalation path for decisions a system should not make alone. A model swap leaves every one of those pieces where it was.
Before You Ask Which Model, Ask Which Layer
The reflex is to go shopping for a replacement. The book replaces the reflex with an order of operations. Set two hypotheses against each other, then run the smallest safe experiment that tells them apart.
HYPOTHESIS ONE
The Model Is the Limit
Capability itself sets the ceiling. Sometimes this is true, and a head-to-head model test is the sharpest experiment available.
HYPOTHESIS TWO
A Surrounding Layer Is the Cause
Context assembly, retrieval, a tool call, validation, or an evaluation case nobody wrote. A model swap changes everything except the cause.
Capability is necessary. Production quality simply continues past it.
THE QUALITY LOCUS
Six Surfaces Around the Model
Production quality is not contained in the deployed model. It is assembled across the system that prepares, surrounds, measures, and controls it. The model sits at the center as the baseline, and six surfaces around it are where quality is won or lost.
Formation
Data and training shaped the model before it shipped. They are the one place a running system cannot reach into without a different model.
Deployed Behavior
Inference, memory, and runtime decide how the frozen model behaves. Test-time compute, what the system retrieves, and where output meets tools, validation, and context assembly.
Control and Learning
Evaluation decides what the system can detect, compare, reject, ship, and improve. It is permanently staffed work, not a gate cleared once at launch.
Seven Steps, Two Opposite Answers
With the map in hand, “the AI is bad” turns into a short triage, and the triage becomes a loop. The book runs it end to end on two synthetic systems, false starts left in.
THE LOOP
Seven steps, no prior about the answer.
1. Scope and reproduce
Pin the failure down before changing anything.
2. State the contract
Name what broke: correctness, trust, or operability.
3. List the hypotheses
The model, the input data, retrieval, context assembly, validation, eval.
4. Measure
Choose the measurement that separates them.
5. Intervene
The smallest safe change that rules out the most hypotheses.
6. Mitigated or resolved
A quieter symptom is not a removed bottleneck.
7. Institutionalize
Return what was learned to the contract.
TWO CASES
Same steps, opposite conclusions.
Case A. A document pipeline starts extracting the wrong date. The room blames the model, and a prompt rewrite cuts the failure rate by more than half without touching the cause. Logging the context the model actually received shows that the answer never reached it.
Case B. The team works through every surface it can reach, and the failure survives all of them. A counterfactual that changes the model and nothing else confirms it. That survival is the diagnosis.
Five Propositions That Outlast the Tools
Tools and models turn over fast. The places where production quality is won or lost change far more slowly. The book closes with a doctrine that names a way of looking rather than a bottleneck.
Composition
Production quality is jointly produced by the model and the system around it, and neither alone determines the outcome.
Displacement
The symptom you observe and the layer that causes it frequently sit in different places. A prompt symptom can reflect a retrieval cause.
Constraint
A layer that falls below what the contract requires sets a ceiling on the whole system, and improving the model does not raise it.
Reallocation
Improving one layer usually reveals the next bottleneck. The need for system design relocates rather than retires.
Boundary
Some failures are genuine model bottlenecks. Telling them apart from system bottlenecks is the diagnostic task itself.
An Owner for Every Surface
The doctrine ends in an operating model. Each surface gets an owner, so a diagnosis has somewhere to land.
When the Model Really Is the Limit
Sometimes capability really is the ceiling. The book treats that verdict as a result the method has to earn, not as a failure of the method. It is earned by a counterfactual that changes the model and nothing else.
A coordinate system that only ever pointed away from the model would not be a diagnosis, only a bias with good manners.
Chapter 12 sets out the conditions as a diagnostic procedure, and an appendix states them in quantitative form. That is where the argument is most exposed, and where a critical reader should push.
What Is Inside
Five parts, thirty diagrams, 109 cited sources from classical learning theory to 2026 papers, a glossary, and a full index. Every chapter ends with what changed, what it means in practice, common failure modes, and questions for technical leaders.
Part I. The Model Is Not Enough
Why the model-centered map of quality, useful as it once was, breaks on modern systems.
Part II. Learning and Inference
Scaling laws and their limits, late generalization, pretraining, post-training, and inference-time compute.
Part III. Assembled Through Systems
Long context, retrieval, the architecture of memory, and agents as closed-loop systems.
Part IV. The Hard Truths
Evaluation, knowing when you do not know, and the constraints under which the smartest model still loses.
Part V. The Enduring Doctrine
The five propositions, and an owner for every surface.
Five Appendices
Double descent in the linear case, calibration and selective prediction, retrieval design decisions, a full diagnostic walkthrough, and a glossary.
Who It Is For
Anyone who has built and operated a production system and now answers for an AI feature running for real users. Senior, staff, and principal engineers, architects and consultants, platform and SRE teams, and the engineering leaders who get paged when it degrades.
It assumes production engineering experience, not a machine learning background. It is about diagnosis rather than implementation, so it does not teach a framework or walk through application code.
Short on time? Read the Declaration, then Chapter 14, then the diagnostic walkthrough in Appendix D. Together they come to under forty pages, and every conclusion the book reaches is inside them.
The Question Behind the Book
The model has not lost its importance. It has lost its monopoly on the answer. If you read this book and your honest answer to one question changes, it will have done its work.
When quality breaks, do you suspect the model first, or the system?
Beyond the Model
A Systems Theory of Modern AI Quality