Session 13.2: Model Updates

Course → Module 13: Protecting Your Work & Staying Ahead

Session 2 of 6

Model Releases Are Not Automatic Upgrades

A new model version drops. The announcements are enthusiastic. Benchmarks show improvement. You are tempted to switch your production pipeline immediately. Do not.

When a new model releases, three things happen simultaneously. Some outputs improve. Some outputs change in unexpected ways. Some prompts that worked reliably on the old model produce different (sometimes worse) results on the new one. This is not a failure of the new model. It is the reality of any system built on a platform that changes underneath you.

The response is not to avoid updates. It is to treat them like a factory treats equipment upgrades: test before deploying.

Model Regression Testing: Running a set of standardized benchmark prompts on a new model and comparing outputs to your reference outputs from the current model. The goal is not to prove the new model is better. The goal is to identify what changed and whether those changes break your pipeline.

The Three Types of Change

flowchart TD A[New Model Released] --> B["Run Benchmark Prompts
(same prompts, same parameters)"] B --> C["Compare to Reference Outputs"] C --> D{"Type of Change?"} D -->|"Improvement
(better quality,
fewer markers)"| E["Accept
Update reference output"] D -->|"Neutral Drift
(different but
equivalent quality)"| F["Evaluate
May need prompt adjustment"] D -->|"Regression
(worse quality,
new problems)"| G["Reject
Stay on current model
or adjust prompts"] style E fill:#6b8f71,color:#111 style F fill:#c8a882,color:#111 style G fill:#c47a5a,color:#111

Change Type	What It Looks Like	Example	Action
Improvement	Better structure, fewer AI markers, more natural voice	New model produces openings that skip "comprehensive guide" preamble	Update to new model. Update reference outputs.
Neutral drift	Different phrasing, different structure, same quality level	New model organizes content with H3 subheadings instead of bold inline headers	Test whether downstream pipeline handles the format change. Adjust if needed.
Regression	More verbose, more hedging, new AI markers, worse adherence to constraints	New model ignores "do not use bullet lists" instruction that old model followed	Stay on old model. Report the regression. Revisit on next update.

Building Your Benchmark Suite

Your benchmark suite is a set of 5-10 prompts that represent your most critical content types. These prompts stay fixed. You run them on every new model and compare the outputs.

Good benchmark prompts have these properties:

Constrained. They include specific formatting requirements, voice constraints, and structural expectations. Unconstrained prompts do not test compliance.
Representative. They cover the range of content your pipeline produces. If you produce product descriptions and blog posts and email copy, your benchmark includes all three.
Scorable. You can evaluate the output using your quality rubric from Session 11.5. Subjective "this feels better" is not a benchmark. A rubric score that moves from 34 to 38 is.
Version-controlled. The prompts are stored in files, not typed from memory. The reference outputs are stored alongside them. Every model version gets its own output file.

The Update Protocol

When a new model is released, follow this sequence:

Do not change your production pipeline. Keep the current model running.
Run benchmark suite on new model. Same prompts, same parameters.
Score all outputs using your rubric. Compare scores to reference.
Categorize each benchmark result as improvement, neutral drift, or regression.
If all benchmarks show improvement or neutral drift: switch production to new model. Update reference outputs.
If any benchmark shows regression: investigate. Can a prompt adjustment fix the regression? If yes, adjust and re-test. If no, stay on current model.
Document the decision in your production log. Include benchmark scores, change categories, and the rationale for switching or staying.

Prompt Versioning

Some model updates require prompt adjustments. A prompt that works on Claude 3.5 may need modification for Claude 4. This is normal. The solution is prompt versioning: maintaining model-specific variants of your prompts that are loaded based on which model is active.

Your prompt library (from Session 5.9) should be organized so that model-specific adjustments are isolated. The core intent of the prompt stays the same. Only the phrasing or formatting instructions change. When you switch models, you switch the prompt variant, not the prompt itself.

Over time, you accumulate a history of which prompts work on which models. This history is valuable. It tells you how models differ in their response to specific instructions and helps you adapt faster when the next update arrives.

Assignment

Create your model benchmark suite: 5 test prompts that represent your most critical content types. Run them on your current model and save the outputs as reference (include rubric scores for each). Store the prompts and outputs in version-controlled files. The next time a new model is released, run the same 5 prompts, score the outputs, and categorize the changes. Document the comparison and your switch/stay decision.

Model Updates