Notes from the lab · What we ran
We shipped 400 assets in a month. Three things broke.
One brand account. 31 days. 412 shipped assets: statics, carousels and short films, posted on a daily cadence. This is the teardown of what failed while that happened, because the failures are more useful than the total.
None of the three breaks below were the model. At volume, the model is rarely the thing that breaks. The plumbing around it is.
Break 1: the 10MB wall nobody reads about
Our asset host rejects files over 10MB. Our 4K renders regularly weigh more than that. For 3 days the pipeline showed green while a chunk of "shipped" carousels silently collapsed on upload, because the size failure happened after the quality gate, in a step nobody was watching.
The fix was boring and permanent. Render at 2K by default, since no feed placement on Instagram or LinkedIn benefits from 4K. Upscale only the masters that go to broadcast. And move the file-size check before the quality gate, so an oversized asset fails loudly and early.
Break 2: QC became the bottleneck, not generation
Generation kept up with the calendar all month. The review queue did not. Our automated quality pass hung on roughly 3% of assets, mostly long video renders, and every hang pulled a human into re-checking work the machine had already half-judged. By week 2, throughput was set by the review queue, not by the models.
The fix was tiering. High-confidence assets auto-pass. Only flagged assets reach a human, with the flag reason attached so the reviewer starts from context instead of from zero. Human review load fell by about 70%, and the queue stopped being the constraint.
Break 3: the look drifted, and a client caught it before we did
Around asset 250, the brand look had moved. Not dramatically. Warmer grade, slightly different framing habits. The cause was two small things compounding: prompt edits accumulating across three weeks, and a model version update mid-month. A client comment flagged it. That is the wrong way to find out.
The fix: the brand file is now locked, and changes to it are versioned like code. Alongside it sits a regression set of 12 reference renders that re-generate daily and get diffed against approved baselines. When the look moves, we now know within a day, with a picture of exactly what moved.
What we changed going into the next month
- Size and format checks run first, before quality judgment.
- QC is tiered. Machines judge everything, humans judge exceptions.
- The brand look is version-controlled and regression-tested daily.
All three fixes are now defaults in every account we run. That is the actual product of a month like this. The 412 assets were the output; the hardened system is the asset.
Want volume without the breakage?
We run the system inside your account. These lessons come pre-installed.