How We Decide to Ship a Model

Why We Don't Do Day-One Integrations

The sample reels at a model vendor's launch are the best takes picked from thousands of generations — nothing to fault there; every industry's sizzle reels work the same way. But creators don't need a model's best day. They need its average day and its worst day: on a thirty-shot project, the three worst shots decide the quality of the final cut.

So our rule is simple: untested models don't enter production workflows. On a film set, even a rented lens gets a test roll to check edge sharpness — and a generative model decides what your entire film looks like. Why would it go into a real project the day it launches? "Day-one integration" looks great in marketing, but when things fall apart, the creator pays. That's math we do on our users' behalf.

Three Tests: Pressure From Real Work

Co-testing isn't benchmarking. Benchmarks measure how a model performs on standardized problems; we measure how it will behave inside your project. Three dimensions, all run on real creative tasks:

• Consistency: Take one character's character sheets (front, profile, 45 degrees), generate ten consecutive shots across different framings and angles, and count how many still show the same face. This is the life-or-death line for serial creation — a model that dazzles on one image but hands you ten different faces across ten images belongs in inspiration exploration, not production.
• Instruction following: Write one prompt with explicit spatial relationships and multi-subject attribute binding ("a boy in a black uniform at frame left; a girl in a white dress in the foreground, back to camera") and score how much of it actually lands. Then test action capacity: pack two consecutive actions into ten seconds and see whether one gets dropped.
• Controllability: Change exactly one variable in the prompt (the camera move, from "push in" to "orbit"), touch nothing else, and check whether that one thing — and only that one thing — changed. With a poorly controllable model, every revision is a fresh gacha pull: ten edits equal ten brand-new generations.

The co-testers are experienced creators, not us. A tool team has blind spots when testing models — we know far too well how to write with the grain of a model. Creators write the way they always write, and that's the real input the model will face once it's live.

After the Pass: Write the Dialect File First, Then Ship

A model that passes co-testing still doesn't show up in the interface. The next step is distilling everything learned during testing into that model's dialect file — internally we call them prompt skill cards, five sections per card:

• Register: What sentence shapes it eats. Seedance eats long narrative sentences, written like you're reading a storyboard aloud to your cinematographer; Nano Banana's official docs explicitly reject keyword stuffing; Midjourney-style models eat tag streams. One brief, three completely different ways to write it.
• Word-count sweet spot: Seedance, for example, officially recommends staying under 500 Chinese characters — go past that and it grabs the highlights and drops the details. Information like this hides in the corners of vendor docs; most people never see it.
• Negation handling: Does it support negative parameters — and if not, how "no X" needs to be rewritten as positive phrasing.
• Hard constraints: API-level rules — reference image caps, mutually exclusive modes, reference syntax. Kling 3O, for instance, requires you to explicitly state what each referenced asset is for; handing it an image without stating the purpose is the number-one failure mode its own documentation warns about.
• Examples: A user's raw input → the ideal rewrite in this model's dialect, covering the common failure modes.

These files earn their keep twice. Right now, they're the knowledge layer under our prompt-optimization feature. Long term, they're a more durable asset than any model — models turn over every six months, but "how to test a model, how to record its dialect" doesn't expire.

What "Shipping" Means: Actions, Not a List

Integration comes last — and it looks different here than on most platforms. A model doesn't get hung by name in a dropdown; it gets orchestrated into creative actions — generate a character, generate a frame, plan a storyboard, block out a space. You choose "give this character a profile character sheet," not "use Model X v2.3."

One immediate payoff: when models turn over, your project doesn't move. The action is still the action; the model underneath changes, the dialect file changes, and your assets, your process, and your habits stay exactly where they were. We've watched too many creators build their workflow around one specific model version — the model gets sunset, and the whole pipeline is scrap. Treating a model as infrastructure is dangerous. It's a consumable.

And yes, some models finish testing and never ship: a dazzling single capability, but consistency below the line, or API constraints that fight the workflow. We keep tracking their releases — the day they clear the bar is the day they ship. The length of the list was never our KPI. Whether creators dare to use it on a real project — that is.

FAQ