REGRESSION TESTING FOR PROMPTS
Prompts are code — but nothing stops a bad edit from shipping. Prismo is the regression gate for prompts: it runs the old and new version on your real inputs, blind-judges the outputs, and blocks the pull request when the change loses. One GitHub Action. No eval suite yet? Prismo writes it for you. Free with your own key (Anthropic, OpenAI, or Google Gemini).
Anthropic · OpenAI · Gemini — your key or ours.
01 / THE GATE
A one-step GitHub Action: on every pull request that touches a prompt, Prismo runs the old and new version on your committed test inputs, judges the outputs blind (both orders, to cancel position bias), and fails the check when the new prompt loses. Ties hold; regressions don't ship. The per-input verdict table lands in the job summary.
Runs on your own provider key — free, or a Prismo license key with nothing else to wire. Needs no GitHub token, reads nothing but the two prompt versions and your committed inputs, and adds ~30–60s and a few cents per gated PR. No test inputs yet? prismo.ci --init writes the suite for you, or the web app generates it ready to commit.
02 / REFRACTION
Paste a draft, pick a goal. Prismo applies the craft — specificity, output format, role framing, constraints — without changing what you asked for.
03 / THREE STRATEGIES
One-shot rewrite through a versioned meta-prompt. Fast enough to feel like autocorrect for prompts.
1 model call · ~2s
A critic names every weakness; a rewriter fixes them. Repeats up to three rounds until quality holds.
critique → rewrite → loop
Three candidates generated at different temperatures, then ranked pairwise by an impartial judge.
3 candidates · swap-debiased judge
04 / RECEIPTS
Don't take its word for it — anyone can claim a prompt got "better." Prismo runs the original and your edit on your real example inputs, judges the outputs blind — both orders, to cancel position bias — and hands you a win rate. Save those inputs as a test-set and re-run it every time you change the prompt or switch models, across Anthropic, OpenAI, and Gemini. If a change quietly regresses, you'll see it before your users do. No test inputs? Prismo generates the suite for you — typical, edge, and adversarial cases plus a grading rubric — and exports it as promptfoo YAML or pytest, so your prompts get tests even if you never wrote an eval in your life. In CI, the prompt gate blocks any pull request whose prompt change loses to the version on main — one GitHub Action, judged on your committed test inputs. And after you ship, drift monitors re-run your prompt weekly against the live model and email you the day its outputs start losing to your baseline — because models change under you.
Every optimization comes with evidence: the original vs. your edit, run on your real inputs and judged blind, both orders — so a win rate is something you can actually trust.
Same receipts whether you bring your own key or run on Prismo's hosted inference — see pricing below.
Shareable receipts — send a "won 5/5 on my inputs" link. No login to view.
05 / SAVE MONEY
One click runs your prompt across Fast, Balanced and Quality on your own inputs, blind-judges every output against the top tier, and recommends the cheapest model that holds quality — with a shareable savings receipt. Most prompts don't need the expensive model; now you can prove which ones.
06 / OBJECTIONS, ANSWERED
You shouldn't trust it blindly, which is why Prismo does the honest version: outputs are judged on your real inputs (not abstract "prompt quality"), in both orders to cancel position bias, with the rationale shown per case — and every side-by-side output is right there to check yourself. The judge doesn't replace your eyes; it makes checking 8 cases cost one click instead of an afternoon.
promptfoo is a great framework — if you're ready to write YAML configs and maintain a test harness (and are comfortable with it being OpenAI-owned now). Prismo is the zero-config on-ramp: it generates the test suite from your prompt, runs the gate with one Action, and stays provider-neutral. And if you outgrow it, Prismo exports your suite as promptfoo YAML — you lose nothing by starting here.
With your own API key, inference goes provider-direct and Prismo's server stores no prompt content — history and test-sets live in your browser. Server-side storage exists only where you explicitly opt in: shared links and drift monitors. Telemetry is content-free counts.
On your own key: pennies. A typical gate run (5 inputs, two prompts, blind judging both ways on the Fast tier) costs a few cents of inference — less than the CI minutes around it. You cap it by committing fewer inputs (8 max per run).
The judge never sees which prompt produced which output, sees them in both orders, and is instructed to ignore verbosity. Ties are counted as "held", not wins — the gate only fails on actual losses. If you still disagree with a verdict, the case view shows you exactly why.
The gate sends exactly two things to the API: the two prompt versions and your committed test inputs — it reads nothing else from the repo and needs no GitHub token (the optional token is only for posting the verdict as a PR comment). Your provider key travels over TLS for that request and is never stored. Want zero third parties? Point the CLI at a self-hosted Prismo.
No. BYOK works without one. An account (Google, one click) adds 100 free hosted runs a month, a dashboard, and a drift monitor. Paying adds hosted inference and 10 daily monitors — that's it, no feature hostage-taking.
07 / PRICING
bring your own key · no account
Launch price — lock it in while you're subscribed.
hosted, no key · Fast + Balanced models
Hosted plans include a monthly fair-use allowance — the app shows a meter with how much you have left. Bring-your-own-key is always unlimited, and heavy Quality users can fall back to BYOK anytime, free.
08 / ALSO NATIVE · macOS
Prefer a desktop app? The signed, notarized DMG puts Prismo one ⌘⇧P away, anywhere. Same engine — use the free hosted trial, or paste your own Anthropic, OpenAI, or Gemini key in Settings.
09 / PRIVACY
Prompts are sent to the Prismo service only to generate a result. Your history and settings stay on your device — in your browser or on your Mac.
never stored · never logged · never sold · never used for training