Pick a prompt
Read it, improve it (by hand or with Phinity), evaluate it against its golden, check the judges and publish it — all from here, all in MLflow.
New here? Press any ? for a short explanation of that part of the page.
Guide what you do → what you get · click a step to go there
The 5 rules the Report checks each one
- 5 test cases or more, each with one expected answer.
- Every AI grader passes 80% or more, on all the cases.
- You check every answer with Yes / No, and agree with the AI on 80% or more.
- Same question, same answer: 90% or more of the repeated answers agree.
- All green → publish. Anything amber → 6 · Improve.
- Each case below is one real moment the live CIO decided, frozen exactly as it received it: the agents' signals, the market context and the trigger.
- Want more? Pick a date range and how many, then press ＋ Add real CIO moments — or ＋ Add a case to write one yourself.
- Open a case (▸) to ✎ Edit or 🗑 Delete it.
- Market did = what BTC actually did after that trigger (1 up · −1 down · 0 neither). Just for reference — the AI graders never see it.
- Check the cases below — they are the questions every version gets graded on.
- Empty? Press ⇪ Seed cases to load the starting set.
- Where the agent must give a specific answer, set the expected label on the case.
- Generate with Phinity: Phinity reads the prompt and drafts test cases — or ＋ Add a case to write one yourself.
- Review the draft: edit any field, untick what you don't want, then Load into golden. Nothing is saved before that.
- Aim for 5–10 cases, including at least one whose answer is not in its context (the agent should say so).
Golden dataset
real moments between andAI review — a model answers each case, AI graders judge each answer
see exactly what one case sends
Result — score per version
% of cases each grader passed · PROMOTABLE = at least as good as the live versionResult — case by case
read left to right: what went in → prompt → the agent's answer → expected → each AI grader (name, version) and its YES / NOLive conversations
The agent's live traces — ⚖ to judge one.Consistency — ask the same thing several times
Result
one row per case · View shows the input and every answer side by sideHuman review — is each answer the expected one? You answer first, blind
Result — you vs the AI grader
Review
answer Yes / No first — the AI grader's verdict is revealed only after, so it cannot sway youReport what already happened to one prompt version
Improve — three better prompts, each fixing what went wrong
The three proposals
each says why and which problems it attacks · save one as a candidate, then evaluate it (2 · AI review)No proposals yet — press “Propose 3 improved prompts”.
- For every golden case, the agent answers; then this judge reads the answer with the agent's evidence and says yes / no with a reason.
- An agent's score for this judge = the % of cases it said yes.
- Its instructions are the Prompt tab — versioned like any agent prompt. Evaluations grade with its @production version.
What it checks
What its prompt receives
Fields you can use in its instructions, and what each one carries.Who it grades
Which agents use this judge. Change it from an agent's 2 · LLM judges tab.- Your grades in any agent's 3 · Human evaluation are the ground truth.
- Pick a version of this judge and press ⚖ Calibrate: it re-grades every labelled answer and measures how often it agrees with you (κ).
- A version can be published only with κ ≥ 0.6 — a judge decides promotions, so an unmeasured one never goes live.
Calibration per version
What is this page?
Every prompt your agents run on — the chat (Phinity), the strategy lenses and the CIO decision — lives in the MLflow Prompt Registry. Here you read them, improve them, test them and choose which version is live.
Pick a prompt on the left to start. The version tagged @production is the one the fleet and the chat actually use.
New prompt
First pick its type: an agent prompt, which goes into a folder (an existing one, or type a new one like Strategies/experiments), or a ⚖ judge, an evaluator that answers yes/no about an agent's answer. A judge is named judge_…, and any prompt can reuse it with ＋ Add evaluator. Phinity writes the right kind of prompt for each.
Three ways to start: a blank page, a copy of an existing prompt, or a file / pasted text you already have.
Then iterate with Phinity on the right: say what the agent should do (or what to change — “shorter”, “more formal”, “I didn't like it”) and it rewrites the whole draft. Every step is listed: View changes shows the diff, ↶ Undo goes back one. ▶ Try it runs the draft on a sample message; your next request sees that answer, so “the answer is too long” just works.
Register v1 and publish saves it as v1 at @production. Use a short lowercase name (letters, digits, _). Then build its test cases (step 1) and evaluate it.
@production and MLflow
@production vN is the live version: the fleet and the chat load it (within ~10 minutes of a change). Other versions are candidates — saved, but not used until you publish them.
MLflow ↗ opens this prompt in the Databricks Prompt Registry, where every version is stored.
The tabs, in the order you use them
Guide — the steps at a glance, with where you are, and the 5 rules.
0 · Prompt — read, compare, edit and publish versions.
1 · Test cases — the fixed cases every version is graded on.
2 · AI review — the AI graders grade each answer.
3 · Human review — you answer Yes / No on the same answers.
4 · Consistency — the same input several times: are the answers the same?
5 · Report — 1–4 in one page, checked against the 5 rules.
6 · Improve — three better prompts that fix what the report found.
0 makes the agent; 1–4 measure it; 5 tells you if it is ready; 6 fixes it. A judge (Judges in the sidebar) has its own tabs: Prompt, What it checks and Calibration.
Each tab opens with a short What to do here guide.
Versions
Every saved version, newest first. PROD marks the live one. Click a version to read it; the chips under it show its latest scores.
Saving never overwrites: each save adds a new version, so you can always go back by publishing an older one.
Read · Diff · Edit
Read shows the selected version. Diff vs production highlights what changed against the live version (green added, red removed). Edit opens the editor — Save as a new version keeps it as a candidate unless you tick publish on save.
Propose improvement asks Phinity to draft the next version from the cases this one failed; nothing is saved until you review it. ★ Publish makes it live. To grade a version, go to 2 · LLM judges.
How each version performs
One row per evaluated version, one column per judge (the % of golden cases it passed), plus its flip rate. PROMOTABLE means it is at least as good as production on the main judge — safe to publish.
To add a row: pick a version and press ▶ Evaluate (takes a few minutes). Consistency is tested separately in 4 · Flipping.
Cases & verdicts
Every golden case of the selected run, with each judge's verdict. Click a verdict to read the judge's reasoning and mark whether it is right (👍/👎). Your marks are stored in MLflow as human labels and are what the judges get calibrated against.
Flipping
The same question asked K times. If the agent's answer changes between runs it "flips" — a sign the prompt is ambiguous. Lower is better.
Run it on the whole golden (that run feeds the Stability card) or on only the cases you pick — e.g. to re-test the ones that flipped after editing the prompt. A partial run never replaces Stability.
Live conversations
Real conversations traced from the running agent (not test cases). Press ⚖ on one to have the judges grade it — useful to catch problems the golden doesn't cover.
Who grades this agent
The judges that score this agent on its golden. An LLM judge is Phinity following written instructions (open it to read them); a code rule is an exact check, no LLM.
Tick or untick judges and press Save as a new version: the choice is stored in this agent's prompt (frontmatter judges:), so it is versioned — evaluate and publish it like any version.
κ next to each judge = how much its live version agrees with your grades. Below the minimum, don't let it decide.
Golden dataset
The fixed set of test cases (question + context, and for some agents the expected answer) that every version is graded on, stored in Databricks. Same cases every time, so scores are comparable across versions.
⇪ Seed cases fills it with the starting set; you can label cases to set the expected answer.
A new prompt? Build its golden right here: Generate with Phinity drafts cases from the prompt for you to review, or ＋ Add a case to write one by hand. Nothing is saved until you load it.
This will: