Hands-on lab · Splunky Finance · 8 steps
Observe, evaluate, and secure the agents, models, and infrastructure behind your AI.
In this lab you get your own copy of a fictional bank, its AI agent wired to Galileo. Build the evaluators that catch a wrong answer, then the guardrail that stops an unsafe one.
Bring your laptop with a browser. Nothing to install.
Lab check-in
Type the passphrase your instructor gives you. Checked in already? Use the same email to pick up where you left off.
How it runs
The passphrase from the room gets you your own copy of the bank and a Galileo account made for your email.
Follow the briefing: point the agent at your Galileo project and at the model, then find your first question as a trace.
Three evaluators that score every answer, and a guardrail that refuses an unsafe action before it runs.
The lab
Sign in to Galileo and create an API key, a project and a log stream for the agent to report into.
Paste your key, project and stream into the app's Setup tab and prove the connection holds.
Point the agent at the model, ask it one real question, and find that question as a trace in Galileo.
Define three LLM-as-a-judge evaluators: the whole question answered, the numbers right, the right customer.
Attach the judges to your stream from the app, so every new turn is scored.
Inject faults one at a time and watch each judge catch the answer it was built for.
Build the guardrail that refuses a transfer out of someone else's account, before it happens.
Ask the agent to move another customer's money and watch the guardrail decide.
House rules
Evaluators live at the organisation level, so put your initials on each one, as the guide shows.
Your bank, your data, your settings. Nothing you do reaches anyone else's.
Judges are models and they vote. If a score looks wrong, run it again before you rewrite the prompt.
Check the troubleshooting page first, then put your hand up.
Ready
Three setup steps, then the lab. Your details are one click away on every page.