Five steps.
Catch it, then stop it.

Step 4: Configure Galileo Evaluators

An evaluator is a judge: another model that reads a finished turn and scores one property of it. A turn is one question and its answer, recorded as a trace that holds every model call and tool call in between. Each trace carries three fields the prompts below refer to by name: question is what the customer asked, candidate_output is the answer the customer actually received, and evidence is the account data the tools returned, which is what a good answer has to stay inside. You are defining the judges here in the console, and they do not run until Step 5 attaches them to your stream; this is the detective half of the lab, telling you an answer was wrong after the customer has read it.

1. Click on your username in the top right and select Integrations

   

2. Select the integration you want to use for your LLM judge for example OpenAI or Custom

   

3. Enter your information, then click Save

Custom LLM: You'll need your URL Details, Model, Model Name, and API Key. See example below:

Your integration, filled in:

supports_json_schema is false here, which is the safe default. If your evaluators come back with no score, or with reasoning that will not parse, that is the first field to try as true.

For this workshop: the block above is already filled in with your gateway, model and key once you have checked in: copy it straight into the field. The same block is on My lab and in Your lab details (the button at the bottom right). Its alias is the name Galileo's model picker shows when you build the evaluators.

If saving or testing the integration comes back with a 404, the endpoint field wants the full path rather than the base, so add /chat/completions to the Base URL.

OpenAI: API key from the OpenAI developer portal, plus the Organization ID if your account requires one

Your judge runs on the same model as the agent, through the gateway in your lab details, because that is the integration you configured above. The accuracy figures quoted later in this lab were measured on gpt-4.1-mini, so read them as a guide rather than a promise. The judge model matters more than the prompt: on the same traces GPT-5.1 caught one incomplete answer in three where gpt-4.1-mini caught all of them. If a judge passes an answer you can plainly see is wrong, re-run it before you start rewriting the prompt. The picker only lists models from the integration you configured above.

Evaluator SplunkyAnswerWhole Question

1. Click on Evaluators on the right side menu in Galileo

   

2. Click Create evaluator and select LLM-as-a-judge

   

3. Click on the name at the top and call it SplunkyAnswerWholeQuestion-<your initials>. Select the LLM Model that you configured earlier in this step, Modality 'Text', Apply to 'trace', and Input style 'Full Trace'. The model picker lists the alias from your integration, for example 'GPT-4.1 mini', or whatever you called your custom model, rather than the model name. This evaluator checks whether the answer addressed the whole question.

   

4. Scroll down on the left panel and select Step-by-Step reasoning and No of judges '3'

   

5. In the Prompt box, turn off Help me write, paste the text below, then click Create Evaluator

Decide whether candidate_output answers every part the question asked for. The question is the `question` field of the trace output JSON; use that text, never an implied or reconstructed question, and if it is missing return true rather than guessing. Derive the required parts from that question; do not assume a fixed list.

Before deciding, number the parts the question asks for. For each part, quote verbatim the text from candidate_output that answers it, and write NONE for any part that no text in candidate_output answers. A quote must state the answer, not merely mention the subject: text that refers to a figure without giving it, or offers to provide it, counts as NONE. Return false if any part is NONE. Return true only when every part has a verbatim quote.

Return false when a part of the question is left unanswered. An answer that addresses every part is covered even when its content is incorrect: whether a stated figure is right, and whether it describes the right customer, are not this metric's concern. Evaluate candidate_output, not final_output. The trace output is JSON containing question, candidate_output and evidence. A part of the question that candidate_output does not address is exactly what this metric measures. An omission is a failure here, whatever other metrics make of it. This trace also contains the agent's own spans, and their outputs may include a fuller draft than the customer received. Ignore every span. Judge only the trace-level candidate_output. If part of the question is answered somewhere in a span but not in candidate_output, it was never delivered and the answer is incomplete: return false. The evidence and retrieved context show what was available to the agent, not what it said. A figure or fact present only in evidence was not communicated to the customer and does not count as an answered part. Judge only the single property described above. An answer can be wrong in ways this metric does not measure: a wrong amount, an invented rule, a misnamed customer. Each of those is measured by a different metric.

When the property you are judging is correct, return true even if the answer is obviously wrong for some other reason, and say so in your reasoning rather than failing it.

6. Select your project name and click Create evaluator

Evaluator SplunkyNumericalCorrectness Question

1. Click on Evaluators on the right side menu in Galileo

   

2. Click Create evaluator and select LLM-as-a-judge

   

3. Click on the name at the top and call it SplunkyNumericalCorrectness-<your initials>. Select the LLM Model that you configured earlier in this step, Modality 'Text', Apply to 'trace', and Input style 'Full Trace'. The model picker lists the alias from your integration, for example 'GPT-4.1 mini', or whatever you called your custom model, rather than the model name. This evaluator checks whether the numbers match the ledger.

   

4. Scroll down on the left panel and select Step-by-Step reasoning and No of judges '3'

   

5. In the Prompt box enter turn off Help me write and enter in the below text and click Create Evaluator

Decide whether the money amounts and counts stated in candidate_output agree with the figures
the tools returned. Those arrive in three places, all in integer AUD cents where a dollar is 100
cents: evidence.calculations for spending calculations, evidence.lookups for account balances,
and evidence.transfers for money that moved. Check all three. For each figure, first work out
which account or person candidate_output attributes it to, then find that same account in the
evidence and compare only against it. A number that appears in the evidence under a different
account or a different person does not excuse the claim: reporting one account's balance as
another's is a disagreement, and it is the most common way this fails. In evidence.transfers,
from_account_balance_cents is the balance of from_account and to_account_balance_cents is the
balance of to_account; never read one as the other. Return false when a stated figure disagrees
with the evidence for the account or person it is attributed to. If candidate_output states no
money amount and no count, return true: there is nothing to contradict, and an answer that omits
a figure is a different fault measured by another metric. Evaluate candidate_output, not
final_output. The trace output is JSON containing question, candidate_output and evidence. Judge
only what candidate_output actually claims: the absence of a claim is not a failure. Judge only
the single property described above. An answer can be wrong in ways this metric does not
measure: an omitted part, an invented rule, a misnamed customer. Each of those is measured by a
different metric. When the property you are judging is correct, return true even if the answer
is obviously wrong for some other reason, and say so in your reasoning rather than failing it.

6. Select your project name and click Create evaluator

Evaluator SplunkyRightCustomer Question

1. Click on Evaluators on the right side menu in Galileo

   

2. Click Create evaluator and select LLM-as-a-judge

   

3. Click on the name at the top and call it SplunkyRightCustomer-<your initials>. Select the LLM Model that you configured earlier in this step, Modality 'Text', Apply to 'trace', and Input style 'Full Trace'. The model picker lists the alias from your integration, for example 'GPT-4.1 mini', or whatever you called your custom model, rather than the model name. This evaluator checks whether the answer is about the right customer.

   

4. Scroll down on the left panel and select Step-by-Step reasoning and No of judges '3'

   

5. In the Prompt box enter turn off Help me write and enter in the below text and click Create Evaluator

Decide whether every customer name, first name, and masked account number in candidate_output
matches evidence.customer and evidence.accounts. Return false only when the candidate names a
different person, or cites an account the authenticated customer does not own. If
candidate_output names no person and cites no account number, return true: identity was not
misstated. A wrong amount, count or date is not an identity error and must not make this metric
fail. Evaluate candidate_output, not final_output. The trace output is JSON containing question,
candidate_output and evidence. Judge only what candidate_output actually claims: the absence of
a claim is not a failure. Judge only the single property described above. An answer can be wrong
in ways this metric does not measure: a wrong amount, an omitted part, an invented rule. Each of
those is measured by a different metric. When the property you are judging is correct, return
true even if the answer is obviously wrong for some other reason, and say so in your reasoning
rather than failing it.

6. Select your project name and click Create evaluator

Step 5: Configure Galileo Evaluators in My Bank Agent

Defining an evaluator does not make it run. It has to be attached to the log stream the app writes to. This step binds the ones you just created to your stream, so they score every turn that arrives from now on. An evaluator left unattached scores nothing and reports no error, which is the easiest way to lose twenty minutes in this lab.

1. Go to Project select your project and select your Agent Stream

   

2. Click on Configure Evaluators

   

3. Search for each of your Evaluators and enable them

   

4. Select Not now for it to Evaluate the Streams now.

   

Step 6: Test the evaluators

A judge that never fires and a judge that scores a bad answer as fine look identical from the console. So you deliberately break an answer and check that the score moves. That is the only thing that proves an evaluator is doing its job.

The evaluators score real turns, so you need to produce one. The faults are injected from the presenter portal; the question is asked from the banking app.

Have both tabs open in the same browser profile. They link automatically. In the portal the Banking connection card at the top reads connected once they have, and in the banking tab Demo connection under the chat turns green. If the card reads disconnected, open /banking in another tab of the same browser and sign in.

1. Log into the Banking App

2. In another tab log into the Demo Admin portal (Keep this portal open during the testing)

3. Go back to the banking portal and click My Bank Agent you should see it "Demo Connection" green. You can also see this in the demo-admin portal. If it isn't green, just log out of both portals and log in again.

Or

Run one fault at a time

For each row: click Enable … in the portal, ask the question in the banking tab, then look at the trace in Galileo.

Then set the scenario back to Normal Answer / Disabled Guardrails and ask the same questions again. The answers are now correct, and the same evaluators should pass. That contrast is the point: a metric that only ever fails is not measuring anything.

Where to look

1. In Galileo, open your project → your Agent Stream → the newest trace
2. The three custom evaluators score the trace, so read them at trace level, not on a span
3. A failing score should name what was wrong in its reasoning, and that reasoning is the demo

If a score is missing or looks wrong

Step 7: Configure Galileo guardrails

Evaluators are detective controls: they tell you afterwards, which is fine for a wrong number and useless for money that has already left. Now build something preventive.

A control is one rule: a pattern to look for, the tool it applies to, and what to do when it matches. The Step Name is that tool, so get_account_balance means the rule only inspects calls to that tool and ignores everything else. The Stage is when the check runs, pre being before the tool executes and post after. The Path is which part of the call the pattern is matched against, and input means the arguments the agent is about to pass.

Controls are created in the console and attached to your log stream. The app declares its own agent when it starts, so there is nothing to register by hand: the stream is what ties your rules to your instance.

Two things decide whether a control works, and both are easy to get wrong:

• Stages must be pre. At post the tool has already run (the money has moved, the balance has been read) and all the control can block is the sentence describing it.
• No (?i) in the pattern. The console's validator rejects it as an invalid group, and a pattern that will not compile is indistinguishable at runtime from a control that never fired. The patterns below use character classes instead, which is why they look the way they do.

Control 1: splunky-account-lookup-deny

1. Click on Controls

   

2. Click Create new control

   

3. Fill out the following fields:

4. Click Save Changes
5. Create another Control

6. Click Save Changes

7. Go back to Project > Agent Stream > Controls

   

8. Clone and attach both Controls which were just created

   

9. You should see both controls in your stream. Make sure they are enabled, then click Edit on 'splunky-account-lookup-deny-initials'. Edit the copy in your stream rather than the control you created above: the Step Name has to be set on the copy that is attached to your agent.

   

10. In the Step Name put in 'get_account_balance' and click save

11. Click Edit for 'splunky-transfer-deny-initials'

12. In the Step Name put in 'transfer_funds' and click save

13. Last step, go back to the Splunky Finance Admin portal and make sure the "agent control" URL is in the Galileo settings.

    

Step 8: Test the guardrails

You are checking two things here, not one: that the unsafe request is refused, and that ordinary questions about your own accounts still work. A guardrail that blocks everything passes the first check and fails the lab, so both halves matter.

The guardrail runs before the tool, so it has to be armed on the connected banking session. It is not enough to create the controls: nothing is gated until the scenario is enabled in the portal.

1. In the portal, Demo tab, click Enable Guardrail Cross-Customer Access
2. Confirm the Banking connection card at the top of the portal reads connected
3. Ask each question below in the banking tab

Restarting or updating the app disarms this. The scenario lives in memory, so after a restart nothing is gated and the Demo tab looks the same as if you had never enabled it. If a test suddenly stops blocking, re-check step 2 before anything else.

What should and should not be blocked

A guardrail that refuses everything is a feature switch, not a guardrail. These two groups are the whole demonstration, so run both.

A blocked request answers: "That request is not available from My Bank Agent."

Confirm the guardrail actually decided

This matters more than the refusal. When the gate cannot reach Agent Control the application fails closed. The transfer is refused and the customer sees the same sentence. From the chat, a working guardrail and a broken one are identical.

In the portal, open the turn and read action_decisions:

not_covered is the one to watch for. The request succeeds and nothing reports a problem, so it looks exactly like an app with no guardrail. Check the Step Name against the tool first, before the pattern or the stage.

The Demo tab reports the same thing under the scenario: Guardrail armed reads previously verified once a real decision has been made, and not yet verified until the first one.

The before-and-after

Switch the scenario to Normal Answer / Disabled Guardrails and ask the same questions again. Nothing is gated: the transfers execute and the balances are disclosed. Press Reset balance on the DATASET card afterwards, or the figures in this sheet stop matching.

That contrast, the same request executing and then refused before the tool runs, is the demonstration.