Agent Platform Eval Flywheel Skill

When to use this skill

• Evaluating GenAI agents or models with the Agent Platform GenAI Evaluation SDK (client.evals.evaluate()).
• Creating evaluation datasets from session traces, pandas DataFrames, or synthetic generation.
• Selecting, configuring, or writing custom evaluation metrics.
• Analyzing rubric verdicts, loss patterns, and clustering failures.
• Suggesting concrete code/prompt improvements based on eval results.
• Evaluating a model served on an Agent Platform endpoint (BYOM) or a Model-as-a-Service (MaaS) model by ID — including deploying the model first if needed. For this case, follow references/deployment.md [blocked] and use the endpoint_evaluation.py / maas_evaluation.py scripts.

Safety & Confirmation Tiers (CRITICAL)

1. Tier R: Read-only (inspect_results.py, compare_results.py, validate_dataset.py, parse_adk_traces.py, render_html_report.py)
   • Rule: No confirmation needed. You may execute these helper scripts immediately to inspect data, validate schemas, parse traces, or compare evaluation results.
2. Tier M: Read-only with Compute Costs (client.evals.run_inference, client.evals.evaluate, client.evals.generate_conversation_scenarios, client.evals.generate_loss_clusters)
   • Rule: These operations invoke LLMs or remote evaluation services that consume compute resources and incur costs. This requires interactive confirmation with 'Yes'/'No' options. Once granted once, you do not have to prompt for future evaluation.
   • Same-turn restriction: Do not run the evaluation in the same turn as presenting the confirmation prompt. End your turn after asking and wait for the user's reply; only execute after explicit 'Yes' / approval. Printing a preview and then calling the tool before the user can answer does not count as obtaining confirmation.

Setup

python3 -c "import vertexai, google.genai, pandas, requests" \
  || pip install 'google-cloud-aiplatform[evaluation]>=1.163.0' 'google-genai>=1.0.0'

Correct SDK entrypoints

import agentplatform
client = agentplatform.Client(project=PROJECT, location=LOCATION)

client.evals.run_inference(model=..., src=...)
client.evals.evaluate(dataset=..., metrics=...)
client.evals.generate_conversation_scenarios(...)

• from agentplatform.types import evals -- ModuleNotFoundError. types is a module, not a package; use from agentplatform import types.
• from vertexai.evaluation import PointwiseMetric, EvalTask -- the superseded SDK. Its classes take different arguments (PointwiseMetric has no system_instruction), so code written against it fails with TypeError rather than an import error. Use agentplatform throughout.

The Quality Flywheel

Shortcuts that waste time

1. Prepare Data

• EvalCase list (single-turn or multi-turn):

  from agentplatform import types
  from google.genai import types as genai_types
  
  # prompt/reference/response values are Content, not str. UserContent and
  # ModelContent wrap a plain string and set the right role.
  dataset = types.EvaluationDataset(eval_cases=[
      types.EvalCase(
          prompt=genai_types.UserContent("What is 2+2?"),
          responses=[types.ResponseCandidate(
              response=genai_types.ModelContent("4"))],
          reference=types.ResponseCandidate(
              response=genai_types.ModelContent("4")),
      ),
      # For multi-turn agent traces, set agent_data instead of prompt/responses.
  ])

  Multi-turn agent traces wrap each conversation in AgentData → ConversationTurn → AgentEvent. See references/dataset_schema.md [blocked] for the full type hierarchy.
• Pandas DataFrame (tabular sources — CSV, BigQuery, Sheets):

  import pandas as pd
  from agentplatform import types
  
  df = pd.DataFrame({
      "prompt":    ["What is 2+2?", "Capital of France?"],
      "response":  ["4",            "Paris"],
      "reference": ["4",            "Paris"],
  })
  dataset = types.EvaluationDataset(eval_dataset_df=df)

  Column names must match the fields the chosen metrics expect (see references/dataset_schema.md [blocked] for the per-metric requirements table).
• Cold start (no data at all): synthesize scenarios server-side with client.evals.generate_conversation_scenarios(agent=..., config=...) -- the parameter is agent or agent_info, not agents, and config is required. The config class is types.evals.UserScenarioGenerationConfig, not types.UserScenarioGenerationConfig. Set its user_scenario_count (1-100): it defaults to None, the client accepts that, and the server rejects the call with 400 INVALID_ARGUMENT. count is a separate field and does not substitute for it. Stage 2 plays the scenarios out.
• Managed Agents (Gemini Agents API): evaluate agents created with the Managed Agents API. Use generate_conversation_scenarios to create test scenarios from the agent's configuration, run_inference to execute the agent, and evaluate to score the traces. These functions now accept managed agents and interaction ids as input. You can also evaluate existing interactions recorded via the Interactions API using InteractionsDataSource. See references/sdk_patterns.md [blocked] Pattern 8 for the full code pattern.

2. Run Inference

# Agent eval — pass a callable wrapping the user's ADK Agent/App.
client.evals.run_inference(model=agent_callable, src=dataset)

# Model eval — pass a model ID directly.
client.evals.run_inference(model="gemini-2.5-flash", src=dataset)

# Synthesized scenarios — let the simulator drive.
client.evals.run_inference(
    model=agent_callable,
    src=dataset,
    user_simulator_config=UserSimulatorConfig(max_turn=10),
)

# DataFrame also works as src= — no EvalCase wrapping needed.
client.evals.run_inference(model="gemini-2.5-flash", src=df)

# Managed Agent — pass an agent resource name.
AGENT_RESOURCE = f"projects/{PROJECT_ID}/locations/global/agents/{AGENT_ID}"
client.evals.run_inference(
    agent=AGENT_RESOURCE,
    src=scenarios,
    config={"user_simulator_config": {"max_turn": 3}},
)

3. Grade (always run)

result = client.evals.evaluate(dataset=dataset, metrics=[...])
result.show()  # Interactive HTML report with scores, rubrics, and traces.

• Predefined: types.RubricMetric.<NAME> — server-side AutoRater, no judge model needed.
• Custom LLM-as-a-judge: types.LLMMetric with prompt_template or types.MetricPromptBuilder for structured rubrics. Always set judge_model; it defaults to None and every case then fails with 400 INVALID_ARGUMENT: Error parsing JSON.
• Custom code: types.CodeExecutionMetric with a custom_function string containing def evaluate(instance: dict) for remote sandboxed execution; or types.Metric with custom_function=<callable> for local execution.

import datetime
from pathlib import Path

from agentplatform._genai import _evals_visualization

out_dir = Path("artifacts/grade_results")
out_dir.mkdir(parents=True, exist_ok=True)
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")

# fallback=str, or a DataFrame-backed dataset raises PydanticSerializationError.
result_json = result.model_dump_json(fallback=str)
(out_dir / f"results_{ts}.json").write_text(result_json)

html = _evals_visualization.get_evaluation_html(result_json)
(out_dir / f"results_{ts}.html").write_text(str(html))

4. Analyze Failures

# Only supports multi_turn_task_success and multi_turn_tool_use_quality.
# Service runs in the global region.
analysis_client = agentplatform.Client(project="PROJECT_ID", location="global")
response = analysis_client.evals.generate_loss_clusters(
    eval_result=result,
    metric="multi_turn_task_success",
    config={"max_top_cluster_count": 5},
)
for r in response.results:
    for cluster in r.clusters:
        print(
            f"[{cluster.taxonomy_entry.l1_category}/"
            f"{cluster.taxonomy_entry.l2_category}] "
            f"{cluster.item_count} cases — {cluster.taxonomy_entry.description}"
        )

5. Optimize & Iterate

Proving your work

• After running eval, print the summary_metrics table (scripts/inspect_results.py).
• After a fix, show before/after via scripts/compare_results.py.
• Before declaring success, confirm ALL cases pass — not just the one you were working on.

Rules of Engagement

1. Always Plan First: Before writing a script, output a <plan> block detailing the steps you are about to take.
2. Step-by-Step Execution: Write the script, execute it, wait for output, then analyze. Don't do everything in one response.
3. Standard Python: Use standard Python imports (import agentplatform, from google.genai import types). Don't use internal import paths.
4. Verify Before Guessing: When unsure about SDK types or metrics, check the SDK source code rather than guessing or hallucinating.

SDK Quick Reference

import agentplatform
from agentplatform import types
from google.genai import types as genai_types
import pandas as pd

# Initialize client
client = agentplatform.Client(project="PROJECT_ID", location="LOCATION")

# --- SINGLE-TURN EVAL (pandas DataFrame) -- RECOMMENDED ---
# The converter wraps plain strings for you.
df = pd.DataFrame({
    "prompt":   ["Q1", "Q2"],
    "response": ["A1", "A2"],
})
dataset = types.EvaluationDataset(eval_dataset_df=df)

# --- SINGLE-TURN EVAL (direct EvalCase) ---
# Verbose and easy to get wrong; see references/dataset_schema.md for the
# exact types before using this form.
dataset = types.EvaluationDataset(eval_cases=[
    types.EvalCase(
        prompt=genai_types.UserContent("Query here"),
        responses=[types.ResponseCandidate(
            response=genai_types.ModelContent("Model response here"))],
        reference=types.ResponseCandidate(
            response=genai_types.ModelContent("Ground truth here")),
    ),
])

# --- MULTI-TURN AGENT EVAL ---
agent_data = types.evals.AgentData(
    agents={"my_agent": types.evals.AgentConfig(
        agent_id="my_agent", instruction="You are helpful.")},
    turns=[types.evals.ConversationTurn(turn_index=0, events=[
        types.evals.AgentEvent(author="user",
            content=genai_types.Content(role="user",
                parts=[genai_types.Part(text="Hello")])),
        types.evals.AgentEvent(author="my_agent",
            content=genai_types.Content(role="model",
                parts=[genai_types.Part(text="Hi! How can I help?")])),
    ])],
)
dataset = types.EvaluationDataset(
    eval_cases=[types.EvalCase(agent_data=agent_data)])

# --- METRICS ---
predefined = types.RubricMetric.MULTI_TURN_TRAJECTORY_QUALITY
custom_llm = types.LLMMetric(name="tone",
    prompt_template="Is this polite? Response: {response}")
custom_code = types.CodeExecutionMetric(name="check",
    custom_function='def evaluate(instance): return {"score": 1.0}')

# --- EVALUATE ---
result = client.evals.evaluate(dataset=dataset, metrics=[predefined])

# --- RESULTS ---
for s in result.summary_metrics:
    print(f"{s.metric_name}: mean={s.mean_score}, pass_rate={s.pass_rate}")
for case in result.eval_case_results:
    for cand in case.response_candidate_results:
        for name, r in cand.metric_results.items():
            print(f"  {name}: score={r.score}, explanation={r.explanation}")

Bundled scripts