Customization

🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)

1. Send voice notification:
   bashCopy

   curl -s -X POST http://localhost:31337/notify \
     -H "Content-Type: application/json" \
     -d '{"message": "Running the WORKFLOWNAME workflow in the Evals skill to ACTION"}' \
     > /dev/null 2>&1 &

2. Output text notification:
   Running the **WorkflowName** workflow in the **Evals** skill to ACTION...

Evals - AI Agent Evaluation Framework

What It Does

The Problem

How It Works

When to Activate

• "run evals", "test this agent", "evaluate", "check quality", "benchmark"
• "regression test", "capability test"
• "run scenario", "multi-turn eval", "simulated user test"
• "create scenario", "simulate conversation"
• Compare agent behaviors across changes
• Validate agent workflows before deployment
• Verify ALGORITHM ISC rows
• Create new evaluation tasks from failures

Core Concepts

Three Grader Types

Evaluation Types

Key Metrics

• pass@k: Probability of at least 1 success in k trials (measures capability)
• pass^k: Probability all k trials succeed (measures consistency/reliability)

Workflow Routing

CLI Quick Reference

Quick Reference

CLI Commands

# Run an eval suite
bun run ${LIFEOS_SKILL_DIR}/Tools/AlgorithmBridge.ts -s <suite>

# Log a failure for later conversion
bun run ${LIFEOS_SKILL_DIR}/Tools/FailureToTask.ts log "description" -c category -s severity

# Convert failures to test tasks
bun run ${LIFEOS_SKILL_DIR}/Tools/FailureToTask.ts convert-all

# Manage suites
bun run ${LIFEOS_SKILL_DIR}/Tools/SuiteManager.ts create <name> -t capability -d "description"
bun run ${LIFEOS_SKILL_DIR}/Tools/SuiteManager.ts list
bun run ${LIFEOS_SKILL_DIR}/Tools/SuiteManager.ts check-saturation <name>
bun run ${LIFEOS_SKILL_DIR}/Tools/SuiteManager.ts graduate <name>

ALGORITHM Integration

# Run eval and update ISC row
bun run ${LIFEOS_SKILL_DIR}/Tools/AlgorithmBridge.ts -s regression-core -r 3 -u

| # | What Ideal Looks Like | Verify |
|---|----------------------|--------|
| 1 | Auth bypass fixed | eval:auth-security |
| 2 | Tests all pass | eval:regression |

Available Graders

Code-Based (Fast, Deterministic)

Model-Based (Nuanced)

Domain Patterns

Task Schema (YAML)

task:
  id: "fix-auth-bypass_1"
  description: "Fix authentication bypass when password is empty"
  type: regression  # or capability
  domain: coding

  graders:
    - type: binary_tests
      required: [test_empty_pw.py]
      weight: 0.30

    - type: tool_calls
      weight: 0.20
      params:
        sequence: [read_file, edit_file, run_tests]

    - type: llm_rubric
      weight: 0.50
      params:
        rubric: prompts/security_review.md

  trials: 3
  pass_threshold: 0.75

Resource Index

Key Principles (from Anthropic)

1. Start with 20-50 real failures - Don't overthink, capture what actually broke
2. Unambiguous tasks - Two experts should reach identical verdicts
3. Balanced problem sets - Test both "should do" AND "should NOT do"
4. Grade outputs, not paths - Don't penalize valid creative solutions
5. Calibrate LLM judges - Against human expert judgment
6. Check transcripts regularly - Verify graders work correctly
7. Monitor saturation - Graduate to regression when hitting 95%+
8. Build infrastructure early - Evals shape how quickly you can adopt new models

Related

• ALGORITHM: Evals is a verification method
• Science: Evals implements scientific method
• Browser: For visual verification graders

Gotchas

• Choose the right grader type: Code-based for deterministic checks (fast, cheap). Model-based for nuanced quality (flexible, expensive). Human for calibration (gold standard, slow).
• pass@k scoring requires multiple runs. A single run doesn't give statistical significance. Default to pass@3 minimum.
• Transcript capture must be enabled BEFORE the test run. Can't retroactively capture transcripts.
• Eval results go to the current work directory — not a global location. Tie evals to the work item.
• Don't evaluate skills with trivial prompts. Simple one-liners may not trigger skill usage. Test prompts must be substantive.

Examples

User: "evaluate which prompt produces better summaries"
→ Creates eval suite with 3+ test cases
→ Runs both prompts against test cases
→ Model-based grader scores quality
→ Reports pass@k and comparative analysis

User: "run evals on the Research skill after the update"
→ Uses existing test fixtures for Research
→ Before/after comparison
→ Reports any quality regressions

Execution Log

echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","skill":"Evals","workflow":"WORKFLOW_USED","input":"8_WORD_SUMMARY","status":"ok|error","duration_s":SECONDS}' >> ~/.claude/LIFEOS/MEMORY/SKILLS/execution.jsonl