AI · 11 — Eval (quality gate)

Score an AI answer against assertions (contains / regex / not-contains) and gate on the result — the pattern for testing prompts before they ship. Deterministic checks here; wire the ◈ Model and add an 'llm-judge' assertion to grade with a model.

What’s inside

6 nodes — every type links to its full reference page.

Step	Node	Type
AI answer	Inject	`input.inject`
Score it	AI Eval	`ai.eval`
Ollama (local)	AI Model	`ai.model`
Passed?	IF	`logic.if`
Ship it	Console	`utility.console`
Block	Console	`utility.console`

Import it

Studio → Import — paste the JSON below (or the URL /docs/templates/ai-11-eval-quality-gate.json).

Or ask the AI Copilot with the JSON pasted after the phrase:

import this workflow json into a new workflow named "AI · 11 — Eval (quality gate)"

Any credentials the nodes need are ${credential.…} references — add them once in Admin → Credentials and the template picks them up. Nothing secret ships in a template.

The workflow document (flowdrome.workflow.v1, 5 node types)

{
  "schemaVersion": "flowdrome.workflow.v1",
  "id": "",
  "name": "AI · 11 — Eval (quality gate)",
  "version": "0",
  "status": "draft",
  "nodes": [
    {
      "id": "seed",
      "type": "input.inject",
      "label": "AI answer",
      "category": "trigger",
      "config": {
        "contentType": "application/json",
        "payload": {
          "answer": "Thanks for reaching out! Your order 1001 ships tomorrow."
        }
      },
      "position": {
        "x": 0,
        "y": 23
      }
    },
    {
      "id": "score",
      "type": "ai.eval",
      "label": "Score it",
      "category": "ai",
      "config": {
        "actualField": "answer",
        "assertions": [
          {
            "type": "contains",
            "expected": "order",
            "field": ""
          },
          {
            "type": "not-contains",
            "expected": "I cannot",
            "field": ""
          },
          {
            "type": "regex",
            "expected": "^[A-Z]",
            "field": ""
          }
        ],
        "provider": "ollama",
        "baseUrl": "",
        "apiKey": "",
        "model": "qwen2.5:1.5b",
        "timeoutMs": 120000
      },
      "position": {
        "x": 530,
        "y": 23
      },
      "attachments": {
        "model": "llm"
      }
    },
    {
      "id": "llm",
      "type": "ai.model",
      "label": "Ollama (local)",
      "category": "ai",
      "config": {
        "provider": "ollama",
        "baseUrl": "",
        "apiKey": "",
        "model": "qwen2.5:1.5b",
        "models": [],
        "temperature": 0.2,
        "maxTokens": 512
      },
      "position": {
        "x": 530,
        "y": 245
      }
    },
    {
      "id": "gate",
      "type": "logic.if",
      "label": "Passed?",
      "category": "flow",
      "config": {
        "path": "pass",
        "operator": "equals",
        "value": true
      },
      "position": {
        "x": 1060,
        "y": 23
      }
    },
    {
      "id": "ok",
      "type": "utility.console",
      "label": "Ship it",
      "category": "utility",
      "config": {
        "message": "✅ PASSED {{ $json.passed }}/{{ $json.total }} (score {{ $json.score }})"
      },
      "position": {
        "x": 1590,
        "y": 0
      }
    },
    {
      "id": "bad",
      "type": "utility.console",
      "label": "Block",
      "category": "utility",
      "config": {
        "message": "❌ FAILED — score {{ $json.score }}"
      },
      "position": {
        "x": 1590,
        "y": 178
      }
    }
  ],
  "edges": [
    {
      "id": "e0",
      "sourceNodeId": "seed",
      "sourcePort": "output",
      "targetNodeId": "score",
      "targetPort": "input"
    },
    {
      "id": "e1",
      "sourceNodeId": "score",
      "sourcePort": "output",
      "targetNodeId": "gate",
      "targetPort": "input"
    },
    {
      "id": "e2",
      "sourceNodeId": "gate",
      "sourcePort": "true",
      "targetNodeId": "ok",
      "targetPort": "input"
    },
    {
      "id": "e3",
      "sourceNodeId": "gate",
      "sourcePort": "false",
      "targetNodeId": "bad",
      "targetPort": "input"
    }
  ],
  "triggers": [
    {
      "id": "t0",
      "nodeId": "seed"
    }
  ],
  "meta": {
    "snippets": [],
    "source": "template-gallery",
    "description": "Score an AI answer against assertions (contains / regex / not-contains) and gate on the result — the pattern for testing prompts before they ship. Deterministic checks here; wire the ◈ Model and add an 'llm-judge' assertion to grade with a model."
  }
}

Generated from the verified demo corpus — this exact document seeds and runs on a fresh Flowdrome. Raw JSON: ai-11-eval-quality-gate.json.