Awesome Agentic Engineering

Production guide · 10 evidence gates

AI Agent Production Readiness Checklist

A demo proves that an agent can complete a task once. A production review asks whether the task is bounded, repeatable, observable, recoverable, affordable, and safe when real users and real tools are involved.

Score your agent Get the templates

How to use this checklist

Score each gate from 0 to 2. A 0 means the control is missing, a 1 means it exists but is incomplete or manual, and a 2 means the team can show repeatable evidence. Record the result in an Agent Card and review it whenever the model, tools, permissions, data sources, or release scope changes.

A total score never overrides a launch blocker. Missing authorization, unbounded destructive tools, exposed secrets, or no recovery path should stop a release regardless of the other points.

The 10 production gates

Gate Evidence for 2/2 Block release when
Goal clarity Named users, bounded task, success metric, and explicit non-goals The agent is expected to handle an undefined class of requests
Tool permissions Per-tool scopes, allowlists, validation, and approval for risky writes A model-controlled action can exceed the requester's authority
Memory Schema, owner, retention, inspection, deletion, and tenant boundaries State can leak across users or cannot be removed
Evals Versioned scenarios for normal, edge, adversarial, and failure paths Release quality is judged only by demo prompts
Failure handling Timeouts, bounded retries, fallback, idempotency, and user recovery A partial failure can silently duplicate or lose a side effect
Security Injection, data-boundary, secret, egress, and unsafe-tool tests Retrieved content can redefine instructions or expose credentials
Observability Trace IDs across model, retrieval, tools, policy, cost, and outcome Operators cannot reconstruct why a consequential action occurred
Cost control Per-run budget, step limits, concurrency quotas, alerts, and aborts A loop can spend without a deterministic ceiling
Human review Risk tiers, authorized approvers, preserved evidence, and resumable state Irreversible or high-impact actions execute without review
Documentation Setup, architecture, ownership, threat model, runbook, and examples Only the original builder can operate or disable the agent

What counts as evidence

A policy sentence is not the same as an enforced control. Prefer artifacts that another engineer can inspect or rerun:

  • Goal: acceptance criteria and non-goals tied to a named workflow.
  • Permissions: tool schemas, scopes, policy decisions, and denied-call tests.
  • Evals: fixtures with expected outcomes and a release threshold.
  • Operations: traces, budget alerts, rollback steps, and an owner for incidents.
  • Human control: an approval record showing who reviewed which evidence before execution.

The example Agent Cards show how to keep this evidence close to the code instead of in a release meeting that cannot be reproduced.

Interpret the 20-point score

Score Stage Recommended action
0–7 Demo only Keep tools read-only or simulated; define the workflow and owners
8–14 Prototype Use internal users, strict limits, and mandatory review
15–18 Limited beta Canary a bounded cohort and close every launch blocker
19–20 Production candidate Run the operational review; expand authority only with evidence

Use the browser scorecard to generate a shareable result and download a machine-readable agent-card.json.

Turn the review into a CI gate

Keep the Agent Card in the consumer repository and run the public Node 24 Action whenever the card, workflow, tools, or evals change. 

- uses: lindixu6-hash/awesome-agentic-engineering@v0
  with:
    card: agent-card.json
    min-score: "15"
    fail-below: "true"
    fail-on-blockers: "true"

Strict blocker mode is opt-in for backward compatibility. The Action writes the numeric score, rating, badge, combined pass result, blocker count, and blocker list as outputs. It adds both gates to the workflow summary, uses no model API, and requires no external key.

For a local check without cloning this project: 

npm exec --yes \
  --package=github:lindixu6-hash/awesome-agentic-engineering#v0 \
  -- agentic-score agent-card.json

Convert incidents into regression tests

Production readiness improves when a public incident, internal failure, vulnerability, or red-team finding becomes a test that can fail before the next release.

  1. Record the source and classify it accurately: observed incident, vulnerability, adversarial test, or research proof of concept.
  2. Write the precondition as Given, the unsafe or failing behavior as When, and the required control as Then.
  3. Add the fixture to the eval suite and define the expected refusal, approval, fallback, or containment behavior.
  4. Gate releases on that fixture so the same class of failure cannot return silently.

See the source-linked incident library for complete Given/When/Then examples.

A practical release review

  1. Score all ten gates and attach evidence for every 2.
  2. List launch blockers separately from the numeric total.
  3. Run normal, adversarial, timeout, partial-failure, and budget tests.
  4. Confirm approvers, rollback owner, disable path, and incident channel.
  5. Start read-only, shadow, or canary; expand authority only after review.

Copy the launch checklist into the release issue so the decision and its evidence remain reviewable.