Agent-Forge

Agent-Forge runs a Karpathy-style propose → eval → score → commit-or-revert loop over a mutable, git-versioned AGENT.md specification — so agents iteratively improve against a measurable eval suite without human intervention between iterations.

It is framework-agnostic (LangChain, LangGraph, CrewAI, AutoGen, or raw SDKs), using an LLM-judge eval harness with held-out validation to guard against overfitting, and git as the safety net: every proposed change is committed or reverted on its score. Built on Python 3.12+ with FastAPI, SQLAlchemy 2.0, Pydantic v2, a Celery + Redis pipeline, pgvector/sentence-transformers retrieval, and MCP — containerized with Docker Compose and fronted by a Next.js trust dashboard.

• Git-native improvement loop with full audit trail
• LLM-judge evals + held-out validation
• Framework-agnostic adapters · pgvector memory · Celery orchestration

Forge agents worth trusting. 201 commits.

Architecture

The system splits into layers, each owning one part of the loop:

• AGENT.md — the mutable, git-versioned spec and single source of truth for agent behavior. Every proposal is a diff against this file.
• Framework adapters — thin wrappers around LangChain, LangGraph, CrewAI, AutoGen, or raw SDKs. These run and evaluate the agent under test during the eval step; they don't generate the proposal. Proposing always runs on a fixed system adapter (Anthropic Claude, falling back to Azure OpenAI depending on which credentials are configured) — independent of whatever framework the agent being improved happens to use.
• Eval harness — the LLM-judge, scoring the candidate against the main eval suite every iteration, plus a periodic held-out check to catch overfitting before it reaches the spec.
• Commit-or-revert gate — git itself as the persistence and rollback layer: passing scores land as commits, failing ones are reverted, so the spec's own history is the audit trail.
• Service layer — FastAPI, SQLAlchemy 2.0, and Pydantic v2 expose the loop as an API; Celery + Redis run propose/eval as background jobs; pgvector/sentence-transformers back a separate semantic-memory layer for live agent conversations elsewhere in the system — the improvement loop's own iteration history is a plain relational log, recalled in-session rather than via vector search; MCP exposes the same surface as callable tools; Docker Compose containerizes it all; a Next.js dashboard surfaces the audit trail.

How It Works

1. An iteration triggers — scheduled or on demand — into the Celery/Redis queue rather than running inline.
2. A fixed system adapter (Claude, or Azure OpenAI as fallback, chosen by whichever API credentials are configured) proposes an edit to AGENT.md: a new instruction, a changed constraint, a removed rule — independent of whatever framework the agent under test itself runs on.
3. The main eval suite scores the candidate via the LLM-judge — this is the score that gates the commit-or-revert decision every iteration.
4. The score is compared against the current committed baseline.
5. Higher score → git commits the candidate as the new AGENT.md, permanently part of the history. Lower or tied → git reverts and the baseline stands.
6. Every 10th iteration by default, a separate held-out validation pass checks the loop's trajectory against examples it never optimized against — purely to catch overfitting, logged as an alert rather than gating that iteration's own commit/revert call.
7. The last several iteration records (score, change description) carry forward in-session so the next proposal has recent context on what's already been tried — a plain history log, not a vector search.
8. The Next.js dashboard and MCP surface expose the audit trail — every commit is a decision an eval made, not a person.
9. The loop repeats with no human between iterations; the only touchpoint left is judging whether the trend is worth continuing.