Human Feedback Loop for AI Agents: RLHF Quick Starter

What you get

• AGENTS — a numbered contract every run follows: ingest feedback, load learnings, do the work, cite what was applied, log the run. A run that skips a step is invalid.
• loop/learnings and loop/classification-rules — what your humans taught, one line each, with the feedback it came from.
• feedback/, projects/*/reports/, runs/ — a worked demo: four project status reports rated red / yellow / green, three human corrections, and a second run that gets them right and says why.

Start in 3 steps

1. Click Use template. The bundle becomes yours.
2. Connect your AI assistant over MCP — Claude, ChatGPT, Cursor or any MCP client.
3. Ask: “Classify this week’s project reports and cite the learnings you applied.”

How it grows

When you disagree, say so in chat or write a note in feedback/. The next run reads it first, turns it into a learning (L-001, L-002 …) and, for a hard threshold, a rule. Every output names the learnings it used, so you can check it listened. The log keeps who changed which rule and why, so the history of what you taught is never lost. Swap the demo reports for support tickets, invoices or code reviews and keep the loop.

Is this real RLHF?

No model is trained. Reinforcement learning from human feedback, in the strict sense, changes a model’s weights. Here your feedback changes what the agent reads before it works — in-context learning you can see, edit and share across every assistant you connect.

About this example. This template is a starting point, not a finished reference. Its content shows how the structure works; it is not complete, may be out of date, and has not been checked for your situation. Projects, people and thresholds are fictional. Replace them with your own, and check anything that matters against a current source or a qualified professional.

Use template, connect your assistant, and ask the first prompt — it takes two minutes.