Telling an agent a constraint is not the same as enforcing it

This post was written for All Things Agentic Hackathon, a global hackathon hosted by Google. It documents the project I built for that entry.

I spent a day building an agent that plans corporate training, and the most useful thing I learned had nothing to do with prompting. It was this: on 38 percent of the plans, the model broke a rule it had been explicitly given in its own instructions.

Not a subtle rule. A number.

The setup

Large companies already write their HR standards down. There is a job architecture — families, series, roles. There is a grade rubric that defines what each level means, usually across a few axes. There is a course catalog with hundreds of entries, tagged with difficulty and cost and effort. All three exist as spreadsheets, and nothing connects them.

So the connecting is done by hand. Someone in L&D opens three files, forms a judgement about where a person sits, and picks courses that feel about right. For a few hundred people that is weeks of work, the reasoning is never written down, and when a manager asks why one person got four courses and another got two, there is no answer that survives the question.

I built Rungs to do the connecting. Google ADK for orchestration, Gemini 3.5 Flash for the judgement calls, Cloud Run and Firestore underneath. Four steps per person: assess them against the rubric, compute the gap, plan a twelve-week roadmap from the catalog, verify the plan.

The part I expected to be decoration

That fourth step — verify — I added because it felt like the responsible thing to do. A plain Python function that checks the plan against the training budget, the learning-load ceiling, the difficulty bounds for the person's grade, how many courses overlap in a week, and whether every axis with a gap actually got a course.

The planner already knows all of this. Its prompt says, in as many words: total fee must stay at or under two hundred dollars, total learning load at or under a hundred and fifty points, never assign a course above the learner's grade plus one.

I assumed the verifier would sit there and pass everything.

The first time I ran the full forty-person roster, it did pass everything — and that turned out to be a bug in my own setup rather than a result. I had set the budget at four hundred dollars and the load ceiling at two hundred and twenty. The most expensive plan any run produced came to three hundred and thirty dollars and two hundred and sixteen points. My constraints sat above the maximum the system could reach. They could never bind. The verifier had nothing to do.

That is worth sitting with for a second, because it is an easy mistake to ship. A green check next to every plan looked like the system working. It was the system not being tested.

What happened when the numbers actually bound

I set the budget to two hundred dollars and the ceiling to a hundred and fifty — figures that a real per-person annual training allowance would plausibly land on, and that a three-course plan can genuinely exceed.

Across every plan the service has produced since — 500 of them, counted out of the audit trails rather than off one run:

• 190 of 500 plans were rejected at least once and sent back to be rewritten
• 140 for going over the learning-load ceiling, 117 for going over budget
• After rewriting, all 500 satisfied every constraint

Roughly two plans in five. From a model that had been told the limits, in its own instructions, in plain language, immediately above the data.

This is not a knock on Gemini specifically. It is a property of the arrangement. An instruction in a prompt is a strong prior, not a gate. If you need a gate, you have to write one.

Where the model earns its place

The temptation, having found this, is to conclude the model cannot be trusted with anything. That would be the wrong lesson, and the numbers say so too.

I planted a test. Thirteen of the forty people on the roster have a deliberate mismatch in their record: their self-assessment describes work one level above or below their assigned grade, on a single axis. The roster the agent reads gives no hint which thirteen. The answer key lives in a separate file that no part of the pipeline loads.

Over ten passes of the roster the assessment finds 72 percent of them, at 87 percent precision, with false alarms on under 2 percent of the eighty-six clean axes.

Here is one it caught. A quality-assurance inspector whose self-assessment arrived as a scan of a paper form — the agent reads the page too — with GL2 written in the grade box. Two axes came back where the form said they would. Expertise came back at GL3, above the grade on the form, because the person had written that they understand why the tolerances are set where they are and decide themselves whether an out-of-band part ships. That is the GL3 definition almost word for word. The agent went with the evidence and quoted the sentence it used.

No deterministic function I could reasonably write would find that. It requires reading a sentence against a rubric and deciding they mean the same thing.

The ones it misses are all the same shape: the evidence sits above the assigned grade and the agent defers to the grade. Under-calling a promotion case is the safer of the two failure directions, but it is still a failure, and it is in the README.

And then I found out I had been measuring wrong

Every number above the line I have just written came from one run of the roster. I ran it again to check something unrelated, and recall came back at 62 percent where it had been 77.

Nothing had changed. Same forty records, same code, same model, same prompt. The assessment runs at temperature 0.2, which is low but is not zero, and forty records times three axes is a small enough sample that the draw moves the number nine points.

So I had been doing exactly what I had caught the system doing in the paragraph above: reporting a green check as though it were evidence. Ten passes later — 400 assessments, all against one deployment — the honest version is 72.3 percent recall, standard deviation 9.0, observed range 62 to 92. Any single run of that supports a headline anywhere in a thirty-point band, and I get to pick which run I quote.

The repeated measurement also bought two things one run could not.

The misses stopped being anonymous. Four axes account for 34 of the 36 misses across ten passes, and one of them was missed in all ten. That is not sampling noise, that is a reproducible blind spot with a name and a direction — and it turns "the agent sometimes misses one" into a defect you can go and work on.

The second reader turned out to be the better reader. I had added Gemma 4 31B as a second opinion on the assumption it would be a weaker check on a stronger model. Scored against the same key over the same ten passes it finds 98 percent of the planted mismatches to Gemini's 72 — and calls roughly twice as many clean axes wrong. One conservative reader, one sensitive one, failing in different directions. That is the only reason a second reader is worth paying for; two models that fail the same way just cost twice as much.

And it produced the one number in the project I would actually stake something on. Route on how many readers depart from the grade on file, and across ten passes and about 1,200 axis judgements, not one planted mismatch ever survived both readers unchallenged. Where Gemini and Gemma both let the record stand, the record was right — every time, in every pass. That is what turns a review queue from a feature into something an L&D team can actually work from: not "here are the ones we flagged", but "the ones we did not flag did not need you".

The other thing worth measuring

Eight of the forty have no self-assessment on file at all. Thin records are the normal case in real HR systems, not an edge case.

Confidence on those: 0.24 mean. Confidence where a self-assessment exists: 0.76. Those two are the steadiest numbers in the whole project — they move by less than 0.02 between runs where everything else moves by nine points.

The agent also returns a list of what it would have needed to judge better — "manager feedback or objective performance data to verify whether they handle routine work independently despite checking everything with their manager." That list is arguably more useful to an L&D team than the grade itself, because it tells them what to go collect.

An assessment that hides its uncertainty is worse than no assessment, because someone will act on it.

One more thing, which I did not go looking for

Early on, the model read the roster's names and started writing "she has completed…".

The records state no such thing. It had inferred gender from a name.

In a document that feeds a promotion decision, a system that infers a protected attribute from a name is a system that will eventually be challenged and lose. The prompt now says the record does not state anyone's gender and a name is not evidence of it. There is a test that greps the assessments for gendered pronouns.

It cost four lines. I only found it because I read the output rather than the pass/fail.

The shape that came out of this

The pipeline ended up split along a line that was not obvious to me at the start:

The model does the reading. Judging where a person sits from their record. Sequencing courses into something that holds together as a plan. These need comprehension, and no function I write will do them.

Code does the arithmetic and the gates. Target grade minus estimated grade. Budget totals. Load totals. Week overlap. Coverage. Every number that someone might later have to defend.

The two halves talk through structured objects — every model output is constrained by a schema, and the verifier's rejections go back to the planner as a typed list of violations, each with a code and a sentence. That list is the whole interface between generation and enforcement.

If I had put the checks in the prompt and trusted them, the system would have shipped over-budget plans to nearly two people in five, and the green check next to each one would have looked exactly the same.

---

Rungs was built for the All Things Agentic Hackathon on Gemini 3.5 Flash, Google ADK, Cloud Run, and Firestore, with Gemma 4 31B as the second reader. Every person, course, and competency statement in it is generated — the company it plans for does not exist. The measurement harness is in the repository and needs no cloud credentials: it drives the public URL and scores against an answer key that ships with the code, so every number above can be recomputed rather than believed.