How to Review AI Generated Code When Agents Write Most Pull Requests
A practical guide to reviewing AI generated code at scale. How we built an AI generated code review pipeline at SignatureAPI with review records tied to commits, review depth that follows the size of a change, and a monthly signed report for SOC 2.
In September 2026, 103 pull requests merged into the development branch at SignatureAPI. A year earlier, a normal month had between 17 and 29. AI coding agents write and review most of that code now. That raises a simple question. How do you review AI generated code when there is this much of it?
My short answer is this. Stop treating review as a click on an “Approve” button. Treat it as a written record. That record belongs to one exact commit. It lists every finding and what happened to each one. Then make the depth of the review follow the size of the change, and keep people on the decisions that really need a person.
I described the full control in a post for the company blog, How we keep every change reviewed when agents write the code. That post explains the rules. This one is the practical side. It covers what I would tell another team that runs development through agents and wants every change reviewed.
Why one approval stopped scaling
Most teams review code with one rule. A person who did not write the change reads it and approves it. GitHub can require that approval on every pull request. The standard SOC 2 policy template uses the same idea. The reviewer must be a different person from the author.
When people write the code, this rule works. The approval tells you that a second human read the diff and agreed with it.
When agents write the code, the same approval says much less. It tells you a person clicked a button. It does not tell you what was checked. It does not list the problems the review found. It does not say whether those problems were fixed or ignored, and why.
There is also a simple time problem. Around a hundred pull requests in a month is a lot of diffs to read line by line. We wanted the hundredth pull request of the month to get the same quality of review as the first one. That cannot depend on a person reading every diff.
So we changed what counts as evidence. Instead of a name on an approval, we wanted a description of the review itself.
AI generated code review pipeline
We turned the new control on at SignatureAPI on October 27, 2026. It has seven parts. Here they are as practical steps you can copy.
Review the exact commit
A review is only useful if you know which code it looked at. A pull request changes with every push. So a review that says “looks good” without a commit is weak evidence.
In our pipeline, a review record is a comment on the pull request. It names the model that ran the review and the procedure it followed. The procedure can be a review command at a given depth, a single reviewer agent, or an orchestrator. An orchestrator is an agent that splits the change among several sub-reviewers. In that case the record lists each sub-reviewer with its model and its focus. The record ends with the findings and a verdict of pass or fail. Our tooling then adds the head commit and the range of commits the review covered.
A pull request into the development branch can merge only when five things are true for its head commit:
- The required CI checks passed (lint, build, unit tests, and a contract check of the API against our OpenAPI spec).
- The title or the branch name has a ticket id.
- A secret scan of the commits finds no secrets, such as API keys.
- A valid review record exists for that exact commit, posted by someone with write access.
- A chain of review records, with no gaps, covers every commit since the branch left the development branch.
When all five hold, an automated reviewer account approves the pull request. A new push makes that approval stale. The check then goes back to “Waiting for review record”. We show that state as pending, not as failed. A push without a record yet is a normal step, not an error.
Write down every decision
Agents find a lot of things. Some are real bugs. Some do not apply. The important part is that nobody decides silently.
Each finding in a record has a number, a severity, a summary, a resolution and a reason. The severity sets the rule:
- Critical findings must be fixed, or dismissed with a written reason.
- Important findings follow the same rule.
- Minor findings can be fixed, dismissed, or left open.
Dismissing a finding is fine. Reviewers often raise things that turn out not to apply. What matters is that the reason is written next to the finding. Later, nobody has to remember why it was ignored.
The gate checks the record as evidence. It checks that the record exists, that it belongs to the current commit, and that it follows these rules. It cannot prove that every word in the record is true. Two things make honesty visible instead. The record must state exactly what ran. And every dismissed blocking finding goes into the monthly report that management signs.
Match depth to size
Here is the trade off every team hits. A pull request rarely lands in one push. Agents fix findings and failing tests, and each fix is a new push.
You could run a full review with your strongest models after every push. That is easy to defend, but it is slow and expensive. Most of that review would read code that was already reviewed. You could also review only the new commits each time. That is cheap, but it judges each push alone.
We do both, based on size. The first record on a pull request is always a full review of the whole change from its merge base (the commit where the branch started). After that, each push is measured against the previous record:
- A small follow-up has at most 200 changed lines in at most 8 files. Only the new commits need review, and any allowed model can do it, including smaller and faster ones.
- A large change is anything bigger, or any size the gate cannot compute. It needs a review of the whole pull request by one of our two most capable models, at medium depth or deeper.
Lockfiles and generated clients do not count toward the size. Merging the development branch into a pull request does not count either, because those commits had their own review. Conflict resolutions in that merge do count, because they are new code.
Count the small pushes
Per push sizing has a hole. Ten small pushes can add up to one big change that no full review ever saw.
So the gate keeps a running total. It adds up everything pushed since the last large full review. When that total goes past the small limit, the gate rejects every follow-up review, however small. A new full review resets the count. Splitting the fixes into more pushes does not help, because the total stays the same.
This rule matters with agents because follow-up pushes are cheap to make. The running total makes sure the whole change still gets a full review.
Judge by base rules
This part is easy to miss. If the gate runs the code from the pull request, the pull request can change the gate.
Our gate always runs its own code from the base branch. It never runs the code from the pull request. The secret scan does the same. Its allowlist (the list of known safe matches) comes only from the base branch.
With agents, this is not a theoretical risk. An agent told to make a check pass may simply edit the check. In our setup, that edit has no effect on its own pull request. It only takes effect after it merges under the old rules. Changes to the gate’s own files also appear in the monthly report.
One side effect is worth knowing. A pull request that changes the record format must still post a record in the old format.
Keep people on releases
Agents can write and review code. A person still decides what goes to production.
A release is a pull request from the development branch to the production branch. Merging it deploys production. The gate checks two things. Our automation must have opened the release. And a full end to end test run must have passed on staging on that exact commit. That suite runs after every staging deploy. It checks every API response against our OpenAPI spec.
The gate never approves a release. A person does, acting as release manager. That approval means “this set of commits goes to production”. It comes after the review records, the CI checks and the staging run are all in place.
Plan the emergency exit
A strict gate needs a planned way out for real emergencies.
Our gate can be bypassed in two cases only. One is an outage of the CI system. The other is a false block on a production fix. A normal urgent fix still uses the normal path, with a label.
The bypass, which we call break glass, works like this:
- Someone comments BREAK-GLASS on the pull request, with a reason and a ticket.
- A repository admin adds a temporary bypass, merges, and removes the bypass right away.
- Within one business day, someone posts a review record for the merged commit and runs the gate again on it.
In the end, the emergency change has the same evidence as every other change. It just gets it later.
SOC 2 code review requirements
SignatureAPI holds a SOC 2 Type II report. SOC 2 expects every change to be approved. Our system description says what really happens to a change. It goes through automated gates, AI assisted review, a staging environment, and a separate authorization before production.
The link to the audit is one document a month. On the 1st of each month, a change report is created automatically. It is due within 10 business days. It lists:
- Pull requests into the development branch that merged without an approval
- Releases, with who prepared, approved and merged each one, and the result of its staging end to end run
- Dismissed blocking findings, with the reason for each
- BREAK-GLASS comments
- Changes to the gate’s own files and to the repository rules
- Drift from the approved baseline of repository settings
Senior management signs this report. They do not need to read code to do it. They read decisions and the names of the people who made them. The signed report goes to our compliance platform as SOC 2 evidence for the approval of changes control.
For an auditor, this gives a clear starting point. They begin with one signed document for the month. For any line, they can go to the repository and find the pull request and its review record.
This is how our control answers the approval question. It is not a full list of what SOC 2 asks for. If you are designing your own, ask your auditor what they need to see.
Questions still open
The gate is young. By the time I wrote the company post, 35 pull requests and 6 releases had gone through it. That is not enough data to tune the numbers.
We picked 200 changed lines in 8 files as the line between small and large. We do not know yet if that line is in the right place for how our pull requests grow. We also do not know how often the running total rule fires in a normal month.
The gate does not keep that data yet. The plan is to count rejected records by the rule that rejected them. We will look at the first full months of counts before we move either number.
If you want the rules in full, with the tables and the design mocks, read the original post on the SignatureAPI blog. The one idea to keep is simple. Review AI generated code by writing down what was checked, for which commit, and what happened to every finding.