Rolando Andrade
Back to Content

How to Review AI Generated Code When Agents Write Most Pull Requests

A practical guide to reviewing AI generated code at scale. How we built an AI generated code review pipeline at SignatureAPI with review records tied to commits, review depth that follows the size of a change, and a monthly signed report for SOC 2.

· 9 min read
How to Review AI Generated Code When Agents Write Most Pull Requests

In September 2026, 103 pull requests merged into the development branch at SignatureAPI. A year earlier, a normal month had between 17 and 29. AI coding agents write and review most of that code now. That raises a simple question. How do you review AI generated code when there is this much of it?

A normal month a year earlier (17 to 29) 17 Oct 29 Nov 25 Dec 25 Jan 24 Feb 28 Mar 9 Apr 31 May 74 Jun 35 Jul 31 Aug 103 Sep 68 Oct 2025 2026 A normal month a year earlier (17 to 29) Oct 2025 17 Nov 2025 29 Dec 2025 25 Jan 2026 25 Feb 2026 24 Mar 2026 28 Apr 2026 9 May 2026 31 Jun 2026 74 Jul 2026 35 Aug 2026 31 Sep 2026 103 Oct 2026 68
Pull requests merged into the development branch each month. September 2026 (103) rises far above the 17 to 29 of a normal month a year earlier.

My short answer is this. Stop treating review as a click on an “Approve” button. Treat it as a written record. That record belongs to one exact commit. It lists every finding and what happened to each one. Then make the depth of the review follow the size of the change, and keep people on the decisions that really need a person.

I described the full control in a post for the company blog, How we keep every change reviewed when agents write the code. That post explains the rules. This one is the practical side. It covers what I would tell another team that runs development through agents and wants every change reviewed.

Why one approval stopped scaling

Most teams review code with one rule. A person who did not write the change reads it and approves it. GitHub can require that approval on every pull request. The standard SOC 2 policy template uses the same idea. The reviewer must be a different person from the author.

When people write the code, this rule works. The approval tells you that a second human read the diff and agreed with it.

When agents write the code, the same approval says much less. It tells you a person clicked a button. It does not tell you what was checked. It does not list the problems the review found. It does not say whether those problems were fixed or ignored, and why.

There is also a simple time problem. Around a hundred pull requests in a month is a lot of diffs to read line by line. We wanted the hundredth pull request of the month to get the same quality of review as the first one. That cannot depend on a person reading every diff.

So we changed what counts as evidence. Instead of a name on an approval, we wanted a description of the review itself.

AI generated code review pipeline

We turned the new control on at SignatureAPI on October 27, 2026. It has seven parts. Here they are as practical steps you can copy.

Break glass, reviewed within one business day An agent pushes a commit Review record for this exact commit Model, procedure, findings and a verdict Every blocking finding resolved Fixed, or dismissed with a written reason Review depth follows size Up to 200 lines in 8 files, new commits onlyBigger, a full review by a top model Rules read from the base branch A pull request cannot change its own gate Merge into development A person authorizes the release After staging passes on the same commit Production
The path of one commit. Every gate checks the same exact commit, and the break glass path gets its review afterwards.

Review the exact commit

A review is only useful if you know which code it looked at. A pull request changes with every push. So a review that says “looks good” without a commit is weak evidence.

In our pipeline, a review record is a comment on the pull request. It names the model that ran the review and the procedure it followed. The procedure can be a review command at a given depth, a single reviewer agent, or an orchestrator. An orchestrator is an agent that splits the change among several sub-reviewers. In that case the record lists each sub-reviewer with its model and its focus. The record ends with the findings and a verdict of pass or fail. Our tooling then adds the head commit and the range of commits the review covered.

A pull request into the development branch can merge only when five things are true for its head commit:

  1. The required CI checks passed (lint, build, unit tests, and a contract check of the API against our OpenAPI spec).
  2. The title or the branch name has a ticket id.
  3. A secret scan of the commits finds no secrets, such as API keys.
  4. A valid review record exists for that exact commit, posted by someone with write access.
  5. A chain of review records, with no gaps, covers every commit since the branch left the development branch.

When all five hold, an automated reviewer account approves the pull request. A new push makes that approval stale. The check then goes back to “Waiting for review record”. We show that state as pending, not as failed. A push without a record yet is a normal step, not an error.

Write down every decision

Agents find a lot of things. Some are real bugs. Some do not apply. The important part is that nobody decides silently.

Each finding in a record has a number, a severity, a summary, a resolution and a reason. The severity sets the rule:

  • Critical findings must be fixed, or dismissed with a written reason.
  • Important findings follow the same rule.
  • Minor findings can be fixed, dismissed, or left open.

Dismissing a finding is fine. Reviewers often raise things that turn out not to apply. What matters is that the reason is written next to the finding. Later, nobody has to remember why it was ignored.

The gate checks the record as evidence. It checks that the record exists, that it belongs to the current commit, and that it follows these rules. It cannot prove that every word in the record is true. Two things make honesty visible instead. The record must state exactly what ran. And every dismissed blocking finding goes into the monthly report that management signs.

Match depth to size

Here is the trade off every team hits. A pull request rarely lands in one push. Agents fix findings and failing tests, and each fix is a new push.

You could run a full review with your strongest models after every push. That is easy to defend, but it is slow and expensive. Most of that review would read code that was already reviewed. You could also review only the new commits each time. That is cheap, but it judges each push alone.

We do both, based on size. The first record on a pull request is always a full review of the whole change from its merge base (the commit where the branch started). After that, each push is measured against the previous record:

  • A small follow-up has at most 200 changed lines in at most 8 files. Only the new commits need review, and any allowed model can do it, including smaller and faster ones.
  • A large change is anything bigger, or any size the gate cannot compute. It needs a review of the whole pull request by one of our two most capable models, at medium depth or deeper.

Lockfiles and generated clients do not count toward the size. Merging the development branch into a pull request does not count either, because those commits had their own review. Conflict resolutions in that merge do count, because they are new code.

Count the small pushes

Per push sizing has a hole. Ten small pushes can add up to one big change that no full review ever saw.

So the gate keeps a running total. It adds up everything pushed since the last large full review. When that total goes past the small limit, the gate rejects every follow-up review, however small. A new full review resets the count. Splitting the fixes into more pushes does not help, because the total stays the same.

This rule matters with agents because follow-up pushes are cheap to make. The running total makes sure the whole change still gets a full review.

Judge by base rules

This part is easy to miss. If the gate runs the code from the pull request, the pull request can change the gate.

Our gate always runs its own code from the base branch. It never runs the code from the pull request. The secret scan does the same. Its allowlist (the list of known safe matches) comes only from the base branch.

With agents, this is not a theoretical risk. An agent told to make a check pass may simply edit the check. In our setup, that edit has no effect on its own pull request. It only takes effect after it merges under the old rules. Changes to the gate’s own files also appear in the monthly report.

One side effect is worth knowing. A pull request that changes the record format must still post a record in the old format.

Keep people on releases

Agents can write and review code. A person still decides what goes to production.

A release is a pull request from the development branch to the production branch. Merging it deploys production. The gate checks two things. Our automation must have opened the release. And a full end to end test run must have passed on staging on that exact commit. That suite runs after every staging deploy. It checks every API response against our OpenAPI spec.

The gate never approves a release. A person does, acting as release manager. That approval means “this set of commits goes to production”. It comes after the review records, the CI checks and the staging run are all in place.

Plan the emergency exit

A strict gate needs a planned way out for real emergencies.

Our gate can be bypassed in two cases only. One is an outage of the CI system. The other is a false block on a production fix. A normal urgent fix still uses the normal path, with a label.

The bypass, which we call break glass, works like this:

  1. Someone comments BREAK-GLASS on the pull request, with a reason and a ticket.
  2. A repository admin adds a temporary bypass, merges, and removes the bypass right away.
  3. Within one business day, someone posts a review record for the merged commit and runs the gate again on it.

In the end, the emergency change has the same evidence as every other change. It just gets it later.

SOC 2 code review requirements

SignatureAPI holds a SOC 2 Type II report. SOC 2 expects every change to be approved. Our system description says what really happens to a change. It goes through automated gates, AI assisted review, a staging environment, and a separate authorization before production.

The link to the audit is one document a month. On the 1st of each month, a change report is created automatically. It is due within 10 business days. It lists:

  • Pull requests into the development branch that merged without an approval
  • Releases, with who prepared, approved and merged each one, and the result of its staging end to end run
  • Dismissed blocking findings, with the reason for each
  • BREAK-GLASS comments
  • Changes to the gate’s own files and to the repository rules
  • Drift from the approved baseline of repository settings

Senior management signs this report. They do not need to read code to do it. They read decisions and the names of the people who made them. The signed report goes to our compliance platform as SOC 2 evidence for the approval of changes control.

Change report Created on the 1st, due in 10 business days Merged without an approval Releases, with who approved them Dismissed blocking findings BREAK-GLASS comments Changes to the gate and its rules Drift from the approved settings Signed by senior management SOC 2 evidence Compliance platform Pull request and its review record Auditor traces a line
One signed document a month. The signer reads decisions and names, and an auditor can follow any line back to its pull request.

For an auditor, this gives a clear starting point. They begin with one signed document for the month. For any line, they can go to the repository and find the pull request and its review record.

This is how our control answers the approval question. It is not a full list of what SOC 2 asks for. If you are designing your own, ask your auditor what they need to see.

Questions still open

The gate is young. By the time I wrote the company post, 35 pull requests and 6 releases had gone through it. That is not enough data to tune the numbers.

We picked 200 changed lines in 8 files as the line between small and large. We do not know yet if that line is in the right place for how our pull requests grow. We also do not know how often the running total rule fires in a normal month.

The gate does not keep that data yet. The plan is to count rejected records by the rule that rejected them. We will look at the first full months of counts before we move either number.

If you want the rules in full, with the tables and the design mocks, read the original post on the SignatureAPI blog. The one idea to keep is simple. Review AI generated code by writing down what was checked, for which commit, and what happened to every finding.