AI code review (Claude, maybe Codex)
Hauke Mehrtens
hauke at hauke-m.de
Mon Apr 6 08:41:05 PDT 2026
Hi,
I added Claude code review recently to OpenWrt, see:
https://github.com/openwrt/openwrt/blob/main/.github/workflows/claude-code-review.yml
This GitHub Action will review a PR when an OpenWrt member adds a
comment with "/claude". This will trigger a GitHub Action which runs
Claude Code to perform the code review.
Summary of the current manual test run
======================================
I think this is helpful and it finds real problems. It is a bit
nitpicky, but the false positive rate is pretty low.
It does not only look at the diff itself, but it starts an agent which
also looks at the full file being modified. When a new DTS file is
added, it often looks at similar DTS files and suggests making the new
one look more like the existing ones. It also reads all existing
comments on the PR.
The review is pretty expensive; a PR review costs between $0.20 and
$1.50, with an average price per review of $0.50. This depends on the
number of turns it does. We can restrict the maximum number of turns,
but it will reduce the quality of the review.
I am using the Claude API with their normal API pricing. The review is
done using Claude Sonnet 4.6.
The claude-code-action looks like beta software to me. We are using a
fork because the normal version does not support reviews of PRs from
forked repositories. This was fixed in version 1.0.49, but then reverted
due to another problem. It is also not well documented what permissions
it needs and how to configure it correctly.
Upstream is already 12 versions ahead of my two-week-old fork.
https://github.com/hauke/claude-code-action/tree/v1.0.77-fixed
Here is the full list of PRs where it ran:
https://github.com/openwrt/openwrt/actions/workflows/claude-code-review.yml?query=is%3Asuccess
Current problems
================
We do not have enough reviewers for all the PRs we receive. Many of the
PRs adding new devices have the same problems. AI can catch most of
them, so we no longer have to handle them manually. I hope AI can help
us reduce the load on PR reviews, increase the quality of PRs, and also
increase the number of PRs we will merge. If we activate it to run
directly after PR creation, contributors will also get fast feedback.
I think AI would have found the recently discovered and fixed security
problems in mdnsd and LuCI already when the PR was created.
Open Source sponsoring
======================
I have applied for the Open Source maintainers programs from Claude and
OpenAI Codex:
https://developers.openai.com/community/codex-for-oss
https://claude.com/contact-sales/claude-for-oss
(The form for Claude does not show for me in Firefox.)
OpenAI Codex is similar to Claude Code as far as I understand.
The Claude program only offers their Claude Max plan, but this does not
work with CI pipelines. We actually need API credits. The OpenAI Codex
program looks like it matches our use case.
I also applied for Codex Security, which should be able to find and fix
security problems.
Future plan
===========
I want to run a good AI agent on all PRs automatically when they are
opened and when they are updated. This should apply to all repositories
under the OpenWrt GitHub organization. If we were to fully pay for
Claude ourselves, this would cost around $200 per month. I hope to get
some sponsorship from Anthropic and/or OpenAI to make this happen.
Once we have that, I would first integrate it into our main repository
and then into LuCI.
Based on the code review results and what it missed or wrongly suggested
we should create documentation the AI can use to improve further
reviews. We should add a CLAUDE.md and AGENTS.md or a special version
for code reviews which we point the AI to.
How do other Open source projects handle this? Do they have a summary of
what they do and how well it works for them?
Does someone know other alternatives to Anthropic Claude Code and OpenAI
Codex? Does someone know other ways to get sponsorship for this?
Do you agree on running this automatically on all PRs?
Feel free to try it yourself by adding a comment with /claude to a PR.
It works for all people with commit permission.
Example Reviews
===============
Here are some recent examples. The num_turns is the number of turns the
AI used for the review. It ran grep to look for similar files, or
requested more context, for example. total_cost_usd is the cost of the
action in USD, excluding taxes.
Sysupgrade fixes, including related to apk#17847
https://github.com/openwrt/openwrt/pull/17847#issuecomment-4190389607
"num_turns": 11, "total_cost_usd": 0.2514354
generic: net: phy: realtek: add 5G and 10G PHY support#22563
https://github.com/openwrt/openwrt/pull/22563#issuecomment-4188211274
"num_turns": 34, "total_cost_usd": 1.01998215,
Fix setup of non-80MHz 802.11s mesh interfaces on EHT hardware#22644
https://github.com/openwrt/openwrt/pull/22644#issuecomment-4185594305
"num_turns": 14, "total_cost_usd": 0.23388720000000002,
It somehow stopped after creating an example patch description which
I requested in an other comment.
netifd/odhcp6c: convert DHCP protocol handlers to ucode#22751
https://github.com/openwrt/openwrt/pull/22751#issuecomment-4184960743
"num_turns": 20, "total_cost_usd": 0.4492875,
ramips: mt7621: fix reset hang and re-enable LIST_HARDENED#22724
https://github.com/openwrt/openwrt/pull/22724#issuecomment-4183374970
"num_turns": 14, "total_cost_usd": 0.38940090000000005,
mediatek: filogic: add support for Hilink RM65 development board#22683
https://github.com/openwrt/openwrt/pull/22683#issuecomment-4164682965
"num_turns": 54, "total_cost_usd": 0.6598969499999999,
mediatek: add Huasifei WH3000 Pro NAND support#22694
https://github.com/openwrt/openwrt/pull/22694#issuecomment-4164295740
"num_turns": 26, "total_cost_usd": 0.48784635000000004,
It did a incorrect suggestion to change the SPDX-License header
realtek: i2c: rtl9300: backport rtl9607c i2c support and speed patches#22663
https://github.com/openwrt/openwrt/pull/22663#issuecomment-4164086813
"num_turns": 33, "total_cost_usd": 0.6598452,
realtek: dsa: move configuration variables around#22438
https://github.com/openwrt/openwrt/pull/22438#issuecomment-4164059604
"num_turns": 15, "total_cost_usd": 0.41437409999999997,
It complained about a change in the PR which was actually not there
Older one:
realtek: stc8: use flexible array member#22506
https://github.com/openwrt/openwrt/pull/22506#issuecomment-4114848378
"num_turns": 8, "total_cost_usd": 0.24136304999999997,
Hauke
More information about the openwrt-devel
mailing list