AI code review (Claude, maybe Codex)

Mon Apr 6 14:45:45 PDT 2026

On 4/6/26 19:33, JP wrote:
> On 6 Apr 2026, at 17:41, Hauke Mehrtens wrote:
> 
>> Hi,
>>
>> I added Claude code review recently to OpenWrt, see:
>> https://github.com/openwrt/openwrt/blob/main/.github/workflows/claude-code-review.yml
>>
>> This GitHub Action will review a PR when an OpenWrt member adds a comment with "/claude". This will trigger a GitHub Action which runs Claude Code to perform the code review.
>>
>>
>> Summary of the current manual test run
>> ======================================
>> I think this is helpful and it finds real problems. It is a bit nitpicky, but the false positive rate is pretty low.
>> It does not only look at the diff itself, but it starts an agent which also looks at the full file being modified. When a new DTS file is added, it often looks at similar DTS files and suggests making the new one look more like the existing ones. It also reads all existing comments on the PR.
>>
>> The review is pretty expensive; a PR review costs between $0.20 and $1.50, with an average price per review of $0.50. This depends on the number of turns it does. We can restrict the maximum number of turns, but it will reduce the quality of the review.
>> I am using the Claude API with their normal API pricing. The review is done using Claude Sonnet 4.6.
>>
>> The claude-code-action looks like beta software to me. We are using a fork because the normal version does not support reviews of PRs from forked repositories. This was fixed in version 1.0.49, but then reverted due to another problem. It is also not well documented what permissions it needs and how to configure it correctly.
>> Upstream is already 12 versions ahead of my two-week-old fork.
>> https://github.com/hauke/claude-code-action/tree/v1.0.77-fixed
>>
>> Here is the full list of PRs where it ran:
>> https://github.com/openwrt/openwrt/actions/workflows/claude-code-review.yml?query=is%3Asuccess
>>
>>
>> Current problems
>> ================
>> We do not have enough reviewers for all the PRs we receive. Many of the PRs adding new devices have the same problems. AI can catch most of them, so we no longer have to handle them manually. I hope AI can help us reduce the load on PR reviews, increase the quality of PRs, and also increase the number of PRs we will merge. If we activate it to run directly after PR creation, contributors will also get fast feedback.
>>
>> I think AI would have found the recently discovered and fixed security problems in mdnsd and LuCI already when the PR was created.
>>
>>
>> Open Source sponsoring
>> ======================
>> I have applied for the Open Source maintainers programs from Claude and OpenAI Codex:
>> https://developers.openai.com/community/codex-for-oss
>> https://claude.com/contact-sales/claude-for-oss
>> (The form for Claude does not show for me in Firefox.)
>>
>> OpenAI Codex is similar to Claude Code as far as I understand.
>> The Claude program only offers their Claude Max plan, but this does not work with CI pipelines. We actually need API credits. The OpenAI Codex program looks like it matches our use case.
>>
>> I also applied for Codex Security, which should be able to find and fix security problems.
>>
>>
>> Future plan
>> ===========
>> I want to run a good AI agent on all PRs automatically when they are opened and when they are updated. This should apply to all repositories under the OpenWrt GitHub organization. If we were to fully pay for Claude ourselves, this would cost around $200 per month. I hope to get some sponsorship from Anthropic and/or OpenAI to make this happen.
>>
>> Once we have that, I would first integrate it into our main repository and then into LuCI.
>>
>> Based on the code review results and what it missed or wrongly suggested we should create documentation the AI can use to improve further reviews. We should add a CLAUDE.md and AGENTS.md or a special version for code reviews which we point the AI to.
>>
>>
>> How do other Open source projects handle this? Do they have a summary of what they do and how well it works for them?
>>
>> Does someone know other alternatives to Anthropic Claude Code and OpenAI Codex? Does someone know other ways to get sponsorship for this?
>>
>> Do you agree on running this automatically on all PRs?
>>
>>
>> Feel free to try it yourself by adding a comment with /claude to a PR. It works for all people with commit permission.
>>
>>
>> Example Reviews
>> ===============
>> Here are some recent examples. The num_turns is the number of turns the AI used for the review. It ran grep to look for similar files, or requested more context, for example. total_cost_usd is the cost of the action in USD, excluding taxes.
>>
>> Sysupgrade fixes, including related to apk#17847
>> https://github.com/openwrt/openwrt/pull/17847#issuecomment-4190389607
>>    "num_turns": 11, "total_cost_usd": 0.2514354
>>
>> generic: net: phy: realtek: add 5G and 10G PHY support#22563
>> https://github.com/openwrt/openwrt/pull/22563#issuecomment-4188211274
>>    "num_turns": 34, "total_cost_usd": 1.01998215,
>>
>> Fix setup of non-80MHz 802.11s mesh interfaces on EHT hardware#22644
>> https://github.com/openwrt/openwrt/pull/22644#issuecomment-4185594305
>>    "num_turns": 14, "total_cost_usd": 0.23388720000000002,
>>    It somehow stopped after creating an example patch description which I requested in an other comment.
>>
>> netifd/odhcp6c: convert DHCP protocol handlers to ucode#22751
>> https://github.com/openwrt/openwrt/pull/22751#issuecomment-4184960743
>>    "num_turns": 20, "total_cost_usd": 0.4492875,
>>
>> ramips: mt7621: fix reset hang and re-enable LIST_HARDENED#22724
>> https://github.com/openwrt/openwrt/pull/22724#issuecomment-4183374970
>>    "num_turns": 14, "total_cost_usd": 0.38940090000000005,
>>
>> mediatek: filogic: add support for Hilink RM65 development board#22683
>> https://github.com/openwrt/openwrt/pull/22683#issuecomment-4164682965
>>     "num_turns": 54, "total_cost_usd": 0.6598969499999999,
>>
>> mediatek: add Huasifei WH3000 Pro NAND support#22694
>> https://github.com/openwrt/openwrt/pull/22694#issuecomment-4164295740
>>    "num_turns": 26, "total_cost_usd": 0.48784635000000004,
>>    It did a incorrect suggestion to change the SPDX-License header
>>
>> realtek: i2c: rtl9300: backport rtl9607c i2c support and speed patches#22663
>> https://github.com/openwrt/openwrt/pull/22663#issuecomment-4164086813
>>    "num_turns": 33, "total_cost_usd": 0.6598452,
>>
>> realtek: dsa: move configuration variables around#22438
>> https://github.com/openwrt/openwrt/pull/22438#issuecomment-4164059604
>>    "num_turns": 15, "total_cost_usd": 0.41437409999999997,
>>     It complained about a change in the PR which was actually not there
>>
>> Older one:
>> realtek: stc8: use flexible array member#22506
>> https://github.com/openwrt/openwrt/pull/22506#issuecomment-4114848378
>>    "num_turns": 8,  "total_cost_usd": 0.24136304999999997,
>>
>> Hauke
> 
> hi
> 
> I'd like to start this out with an immediate acknowledgement of my minimal involvement in owrt dev at this stage, so my opinion here may count for cents. I'd like to follow that up by saying that my involvement has been because of life-limiting factors (a problem which we all know), and I've been managing to reduce those somewhat
> 
> that said: this mail/action was an extremely dismaying thing to see

We already did 50 reviews using Claude code in the last 2 weeks, it 
automatically added a comment to the PRs.

> dimensions of why:

AI is a highly polarizing topic in some groups. Keep that in mind when 
reading about it. Also AI != AI, there are a lot of different use sages 
and systems.

>   - these platforms are subsidised (in the extreme) by (provably society-damaging) VC-funds; any attempt at building infrastructure upon this without significant review/planning/estimation strikes me as potentially high risk

Isn't this good? OpenWrt can profit from these VC-funds.

As far as I understand the LLM business model, the taring of new LLMs is 
very expensive. I think most providers make money with executing them 
over their APIs, but they do not make enough money to finance the 
training. The companies are private and I do not know their financial 
details. This is just an addition and we would not depend on it. Github 
is a similar problem, GitHub actions would already cost $1k just for our 
usage in April this year till now.

>   - damned near _every_ quantitative study observing reliance upon these systems have found them to be a cognitive net-negative for those relying upon them

Currently not enough people do PR reviews, so it is open for you to 
train your brain.

>   - the (very obvious) aspect of _everything_ about these systems being built on a strip-mined commons (with no intent to contribute back);
>     - the intent of this is not to get into litigating the thesis and dimensions of open source, but simply to indicate: _these companies do not give a single fuck_

At least the two mentioned companies have an Open source maintainers 
sponsoring. There are also companies making billions with products based 
on OpenWrt without contributing much like many SoC vendors.

> what is your proposition for how the openwrt project should handle "review feedback" with model-generated code (with the copyright licensing implications therein)?

Normally these are very small snippets which probably do not really 
cover copyright. At least in the US it looks like LLMs are covered by 
fair use.

> how do you propose handling the inevitable iteration-loop-increase that the "oh, openwrt reviews PRs with models, it'll be fast now" dynamic is almost certain to result in?
> 
> I note in your mail that you indicate that there has been a lack of reviwers, so I want to ask:
> 
>   - what type of review is required?
>   - what expectations of competency/exposure are wanted?
>   - what shape of CTA (call to action) has been attempted to find more/applicable reviewers?
> 
> since I am the one sending this mail, I will outright offer myself as tribute: I'm down to take on some more review capacity. I've got a bit of smarts and clue all over, and where I don't know I ask/learn

We tried to invite more people, but there are not enough people doing 
reviews, just look at some of the 917 open PRs which interest you and 
look if they look fine and maybe test them and report the results back.
https://github.com/openwrt/openwrt/pulls

Currently we have 917 open PR for the main repository.

We had a discussion about no review happening in over 1 year some time 
ago, see 
https://lists.openwrt.org/pipermail/openwrt-devel/2025-November/044326.html 
There were other discussions mostly started by people complaining that 
their open PR got new reaction for a long time.

We also did not see the buffer overflow added in:
https://github.com/openwrt/mdnsd/pull/13/changes
I asked Claude code to do a review and it found CVE-2026-30871 and other 
problems in it.

Hauke