---
title: "Inference-Healed Code Review Reward"
summary: "Replaces a binary tests-passed reward with a critic that scores correctness, style, performance, and security, explains low scores, and combines them"
status: "proposed"
category: "Feedback Loops"
url: "https://agentic-patterns.com/patterns/inference-healed-code-review-reward"
source: "https://www.youtube.com/watch?v=Xkwok_XXQgw"
authors: ["Nikola Balic (@nibzard)"]
tags: ["reward-modeling","code-review","inference-healing","quality-assessment"]
signals: ["RL coding agents pass tests but produce low-quality patches","The agent needs to know which quality aspect caused a low reward","Linters, benchmarks, and static analyzers are available to feed sub-scores"]
anti_signals: ["Test pass/fail fully captures what you care about","Reward latency and compute from extra checks is not acceptable","No labeled patch data to train or calibrate the critic"]
updated_at: "2026-10-13"
---

# Inference-Healed Code Review Reward

> Replaces a binary tests-passed reward with a critic that scores correctness, style, performance, and security, explains low scores, and combines them

## Use when

- RL coding agents pass tests but produce low-quality patches
- The agent needs to know which quality aspect caused a low reward
- Linters, benchmarks, and static analyzers are available to feed sub-scores

## Avoid when

- Test pass/fail fully captures what you care about
- Reward latency and compute from extra checks is not acceptable
- No labeled patch data to train or calibrate the critic

## Problem

Simple reward functions that only check for "all tests passed" fail to capture nuanced code quality issues (e.g., performance regressions, style violations, missing edge-case handling). A single binary signal at the end cannot guide the agent to produce maintainable, high-quality code.

- Verifying **only** final correctness misses suboptimal commits (e.g., a patch that removes error handling but still passes tests).
- Reward models that produce a single scalar lack **explainability** to tell the agent which aspect of the code needs improvement.

## Solution

Use an **inference-healed reward model**—a code-review critic that:

> Naming note: "inference healing" is this catalogue's coinage. The source talk's auto-captions render the phrase as "inference feeling," and the concept actually discussed is inference-time scaling for reward models that reason over sub-criteria rubrics. In the literature this family is called generative or rubric-based reward models.

The concrete instantiation below (linter/benchmark/security sub-scores with these weights and a 1-2B critic) is an extrapolation for code review, not something the cited work describes.

**1. Decomposes Code Quality into Subcriteria**
- **Correctness:** Does the code pass all existing and newly added tests?
- **Style:** Are linters (e.g., ESLint, pylint) satisfied (zero or minimal warnings)?
- **Performance:** Are there clear performance regressions gauged by simple benchmarks?
- **Security:** Does a static analyzer (e.g., Bandit, SonarQube) flag no critical issues?

**2. Runs Internal Chain-of-Thought (CoT) Reasoning**
- If uncertain about a subcriterion (e.g., performance), the critic runs a short CoT inside itself:
  ```text
  "Step: performance check. Baseline runtime: 50ms. New code runtime: 65ms.
  Regression > 20%. Score: 0.4."
  ```
- This "inference healing" allows the reward model to **explain** each sub-score.
- Use smaller critic models (1–2B parameters) to keep CoT generation cost-efficient.

**3. Aggregates Subscores**
- Each subcriterion returns a float ∈ [0, 1].
- A weighted sum (e.g., 0.4 × correctness + 0.2 × style + 0.2 × performance + 0.2 × security) yields the final code-review score.

**4. Generates Human-Readable Feedback**
- Alongside a numerical score, return a short analysis:
  ```json
  {
    "correctness": 1.0,
    "style": 0.8,
    "performance": 0.4,
    "security": 0.6,
    "comments": "Performance regression due to O(n²) loop."
  }
  ```

## Example

```python
# Pseudo-code for one code-review reward invocation
subscores = {
    "correctness": test_critic.score(patch),
    "style": linter_critic.score(patch),
    "performance": perf_critic.score(patch),
    "security": security_critic.score(patch),
}
final_score = sum(weight[k]*subscores[k] for k in subscores)
return final_score, subscores, comments
```

## How to use it

- **Critic Dataset Collection:** Gather examples of good vs. bad code patches, labeled along each subcriterion.
- **Critic Training:** Fine-tune a small LLM (e.g., 1–2 B parameters) to produce sub-scores and CoT justifications.
- **Integration into RL Loop:** Replace or augment the existing binary "tests-passed" reward with `inference_healed_reward(patch)`.
- **Selective Healing:** Generate CoT explanations only for subscores below a threshold (e.g., < 0.8) to reduce latency and cost.
- **Parallel Execution:** Run tests, linters, benchmarks, and security scans concurrently to reduce total evaluation time.
- **Human-in-the-Loop Checkpoints:** If a patch is borderline (e.g., final_score ∈ [0.5, 0.7]), route it for manual code review to generate better labels for future training.

## Trade-offs

- **Pros:**
  - **Explainable Feedback:** The agent knows *why* a patch scored poorly, allowing targeted improvements.
  - **Higher Code Quality:** Incorporates non-functional criteria (performance, security), leading to more robust code.
- **Cons/Considerations:**
  - **Compute Overhead:** Each reward invocation may involve running tests, linters, benchmarks, and a static analysis, adding latency.
  - **Critic Maintenance:** As coding standards or security rules evolve, retrain or update the critic models and rubrics.

## References

- Primary source: https://www.youtube.com/watch?v=Xkwok_XXQgw (Will Brown, Prime Intellect, "Open Questions in Agentic RL") - reward models as reasoners that break tasks into numerically scored sub-criteria rubrics; credits a recent DeepSeek paper.
- [Inference-Time Scaling for Generalist Reward Modeling](https://arxiv.org/abs/2504.02495) (DeepSeek-GRM, 2025) - the rubric-based generalist reward model line the talk credits.
- [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://arxiv.org/abs/2408.15240) (DeepMind, ICLR 2025) - verifiers that generate reasoning before scoring.
- "CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning" ([arXiv 2207.01780](https://arxiv.org/abs/2207.01780), NeurIPS 2022) - critic-based reward signals for code generation.

Distinct from [Anti-Reward-Hacking Grader Design](anti-reward-hacking-grader-design.md), which is about making graders hard to game, and from [CriticGPT-Style Code Review](criticgpt-style-evaluation.md), which trains a critique model for human-facing review rather than for an RL reward signal.