--- title: "CriticGPT-Style Code Review" summary: "Runs a specialized critic model over generated code to find bugs, security flaws, and quality issues, and feeds critical findings back for regeneration" status: "emerging" category: "Reliability & Eval" url: "https://agentic-patterns.com/patterns/criticgpt-style-evaluation" source: "https://arxiv.org/abs/2407.00215" authors: ["Nikola Balic (@nibzard)"] tags: ["evaluation","code-review","critique","quality-assurance","bug-detection","gpt-4"] signals: ["High volume of AI-generated code overwhelms human reviewers","Security-sensitive code needs vulnerability checks before merge","CI/CD pipelines need automated pre-commit review"] anti_signals: ["Review depends on full business context only humans have","You cannot afford human verification of false positives","Low commit volume where human review is sufficient"] updated_at: "2026-10-13" --- # CriticGPT-Style Code Review > Runs a specialized critic model over generated code to find bugs, security flaws, and quality issues, and feeds critical findings back for regeneration ## Use when - High volume of AI-generated code overwhelms human reviewers - Security-sensitive code needs vulnerability checks before merge - CI/CD pipelines need automated pre-commit review ## Avoid when - Review depends on full business context only humans have - You cannot afford human verification of false positives - Low commit volume where human review is sufficient ## Problem As AI-generated code becomes more sophisticated, it becomes increasingly difficult for human reviewers to catch subtle bugs, security issues, or quality problems. Traditional code review processes may miss issues in AI-generated code because: - The volume of generated code can overwhelm human reviewers - Subtle bugs may appear correct at first glance - Security vulnerabilities may be non-obvious - Style and best practice violations may be inconsistent ## Solution Train or deploy a specialized critic model whose only job is finding problems in code output. CriticGPT, the reference instance, is a GPT-4-based critic trained with reinforcement learning from human feedback to write natural-language critiques of ChatGPT code answers. In the paper it supports human RLHF evaluators, who preferred its critiques over purely human-written critiques 63% of the time and found more real bugs with it. Two caveats travel with the result: the critic sometimes writes convincing critiques of bugs that do not exist, and human-plus-critic teams hallucinate fewer such bugs than the critic alone — so the human stays in the loop by design. A critic used this way can: 1. **Identify bugs** that human reviewers might miss 2. **Detect security vulnerabilities** in generated code 3. **Suggest improvements** for code quality and efficiency 4. **Verify correctness** of implemented solutions 5. **Check adherence** to coding standards and best practices **Scope note:** the published evidence covers critics assisting human evaluators on model outputs, not a standalone reviewer. Running such a critic as an autonomous CI/CD gate, as sketched below, is an extrapolation of the paper's results; treat its findings as input to human review rather than a verdict. For the broader technique of AI flagging issues during human review, see the sibling pattern [AI-Assisted Code Review / Verification](ai-assisted-code-review-verification.md) — the difference is that this pattern uses a model trained specifically for critique, while that one uses general-purpose review agents. ## Example ```python class CriticGPTReviewer: def __init__(self, critic_model, severity_threshold=0.7): self.critic = critic_model self.severity_threshold = severity_threshold def review_code(self, code, context=None, language="python"): """Comprehensive code review using specialized critic model""" reviews = { 'bugs': self.check_for_bugs(code, context, language), 'security': self.security_audit(code, language), 'quality': self.quality_review(code, language), 'performance': self.performance_analysis(code, language), 'best_practices': self.best_practices_check(code, language) } # Aggregate findings all_issues = [] for category, findings in reviews.items(): for issue in findings: issue['category'] = category all_issues.append(issue) # Sort by severity all_issues.sort(key=lambda x: x['severity'], reverse=True) return { 'issues': all_issues, 'summary': self.generate_summary(all_issues), 'recommended_action': self.recommend_action(all_issues) } def check_for_bugs(self, code, context, language): prompt = f""" Review this {language} code for bugs: Context: {context or 'General purpose code'} Code: ```{language} {code} ``` Identify any bugs including: - Logic errors - Off-by-one errors - Null/undefined reference errors - Type mismatches - Resource leaks - Race conditions - Edge case handling For each bug found, provide: 1. Line number(s) 2. Description of the bug 3. Severity (0-1) 4. Suggested fix """ response = self.critic.analyze(prompt) return self.parse_bug_findings(response) def security_audit(self, code, language): prompt = f""" Perform security analysis on this {language} code: ```{language} {code} ``` Check for vulnerabilities including: - SQL injection - XSS vulnerabilities - Command injection - Path traversal - Insecure cryptography - Hardcoded secrets - Authentication/authorization issues - Input validation problems Report format: [line, vulnerability type, severity, fix] """ response = self.critic.analyze(prompt) return self.parse_security_findings(response) def quality_review(self, code, language): prompt = f""" Review code quality for this {language} code: ```{language} {code} ``` Evaluate: - Code clarity and readability - Function/variable naming - Code organization - Documentation completeness - DRY principle violations - Coupling and cohesion - Error handling quality """ response = self.critic.analyze(prompt) return self.parse_quality_findings(response) def suggest_fixes(self, code, issue): """Generate specific fix for identified issue""" prompt = f""" Code with issue: ``` {code} ``` Issue: {issue['description']} Location: Line {issue['line']} Provide a corrected version of the relevant code section. """ return self.critic.generate(prompt) class IntegratedCodeGeneration: """Example of critic integration with code generation""" def __init__(self, generator, critic): self.generator = generator self.critic = CriticGPTReviewer(critic) def generate_and_review(self, task_description, max_iterations=3): # Initial generation code = self.generator.generate_code(task_description) for i in range(max_iterations): # Review generated code review = self.critic.review_code( code, context=task_description ) # If no critical issues, we're done critical_issues = [ issue for issue in review['issues'] if issue['severity'] > 0.8 ] if not critical_issues: break # Otherwise, regenerate with feedback feedback = self.format_feedback(critical_issues) refinement_prompt = f""" Original task: {task_description} Generated code has these issues: {feedback} Generate improved code addressing these issues: """ code = self.generator.generate_code(refinement_prompt) return { 'code': code, 'final_review': review, 'iterations': i + 1 } ``` ```mermaid sequenceDiagram participant User participant Generator as Code Generator participant Critic as CriticGPT participant Human as Human Reviewer User->>Generator: Request code for task Generator->>Generator: Generate initial code Generator->>Critic: Submit code for review loop Until code passes or max iterations Critic->>Critic: Analyze for bugs Critic->>Critic: Security audit Critic->>Critic: Quality review Critic->>Generator: Return issues found alt Critical issues found Generator->>Generator: Refine code based on feedback Generator->>Critic: Submit revised code else No critical issues Generator->>Human: Present code with review end end Human->>User: Approve/modify final code ``` ## Benefits - **Catches More Bugs**: Specialized training helps identify subtle issues - **Consistent Reviews**: No fatigue or oversight like human reviewers - **Fast Feedback**: Near-instantaneous review of generated code - **Learning Tool**: Helps developers understand potential issues - **Reduces Security Risks**: Proactive vulnerability detection ## Trade-offs **Pros:** - Scalable code review process - Consistent quality standards - Catches issues early in development - Can review code 24/7 without breaks - Improves over time with more training **Cons:** - May have false positives requiring human verification - Training specialized critic models is resource-intensive - Cannot understand full business context like humans - May miss novel vulnerability types - Requires integration into existing workflows - Risk of evaluator-model collusion in self-critique loops (mitigate with anchor sets and adversarial examples) ## How to use it - Use for automated code review workflows with high commit volumes - Deploy in CI/CD pipelines for pre-commit quality checks, with critic findings routed to a human as review input - Apply to security-sensitive code requiring vulnerability detection - Integrate via git webhooks (GitHub, GitLab) for event-driven review - Use 3-4 iterations of critique-refinement loops for complex changes - Mitigate collapse risks: decouple evaluation/generation prompts, benchmark against human-labeled anchor sets ## References - McAleese et al., [LLM Critics Help Catch LLM Bugs](https://arxiv.org/abs/2407.00215) (arXiv:2407.00215, June 2024, incl. Jan Leike) — the CriticGPT paper: RL-trained GPT-4 critic assisting human RLHF evaluators; 63% preference over human critiques, hallucinated-bug caveats. (The original `openai.com/research/criticgpt` announcement page is no longer available.) - [Constitutional AI: Harmlessness from AI Feedback (Anthropic, 2022)](https://arxiv.org/abs/2212.08073) - RLAIF foundation: replacing human annotation with AI feedback at lower cost - [Self-Taught Evaluators (Meta AI, 2024)](https://arxiv.org/abs/2408.02666) - Bootstrap critic models from synthetic data - [Evaluating LLMs for Code Review (2025)](https://arxiv.org/abs/2505.20206) - GPT-4o: 68.50% classification accuracy with context