Plugin Directory: Add automated plugin review job#586
Open
obenland wants to merge 9 commits intoWordPress:trunkfrom
Open
Plugin Directory: Add automated plugin review job#586obenland wants to merge 9 commits intoWordPress:trunkfrom
obenland wants to merge 9 commits intoWordPress:trunkfrom
Conversation
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Core Committers: Use this line as a base for the props when committing in SVN: To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
There was a problem hiding this comment.
Pull request overview
Adds an automated (AI-assisted) review job to the Plugin Directory submission workflow, aiming to produce a structured verdict and findings for reviewer internal notes using a triage → batch → synthesis loop.
Changes:
- Introduces a new cron/AJAX-driven job (
Plugin_Automated_Review) that extracts the uploaded ZIP, batches source files, calls the WordPress AI Client API, and logs/stores results. - Wires the job into existing job infrastructure (colon-based cron hooks) and plugin upload flow, plus adds an admin metabox button to trigger reviews on-demand.
- Adds bundled prompt/reference markdown files, including a guidelines fallback when DevHub content can’t be fetched.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
jobs/class-plugin-automated-review.php |
Implements the multi-phase automated review loop, batching, AI calls, result storage, and audit logging. |
jobs/class-manager.php |
Registers the new colon-based cron task handler mapping and queues review on plugin_upload. |
jobs/automated-review/triage-prompt.md |
Triage system prompt template. |
jobs/automated-review/batch-prompt.md |
Batch review system prompt template + security/guideline rules. |
jobs/automated-review/synthesis-prompt.md |
Synthesis system prompt template for final report. |
jobs/automated-review/guidelines.md |
Bundled guidelines fallback when DevHub is unreachable. |
admin/metabox/class-review-tools.php |
Adds “Run Automated Review” button + AJAX call in the review metabox. |
admin/class-customizations.php |
Registers the new admin-ajax action for on-demand review runs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Introduces a 3-phase automated review system (triage → batch → synthesis) that reviews plugin submissions against directory guidelines, security checklists, and code quality standards using the WordPress AI Client API. - New `Plugin_Automated_Review` job class with cron and AJAX entry points - Bundled prompt references: security checklist, guidelines, 13 micro-prompts encoding experienced reviewer judgment rules - Integrates with existing Plugin Check (PCP) results for cross-referencing - Falls back to bundled guidelines when DevHub is unreachable - Enforces verdict consistency (blockers always mean reject) - Admin UI button gated to review-eligible statuses with proper nonce/cap checks - WPCS formatting cleanup in class-review-tools.php (short → long array syntax) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
obenland
force-pushed
the
add/plugin-directory-automated-review
branch
from
0c2e01a to
c87848e
Compare
- Pass slug args to wp_next_scheduled() so duplicate detection works - Check wp_supports_ai() in cron_trigger() and async AJAX path - Cast file_get_contents() returns to string to avoid TypeError with strict_types - Wrap AI client call in try/finally to ensure timeout filter is always removed Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Escape file paths in XML tag attributes with esc_attr() - Add untrusted-content warning to triage system prompt - Cast batch-prompt.md file_get_contents to string for strict_types safety Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix collect_files() to skip vendor/ and node_modules/ inside nested plugin directories (e.g., my-plugin/vendor/) not just at the root - Extract determine_verdict() as single source of truth for verdict logic, used by both the AI synthesis path and the fallback path - Remove unreachable verdict check (build_fallback_result always sets it) - Incomplete batch coverage now adds a warning finding so determine_verdict() naturally produces needs_changes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
obenland
force-pushed
the
add/plugin-directory-automated-review
branch
from
f76caa7 to
a32e8c5
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
obenland
force-pushed
the
add/plugin-directory-automated-review
branch
from
22f0274 to
bb7004e
Compare
- Use end() for ZIP attachment selection instead of max(array_keys()) for more reliable "newest attachment" semantics - Include vendor/node_modules paths in all_files list (still excluded from source_files) so structure analysis can detect dev artifacts - Pass full file list to batch prompts for disallowed-file detection - Rewrite synthesis prompt to request JSON output matching the schema, fixing mismatch with as_json_response() that could force fallback path - Restore leading backslash on use import to match file convention Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
obenland
force-pushed
the
add/plugin-directory-automated-review
branch
from
bb7004e to
ec81148
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…tches - Inject incomplete-coverage warning in run_review() before determine_verdict(), covering both AI synthesis and fallback paths - Remove duplicate incomplete-coverage logic from build_fallback_result() - Cast guidelines fallback file_get_contents to (string) - Fix batch prompt: replace per-guideline PASS/FAIL/WARN/N/A instruction with "only create findings for violations" to match JSON schema - Mark common-issues section as internal reference only to prevent fix suggestions from leaking into findings output Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
39 tests covering the deterministic logic methods: - determine_verdict(): blockers→reject, warnings→needs_changes, clean→approve - build_fallback_result(): severity bucketing, field defaults, multi-batch aggregation, unknown severity routing - normalize_file_priorities(): array-of-objects and map form handling - build_default_triage(): extension-based priority, PCP error promotion - build_batches(): skip filtering, oversized isolation, priority sorting, batch size splitting - collect_files(): vendor/node_modules skipping, extension filtering - summarize_pcp_results(): error/warning counting, formatted output - format_as_note(): verdict display, HTML output, finding rendering - format_pcp_for_file(): forward and reverse path matching - find_readme_content(): regex matching Pure-logic methods made public for direct testability. Uses Yoast PHPUnit Polyfills XTestCase for PHPUnit 10 compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
obenland
force-pushed
the
add/plugin-directory-automated-review
branch
from
411facf to
d245ca6
Compare
Catch missing or invalid attachment file paths early with an explicit check, rather than relying on Filesystem::unzip() to fail gracefully. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
obenland
force-pushed
the
add/plugin-directory-automated-review
branch
from
d245ca6 to
965c707
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
wp_supports_ai()is availablecd environments && npm run plugins:test🤖 Generated with Claude Code