Skip to content

ROX-32888: Use ACS bot token to create PR#115

Merged
mtodor merged 1 commit intomainfrom
mtodor/ROX-32888-use-acs-bot-token
Apr 20, 2026
Merged

ROX-32888: Use ACS bot token to create PR#115
mtodor merged 1 commit intomainfrom
mtodor/ROX-32888-use-acs-bot-token

Conversation

@mtodor
Copy link
Copy Markdown
Collaborator

@mtodor mtodor commented Apr 20, 2026

Description

Uses RHACS_BOT_GITHUB_TOKEN instead of the default GITHUB_TOKEN in the peter-evans/create-pull-request step of the model-evaluation workflow. PRs created with the default token do not trigger pull_request workflows (GitHub's design to prevent infinite loops), which meant required CI checks (build, test, e2e, style, smoke, wiremock-test) were not running on auto-generated evaluation PRs.

Also changes the weekly schedule from Monday to Tuesday at 6:00 UTC - to test trigger.

Validation

  • actionlint passes on the modified workflow file
  • Verify CI checks trigger on the next auto-generated evaluation PR (after merge)

AI-assisted development prompts

This PR was developed with AI assistance. Below are the prompts used:

Prompt: "Weekly job is running now, but we have a problem that created PR is not running tests. Here is an example: #114 - if you investigate that PR, you will see that required tests are not triggered. What should be done to fix that?"
Response: Identified the root cause: peter-evans/create-pull-request using the default GITHUB_TOKEN does not trigger pull_request workflows. Recommended using a PAT or GitHub App token instead.

Prompt: "Ok. I have token in secret: RHACS_BOT_GITHUB_TOKEN - use that"
Response: Added token: ${{ secrets.RHACS_BOT_GITHUB_TOKEN }} to the Create Pull Request step.

Prompt: "Let's switch to Tuesday - so that we can test it next day."
Response: Changed cron schedule from 0 6 * * 1 (Monday) to 0 6 * * 2 (Tuesday).

@mtodor mtodor requested a review from janisz as a code owner April 20, 2026 14:33
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Apr 20, 2026

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
361 2 359 12
View the full list of 2 ❄️ flaky test(s)
::policy 1

Flake rate in main: 100.00% (Passed 0 times, Failed 16 times)

Stack Traces | 0s run time
- test violation 1
- test violation 2
- test violation 3
::policy 4

Flake rate in main: 100.00% (Passed 0 times, Failed 16 times)

Stack Traces | 0s run time
- testing multiple alert violation messages 1
- testing multiple alert violation messages 2
- testing multiple alert violation messages 3

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 20, 2026

E2E Test Results

Commit: fa3d295
Workflow Run: View Details
Artifacts: Download test results & logs

=== Evaluation Summary ===

  ✓ list-clusters (assertions: 3/3)
  ✓ cve-detected-workloads (assertions: 3/3)
  ✓ cve-detected-clusters (assertions: 3/3)
  ~ cve-nonexistent (assertions: 2/3)
      - MaxToolCalls: Too many tool calls: expected <= 5, got 7
  ✓ cve-cluster-does-exist (assertions: 3/3)
  ~ cve-cluster-does-not-exist (assertions: 2/3)
      - ToolsUsed: Required tool not called: server=stackrox-mcp, tool=, pattern=list_clusters
  ✗ cve-clusters-general (assertions: 3/3)
      one or more verification steps failed
  ✓ cve-cluster-list (assertions: 3/3)
  ✓ cve-log4shell (assertions: 3/3)
  ✓ cve-multiple (assertions: 3/3)
  ✓ rhsa-not-supported (assertions: 2/2)

Tasks:      10/11 passed (90.91%)
Assertions: 30/32 passed (93.75%)
Tokens:     ~58232 (estimate - excludes system prompt & cache)
MCP schemas: ~12738 (included in token total)
Agent used tokens:
  Input:  14453 tokens
  Output: 22787 tokens
Judge used tokens:
  Input:  62504 tokens
  Output: 68610 tokens

@mtodor mtodor force-pushed the mtodor/ROX-32888-use-acs-bot-token branch from 1c6572e to fa3d295 Compare April 20, 2026 15:50
@mtodor mtodor merged commit 7bc6074 into main Apr 20, 2026
10 checks passed
@mtodor mtodor deleted the mtodor/ROX-32888-use-acs-bot-token branch April 20, 2026 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants