NO-ISSUE: Add long-duration test script and documents

[jianzhangbjz]

Add a Python script to run a long-duration test for the OLMv1. You can store the test results into the the sheet OLM Overall Performance Test Report of OLM Quality Assessment

jiazha-mac:long_duration jiazha$ python run_e2e.py --help
usage: run_e2e.py [-h] -b BINARY [-i INTERVAL] [-l LOGFILE] [-o] [-s] [-d DAYS] [-m MAX_RUNS] [--slack-webhook SLACK_WEBHOOK] [--min-pass-rate MIN_PASS_RATE]
                  [--anomaly-threshold ANOMALY_THRESHOLD] [--include-patterns INCLUDE_PATTERNS [INCLUDE_PATTERNS ...]]
                  [--exclude-patterns EXCLUDE_PATTERNS [EXCLUDE_PATTERNS ...]] [--suite SUITE] [--max-concurrency MAX_CONCURRENCY]
                  [--kube-burner-binary KUBE_BURNER_BINARY] [--kube-burner-args ...] [--check-olmv1-health] [--google-sheet-id GOOGLE_SHEET_ID]
                  [--google-sheet-name GOOGLE_SHEET_NAME] [--google-sheet-start-cell GOOGLE_SHEET_START_CELL] [--google-sheet-per-run]
                  [--google-sheet-per-run-name GOOGLE_SHEET_PER_RUN_NAME] [--google-credentials GOOGLE_CREDENTIALS] [--disable-fishers-test]
                  [--fishers-historical-window FISHERS_HISTORICAL_WINDOW] [--fishers-recent-window FISHERS_RECENT_WINDOW] [--fishers-alpha FISHERS_ALPHA]
                  [--enable-consecutive-failure-check] [--consecutive-failure-threshold CONSECUTIVE_FAILURE_THRESHOLD] [--must-gather-on-error]
                  [--must-gather-dir MUST_GATHER_DIR]

Trigger extended-platform-tests, olmv1-tests-ext, or kube-burner-ocp periodically

options:
  -h, --help            show this help message and exit
  -b BINARY, --binary BINARY
                        Path to the binary (e.g., /path/to/extended-platform-tests, /path/to/olmv1-tests-ext, or /usr/local/bin/kube-burner-ocp)
  -i INTERVAL, --interval INTERVAL
                        Interval between executions in seconds (default: 600 = 10 minutes)
  -l LOGFILE, --logfile LOGFILE
                        Path to log file (optional)
  -o, --once            Run only once and exit
  -s, --stats           Show test pass rate statistics and calculate average across runs
  -d DAYS, --days DAYS  Total duration to run in days (e.g., 5 for 5 days, 0.5 for 12 hours)
  -m MAX_RUNS, --max-runs MAX_RUNS
                        Maximum number of runs before exiting (default: unlimited)
  --slack-webhook SLACK_WEBHOOK
                        Slack Webhook URL for notifications (required for Slack alerts)
  --min-pass-rate MIN_PASS_RATE
                        Minimum pass rate threshold in percentage (default: 85.0). Only checked on first run
  --anomaly-threshold ANOMALY_THRESHOLD
                        Bayesian anomaly detection threshold in standard deviations (default: 3.0). Higher values = less sensitive
  --include-patterns INCLUDE_PATTERNS [INCLUDE_PATTERNS ...]
                        Test case patterns to include (default: 'OLM v1'). Only tests containing ALL of these strings will be selected. Example: --include-patterns high
                        olm jiazha
  --exclude-patterns EXCLUDE_PATTERNS [EXCLUDE_PATTERNS ...]
                        Test case patterns to exclude (default: DEPRECATED VMonly Stress Disruptive ChkUpgrade). Tests containing any of these strings will be excluded
  --suite SUITE         Test suite name for olmv1-tests-ext binary (e.g., 'olmv1/extended/candidate/parallel'). If not specified, tests are filtered by include/exclude
                        patterns
  --max-concurrency MAX_CONCURRENCY
                        Maximum number of tests to run in parallel (default: 6)
  --kube-burner-binary KUBE_BURNER_BINARY
                        Path to kube-burner-ocp binary. If specified, kube-burner-ocp will run in the background while executing the main binary
  --kube-burner-args ...
                        Extra arguments passed to kube-burner-ocp (only used with --kube-burner-binary)
  --check-olmv1-health  Check OLM v1 health before each test run. Stop and send Slack notification if OLM v1 is not healthy
  --google-sheet-id GOOGLE_SHEET_ID
                        Google Spreadsheet ID (from URL) to write test results
  --google-sheet-name GOOGLE_SHEET_NAME
                        Name of the sheet/tab in Google Spreadsheet (default: 'OLM Overall Performance Test Report')
  --google-sheet-start-cell GOOGLE_SHEET_START_CELL
                        Start cell for writing data (e.g., 'B30'). If not specified, appends to the end of the sheet.
  --google-sheet-per-run
                        Write per-run results to Google Sheets after each test run
  --google-sheet-per-run-name GOOGLE_SHEET_PER_RUN_NAME
                        Name of the sheet/tab for per-run raw results (default: 'Long-duration Test Results')
  --google-credentials GOOGLE_CREDENTIALS
                        Path to Google service account credentials JSON file (required if --google-sheet-id is specified)
  --disable-fishers-test
                        Disable Fisher's Exact Test for individual test case regression detection (enabled by default)
  --fishers-historical-window FISHERS_HISTORICAL_WINDOW
                        Fisher's Exact Test: number of historical runs for baseline (default: 20)
  --fishers-recent-window FISHERS_RECENT_WINDOW
                        Fisher's Exact Test: number of recent runs to compare (default: 10)
  --fishers-alpha FISHERS_ALPHA
                        Fisher's Exact Test: significance level before Bonferroni correction (default: 0.05)
  --enable-consecutive-failure-check
                        Enable consecutive failure detection for individual test cases (disabled by default)
  --consecutive-failure-threshold CONSECUTIVE_FAILURE_THRESHOLD
                        Alert if a test case fails N consecutive times (default: 3). Only used with --enable-consecutive-failure-check
  --must-gather-on-error
                        Run 'oc adm must-gather' when the program exits abnormally (errors, health check failures, etc.)
  --must-gather-dir MUST_GATHER_DIR
                        Directory to store must-gather output (default: ./must-gather)

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jianzhangbjz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• DOWNSTREAM_OWNERS [jianzhangbjz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kuiwang02]

/lgtm

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Review skipped — only excluded labels are configured. (1)

• do-not-merge/work-in-progress

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: f2e8f042-59ec-4d6f-9d0d-19be743a07a3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[jianzhangbjz]

/verified by @jianzhangbjz

[openshift-ci-robot]

@jianzhangbjz: This PR has been marked as verified by @jianzhangbjz.

Details

In response to this:

/verified by @jianzhangbjz

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jianzhangbjz]

/retitle NO-ISSUE: Add long-duration test script and documents

[openshift-ci-robot]

@jianzhangbjz: This pull request explicitly references no jira issue.

Details

In response to this:

Add a Python script to run a long-duration test for the OLMv1. You can store the test results into the the sheet OLM Overall Performance Test Report of OLM Quality Assessment

jiazha-mac:long_duration jiazha$ python run_e2e.py --help
usage: run_e2e.py [-h] -b BINARY [-i INTERVAL] [-l LOGFILE] [-o] [-s] [-d DAYS] [-m MAX_RUNS] [--slack-webhook SLACK_WEBHOOK] [--min-pass-rate MIN_PASS_RATE]
                 [--anomaly-threshold ANOMALY_THRESHOLD] [--include-patterns INCLUDE_PATTERNS [INCLUDE_PATTERNS ...]]
                 [--exclude-patterns EXCLUDE_PATTERNS [EXCLUDE_PATTERNS ...]] [--suite SUITE] [--max-concurrency MAX_CONCURRENCY]
                 [--kube-burner-binary KUBE_BURNER_BINARY] [--kube-burner-args ...] [--check-olmv1-health] [--google-sheet-id GOOGLE_SHEET_ID]
                 [--google-sheet-name GOOGLE_SHEET_NAME] [--google-sheet-start-cell GOOGLE_SHEET_START_CELL] [--google-sheet-per-run]
                 [--google-sheet-per-run-name GOOGLE_SHEET_PER_RUN_NAME] [--google-credentials GOOGLE_CREDENTIALS] [--disable-fishers-test]
                 [--fishers-historical-window FISHERS_HISTORICAL_WINDOW] [--fishers-recent-window FISHERS_RECENT_WINDOW] [--fishers-alpha FISHERS_ALPHA]
                 [--enable-consecutive-failure-check] [--consecutive-failure-threshold CONSECUTIVE_FAILURE_THRESHOLD] [--must-gather-on-error]
                 [--must-gather-dir MUST_GATHER_DIR]

Trigger extended-platform-tests, olmv1-tests-ext, or kube-burner-ocp periodically

options:
 -h, --help            show this help message and exit
 -b BINARY, --binary BINARY
                       Path to the binary (e.g., /path/to/extended-platform-tests, /path/to/olmv1-tests-ext, or /usr/local/bin/kube-burner-ocp)
 -i INTERVAL, --interval INTERVAL
                       Interval between executions in seconds (default: 600 = 10 minutes)
 -l LOGFILE, --logfile LOGFILE
                       Path to log file (optional)
 -o, --once            Run only once and exit
 -s, --stats           Show test pass rate statistics and calculate average across runs
 -d DAYS, --days DAYS  Total duration to run in days (e.g., 5 for 5 days, 0.5 for 12 hours)
 -m MAX_RUNS, --max-runs MAX_RUNS
                       Maximum number of runs before exiting (default: unlimited)
 --slack-webhook SLACK_WEBHOOK
                       Slack Webhook URL for notifications (required for Slack alerts)
 --min-pass-rate MIN_PASS_RATE
                       Minimum pass rate threshold in percentage (default: 85.0). Only checked on first run
 --anomaly-threshold ANOMALY_THRESHOLD
                       Bayesian anomaly detection threshold in standard deviations (default: 3.0). Higher values = less sensitive
 --include-patterns INCLUDE_PATTERNS [INCLUDE_PATTERNS ...]
                       Test case patterns to include (default: 'OLM v1'). Only tests containing ALL of these strings will be selected. Example: --include-patterns high
                       olm jiazha
 --exclude-patterns EXCLUDE_PATTERNS [EXCLUDE_PATTERNS ...]
                       Test case patterns to exclude (default: DEPRECATED VMonly Stress Disruptive ChkUpgrade). Tests containing any of these strings will be excluded
 --suite SUITE         Test suite name for olmv1-tests-ext binary (e.g., 'olmv1/extended/candidate/parallel'). If not specified, tests are filtered by include/exclude
                       patterns
 --max-concurrency MAX_CONCURRENCY
                       Maximum number of tests to run in parallel (default: 6)
 --kube-burner-binary KUBE_BURNER_BINARY
                       Path to kube-burner-ocp binary. If specified, kube-burner-ocp will run in the background while executing the main binary
 --kube-burner-args ...
                       Extra arguments passed to kube-burner-ocp (only used with --kube-burner-binary)
 --check-olmv1-health  Check OLM v1 health before each test run. Stop and send Slack notification if OLM v1 is not healthy
 --google-sheet-id GOOGLE_SHEET_ID
                       Google Spreadsheet ID (from URL) to write test results
 --google-sheet-name GOOGLE_SHEET_NAME
                       Name of the sheet/tab in Google Spreadsheet (default: 'OLM Overall Performance Test Report')
 --google-sheet-start-cell GOOGLE_SHEET_START_CELL
                       Start cell for writing data (e.g., 'B30'). If not specified, appends to the end of the sheet.
 --google-sheet-per-run
                       Write per-run results to Google Sheets after each test run
 --google-sheet-per-run-name GOOGLE_SHEET_PER_RUN_NAME
                       Name of the sheet/tab for per-run raw results (default: 'Long-duration Test Results')
 --google-credentials GOOGLE_CREDENTIALS
                       Path to Google service account credentials JSON file (required if --google-sheet-id is specified)
 --disable-fishers-test
                       Disable Fisher's Exact Test for individual test case regression detection (enabled by default)
 --fishers-historical-window FISHERS_HISTORICAL_WINDOW
                       Fisher's Exact Test: number of historical runs for baseline (default: 20)
 --fishers-recent-window FISHERS_RECENT_WINDOW
                       Fisher's Exact Test: number of recent runs to compare (default: 10)
 --fishers-alpha FISHERS_ALPHA
                       Fisher's Exact Test: significance level before Bonferroni correction (default: 0.05)
 --enable-consecutive-failure-check
                       Enable consecutive failure detection for individual test cases (disabled by default)
 --consecutive-failure-threshold CONSECUTIVE_FAILURE_THRESHOLD
                       Alert if a test case fails N consecutive times (default: 3). Only used with --enable-consecutive-failure-check
 --must-gather-on-error
                       Run 'oc adm must-gather' when the program exits abnormally (errors, health check failures, etc.)
 --must-gather-dir MUST_GATHER_DIR
                       Directory to store must-gather output (default: ./must-gather)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD c31c511 and 2 for PR HEAD f0c6148 in total

[openshift-ci]

@jianzhangbjz: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.