fix(automate): name App Automate sessions per test instead of only at worker teardown (SDK-7270) - #131
fix(automate): name App Automate sessions per test instead of only at worker teardown (SDK-7270)#131anish353 wants to merge 2 commits into
Conversation
… worker teardown (SDK-7270) Since 9.27 the SDK self-bootstraps the platform binary, so BrowserstackCLI.isRunning() is true on every run and service.ts skips the legacy per-test rename. Ownership moved to automateModule, which records the per-test name in sessionMap but issues no PUT until onAfterExecute -- reached only from service.ts's after() hook, i.e. once at worker teardown. Every session's name therefore depended on a single event at the very end of the worker. Suites that reload the session per test have already closed those sessions by then, and a worker that never reaches after() (interrupted run, hard exit, crash) never fires onAfterExecute at all -- leaving every session on its creation-time sessionName capability. Name the session from onBeforeTest, while it is still the live session, via a new flushSessionName() helper de-duped on SessionData.appliedName. onAfterExecute now calls the same helper, becoming a final sweep that no-ops for sessions already named. Restores the pre-9.27 per-test timing with no extra API calls in the steady state. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
RUN_TESTS |
|
🔴 SDK PR Review gate is red. Pending:
It turns green once the latest SDK PR Review Agent run reports GTG on the current head commit. A native reviewer approval is separately required by branch protection before merge. |
|
[SDK Wdio Test] TRA build state: failed | Stability 98% — verdict: success. Passed: 85, Failed: 2, Aggregate: 87. TRA: https://observability.browserstack.com/builds/falsds1jgm1yqrnixtx9fionzmyyxrj9lph4jinf |
anish353
left a comment
There was a problem hiding this comment.
Claude Code Review (automated) — 5 inline finding(s). Full report in the PR comment below. Verdict: Passed.
| user: this.config.userName as string, | ||
| key: this.config.accessKey as string | ||
| }) | ||
| sessionData.appliedName = name |
There was a problem hiding this comment.
[Medium] This records attempted, not applied — so a failed PUT disarms the final sweep
markSessionName never checks response.ok (:274-276 — await fetch(...) → await response.json() → debug-log, no status check) and swallows everything in its catch (:277-279). A 401/404/429/5xx is indistinguishable from success. This line then sets appliedName unconditionally, so the teardown sweep at :220-221 no-ops. The field comment at :28 says "last name successfully PUT" — something the code can't actually know.
Failure scenario: the reload-per-test shape this PR targets puts one test per session. Test A's onBeforeTest PUT hits a 429 or a socket blip → appliedName = "A" → sweep suppressed → that session permanently keeps its creation-time sessionName capability. That's the exact bug this PR fixes, reintroduced under transient failure — while the new "Final sweep" comment advertises a guarantee it no longer provides. It also compounds with the per-test PUT volume: more requests make rate-limiting more likely.
Suggestion: have markSessionName return response.ok and set appliedName only on 2xx. Alternatively, make the teardown sweep unconditional and de-dupe only the per-test calls — that costs one extra PUT per session, which is exactly what main does today, so it's a safe fallback.
Reviewer: fallback independent reviewer
| // that never reaches `after()` — interrupted run, hard exit, crash — never fires | ||
| // onAfterExecute at all, leaving every session on the creation-time `sessionName` | ||
| // capability. Restores the pre-9.27 behaviour, where the rename was issued per test. | ||
| await this.flushSessionName(sessionId) |
There was a problem hiding this comment.
[Medium] Skipped tests also fire TEST/PRE, so a hook failure fans this out into hundreds of blocking PUTs
TEST/PRE isn't fired only by service.ts:498. skipReporter.ts:58 awaits framework.trackEvent(TEST, PRE, …) for every skipped test, and reportSuiteSkipped (:72-92) walks every remaining test and nested suite calling reportSkippedTest when a before all / beforeEach hook fails — all inside the awaited afterHook. Each of those now issues a blocking rename PUT, and because the titles are distinct the appliedName de-dupe collapses none of them.
Failure scenario: a failed before all in a 300-test suite adds ~300 sequential HTTPS PUTs to teardown — roughly a minute at typical latency — inside an awaited WDIO hook. Statically-skipped tests also transiently rename the live session to a test that never ran.
UNVERIFIED: whether this trips the WDIO afterHook timeout — I didn't confirm the configured value. If it's in the tens of seconds, this escalates to High, so worth a check on your side.
Suggestion: pass a marker in reportSkippedTest's TEST/PRE args (e.g. skipped: true) and skip the flush for it; or make the per-test PUT fire-and-forget and let the awaited sweep stay authoritative.
Reviewer: fallback independent reviewer
| } | ||
|
|
||
| const name = sessionData.lastTestName | ||
| await this.markSessionName(sessionId, name, { |
There was a problem hiding this comment.
[Medium] Awaited network round trip now on the per-test hot path, with no timeout
Under Mocha with defaults name embeds the test title (:80-85), so lastTestName changes every test and the de-dupe never suppresses. A conventional suite — one session, N tests, no reload — goes from 1 PUT at teardown to N PUTs, each awaited before the test body runs. _fetch (fetchWrapper.ts:19-35) sets no timeout, no retry, and no abort signal, so a slow or hanging API stalls every test start. At ~150–300 ms that's roughly 1–2.5 min added per worker on a 500-test suite.
This also makes the PR body's "no extra API calls in the steady state" inaccurate for the non-reload case — it holds only when consecutive titles repeat.
Mitigating, and worth saying: this is parity with the legacy non-CLI path — service.ts:503 awaits _setSessionName per test, de-duped via _fullTitle at :1006. So it's restored prior behavior, not a new design.
Suggestion: bound it with AbortSignal.timeout(~5000), or make the onBeforeTest flush fire-and-forget and keep the teardown sweep authoritative.
Reviewer: fallback independent reviewer
| */ | ||
| private async flushSessionName(sessionId: string): Promise<void> { | ||
| const testContextOptions = this.config.testContextOptions as TestContextOptions | ||
| if (testContextOptions.skipSessionName) { |
There was a problem hiding this comment.
[Low] Unguarded testContextOptions deref (pre-existing pattern, reproduced here)
testContextOptions is genuinely reachable as undefined: unConfigureModules() (cli/index.ts:427) re-calls configure() with no 4th arg so config defaults to {}; setConfig swallows a parse failure leaving {}; and the binary returns a degenerate config on auth failure. The as TestContextOptions cast hides all of that from the compiler.
No new exposure — :66 already derefs before this line is reachable from onBeforeTest, and the onAfterExecute path derefs inside a try/catch in both old and new code. So this isn't a regression; it just reproduces the pattern where ?. would cost nothing.
Note this guard is also unreachable from onBeforeTest specifically (:66 has already returned by then) — it's live and correct only on the onAfterExecute path.
Reviewer: fallback independent reviewer
| "@wdio/browserstack-service": patch | ||
| --- | ||
|
|
||
| - Fixed App Automate and Automate session names staying on the static `sessionName` capability instead of the test title, for suites that reload the session between tests or whose run ends before the WebdriverIO `after` hook. |
There was a problem hiding this comment.
[Medium] The reload clause is contradicted by this PR's own evidence
The note credits the fix to "suites that reload the session between tests or whose run ends before the WebdriverIO after hook." Row 1 of your own evidence table disproves the first clause: 9.33.1 (shipped code), after() reached → 3 renames, correct per-test titles — on a build you describe as the customer's exact reload-per-test shape.
The mechanism agrees: onReload refreshes KEY_FRAMEWORK_SESSION_ID (service.ts:809-812), so each reloaded session already got its own sessionMap entry and the old sweep already PUT a name for every one. Reload alone was never a failure mode. The real cause is solely the second clause — the worker never reaching after() (service.ts:584 is the only EXECUTE/POST trigger; the customer log shows EXECUTE trackEvent = 0).
This matters because it's the customer-facing release note: users who reload per test but do reach after() would read this as fixing something that was never broken for them.
Suggestion: drop the reload clause here and in the internal note. Or substantiate it — it would hold only if the Automate REST API refuses to rename a terminated session, which is worth confirming either way since it's a useful fact.
Reviewer: fallback independent reviewer
Claude Code PR ReviewPR: #131 • Head: 0986e13 • Reviewers: fallback independent reviewer — this repo has no SummaryMoves the Automate/App-Automate session rename from the single The diagnosis is correct and the mechanism is sound. The PR's evidence table isolates the defect cleanly, and the two things most likely to make this change dangerous both came back clean under direct verification — there is no off-by-one in the name, and a failed PUT cannot break a customer's test. Five Medium findings, three Low. None is High or Critical. The Mediums are robustness, blast radius, and release-note accuracy — not broken logic. Review Table
Findings1.
|
|
🔴 SDK PR Review gate is red. Pending:
It turns green once the latest SDK PR Review Agent run reports GTG on the current head commit. A native reviewer approval is separately required by branch protection before merge. |
What is this about?
App Automate session names show the static
sessionNamecapability instead of the per-test title. This is the follow-up to #100 — that fix shipped in 9.33.1 and the customer still reproduced on it.Why #100 didn't resolve it. #100 fixed endpoint routing (
isAppAutomate()now detectsappium:app, so the PUT targets/app-automate/...instead of 404-ing against/automate/...). That fix is present and working — the customer's 9.33.1 log shows the correct App-Automate endpoint in use. But routing only matters once a rename is actually issued, and in this customer's shape none ever is.Actual cause. Since 9.27 the SDK self-bootstraps the platform binary, so
BrowserstackCLI.getInstance().isRunning()is true on every run. That flips session naming off the legacy per-test path:service.tsskips the legacy rename entirely when the CLI runs —beforeSuiteis guarded (service.ts:385),beforeTestearly-returns before_setSessionName(service.ts:495-500), and_setSessionName's_updateJob({ name })is itself guarded by!isRunning()(service.ts:1006-1009). Customer log:Update job with sessionId= 0.automateModule, which records the per-test name insessionMapbut issues no PUT untilonAfterExecute— reached only fromservice.ts:584inside WDIO'safter()hook, i.e. once, at worker teardown.So every session's name depends on a single event at the very end of the worker. Suites that reload the session per test (the customer's does — 28
Session Reloadedin the captured run) have already closed those sessions by then, and a worker that never reachesafter()— interrupted run, hard exit, crash — never firesonAfterExecuteat all. Customer log confirms:trackEvent: automationFrameworkState=…EXECUTE= 0, ending atHandling CLI cleanup in exit handler. Result: zero renames, every session keeps its creation-timesessionNamecapability.Change. Name the session from
onBeforeTest, while it is still the live session, via a newflushSessionName()helper de-duped onSessionData.appliedName.onAfterExecutenow calls the same helper, becoming a final sweep that no-ops for sessions already named. Restores the pre-9.27 per-test timing; no extra API calls in the steady state.Verified on real App Automate builds (
appium:appcapability,reloadSession()per test — the customer's exact shape):after()reachedafter()not reached"STATIC CAP NAME…"← reported symptomafter()not reachedThe first row isolates the defect: on shipped code the rename works only if the worker reaches
after().Full vitest suite is identical to the clean-tree control (same 7 files / 70 pre-existing failures with and without this change — zero new);
automateModule.test.ts31/31; build + eslint clean.Related Jira task/s
https://browserstack.atlassian.net/browse/SDK-7270 (clone of https://browserstack.atlassian.net/browse/SDK-7093)
Release (mandatory for every PR — required for the
ready-for-reviewlabel)Version bump: (required — tick exactly one)
Release notes type: (optional)
Release notes (customer-facing): (optional but encouraged)
sessionNamecapability instead of the test title, for suites that reload the session between tests or whose run ends before the WebdriverIOafterhook.Release notes (internal): (required — engineer-facing; what actually changed / why)
automateModuleissued the session-name PUT only fromonAfterExecute(WDIOafter(), once per worker at teardown). Sessions reloaded per test were already closed by then, and a worker that never reachesafter()never named anything.flushSessionName()now fires fromonBeforeTestwhile the session is live, de-duped viaSessionData.appliedName;onAfterExecutereuses it as a no-op final sweep.Checklist
PR Validations
Run Tests: Comment RUN_TESTS to trigger sanity tests.