chore: pin FluidAudio 0.15.4 → 0.15.5 (#122) - #142
Conversation
Unblocks #123 / #124 — the Parakeet Unified and TDT-ja models exist only in 0.15.5. #110 flagged DownloadUtils → ModelHub as breaking. It is not breaking for this repo: the rewrite is internal to the download layer and this repo has zero references to it. Both questions the issue left open were answered by compiling rather than by reading release notes — the AsrModels signatures are unchanged, and the Chinese-family managers used here are not the experimental zh-CN CTC / Qwen3 backends dropped in 0.15.3. Accuracy is unchanged: every overlapping before/after pair matches to the precision the report prints, across parakeet, sensevoice and paraformer. Throughput moved, but those are single unaveraged runs and are not evidence either way. Carrying #110's residue: the schema records app_version but not the FluidAudio version, so the two batches are indistinguishable in the store once written and are separated here only by capture time. Refs #122
0b736de to
bd35b9c
Compare
Verify Report — PR #142Enginemanual fan-out (3 lens Agents, model: opus, file-based output) + Codex ( Two deliberate deviations from the default ensemble, both stated rather than silently applied:
Diff-freshness gate (#228): PASS — frozen Process GapsThe Devil's Advocate leg did not run. Two attempts, neither produced output:
Per the skill's second-idle fallback, the coordinator performed the adversarial review in its place and the substitution is recorded here rather than absorbed. This is a lower-quality substitute and should be read as one: the coordinator had already read all four findings files before reviewing, so it lacks the cognitive isolation the role depends on — an adversarial pass works because it does not share the other reviewers' framing, and this one does. The coordinator's contributions are marked Two findings that a real adversarial pass would likely have interrogated further, and did not get to: whether the six bestASR versions between the two batches contain anything that touches the Working-tree contention. One lens ran a before/after warning comparison by flipping AggregateFAIL — 2 blocking, 6 follow-up. The dependency change itself is clean; the security review is a substantive PASS with rigorous method. What fails is the PR's account of itself: three of its factual claims are false as written, and its headline evidence cannot support its headline conclusion. Requirements coverage — issue #122
Findings (merged, deduped; the stacking effect is collapsed — see the note below the table)
On the stacking effect. Three lenses plus the cross-model leg converged hard on "the documentation is false", which makes the count look larger than the problem. The genuinely distinct claims are collapsed above: "models only in 0.15.5", "signatures unchanged" and "consumed surface entirely high-level" are one finding (F2) reported by three sources, not three findings. The evidence problems, by contrast, really are separate — F1(a) is provenance and F1(b) is coverage, and closing one leaves the other open. Adversarial adjudication
|
| corpus | backend | 2026-07-05/06 (app 0.10.0) | 2026-08-02 (app 0.16.0) |
|---|---|---|---|
59dfb9a4acb3 (jfk) |
parakeet | 0.0000 % | 0.0000 % |
29007d2e6418 |
parakeet | 93.5484 % | 93.5484 % |
33dd468f27c6 |
sensevoice | 14.0845 % | 14.0845 % |
af3571b78130 |
sensevoice | 47.2222 % | 47.2222 % |
29007d2e6418 |
sensevoice | 11.2903 % | 11.2903 % |
33dd468f27c6 |
paraformer | 178.8732 % | 178.8732 % |
af3571b78130 |
paraformer | 181.9444 % | 181.9444 % |
This is stronger than the CHANGELOG's own table implies. The cross-model leg's objection — that two differently-rounded numbers cannot establish equality — is correct about what the table shows and wrong about what the data holds.
B. The 2026-07-19 batch (app 0.14.0, 24 rows) shares zero corpora with the after-batch.
Its corpus IDs (12669b3d0de8, 604acbc2d096, c4120b0b7d67, …) and the after-batch's (59dfb9a4acb3, 33dd468f27c6, af3571b78130, 29007d2e6418) do not intersect at all. So the choice of the older, smaller 07-05/06 batches as "before" was forced by overlap, not carelessness — a fact that reads as an implicit accusation in one lens report and should not.
Adjudication. A defuses part of F1(a) and does nothing for F1(b).
The bit-identity is real evidence that the decode paths those corpora exercise are deterministic and unmoved — across six bestASR versions as well as across the FluidAudio bump. So "the comparison confounds six bestASR versions" overstates the practical damage: the confound existed and demonstrably did not materialise. On that basis I record F1 at HIGH rather than CRITICAL, which is a downgrade from the requirements lens's rating.
But it rescues nothing on F1(b), and arguably sharpens it. If the changed branch cannot fire on single-chunk audio or on a language without case, then bit-identity on exactly those corpora is the predicted outcome whether or not 0.15.5 changed anything — it is not evidence about the change, it is evidence about the absence of the change's preconditions. Identity where the code cannot differ says nothing about where it can.
So both statements are true at once, and the honest form of the claim is narrower than either the PR's or the harshest lens reading:
On the seven overlapping corpus/backend pairs, error rates are bit-identical between the 2026-07-05/06 baseline (bestASR 0.10.0) and the post-bump sweep (bestASR 0.16.0). None of those pairs exercises 0.15.5's new seam-duplicate collapse, which requires multi-chunk audio containing case-differing Latin text; the English multi-chunk corpora that would are on disk and were not measured. No pre-upgrade sweep was run, and the store cannot attribute a row to a FluidAudio version.
On the caveat (F4) I also differ from the requirements lens, which rated it CRITICAL partly on the reading that it conceals the real confound. The sentence is false about the pair it names and that is worth correcting, but it is false in the direction of understating the author's own diligence gap rather than overstating a quality claim. MEDIUM.
What the security review establishes (and it is substantial)
Recorded because a FAIL verdict should not flatten a genuinely clean result on the axis where this PR is actually risky.
The pin is correct and the supply-chain posture is unchanged or better. Tag → revision was cross-checked against the GitHub API rather than the local mirror alone, on the stated reasoning that a mirror fetched by this machine would faithfully reproduce upstream tampering — 19600a48… is the real v0.15.5. swift package resolve against the committed tree is a no-op, so the lockfile is what SwiftPM computes rather than a hand edit. Across a large upstream delta (41 commits, 156 files, a wholesale rewrite of the download layer), there are no new transitive dependencies (FluidAudio declares none, and its Package.swift is byte-identical between tags), no network-endpoint change (ModelRegistry is byte-identical), no TLS-verification weakening, no new credential sink, no telemetry — and the offline-egress gate got stronger, gaining per-request re-checks inside FileDownloader and HFTreeLister.
Weight pins were confirmed empirically rather than assumed: hashing the live cache against the committed manifest gives 42/42 files clean across all three pinned repos. And the PR is correct not to re-pin, which is worth stating so a later reviewer does not "fix" it reflexively — re-running the pin script here would have TOFU-laundered any drift, and the check above shows there was none to record. Leaving the manifest untouched is what preserved the ability to detect drift.
Scope check
Clean. Three files, +43/−4. The re-cut that moved Package.swift / Package.resolved here from PR #140 left no inconsistency: CHANGELOG text, manifest pin and lockfile revision now agree, the ### Changed block sits correctly in the unreleased section, and nothing from the old split leaked through. The blocker relationship to #110 is genuinely lifted — ModelGrid still carries only 0.6b-v3, which is correct, since adding tdt-ja / unified rows is #123/#124's work, and an unmapped grid model throws loudly rather than falling back silently.
Next
F1 and F2 gate the merge. F1(b) closes at the cost of a single benchmark run on one English multi-chunk corpus (osr-harvard-1, 33 s); F1(a) closes either by checking out 0.15.4 for a real before-sweep, or by relabelling the table with both app versions and capture dates and weakening the conclusion to the honest form above. F2 is three sentence-level corrections to the PR body and CHANGELOG.
Verify was run at bd35b9c; re-run after the blocking findings are addressed — and note that the Devil's Advocate leg owes this PR a pass it did not get.
…122) Verification found the dependency change itself clean and the entry describing it wrong in four places. The pin, the lockfile revision and the supply-chain posture are unchanged by this commit; only CHANGELOG.md is. The accuracy evidence could not support its conclusion. 0.15.5 adds seam-duplicate collapse to the Parakeet TDT chunk merger, and that branch fires only on audio longer than one ~14.96s chunk AND only when the seam duplicate is case-differing Latin text. The two parakeet rows offered as proof were jfk (11.0s, single chunk, the branch is never entered) and a Chinese corpus (no case, so the predicate can essentially never hold). Both are structurally blind to the only behavioural change on that path: identical numbers there were the predicted outcome whether or not anything changed. Measured on a corpus that CAN exercise it -- osr-harvard-1, 33.6s of English -- by building the same tree against each pin in turn: 0.15.4 (b9d43724) osr-harvard-1 parakeet WER 0.037500 0.15.5 (19600a48) osr-harvard-1 parakeet WER 0.037500 Bit-identical, not equal-after-rounding. The 0.15.4 side was confirmed genuinely 0.15.4 by the ABSENCE of collapseSeamWordDuplicates from the checked-out source, which is stronger than the resolved version string: a stale build artifact can survive a pin change, but code that is not on disk cannot run. The earlier table was not a before/after comparison at all and is removed rather than repaired. Its "0.15.4" column reconciles to store rows from 2026-07-05/06 under app_version 0.10.0; its "0.15.5" column is the 2026-08-02 batch under 0.16.0 -- four weeks and six bestASR versions apart, with no fluid-* batch in between. Issue #122 asked for a sweep on each side of the upgrade; only the after side was ever run. The seven pairs are in fact bit-identical in the store, which the mismatched 2dp/1dp rendering obscured -- real evidence that those paths are deterministic across six app versions, and simply not evidence about this bump. Three factual claims corrected: - "models exist only in 0.15.5" is wrong for TDT-ja. AsrModelVersion.tdtJa and the parakeet-0.6b-ja-coreml repo are present and byte-identical in 0.15.4, as are the Unified managers. What 0.15.5 completes is the Unified FRONTEND (native Swift mel replacing the CoreML preprocessor, per-latency-tier streaming encoder). There is a genuine reason to prefer 0.15.5 for #123 -- just not the stated one -- and the error put #124's "blocked by #122" premise in doubt with it. - "the AsrModels signatures are unchanged" is false: progressHandler's type changed on every public entry point and on every consumed factory. Source-compatible here only because this repo never passes one, which is a different statement, and the issue's open question asked for the first. - "the consumed surface is entirely high-level" omits DiarizerModels.downloadIfNeeded(), AudioConverter.resampleAudioFile, DiarizerManager.initialize / .performCompleteDiarization, and AsrModelVersion. The first is a download-layer entry point, which undercuts the framing that the rewrite cannot reach this repo. Also corrected: the figures are error rates, not "accuracy" (178% is impossible under any bounded accuracy definition, and the heading inverted the direction as well as the meaning); two of the seven rows were fluid-paraformer, which the grid marks priority 2 / verified false and the sweep excludes by default as a known upstream decode bug, so stable numbers there evidence a stable bug rather than preserved quality; the comparability caveat said the two batches were "indistinguishable in the store" when they differ in app_version and decode_deterministic -- describing a hypothetical confound while the real one went unstated; and calling the 21.7% throughput decline "not evidence either way" was too strong, since one uncontrolled run cannot attribute a difference but does observe one. Recorded as unestablished rather than unchanged: the diarizer subtree also moved substantially upstream and this repo consumes diarization, but no DER row exists on either side and validate-diarization.sh was not run. 445 tests / 88 suites green. Pin, lockfile and weight manifest untouched. Refs #122
Round-2 verification found the previous correction had fixed three false claims and introduced two more, and that my first attempt at fixing THOSE introduced a third. This is the third consecutive round on this entry where a correction claimed more than its evidence; the changes here are mostly subtraction. "with no fluid-* batch in between" was false. The store holds 24 fluid-* rows on 2026-07-19 under app_version 0.14.0, squarely in between. Worse, the accurate fact it displaced was the one that defended the choice: that batch shares ZERO corpora with the after-batch, which is why the older 07-05/06 rows were used as "before" — constrained by overlap, not careless. I had established that in round 1 and compressed it into a falsehood. "the only behavioural change on that path" was also false, and the recount that replaced it was still short. ChunkProcessor carries FIVE changes across two commits: 7e856da4 adds the case-gated collapseSeamWordDuplicates AND threads case-folding into the overlap matcher — tokensMatch went from exact token-ID equality to case-folded equality, which changes where the seam is CUT rather than what is removed after it — and 0ac0e414 adds three word-boundary fallbacks. The correction to that then overstated in the other direction, asserting the fallbacks "run on any multi-chunk audio" and were therefore exercised. They are strictly conditional, differing from 0.15.4 only where no splice-safe token exists at the seam. Upstream's own instrumentation, in that commit message, reports ~67 chunk-merge events over ~15 minutes of deliberately adversarial agglutinative audio hitting the guarded logic on ~70% of seams and never falling through to any of the three. Asserting the enclosing function ran is not asserting the changed branch was taken — the same category error the original "structurally blind" sentence made. So the headline is wrong and is replaced. Byte-identical transcripts are themselves evidence that NONE of the five changes was reached. What the run establishes is that a corpus whose shape makes all five reachable in principle comes out identical, with no evidence any was entered. Weaker than "accuracy is unchanged"; stronger than the removed table could support. "code that is not on disk cannot run" was unsound — a stale build artifact survives a pin change and survives the checkout too. The artifact to interrogate is the binary: nm reports ChunkProcessor.caseVariantCanonicalIds present under the 0.15.5 pin and absent under 0.15.4. The conclusion holds; the argument did not, and it had been copied into the evidence file as well. Also corrected: "as are the Unified* managers" claimed byte-identity for five files of ~300 changed lines, in the same paragraph that says 0.15.5 rewrites their frontend. Beyond the record: the A/B wrote two rows into the local measurement store, and the 0.15.4-side row was stamped app_version 0.16.0 — the version shipping with 0.15.5 — because the schema has no dependency field. That is the first real instance of the confound this very entry documents, created by the run that documents it. The row was removed (backup retained) and the measurement now lives in benchmarks/evidence/issue-122-fluidaudio-ab.json, which also fixes the headline number being reproducible only from a temp directory. Nine live in-tree 0.15.4 references are updated. Version pointers move to 0.15.5; the two BEHAVIOURAL claims (duration-0 fallback, paraformer BPE) keep their 0.15.4 attribution and now say they were not re-verified at 0.15.5, because bumping the number there would assert something unchecked. 445 tests / 88 suites green. Pin, lockfile and weight manifest untouched by this commit. Refs #122
Verify Report — PR #142, Round 2Re-verify after the round-1 blocking findings were addressed. This report describes Engine — degraded, and it matters4 verifiers, not 6. 2 lens Agents (opus) + Devil's Advocate + coordinator. The cross-model leg was unavailable: That gap is worth stating rather than noting, because in round 1 the cross-model leg produced this PR's two sharpest findings — that a table of 178 % values cannot be headed "Accuracy" (they are error rates, and the heading inverted the direction as well as the meaning), and that comparing As in round 1, the logic lens was deliberately not dispatched — the diff is prose. Diff-freshness gate (#228): PASS at Process note — a third party was editing the treeMid-review, an unrelated branch ( The Devil's Advocate independently detected the tree moving ("at least five times… several sibling conclusions were true when written and are not true now") and filed it. Both it and the coordinator attributed the movement to the coordinator's own edits. Neither checked who else was present. Recorded because AggregateFAIL at Round-1 findings — disposition
No finding was falsely claimed as fixed. Both lenses checked every closure against the artifact. The adjudication: what the A/B establishesRound 1 said the evidence could not exercise the changed path. The correction ran a real A/B — the same tree built against each pin, on But three successive statements about what changed have each been wrong, in different directions. Round 1's write-up named one behavioural change (the case-gated The Devil's Advocate established the actual surface: five changes across two commits. And the claim that those fallbacks were exercised is false for the same reason the original "structurally blind" sentence was: it conflates the enclosing function running with the changed branch being taken. All three are strictly conditional, differing from 0.15.4 only where no splice-safe token exists at the seam. Two independent lines of evidence say none fired:
So the honest position is narrower than either draft: the run shows that a corpus whose shape makes all five changes reachable in principle — unlike This re-opens F1(b) in weakened form, overturning the lenses' "CLOSED". Findings (merged)
Negative space — attacks that failed
What changed in response (
|
1 participant
Pins FluidAudio
0.15.4 → 0.15.5.Refs #122
Why
0.15.5 completes the Parakeet Unified frontend — the CoreML preprocessor bundle is dropped in favour of a native Swift mel extractor, and the streaming encoder gains a per-latency-tier context suffix. That is what #123 needs.
TDT-ja was already reachable at 0.15.4.
AsrModelVersion.tdtJaand theparakeet-0.6b-ja-coremlrepo are present and byte-identical there, as are theUnified*managers. An earlier version of this PR said the models "exist only in 0.15.5" — that was wrong, and it puts #124's blocked by #122 premise in doubt along with it.Accuracy: an identical transcript on a corpus that could have differed
The first version of this PR offered seven before/after pairs and concluded "Accuracy is unchanged". Verification found that table could not support the claim, for two independent reasons.
It was not a before/after comparison. The "0.15.4" column reconciles to store rows captured
2026-07-05/06under app_version 0.10.0; the "0.15.5" column is the2026-08-02batch under 0.16.0 — four weeks and six bestASR versions apart. Afluid-*batch does exist in between (24 rows on2026-07-19under0.14.0), and it shares zero corpora with the after-batch — which is what forced the older rows into the "before" column: the choice was constrained by overlap, not made carelessly. Issue #122 asks for a sweep on each side of the upgrade; only the after side was ever run.And the corpora could not exercise what changed — though not for the reason the first write-up gave.
ChunkProcessorcarries five changes across two upstream commits:7e856da4adds the case-gatedcollapseSeamWordDuplicatesand threads case-folding into the overlap matcher (changing where the seam is cut, not just what is removed after);0ac0e414adds three word-boundary fallbacks that are neither case- nor Latin-gated but are strictly conditional — upstream's own instrumentation reports 67 adversarial seams with zero fall-throughs. Of the seven removed rows,jfk(11.0 s) is single-chunk so the merger never runs; the three sensevoice and two paraformer rows never enterChunkProcessorat all; onlycv-zhtw-4parakeet (25.68 s, two chunks) reaches it, and it confounds six bestASR versions. The table could not support its conclusion — but it was not empty, and the reason five of its rows carried no information was the ASR subtree they run in, not the absence of case.So the table is removed rather than repaired, and replaced with a measurement that could have come out differently — the same tree built against each pin in turn, on an English corpus long enough to be multi-chunk:
b9d43724)osr-harvard-1(33.6 s, en)19600a48)osr-harvard-1(33.6 s, en)Bit-identical, not equal-after-rounding. Which code actually ran was settled on the binary, not the checkout:
nmreportsChunkProcessor.caseVariantCanonicalIdspresent under the 0.15.5 pin and absent under 0.15.4. Neither the resolved version string nor the checked-out source settles it — a stale build artifact survives both.What it establishes is narrower than "accuracy is unchanged", and narrower than an earlier draft of this section claimed. Byte-identical transcripts are themselves evidence that none of the five changes was reached: the collapse is case-gated and a clean reading passage has no case-variant seam duplicate; the fallbacks differ only where no splice-safe token exists, and any real entry would have kept content 0.15.4 dropped. So the run shows that a corpus whose shape makes all five reachable in principle comes out identical, with no evidence any was entered — a weaker claim than the first draft made and a stronger one than the removed table could support.
Raw outputs are committed at
benchmarks/evidence/issue-122-fluidaudio-ab.json; the first version of this measurement existed only in a temp directory.(Footnote on the old table, since it was not worthless: those seven pairs are in fact bit-identical in the store — the mismatched 2 dp / 1 dp rendering obscured it. That is real evidence those paths are deterministic and unmoved across six app versions. It is simply not evidence about this dependency bump.)
Compatibility: what compiling did and did not establish
#110 flagged
DownloadUtils → ModelHubas breaking. It is not breaking for this repo — no compiled call site references either symbol, verified repo-wide rather than inferred from a green build. But two narrower claims in the first version of this PR were false:AsrModelssignatures did change. Every public entry point tookprogressHandler: DownloadUtils.ProgressHandler?and now takesprogressHandler: ProgressHandler?, as did every other consumed factory. Source-compatible here only because this repo never passes one — a different statement from "unchanged", and the issue's open question asked for the first.DiarizerModels.downloadIfNeeded(),AudioConverter.resampleAudioFile,DiarizerManager.initialize/.performCompleteDiarization, andAsrModelVersion. The first is a download-layer entry point, which undercuts the framing that the download-layer rewrite cannot reach this repo: it reaches it through a facade rather than by name.Two unmentioned behaviour changes on call sites this repo does use, both benign:
AsrModels.downloadnow performs an extra fetch of the vocab JSON (upstream #748 — the weight manifest already pins both vocab files, so the newly-guaranteed fetch lands inside the pinned set), andperformCompleteDiarizationgained a defaulted trailingprogressHandler.Supply chain
Clean, and verified rather than assumed:
19600a48…is the realv0.15.5.swift package resolveagainst the committed tree is a no-op, so the lockfile is what SwiftPM computes rather than a hand edit.ModelRegistryis byte-identical between the tags), no TLS weakening, no new credential sink, no telemetry. The offline-egress gate got stronger, gaining per-request re-checks.pin-weights.shhere would TOFU-launder any drift; the check above shows there was none to record. Leaving the manifest untouched is what preserved the ability to detect drift — please do not "fix" this reflexively.Labelling corrections
The figures are error rates, not accuracy: values like 178 % are impossible under any bounded accuracy definition, and the earlier heading inverted the direction as well as the meaning. Two of the seven rows were
fluid-paraformer, which the grid markspriority: 2, verified: falseandrelease-sweep.shexcludes by default as "demoted for a known upstream decode bug" — stable numbers there evidence a stable bug, not preserved quality. And the comparability caveat said the two batches were "indistinguishable in the store" when they differ inapp_versionanddecode_deterministic; it described a hypothetical confound while the real one went unstated. Throughput moved 161.6× → 126.5× onjfk(a 21.7 % single-run decline) — one uncontrolled run cannot attribute that to the dependency, but calling it "not evidence either way" was too strong.Not established here
KMeansClustering,OfflineReconstruction, newZeroVoteReembedder/OfflineEmbeddingExtractor/OfflineSortformerDiarizer) and this repo consumes diarization. No DER row exists on either side of the bump andscripts/validate-diarization.shwas not run, so that surface is unestablished rather than unchanged.weights-manifest.jsonstill pinssilero-vad-unified-256ms-v6.0.0; 0.15.5 moved tov6.2.1. Inert today (no VAD usage, no seam verifies that repo), recorded because it demonstrates the mechanism's blind spot: an upstream rename degrades a repo from "pinned" to "effectively unverified" without failing loudly, sinceWeightVerifieriterates manifest entries only and extra cache files never fail.build_idor lockfile hash on the row — measurement schema 增 run_kind / decode-determinism provenance 欄位（follow-up from #109 verify） #111 / mlx-audio determinism 表示法抉擇：Bool?+nil vs per-backend enum vs document-only（deferred from #111） #118 territory, not this change's.Tests
445 tests / 88 suites green. Pin, lockfile and weight manifest untouched by the correction commit.