Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions .github/workflows/dotnet.yml
Original file line number Diff line number Diff line change
Expand Up @@ -435,3 +435,94 @@ jobs:
env:
DOTNET_ROLL_FORWARD: Major
FOLDERDIFF_RUN_E2E: true

test-macos:
name: macOS stable tests and CLI smoke
runs-on: macos-latest
timeout-minutes: 20
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0

- name: Set up .NET
uses: actions/setup-dotnet@a98b56852c35b8e3190ac28c8c2271da59106c68 # v6.0.0
with:
global-json-file: global.json

- name: Cache NuGet packages
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.nuget/packages
key: ${{ runner.os }}-nuget-${{ hashFiles('**/*.csproj') }}
restore-keys: |
${{ runner.os }}-nuget-

- name: Restore dependencies
run: dotnet restore FolderDiffIL4DotNet.sln

- name: Build
run: dotnet build FolderDiffIL4DotNet.sln --configuration Release --no-restore

- name: Run stable tests
run: >-
dotnet test FolderDiffIL4DotNet.Tests/FolderDiffIL4DotNet.Tests.csproj
--configuration Release
--no-build
--nologo
--filter "Category!=E2E&Category!=Performance"
--logger "trx;LogFileName=macos_test_results.trx"
--results-directory ./MacOsTestResults

- name: Run non-interactive CLI smoke
shell: bash
run: |
set -euo pipefail

smoke_root="$RUNNER_TEMP/nildiff-macos-smoke"
old_dir="$smoke_root/old folder"
new_dir="$smoke_root/new folder"
report_root="$smoke_root/reports"
cli_path="$GITHUB_WORKSPACE/bin/Release/net8.0/nildiff"

mkdir -p "$old_dir" "$new_dir" "$report_root"
printf 'old only\n' > "$old_dir/only-old.txt"
printf 'before\n' > "$old_dir/CaseProbe.txt"
printf 'after\n' > "$new_dir/caseprobe.txt"

test -x "$cli_path"
"$cli_path" --doctor --skip-il --no-banner --no-pause

set +e
"$cli_path" \
"$old_dir" \
"$new_dir" \
macos-smoke \
--output "$report_root" \
--skip-il \
--no-banner \
--no-pause \
--fail-on-diff
smoke_exit=$?
set -e

if [ "$smoke_exit" -ne 5 ]; then
echo "::error::Expected reportable-difference exit code 5, got $smoke_exit."
exit 1
fi

for artifact in diff_report.md diff_report.html audit_log.json; do
test -s "$report_root/macos-smoke/$artifact"
done

grep -Fq "$old_dir" "$report_root/macos-smoke/diff_report.md"
grep -Fq "$new_dir" "$report_root/macos-smoke/diff_report.md"
grep -Fq '| `[ * ]` | CaseProbe.txt |' "$report_root/macos-smoke/diff_report.md"

- name: Upload macOS test results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: MacOsTestResults
path: MacOsTestResults/*.trx
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

#### Added

- **Blocking macOS CI coverage** — A new `test-macos` job runs on `macos-latest` for every configured pull request, supported-branch push, and manual dispatch. It builds Release, runs the stable suite with only `E2E` and `Performance` excluded, and executes the generated apphost to verify executable permission, tool discovery, temporary POSIX paths with spaces and case variants, report artifacts, and reportable-difference exit code `5`. The blocking job has a 20-minute timeout, uploads TRX results, and documents why real-disassembler E2E remains deferred on hosted macOS. Affected: `.github/workflows/dotnet.yml`, `FolderDiffIL4DotNet.Tests/Architecture/CiAutomationConfigurationTests.cs`, `doc/TESTING_GUIDE.md`. Issue: #224.

- **Transitive NuGet vulnerability CI gate** — The main CI workflow now audits every solution project for direct and transitive NuGet advisories, publishes the complete result in the log and GitHub job summary, and fails closed on High/Critical findings or invalid audit output. `FsCheck.Xunit` was upgraded from 2.16.6 to 3.3.3, removing the vulnerable `System.Net.Http` and `System.Security.Cryptography.X509Certificates` test dependency chain; the property tests were migrated to the FsCheck 3 fluent API. Local reproduction and the temporary-exception policy are documented in the testing guide. Affected: `.github/workflows/dotnet.yml`, `.gitignore`, `scripts/nuget_audit_gate.py`, `scripts/tests/test_nuget_audit_gate.py`, `FolderDiffIL4DotNet.Tests/FolderDiffIL4DotNet.Tests.csproj`, `FolderDiffIL4DotNet.Tests/PropertyBased/PropertyBasedTests.cs`, `FolderDiffIL4DotNet.Tests/Architecture/CiAutomationConfigurationTests.cs`, `doc/TESTING_GUIDE.md`.

- **First-class npm/Jest CI gate** — The main CI workflow now pins Node.js through `.node-version`, restores the exact `package-lock.json` graph with cached `npm ci`, runs the complete HTML-report Jest suite, and rejects High/Critical npm advisories. The lockfile refresh remediates the five findings reported when this work began. A newly published `brace-expansion` advisory (`GHSA-mh99-v99m-4gvg`) is isolated to development-only Jest tooling and has a documented exception that expires on 2026-08-31; every other High/Critical finding still fails the gate. Dependabot now opens weekly npm update pull requests that run the same checks. Affected: `.node-version`, `.github/workflows/dotnet.yml`, `.github/dependabot.yml`, `package.json`, `package-lock.json`, `npm-audit-exceptions.json`, `scripts/npm-audit-gate.js`, `doc/TESTING_GUIDE.md`. Tests: `npm_audit_gate.test.js`, `CiAutomationConfigurationTests`.
Expand Down Expand Up @@ -1694,6 +1696,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

#### 追加

- **ブロッキングな macOS CI カバレッジ** — 新しい `test-macos` ジョブを追加し、設定済みのすべての pull request、対象ブランチへの push、手動実行で `macos-latest` を使うようにしました。Release ビルド、`E2E` と `Performance` だけを除外した安定テスト、生成 apphost の直接実行を行い、実行権限、ツール探索、空白と大文字小文字差を含む一時 POSIX パス、レポート artifact、差分検出時の終了コード `5` を検証します。このブロッキングジョブは 20 分でタイムアウトし、TRX 結果をアップロードします。macOS hosted runner 上の実逆アセンブラ E2E を延期する理由も文書化しました。対象: `.github/workflows/dotnet.yml`, `FolderDiffIL4DotNet.Tests/Architecture/CiAutomationConfigurationTests.cs`, `doc/TESTING_GUIDE.md`。Issue: #224。

- **推移的 NuGet 脆弱性 CI ゲート** — メイン CI ワークフローはソリューション内の全 project を対象に NuGet の直接・推移的 advisory を監査し、完全な結果をログと GitHub job summary に公開して、High/Critical の検出または不正な監査出力を fail-closed で失敗させるようになりました。`FsCheck.Xunit` を 2.16.6 から 3.3.3 へ更新し、脆弱な `System.Net.Http` と `System.Security.Cryptography.X509Certificates` のテスト依存チェーンを解消しました。プロパティテストは FsCheck 3 fluent API へ移行しています。ローカル再現手順と一時例外ポリシーはテストガイドに記載しました。対象: `.github/workflows/dotnet.yml`, `.gitignore`, `scripts/nuget_audit_gate.py`, `scripts/tests/test_nuget_audit_gate.py`, `FolderDiffIL4DotNet.Tests/FolderDiffIL4DotNet.Tests.csproj`, `FolderDiffIL4DotNet.Tests/PropertyBased/PropertyBasedTests.cs`, `FolderDiffIL4DotNet.Tests/Architecture/CiAutomationConfigurationTests.cs`, `doc/TESTING_GUIDE.md`。

- **npm/Jest を第一級 CI ゲート化** — メイン CI ワークフローは `.node-version` で Node.js を固定し、`package-lock.json` の依存グラフをキャッシュ付き `npm ci` で厳密に復元して、HTML レポートの Jest 全テストと High/Critical npm advisory の拒否を実行するようになりました。ロックファイル更新により着手時点で報告されていた High 5 件を解消しました。作業中に新規公開された `brace-expansion` advisory(`GHSA-mh99-v99m-4gvg`)は開発専用の Jest ツールに限定されるため、2026-08-31 に失効する例外として明記し、それ以外の High/Critical はすべて引き続きゲートを失敗させます。Dependabot は同じチェックを通る npm 更新 PR を週次で作成します。対象: `.node-version`, `.github/workflows/dotnet.yml`, `.github/dependabot.yml`, `package.json`, `package-lock.json`, `npm-audit-exceptions.json`, `scripts/npm-audit-gate.js`, `doc/TESTING_GUIDE.md`。テスト: `npm_audit_gate.test.js`, `CiAutomationConfigurationTests`。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -593,6 +593,39 @@ public void Workflows_EnableRealDisassemblerE2EInCi()
Assert.Contains("DOTNET_ILDASM_VERSION: 0.12.2", releaseWorkflow, StringComparison.Ordinal);
}

/// <summary>
/// Verifies that the macOS CI job remains blocking, bounded, and focused on stable tests plus CLI smoke coverage.
/// macOS CI ジョブがブロッキングかつ時間制限付きで、安定テストと CLI スモークを対象にし続けることを検証します。
/// </summary>
[Fact]
[Trait("Category", "Unit")]
public void DotNetWorkflow_BuildsAndSmokeTestsOnMacOs()
{
var workflow = File.ReadAllText(GetRepositoryFilePath(".github", "workflows", "dotnet.yml"));
var jobStart = workflow.IndexOf("\n test-macos:", StringComparison.Ordinal);

Assert.True(jobStart >= 0, "The dotnet workflow must define a test-macos job.");

var macOsJob = workflow[jobStart..];

Assert.Contains("name: macOS stable tests and CLI smoke", macOsJob, StringComparison.Ordinal);
Assert.Contains("runs-on: macos-latest", macOsJob, StringComparison.Ordinal);
Assert.Contains("timeout-minutes: 20", macOsJob, StringComparison.Ordinal);
Assert.Contains("\"Category!=E2E&Category!=Performance\"", macOsJob, StringComparison.Ordinal);
Assert.Contains("test -x \"$cli_path\"", macOsJob, StringComparison.Ordinal);
Assert.Contains("\"$cli_path\" --doctor --skip-il --no-banner --no-pause", macOsJob, StringComparison.Ordinal);
Assert.Contains("smoke_exit=$?", macOsJob, StringComparison.Ordinal);
Assert.Contains("diff_report.md diff_report.html audit_log.json", macOsJob, StringComparison.Ordinal);
Assert.Contains("grep -Fq \"$old_dir\"", macOsJob, StringComparison.Ordinal);
Assert.Contains("grep -Fq \"$new_dir\"", macOsJob, StringComparison.Ordinal);
Assert.Contains("CaseProbe.txt", macOsJob, StringComparison.Ordinal);
Assert.Contains("name: MacOsTestResults", macOsJob, StringComparison.Ordinal);
Assert.DoesNotContain("FOLDERDIFF_RUN_E2E: true", macOsJob, StringComparison.Ordinal);
Assert.DoesNotContain("dotnet tool install --global dotnet-ildasm", macOsJob, StringComparison.Ordinal);
Assert.DoesNotContain("continue-on-error: true", macOsJob, StringComparison.Ordinal);
Assert.DoesNotContain("\n if:", macOsJob, StringComparison.Ordinal);
}

/// <summary>
/// Verifies that the benchmark regression workflow detects performance degradation on PRs.
/// ベンチマークリグレッションワークフローが PR でパフォーマンス劣化を検知することを検証します。
Expand Down
8 changes: 6 additions & 2 deletions doc/TESTING_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,9 @@ Workflow/config files: [`.github/workflows/dotnet.yml`](../.github/workflows/dot

- DocFX site generation runs before tests and publishes `_site/` as the `DocumentationSite` artifact.
- Tests and coverage run only when [`FolderDiffIL4DotNet.Tests/FolderDiffIL4DotNet.Tests.csproj`](../FolderDiffIL4DotNet.Tests/FolderDiffIL4DotNet.Tests.csproj) exists.
- .NET validation runs in two jobs: the `build` job (Ubuntu) installs a real [`dotnet-ildasm`](https://www.nuget.org/packages/dotnet-ildasm/) tool, adds the global-tool directory to `PATH`, and runs the test step with both `DOTNET_ROLL_FORWARD=Major` and `FOLDERDIFF_RUN_E2E=true`; the `test-windows` job (Windows) does the same on `windows-latest`. This makes the real-disassembler E2E test part of the blocking CI gate instead of an opt-in skip.
- .NET validation runs in three blocking jobs. The `build` job (Ubuntu) installs a real [`dotnet-ildasm`](https://www.nuget.org/packages/dotnet-ildasm/) tool, adds the global-tool directory to `PATH`, and runs the test step with both `DOTNET_ROLL_FORWARD=Major` and `FOLDERDIFF_RUN_E2E=true`; the `test-windows` job (Windows) does the same on `windows-latest`.
- The `test-macos` job runs on `macos-latest` for every pull request, supported-branch push, and manual dispatch configured for the workflow. It builds Release, runs the stable suite, and directly executes the generated apphost. The smoke test checks executable permission, `--doctor` tool discovery, temporary POSIX paths containing spaces and case variants, report generation, and the expected reportable-difference exit code `5`.
- macOS excludes only `E2E` tests because hosted-runner behavior for the external disassembler is not yet proven stable, and `Performance` tests because timing assertions are unsuitable for shared hosted runners. Unit coverage for path/case handling, temporary directories, and deterministic fake-tool discovery remains enabled. The job has no job-level condition or `continue-on-error`, so it is blocking; a 20-minute timeout and the two narrow exclusions bound its cost. Real-disassembler macOS E2E remains deferred until hosted-runner stability is established.
- The blocking `javascript-tests` job uses the exact Node.js version in [`.node-version`](../.node-version), keys the npm cache from [`package-lock.json`](../package-lock.json), restores with `npm ci`, runs the complete Jest suite, and fails on newly introduced High/Critical npm advisories through `npm run audit:high`. The gate permits only advisory dependency chains rooted in an unexpired exact match from [`npm-audit-exceptions.json`](../npm-audit-exceptions.json); unrelated findings, expired exceptions, and invalid audit responses fail CI.
- `TestAndCoverage` artifact includes TRX and coverage outputs.
- `CoverageReport/SummaryGithub.md` is appended to GitHub Step Summary when present.
Expand Down Expand Up @@ -456,7 +458,9 @@ dotnet tool run reportgenerator -reports:"TestResults/**/coverage.cobertura.xml"

- テスト前に DocFX サイト生成を実行し、`_site/` を `DocumentationSite` artifact として公開します。
- [`FolderDiffIL4DotNet.Tests/FolderDiffIL4DotNet.Tests.csproj`](../FolderDiffIL4DotNet.Tests/FolderDiffIL4DotNet.Tests.csproj) が存在する場合のみテスト/カバレッジを実行します。
- .NET の検証は 2 つのジョブで構成されます。`build` ジョブ(Ubuntu)はテスト前に実 [`dotnet-ildasm`](https://www.nuget.org/packages/dotnet-ildasm/) をインストールし、グローバルツールディレクトリを `PATH` に追加したうえで、`DOTNET_ROLL_FORWARD=Major` と `FOLDERDIFF_RUN_E2E=true` 付きでテストを実行します。`test-windows` ジョブ(Windows)も `windows-latest` 上で同様に検証します。これにより、実逆アセンブラ E2E がブロッキングな CI ゲートに入ります。
- .NET の検証は 3 つのブロッキングジョブで構成されます。`build` ジョブ(Ubuntu)はテスト前に実 [`dotnet-ildasm`](https://www.nuget.org/packages/dotnet-ildasm/) をインストールし、グローバルツールディレクトリを `PATH` に追加したうえで、`DOTNET_ROLL_FORWARD=Major` と `FOLDERDIFF_RUN_E2E=true` 付きでテストを実行します。`test-windows` ジョブ(Windows)も `windows-latest` 上で同様に検証します。
- `test-macos` ジョブは、ワークフローで設定されたすべての pull request、対象ブランチへの push、手動実行に対して `macos-latest` 上で動きます。Release ビルドと安定テストを実行し、生成した apphost を直接起動します。CLI スモークでは実行権限、`--doctor` によるツール探索、一時領域上の空白と大文字小文字差を含む POSIX パス、レポート生成、差分検出時の想定終了コード `5` を確認します。
- macOS で除外するのは、外部逆アセンブラの hosted runner 上での安定性が未確認である `E2E` と、共有 hosted runner で時間判定に適さない `Performance` だけです。パス/大文字小文字処理、一時ディレクトリ、決定的な偽ツール探索の unit テストは引き続き実行します。ジョブレベルの条件や `continue-on-error` はないためブロッキングであり、20 分のタイムアウトと 2 つの限定的な除外でコストを抑えます。macOS の実逆アセンブラ E2E は hosted runner 上での安定性を確認できるまで延期します。
- ブロッキングな `javascript-tests` ジョブは、[`.node-version`](../.node-version) の Node.js バージョンを厳密に使い、[`package-lock.json`](../package-lock.json) をキーに npm キャッシュを構成し、`npm ci` で復元して Jest 全件を実行します。さらに `npm run audit:high` により、新しく持ち込まれた High/Critical の npm advisory で失敗します。このゲートが許可するのは、[`npm-audit-exceptions.json`](../npm-audit-exceptions.json) の未失効かつ完全一致する advisory を起点とした依存チェーンだけであり、無関係な検出、期限切れの例外、無効な監査レスポンスは CI を失敗させます。
- `TestAndCoverage` アーティファクトに TRX とカバレッジ関連ファイルを格納します。
- `CoverageReport/SummaryGithub.md` があれば GitHub Step Summary に追記されます。
Expand Down
Loading