[Hackathon] feat: Reproducibility Validation for Workflow Versions

[SarahAsad23]

Overview

This PR introduces a framework for comparing different versions of a workflow execution to help identify where executions begin to diverge. The goal of this project is to improve workflow reproducibility and debugging by making it easier to detect differences across workflow versions, operator outputs, and intermediate states.

The system allows workflow executions to be run side-by-side and analyzed for inconsistencies throughout execution rather than only comparing final outputs.

0516.1.mp4

Motivation

As workflows evolve over time, even small changes to operators, dependencies, configurations, or execution environments can lead to unexpected differences in results. Currently, it can be difficult to determine exactly where two workflow executions stop behaving the same way.

This project explores the idea of execution-level differencing for workflows by introducing tooling to:

• Compare executions across workflow versions
• Detect divergence points between executions
• Surface intermediate operator differences
• Support reproducibility analysis and debugging

Features

• Added support for comparing multiple workflow executions
• Introduced execution differencing logic for detecting mismatched states
• Added operator-level comparison support
• Added infrastructure for tracking divergence points during execution

[codecov-commenter]

Codecov Report

❌ Patch coverage is 16.16162% with 83 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.76%. Comparing base (df23815) to head (a80774b).
⚠️ Report is 24 commits behind head on main.

Files with missing lines	Patch %	Lines
...ard/user/workflow/WorkflowExecutionsResource.scala	16.16%	82 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               main    #5102    +/-   ##
==========================================
  Coverage     42.75%   42.76%            
- Complexity     2197     2206     +9     
==========================================
  Files          1045     1045            
  Lines         40094    40258   +164     
  Branches       4232     4254    +22     
==========================================
+ Hits          17143    17217    +74     
- Misses        21887    21969    +82     
- Partials       1064     1072     +8     

Flag	Coverage Δ		*Carryforward flag
access-control-service	39.53% <ø> (ø)		Carriedforward from df23815
agent-service	33.72% <ø> (ø)		Carriedforward from df23815
amber	43.61% <16.16%> (+0.01%)	⬆️
computing-unit-managing-service	0.00% <ø> (ø)		Carriedforward from df23815
config-service	0.00% <ø> (ø)		Carriedforward from df23815
file-service	32.18% <ø> (ø)		Carriedforward from df23815
frontend	33.92% <ø> (-0.01%)	⬇️	Carriedforward from df23815
python	88.84% <ø> (ø)		Carriedforward from df23815
workflow-compiling-service	47.72% <ø> (ø)		Carriedforward from df23815

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.