perf: chan's parallel mean-var algorithm for dask-backed arrays (sparse/dense)

[ilan-gold]

See https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm

Based on a #4118 (comment) with @zboldyga

This has two benefits - it allows us to calculate mean/var in one pass instead of effectively two (square sum and sum squared) and gets rid of a numerical instability issue that @zboldyga found the solution to (see removed comment)

• Closes #
• Tests included or not required because:

• Release notes not necessary because:

[codecov]

Codecov Report

❌ Patch coverage is 95.83333% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.71%. Comparing base (d96d91d) to head (7bf2db4).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/scanpy/get/_aggregated.py	95.83%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           main    #4143       +/-   ##
=========================================
+ Coverage      0   79.71%   +79.71%     
=========================================
  Files         0      120      +120     
  Lines         0    12830    +12830     
=========================================
+ Hits          0    10227    +10227     
- Misses        0     2603     +2603     

Flag	Coverage Δ
hatch-test.low-vers	78.74% <39.58%> (?)
hatch-test.pre	79.57% <95.83%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/scanpy/get/_aggregated.py	92.94% <95.83%> (ø)

... and 119 files with indirect coverage changes

[zboldyga]

@ilan-gold I added njit support, see: #4153 . This enables rank_gene_groups to use njit.

I integrated this to the rank_gene_groups PR and benchmarked there as well as here and it gives a speedup on both at normal group x gene sizes.

[ilan-gold]

Nice commented there about something, but once you got the pre-commit fixed as well, I'll merge into this

[scverse-benchmark]

Benchmark changes

Change	Before [d96d91d]	After [7bf2db4]	Ratio	Benchmark (Parameter)
-	1.96G	381M	0.19	preprocessing_counts.Agg.peakmem_agg('var', False, True)
-	898±30ms	736±5ms	0.82	preprocessing_counts.Agg.time_agg('count_nonzero', False, False)
-	2.58±0.01s	19.9±0.7ms	0.01	preprocessing_counts.Agg.time_agg('var', False, True)
-	23.6±0.7ms	16.3±0.3ms	0.69	preprocessing_counts.Agg.time_agg('var', True, True)
-	8.48±0.1ms	7.32±0.3ms	0.86	preprocessing_counts.FastSuite.time_log1p('pbmc3k', 'counts-off-axis')

Comparison: https://github.com/scverse/scanpy/compare/d96d91de3162f29d901194ac56fd732459389784..7bf2db4aa11c28d5e6ed01644453bb742bab6375
Last changed: Thu, 25 Jun 2026 11:24:00 +0000

More details: https://github.com/scverse/scanpy/pull/4143/checks?check_run_id=83408969507

[ilan-gold]

I have no idea why this CI job is failing but the one in #4013 passes and contains this branch