Fourier filtering, covariance fixes and masks by adrien-laposta · Pull Request #70 · simonsobs/SOOPERCOOL

adrien-laposta · 2026-05-11T14:58:25Z

This PR merges three different branches

ML_maps: allows for an additional Fourier filtering step on maps (can be either F+B or ML maps). Applying the Fourier filter might require being careful about sharp features in our maps. This new feature is therefore coupled to the mask refactoring recent changes.
mask-validation: This branch (see Mask validation #60) merges changes from the refactor_mask branch as it is complementary to the validation script introduced. Fourier filtering step requires the cropped version of the binary mask generated with this new mask definition script.
covfix: This was just a branch started mostly to fix an issue with the binning of beam window functions when using compute_Dl: True in the parameter file. It has been merged and tested on ML maps, with changes introduced in this branch.

commit ac76d04 Author: kwolz <kevin.wolz93@gmail.com> Date: Sat Mar 14 19:59:24 2026 -0400 remove unnecessary line commit 50c011f Author: kwolz <kevin.wolz93@gmail.com> Date: Sat Mar 14 19:56:58 2026 -0400 removed unused script commit ffefc79 Author: kwolz <kevin.wolz93@gmail.com> Date: Sat Mar 14 19:45:43 2026 -0400 lint commit b9e9116 Merge: 3cd5550 6596179 Author: kwolz <kevin.wolz93@gmail.com> Date: Sat Mar 14 19:42:10 2026 -0400 merging from remote commit 3cd5550 Author: kwolz <kevin.wolz93@gmail.com> Date: Sat Mar 14 19:34:14 2026 -0400 add mask validation script, small refactoring of map_utils commit b17450d Author: kwolz <kevin.wolz93@gmail.com> Date: Sat Mar 14 19:31:50 2026 -0400 refactor & doc get_analysis_mask commit 7248903 Merge: 0ce6c85 329a7bc Author: kwolz <kevin.wolz93@gmail.com> Date: Thu Feb 19 13:19:03 2026 -0500 resolve merger conflicts commit 329a7bc Author: Adrien La Posta <adrien.la.posta@gmail.com> Date: Wed Feb 18 17:44:46 2026 +0000 first commit, clean smooth mask commit 6596179 Author: kwolz <kevin.wolz93@gmail.com> Date: Sun Sep 28 14:41:24 2025 +0100 improve plot transfer function commit da91814 Author: kwolz <kevin.wolz93@gmail.com> Date: Sun Sep 28 14:40:46 2025 +0100 mask spin derivatives commit 5c49012 Author: Jason Leung <leung@astro.utoronto.ca> Date: Wed Aug 6 10:00:00 2025 +0100 Improve plots for mask validation commit 874ee89 Author: Jason Leung <leung@astro.utoronto.ca> Date: Tue Aug 5 12:00:00 2025 +0100 Change argument name from 'puretype' to 'simtype' since 'puretype' refers to power-law Gaussian sims whereas mask validation uses CMB sims as input. commit 83ad1d8 Author: Jason Leung <leung@astro.utoronto.ca> Date: Mon Aug 4 16:00:00 2025 +0100 Default to validate only the mask listed in config Only if that mask is not found, look for masks in the mask directory. Print plot directory on screen. commit 0ce6c85 Author: kwolz <kevin.wolz93@gmail.com> Date: Thu Jul 24 04:29:13 2025 -0400 changed car apodization commit f3b8c86 Author: kwolz <kevin.wolz93@gmail.com> Date: Wed Jul 16 18:25:38 2025 -0400 mostly naming conventions commit 4f2b68a Author: kwolz <kevin.wolz93@gmail.com> Date: Wed Jul 16 18:24:27 2025 -0400 script to coadd signal and noise sims commit a29bcc4 Author: kwolz <kevin.wolz93@gmail.com> Date: Wed Jul 16 18:23:53 2025 -0400 noise curve fitting and plotting commit 4af8558 Author: kwolz <kevin.wolz93@gmail.com> Date: Wed Jul 16 18:23:26 2025 -0400 add approximate Gaussian ISO beams commit 927febe Author: kwolz <kevin.wolz93@gmail.com> Date: Wed Jul 16 18:22:39 2025 -0400 plotting improvements and car apodzation commit 4010704 Author: kwolz <kevin.wolz93@gmail.com> Date: Wed Jul 16 18:16:13 2025 -0400 mask: added weights option; shifted smoothing before apodization commit 354efe2 Author: kwolz <kevin.wolz93@gmail.com> Date: Fri Jul 11 11:06:08 2025 -0400 update N_ell fitting routine commit d884836 Author: kwolz <kevin.wolz93@gmail.com> Date: Fri Jul 11 11:03:35 2025 -0400 lmax management commit b94cb53 Author: kwolz <kevin.wolz93@gmail.com> Date: Thu Jul 24 04:36:09 2025 -0400 change car apodization commit 0ea3700 Author: kwolz <kevin.wolz93@gmail.com> Date: Fri Jul 11 10:58:21 2025 -0400 modernized mpi_utils commit 199b38e Author: kwolz <kevin.wolz93@gmail.com> Date: Fri Jul 11 10:55:21 2025 -0400 add alternative (temporary) mask generation commit 18fd459 Author: kwolz <kevin.wolz93@gmail.com> Date: Fri Jul 11 10:54:02 2025 -0400 lmax management in MCM computation commit c8f532c Author: Jason Leung <leung@astro.utoronto.ca> Date: Mon Jun 23 09:00:00 2025 +0100 Fix plot units Cls have units temp^2 commit 3e1b791 Author: Jason Leung <leung@astro.utoronto.ca> Date: Fri Jun 20 14:00:00 2025 +0100 Add mask validation script Scans a directory for analysis masks, applies them to CMB sims, and passes the masked maps into NaMaster to obtain the power spectra. Outputs plots in directory specified in pipeline/get_analysis_mask.py

…rders are not sharp and within the footprint

kwolz

A few comments. Looks good overall.

kwolz · 2026-05-11T17:27:13Z

-        out_unf = f"{pcls_tf_est_dir}/pcls_mat_tf_est_{ftag1}_x_{ftag2}_unfiltered_{id_sim:04d}.npz"  # noqa
+        preproc_ftag1, kspace_tag1 = ftag1
+        preproc_ftag2, kspace_tag2 = ftag2
+


These names are getting long! Can we add a (...) around each lab to help identifying their content from the file name?

Not sure I fully understood this, you want to explicitely separate them in the fname string using brackets ? (Overall I agree with you on file name length - that's the price to pay to stay explicit)

Yes, something like {ftag1}_x_{ftag2} --> ({ftag1})_x_({ftag2})

kwolz · 2026-05-11T17:45:47Z

-        print(f"{tf_dir}/transfer_function_{ftag1}_x_{ftag2}.npz")
+        lab1 = f"{ftag1[0]}_{ftag1[1]}"
+        lab2 = f"{ftag2[0]}_{ftag2[1]}"
+        print(f"{tf_dir}/transfer_function_{lab1}_x_{lab2}.npz")


Same suggestion as above to put brackets around lab1 and lab2 in the file names

kwolz · 2026-05-11T17:50:09Z

-        tf = np.load(f"{tf_dir}/transfer_function_{ftag1}_x_{ftag2}.npz")
+        ktag1 = meta.kspace_tag_from_map_set(ms1)
+        ktag2 = meta.kspace_tag_from_map_set(ms2)
+        tf = np.load(f"{tf_dir}/transfer_function_{ftag1}_{ktag1}_x_{ftag2}_{ktag2}.npz") # noqa


adrien-laposta · 2026-05-14T09:22:15Z

Regarding all comments related to the naming convention length.

A quick grep looking for occurrences of ftag gives a lot of results: it feel unsafe to propagate all changes there.
This naming convention length issue will also occur for spectra (if map_sets are quite long and descriptive), covariances (pairs of map_set pairs) and other products such as coupling matrices.
I'd rather open an issue on this, to see if we find any elegant way of compressing these labels to make pipeline output more human-readable and apply it to all outputs all at once.

adrien-laposta · 2026-05-14T09:23:31Z

Regarding all comments related to the naming convention length.

* A quick grep looking for occurrences of `ftag` gives a lot of results: it feel unsafe to propagate all changes there.

* This naming convention length issue will also occur for spectra (if map_sets are quite long and descriptive), covariances (pairs of `map_set` pairs) and other products such as coupling matrices.

* I'd rather open an issue on this, to see if we find any elegant way of compressing these labels to make pipeline output more human-readable and apply it to all outputs all at once.

An issue has been opened there #75

chervias · 2026-05-22T15:20:10Z

This PR is missing applyting the k-space filter to the signflip spectra estimation that happens on pipeline/compute_signflip_sims_pseudo_cells.py

chervias · 2026-05-22T15:55:05Z

+
+    def almtomap(alm):
+        return alm2map(alm, pix_type=pix_type, nside=nside,
+                       car_map_template=temp)


this is a bug, alm2map has parameter car_template, not car_map_template

chervias · 2026-05-22T21:10:36Z



-def plot_transfer_function(lb, tf_dict, lmin, lmax, field_pairs, file_name):
+def plot_transfer_function(lb, tf_dict, lmin, lmax, field_pairs,


Here tf_dict is assumed to be a dictionary, but the way it is used you actually pass a npz file here, which is not the same, so this function is broken as is. I'm not sure how you want to handle it, I will deactivate plots for now for me.

…seudo_cells.py

adrien-laposta added 23 commits February 25, 2026 12:18

adapt to multi hit maps

875c374

fix binning

a1fa790

fix bin

e876a13

some fft tools

0c3cd2d

fft filtering

49096e6

add todo mask

94f5ab4

fix output naming

8a74a2a

adapt to multi-filtering

979cd87

fix fname typo

c2b2131

on-the-fly filtering in cl estimation

4c3008b

todo

6f5669a

testing is useful

c046592

fix fft

af4473f

coupling updates

fc9708a

pcls fix and mask

4bf0b16

flake8

7d938ba

Merge remote-tracking branch 'origin/covfix' into ML_maps

4f700df

fix cov

8e857ed

fix doc

4d618df

there is no need to smooth the analysis mask, cropping ensure that bo…

d4b4fa0

…rders are not sharp and within the footprint

flake8

c279b36

flaked

16f6720

adrien-laposta commented May 11, 2026

View reviewed changes

Comment thread pipeline/misc/validate_mask.py

kwolz requested changes May 11, 2026

View reviewed changes

a-rizzieri and others added 5 commits May 12, 2026 04:50

file name to save mask figures build step by step

7bf17a9

remove useless comments

0ab5f6a

Merge branch 'ML_maps' of github.com:simonsobs/SOOPERCOOL into ML_maps

5dce8f1

remove useless comments

2895d15

relabel hit map list

a7011ef

fix hp/car treatment for hits/weights

c51d10f

This was referenced May 12, 2026

Not save maps/spectra for mask-validation script #71

Open

Revive TF validation routines for both FB and FFT filters #72

Open

Merge sacc file creation for MC/AN cov #73

Open

Generalize cov couplings for noise to T hits #74

Open

add error for old paramfiles

be0003b

kwolz approved these changes May 12, 2026

View reviewed changes

remove unused argument

8f09d01

adrien-laposta mentioned this pull request May 14, 2026

Find and implement elegant naming convention to not have long filenames #75

Open

chervias reviewed May 22, 2026

View reviewed changes

fixing a bug and changing the default of K_to_uK

12dbaaf

chervias reviewed May 22, 2026

View reviewed changes

chervias added 2 commits May 22, 2026 19:32

changes required to bring kspace filtering to compute_signflip_sims_p…

0cfe640

…seudo_cells.py

fixing a bug for kspace filter on compute_signflip_sims_pseudo_cells.py

5b114d6

adrien-laposta mentioned this pull request May 26, 2026

Fix transfer function plots #77

Open

flake8

fd5a8a2

adrien-laposta merged commit 2428c7a into main May 26, 2026
1 check passed



		def plot_transfer_function(lb, tf_dict, lmin, lmax, field_pairs, file_name):
		def plot_transfer_function(lb, tf_dict, lmin, lmax, field_pairs,

Conversation

adrien-laposta commented May 11, 2026

Uh oh!

Uh oh!

kwolz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kwolz May 11, 2026

Choose a reason for hiding this comment

Uh oh!

adrien-laposta May 12, 2026

Choose a reason for hiding this comment

Uh oh!

kwolz May 12, 2026

Choose a reason for hiding this comment

Uh oh!

kwolz May 11, 2026

Choose a reason for hiding this comment

Uh oh!

kwolz May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adrien-laposta commented May 14, 2026

Uh oh!

adrien-laposta commented May 14, 2026

Uh oh!

chervias commented May 22, 2026

Uh oh!

chervias May 22, 2026

Choose a reason for hiding this comment

Uh oh!

chervias May 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants