Too many edges for number of nodes

[rbdavid]

We're seeing failures of the Convergence Ratio analysis runs with the below error. After some digging, the fatal issue is that, by the time convergence ratio is calculated, the blast parquet file contains self-alignments still, resulting in convergence ratio values greater than 1 (which should not be possible).

This continues the work I initiated a couple months ago in #258

The flow needs to be:

• Condense sequences. (ALL_BY_ALL workflow)
• Run all-by-all BLAST calcs. (ALL_BY_ALL workflow)
• Run blastreduce to take the top triangle of the all-by-all BLAST results. Keep self-alignments here. (ALL_BY_ALL workflow)
• Restore from the condensed sequence set to the full sequence set. Remove self-alignments here. (ALL_BY_ALL workflow)
  • a DuckDB call happens that outputs condensed.out.
  • a restore_condensed_sequences.py call happens that outputs 1.out. This is likely where the self-alignments need to be removed.
  • a transcode_restored_blast.py call happens that outputs the 1.out.parquet file.
• Calculate convergence ratio. (REPORTING workflow)

[rbdavid]

I believe this is fixed with #329 now merged but testing with John's use case in mind needs to happen still.

Note, in #329, the SQL query to remove self-alignments was implemented in the restore_condensed_sequences.py script (restore_condensed proccess block). In this script, a CLI argument also controls whether intra- CD-HIT cluster edges are created or not. It should be emphasized that intra-cluster edges are the most important edges (since the sequences in a CD-HIT cluster are all 100% sequence identical) in the network. If we don't include them, our degree centrality and convergence ratio metrics as well as the network topology, when visualized in cytoscape, will be incorrect. Omission of these edges may or may not have an effect on the final scientific hypotheses being developed from the SSN.

Currently, the use of the "old method" (omitting intra-cluster edges) is turned on by default. This should be changed once the code is in a better state to fix the bug.