[desc] Execute pre & post process in the farm

[Alxiice]

PR description

We introduce a new feature that enables preprocessing and postprocessing tasks.
In this new version, these tasks are considered as new chunks and can be implemented on custom nodes.
On the scheduling/submitter part, preprocessing are executed before all chunks, and postprocessing are executed after them.
This way these can be used to prepare data for processing in chunks (preprocessing), and then concatenate data from all chunks (postprocessing).

Adding preprocessing/postprocessing tasks

It's very simple and follows what we already use with process/processChunk

class PluginSubmitterDPrePost(desc.Node):
    category = 'MyPlugins'
    inputs = INPUTS
    outputs = OUTPUTS

    def preprocess(self, node):
        LOGGER.info(f"> [{node.name}](preprocess) Start")
        # TODO
        LOGGER.info(f"> [{node.name}](preprocess) Done")

    def postprocess(self, node):
        LOGGER.info(f"> [{node.name}](postprocess) Start")
        # TODO
        LOGGER.info(f"> [{node.name}](postprocess) Done")

Impact on the UI

If there no preprocess/postprocess the chunks will not be displayed on the UI.

Implementation notes

• Create a add_method_flag decorator to be able to check whether a method is overloaded or not. preprocess and postprocess methods are flagged with it, and we know if we have a pre/postprocess using this system
• On the submitter part a new task graph system is introduced to rebuild the list of tasks. This way when we get to the submitter it's extremely simple to send tasks to the submitter.

Note to the reviewers

• Check the name of classes (Ordered...) on the submitter

If you use rez you can add this to the package commands :

def commands():
    import os
    ...
    enabled_options = ('y', 'yes', 't', 'true', 'on', '1')
    if os.environ.get("ENABLE_LOCAL_SUBMITTERS", "0") in enabled_options:
        env.MESHROOM_SUBMITTERS_PATH.append('{base}/repo/meshroom/submitters')
    ...

Merging

• First a PR to make sure nothing is changes on the submitter side : mrSubmitters/4
• Then this PR
• Finally a PR on the submitter side to use the new API : TODO

[codecov]

Codecov Report

❌ Patch coverage is 82.84600% with 88 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.69%. Comparing base (a4ffad7) to head (192652a).
⚠️ Report is 19 commits behind head on develop.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
meshroom/core/node.py	68.98%	58 Missing ⚠️
meshroom/core/submitter.py	87.57%	20 Missing ⚠️
meshroom/core/desc/node.py	54.54%	5 Missing ⚠️
...lugins/meshroom/pluginSubmitter/PluginSubmitter.py	62.50%	3 Missing ⚠️
tests/test_compute.py	98.73%	1 Missing ⚠️
tests/test_submit.py	98.43%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2984      +/-   ##
===========================================
+ Coverage    83.47%   83.69%   +0.22%     
===========================================
  Files           81       81              
  Lines        10301    10698     +397     
===========================================
+ Hits          8599     8954     +355     
- Misses        1702     1744      +42     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR introduces preprocessing and postprocessing capabilities to Meshroom's node execution system. These new stages allow nodes to perform setup work before chunk processing and cleanup/aggregation work after all chunks complete. The implementation treats pre/post-processing as special chunks with dedicated indices, integrates them into the task submission system, and updates the UI to display these stages.

Changes:

• Added ChunkIndex enum to distinguish preprocess (-2), postprocess (-1), and standard chunks (0+)
• Implemented task ordering system (OrderedTasks, OrderedTask, OrderedNode) to schedule pre/post-processing relative to chunks
• Updated UI (QML) to display preprocess/postprocess chunks in the chunks list view

Reviewed changes

Copilot reviewed 14 out of 16 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
meshroom/core/desc/node.py	Added add_method_flag decorator to detect overridden preprocess/postprocess methods
meshroom/core/node.py	Implemented ChunkIndex enum, created pre/post-process chunks, updated chunk processing logic
meshroom/core/submitter.py	Added OrderedTask/OrderedNode/OrderedTasks classes for task dependency ordering
meshroom/core/taskManager.py	Updated to pass forceCompute to pre/postprocess calls
meshroom/core/graph.py	Added debug print statements for graph execution (should be removed)
meshroom/ui/qml/Utils/Colors.qml	Added null check for chunk color handling
meshroom/ui/qml/GraphEditor/NodeLog.qml	Simplified chunk log file access
meshroom/ui/qml/GraphEditor/NodeEditor.qml	Changed to use allChunks property, increased width for "postprocess" text
meshroom/ui/qml/GraphEditor/ChunksListView.qml	Refactored to handle pre/postprocess chunks with ChunkIndex enum
bin/meshroom_compute	Added --preprocess and --postprocess flags, updated chunk iteration handling
bin/meshroom_createChunks	Moved preprocess/postprocess calls outside chunk loop
localfarm/localFarm.py	Improved logging format
localfarm/README.md	Added comprehensive documentation for local farm (contains typo: "famr")
tests/test_submit.py	Added test_orderTasks to verify task ordering logic
tests/test_compute.py	Updated to call pre/postprocess outside chunk loop, added duplicate prepareLogger call
tests/plugins/meshroom/pluginSubmitter/PluginSubmitter.py	Updated test plugins to demonstrate pre/postprocess usage

Comments suppressed due to low confidence (4)

localfarm/README.md:8

• Spelling error: "famr" should be "farm"

> Note that the local famr only works in Unix for now because we use `fork` for daemonization.

meshroom/core/submitter.py:76

• The comment for EXPANDING is identical to PREPROCESS. It should describe that this task executes a node's expanding process (for nodes whose chunk count is not determined yet).

    EXPANDING = auto()
    """Task that executes a node preprocess method"""

meshroom/core/submitter.py:78

• The comment for CHUNK is misleading. Based on the code context, CHUNK represents a task that executes a single chunk of a parallelized node, not a task that will expand. The "expanding" behavior is handled by the EXPANDING task type.

    CHUNK = auto()
    """Task that will expand during the processing"""

meshroom/core/submitter.py:184

• Spelling error in documentation: "orger" should be "order"

        Take all the nodes and connections and orger them by processing step

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[servantftransperfect]

Can't you use simple introspection ?

is_overridden = type(self).preprocess is not Node.preprocess

not sure of the grammar