Question: Best practices to limit CPU usage?

[jashapiro]

I have been starting some work running CytoTable to work with some data on CellPainting Gallery, and I am wrapping it as part of a Nextflow workflow. One of the things I would like to be able to do is to limit the CPU usage of cytotable.convert() while maintaining its efficiency, and I was not quite clear on the best way to do that.

I can limit the number of parsl processes/threads with its config, but duckdb and pyarrow seem to have their own CPU discovery that default to the full set of available cores, so I still end up with quite a large number of threads using that setting alone. It seems like I should be able to also set CYTOTABLE_MAX_THREADS and that would limit duckdb, but perhaps not pyarrow? I think pyarrow can also be limited on its own with set_cpu_count() and set_io_thread_count().

So there seem to be a number of possible places to limit the CPU use, and I what I am trying to work out is how best to balance those. Do you have any recommendations for balancing the CPU allocation among parsl, duckdb, and pyarrow?

Secondarily, for a system running within a single Docker container for import, do you have a recommendation between HighThroughputExecutor and ThreadPoolExecutor? It seems from the paper that multithreading is generally a bit more efficient, but I wanted to confirm that was what you would recommend.

Thanks for your help!

[d33bs]

Thanks @jashapiro for reaching out and sorry to hear about the CPU use challenges! I think you're on the right track in terms of thinking through this.

• I recommend trying HighThroughputExecutor which has more reliable performance generally. ThreadPoolExecutor is more efficient to a point but also may suffer from poor memory management contingent on the system.
• CYTOTABLE_MAX_THREADS will help when it comes to duckdb, spot on.
• set_io_thread_count() and set_cpu_count() should help with arrow allocations. That said, I have not tested these as much directly.

Here's some pseudocode below to reference in terms of how you might optimize things. I think the thing to tinker with here is the task CPU's available and duckdb_threads

• If CPU is below the max and memory is fine, try increasing duckdb_threads (or raise workers).
• If CPU is high but runtime doesn’t improve, back things down.

# import non-cytotable packages first to avoid setting defaults automatically in cytotable
import os
import pyarrow as pa
from parsl.config import Config
from parsl.executors import HighThroughputExecutor

# set a certain number of cpus
task_cpus = int(os.environ.get("NXF_TASK_CPUS", "8"))
# set number of threads for duckdb to use
duckdb_threads = 2
# set a floor to how many workers parsl may use based on duckdb threads and cpus available
workers = max(1, task_cpus // duckdb_threads)

# set the environment variable for max duckdb threads in cytotable
os.environ["CYTOTABLE_MAX_THREADS"] = str(duckdb_threads)  # set before importing cytotable

# configure pyarrow to respect the same number of duckdb threads
pa.set_cpu_count(duckdb_threads)
pa.set_io_thread_count(duckdb_threads)

# import cytotable (which should inherit the above)
import cytotable

# create a custom Parsl HTEX configuration with the max workers
parsl_cfg = Config(
    executors=[HighThroughputExecutor(label="htex", max_workers_per_node=workers)]
)

# run convert with the custom config (other settings pass via environment variables or other internals)
cytotable.convert(..., parsl_config=parsl_cfg)

As a follow up question just in case this doesn't help (and also because I'm curious!):
What CPU and memory does your Nextflow process request per task, and what thread settings were you currently using for each layer (Parsl max_workers, CYTOTABLE_MAX_THREADS, and PyArrow CPU/IO threads)?

[jashapiro]

Thanks much for the guidance!

If I'm reading your pseudocode correctly, it seems like you would tend to recommend fewer duckdb threads and more workers for efficiency, or is that just meant as a starting point? (I'm planning to test some for sure.)

As a follow up question just in case this doesn't help (and also because I'm curious!):
What CPU and memory does your Nextflow process request per task, and what thread settings were you currently using for each layer (Parsl max_workers, CYTOTABLE_MAX_THREADS, and PyArrow CPU/IO threads)?

At first I was not setting any of the relevant parameters, so running convert() on my local machine was taking over completely... it did seem to run pretty fast though. I'll spend a bit of time doing some more tests, this time in docker containers with limited cpus to see if I can find the right balance with your recommendations as a starting point.

[jashapiro]

So after some testing, I have settled on the following settings, which seems to be working reasonably well to match the actual CPU usage to the external request, and is surprisingly simple: 1 worker per cpu core, and 2 threads for duckdb.
Almost anything with approximately 2 duckdb threads/core resulted in ~full CPU usage, but the higher worker count completed the job faster.

[d33bs]

That’s a really nice result—thanks for sharing what you landed on @jashapiro! I like the simplicity of it.

Curious if you noticed any tradeoffs in memory or IO as you increased the worker count, or if things stayed pretty stable?

Also, would you be open to us capturing this in the documentation—maybe under a "performance tuning" or "CPU/threading configuration" section as a recommended starting point?

[jashapiro]

Curious if you noticed any tradeoffs in memory or IO as you increased the worker count, or if things stayed pretty stable?

I didn't really track carefully; memory usage didn't seem to be an issue locally and for AWS instances the memory will scale with core count, so I wasn't too motivated to worry about it. I didn't look at IO either; it is quite possible that I might saturate IO at some point, but I didn't seem to reach that point, at least not judging by performance

Also, would you be open to us capturing this in the documentation—maybe under a "performance tuning" or "CPU/threading configuration" section as a recommended starting point?

Absolutely!