Add `profile` feature to generic-worker

[ahal]

Problem

This expands on the idea originally put forth in:
taskcluster/taskcluster#7426

Currently, many tasks in Gecko have profiles that automatically get uploaded. You can even click a button in Treeherder to open the profiles in the Firefox Profiler. It's a great feature! The problem is that the thing responsible for the profile is mozsystemonitor in-tree. This means there are large swathes of the task that can't be profiled. E.g, anything that run-task does such as cloning the repo or downloading fetches. It also means it only for Gecko tasks.

We could find a way to run mozsystemmonitor in run-task, but that still has blind spots. Such as anything that generic-worker handles like fetching the docker image or setting up caches.

I'd like a way to get a profile of the entire task lifecycle. And I'd like it to be something that's available to all tasks in Firefox-CI. Regardless of which project they're in, and regardless of whether they use run-task or not.

This means we need to make profiling part of the platform.

Idea

The high level idea here is that generic-worker will provide a new feature that tasks can opt-in to, whereupon it'll enable sampling throughout the task's lifecycle, and emit a profile as an artifact upon task completion. The flow looks something like:

1. Task has features: {"profile": true} in the task definition
2. generic-worker starts sampling resource usage using something like gopsutil (CPU, memory, network IO etc)
3. generic-worker starts the task and creates lifecycle markers for things like downloading docker images, or setting up caches. This way we can track how long each of these steps take
4. When task is finished generic-worker outputs all profile data as an artifact

Custom markers

There's a big gap here though. Generic-worker has no visibility into the task payload, so it can't know e.g, how long a clone or a build took. For this it will need to delegate to the task itself. There are likely many ways to solve this, but one simple way would be for generic-worker to set a TASKCLUSTER_PROFILE_DIR variable. Then task processes can write their own markers in files like $TASKCLUSTER_PROFILE_DIR/<pid>.jsonl (one file per process to avoid racing).

Then, generic-worker simply has to merge in the JSON objects in these files to the final profile. In this way, run-task can emit data about how long a clone took. Or build scripts can emit data on how long various stages of the build were. The final profile will contain both the internal markers, and custom markers. As well as counters for resource usage (so the final profile can map various segments of the task to usage).

Profile Format

The format of the profiles should follow the Firefox Profiler format. While it has Firefox in the name, the format is actually agnostic to Mozilla. It's used by other browser vendors, and other companies that have nothing to do with browsers. It's well documented and several implementations exist (though unfortunately none in Go yet).

Yes, we partly pick this format because it's already what Mozilla uses.. But it's also a fantastic tool used across many companies and open source projects. So it makes a lot of sense for this use case.

Future Worker Profiling

In the future, it might be nice to be able to profile the entire worker lifecycle even between tasks. I think that should be out-of-scope for now, but I could imagine a daemon that runs on the workers sampling everything. If it's present, generic-worker would know to how to interface with it such that it can retrieve samples from it and include them in the per-task profiles it generates. We could emit per worker profiles somehow, e.g if we wanted to better track idle time between tasks.

Maybe it would make sense to go this route even if we don't do full worker sampling, and that way generic-worker could delegate resource usage monitoring to a different tool?

---

I'm interested in people's thoughts! If reception to the overall idea is generally positive, I can follow-up with a proper RFC that goes into more specifics on the implementation.

[ahal]

Over in taskcluster/taskcluster#7951 (comment), padenot mentioned https://github.com/padenot/prometheus-resourcemonitor as a tool that could do the sampling.

I'm unsure of how this and generic-worker could work together but it seems like there might be something there.

[bhearsum]

I think this is a great idea overall! And there's already some groundwork laid in generic-worker, eg: with relatively recent changes to monitor ram usage.

In this way, run-task can emit data about how long a clone took.

Does run-task (or anything else running in the payload) need to do any calculations? I would've thought that all they need to do is emit markers, which would then be overlayed on timings, resource usage, etc. (This seems like it would be preferable, as it would allow all of the timing and other data collection to be centralized and unified.)

[ahal]

That's a good question, and I'm not sure of the answer. I know mozsystemmonitor internally tracks the start/stop of a marker and only emits data on the call to stop:
https://searchfox.org/firefox-main/source/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py#1313

Whether that's due to a limitation in the Firefox Profiler data format or some other reason I'm not sure.

[Callek]

I was just talking about this in last weeks community meeting. My deployment of taskcluster doesn't use taskgraph or run-task, and my tasks are not suitable for the mozsystemmonitor. However we have a strong desire to be able to see data about the task resource-usage throughout its run. (We sometimes have issues with OOM, and such, - which we currently solve by increasing the machine type for the pool)

Having this data would allow us to better right-size the worker pools for the task types that run on them.

Our logs are not formatted in a way that currently gives great data to the current task profiler feature in taskcluster, but if this system monitoring profile was built in, and taskcluster's UX could easily open the task profile with this extra data, it would be hugely useful to us.

[ahal]

Tbh, I think Taskcluster's UI opening the profiles should be a non-goal, there's nothing that will beat the Firefox Profiler for this. But if you have a Firefox Profiler artifact, you can simply pass the URL to it here:
https://profiler.firefox.com/

If your profile artifacts are private, then you can either upload profiles from your local machine, or if they're sensitive and you don't want to do that.. you might be able to run your own instance of the Firefox Profiler (but I'm just assuming :p)

[Callek]

Tbh, I think Taskcluster's UI opening the profiles should be a non-goal, there's nothing that will beat the Firefox Profiler for this. But if you have a Firefox Profiler artifact, you can simply pass the URL to it here: https://profiler.firefox.com/

To be clear, Taskcluster already has UI for opening a profile based on the task. which opens a view like this fxci task today

I'm suggesting that if a taskcluster feature exposes profiler information (even if its stored as an artifact) then it should use that information when it passes its data along to the Firefox Profiler, as it already does for the rest of the log.

I'm not suggesting that Taskcluster itself should be the viewer for the profile.