Local Python env Cluster (rebased)#9332
Conversation
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 40 files ± 0 40 suites ±0 14h 46m 47s ⏱️ + 11m 43s For more details on these failures, see this check. Results for commit 53dc38b. ± Comparison against base commit 40fcd99. ♻️ This comment has been updated with latest results. |
| '''ignore:unclosed transport <_SelectorSocketTransport.*:ResourceWarning''', | ||
| '''ignore:unclosed transport <asyncio\.sslproto\..*:ResourceWarning''', | ||
| '''ignore:unclosed cluster SSHCluster.*:ResourceWarning''', | ||
| '''ignore:unclosed transport <_UnixReadPipeTransport.*:ResourceWarning''', |
There was a problem hiding this comment.
These warnings were being thrown by test_job_submission. I'm assuming it's safe to ignore them.
|
Tests mostly passing:
The remaining linting ( |
|
It looks like subprocess support in Windows is only available using the I'll raise a |
|
Looking into subprocess support in Windows I saw that I think the only differences with
So perhaps a more lightweight change would be to simply make the python interpreter configurable in Pinging @crusaderky as I noticed you are currently working on |
| def __repr__(self): | ||
| return f"<LocalEnv {type(self).__name__}: status={self.status}>" | ||
|
|
||
| async def _set_env_helper(self): |
There was a problem hiding this comment.
This logic to forward config is taken from the SSHCluster. I'm not sure why it is necessary: if I see correctly, LocalEnv is meant to start workers/scheduler in a different venv on the same host. Assuming the processes using python in the second venv always run as the same user as the python interpreter that runs this code, why do we need to forward config? Won't the spawned workers get the same config anyway when they start up?
I'm not actively working on it. I tried, realised it was not a quick win, and abandoned the effort. |
|
Understood. Could someone let me know if this PR, or a more simple version that makes the command in |
2 participants
Rebase of #5013
I refer to explanation and discussion in that PR.
#5013 was never finished because merge conflicts appear. That seems a shame: the conflicts were relatively easy to solve. Opening this PR to see if the tests still pass.
I take no credit for @DPeterK's amazing work -- git adds me as a co-author for their commits because of the rebase, I hope that's ok.