Skip to content

Document commands to use if jobs are held. #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ktlim wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Document commands to use if jobs are held. #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions doc/lsst.ctrl.bps.htcondor/userguide.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,22 @@ initially run with 2 GB of memory and failed because of exceeding the limit,
result the entire workflow fails again due to other reasons, the job will ask
for 2 GB of memory during the first execution after the workflow is restarted.

If you did not set the ``memoryMultiplier`` option, this command:

.. code-block:: sh

condor_q -hold $USER

will show any held jobs and the reasons for their being held, including running out of memory.

This command:

.. code-block:: sh

condor_q $USER | awk '{if($6 == "H") print "condor_qedit",$1,"RequestMemory 4096; condor_release",$1}' | bash

@MichelleGower MichelleGower May 19, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assumes all of the user's held jobs are from runs submitted without automatic tries and are held because of memory issues. But based on questions I've seen on slack this is more than likely the case. We can update this later to be more complicated but more accurate if actually need it.


will take any held jobs, change their requested memory size to 4096 MiB, and release them to be run again.

.. _htc-plugin-troubleshooting:

Troubleshooting
Expand Down