diff --git a/.gitignore b/.gitignore index cb4c6d027..20a4572d9 100644 --- a/.gitignore +++ b/.gitignore @@ -41,3 +41,4 @@ book.pdf *~ build venv +.DS_Store diff --git a/README.md b/README.md index 1f4d9c259..3c8a78753 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ If you'd just like to learn about how to use the FCC Software, ```{eval-rst} .. toctree:: - :numbered: + :numbered: 4 :maxdepth: 2 :includehidden: :caption: Contents: @@ -49,14 +49,13 @@ If you'd just like to learn about how to use the FCC Software, CONTRIBUTING.md .. toctree:: - :numbered: - :maxdepth: 2 :includehidden: :caption: External links: FCC Software + Key4hep FCC-ee Detector Full Sim FCCAnalyses - Analysis essentials + HSF Analysis essentials FCC Software glossary ``` diff --git a/distributed-computing/README.md b/distributed-computing/README.md index 7a1ee98bc..72e9c15dc 100644 --- a/distributed-computing/README.md +++ b/distributed-computing/README.md @@ -1,20 +1,28 @@ # Distributed computing -These pages provide --- and dissect --- examples of workflows to be run on distributed resources with the [DIRAC Interware system][dirac] -through [iLCDirac][ilcdirac], the extension developed by the Linear Collider community and used also by CALICE. +> +> Original author: Gerardo Ganis +> -The procedure to be enabled to use the FCC resources through DIRAC is first described. Additional information about the use of -iLCDirac can be found in the [CLIC][wikiclic] and [ILC][wikiilc] dedicated Wiki pages. +These pages provide — and dissect — examples of workflows to be run +on distributed resources with the [DIRAC Interware system][dirac] through +[iLCDirac][ilcdirac], the extension developed by the Linear Collider community +and used also by [CALICE][calice]. + +The procedure how to use the FCC resources through DIRAC is described. +Additional information about the use of iLCDirac can be found in the +dedicated [iLCDirac Documentation][ilcdirac-docs]. Unless specified, in the rest of this section the word `DIRAC` refers to the `iLCDirac` extension introduced above. [dirac]: https://dirac.readthedocs.io/en/latest/ [ilcdirac]: https://iopscience.iop.org/article/10.1088/1742-6596/513/3/032077/meta -[wikiclic]: https://twiki.cern.ch/twiki/bin/view/CLIC/DiracForUsers -[wikiilc]: https://flcwiki.desy.de/ILCDirac +[calice]: https://twiki.cern.ch/twiki/bin/view/CALICE/ +[ilcdirac-docs]: https://ilcdirac-doc.web.cern.ch/index.html ```{eval-rst} .. toctree:: + :maxdepth: 2 :caption: Contents: RegisteringToFccVO.md diff --git a/distributed-computing/RegisteringToFccVO.md b/distributed-computing/RegisteringToFccVO.md index b9139d663..3204c9e17 100644 --- a/distributed-computing/RegisteringToFccVO.md +++ b/distributed-computing/RegisteringToFccVO.md @@ -1,24 +1,30 @@ # Getting started with FCC distributed computing +> +> Original author: Gerardo Ganis +> + +For more detailed documentation how user's can use the distributed resources +managed by iLCDirac, refer to the iLCDirac's +[User Guide](https://ilcdirac-doc.web.cern.ch/DOC/Files/UserGuide/index.html). + ## Registering to the FCC VO -The [standard Grid VO registration procedure][signup] -should be followed to be enable to use the resources connected with the FCC VO. +The [standard Grid VO registration procedure][signup] should be followed to be +able to use the resources connected with the FCC Virtual Organization (VO). :::{admonition} Note :class: callout -You need to use a browser where you have installed your certificate and the -[CERN CA][cernca] certificates. Firefox usually works fine, Google Chrome -usually does not work. Safari might also work. +As for the VO registration, you need to use a browser where you can import your +GRID certificate together with the [CERN CA][cernca] certificates. Firefox +usually works fine, Google Chrome usually does not work. Safari might also work. ::: -[signup]: https://fcc-auth.cern.ch/dashboard#!/newClient - ## Enabling DIRAC DIRAC is available on CernVM-FS. To enable the relevant applications and -scripts, the following setup script needs to be sourced first +scripts, the following setup script needs first to be sourced ```bash source /cvmfs/clicdp.cern.ch/DIRAC/bashrc @@ -29,7 +35,9 @@ To submit jobs through DIRAC a proxy needs to be created and uploaded: ```bash dirac-proxy-init -g fcc_user ``` + A successful creation looks like this: + ``` Generating proxy... Enter Certificate password: @@ -51,11 +59,15 @@ Proxies uploaded: DN | Group | Until (GMT) /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ganis/CN=393971/CN=Gerardo Ganis | | 2022/05/13 12:12 ``` + The last section shows the valid proxies upload to the DIRAC system. It can also be checked with + ```bash dirac-proxy-info -m ``` + with output similar to + ``` subject : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ganis/CN=393971/CN=Gerardo Ganis/CN=2178341058/CN=3000266373 issuer : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ganis/CN=393971/CN=Gerardo Ganis/CN=2178341058 @@ -78,12 +90,14 @@ If everything worked fine, your proxy should be mapped to the `fcc001` user. Thi export EOS_MGM_URL=root://eospublic.cern.ch XrdSecPROTOCOL=gsi,unix eos whoami ``` + the result should look similar to this: + ``` Virtual Identity: uid=140035 (99,140035) gid=2855 (99,2855) [authz:gsi] host=lxplus743.cern.ch domain=cern.ch geo-location=0513 ``` -At CERN the uid of `fcc001` is 140035. +At CERN the uid of `fcc001` is `140035`. ## Copying, browsing, accessing files @@ -107,7 +121,7 @@ $ ls -lt /eos/experiment/fcc/prod/fcc/user/g/ganis total 9545 -rw-r--r--. 1 fcc001 fcc-cg 9768981 Nov 3 2021 edm4hep_test_output.root ``` -Files can be replicated to another STorage Element, e.g. CNAF-DISK, using +Files can be replicated to another Storage Element, e.g. CNAF-DISK, using `dirac-dms-replicate-lfn`: ``` $ dirac-dms-replicate-lfn /fcc/user/g/ganis/edm4hep_test_output.root CNAF-DISK @@ -134,17 +148,13 @@ root [1] ## The web portal -The [DIRAC web portal][diracweb] is available to check the status of things. It shows all the jobs submitted and the -files registered. +The [DIRAC web portal][diracweb] is available to check the status of things. It +shows all the jobs submitted and the files registered. An example screenshot +is shown below. +![ILCDirac Web](images/dirac-web-job-listing.png) -:::{admonition} Note -:class: callout - -As for the VO registration, you need to use a browser where you have installed -your certificate and the [CERN CA][cernca] certificates. Firefox usually works -fine, Safari might also work. Google Chrome usually does not work. -::: -[diracweb]: https://voilcdiracwebapp2.cern.ch/DIRAC/?view=tabs&theme=Crisp&url_state=1|*DIRAC.JobMonitor.classes.JobMonitor +[signup]: https://fcc-auth.cern.ch/start-registration +[diracweb]: https://voilcdiracwebapp2.cern.ch/DIRAC/?theme=Classic&url_state=1|*DIRAC.JobMonitor.classes.JobMonitor:, [cernca]: https://ca.cern.ch/cafiles/certificates/Grid.aspx diff --git a/distributed-computing/Workflows.md b/distributed-computing/Workflows.md index c090fd2c6..fa23e596c 100644 --- a/distributed-computing/Workflows.md +++ b/distributed-computing/Workflows.md @@ -1,14 +1,21 @@ # FCC DIRAC example workflows -These pages provide - and dissect - examples of workflows to be run on distributed resources with the [DIRAC Interware system][dirac]. +> +> Original author: Gerardo Ganis +> -The python scripts discussed in these pages are available at [https://github.com/HEP-FCC/FCCDIRAC][fccdirac]. +These pages provide — and dissect — examples of workflows to be run +on distributed resources of FCC with the [DIRAC Interware system][dirac]. + +The python scripts discussed in these pages are available at +[https://github.com/HEP-FCC/FCCDIRAC][fccdirac]. [dirac]: https://dirac.readthedocs.io/en/latest/ -[fccdirac]: https://github.com/HEP-FCC/FCCDIRAC +[fccdirac]: https://github.com/HEP-FCC/FCCDIRAC/tree/master/workflows ```{eval-rst} .. toctree:: + :maxdepth: 2 :caption: Contents: workflows/Overview.md diff --git a/distributed-computing/images/dirac-web-job-listing.png b/distributed-computing/images/dirac-web-job-listing.png new file mode 100644 index 000000000..cf7cd58d3 Binary files /dev/null and b/distributed-computing/images/dirac-web-job-listing.png differ diff --git a/distributed-computing/workflows/01/DiTauKKMCeeDelphesStandAlone.md b/distributed-computing/workflows/01/DiTauKKMCeeDelphesStandAlone.md index a5458c99f..a9a2d7786 100644 --- a/distributed-computing/workflows/01/DiTauKKMCeeDelphesStandAlone.md +++ b/distributed-computing/workflows/01/DiTauKKMCeeDelphesStandAlone.md @@ -1,15 +1,26 @@ -# Workflow 1 +# Workflow 1: Ditau with KKMCee + Delphes -## Purpose +> +> Original author: Gerardo Ganis +> -Demonstrate the use of DIRAC generate 10000 tau+tau- events @91.2 GeV with KKMC and process through Delphes/IDEA. +:::{admonition} Purpose +:class: objectives -## Output files +Demonstrate the use of DIRAC to generate 10'000 $\tau^{+}\tau^{-}$ events +@ $91.2~\mathrm{GeV}$ with KKMC and pass them through the IDEA detector with the +help of Delphes framework. +::: -**Local mode**
-The output file is located under a sub-directory name `Local__JobDir` created under the current directory. +## Location of the output files -**WMS mode**
+The placement of the job output files depends on where the job runs. + +**Local mode** +The output file is located under a sub-directory name `Local__JobDir` +created under the current directory. + +**DIRAC WMS mode** In WMS mode the file will be located under ``` @@ -22,36 +33,41 @@ e.g. at CERN (storage element `CERN-DST-EOS`): /eos/experiment/fcc/prod/fcc/user/g/ganis/2021_07/59821/59821752/edm4hep_test_output.root ``` + ## DIRAC components involved -This exercise constist of two steps, the event generation with `KKMCee` and the `Delphes` simulation. +This exercise consists of two steps, the event generation with [KKMCee][kkmcee] +and afterwards the simulation of the detector response with [Delphes][delphes]. -For the first step we need the `KKMC` DIRAC application which we configure with the process, the number of events, the energy and -the name of the output file. +For the first step we need the [KKMC DIRAC application][kkmc-dirac-app] which +we configure with the process, the number of events, the energy and the name of +the output file. -The second step consists in running the `DelphesPythia8_EDM4HEP` standalone application with arguments +The second step consists of running the standalone `DelphesPythia8_EDM4HEP` +executable from the Key4hep stack as an +[DIRAC Generic Application][generic-dirac-app] while manually passing all the +needed arguments: 1. The `Delphes` card implementing the IDEA detector concept; 2. The definition of the EDM4hep output; -3. The pythia card reading LHE file formats; +3. The Pythia card reading LHE file formats; 4. The output file. For this we use the generic application DIRAC interface. -## The script, dissected +## The workflow script, dissected -The submission script for this workflow is called [FCC_Dirac_Workflow1.py][workflow1-py]. -The script accepts one argument to control where the job is executed: +The submission script for this workflow is called +[FCC_Dirac_Workflow1.py][workflow1-py]. The script accepts only one argument to +control where the job is executed: -``` +```bash python FCC_Dirac_Workflow1.py -h ``` ``` - Usage: - - FCC_Dirac_Workflow1.py (|)* + FCC-Dirac-Workflow1 [options] ... General options: -o --option : Option=value to add @@ -64,295 +80,468 @@ General options: -h --help : Shows this help Options: - -w --wms : WMS where to run + -w --wms : Run on DIRAC WMS + -l --local : Run locally ``` -[workflow1-py]: https://raw.githubusercontent.com/HEP-FCC/FCCDIRAC/master/workflows/1/FCC_Dirac_Workflow1.py -### The utility function +### Helper copy function -```python -# Create sandbox files -import os -from shutil import copy2 -import array +In order to ease the preparation of the job input files a helper function, which +aids with string replacements is defined at the top of the script. -# Utility function -def copywithreplace(filein, fileout, repls): +```python +def copywithreplace(filein: str, fileout: str, + repls: list[tuple[str, str]] = []): + ''' + Copy the contents of the file with possible string replacements. + ''' # If no replacements, just copy the file - if len(repls) == 0: + if repls is None: copy2(filein, fileout) return - # input file - fin = open(filein, "rt") - # output file to write the result to - fout = open(fileout, "wt") - # for each line in the input file - for line in fin: - # Apply each requested replacement - ltmp = line - for r in repls: - lout = ltmp.replace(str(r[0]), str(r[1])) - ltmp = lout - fout.write(lout) - # close input and output files - fin.close() - fout.close() + + # Load the contents of the input file + with open(filein, 'rt', encoding='utf-8') as infile: + # open the output file to write the result to + lines = infile.readlines() + with open(fileout, 'wt', encoding='utf-8') as outfile: + # for each line in the input file + for line in lines: + # Apply each requested replacement + lout = line + for rpl in repls: + lout = lout.replace(str(rpl[0]), str(rpl[1])) + outfile.write(lout) ``` + ### Adding the `--wms` switch +As described in the [](../Overview.md) we define a simple switch to have +command line control over where the job will be executed. + ```python -from DIRAC import S_OK, S_ERROR +from DIRAC import gLogger, S_OK from DIRAC.Core.Base import Script +... + # Define a simple class to hold the script parameters -class Params(object): - def __init__(self): - self.wms = 'wms' - def setWMS(self, value): - self.wms = value - return S_OK() +class Params: + def __init__(self): + self.where = 'local' + def run_on_wms(self, _): + self.where = 'wms' + return S_OK() + def run_locally(self, _): + self.where = 'local' + return S_OK() -# Instantiate the params class -cliParams = Params() -Script.registerSwitch('w', 'wms', "WMS where to run", cliParams.setWMS) -Script.parseCommandLine(ignoreErrors=False) -# Get the list of services (the switch above appearer as servicesList[0]) -servicesList = Script.getPositionalArgs() -print servicesList -``` +... -### The DIRAC API instance + # Setup argument parsing + cli_params = Params() -```python -from ILCDIRAC.Interfaces.API.DiracILC import DiracILC + Script.registerSwitch('w', 'wms', 'Run on DIRAC WMS', + cli_params.run_on_wms) + Script.registerSwitch('l', 'local', 'Run locally', + cli_params.run_locally) -dIlc = DiracILC() + Script.parseCommandLine() ``` + ### The DIRAC Job manager ```python from ILCDIRAC.Interfaces.API.NewInterface.UserJob import UserJob +... + job = UserJob() job.setOutputSandbox(['*.log', '*.sh', '*.py', '*.xml']) outputdatafile='kktautau_delphes_edm4hep_output.root' -job.setOutputData(outputdatafile, '','CERN-DST-EOS' ) +job.setOutputData(outputdatafile, '', 'CERN-DST-EOS' ) -job.setJobGroup( "KKMC_EDM4HEP_Run" ) -job.setName( "KKMC_EDM4HEP" ) +job.setJobGroup("KKMC_EDM4HEP_Run") +job.setName("KKMC_EDM4HEP") job.setLogLevel("DEBUG") ``` ### The `KKMC` application instance +The [KKMC DIRAC application][kkmc-dirac-app] requires a few settings. + ```python -from ILCDIRAC.Interfaces.API.NewInterface.Applications import KKMC +from ILCDIRAC.Interfaces.API.NewInterface import Applications -kkmc = KKMC() -kkmc.setVersion('Key4hep-2021-04-30') -kkmc.setEvtType('Tau') -kkmc.setEnergy(91.2) -nevts = 10000 -outputfile = 'kktautau_delphes_' + str(nevts) + '.LHE' -kkmc.setNumberOfEvents(nevts) -kkmc.setOutputFile(outputfile) +... + + kkmc_app = Applications.KKMC() + kkmc_app.setVersion('key4hep_250128') + kkmc_app.setEvtType('Tau') + kkmc_app.setEnergy(91.2) + nevts = 10000 + outputfile = f'kktautau_delphes_{nevts}.lhe' + kkmc_app.setNumberOfEvents(nevts) + kkmc_app.setOutputFile(outputfile) -job.append(kkmc) + # Register KKMC application to the Job instance + job.append(kkmc_app) ``` -### The steering files for `Delphes` + +### Preparing the `Delphes` input cards + +First, all the needed input cards need to be prepared in the submission +directory. In this example workflow the cards are provided alongside the +submission script and are copied (and adjusted) to the submission directory. ```python -# Delphes card -delphescardpath=os.path.expandvars('$DELPHES/cards/delphes_card_IDEA.tcl') -delphescard=os.path.basename(delphescardpath) -copy2(delphescardpath, delphescard) -# EDM4hep output definition -edm4hepoutdefpath=os.path.expandvars('$K4SIMDELPHES/edm4hep_output_config.tcl') -edm4hepoutdef=os.path.basename(edm4hepoutdefpath) -copy2(edm4hepoutdefpath, edm4hepoutdef) -# Pythia card -pythiacardpath=os.path.expandvars('$K4GEN/Pythia_LHEinput.cmd') -pythiacard=os.path.basename(pythiacardpath) -replacements = [['Main:numberOfEvents = 100','Main:numberOfEvents = ' + str(nevts)], - ['Beams:LHEF = Generation/data/events.lhe','Beams:LHEF = ' + outputfile]] -copywithreplace(pythiacardpath, pythiacard, replacements) + script_dir = os.path.dirname(os.path.realpath(__file__)) + + # Delphes IDEA card + # copy of $DELPHES/cards/delphes_card_IDEA.tcl + idea_card = 'delphes_card_IDEA.tcl' + copy2(os.path.join(script_dir, idea_card), idea_card) + # Delphes EDM4hep output card + # copy of $K4SIMDELPHES/edm4hep_output_config.tcl + edm4hep_output_def = 'edm4hep_output_config.tcl' + copy2(os.path.join(script_dir, edm4hep_output_def), edm4hep_output_def) + # Pythia card + # copy of $K4GEN/Pythia_LHEinput.cmd + pythia_card = 'Pythia_LHEinput.cmd' + replacements = [ + ('Main:numberOfEvents = 100', + f'Main:numberOfEvents = {nevts}'), + ('Beams:LHEF = Generation/data/events.lhe', + f'Beams:LHEF = {outputfile}') + ] + copywithreplace(os.path.join(script_dir, pythia_card), pythia_card, + replacements) ``` + ### Completing the sandbox for `Delphes` +In order for the job to be able to pick up our input cards we need to register +them for the input sandbox. + ```python -# Set the sandbox content -job.setInputSandbox(['./' + delphescard, './' + edm4hepoutdef, './' + pythiacard]) + # Set the sandbox content + job.setInputSandbox( + ['./' + idea_card, './' + edm4hep_output_def, './' + pythia_card] + ) ``` -### Standalone `Delphes` using the DIRAC generic application +### Standalone `Delphes` using the generic DIRAC application ```python -from ILCDIRAC.Interfaces.API.NewInterface.Applications import GenericApplication +from ILCDIRAC.Interfaces.API.NewInterface import Applications -ga = GenericApplication() -ga.setSetupScript("/cvmfs/sw.hsf.org/spackages2/key4hep-stack/2021-04-30/x86_64-centos7-gcc8.3.0-opt/t5gcd6ltt2ikybap2ndoztsg5uyorxzg/setup.sh") -ga.setScript("/cvmfs/sw.hsf.org/spackages2/k4simdelphes/00-01-05/x86_64-centos7-gcc8.3.0-opt/beesqo4r5wuqrrijyz57kxbqcdp5pp4v/bin/DelphesPythia8_EDM4HEP") -ga.setArguments(delphescard + ' ' + edm4hepoutdef + ' ' + pythiacard + ' ' + outputdatafile) +... -job.append(ga) + ga = Applications.GenericApplication() + ga.setSetupScript( + '/cvmfs/sw.hsf.org/key4hep/releases/2025-01-28/' + 'x86_64-almalinux9-gcc14.2.0-opt/key4hep-stack/' + '2025-01-28-q6hyek/setup.sh' + ) + ga.setScript( + '/cvmfs/sw.hsf.org/key4hep/releases/2025-01-28/' + 'x86_64-almalinux9-gcc14.2.0-opt/k4simdelphes/00-07-04-naw5vm/' + 'bin/DelphesPythia8_EDM4HEP' + ) + ga.setArguments( + f'{idea_card} {edm4hep_output_def} {pythia_card} {outputdatafile}' + ) + + job.append(ga) ``` -### Submitting the job to the chosen WMS + +### Submitting the job ```python -submitmode='wms' -if len(servicesList) > 0: - submitmode= servicesList[0] -print job.submit(dIlc, mode=submitmode) +from ILCDIRAC.Interfaces.API.DiracILC import DiracILC + +... + + dilc = DiracILC() + print(job.submit(dilc, mode=cli_params.where)) ``` -## Running the script on lxplus -Suggestion is, after having cloned the repository and initialized the environment, to go to the workflow sub-directory, created a 'run' sub-directory and run -from there: +## Running the job submission script -``` -$ cd workflow/1 -$ mkdir run; cd run +After having cloned the repository and initialized the environment, go to the +workflow sub-directory, create a submission directory `run` and launch the +submission script from there: + +```bash +cd workflow/1 +mkdir run; cd run ``` ### Local submission +Run the submission script with the `--local` switch + ```bash -$ python ../FCC_Dirac_Workflow1.py --wms local +python ../FCC_Dirac_Workflow1.py --local ``` +The summary of the job will be printed for the inspection, afterwards confirm or +decline the submission. The whole output should look similar to this: + ``` -['local'] -kkmc Key4hep-2021-04-30 +Submitting Example Workflow 1 + - execution location: local +kkmc key4hep_250128 Attribute list : - forgetAboutInput: Not defined - randomSeed: -1 - outputSE: Not defined - seedFile: Not defined + ... + version: key4hep_250128 + steeringFile: Not defined + inputFile: Not defined + outputFile: kktautau_delphes_10000.lhe + ... + logFile: kkmc_key4hep_250128_Step_1.log + ... + numberOfEvents: 10000 energy: 91.2 ... - logFile: kkmc_Key4hep-2021-04-30_Step_1.log + _extension: hepmc + ApplicationScript Attribute list : - forgetAboutInput: Not defined - outputSE: Not defined - energy: 91.2 + script: /cvmfs/sw.hsf.org/key4hep/releases/2025-01-28/x86_64-almalinux9-gcc14.2.0-opt/k4simdelphes/00-07-04-naw5vm/bin/DelphesPythia8_EDM4HEP + setupScript: /cvmfs/sw.hsf.org/key4hep/releases/2025-01-28/x86_64-almalinux9-gcc14.2.0-opt/key4hep-stack/2025-01-28-q6hyek/setup.sh + arguments: delphes_card_IDEA.tcl edm4hep_output_config.tcl Pythia_LHEinput.cmd kktautau_delphes_edm4hep_output.root ... logFile: ApplicationScript_Step_2.log + ... + energy: 91.2 + ... Proceed and submit job(s)? y/[n] : y -[long output] +... + +... -2021-07-28 16:27:53 UTC dirac-jobexec/ILCDIRAC.Workflow.Modules.UserJobFinalization INFO: GUID = CCADDA80-3B2E-7E9C-3C14-5A493AB48BD4 -2021-07-28 16:27:53 UTC dirac-jobexec DEBUG: Workflow execution successful, exiting +2026-01-20 09:29:32 UTC dirac-jobexec DEBUG: Workflow execution successful, exiting +Standard output written to std.out {'OK': True, 'Value': 'Execution completed successfully'} ``` -The local sandbox should contain the following: +After the execution the submission directory should contain the following files: -``` -$ ls -lt -``` -``` -total 32 -drwx------. 2 ganis sf 2048 Jul 28 18:27 Local_pQl06k_JobDir --rw-r--r--. 1 ganis sf 1483 Jul 28 18:25 Pythia_LHEinput.cmd --rw-r--r--. 1 ganis sf 587 May 10 15:31 edm4hep_output_config.tcl --rw-r--r--. 1 ganis sf 27219 Apr 30 14:50 delphes_card_IDEA.tcl +```bash +ls -lt ``` -and the output directory: - ``` -$ ls -lt Local_pQl06k_JobDir/kktautau* --rw-r--r--. 1 ganis sf 9642099 Jul 28 18:27 Local_pQl06k_JobDir/kktautau_delphes_edm4hep_output.root --rw-r--r--. 1 ganis sf 32586567 Jul 28 18:27 Local_pQl06k_JobDir/kktautau_delphes_10000.LHE -$ ls -lt Local_pQl06k_JobDir/*.log --rw-r--r--. 1 ganis sf 257109 Jul 28 18:27 Local_pQl06k_JobDir/ApplicationScript_Step_2.log --rw-r--r--. 1 ganis sf 44250 Jul 28 18:27 Local_pQl06k_JobDir/kkmc_Key4hep-2021-04-30_Step_1.log --rw-r--r--. 1 ganis sf 900532 Jul 28 18:27 Local_pQl06k_JobDir/localEnv.log +total 44 +drwx------. 3 jsmiesko jsmiesko 4096 Jan 20 13:31 Local_dp5xm4kz_JobDir +-rw-r--r--. 1 jsmiesko jsmiesko 1483 Jan 20 13:30 Pythia_LHEinput.cmd +-rw-r--r--. 1 jsmiesko jsmiesko 580 Jan 20 09:49 edm4hep_output_config.tcl +-rw-r--r--. 1 jsmiesko jsmiesko 30239 Jan 20 09:49 delphes_card_IDEA.tcl ``` -### WMS submission +in this example workflow it should contain the input cards and the local output +directory: ```bash -python ../FCC_Dirac_Workflow1.py +ls -lt Local_dp5xm4kz_JobDir/ +``` +```txt +total 39712 +-rw-r--r--. 1 jsmiesko jsmiesko 1 Jan 20 13:31 std.err +-rw-r--r--. 1 jsmiesko jsmiesko 1 Jan 20 13:31 std.out +-rw-r--r--. 1 jsmiesko jsmiesko 32268 Jan 20 13:31 ApplicationScript_Step_2.log +-rw-r--r--. 1 jsmiesko jsmiesko 135662 Jan 20 13:31 kktautau_delphes_edm4hep_output.root +-rw-r--r--. 1 jsmiesko jsmiesko 318074 Jan 20 13:30 localEnv.log +-rw-r--r--. 1 jsmiesko jsmiesko 176236 Jan 20 13:30 kkmc_key4hep_250128_Step_1.log +-rw-r--r--. 1 jsmiesko jsmiesko 19609219 Jan 20 13:30 events.hepmc +drwxr-xr-x. 2 jsmiesko jsmiesko 146 Jan 20 13:30 KKMCee-20Jan2026-130159 +-rw-r--r--. 1 jsmiesko jsmiesko 19609219 Jan 20 13:30 kktautau_delphes_10000.lhe +-rwxr-xr-x. 1 jsmiesko jsmiesko 613 Jan 20 13:30 kkmc_key4hep_250128_Run_1.sh +-rwxr-xr-x. 1 jsmiesko jsmiesko 446528 Jan 20 13:30 DelphesPythia8_EDM4HEP +-rw-r--r--. 1 jsmiesko jsmiesko 154244 Jan 20 13:30 setup.sh +-rw-r--r--. 1 jsmiesko jsmiesko 580 Jan 20 13:30 edm4hep_output_config.tcl +-rw-r--r--. 1 jsmiesko jsmiesko 1483 Jan 20 13:30 Pythia_LHEinput.cmd +-rw-r--r--. 1 jsmiesko jsmiesko 30239 Jan 20 13:30 delphes_card_IDEA.tcl +-rw-r--r--. 1 jsmiesko jsmiesko 18882 Jan 20 13:30 jobDescription.xml +``` + + +### DIRAC WMS submission + +Run the submission script with the `--wms` switch +```bash +python ../FCC_Dirac_Workflow1.py --wms ``` +The initial output will be similar to the local mode, but after confirming the +submission you should get JSON dump which should contain the job ID. ``` ... Proceed and submit job(s)? y/[n] : y ... -'Value': 59838136, 'JobID': 59838136} +{'OK': True, 'Value': 66251838, 'JobID': 66251838, ... ``` -The `JobID` defines uniquely the job and can be used for any operation, for example to check the status: +#### Monitoring the submitted job -``` -dirac-wms-job-status 59838136 -``` +The `JobID` defines uniquely the job and can be used, for example, to check the +status of the job with `dirac-wms-job-status`: +```bash +dirac-wms-job-status 66251838 ``` -JobID=59838136 Status=Waiting; MinorStatus=Pilot Agent Submission; Site=ANY; + +```txt +JobID=66251838 ApplicationStatus=Unknown; MinorStatus=Pilot Agent Submission; Status=Waiting; Site=ANY; ``` -or, when the job is finished, get the job files: +To more closely monitor the execution of the job one can also use +`dirac-wms-job-logging-info`: +```bash +dirac-wms-job-logging-info 66251838 +``` +```txt +Source Status MinorStatus ApplicationStatus DateTime +================================================================================================= +JobManager Received Job accepted Unknown 2026-01-20 10:31:01 +JobPath Checking JobSanity Unknown 2026-01-20 10:31:01 +JobSanity Checking SoftwareVersions Unknown 2026-01-20 10:31:01 +SoftwareVersions SoftwareCheck Done Unknown 2026-01-20 10:31:01 +SoftwareVersions Checking JobScheduling Unknown 2026-01-20 10:31:01 +JobScheduling Waiting Pilot Agent Submission Unknown 2026-01-20 10:31:02 +... +... ``` -$ dirac-wms-job-get-output 59838136 + +#### Retrieving job output + +When the job is finished, one can get the job output files (includes job +description, log files, application scripts, ...): + +```bash +dirac-wms-job-get-output 66251838 ``` + +```txt +Files retrieved and extracted in /home/jsmiesko/dirac-workflows/FCCDIRAC/workflows/1/run2/66251838 +Job output sandbox retrieved in /home/jsmiesko/dirac-workflows/FCCDIRAC/workflows/1/run2/66251838/ ``` -Job output sandbox retrieved in /afs/cern.ch/user/g/ganis/local/dirac/GIT/FCCDIRAC/workflows/1/run/59838136/ -$ ls -lt 59838136/ -total 1295 --rw-r--r--. 1 ganis sf 273338 Jul 28 19:13 std.out --rw-r--r--. 1 ganis sf 38583 Jul 28 19:13 std.err --rw-r--r--. 1 ganis sf 256445 Jul 28 19:13 ApplicationScript_Step_2.log --rw-r--r--. 1 ganis sf 464310 Jul 28 19:12 localEnv.log --rw-r--r--. 1 ganis sf 43690 Jul 28 19:12 kkmc_Key4hep-2021-04-30_Step_1.log --rwxr-xr-x. 1 ganis sf 559 Jul 28 19:12 kkmc_Key4hep-2021-04-30_Run_1.sh --rw-r--r--. 1 ganis sf 227185 Apr 30 15:47 setup.sh --rw-r--r--. 1 ganis sf 19080 Jan 1 1970 jobDescription.xml + +```bash +ls -lt 66251838/ ``` -or get the output data: ``` -$ dirac-wms-job-get-output-data 59838136 +total 832 +-rw-r--r--. 1 jsmiesko jsmiesko 109703 Jan 20 11:37 std.out +-rw-r--r--. 1 jsmiesko jsmiesko 26159 Jan 20 11:37 std.err +-rw-r--r--. 1 jsmiesko jsmiesko 32382 Jan 20 11:37 ApplicationScript_Step_2.log +-rw-r--r--. 1 jsmiesko jsmiesko 311550 Jan 20 11:37 localEnv.log +-rw-r--r--. 1 jsmiesko jsmiesko 183261 Jan 20 11:37 kkmc_key4hep_250128_Step_1.log +-rwxr-xr-x. 1 jsmiesko jsmiesko 620 Jan 20 11:37 kkmc_key4hep_250128_Run_1.sh +-rw-r--r--. 1 jsmiesko jsmiesko 154244 Jan 28 2025 setup.sh +-rw-r--r--. 1 jsmiesko jsmiesko 18882 Jan 1 1970 jobDescription.xml ``` -``` -Job 59838136 output data retrieved -$ ls -lt kktautau_delphes_edm4hep_output.root --rwxr-xr-x. 1 ganis sf 9652641 Jul 29 11:30 kktautau_delphes_edm4hep_output.root +#### Accessing the job results + +The files with the job results (ROOT files, LHE files, ...) can be retrieved +using the `dirac-wms-job-get-output-data`: + +```bash +dirac-wms-job-get-output-data 66251838 ``` -The output data are also available on storage element: +```txt +Attempting to retrieve /fcc/user/j/jsmiesko/2026_01/66251/66251838/kktautau_delphes_edm4hep_output.root +Trying to download root://x509up_u1000@eospublic.cern.ch//eos/experiment/fcc/prod/fcc/user/j/jsmiesko/2026_01/66251/66251838/kktautau_delphes_edm4hep_output.root to /home/jsmiesko/dirac-workflows/FCCDIRAC/workflows/1/run2/kktautau_delphes_edm4hep_output.root +Job 66251838 output data retrieved +``` +Now the resulting root file should be located in your local directory: +```bash +ls -lt kktautau_delphes_edm4hep_output.root ``` -$ ls -lt /eos/experiment/fcc/prod/fcc/user/g/ganis/2021_07/59838/59838136/ + +```txt +-rw-r--r--. 1 jsmiesko jsmiesko 135662 Jan 20 13:52 kktautau_delphes_edm4hep_output.root ``` + +Since this example workflow defines `CERN-DST-EOS` Storage Element as its +output data location, we can also access the resulting file through EOS at CERN: +```bash +ls -lt /eos/experiment/fcc/prod/fcc/user/j/jsmiesko/2026_01/66251/66251838/ ``` -total 9427 --rw-r--r--. 1 fcc001 fcc-cg 9652641 Jul 28 19:13 kktautau_delphes_edm4hep_output.root + +```txt +total 133 +-rw-r--r--. 1 140035 2855 135662 Jan 20 11:37 kktautau_delphes_edm4hep_output.root ``` -The job id of the user jobs get also be retrieved with the `dirac-wms-select-jobs` command, e.g. +#### Listing jobs of the user + +The listing of the jobs of the user can be retrieved with the +`dirac-wms-select-jobs` command, e.g. + ``` -$ dirac-wms-select-jobs --Date=2021-07-28 --Owner="ganis" +dirac-wms-select-jobs --Date=2021-07-28 --Owner="ganis" ``` + ``` ==> Selected 1 jobs with conditions: Date = 2021-07-28, Owner = ganis 59838136 ``` -or from the web portal. +All possible selection options can be listed by providing `--help` argument +```txt +dirac-wms-select-jobs --help + +Select DIRAC jobs matching the given conditions + + +Usage: + dirac-wms-select-jobs [options] ... + +General options: + -o --option : Option=value to add + -s --section : Set base section for relative parsed options + -c --cert : Use server certificate to connect to Core Services + -d --debug : Set debug mode (-ddd is extra debug) + - --cfg= : Load additional config file + - --autoreload : Automatically restart if there's any change in the module + - --license : Show DIRAC's LICENSE + -h --help : Shows this help + +Options: + - --Status= : Primary status + - --MinorStatus= : Secondary status + - --ApplicationStatus= : Application status + - --Site= : Execution site + - --Owner= : Owner (DIRAC nickname) + - --JobGroup= : Select jobs for specified job group + - --Date= : Date in YYYY-MM-DD format, if not specified default is today + - --Maximum= : Maximum number of jobs shown (default 100, 0 means all) +``` + +Another possibility is to use the [iLCDirac web portal][diracweb]. + +[delphes]: https://delphes.github.io/ +[kkmcee]: https://kkmcee.docs.cern.ch/ +[kkmc-dirac-app]: https://gitlab.cern.ch/CLICdp/iLCDirac/ILCDIRAC/-/blob/Rel-v35r0/src/ILCDIRAC/Interfaces/API/NewInterface/Applications/KKMC.py +[generic-dirac-app]: https://gitlab.cern.ch/CLICdp/iLCDirac/ILCDIRAC/-/blob/Rel-v35r0/src/ILCDIRAC/Interfaces/API/NewInterface/Applications/GenericApplication.py +[workflow1-py]: https://raw.githubusercontent.com/HEP-FCC/FCCDIRAC/master/workflows/1/FCC_Dirac_Workflow1.py +[diracweb]: https://voilcdiracwebapp2.cern.ch/DIRAC/?theme=Classic&url_state=1|*DIRAC.JobMonitor.classes.JobMonitor:, diff --git a/distributed-computing/workflows/Overview.md b/distributed-computing/workflows/Overview.md index cd9af2885..00b8f046b 100644 --- a/distributed-computing/workflows/Overview.md +++ b/distributed-computing/workflows/Overview.md @@ -1,82 +1,107 @@ -# Overview of the submission scripts +# Overview of the job submission script -Command line submission to DIRAC is performed using python scripts instantiating the relevant classes. -The general structure of the script is the following: +> +> Original author: Gerardo Ganis +> -1. Instantiation of the interface to DIRAC; -2. Creation of a Job manager instance, including input and output sandbox, and all relevant config and data files; -3. Creation and configuration of the application to be run and their registration to the job manager instance; +Command line user submission to DIRAC WMS (Workload Management System) is +performed using Python scripts instantiating the relevant classes. The general +structure of the script is following: + +1. Instantiation of the interface to the iLCDirac; +2. Creation of a Job manager instance, including input and output sandbox, and + all relevant configuration and data files; +3. Creation and configuration of the application(s) to be run and their + registration to the job manager instance; 4. Job submission -The script may contain or import all the code relevant to the correct definition of the various steps above. -DIRAC also provides some standard tooling for parsing arguments and homogenize the submission script experience. +The script may contain or import all the code relevant to the correct +definition of the various steps above. DIRAC also provides some standard +tooling for parsing arguments to homogenize the submission script experience. -The parser is defined the DIRAC core, is part of the generic definition of [Script][script] and provides a callback for -customizing the actions. Typical usage looks like this: +The parser is defined in the DIRAC core, is part of the generic definition of +[Script][script] and provides a callback for customizing the actions. Typical +usage looks like this: ```python -from DIRAC import S_OK, S_ERROR +from DIRAC import S_OK, gLogger from DIRAC.Core.Base import Script # Define a simple class to hold the script parameters -class Params(object): - def __init__(self): - self.wms = 'wms' - def setWMS(self, value): - self.wms = value - return S_OK() - -# Instantiate the params class -cliParams = Params() -Script.registerSwitch('w', 'wms', "WMS where to run", cliParams.setWMS) -Script.parseCommandLine(ignoreErrors=False) - -# Get the list of services (the switch above appearer as servicesList[0]) -servicesList = Script.getPositionalArgs() -# The value for argument 'wms' is the first entry in servicelist list +class Params: + def __init__(self): + self.where = 'local' + def run_on_wms(self, _): + self.where = 'wms' + return S_OK() + def run_locally(self, _): + self.where = 'local' + return S_OK() -``` -[script]: https://dirac.readthedocs.io/en/latest/CodeDocumentation/Core/Base/Script.html +def main(): + # Instantiate the params class + cli_params = Params() + + # Register simple on/off switches + Script.registerSwitch('w', 'wms', 'Run on DIRAC WMS', + cli_params.run_on_wms) + Script.registerSwitch('l', 'local', 'Run locally', + cli_params.run_locally) + + # Parse the command line and initialize DIRAC + Script.parseCommandLine() + + # The result of the command line parsing is stored in the params class + gLogger.notice('cli_params.where: ', cli_params.where) -The DIRAC interface is controlled by the API interface class `DiracILC`, which derives from the upstream [DIRAC API][diracapi]. -Typical usage is the following: + +if __name__ == '__main__': + main() +``` + +The DIRAC interface is controlled by the API interface class `DiracILC`, which +derives from the upstream [DIRAC API][diracapi]. Typical usage is the following: ```python from ILCDIRAC.Interfaces.API.DiracILC import DiracILC + ... -dIlc = DiracILC() -``` -The returned `dILc` variable contais the API context to be used when relevant. +dilc = DiracILC() +``` -[diracapi]: https://raw.githubusercontent.com/DIRACGrid/DIRAC/integration/src/DIRAC/Interfaces/API/Dirac.py +The returned `dILc` variable contains the API context to be used when relevant. -The job manager is an instance of `UserJob` - which derives from [Job][job] - is instantiated next. -Typical usage is the following: +The job manager is an instance of `UserJob` - which derives from [Job][job] +— is instantiated next. Typical usage is the following: ```python from ILCDIRAC.Interfaces.API.NewInterface.UserJob import UserJob + ... + job = UserJob() job.setOutputSandbox(['*.log', '*.sh', '*.py', '*.xml']) outputdatafile='kktautau_delphes_edm4hep_output.root' -job.setOutputData(outputdatafile, '','CERN-DST-EOS' ) -job.setJobGroup( "KKMC_EDM4HEP_Run" ) -job.setName( "KKMC_EDM4HEP" ) -job.setLogLevel("DEBUG") +job.setOutputData(outputdatafile, '', 'CERN-DST-EOS') +job.setJobGroup('KKMC_EDM4HEP_Run') +job.setName('KKMC_EDM4HEP') +job.setLogLevel('DEBUG') + ... -# Information can be added thorugh out the script -delphescard='delphes_card_IDEA.tcl' + +# Information can be added through out the script +delphescard = 'delphes_card_IDEA.tcl' + ... + # Set the sandbox content job.setInputSandbox(['./' + delphescard, './' + edm4hepoutdef, './' + pythiacard]) ``` -[job]: https://dirac.readthedocs.io/en/latest/UserGuide/GettingStarted/UserJobs/ - -Applications are created, configured and added to the job manager in the order of running: -An example is the following: +Applications are created, configured and added to the job manager in the order +of running. An example is the following: ```python from ILCDIRAC.Interfaces.API.NewInterface.Applications import KKMC @@ -93,14 +118,12 @@ kkmc.setOutputFile(outputfile) job.append(kkmc) ``` -Available applications defined in [here][diracapp]. - -[diracapp]: https://gitlab.cern.ch/CLICdp/iLCDirac/ILCDIRAC/-/tree/Rel-v31r0/Interfaces/API/NewInterface +Available applications can be found [here][diracapp]. -Finally the job is submitted: +Finally the job is submitted with: ```python -print job.submit(dIlc, mode='wms') +print job.submit(dilc, mode='wms') # Use wms='local' for running on the local computer ``` @@ -108,48 +131,58 @@ Local submission can be used for testing. ## Before starting: cloning of workflows repository -The example scripts described in these pages, together with the relevant setup scripts, are available from the -[FCCDIRAC][fccdirac] repository. - -The following steps must be executed (only once!) before trying to execute any of the workflows: - -```bash -$ git clone https://github.com/HEP-FCC/FCCDIRAC -$ cd FCCDIRAC - -$ source init_fcc.sh -Setting up the latest Key4HEP software stack from CVMFS ... - ... Key4HEP release: key4hep-stack/2021-07-16 - ... Use the following command to reproduce the current environment: - ... - source /cvmfs/sw.hsf.org/spackages2/key4hep-stack/2021-07-16/x86_64-centos7-gcc8.3.0-opt/wxwfgu65rjnk7s6frj25qsoq5miay4ft/setup.sh - ... - ... done. - -$ source init_dirac.sh -Setting the iLCDirac environment ... - -$ source init_dirac_proxy.sh -Initializing the DIRAC/Grid proxy ... -Generating proxy... -Enter Certificate password: -Added VOMS attribute /fcc -Uploading proxy.. -Proxy generated: -subject : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ganis/CN=393971/CN=Gerardo Ganis/CN=2888907760/CN=1791020771 -issuer : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ganis/CN=393971/CN=Gerardo Ganis/CN=2888907760 -identity : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ganis/CN=393971/CN=Gerardo Ganis -timeleft : 23:53:59 -DIRAC group : fcc_user -path : /tmp/x509up_u2759 -username : ganis -properties : NormalUser -VOMS : True -VOMS fqan : ['/fcc'] - -Proxies uploaded: - DN | Group | Until (GMT) - /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ganis/CN=393971/CN=Gerardo Ganis | | 2022/05/13 12:12 -``` +The example scripts described in these pages, together with the relevant setup +scripts, are available from the [FCCDIRAC][fccdirac] repository. + +The following steps have to be executed before trying to execute any +of the example workflows: + +1. Clone the FCCDIRAC repository and enter the `workflows` directory within the + repository + ```bash + git clone https://github.com/HEP-FCC/FCCDIRAC + cd FCCDIRAC/workflows + ``` + +2. Setup the iLCDirac/DIRAC environment + ```bash + source setup_dirac.sh + ``` + The expected output: + ``` + Setting the iLCDirac environment ... + ``` + +3. Setup the GRID user proxy + ```bash + source setup_proxy.sh + ``` + The expected output: + ``` + Initializing the DIRAC/Grid proxy ... + Generating proxy... + Enter Certificate password: + Added VOMS attribute /fcc + Uploading proxy.. + Proxy generated: + subject : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ganis/CN=393971/CN=Gerardo Ganis/CN=2888907760/CN=1791020771 + issuer : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ganis/CN=393971/CN=Gerardo Ganis/CN=2888907760 + identity : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ganis/CN=393971/CN=Gerardo Ganis + timeleft : 23:53:59 + DIRAC group : fcc_user + path : /tmp/x509up_u2759 + username : ganis + properties : NormalUser + VOMS : True + VOMS fqan : ['/fcc'] + + Proxies uploaded: + DN | Group | Until (GMT) + /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ganis/CN=393971/CN=Gerardo Ganis | | 2022/05/13 12:12 + ``` +[script]: https://dirac.readthedocs.io/en/latest/CodeDocumentation/Core/Base/Script.html +[diracapi]: https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/Interfaces/API/Dirac.py +[job]: https://dirac.readthedocs.io/en/latest/UserGuide/GettingStarted/UserJobs/ +[diracapp]: https://gitlab.cern.ch/CLICdp/iLCDirac/ILCDIRAC/-/tree/Rel-v35r0/src/ILCDIRAC/Interfaces/API/NewInterface/Applications [fccdirac]: https://github.com/HEP-FCC/FCCDIRAC