refactor: consolidate tooling, fix pipeline, modernize examples

[jdsika]

Summary

Major consolidation of sl-5-8-asset-tools: modernized build system, overhauled pipeline with OMB-backed validation, structured logging, CID-named archives, Gaia-X metadata enrichment, and opt-in deterministic mode. Includes a pure-Python OpenDRIVE-to-GeoJSON converter, SHACL-driven CLI wizard, and comprehensive code quality improvements.

Build System & Tooling

• Add pyproject.toml replacing 16 scattered requirements.txt files
• Rename external/ to submodules/ for consistency
• Rewrite Makefile as central command center (OMB pattern)
  • Space-separated subcommands: make generate opendrive, make check format
  • OS detection for cross-platform support (Windows + Unix)
  • make validate auto-discovers and validates generated examples
• Add .pre-commit-config.yaml with make-target-only hooks
• Add CI workflow with multi-OS matrix and pipeline smoke tests
• Replace black+flake8+isort with ruff
• Add .gitattributes enforcing LF line endings
• Add .github/copilot-instructions.md

Pipeline Overhaul

Validation

• Replace legacy RDF/SHACL validation with OMB-backed offline loading (utils/rdf.py)
• Simplify jsonLD_validator to thin OMB wrapper (from ~200 lines to ~50)
• Fix recursive SHACL dependency loading in jsonLD_creator (BFS over imports, filtered to ENVITED-X/Gaia-X namespaces)
• Add split JSON syntax validation stages: extractor output + structure output checked separately

Logging & Reporting

• New utils/pipeline_reporting.py (~530 lines): stage-oriented output with [01/15] START/OK/FAIL formatting, tool-specific summarizers for OMB validation, QC results, GeoJSON conversion
• New utils/log_config.py: SL58_LOG_MODE=debug for verbose subprocess diagnostics
• Refactored utils/subprocess.py: CommandResult/CommandError types
• ASCII-only output throughout

CID-Named Archives

• New utils/cid.py: compute CIDv1 (base32, raw, SHA-256) from file content
• Archive output renamed from asset.zip to <CID>.zip in the example directory

Gaia-X Metadata Enrichment

• New meta_data_extractor/gaiax.py: extracts license, copyright owner, and resource policy from sibling LICENSE files
• Wired into both XODR and XOSC extractors
• Generates real manifest UUID (replaces placeholder did:web:registry.gaia-x.eu:Manifest:uuid)

Deterministic Mode (Opt-in)

• SL58_DETERMINISTIC=1 enables reproducible output: same input → same UUIDs, timestamps, ZIP bytes, CID
• utils/ids.py: UUID5 seeded by SL58_INPUT_HASH with counter for uniqueness
• structure_creator/main.py: SL58_SOURCE_MTIME for generated file timestamps
• create_zip(): sorted file iteration + fixed ZIP entry timestamps
• Default mode: random UUID4 + real timestamps (each generation is a unique asset instance)

QC Checker

• Pin qc-opendrive to fork fixing crash on roads without <link> element (upstream PR Handle missing optional road links in contact-point check asam-ev/qc-opendrive#139)
• Improved XQAR parsing and internal error detection in qualitychecker_caller

New Modules

Pure-Python OpenDRIVE-to-GeoJSON Converter

• Full xodr_to_geojson_caller rewrite: parser, geometry engine (curves, elevation, discretisation), generators (road, lane, object, signal, roadmark), GeoJSON converter
• Golden file regression tests for feature coverage proof
• Properly formatted output (indent=2)

SHACL-Driven CLI Wizard

• wizard_caller/shacl_wizard.py: interactive SHACL-driven metadata wizard
• sd-creation-wizard-api and sd-creation-wizard-frontend submodules

Other Improvements

• input_manifest.json pipeline replacing uploadedFiles.json (JSON-LD format detection)
• External asset reference support in scenario pipeline
• Migrate remaining os.path usage to pathlib
• Fix platform compatibility: os.sep in display paths, quoted $(MAKE) for Windows
• Modernize examples with input/+output/ structure
• Fix SPDX license identifier from EPL-2.0 to Apache-2.0
• Update OMB submodule pointer (3 upstream fixes)

---

Diff Stats (vs main)

145 files changed, 8211 insertions(+), 1077 deletions(-)

Key new files:

• utils/pipeline_reporting.py (529 lines)
• utils/cid.py, utils/ids.py, utils/input_manifest.py
• meta_data_extractor/gaiax.py (127 lines)
• wizard_caller/shacl_wizard.py (321 lines)
• scripts/resolve_references.py (130 lines)
• xodr_to_geojson_caller/ — 20+ new modules (parser, geometry, generators, tests)

Commits

• 1758dd7 feat: overhaul pipeline with OMB validation, structured logging, and deterministic mode
• 910351f fix: make bug
• 91fe366 fix: address remaining audit gaps
• c671286 fix: remediate audit findings
• 6a7f6e6 chore: update ontology-management-base submodule pointer
• 02360ed feat: support external asset references in scenario pipeline
• 82b7511 docs: update Makefile targets and README for new example structure
• e0fbec2 feat: modernize examples with input/output structure
• bc4f308 refactor: migrate remaining os.path usage to pathlib
• d97b801 fix: critical pipeline bugs in structure_creator and asset_extraction
• 2ec3d8a fix: rename output files and fix stale paths in generated assets
• 1d13f4e docs: update stale black/flake8 references to ruff
• d3961b6 feat: rewrite wizard_caller as SHACL-driven CLI wizard
• 889995b feat: add eclipse-xfsc SD Creation Wizard as submodules
• e0fec89 refactor: rename wizard-caller to wizard_caller
• 7e006b9 fix: correct SPDX license identifier from EPL-2.0 to Apache-2.0
• 9940733 fix: make OMB install conditional for CI without submodules
• 5866bfc refactor: replace black+flake8+isort with ruff
• eaaa28c style: run Black on xodr_to_geojson_caller (21 files)
• 8ad4688 style: run Black formatter on 3 files
• fc36cc0 test: add golden file regression tests for feature coverage proof
• 4eb7fac feat: re-implement OpenDRIVE-to-GeoJSON converter in pure Python
• 6ba5ca1 fix: structure_creator filename bugs and bjson output path
• e72c98d feat: replace uploadedFiles.json with input_manifest.json
• 008245d fix: image index counter and register_folder early return
• a1a981d fix: resolve LICENSE path bug in structure_creator
• 28c70b9 refactor: consolidate tooling with pyproject.toml, Makefile, and CI

Known Follow-ups

• OMB Makefile quoting fixes need a separate upstream PR to ASCS-eV/ontology-management-base
• QC fork pin should be updated to @main once upstream PR asam-ev/qc-opendrive#139 is merged

[MircoNierenz]

I recreated the workflow in Readme.md; here are my findings:

1. Installation:
   Worked in WSL and Git Bash.
   Doesn't work in Windows PowerShell—the README says it has to be POSIX—so that makes sense.

2. make generate opendrive
   Error: qc_opendrive reported internal checker errors: check_asam_xodr_road_geometry_contact_point: Error: 'NoneType' object has no attribute 'find'.
   I'll take a closer look at it in ASAM QualityChecker—and report it if necessary.

3. make generate openscenario
   It seems as though it isn't jumping to OpenScenario but is executing the OpenDrive path:
   [INFO] OpenDRIVE asset not built yet -- building dependency...
   make[1]: Entering directory 'E:/Data/Customer/OpenMSL/sl-5-8-asset-tools'
   [INFO] Running OpenDRIVE pipeline...
   ...

4. I'm not sure how to process my own assets using my input_manifest.json. I can't find anything about this in the README.
   I tried the following:
   python -m asset_extraction.main examples/OpenDRIVE/input/input_manifest.json -out examples/OpenDRIVE/output -config ./configs
   I then get an error:
   INFO utils.input_manifest: Detected input_manifest.json format (JSON-LD) - converting to legacy format
   Traceback (most recent call last):
   File "", line 198, in _run_module_as_main
   File "", line 88, in _run_code
   File "E:\Data\Customer\OpenMSL\sl-5-8-asset-tools\asset_extraction\main.py", line 444, in
   main()
   File "E:\Data\Customer\OpenMSL\sl-5-8-asset-tools\asset_extraction\main.py", line 324, in main
   raise FileNotFoundError(f"asset file {asset_file} not exists")
   FileNotFoundError: asset file E:\Data\Customer\OpenMSL\sl-5-8-asset-tools\StraightRoad_NCAP_R
   It doesn't look for the file in the input_manifest.json folder, but in the current directory.oadmarks.xodr not exists.

5. Debugmode
   $env:SL58_LOG_MODE = "debug"
   It didn't work for me, but
   export SL58_LOG_MODE=debug
   in WSl and git bash

6. SD Creation Wizard
   ran make wizard, installed Podman through it, started it, and completed the setup.
   Then I opened a new Git Bash and ran make wizard again,
   I'm getting the following errors:
   [INFO] Starting Podman machine...
   Starting machine "podman-machine-default"
   API forwarding for Docker API clients is not available due to the following startup failures.
   CreateFile \.\pipe\docker_engine: All pipe instances are busy.

Podman clients are still able to connect.
Error: machine did not transition into running state: ssh error: machine not in running state

I already asked ChatGPT about it, but it didn't work.

[jdsika]

Point 2) is strange as I fixed that upstream and pinned the Version. Maybe i did a mistake or you still have the old dependencies installed?

[jdsika]

3. the open scenario file is referencing the open drive map, therefore having the open drive is a prerequisite to create the open scenario?

[MircoNierenz]

2. I removed the existing .venv in WSL and reinstalled everything using “make setup”—unfortunately, I got the same error.

[jdsika]

Hey @MircoNierenz, thanks for the thorough testing! I've pushed a batch of fixes addressing all your findings. Here's what changed and how to retest:

What was fixed

#1 (PowerShell) — The README now has a dedicated Prerequisites section with Windows install steps, plus an "Option B" showing how to invoke the pipeline directly from PowerShell without Make. Both Makefiles also now set SHELL := sh on Windows, so make should work from PowerShell too (as long as Git for Windows is installed, which provides sh.exe).

#2 (qc_opendrive contact_point error) — This is pinned to my fork (jdsika/qc-opendrive@fix-contact-point-missing-road-link) which fixes the missing <link> element crash. Since you still see the error after a clean reinstall, please verify the fork is actually installed:

pip show asam-qc-opendrive

The "Location" should point to the fork. If it still shows the upstream version, try:

make clean all && make setup

If the error persists even with the fork installed, I'll need your .xodr file to reproduce — the fork fix handles missing <road><link> elements but there may be another code path hitting the same NoneType pattern. I have an upstream PR open for this: asam-ev/qc-opendrive#139.

#3 (OpenSCENARIO builds OpenDRIVE first) — This is expected: the OpenSCENARIO example references an OpenDRIVE map, so the pipeline auto-builds the dependency first. The message has been improved to make this clearer.

#4 (custom input path resolution) — This was the real bug. download_or_get_file() in utils/http.py resolved relative paths against CWD instead of the manifest directory. Fixed: relative paths now resolve against the output directory (where the manifest lives). The Makefile also no longer needs a cd workaround.

Additionally, there's now a generic INPUT_DIR/OUTPUT_DIR mode:

make generate INPUT_DIR=path/to/your/input OUTPUT_DIR=path/to/your/output

#5 (debug mode in PowerShell) — The README now documents both syntaxes:

# Bash / Git Bash
SL58_LOG_MODE=debug make generate opendrive

# PowerShell
$env:SL58_LOG_MODE = "debug"; make generate opendrive

#6 (Podman pipe error) — Added a troubleshooting table to the README covering the CreateFile \\.\pipe\docker_engine error (quit Docker Desktop first) and the machine not in running state error (podman machine rm then reinit).

How to retest

# 1. Start fresh
git fetch origin && git checkout feature/consolidate-tooling
git pull
make clean all

# 2. Reinstall everything
make setup

# 3. Run the built-in examples
make generate opendrive
make generate openscenario
make validate

# 4. Test with your own input (the bug you hit in #4)
python -m asset_extraction.main your/input_manifest.json -out your/output -config ./configs
# Or via Make:
make generate INPUT_DIR=your/input OUTPUT_DIR=your/output

If #2 still reproduces after step 2, run pip show asam-qc-opendrive and share the output + the .xodr file that triggers it.

[KR-VUFO]

Tried to create a simulation asset, but it was only possible with some changes in the process.json file.

Installation

• used the feature/consolidate-tooling branch
• make setup was executed successfully with the following last output: [OK] Setup complete. Activate with: use the activation script under .venv/Scripts for your shell

Testing with contained example

• pipeline works until step 7 --> Run ASAM OpenDRIVE checks did not work --> OpenDRIVE checker was not installed in the Python .\venv --> needed to manually install it there with pip --> issue was fixed
• same issue with OpenMSL OpenDRIVE checks --> but installation using pip was not possible this time --> cannot fix this issue right now --> deactivated this check in process.json
• same issue with OpenSCENARIO checks but manual installation with pip fixed the issue again (like OpenDRIVE Checker)
• Simulation Asset was successfully created after this manual installation processes and the deactivation of one Check

Testing own example

• with the same setting like mentioned above, a Simulation Asset was created successfully

[jdsika]

Maybe I forgot that you need to "make setup qc "?

Can you look for that in make help?

[KR-VUFO]

Following your instructions from above for a clean reinstall. I just added a make install qc (command is refered in the help) after executing make setup. Unfortunately the process aborted with the following error message:

ERROR: Failed to build 'asam-qc-openscenarioxml' when git clone --filter=blob:none --quiet https://github.com/asam-ev/qc-openscenarioxml 'c:\users\admin_temp\appdata\local\temp\pip-install-v9ib616r\asam-qc-openscenarioxml_655c36674f3d439b840840384a8adc36'

[jdsika]

Hey @KR-VUFO, thanks for testing and reporting! We pushed fixes that address all your issues:

What changed

1. make setup now installs QC tools automatically — you no longer need a separate make install qc step. Just make setup and everything is ready.

2. Windows long-path fix — the git clone failure you saw with asam-qc-openscenarioxml was caused by filenames exceeding the Windows 260-char limit. make setup now sets git config --global core.longpaths true before installing QC packages, so pip clones succeed.

3. Simplified workflow — make install qc has been removed (no longer needed). The only commands you need are:

   make clean all      # fresh start
   make setup          # creates venv + installs everything (dev, QC, OMB)
   make generate opendrive
   make generate openscenario
   

How to retest

git fetch origin && git checkout feature/consolidate-tooling && git pull
make clean all
make setup
make generate opendrive
make generate openscenario

CI is green on all 3 OSes (Ubuntu, Windows, macOS) × both pipelines.