ckanext-tmforum

CKAN harvester extension for TM Forum APIs.

It harvests product offerings/specifications, enriches them with related service specifications, resolves owner organizations from TMF Party data, and creates/updates datasets in CKAN.

Compatibility

• CKAN: 2.11 (current target)
• Python: 3.10
• Requires ckanext-harvest

Harvester Info

Harvester type name:

• tmforum

This is the value to use when creating a Harvest Source in CKAN.

Harvest Flow

1. gather_stage

• Reads launched product offerings from TMF Product Catalog (/productOffering).
• Creates harvest objects using product specification id as GUID.

2. fetch_stage

• Fetches product specification (/productSpecification/{id}).
• Fetches related service specifications from configured service catalog.
• Stores normalized content on harvest object.

3. import_stage

• Resolves owner party from relatedParty using configured owner_role.
• Resolves/creates owner organization in CKAN.
• Builds CKAN dataset and resources.
• Creates or updates package through harvest base logic.

Configuration

Configuration is read from the Harvest Source JSON config (source.config) for runtime behavior.

Expected keys:

• party_url (required)
  • Base TMF Party endpoint.
• service_catalog_url (required)
  • Base TMF Service Catalog endpoint.
• owner_role (optional)
  • Default: owner
  • Used to select owner party from relatedParty.
• marketplace_url (optional)
  • If set, adds an extra resource access URL: {marketplace_url}/search/{offer_id}.
• edp_url (optional)
  • If set, adds an extra resource access URL pointing to EDP.

The harvest source URL (source.url) is used as TMF Product Catalog base.

Example Harvest Source Config

{
  "party_url": "https://tmf.example.com/party",
  "service_catalog_url": "https://tmf.example.com/serviceCatalogManagement/v4",
  "owner_role": "Seller",
  "marketplace_url": "https://marketplace.example.com",
  "edp_url": "https://edc.example.com"
}

Resource Mapping

Per service specification, resources are created with:

• name, description
• access_url (TMF product offering URL)
• conforms_to (TM Forum URIs)

Known specCharacteristic mappings:

• URL -> download_url
• Format -> format
• Media Type -> mimetype

Other characteristics are mapped to flattened resource keys:

• tmf_char_<normalized_name>
• if multiple values: tmf_char_<normalized_name>_1, ..._2, etc.

Duplicate characteristic names are ignored after first occurrence.

Organization Resolution

Owner organization is resolved from party data using DID in this order:

1. externalReference where externalReferenceType == "idm_id" and name starts with did
2. partyCharacteristic where name == "did"
3. fallback to party_id

Generated CKAN org name format:

• vc-org-<sha256(did)[:16]>

Installation

git clone https://github.com/SEAMWARE/ckanext-tmforum.git
cd ckanext-tmforum
pip install -e .
pip install -r requirements.txt

Enable plugin:

• ckan.plugins = ... harvest tmforum ...

Running Harvest Jobs (ckan-docker)

Examples:

docker compose exec ckan-dev bash -lc "ckan -c /srv/app/ckan.ini harvester sources" docker compose exec ckan-dev bash -lc "ckan -c /srv/app/ckan.ini harvester jobs" docker compose exec ckan-dev bash -lc "ckan -c /srv/app/ckan.ini harvester run-test <source_id_or_name>" docker compose exec ckan-dev bash -lc "ckan -c /srv/app/ckan.ini harvester gather-consumer" docker compose exec ckan-dev bash -lc "ckan -c /srv/app/ckan.ini harvester fetch-consumer"

Tests

Run all tests:

pytest --ckan-ini=test.ini

Run unit tests only:

pytest --ckan-ini=test.ini ckanext/tmforum/tests/unit

Releasing

1. Update version in pyproject.toml

2. Build and validate:

   python -m build && twine check dist/*

3. Upload:

   twine upload dist/*

4. Commit and tag:

git commit -a -m "Release x.y.z"
git push
git tag x.y.z
git push --tags

License

AGPL