Fix transitive dependency detection for Python CVE analysis

[etsien]

Summary

The vulnerability analysis pipeline was producing false-positive code_not_present verdicts
for CVEs affecting transitive Python dependencies (e.g., jinja2 pulled in by ansible, urllib3
pulled in by requests). The agent searched the application source for import jinja2, found
nothing because the app never imports it directly, and concluded the library was absent.

Three related problems caused this:

Manifest discovery gap. PythonDependencyTreeBuilder.install_dependencies opened
requirements.txt unconditionally and raised FileNotFoundError for any project that uses
pyproject.toml, setup.py, uv.lock, poetry.lock, setup.cfg, or Pipfile. The
exception was silently swallowed by the VDB builder, leaving the code index empty and making
every transitive CVE a false positive for those projects.

Site-packages not indexed. Even when installation succeeded, the packages in
transitive_env/lib/*/site-packages/ were outside the repo tree and never added to the
document index. Code Keyword Search and the Call Chain Analyzer (CCA) had no visibility into
the source of installed packages such as jinja2 or urllib3.

Installed package list unavailable. There was no indexed record of what was actually
installed in the container environment, so the agent had no way to confirm library presence
short of finding a direct import in the application source.

Changes

src/exploit_iq_commons/utils/dep_tree.py

• detect_ecosystem now recognises uv.lock, poetry.lock, setup.cfg, and Pipfile as
  Python project indicators, so the installation step is triggered for projects that use those
  formats.

• PythonDependencyTreeBuilder.install_dependencies is restructured around a manifest fallback
  chain. It tries each format in priority order and logs which one succeeded. The
  requirements.txt path is unchanged in behaviour. Lock-file projects (uv.lock,
  poetry.lock) are handled via uv export. pyproject.toml/setup.py/setup.cfg projects
  use uv pip install .. Pipfile projects use pipenv requirements.

• After installation, _write_installed_packages runs pip list --format=freeze and writes
  the result to installed_packages.txt in the repository root. This file is a freeze-format
  snapshot of everything installed in the venv.

• PythonDependencyTreeBuilder.build_tree no longer hard-codes a read of requirements.txt
  to determine direct dependencies. The new _get_direct_dependencies helper reads
  requirements.txt when present, and falls back to the indent-level-0 packages in the
  deptree output when it is not.

src/exploit_iq_commons/utils/source_code_git_loader.py

• SourceCodeGitLoader.yield_blobs unconditionally adds installed_packages.txt to the
  include set when the file is present. This makes Code Keyword Search able to confirm library
  presence (e.g., urllib3==2.2.1) without needing to find a direct import in application
  source.

• The new _add_site_packages_blobs static method scans transitive_env/lib/*/site-packages/
  and adds Python source files to the include set for any package directory containing at most
  150 .py files. This lets the CCA and Code Keyword Search trace call chains across package
  boundaries (for example, confirming that an operator's use of Ansible transitively reaches
  the vulnerable jinja2 template path). Packages exceeding the file-count limit,
  .dist-info/.egg-info metadata directories, ansible_collections, test directories, and
  __pycache__ are excluded.

[tmihalac]

A small suggestion, add this as well:

if "UV_CACHE_DIR" not in os.environ:
            fallback_cache = manifest_path / ".uv_cache"
            try:
                fallback_cache.mkdir(exist_ok=True)
                os.environ["UV_CACHE_DIR"] = str(fallback_cache)
            except OSError:
                pass

When uv runs without UV_CACHE_DIR set, it defaults to ~/.cache/uv. In
OpenShift containers the home directory is often read-only (non-root UID,
restricted SecurityContextConstraints), so uv venv or uv pip install fails
with a permission error trying to create its cache.

The fallback creates .uv_cache inside manifest_path (the cloned repo
directory, which is writable on the PVC) and sets UV_CACHE_DIR to point
there. The try/except OSError handles the edge case where even that
directory can't be created.

[zvigrinberg]

A small suggestion, add this as well:

if "UV_CACHE_DIR" not in os.environ:
            fallback_cache = manifest_path / ".uv_cache"
            try:
                fallback_cache.mkdir(exist_ok=True)
                os.environ["UV_CACHE_DIR"] = str(fallback_cache)
            except OSError:
                pass

When uv runs without UV_CACHE_DIR set, it defaults to ~/.cache/uv. In OpenShift containers the home directory is often read-only (non-root UID, restricted SecurityContextConstraints), so uv venv or uv pip install fails with a permission error trying to create its cache.

The fallback creates .uv_cache inside manifest_path (the cloned repo directory, which is writable on the PVC) and sets UV_CACHE_DIR to point there. The try/except OSError handles the edge case where even that directory can't be created.

@etsien This is important...
But just instead of mutating the UV_CACHE_DIR environment variable of the agent process globally, it's better to pass it as env var to the subprocess.run that invokes the uv package manager binary ( in order to prevent potential race conditions and errors between threads that run python analysis jobs concurrently).

[zvigrinberg]

@etsien This is a great idea... good job.
Did you test it successfully for failed scenarios?
Please see @tmihalac comment and mine following one.

[etsien]

A small suggestion, add this as well:

if "UV_CACHE_DIR" not in os.environ:
            fallback_cache = manifest_path / ".uv_cache"
            try:
                fallback_cache.mkdir(exist_ok=True)
                os.environ["UV_CACHE_DIR"] = str(fallback_cache)
            except OSError:
                pass

When uv runs without UV_CACHE_DIR set, it defaults to ~/.cache/uv. In OpenShift containers the home directory is often read-only (non-root UID, restricted SecurityContextConstraints), so uv venv or uv pip install fails with a permission error trying to create its cache.
The fallback creates .uv_cache inside manifest_path (the cloned repo directory, which is writable on the PVC) and sets UV_CACHE_DIR to point there. The try/except OSError handles the edge case where even that directory can't be created.

@etsien This is important... But just instead of mutating the UV_CACHE_DIR environment variable of the agent process globally, it's better to pass it as env var to the subprocess.run that invokes the uv package manager binary ( in order to prevent potential race conditions and errors between threads that run python analysis jobs concurrently).

ok will do

[etsien]

@etsien This is a great idea... good job. Did you test it successfully for failed scenarios? Please see @tmihalac comment and mine following one.

Yes, I had 4 python samples consistently fail, silent failures, because it can't unpack the imports. After the changes, they are now finding and indexing the 3rd party packages to trace the dependencies. This is reflected in the agent traces inside the checklist.

[etsien]

/test-heavy

[tmihalac]

@etsien Updated suggestion — the previous approach (_detect_existing_site_packages_version) didn't work reliably. After extensive testing on the cluster, here's what actually works:

Root cause (full chain)

1. lint-and-test and integration-test sidecar share the same PVC but use different container images
2. uv venv creates bin/python as a symlink to the container-local Python path
3. Different images = different paths = broken symlink across containers
4. uv venv without --python downloads latest Python (3.13) in the sidecar since it has no system Python
5. Packages installed under python3.12/site-packages/ are invisible to Python 3.13

Working solution: store Python on the shared PVC

Tekton changes (on-pull-request.yaml):

1. Switch lint-test image to ubi9/ubi-minimal (no system Python — forces uv to use managed Python from PVC):

image: registry.access.redhat.com/ubi9/ubi-minimal:latest

2. Set UV_PYTHON_INSTALL_DIR to the same absolute PVC path in both lint-test and sidecar:

# lint-test env
- name: UV_PYTHON_INSTALL_DIR
  value: "/exploit-iq-data/python"
# sidecar env (same value)
- name: UV_PYTHON_INSTALL_DIR
  value: "/exploit-iq-data/python"

3. Mount PVC at /exploit-iq-data in lint-test (matching the sidecar):

volumeMounts:
  - name: $(workspaces.unit-test-cache.volume)
    mountPath: /exploit-iq-data

4. Explicitly install Python to PVC in lint-test script:

uv python install 3.12

Important: just setting UV_PYTHON_INSTALL_DIR is NOT enough — uv venv --python 3.12 won't auto-download to UV_PYTHON_INSTALL_DIR. You must run uv python install explicitly first.

5. Install system packages needed by ubi-minimal:

microdnf install -y git tar gzip findutils make gcc gcc-c++
microdnf module enable -y nodejs:20
microdnf install -y nodejs npm

Code changes (dep_tree.py):

def _ensure_venv(self, manifest_path: Path) -> str:
    """Ensure transitive_env exists with a working python binary."""
    venv_python = f'{manifest_path}/{TRANSITIVE_ENV_NAME}/bin/python'
    if Path(venv_python).exists():
        return venv_python
    logger.warning("Venv python not found at %s — creating venv", venv_python)
    self._ensure_uv_cache_dir(manifest_path)
    python_version = self.determine_python_version(str(manifest_path))
    if not python_version:
        import sys
        python_version = f"{sys.version_info.major}.{sys.version_info.minor}"
        logger.info("Python version undetermined; using current interpreter %s", python_version)
    logger.info("Creating transitive_env with Python %s using uv", python_version)
    cmd = f"cd {manifest_path} && uv venv {TRANSITIVE_ENV_NAME} --python {python_version}"
    run_command(cmd)
    return venv_python

Key points:

• Extract venv creation from install_dependencies into shared _ensure_venv
• Always pass --python explicitly (never let uv pick default)
• sys.version_info fallback when determine_python_version returns None
• Both build_tree and install_dependencies call _ensure_venv

Why the previous suggestions didn't work

• _detect_existing_site_packages_version: got confused by polluted python3.13/ directories from failed runs
• uv venv without --python: downloads latest Python (3.13) in the sidecar
• UV_PYTHON_INSTALL_DIR alone: doesn't affect uv venv, only uv python install
• Different PVC mount paths (/workspace/unit-test-cache/ vs /exploit-iq-data/): symlinks encode absolute paths, must match

Verified on cluster

• Cold run: lint-test installs Python 3.12 to PVC, creates venv, installs packages. Sidecar finds Python at same PVC path — no broken symlink. 6/6 integration tests PASS.
• Hot run (second pipeline): lint-test finds Python on PVC (cache hit). Sidecar finds same Python — symlink still valid. 6/6 PASS.

[etsien]

@etsien Updated suggestion — the previous approach (_detect_existing_site_packages_version) didn't work reliably. After extensive testing on the cluster, here's what actually works:

Root cause (full chain)

1. `lint-and-test` and `integration-test` sidecar share the same PVC but use different container images

2. `uv venv` creates `bin/python` as a symlink to the container-local Python path

3. Different images = different paths = broken symlink across containers

4. `uv venv` without `--python` downloads latest Python (3.13) in the sidecar since it has no system Python

5. Packages installed under `python3.12/site-packages/` are invisible to Python 3.13

Working solution: store Python on the shared PVC

Tekton changes (on-pull-request.yaml):

1. Switch lint-test image to `ubi9/ubi-minimal` (no system Python — forces `uv` to use managed Python from PVC):

image: registry.access.redhat.com/ubi9/ubi-minimal:latest

2. Set `UV_PYTHON_INSTALL_DIR` to the **same absolute PVC path** in both lint-test and sidecar:

# lint-test env
- name: UV_PYTHON_INSTALL_DIR
  value: "/exploit-iq-data/python"
# sidecar env (same value)
- name: UV_PYTHON_INSTALL_DIR
  value: "/exploit-iq-data/python"

3. Mount PVC at `/exploit-iq-data` in lint-test (matching the sidecar):

volumeMounts:
  - name: $(workspaces.unit-test-cache.volume)
    mountPath: /exploit-iq-data

4. Explicitly install Python to PVC in lint-test script:

uv python install 3.12

Important: just setting UV_PYTHON_INSTALL_DIR is NOT enough — uv venv --python 3.12 won't auto-download to UV_PYTHON_INSTALL_DIR. You must run uv python install explicitly first.

5. Install system packages needed by ubi-minimal:

microdnf install -y git tar gzip findutils make gcc gcc-c++
microdnf module enable -y nodejs:20
microdnf install -y nodejs npm

Code changes (dep_tree.py):

def _ensure_venv(self, manifest_path: Path) -> str:
    """Ensure transitive_env exists with a working python binary."""
    venv_python = f'{manifest_path}/{TRANSITIVE_ENV_NAME}/bin/python'
    if Path(venv_python).exists():
        return venv_python
    logger.warning("Venv python not found at %s — creating venv", venv_python)
    self._ensure_uv_cache_dir(manifest_path)
    python_version = self.determine_python_version(str(manifest_path))
    if not python_version:
        import sys
        python_version = f"{sys.version_info.major}.{sys.version_info.minor}"
        logger.info("Python version undetermined; using current interpreter %s", python_version)
    logger.info("Creating transitive_env with Python %s using uv", python_version)
    cmd = f"cd {manifest_path} && uv venv {TRANSITIVE_ENV_NAME} --python {python_version}"
    run_command(cmd)
    return venv_python

Key points:

* Extract venv creation from `install_dependencies` into shared `_ensure_venv`

* Always pass `--python` explicitly (never let `uv` pick default)

* `sys.version_info` fallback when `determine_python_version` returns None

* Both `build_tree` and `install_dependencies` call `_ensure_venv`

Why the previous suggestions didn't work

* `_detect_existing_site_packages_version`: got confused by polluted `python3.13/` directories from failed runs

* `uv venv` without `--python`: downloads latest Python (3.13) in the sidecar

* `UV_PYTHON_INSTALL_DIR` alone: doesn't affect `uv venv`, only `uv python install`

* Different PVC mount paths (`/workspace/unit-test-cache/` vs `/exploit-iq-data/`): symlinks encode absolute paths, must match

Verified on cluster

* **Cold run**: lint-test installs Python 3.12 to PVC, creates venv, installs packages. Sidecar finds Python at same PVC path — no broken symlink. 6/6 integration tests PASS.

* **Hot run (second pipeline)**: lint-test finds Python on PVC (cache hit). Sidecar finds same Python — symlink still valid. 6/6 PASS.

Can you push these changes so I can inspect?

[tmihalac]

@etsien Updated suggestion — the previous approach (_detect_existing_site_packages_version) didn't work reliably. After extensive testing on the cluster, here's what actually works:

Root cause (full chain)

1. `lint-and-test` and `integration-test` sidecar share the same PVC but use different container images

2. `uv venv` creates `bin/python` as a symlink to the container-local Python path

3. Different images = different paths = broken symlink across containers

4. `uv venv` without `--python` downloads latest Python (3.13) in the sidecar since it has no system Python

5. Packages installed under `python3.12/site-packages/` are invisible to Python 3.13

Working solution: store Python on the shared PVC

Tekton changes (on-pull-request.yaml):

1. Switch lint-test image to `ubi9/ubi-minimal` (no system Python — forces `uv` to use managed Python from PVC):

image: registry.access.redhat.com/ubi9/ubi-minimal:latest

2. Set `UV_PYTHON_INSTALL_DIR` to the **same absolute PVC path** in both lint-test and sidecar:

# lint-test env
- name: UV_PYTHON_INSTALL_DIR
  value: "/exploit-iq-data/python"
# sidecar env (same value)
- name: UV_PYTHON_INSTALL_DIR
  value: "/exploit-iq-data/python"

3. Mount PVC at `/exploit-iq-data` in lint-test (matching the sidecar):

volumeMounts:
  - name: $(workspaces.unit-test-cache.volume)
    mountPath: /exploit-iq-data

4. Explicitly install Python to PVC in lint-test script:

uv python install 3.12

Important: just setting UV_PYTHON_INSTALL_DIR is NOT enough — uv venv --python 3.12 won't auto-download to UV_PYTHON_INSTALL_DIR. You must run uv python install explicitly first.

5. Install system packages needed by ubi-minimal:

microdnf install -y git tar gzip findutils make gcc gcc-c++
microdnf module enable -y nodejs:20
microdnf install -y nodejs npm

Code changes (dep_tree.py):

def _ensure_venv(self, manifest_path: Path) -> str:
    """Ensure transitive_env exists with a working python binary."""
    venv_python = f'{manifest_path}/{TRANSITIVE_ENV_NAME}/bin/python'
    if Path(venv_python).exists():
        return venv_python
    logger.warning("Venv python not found at %s — creating venv", venv_python)
    self._ensure_uv_cache_dir(manifest_path)
    python_version = self.determine_python_version(str(manifest_path))
    if not python_version:
        import sys
        python_version = f"{sys.version_info.major}.{sys.version_info.minor}"
        logger.info("Python version undetermined; using current interpreter %s", python_version)
    logger.info("Creating transitive_env with Python %s using uv", python_version)
    cmd = f"cd {manifest_path} && uv venv {TRANSITIVE_ENV_NAME} --python {python_version}"
    run_command(cmd)
    return venv_python

Key points:

* Extract venv creation from `install_dependencies` into shared `_ensure_venv`

* Always pass `--python` explicitly (never let `uv` pick default)

* `sys.version_info` fallback when `determine_python_version` returns None

* Both `build_tree` and `install_dependencies` call `_ensure_venv`

Why the previous suggestions didn't work

* `_detect_existing_site_packages_version`: got confused by polluted `python3.13/` directories from failed runs

* `uv venv` without `--python`: downloads latest Python (3.13) in the sidecar

* `UV_PYTHON_INSTALL_DIR` alone: doesn't affect `uv venv`, only `uv python install`

* Different PVC mount paths (`/workspace/unit-test-cache/` vs `/exploit-iq-data/`): symlinks encode absolute paths, must match

Verified on cluster

* **Cold run**: lint-test installs Python 3.12 to PVC, creates venv, installs packages. Sidecar finds Python at same PVC path — no broken symlink. 6/6 integration tests PASS.

* **Hot run (second pipeline)**: lint-test finds Python on PVC (cache hit). Sidecar finds same Python — symlink still valid. 6/6 PASS.

Can you push these changes so I can inspect?

I have i

[tmihalac]

I have those changes as part of a larger PR, that I am still working on, you can have a look if you like:
#243