Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .zuul.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,37 @@
pre-run: playbooks/pre.yml
run: playbooks/test-integration.yml

# End-to-end golden test for the SONiC config generator: provisions NetBox
# with a docker compose stack, seeds it from the in-repo fixtures under
# tests/e2e/scenario/, runs sync_sonic() and compares the exported config_db
# files against tests/e2e/golden/. In check the job only runs when files
# that can change the generated output (or the harness itself) are touched;
# periodic-daily runs it unconditionally. netbox-manager is still pulled at
# tip-of-main via required-projects because it remains the seeding tool, so
# a Depends-On is honored for its code -- but its example/ seed data is no
# longer used, so this job no longer detects drift in that data.
- job:
name: python-osism-sonic-e2e
nodeset: ubuntu-noble
pre-run: playbooks/pre-sonic-e2e.yml
run: playbooks/test-sonic-e2e.yml
required-projects:
- osism/netbox-manager
timeout: 2400
files:
- ^\.zuul\.yaml$
- ^Makefile$
- ^Pipfile\.lock$
- ^files/sonic/.*
- ^osism/settings\.py$
- ^osism/tasks/conductor/.*
- ^osism/utils/.*
- ^playbooks/(pre-|test-)sonic-e2e\.yml$
- ^requirements\.ansible\.txt$
- ^requirements\.txt$
- ^setup\.cfg$
- ^tests/e2e/.*

- project:
merge-mode: squash-merge
default-branch: main
Expand All @@ -154,6 +185,7 @@
- python-osism-test-setup
- python-osism-unit-tests
- python-osism-integration-tests
- python-osism-sonic-e2e
periodic-daily:
jobs:
- flake8
Expand All @@ -163,6 +195,7 @@
- python-osism-test-setup
- python-osism-unit-tests
- python-osism-integration-tests
- python-osism-sonic-e2e
periodic-midnight:
jobs:
- container-image-python-osism-push
Expand Down
8 changes: 7 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,10 @@ sonic-e2e-up:
sonic-e2e-down:
docker compose -f tests/e2e/compose.yaml down --volumes --remove-orphans

.PHONY: sonic-e2e sonic-e2e-regen sonic-e2e-up sonic-e2e-down
# Report config_db table coverage of the golden set (tests/e2e/coverage.py).
# A reporting tool only -- not part of the gating check, which stays the
# golden comparison run by sonic-e2e above.
sonic-e2e-coverage:
pipenv run python -m tests.e2e.coverage

.PHONY: sonic-e2e sonic-e2e-regen sonic-e2e-up sonic-e2e-down sonic-e2e-coverage
57 changes: 57 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,60 @@ REDIS_HOST=localhost REDIS_DB=15 pipenv run pytest tests/integration
> above, or `OSISM_ALLOW_DEFAULT_REDIS_DB=1` if the Redis itself is disposable.
> `REDIS_DB` moves the direct client, the Celery broker and the result backend
> together.

## Running the SONiC E2E golden test

The end-to-end test in `tests/e2e/` provisions NetBox with a docker compose
stack, seeds it from the fixtures in `tests/e2e/scenario/`, generates the SONiC
`config_db.json` files and compares them against the goldens in
`tests/e2e/golden/`. Besides the development dependencies it needs docker with
the compose plugin, `openssl`, and a `netbox-manager` checkout for the seeding
CLI — a sibling directory by default, `NETBOX_MANAGER_DIR` otherwise.

```
pipenv install --dev
make sonic-e2e
```

A cold run takes roughly ten minutes, most of it starting NetBox. To iterate
without paying that each time, bring the stack up separately and leave it
running:

```
make sonic-e2e-up # start NetBox and leave it up; sonic-e2e reuses it
make sonic-e2e-down # stop it again and remove its volumes
```

After an intentional generator change, rewrite the goldens and review the diff
before committing it. Regeneration deliberately refuses to run against a stack
left over from an earlier run, because applying the fixtures over a populated
database can produce goldens that CI — which always starts fresh — would not
reproduce:

```
make sonic-e2e-down
make sonic-e2e-regen
```

How much of the generated config the golden set actually covers is reported
separately, because nothing in CI reports it:

```
make sonic-e2e-coverage
```

That compares the `config_db` tables the generator can emit against the tables
that are non-empty in at least one golden, and names any that no golden covers.
It exits non-zero while that list is non-empty, so it is worth running after
adding a scenario to confirm the new tables landed. It gates nothing on its own
— the golden comparison above is the only check that fails a run.

`tests/e2e/sonic_golden_test.sh` documents the remaining environment overrides
(`NETBOX_PORT`, `KEEP_STACK`, `SEED_PARALLEL` and the regeneration escape
hatch).

> **Warning:** Seeding applies *every* file under
> `tests/e2e/scenario/resources/`, tracked or not, so a stray file there joins
> the fixture set — which either breaks the run or silently changes the
> goldens. Check that directory with `git status --ignored` before regenerating
> or debugging a mismatch.
74 changes: 74 additions & 0 deletions playbooks/pre-sonic-e2e.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
---
# Node preparation for the SONiC config-generation E2E golden test.
#
# The test's NetBox fixture is a docker compose stack (tests/e2e/compose.yaml),
# so the node only needs Docker: ensure-docker installs docker-compose-plugin
# along with docker-ce. No kind, kubectl or helm. netbox-manager's own E2E job
# still provisions NetBox on kind, so there is nothing to keep in sync here.
- name: Prepare the SONiC E2E node
hosts: all

pre_tasks:
# This CI node is IPv6-only and learns its address and default route via
# SLAAC / Router Advertisements. If Docker enables
# net.ipv6.conf.all.forwarding=1, the kernel stops honouring RAs at the
# default accept_ra=1, so the SLAAC default route expires and the node
# drops off the network a few minutes into the run. accept_ra=2 keeps RAs
# honoured even while forwarding is on, preserving the default route.
# Compose's default bridge network is IPv4-only, so Docker probably will
# not turn on IPv6 forwarding here -- but this guard is cheap insurance
# against a severe failure mode (an unreachable CI node) and is kept
# deliberately; removing it should be its own separate experiment, not a
# side effect of some other change. Set it here -- before Docker enables
# forwarding -- so the route never lapses. See
# https://docs.docker.com/engine/daemon/ipv6/ and
# https://forums.docker.com/t/docker-removes-host-ipv6-default-route/83238
- name: Keep accepting IPv6 RAs after Docker enables forwarding (preserve default route)
become: true
ansible.builtin.copy:
dest: /etc/sysctl.d/99-sonic-e2e-accept-ra.conf
owner: root
group: root
mode: "0644"
content: |
net.ipv6.conf.all.accept_ra = 2
net.ipv6.conf.default.accept_ra = 2
{% if ansible_default_ipv6.interface is defined %}
net.ipv6.conf.{{ ansible_default_ipv6.interface }}.accept_ra = 2
{% endif %}

- name: Apply the accept_ra sysctl settings now
become: true
ansible.builtin.command:
cmd: sysctl -p /etc/sysctl.d/99-sonic-e2e-accept-ra.conf
changed_when: true
# A node without an ansible_default_ipv6.interface fact (or otherwise
# missing these IPv6 keys) fails sysctl -p; this is best-effort
# insurance for the common IPv6-only case, not a hard requirement, so
# do not fail pre-run over it.
failed_when: false

roles:
- ensure-pip
- ensure-pipenv
- ensure-docker

tasks:
- name: Ensure the Docker service is running
become: true
ansible.builtin.service:
name: docker
state: started
enabled: true

# openssl is used by tests/e2e/sonic_golden_test.sh directly. curl is used
# by the NetBox container's healthcheck rather than by the script itself,
# but ensure-docker needs it anyway. python3-venv provides the venv module
# the script uses for the seeding venv.
- name: Install required packages
become: true
ansible.builtin.apt:
name:
- curl
- openssl
- python3-venv
40 changes: 40 additions & 0 deletions playbooks/test-sonic-e2e.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
---
- name: Run the SONiC config-generation E2E golden test
hosts: all

vars:
python_venv_dir: /tmp/venv

tasks:
- name: Install dependencies
ansible.builtin.shell:
executable: /bin/bash
chdir: "{{ zuul.project.src_dir }}"
cmd: |
set -e
set -o pipefail
set -x

{{ python_venv_dir }}/bin/pipenv install --dev --deploy
{{ python_venv_dir }}/bin/pipenv run pip install .

- name: Run the E2E golden test
ansible.builtin.shell:
executable: /bin/bash
chdir: "{{ zuul.project.src_dir }}"
cmd: |
set -e
set -o pipefail
set -x

# The script invokes bare `pipenv`; the netbox-manager checkout comes
# from Zuul's required-projects, so a Depends-On change to it is
# tested against the changed code and seed data.
export PATH="{{ python_venv_dir }}/bin:${PATH}"
export NETBOX_MANAGER_DIR="{{ ansible_user_dir }}/{{ zuul.projects['github.com/osism/netbox-manager'].src_dir }}"

# Serial by default: concurrent seeding deadlocks on dcim_device FK
# row locks (see tests/e2e/sonic_golden_test.sh, Phase 2). Set
# `sonic_e2e_seed_parallel: 4` in the job vars to opt back in.
export SEED_PARALLEL="{{ sonic_e2e_seed_parallel | default(1) }}"
tests/e2e/sonic_golden_test.sh
89 changes: 89 additions & 0 deletions tests/e2e/coverage.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# SPDX-License-Identifier: Apache-2.0

"""config_db table coverage report for the SONiC E2E golden set.

This is a reporting tool, not a gate: it is not invoked from
sonic_golden_test.sh, and the golden comparison (tests/e2e/compare.py)
remains the only check that actually fails a run. It exists so the "N of M
config_db tables covered" claim made when the golden set is extended can be
re-derived by anyone, at any time, instead of only having existed as an
ad-hoc one-off script run once during development.

It works in two independent steps:

1. Derive the set of tables the generator can emit by grepping
osism/tasks/conductor/sonic/ (excluding the generated schema package,
_generated/, which is data rather than emission logic) for direct
``config["TABLE"]``/``cfg["TABLE"]`` assignments -- including a nested
item assignment such as ``config["ACL_TABLE"]["SSH_ONLY"] = ...`` or a
``.update(...)`` call, but not a mere read such as
``"x" in config["VERSIONS"]``. This is a static, syntactic approximation
of "tables the generator can populate", not a guarantee every branch
that reaches it is exercised.
2. Collect the set of tables that are non-empty in at least one file under
tests/e2e/golden/*.json.

The report prints both counts and, if any emitted table is never non-empty
across the golden set, lists them and exits 1.
"""

import re
import sys
from pathlib import Path

REPO_ROOT = Path(__file__).resolve().parents[2]
GENERATOR_DIR = REPO_ROOT / "osism" / "tasks" / "conductor" / "sonic"
GOLDEN_DIR = REPO_ROOT / "tests" / "e2e" / "golden"

# Matches config["TABLE"] or cfg["TABLE"], optionally followed by one or
# more ["key"]/[expr] accessors, then either an assignment ("=" but not
# "==") or a .update(...) call -- i.e. the table is a write target, not
# merely read.
_TABLE_ASSIGNMENT_RE = re.compile(
r'(?:config|cfg)\["([A-Z][A-Z0-9_]*)"\](?:\[[^\]]*\])*\s*(?:=(?!=)|\.update\()'
)


def emitted_tables(generator_dir=GENERATOR_DIR):
"""Tables the generator can emit, derived from source assignments."""
tables = set()
for path in sorted(Path(generator_dir).rglob("*.py")):
if "_generated" in path.parts:
continue
for line in path.read_text().splitlines():
if line.lstrip().startswith("#"):
continue
for match in _TABLE_ASSIGNMENT_RE.finditer(line):
tables.add(match.group(1))
return tables
Comment thread
sourcery-ai[bot] marked this conversation as resolved.


def covered_tables(golden_dir=GOLDEN_DIR):
"""Tables that are non-empty in at least one golden file."""
import json

tables = set()
for path in sorted(Path(golden_dir).glob("*.json")):
config = json.loads(path.read_text())
for table, value in config.items():
if value:
tables.add(table)
return tables


def main(argv=None):
emitted = emitted_tables()
covered = covered_tables()
missing = sorted(emitted - covered)

print(f"generator emits {len(emitted)} tables; {len(covered)} non-empty")
if missing:
print("not covered by any golden file:")
for table in missing:
print(f" {table}")
return 1
return 0


if __name__ == "__main__":
sys.exit(main())
Loading