code-organization.md

Code Organization Guide

Purpose

This document defines how the plugin-ipc codebase is structured, how to add new transports, message types, and snapshot helpers, and how to preserve separation of concerns between the architectural layers.

This guide is derived from the Level 1, Codec, Level 2, and Level 3 specifications. It is not an independent design — it enforces the boundaries those specs define.

Core Service Rule

The repository is organized around service kinds, not plugin identity.

• clients connect to services, not plugins
• one service endpoint serves one request kind only
• service names are the stable public contract
• provider plugin/process identity is an internal deployment detail

Examples of service kinds:

• cgroups-snapshot
• ip-to-asn
• pid-traffic

Global Implementation Rules

These rules apply to every service and every supported implementation language:

• C, Rust, and Go must expose the same wire contract and typed behavior.
• NetIPC has no backward-compatibility, forward-compatibility, or best-effort compatibility mode for provider/client drift. Method, layout, status, echoed-key, and generation contracts must match exactly or the call fails.
• Mixed-generation stitched lookup responses are globally unsupported. Do not add service-local compatibility shims or partial stitching rules.
• Lookup PAYLOAD_EXCEEDED and OVERSIZED_ITEM are standard response outcomes, not language-local behavior. Level 2 consumes PAYLOAD_EXCEEDED internally and treats OVERSIZED_ITEM as a final per-item outcome.
• Payload budgets and logical lookup ceilings are initialization policy. Implementations may provide named defaults, but call paths must read configured values or named defaults instead of embedding deployment-size literals.
• Named defaults are not protocol hard limits. Small deployments can choose smaller buffers; large-memory deployments can opt into larger budgets.

Repository layout

The repository mirrors Netdata's destination structure so that future integration requires minimal structural changes:

src/
  libnetdata/netipc/          # C library
    include/netipc/           # C public headers
    src/
      protocol/               # Codec: wire format encode/decode/builders
      transport/
        posix/                # L1: UDS SEQPACKET, POSIX SHM
        windows/              # L1: Named Pipe, Windows SHM
      service/                # L2/L3: typed client/server helpers
        netipc_service.c      # POSIX platform fault/memory/timing helpers
        netipc_service_posix_client.c         # POSIX public client API/config
        netipc_service_posix_client_connect.c # POSIX client connect/reconnect
        netipc_service_posix_client_call.c    # POSIX client raw-call flow
        netipc_service_posix_server.c         # POSIX server lifecycle/accept
        netipc_service_posix_server_session.c # POSIX server session loop
        netipc_service_win.c                  # Windows platform helpers
        netipc_service_win_client*.c          # Windows client API/connect/call
        netipc_service_win_server*.c          # Windows server lifecycle/session

  crates/netipc/              # Rust library (Cargo crate)
    src/
      protocol/               # Codec
        mod.rs                # Core wire primitives and codec re-exports
        increment.rs          # increment codec
        string_reverse.rs     # string-reverse codec
        cgroups_snapshot.rs   # cgroups-snapshot codec
        lookup/               # Lookup codec family
          common.rs           # Shared lookup helpers
          cgroups_lookup.rs   # cgroups lookup codec
          apps_lookup.rs      # apps lookup codec
      transport/
        posix.rs              # L1: UDS, SHM
        windows.rs            # L1: Named Pipe, SHM
      service/                # L2/L3: typed client/server helpers
        cgroups_snapshot.rs   # cgroups-snapshot public typed facade
        cgroups_lookup.rs     # cgroups-lookup public typed facade
        apps_lookup.rs        # apps-lookup public typed facade
        cgroups.rs            # legacy re-exports for historical imports
        raw.rs                # Internal raw helper wrapper/re-exports
        raw/                  # Shared raw infrastructure plus per-method helpers
          client.rs           # Raw client state and public lifecycle
          client_call.rs      # Shared retry/envelope/send-receive flow
          client_unix.rs      # POSIX connect and SHM attach
          client_windows.rs   # Windows connect and SHM attach
          server.rs           # Managed-server state and public lifecycle
          server_unix.rs      # POSIX accept and prepared-SHM setup
          server_windows.rs   # Windows accept and prepared-SHM setup
          cgroups_snapshot.rs # cgroups-snapshot raw call/dispatch
          cgroups_lookup.rs   # cgroups-lookup raw call/dispatch
          apps_lookup.rs      # apps-lookup raw call/dispatch
          cgroups_cache.rs    # cgroups-snapshot Level 3 cache

  go/pkg/netipc/              # Go library (Go package)
    protocol/                 # Codec
      frame.go                # Core wire primitives
      increment.go            # increment codec
      string_reverse.go       # string-reverse codec
      cgroups_snapshot.go     # cgroups-snapshot codec
      lookup_common.go        # Shared lookup helpers
      cgroups_lookup.go       # cgroups lookup codec
      apps_lookup.go          # apps lookup codec
    transport/
      posix/                  # L1: UDS, SHM
      windows/                # L1: Named Pipe, SHM
    service/                  # L2/L3: typed client/server helpers
      cgroups_snapshot/       # cgroups-snapshot public typed facade
      cgroups_lookup/         # cgroups-lookup public typed facade
      apps_lookup/            # apps-lookup public typed facade
      cgroups/                # legacy aliases for historical imports
      raw/                    # Internal raw helper infrastructure
        client.go             # shared raw client retry/envelope flow
        client_unix.go        # POSIX client connect/send/receive
        client_windows.go     # Windows client connect/send/receive
        server.go             # shared server dispatch helpers
        server_unix.go        # POSIX server accept/session loop
        server_windows.go     # Windows server accept/session loop
        *_unix.go             # per-method POSIX constructors
        *_windows.go          # per-method Windows constructors
        cgroups_cache*.go     # cgroups-snapshot Level 3 cache

tests/
  fixtures/                   # Helper binaries for live testing
    c/
    rust/
    go/
  *.sh                        # Validation and smoke scripts

bench/
  drivers/                    # Benchmark helper binaries

docs/                         # Specifications (this directory)

Module boundaries

Codec modules (protocol/)

Codec modules contain:

• Wire format constants (method codes, layout versions, sizes)
• Encode functions (typed structure to payload bytes)
• Decode functions (payload bytes to ephemeral view)
• Response builders
• Copy/materialize helpers

Each service-kind codec must have its own implementation file in each language. Shared wire helpers, directory parsing, label-table handling, alignment helpers, and other reusable codec infrastructure may live in common files, but custom request/response code for one codec must not be mixed with custom request/response code for another codec.

Examples:

• increment codec code lives in increment-specific files.
• string_reverse codec code lives in string-reverse-specific files.
• cgroups_snapshot codec code lives in cgroups-snapshot-specific files.
• cgroups_lookup codec code lives in cgroups-lookup-specific files.
• apps_lookup codec code lives in apps-lookup-specific files.
• shared lookup label/layout helpers live in lookup-common files.

Codec modules must NOT contain:

• Any transport or I/O code
• Any connection or session state
• Any Level 2 or Level 3 logic
• Any OS-specific code (codec is platform-independent)

Codec modules must NOT import transport modules.

Transport modules (transport/)

Transport modules contain:

• Connection lifecycle (connect, listen, accept, close)
• Handshake implementation (auth, profile negotiation, limits)
• Message send/receive (outer envelope framing)
• Batch assembly/extraction (item directory management)
• Chunking (transparent splitting/reassembly)
• Sequencing and message_id tracking
• Transport-specific mechanics (socket ops, pipe ops, SHM regions)
• Native wait-object exposure

Transport modules must NOT contain:

• Typed payload knowledge (no service-kind codec encode/decode)
• Callback dispatch or handler registration
• Retry or reconnect policy
• Cache or snapshot logic

Transport modules import the protocol module for L1 wire primitives (header encode/decode, chunk header, batch directory validation) but must NOT use service-kind codec functions. Level 1 treats all payloads as opaque bytes.

Service modules (service/)

Service modules contain:

• Level 2: typed client contexts, managed server mode, retry policy, and service-specific request/response orchestration
• Level 3: snapshot refresh helpers, local cache management, lookup functions

Service modules import both transport and codec modules. They compose Level 1 + Codec into the convenience surface.

Each service module should correspond to one service kind. The public L2/L3 shape must not drift into “one server exports many unrelated request kinds”.

Internal raw helpers may share connection lifecycle, retry policy, transport send/receive, managed server accept/session loops, and generic envelope validation. Custom typed client calls, typed handler aliases, dispatch adapters, and Level 3 cache logic for one service kind must live in service-kind-specific files so adding new service kinds does not expand a shared catch-all module.

When shared raw/helper files grow beyond one responsibility, split by goal before adding a new service kind. Keep platform fault/memory helpers, public client API/config mapping, client connect/reconnect, raw call send/receive/retry flow, server lifecycle/accept, and server session loops in separate files where the language layout supports it. POSIX and Windows implementations should remain explicit files when their wait, SHM, close, or wake-up behavior differs.

Service modules must NOT contain:

• Transport implementation details (no direct socket/pipe/SHM code)
• Wire format encoding/decoding (that belongs in codec)

Separation rules

Protocol module is shared infrastructure

The protocol module (protocol/) contains both L1 wire primitives (header, chunk header, batch directory) and service-kind codec functions. Transport modules import the protocol module for L1 wire primitives only. They must not call service-kind codec functions. Codec modules never call transport functions.

No upward dependencies

• Protocol depends on nothing
• Level 1 depends on protocol (wire primitives only)
• Level 2 depends on Level 1 + protocol (wire + method codecs)
• Level 3 depends on Level 2

No module may depend on a higher layer. Transport code must never import service code. Codec code must never import service code.

One transport implementation is the source of truth

High-level managed server mode (Level 2) is a thin wrapper over Level 1 listener/session primitives. It must not reimplement transport, framing, or negotiation logic. If the managed server needs a transport capability, that capability must exist as a Level 1 primitive.

Service modules are strictly layered

Level 3 snapshot helpers must use Level 2 typed call functions. They must not bypass Level 2 to call Level 1 directly. This ensures that retry policy, connection management, and typed dispatch remain consistent.

How to add a new transport

1. Add the transport implementation under transport/<platform>/ in each language.
2. The transport must implement the Level 1 transport interface: connect, listen, accept, send, receive, close, wait-object exposure.
3. The transport must support the handshake protocol (auth, profile negotiation, directional limits, packet size).
4. The transport must handle transparent chunking if it has packet size limits.
5. Register the transport profile in the negotiation bitmask.
6. Add the transport contract document under docs/level1-<name>.md.
7. Add tests: unit, interop (all language pairs), fuzz (header/chunk parsing), abnormal path (disconnect, stale recovery).
8. No codec or service module changes are needed — transports carry opaque bytes.

How to add a new message type

1. Define the wire layout contract in a new docs/codec-<name>.md file: method code, request payload layout, response payload layout, result codes, validation rules.
2. Add the encode/decode/builder functions in the codec module (protocol/) of each language (C, Rust, Go).
3. These codec functions are immediately available to Level 1 integrators.
4. Add Level 2 typed wrappers in the service module if convenience client/server support is needed. Public Level 2 client calls must accept and return typed values or typed views only. Public Level 2 server APIs must register typed callbacks only. Any request/response scratch buffers remain internal to Level 2.
5. Add tests: round-trip (all languages), cross-language interop (all pairs), fuzz (decode), boundary, validation rejection.
6. No transport module changes are needed — the new message type uses the existing Level 1 send/receive with opaque payloads.

How to add a Level 3 snapshot helper

1. The snapshot helper sits in the service module, built on Level 2 typed calls.
2. It manages: periodic refresh, local cache construction, lookup by key, cache preservation on refresh failure.
3. It must not bypass Level 2 for transport access.
4. Add the helper specification in docs/level3-<name>.md.
5. Add tests: refresh success, refresh failure with cache preservation, reconnect after failure, lookup correctness, empty cache behavior, large dataset behavior.

File size discipline

Transport and protocol modules must not grow into monolithic files that mix multiple concerns. Guidelines:

• Keep one service-kind codec API per implementation file in C, Rust, and Go.
• Separate client and server types into different files where the language supports it.
• Separate SHM transport from baseline transport into distinct files.
• Keep handshake/negotiation logic isolated from data-plane send/receive.
• Keep tests in dedicated test files (or test modules in Rust).

If a single file exceeds ~500 lines, consider whether it mixes concerns that should be separated.

Build system

CMake is the top-level orchestrator. Language-native manifests remain for their respective ecosystems:

• CMakeLists.txt: top-level build, test registration, benchmark targets
• Cargo.toml: Rust crate metadata (used by CMake via cargo invocation)
• go.mod: Go module metadata (used by CMake via go invocation)

All validation and benchmark workflows must be runnable through CMake targets. Shell scripts remain thin orchestrators that CMake invokes. Local validation and benchmark scripts self-reexec through tests/run-low-priority.sh so they run with low CPU/I/O scheduler priority on developer workstations.

Cross-language consistency

All three language implementations must:

• Produce identical wire bytes for the same typed input
• Accept identical wire bytes and produce equivalent typed views
• Use the same method codes, layout versions, and field offsets
• Apply the same validation rules during decoding
• Use the same negotiation bitmasks and profile constants
• Pass the same cross-language interop test suite

The wire contract documents in docs/ are the authority. If two implementations produce different bytes, at least one is wrong — check it against the spec.