Skip to content

Detect plugin process exit and reflect it in piped's /healthz - #7113

Open
vipulpandey21 wants to merge 1 commit into
pipe-cd:masterfrom
vipulpandey21:feat/detect-plugin-process-exit
Open

Detect plugin process exit and reflect it in piped's /healthz#7113
vipulpandey21 wants to merge 1 commit into
pipe-cd:masterfrom
vipulpandey21:feat/detect-plugin-process-exit

Conversation

@vipulpandey21

@vipulpandey21 vipulpandey21 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

What this PR does:

Makes piped actually notice when a plugin's process dies on its own, and makes /healthz say so, instead of only reacting when piped itself is shutting down.

Why we need it:

Right now, if a plugin process crashes (or gets OOM-killed, or anything else that kills it unexpectedly), piped does not notice at all. It keeps running, /healthz keeps saying ok, and any deployment that needs that plugin just keeps failing quietly. Kubernetes is already set up to restart piped whenever /healthz says something is wrong, so this fix mostly just makes that existing health check tell the truth. Full explanation with code references is in #7112.

What changed, in plain terms

  1. pkg/lifecycle/binary.go — piped already had a way to know when a plugin process exits (a channel that closes when it happens), it just was not exposed outside that file. I added two small getters:

    • Done() — a channel you can wait on, closes when the process exits, for any reason.
    • Err() — tells you the error it exited with, if any.
      Nothing existing changes, this just exposes something that was already being tracked.
  2. pkg/app/pipedv1/cmd/piped/pluginhealth.go (new file) — a very small helper that keeps a list of "plugins that are currently known to be dead". Nothing fancy, just a map with a lock around it, and its own tests.

  3. pkg/app/pipedv1/cmd/piped/piped.go:

    • /healthz now checks that list. If any plugin is dead, it returns a 503 and says which plugin(s) are down, instead of always saying ok.
    • Each plugin now gets watched the whole time piped is running, not just at shutdown. If a plugin dies early, piped logs it clearly and marks it as unhealthy. If piped is the one shutting down (normal case), it stops the plugin the same way as before.

I did not make piped try to automatically restart a crashed plugin. If a plugin keeps crashing, silently restarting it over and over could hide a real bug from whoever is running it. Instead, this just makes /healthz honest, and lets Kubernetes do what it is already set up to do: restart the whole pod when something is wrong.

How I tested it

Added tests for both new pieces:

  • pkg/lifecycle/binary_test.go — checks that the new Done()/Err() correctly reflect a normal exit, a crash (non-zero exit code), and a graceful stop, using real short-lived processes the same way the existing tests in that file already do.
  • pkg/app/pipedv1/cmd/piped/pluginhealth_test.go — checks marking a plugin unhealthy, marking the same one twice (should not duplicate), multiple plugins, and that it is safe if many goroutines touch it at once.
go test ./pkg/lifecycle/... ./pkg/app/pipedv1/cmd/piped/... -v
--- PASS: TestCommandDone
--- PASS: TestPluginHealth
--- PASS: TestPluginHealthConcurrentAccess
ok  	github.com/pipe-cd/pipecd/pkg/lifecycle
ok  	github.com/pipe-cd/pipecd/pkg/app/pipedv1/cmd/piped

Also ran:

  • go test -race on both packages — passed
  • go vet on both packages — clean
  • go build ./pkg/app/pipedv1/... ./cmd/... — to make sure nothing else broke
  • gofmt — clean

I did not add a full end-to-end test that starts a real plugin binary, kills it, and checks /healthz over HTTP. The function that wires up the whole piped agent (run() in piped.go) did not have any test coverage before this change either, so instead of trying to add the first big integration test for it in this same PR, I kept all the new logic in small pieces that are fully testable on their own (Command.Done/Err, pluginHealth).

Which issue(s) this PR fixes:

Fixes #7112

Does this PR introduce a user-facing change?:

  • How are users affected by this change: /healthz now returns 503 (and lists which plugin) if a configured plugin's process has died unexpectedly, instead of always returning 200 ok. Anyone using the standard Helm chart or quickstart manifests already has a liveness/readiness probe pointed at /healthz, so this means the pod now actually gets restarted automatically in that situation, instead of silently staying broken.
  • Is this breaking change: No. Behavior is unchanged as long as all plugins keep running normally, which is the common case.
  • How to migrate (if breaking change): Not applicable

Piped launches each plugin as a separate process and never checks on
it again until its own shutdown. If a plugin process dies on its own
(crash, OOM kill, etc.), piped keeps running, /healthz keeps saying
ok, and every deployment needing that plugin fails until someone
notices and restarts the pod by hand.

lifecycle.Command already tracks process exit internally through
stoppedCh, it was just never exposed or watched outside of shutdown.
This adds Done()/Err() accessors to read that signal, watches each
plugin for an unexpected exit alongside the existing shutdown path,
and lets /healthz report unhealthy plugins so the livenessProbe and
readinessProbe already pointed at /healthz in the shipped manifests
can actually restart the pod.

Signed-off-by: Vipul Subhash Pandey <vipulpandey7917@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

piped never notices when a plugin process exits, /healthz stays ok

1 participant