[ZEPPELIN-6575] Reclaim idle interpreters on the server with a per-setting timeout

[big-cir]

What is this PR for?

An interpreter setting cannot have its own idle timeout today, and worse, trying to give it one fails silently.

Idle reclaim is decided inside the interpreter process by TimeoutLifecycleManager, which reads its threshold from the configuration map the server pushes over Thrift when the process starts. That map is built by ZeppelinConfiguration#getCompleteConfiguration():

for (ConfVars c : ConfVars.values()) {
  if (getString(c) != null) {
    completeConfiguration.put(c.getVarName(), getString(c));
  }
}

Its key set is closed over the ConfVars enum, so an interpreter setting property has no slot to travel in. An operator can put the threshold property on a single interpreter's settings, it is stored, it is shown again when the form is reopened - and the process still starts with the global value. No error, no warning.

So the threshold is effectively all-or-nothing across every interpreter, which does not match how they differ in cost. A Spark interpreter holding tens of gigabytes of cluster memory is worth reclaiming aggressively; a JDBC interpreter that only keeps a few connections open is usually worth keeping. Today an operator who enables idle reclaim gets one number for both.

This moves the decision to the server, which already knows the per-setting value:

• ManagedInterpreterGroup records when the group was last used. The three hooks mirror the ones RemoteInterpreterServer already calls in-process (interpret, getProgress, getStatus), so no new notion of activity is introduced. getStatus matters most: RemoteScheduler.JobStatusPoller calls it while a paragraph runs, which is what keeps a long-running paragraph from having its interpreter pulled out from under it.
• IdleInterpreterReclaimer walks getAllInterpreterGroup() on a timer, resolves the threshold from the owning interpreter setting (falling back to the global property), and closes the group by reusing ManagedInterpreterGroup.close(). It reads in-memory state only and never calls isAlive()/isRunning(), whose cost depends on the launcher - a socket connect with a 1s timeout for docker, an unbounded kube-apiserver round trip for k8s - and this runs over every group on a timer.
• A group whose process is still launching is skipped. The handle field is assigned before start(), while the idle clock has been running since the group was created, so without this a launch slower than the threshold gets killed while coming up. Spark on YARN takes minutes to launch, so this is not hypothetical.

No new configuration property, and nothing changes unless asked for. This follows the existing lifecycle manager class property, which already expresses whether idle reclaim is wanted at all. Its NullLifecycleManager default means an existing deployment sees no change whatsoever; TimeoutLifecycleManager now also enables server-driven reclaim. Any other implementation is left untouched. ZeppelinConfiguration#getLifecycleManagerClass() had no caller before this change.

An interpreter setting overrides the threshold by carrying the same property name, where 0 or below means never reclaimed. This is opt-in per interpreter: a setting that carries nothing keeps following the global threshold exactly as before, so the JDBC interpreter above is still reclaimed on the global schedule until an operator marks it as exempt. What the change adds is the ability to say it at all.

TimeoutLifecycleManager is kept, not replaced. It still runs in the interpreter process, because that is what shuts a process down if the Zeppelin server itself exits unexpectedly and can no longer reclaim anything. It is handed the same threshold resolved here rather than the global one, so the two sides agree instead of one of them shutting a process down on the wrong schedule. Closing a group twice cannot happen either: if the process shuts itself down it unregisters, which removes the group from getAllInterpreterGroup(); if the server closes the group first, the process and its scheduler are gone.

That fallback is intentionally given up for one case only - a setting the operator marked as never reclaimed (0), whose process receives Long.MAX_VALUE and therefore will not self-terminate either. TimeoutLifecycleManager has no way to express "never" and would read a threshold of 0 as "shut down at the next check".

What type of PR is it?

Feature

Todos

• - Track last-used time per interpreter group on the server
• - Close groups idle beyond the threshold, reusing ManagedInterpreterGroup.close()
• - Resolve the threshold per interpreter setting, falling back to the global property
• - Keep the in-process fallback consistent by pushing the resolved threshold to the process
• - Skip groups whose process is still launching
• - Unit tests, including guards against probing and against reclaiming a launching group
• - Document the per-interpreter threshold in docs/usage/interpreter/overview.md and conf/zeppelin-site.xml.template

What is the Jira issue?

• ZEPPELIN-6575, a sub-task of ZEPPELIN-6568

How should this be tested?

IdleInterpreterReclaimerTest (9 tests) covers both directions of the override, the launching guard, the no-probe guard, threshold resolution, and that the default lifecycle manager changes nothing.

export JAVA_HOME=$(/usr/libexec/java_home -v 11)
./mvnw package -pl zeppelin-server --am \
  -Dtest=IdleInterpreterReclaimerTest,TimeoutLifecycleManagerTest -DfailIfNoTests=false

The two override tests fail before the change, because the per-setting value never reaches the process:

perSettingThresholdReclaimsEarlierThanTheGlobalOne
  the group should be reclaimed after the per setting threshold of 10s
  ==> expected: <0> but was: <1>

perSettingThresholdCanOptOutOfAShortGlobalThreshold
  the setting opted out of reclaim, so the short global threshold must not apply
  ==> expected: <1> but was: <0>

aGroupBeingLaunchedIsNotReclaimed fails without the launching guard, and scanNeverProbesTheInterpreterProcess fails if the scan is written with isAlive()/isRunning().

Local runs, all passing:

Scope	Result
org.apache.zeppelin.interpreter.** (19 classes, includes recovery and launcher tests)	pass
notebook, rest, service, socket, server, notebook.repo (39 classes, 318 tests)	pass
TimeoutLifecycleManagerTest (existing idle reclaim behaviour)	pass, no regression
./mvnw clean org.apache.rat:apache-rat-plugin:check -Prat	Unapproved: 0

Manual steps on a running server. This uses two of the lightweight built-in interpreters, md (markdown) and sh (shell), so that two interpreters run side by side under one server and one global threshold:

1. In zeppelin-site.xml set the lifecycle manager class to TimeoutLifecycleManager, the global threshold to 10s, and the check interval to 5s.
2. On the md interpreter setting only, add the threshold property with the value 0 to mark it as never reclaimed. Leave the sh setting untouched so that it follows the global 10s.
3. Run one %md paragraph and one %sh paragraph, then leave the note idle.

Each interpreter setting gets its own process, and each process logs the threshold it was handed:

logs/zeppelin-interpreter-md-shared_process-*.log
  TimeoutLifecycleManager is started with checkInterval: 5000, timeoutThreshold: 9223372036854775807

logs/zeppelin-interpreter-sh-shared_process-*.log
  TimeoutLifecycleManager is started with checkInterval: 5000, timeoutThreshold: 10000

Before this change both processes were handed the global 10000 and both were shut down after 10s of idle time; the 0 on the md setting had no effect at all. Now only sh is reclaimed, and the server log records why:

logs/zeppelin-*.log
  Reclaiming interpreter group sh-shared_process of interpreter setting sh:
    idle for 11603ms which exceeds its threshold of 10000ms

ps confirmed the sh process was gone about 15s after its last use, while the md process was still running after 33s of idle time. Both processes kept the lifecycle manager class the operator configured; only the threshold they received differed.

Not verified locally, left to CI and to deployments that have the runtimes: the docker, k8s and yarn launchers. The reclaim path does not call into a launcher - it neither probes nor launches, only closes - so the exposure is limited to close(), which the existing restart endpoint already uses.

Screenshots (if appropriate)

N/A

Questions:

• Does the license files need to update? No. The two new files carry the ASF header and apache-rat-plugin:check reports no unapproved files.
• Is there breaking changes for older versions? No. With the default NullLifecycleManager nothing is scheduled and no configuration is overridden for the interpreter process, so an untouched deployment behaves exactly as before.
• Does this needs documentation? Yes, and it is included. docs/usage/interpreter/overview.md gains a "Per interpreter idle threshold" section, and the descriptions in conf/zeppelin-site.xml.template are extended. No values in the template were changed.