From ec2079ac4539200f869b76bbe1c436ce81ed5d3d Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 17:39:41 -0700 Subject: [PATCH 01/65] feat: introduce macro operators (logical-plan-level skeleton) KNIME-metanode-style composite operators for Texera. Macros live purely at the logical-plan layer: a new MacroExpander pre-pass inlines each MacroOpDesc into a flat LogicalPlan before physical-plan compilation, so PhysicalPlan, PhysicalOpIdentity, and the Amber engine remain unchanged. Backend (new): - MacroOpDesc, MacroInputOp, MacroOutputOp LogicalOps registered in Jackson @JsonSubTypes; getPhysicalPlan throws to signal a missed expansion pass. - MacroBody, MacroLink, MacroPortSpec, MacroFusion data classes. - MacroExpander: inlines each macro by splicing inner ops/links via boundary markers and prefixes inner-op IDs with the instance ID (\${macroInstanceId}/\${innerOpId}), so per-macro telemetry can be aggregated purely from the operator-ID prefix. Cycle and depth-16 guards via MacroCompileContext. Pluggable MacroRegistry (Empty / inMemory; persistence-backed impl is a later step). - WorkflowCompiler (workflow-compiling-service) calls MacroExpander.expand before scan-source resolution. Backward- compatible: new ctor param defaults to MacroRegistry.Empty. - TODO note in amber WorkflowCompiler; execution-time expansion is a later step. Until then, MacroOpDesc.getPhysicalPlan throwing surfaces unexpanded plans as a loud compile error rather than silently broken execution. Tests (14 passing): - MacroOpDescSpec: JSON round-trip, throws on compile, ports match inputPortCount/outputPortCount. - MacroExpanderSpec: pass-through plan, single-port inline, LIVE registry fetch, nested macros with concatenated prefix, cycle detection, depth-bomb, double-instantiation, input-marker fan-out, missing-LIVE error, snapshot immutability across two expansions. Also includes hackathon-proposal.md (Texera Agent Hackathon submission) covering the AI suggestion and AI fusion features that layer on top of this skeleton in later steps. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../texera/workflow/WorkflowCompiler.scala | 5 + .../texera/amber/operator/LogicalOp.scala | 6 +- .../amber/operator/macroOp/MacroBody.scala | 33 ++ .../amber/operator/macroOp/MacroFusion.scala | 31 ++ .../amber/operator/macroOp/MacroInputOp.scala | 70 ++++ .../amber/operator/macroOp/MacroLink.scala | 33 ++ .../amber/operator/macroOp/MacroOpDesc.scala | 106 +++++ .../operator/macroOp/MacroOutputOp.scala | 70 ++++ .../operator/macroOp/MacroPortSpec.scala | 25 ++ .../operator/macroOp/MacroOpDescSpec.scala | 133 ++++++ hackathon-proposal.md | 25 ++ .../amber/compiler/WorkflowCompiler.scala | 22 +- .../macroOp/MacroCompileContext.scala | 56 +++ .../compiler/macroOp/MacroExpander.scala | 236 +++++++++++ .../compiler/macroOp/MacroRegistry.scala | 45 ++ .../compiler/macroOp/MacroExpanderSpec.scala | 388 ++++++++++++++++++ 16 files changed, 1279 insertions(+), 5 deletions(-) create mode 100644 common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroBody.scala create mode 100644 common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroFusion.scala create mode 100644 common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroInputOp.scala create mode 100644 common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroLink.scala create mode 100644 common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOpDesc.scala create mode 100644 common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOutputOp.scala create mode 100644 common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroPortSpec.scala create mode 100644 common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/macroOp/MacroOpDescSpec.scala create mode 100644 hackathon-proposal.md create mode 100644 workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroCompileContext.scala create mode 100644 workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala create mode 100644 workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroRegistry.scala create mode 100644 workflow-compiling-service/src/test/scala/org/apache/texera/amber/compiler/macroOp/MacroExpanderSpec.scala diff --git a/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala b/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala index b93aa3e4db3..effca5568d7 100644 --- a/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala +++ b/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala @@ -141,6 +141,11 @@ class WorkflowCompiler( def compile( logicalPlanPojo: LogicalPlanPojo ): Workflow = { + // TODO(macro-operators): macro expansion via MacroExpander needs to run here too + // before execution. The compiling-service compiler already does this; this path + // is used at execution time and must be plumbed in a later step. Until then, + // MacroOpDesc.getPhysicalPlan throws IllegalStateException, which surfaces as a + // loud compilation error rather than silently broken execution. // 1. convert the pojo to logical plan val logicalPlan: LogicalPlan = LogicalPlan(logicalPlanPojo) diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/LogicalOp.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/LogicalOp.scala index 4e9d6c6e2cd..2082514acc7 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/LogicalOp.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/LogicalOp.scala @@ -60,6 +60,7 @@ import org.apache.texera.amber.operator.machineLearning.sklearnAdvanced.KNNTrain } import org.apache.texera.amber.operator.machineLearning.sklearnAdvanced.SVCTrainer.SklearnAdvancedSVCTrainerOpDesc import org.apache.texera.amber.operator.machineLearning.sklearnAdvanced.SVRTrainer.SklearnAdvancedSVRTrainerOpDesc +import org.apache.texera.amber.operator.macroOp.{MacroInputOp, MacroOpDesc, MacroOutputOp} import org.apache.texera.amber.operator.metadata.{OPVersion, OperatorInfo, PropertyNameConstants} import org.apache.texera.amber.operator.projection.ProjectionOpDesc import org.apache.texera.amber.operator.randomksampling.RandomKSamplingOpDesc @@ -428,7 +429,10 @@ trait StateTransferFunc value = classOf[SklearnAdvancedSVRTrainerOpDesc], name = "SVRTrainer" ), - new Type(value = classOf[SklearnTestingOpDesc], name = "SklearnTesting") + new Type(value = classOf[SklearnTestingOpDesc], name = "SklearnTesting"), + new Type(value = classOf[MacroOpDesc], name = "Macro"), + new Type(value = classOf[MacroInputOp], name = "MacroInput"), + new Type(value = classOf[MacroOutputOp], name = "MacroOutput") ) ) abstract class LogicalOp extends PortDescriptor with Serializable { diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroBody.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroBody.scala new file mode 100644 index 00000000000..719cb1b16aa --- /dev/null +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroBody.scala @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.operator.macroOp + +import org.apache.texera.amber.operator.LogicalOp + +// The inner subgraph of a macro: inner operators (including MacroInputOp / +// MacroOutputOp boundary markers), internal links, and the declared external +// port specs. Serialized as JSON inside MacroOpDesc.snapshot (for SNAPSHOT +// mode) or returned by MacroRegistry.fetch (for LIVE mode). +case class MacroBody( + operators: List[LogicalOp], + links: List[MacroLink], + inputs: List[MacroPortSpec] = Nil, + outputs: List[MacroPortSpec] = Nil +) diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroFusion.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroFusion.scala new file mode 100644 index 00000000000..94a3fb1ce1d --- /dev/null +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroFusion.scala @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.operator.macroOp + +// AI-fusion payload (Section 9.2). When `verified = true`, MacroExpander substitutes +// the MacroOpDesc with a single PythonUDFOpDescV2 built from `code` instead of inlining +// the macro body. `sampleSize` records how many rows the sample-run diff matched on; +// `verifiedAt` is the epoch millis when verification passed. +case class MacroFusion( + code: String, + verified: Boolean = false, + sampleSize: Int = 0, + verifiedAt: Long = 0L +) diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroInputOp.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroInputOp.scala new file mode 100644 index 00000000000..8bc4627c233 --- /dev/null +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroInputOp.scala @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.operator.macroOp + +import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription} +import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle +import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity} +import org.apache.texera.amber.core.workflow.{OutputPort, PhysicalPlan, PortIdentity} +import org.apache.texera.amber.operator.LogicalOp +import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo} + +// Boundary marker that lives only inside a macro body. Represents external input port +// `portIndex` of the macro: tuples coming into the macro at that port flow out of this +// marker into the inner subgraph. MacroExpander consumes these markers when splicing +// the body into the parent plan and drops them from the expanded plan. +class MacroInputOp extends LogicalOp { + + @JsonProperty(value = "portIndex", required = true) + @JsonSchemaTitle("Port Index") + @JsonPropertyDescription("Which external input port (0-based) this marker represents.") + var portIndex: Int = 0 + + @JsonProperty(value = "displayName") + @JsonSchemaTitle("Display Name") + var displayName: String = "" + + override def getPhysicalOp( + workflowId: WorkflowIdentity, + executionId: ExecutionIdentity + ) = + throw new IllegalStateException( + s"MacroInputOp(portIndex=$portIndex) must be consumed by MacroExpander before " + + s"physical-plan compilation. Markers cannot be compiled directly." + ) + + override def getPhysicalPlan( + workflowId: WorkflowIdentity, + executionId: ExecutionIdentity + ): PhysicalPlan = + throw new IllegalStateException( + s"MacroInputOp(portIndex=$portIndex) must be consumed by MacroExpander before " + + s"physical-plan compilation. Markers cannot be compiled directly." + ) + + override def operatorInfo: OperatorInfo = OperatorInfo( + userFriendlyName = if (displayName.nonEmpty) displayName else s"Input $portIndex", + operatorDescription = + "Macro input boundary marker. External input port; consumed by MacroExpander.", + operatorGroupName = OperatorGroupConstants.UTILITY_GROUP, + inputPorts = List.empty, + outputPorts = List(OutputPort(PortIdentity(0))) + ) +} diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroLink.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroLink.scala new file mode 100644 index 00000000000..98beed015ae --- /dev/null +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroLink.scala @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.operator.macroOp + +import com.fasterxml.jackson.annotation.JsonProperty +import org.apache.texera.amber.core.workflow.PortIdentity + +// Mirrors LogicalLink's shape but lives in workflow-operator (which doesn't depend +// on workflow-compiling-service where LogicalLink lives). MacroExpander converts +// MacroLink → LogicalLink when inlining a macro body into the parent plan. +case class MacroLink( + @JsonProperty("fromOpId") fromOpId: String, + fromPortId: PortIdentity, + @JsonProperty("toOpId") toOpId: String, + toPortId: PortIdentity +) diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOpDesc.scala new file mode 100644 index 00000000000..f97455f6048 --- /dev/null +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOpDesc.scala @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.operator.macroOp + +import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription} +import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle +import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity} +import org.apache.texera.amber.core.workflow.{InputPort, OutputPort, PhysicalPlan, PortIdentity} +import org.apache.texera.amber.operator.LogicalOp +import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo} + +// A macro instance on the parent canvas. Carries identity + link mode + (optionally) +// an embedded body. MacroOpDesc never reaches physical-plan compilation: MacroExpander +// (in workflow-compiling-service) consumes it as a pre-compile pass and replaces it +// with the inlined body or, if `fusion` is verified, a single PythonUDFOpDescV2. +class MacroOpDesc extends LogicalOp { + + @JsonProperty(value = "macroId", required = true) + @JsonSchemaTitle("Macro ID") + @JsonPropertyDescription("Identifier of the macro definition (workflow ID).") + var macroId: String = "" + + @JsonProperty(value = "macroVersion") + @JsonSchemaTitle("Macro Version") + @JsonPropertyDescription("Pinned version (vid) of the macro definition. Used only in LIVE mode.") + var macroVersion: Int = 0 + + @JsonProperty(value = "linkMode", required = true) + @JsonSchemaTitle("Link Mode") + @JsonPropertyDescription("LIVE = referenced by (macroId, macroVersion); SNAPSHOT = embedded body.") + var linkMode: String = MacroOpDesc.LIVE + + @JsonProperty(value = "snapshot") + @JsonSchemaTitle("Snapshot") + @JsonPropertyDescription("Embedded macro body; present only when linkMode = SNAPSHOT.") + var snapshot: Option[MacroBody] = None + + @JsonProperty(value = "inputPortCount", required = true) + @JsonSchemaTitle("Input Port Count") + var inputPortCount: Int = 0 + + @JsonProperty(value = "outputPortCount", required = true) + @JsonSchemaTitle("Output Port Count") + var outputPortCount: Int = 0 + + @JsonProperty(value = "displayName") + @JsonSchemaTitle("Display Name") + var displayName: String = "" + + @JsonProperty(value = "fusion") + @JsonSchemaTitle("Fusion") + @JsonPropertyDescription( + "AI-fused single-UDF replacement (Section 9.2). When verified, MacroExpander uses this " + + "instead of inlining the body." + ) + var fusion: Option[MacroFusion] = None + + override def getPhysicalOp( + workflowId: WorkflowIdentity, + executionId: ExecutionIdentity + ) = + throw new IllegalStateException( + s"MacroOpDesc[$macroId] must be expanded by MacroExpander before physical-plan " + + s"compilation. This is a programmer error: the pre-compile expansion pass did not run." + ) + + override def getPhysicalPlan( + workflowId: WorkflowIdentity, + executionId: ExecutionIdentity + ): PhysicalPlan = + throw new IllegalStateException( + s"MacroOpDesc[$macroId] must be expanded by MacroExpander before physical-plan " + + s"compilation. This is a programmer error: the pre-compile expansion pass did not run." + ) + + override def operatorInfo: OperatorInfo = OperatorInfo( + userFriendlyName = if (displayName.nonEmpty) displayName else "Macro", + operatorDescription = "Composite operator: a reusable, encapsulated sub-workflow.", + operatorGroupName = OperatorGroupConstants.UTILITY_GROUP, + inputPorts = (0 until inputPortCount).toList.map(i => InputPort(PortIdentity(i))), + outputPorts = (0 until outputPortCount).toList.map(i => OutputPort(PortIdentity(i))) + ) +} + +object MacroOpDesc { + // Link modes — strings rather than an enum to keep Jackson serialization trivial. + val LIVE: String = "LIVE" + val SNAPSHOT: String = "SNAPSHOT" +} diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOutputOp.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOutputOp.scala new file mode 100644 index 00000000000..c2492afe911 --- /dev/null +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOutputOp.scala @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.operator.macroOp + +import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription} +import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle +import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity} +import org.apache.texera.amber.core.workflow.{InputPort, PhysicalPlan, PortIdentity} +import org.apache.texera.amber.operator.LogicalOp +import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo} + +// Boundary marker that lives only inside a macro body. Represents external output port +// `portIndex` of the macro: tuples flowing into this marker are emitted out of that +// external port. MacroExpander consumes these markers when splicing the body into the +// parent plan and drops them from the expanded plan. +class MacroOutputOp extends LogicalOp { + + @JsonProperty(value = "portIndex", required = true) + @JsonSchemaTitle("Port Index") + @JsonPropertyDescription("Which external output port (0-based) this marker represents.") + var portIndex: Int = 0 + + @JsonProperty(value = "displayName") + @JsonSchemaTitle("Display Name") + var displayName: String = "" + + override def getPhysicalOp( + workflowId: WorkflowIdentity, + executionId: ExecutionIdentity + ) = + throw new IllegalStateException( + s"MacroOutputOp(portIndex=$portIndex) must be consumed by MacroExpander before " + + s"physical-plan compilation. Markers cannot be compiled directly." + ) + + override def getPhysicalPlan( + workflowId: WorkflowIdentity, + executionId: ExecutionIdentity + ): PhysicalPlan = + throw new IllegalStateException( + s"MacroOutputOp(portIndex=$portIndex) must be consumed by MacroExpander before " + + s"physical-plan compilation. Markers cannot be compiled directly." + ) + + override def operatorInfo: OperatorInfo = OperatorInfo( + userFriendlyName = if (displayName.nonEmpty) displayName else s"Output $portIndex", + operatorDescription = + "Macro output boundary marker. External output port; consumed by MacroExpander.", + operatorGroupName = OperatorGroupConstants.UTILITY_GROUP, + inputPorts = List(InputPort(PortIdentity(0))), + outputPorts = List.empty + ) +} diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroPortSpec.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroPortSpec.scala new file mode 100644 index 00000000000..92099fc7a56 --- /dev/null +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroPortSpec.scala @@ -0,0 +1,25 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.operator.macroOp + +case class MacroPortSpec( + index: Int, + displayName: String = "" +) diff --git a/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/macroOp/MacroOpDescSpec.scala b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/macroOp/MacroOpDescSpec.scala new file mode 100644 index 00000000000..3feb4aa8471 --- /dev/null +++ b/common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/macroOp/MacroOpDescSpec.scala @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.operator.macroOp + +import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity} +import org.apache.texera.amber.core.workflow.PortIdentity +import org.apache.texera.amber.operator.LogicalOp +import org.apache.texera.amber.operator.limit.LimitOpDesc +import org.apache.texera.amber.util.JSONUtils.objectMapper +import org.scalatest.flatspec.AnyFlatSpec +import org.scalatest.matchers.should.Matchers + +class MacroOpDescSpec extends AnyFlatSpec with Matchers { + + "MacroOpDesc" should "round-trip through Jackson with all fields preserved" in { + val inner = new LimitOpDesc + inner.limit = 5 + inner.setOperatorId("inner-limit") + + val body = MacroBody( + operators = List(makeInputMarker(0, "in"), inner, makeOutputMarker(0, "out")), + links = List( + MacroLink("in", PortIdentity(0), "inner-limit", PortIdentity(0)), + MacroLink("inner-limit", PortIdentity(0), "out", PortIdentity(0)) + ), + inputs = List(MacroPortSpec(0, "the-input")), + outputs = List(MacroPortSpec(0, "the-output")) + ) + + val m = new MacroOpDesc + m.macroId = "wid-42" + m.macroVersion = 7 + m.linkMode = MacroOpDesc.SNAPSHOT + m.snapshot = Some(body) + m.inputPortCount = 1 + m.outputPortCount = 1 + m.displayName = "MyMacro" + m.setOperatorId("macro-instance-1") + + val json = objectMapper.writeValueAsString(m.asInstanceOf[LogicalOp]) + val restored = objectMapper.readValue(json, classOf[LogicalOp]) + + restored shouldBe a[MacroOpDesc] + val r = restored.asInstanceOf[MacroOpDesc] + r.macroId shouldBe "wid-42" + r.macroVersion shouldBe 7 + r.linkMode shouldBe MacroOpDesc.SNAPSHOT + r.inputPortCount shouldBe 1 + r.outputPortCount shouldBe 1 + r.displayName shouldBe "MyMacro" + r.operatorIdentifier.id shouldBe "macro-instance-1" + + r.snapshot shouldBe defined + val rb = r.snapshot.get + rb.operators should have size 3 + rb.links should have size 2 + rb.inputs shouldBe body.inputs + rb.outputs shouldBe body.outputs + + // Polymorphic round-trip: inner ops keep their concrete types. + rb.operators.collect { case l: LimitOpDesc => l.limit } shouldBe List(5) + rb.operators.collect { case i: MacroInputOp => i.portIndex } shouldBe List(0) + rb.operators.collect { case o: MacroOutputOp => o.portIndex } shouldBe List(0) + } + + it should "throw on getPhysicalPlan / getPhysicalOp because expansion must run first" in { + val m = new MacroOpDesc + m.macroId = "x" + val wid = WorkflowIdentity(0L) + val eid = ExecutionIdentity(0L) + assertThrows[IllegalStateException] { m.getPhysicalPlan(wid, eid) } + assertThrows[IllegalStateException] { m.getPhysicalOp(wid, eid) } + } + + "MacroInputOp / MacroOutputOp" should "round-trip and throw on compile" in { + val in = makeInputMarker(2, "in-2") + val out = makeOutputMarker(3, "out-3") + val inJson = objectMapper.writeValueAsString(in.asInstanceOf[LogicalOp]) + val outJson = objectMapper.writeValueAsString(out.asInstanceOf[LogicalOp]) + + val restoredIn = + objectMapper.readValue(inJson, classOf[LogicalOp]).asInstanceOf[MacroInputOp] + val restoredOut = + objectMapper.readValue(outJson, classOf[LogicalOp]).asInstanceOf[MacroOutputOp] + restoredIn.portIndex shouldBe 2 + restoredOut.portIndex shouldBe 3 + + val wid = WorkflowIdentity(0L) + val eid = ExecutionIdentity(0L) + assertThrows[IllegalStateException] { restoredIn.getPhysicalPlan(wid, eid) } + assertThrows[IllegalStateException] { restoredOut.getPhysicalPlan(wid, eid) } + } + + "MacroOpDesc.operatorInfo" should "expose ports matching inputPortCount/outputPortCount" in { + val m = new MacroOpDesc + m.inputPortCount = 2 + m.outputPortCount = 3 + val info = m.operatorInfo + info.inputPorts.map(_.id.id) shouldBe List(0, 1) + info.outputPorts.map(_.id.id) shouldBe List(0, 1, 2) + } + + private def makeInputMarker(idx: Int, id: String): MacroInputOp = { + val m = new MacroInputOp + m.portIndex = idx + m.setOperatorId(id) + m + } + + private def makeOutputMarker(idx: Int, id: String): MacroOutputOp = { + val m = new MacroOutputOp + m.portIndex = idx + m.setOperatorId(id) + m + } +} diff --git a/hackathon-proposal.md b/hackathon-proposal.md new file mode 100644 index 00000000000..9264cdc8f11 --- /dev/null +++ b/hackathon-proposal.md @@ -0,0 +1,25 @@ +# AI-Augmented Macro Operators for Texera + +## Problem +Texera workflows grow into 20–50+ operator DAGs with no encapsulation. Users copy-paste the same subgraphs across projects, and pipelines run slower than they need to because of inter-operator serialization. + +## What we'll build +- **Macro operators** — collapse a selection of operators into one reusable, version-pinned node (KNIME wrapped-metanode style). Drill-down to edit; drag from a library to reuse. +- **Agent tool: `suggestMacros`** — the agent inspects the `LogicalPlan` and proposes ranked subgraphs to encapsulate, each with a one-line rationale ("looks like a reusable text-preprocessing block"). Highlights candidates on the canvas; one click materializes. +- **Agent tool: `fuseMacro`** — for a macro whose internals the user no longer needs to inspect, the agent synthesizes an equivalent `PythonUDFOpDescV2`, runs original and fused on a sample, diffs outputs, and only swaps in after verification passes. + +## Why it fits the Agent Hackathon +- Plugs straight into the existing `agent-service` (Vercel AI SDK + ReAct loop + tool framework). Two new tools, no new LLM plumbing. +- The agent doesn't just suggest — it **verifies** (sample-run diff for fusion) and rolls back on mismatch. Concrete, measurable correctness. +- Showcases capabilities a generic chatbot can't: structural reasoning over a DAG, codegen for a known runtime, and a built-in verification harness. + +## Demo (~3 min) +1. Open a 15-operator workflow → "Suggest Macros (AI)" → three highlighted candidates appear with rationales. +2. Accept one → subgraph collapses into a single macro node. +3. Run → note baseline time. +4. Right-click macro → "Fuse for performance" → agent generates UDF, verifies ("matched on 1000 sample rows"), swaps in. +5. Re-run → show **2–5× speedup** on the stateless chain. + +## Stretch +- Cross-workflow pattern mining: "you've built this subgraph 4 times — save as a macro?" +- Auto-publish recurring patterns to the workflow hub as community macros. diff --git a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/WorkflowCompiler.scala b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/WorkflowCompiler.scala index 25166e7ac52..65e03bea062 100644 --- a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/WorkflowCompiler.scala +++ b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/WorkflowCompiler.scala @@ -25,6 +25,7 @@ import org.apache.texera.amber.compiler.WorkflowCompiler.{ collectOutputSchemaFromPhysicalPlan, convertErrorListToWorkflowFatalErrorMap } +import org.apache.texera.amber.compiler.macroOp.{MacroExpander, MacroRegistry} import org.apache.texera.amber.compiler.model.{LogicalPlan, LogicalPlanPojo} import org.apache.texera.amber.core.tuple.Schema import org.apache.texera.amber.core.virtualidentity.OperatorIdentity @@ -122,7 +123,8 @@ case class WorkflowCompilationResult( ) class WorkflowCompiler( - context: WorkflowContext + context: WorkflowContext, + macroRegistry: MacroRegistry = MacroRegistry.Empty ) extends LazyLogging { // function to expand logical plan to physical plan @@ -205,12 +207,24 @@ class WorkflowCompiler( val errorList = new ArrayBuffer[(OperatorIdentity, Throwable)]() var opIdToOutputSchema: Map[OperatorIdentity, Map[PortIdentity, Option[Schema]]] = Map() // 1. convert the pojo to logical plan - val logicalPlan: LogicalPlan = LogicalPlan(logicalPlanPojo) + val rawLogicalPlan: LogicalPlan = LogicalPlan(logicalPlanPojo) - // 2. resolve the file name in each scan source operator + // 2. expand any macro operators into a flat logical plan. Macros are a purely + // logical-plan-level abstraction; after this pass the rest of the pipeline never + // sees a MacroOpDesc / MacroInputOp / MacroOutputOp. + val logicalPlan: LogicalPlan = + try { + MacroExpander.expand(rawLogicalPlan, macroRegistry) + } catch { + case e: Throwable => + errorList.append((OperatorIdentity("__macro_expander__"), e)) + rawLogicalPlan + } + + // 3. resolve the file name in each scan source operator logicalPlan.resolveScanSourceOpFileName(Some(errorList)) - // 3. expand the logical plan to the physical plan + // 4. expand the logical plan to the physical plan val physicalPlan = expandLogicalPlan(logicalPlan, Some(errorList)) // 4. collect the output schema for each logical op diff --git a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroCompileContext.scala b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroCompileContext.scala new file mode 100644 index 00000000000..4c1ccdca3e0 --- /dev/null +++ b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroCompileContext.scala @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.compiler.macroOp + +// Threaded through MacroExpander to detect macro recursion and depth bombs. +// `visited` is the set of (macroId, version) pairs on the current expansion path; +// reappearance means a cycle. +case class MacroCompileContext( + visited: Set[(String, Int)], + depth: Int +) { + + def guardAgainstCycle(macroId: String, version: Int): Unit = { + if (visited.contains((macroId, version))) { + val path = visited.map { case (id, v) => s"$id@v$v" }.mkString(" -> ") + throw new IllegalStateException( + s"Macro cycle detected: $macroId@v$version is already being expanded on this path " + + s"(visited: $path)" + ) + } + } + + def guardAgainstDepth(): Unit = { + if (depth >= MacroCompileContext.MaxDepth) { + throw new IllegalStateException( + s"Macro expansion depth limit (${MacroCompileContext.MaxDepth}) exceeded — " + + s"likely a self-referential macro chain." + ) + } + } + + def descend(macroId: String, version: Int): MacroCompileContext = + MacroCompileContext(visited + ((macroId, version)), depth + 1) +} + +object MacroCompileContext { + val MaxDepth: Int = 16 + def root: MacroCompileContext = MacroCompileContext(Set.empty, 0) +} diff --git a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala new file mode 100644 index 00000000000..92122f0ddf1 --- /dev/null +++ b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala @@ -0,0 +1,236 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.compiler.macroOp + +import org.apache.texera.amber.compiler.model.{LogicalLink, LogicalPlan} +import org.apache.texera.amber.core.virtualidentity.OperatorIdentity +import org.apache.texera.amber.core.workflow.PortIdentity +import org.apache.texera.amber.operator.LogicalOp +import org.apache.texera.amber.operator.macroOp.{ + MacroBody, + MacroInputOp, + MacroLink, + MacroOpDesc, + MacroOutputOp +} +import org.apache.texera.amber.util.JSONUtils.objectMapper + +// Pre-compile pass: walks a LogicalPlan, inlines every MacroOpDesc by splicing its +// body's inner operators and links into the parent, and produces a flat LogicalPlan +// with no MacroOpDesc / MacroInputOp / MacroOutputOp nodes. Inner-op IDs are rewritten +// to "${macroInstanceId}/${innerOpId}" so telemetry can be aggregated per macro +// purely from the operator-ID prefix — the physical-plan layer remains macro-unaware. +object MacroExpander { + + def expand(plan: LogicalPlan, registry: MacroRegistry): LogicalPlan = + expand(plan, registry, MacroCompileContext.root) + + private def expand( + plan: LogicalPlan, + registry: MacroRegistry, + ctx: MacroCompileContext + ): LogicalPlan = { + // Each iteration picks the first remaining MacroOpDesc and inlines it. After + // inlining, the plan shape changes; loop re-scans the fresh `acc`. + var acc = plan + while (acc.operators.exists(_.isInstanceOf[MacroOpDesc])) { + val m = acc.operators.collectFirst { case x: MacroOpDesc => x }.get + acc = inlineMacro(acc, m, registry, ctx) + } + acc + } + + private def inlineMacro( + parent: LogicalPlan, + m: MacroOpDesc, + registry: MacroRegistry, + ctx: MacroCompileContext + ): LogicalPlan = { + ctx.guardAgainstCycle(m.macroId, m.macroVersion) + ctx.guardAgainstDepth() + + // TODO §9.2: if (m.fusion.exists(_.verified)) substitute a single + // PythonUDFOpDescV2 instead of fetching/inlining the body. Wired up in the + // AI-fusion step. + + val body: MacroBody = m.linkMode match { + case MacroOpDesc.SNAPSHOT => + m.snapshot.getOrElse( + throw new IllegalArgumentException( + s"MacroOpDesc[${m.macroId}] has linkMode=SNAPSHOT but no embedded snapshot" + ) + ) + case MacroOpDesc.LIVE => + registry + .fetch(m.macroId, m.macroVersion) + .getOrElse( + throw new IllegalArgumentException( + s"MacroOpDesc[${m.macroId}@v${m.macroVersion}] not found in registry " + + s"(LIVE link). The macro may be deleted or inaccessible." + ) + ) + case other => + throw new IllegalArgumentException( + s"MacroOpDesc[${m.macroId}] has unknown linkMode '$other'" + ) + } + + val expandedBody = expand( + LogicalPlan(body.operators, body.links.map(toLogicalLink)), + registry, + ctx.descend(m.macroId, m.macroVersion) + ) + + spliceIntoParent(parent, m, expandedBody) + } + + private def toLogicalLink(ml: MacroLink): LogicalLink = + LogicalLink( + OperatorIdentity(ml.fromOpId), + ml.fromPortId, + OperatorIdentity(ml.toOpId), + ml.toPortId + ) + + private def spliceIntoParent( + parent: LogicalPlan, + m: MacroOpDesc, + body: LogicalPlan + ): LogicalPlan = { + val instanceId = m.operatorIdentifier.id + val mId = m.operatorIdentifier + + val inputMarkers: Map[Int, MacroInputOp] = + body.operators.collect { case b: MacroInputOp => b.portIndex -> b }.toMap + val outputMarkers: Map[Int, MacroOutputOp] = + body.operators.collect { case b: MacroOutputOp => b.portIndex -> b }.toMap + + val markerIds: Set[OperatorIdentity] = + inputMarkers.values.map(_.operatorIdentifier).toSet ++ + outputMarkers.values.map(_.operatorIdentifier).toSet + + // Deep-clone non-marker inner ops via JSON round-trip and prefix their IDs. + val innerOps: List[LogicalOp] = body.operators.collect { + case op if !op.isInstanceOf[MacroInputOp] && !op.isInstanceOf[MacroOutputOp] => + deepClone(op) + } + + // (originalId → prefixedId) captured before mutating the cloned ops. + val idRewrite: Map[OperatorIdentity, OperatorIdentity] = innerOps.map { op => + val originalId = op.operatorIdentifier + val newId = s"$instanceId/${op.operatorIdentifier.id}" + op.setOperatorId(newId) + originalId -> op.operatorIdentifier + }.toMap + + def rewriteInnerId(id: OperatorIdentity): OperatorIdentity = + idRewrite.getOrElse( + id, + throw new IllegalStateException( + s"MacroExpander: link references unknown inner op '${id.id}' (instance=$instanceId)" + ) + ) + + // 1. Internal body links (non-marker → non-marker), with prefixed IDs. + val internalLinks: List[LogicalLink] = body.links.collect { + case l if !markerIds.contains(l.fromOpId) && !markerIds.contains(l.toOpId) => + LogicalLink(rewriteInnerId(l.fromOpId), l.fromPortId, rewriteInnerId(l.toOpId), l.toPortId) + } + + // 2. For each external input port, list the inner consumers connected via + // MacroInputOp_i. A port may fan out to multiple consumers. + val inputConsumers: Map[Int, List[(OperatorIdentity, PortIdentity)]] = + inputMarkers.map { + case (portIndex, marker) => + val markerId = marker.operatorIdentifier + val consumers = body.links + .filter(_.fromOpId == markerId) + .map(l => (rewriteInnerId(l.toOpId), l.toPortId)) + portIndex -> consumers + } + + // 3. For each external output port, the single inner producer feeding + // MacroOutputOp_j. More than one producer is a malformed body. + val outputProducers: Map[Int, (OperatorIdentity, PortIdentity)] = + outputMarkers.map { + case (portIndex, marker) => + val markerId = marker.operatorIdentifier + val producers = body.links + .filter(_.toOpId == markerId) + .map(l => (rewriteInnerId(l.fromOpId), l.fromPortId)) + producers match { + case single :: Nil => portIndex -> single + case Nil => + throw new IllegalStateException( + s"MacroOutputOp(portIndex=$portIndex) in macro $instanceId has no producer" + ) + case many => + throw new IllegalStateException( + s"MacroOutputOp(portIndex=$portIndex) in macro $instanceId has " + + s"${many.size} producers; expected exactly one." + ) + } + } + + // 4. Rewrite parent links that touch this macro instance. + val rewrittenParentLinks: List[LogicalLink] = parent.links.flatMap { link => + if (link.toOpId == mId) { + val portIndex = link.toPortId.id + inputConsumers.get(portIndex) match { + case Some(consumers) => + consumers.map { + case (innerOp, innerPort) => + LogicalLink(link.fromOpId, link.fromPortId, innerOp, innerPort) + } + case None => + throw new IllegalStateException( + s"Parent link into ($instanceId, port=$portIndex) has no matching " + + s"MacroInputOp inside the macro body." + ) + } + } else if (link.fromOpId == mId) { + val portIndex = link.fromPortId.id + outputProducers.get(portIndex) match { + case Some((innerOp, innerPort)) => + List(LogicalLink(innerOp, innerPort, link.toOpId, link.toPortId)) + case None => + throw new IllegalStateException( + s"Parent link out of ($instanceId, port=$portIndex) has no matching " + + s"MacroOutputOp inside the macro body." + ) + } + } else { + List(link) + } + } + + val newOps = + parent.operators.filterNot(_.operatorIdentifier == mId) ++ innerOps + val newLinks = rewrittenParentLinks ++ internalLinks + LogicalPlan(newOps, newLinks) + } + + // Deep-clone via JSON round-trip. Avoids mutating the persisted body when we + // rewrite inner-op IDs in spliceIntoParent. + private def deepClone(op: LogicalOp): LogicalOp = { + val json = objectMapper.writeValueAsString(op) + objectMapper.readValue(json, classOf[LogicalOp]) + } +} diff --git a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroRegistry.scala b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroRegistry.scala new file mode 100644 index 00000000000..59ac5cbf5f3 --- /dev/null +++ b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroRegistry.scala @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.compiler.macroOp + +import org.apache.texera.amber.operator.macroOp.MacroBody + +// Looks up a macro definition's body by (macroId, version). The persistence-backed +// implementation lives in the amber service and queries workflow_version; tests and +// services without persistence can use Empty or inMemory. +trait MacroRegistry { + def fetch(macroId: String, version: Int): Option[MacroBody] +} + +object MacroRegistry { + + // Always returns None. Use when persistence is not wired up — SNAPSHOT macros still + // work since their body is embedded; LIVE macros fail with "not found in registry". + object Empty extends MacroRegistry { + override def fetch(macroId: String, version: Int): Option[MacroBody] = None + } + + // For tests: a fixed table of bodies keyed by (id, version). + def inMemory(bodies: Map[(String, Int), MacroBody]): MacroRegistry = + new MacroRegistry { + override def fetch(macroId: String, version: Int): Option[MacroBody] = + bodies.get((macroId, version)) + } +} diff --git a/workflow-compiling-service/src/test/scala/org/apache/texera/amber/compiler/macroOp/MacroExpanderSpec.scala b/workflow-compiling-service/src/test/scala/org/apache/texera/amber/compiler/macroOp/MacroExpanderSpec.scala new file mode 100644 index 00000000000..2a57c3b1548 --- /dev/null +++ b/workflow-compiling-service/src/test/scala/org/apache/texera/amber/compiler/macroOp/MacroExpanderSpec.scala @@ -0,0 +1,388 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.compiler.macroOp + +import org.apache.texera.amber.compiler.model.{LogicalLink, LogicalPlan} +import org.apache.texera.amber.core.workflow.PortIdentity +import org.apache.texera.amber.operator.limit.LimitOpDesc +import org.apache.texera.amber.operator.macroOp._ +import org.scalatest.flatspec.AnyFlatSpec +import org.scalatest.matchers.should.Matchers + +class MacroExpanderSpec extends AnyFlatSpec with Matchers { + + "MacroExpander" should "leave non-macro plans unchanged" in { + val src = limit("src", 0) + val sink = limit("sink", 1) + val plan = LogicalPlan( + operators = List(src, sink), + links = List( + LogicalLink(src.operatorIdentifier, PortIdentity(0), sink.operatorIdentifier, PortIdentity(0)) + ) + ) + val out = MacroExpander.expand(plan, MacroRegistry.Empty) + out.operators.map(_.operatorIdentifier.id).toSet shouldBe Set("src", "sink") + out.links.size shouldBe 1 + } + + it should "inline a single-port SNAPSHOT macro and prefix inner-op IDs" in { + val body = MacroBody( + operators = List(inMarker(0, "in"), limit("inner", 10), outMarker(0, "out")), + links = List( + MacroLink("in", PortIdentity(0), "inner", PortIdentity(0)), + MacroLink("inner", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val inst = snapshotInstance("MyMacro-1", "macro-A", body) + val src = limit("src", 0); val sink = limit("sink", 1) + val plan = LogicalPlan( + operators = List(src, inst, sink), + links = List( + LogicalLink(src.operatorIdentifier, PortIdentity(0), inst.operatorIdentifier, PortIdentity(0)), + LogicalLink(inst.operatorIdentifier, PortIdentity(0), sink.operatorIdentifier, PortIdentity(0)) + ) + ) + val out = MacroExpander.expand(plan, MacroRegistry.Empty) + + out.operators.exists(_.isInstanceOf[MacroOpDesc]) shouldBe false + out.operators.exists(_.isInstanceOf[MacroInputOp]) shouldBe false + out.operators.exists(_.isInstanceOf[MacroOutputOp]) shouldBe false + + out.operators.collect { case l: LimitOpDesc => l.operatorIdentifier.id }.toSet shouldBe + Set("src", "sink", "MyMacro-1/inner") + + val edges = out.links.map(l => (l.fromOpId.id, l.toOpId.id)).toSet + edges shouldBe Set("src" -> "MyMacro-1/inner", "MyMacro-1/inner" -> "sink") + } + + it should "fetch a LIVE-linked macro body from the registry" in { + val body = MacroBody( + operators = List(inMarker(0, "in"), limit("inner", 3), outMarker(0, "out")), + links = List( + MacroLink("in", PortIdentity(0), "inner", PortIdentity(0)), + MacroLink("inner", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val registry = MacroRegistry.inMemory(Map(("live-id", 4) -> body)) + + val inst = new MacroOpDesc + inst.macroId = "live-id" + inst.macroVersion = 4 + inst.linkMode = MacroOpDesc.LIVE + inst.inputPortCount = 1 + inst.outputPortCount = 1 + inst.setOperatorId("L-inst") + val src = limit("src", 0); val sink = limit("sink", 1) + val plan = LogicalPlan( + operators = List(src, inst, sink), + links = List( + LogicalLink(src.operatorIdentifier, PortIdentity(0), inst.operatorIdentifier, PortIdentity(0)), + LogicalLink(inst.operatorIdentifier, PortIdentity(0), sink.operatorIdentifier, PortIdentity(0)) + ) + ) + val out = MacroExpander.expand(plan, registry) + out.operators.collect { case l: LimitOpDesc => l.operatorIdentifier.id }.toSet shouldBe + Set("src", "sink", "L-inst/inner") + } + + it should "expand nested macros with concatenated ID prefixes" in { + val innerBody = MacroBody( + operators = List(inMarker(0, "in"), limit("inner-inner", 7), outMarker(0, "out")), + links = List( + MacroLink("in", PortIdentity(0), "inner-inner", PortIdentity(0)), + MacroLink("inner-inner", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val innerInst = snapshotInstance("Inner", "macro-inner", innerBody) + val outerBody = MacroBody( + operators = List(inMarker(0, "oin"), innerInst, outMarker(0, "oout")), + links = List( + MacroLink("oin", PortIdentity(0), "Inner", PortIdentity(0)), + MacroLink("Inner", PortIdentity(0), "oout", PortIdentity(0)) + ) + ) + val outer = snapshotInstance("Outer", "macro-outer", outerBody) + val src = limit("src", 0); val sink = limit("sink", 1) + val plan = LogicalPlan( + operators = List(src, outer, sink), + links = List( + LogicalLink(src.operatorIdentifier, PortIdentity(0), outer.operatorIdentifier, PortIdentity(0)), + LogicalLink(outer.operatorIdentifier, PortIdentity(0), sink.operatorIdentifier, PortIdentity(0)) + ) + ) + val out = MacroExpander.expand(plan, MacroRegistry.Empty) + val ids = out.operators.collect { case l: LimitOpDesc => l.operatorIdentifier.id }.toSet + ids should contain("Outer/Inner/inner-inner") + ids should contain("src") + ids should contain("sink") + val edges = out.links.map(l => (l.fromOpId.id, l.toOpId.id)).toSet + edges should contain("src" -> "Outer/Inner/inner-inner") + edges should contain("Outer/Inner/inner-inner" -> "sink") + } + + it should "detect a self-referential macro cycle" in { + val cycleId = "loop" + // A body that references the same macro again. + val recurInst = new MacroOpDesc + recurInst.macroId = cycleId + recurInst.macroVersion = 1 + recurInst.linkMode = MacroOpDesc.LIVE + recurInst.inputPortCount = 1 + recurInst.outputPortCount = 1 + recurInst.setOperatorId("self") + val body = MacroBody( + operators = List(inMarker(0, "in"), recurInst, outMarker(0, "out")), + links = List( + MacroLink("in", PortIdentity(0), "self", PortIdentity(0)), + MacroLink("self", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val registry = MacroRegistry.inMemory(Map((cycleId, 1) -> body)) + + val outer = new MacroOpDesc + outer.macroId = cycleId + outer.macroVersion = 1 + outer.linkMode = MacroOpDesc.LIVE + outer.inputPortCount = 1 + outer.outputPortCount = 1 + outer.setOperatorId("outer") + val src = limit("src", 0); val sink = limit("sink", 1) + val plan = LogicalPlan( + operators = List(src, outer, sink), + links = List( + LogicalLink(src.operatorIdentifier, PortIdentity(0), outer.operatorIdentifier, PortIdentity(0)), + LogicalLink(outer.operatorIdentifier, PortIdentity(0), sink.operatorIdentifier, PortIdentity(0)) + ) + ) + val ex = intercept[IllegalStateException] { MacroExpander.expand(plan, registry) } + ex.getMessage.toLowerCase should include("cycle") + } + + it should "fail with a depth-limit error on a long non-cyclic macro chain" in { + // Build a chain chain-0 → chain-1 → ... → chain-N (where each chain-i's body + // contains a macro instance referencing chain-(i+1)). Distinct macroIds, so the + // cycle guard cannot fire; depth guard must. + val n = MacroCompileContext.MaxDepth + 5 + val bodies: Map[(String, Int), MacroBody] = (0 until n).map { i => + val nextId = s"chain-${i + 1}" + val innerOp = + if (i < n - 1) { + val m = new MacroOpDesc + m.macroId = nextId + m.macroVersion = 1 + m.linkMode = MacroOpDesc.LIVE + m.inputPortCount = 1 + m.outputPortCount = 1 + m.setOperatorId(s"inst-$i") + m + } else { + limit(s"leaf-$i", 1) + } + val body = MacroBody( + operators = List(inMarker(0, s"in-$i"), innerOp, outMarker(0, s"out-$i")), + links = List( + MacroLink(s"in-$i", PortIdentity(0), innerOp.operatorIdentifier.id, PortIdentity(0)), + MacroLink(innerOp.operatorIdentifier.id, PortIdentity(0), s"out-$i", PortIdentity(0)) + ) + ) + (s"chain-$i", 1) -> body + }.toMap + + val registry = MacroRegistry.inMemory(bodies) + val outer = new MacroOpDesc + outer.macroId = "chain-0" + outer.macroVersion = 1 + outer.linkMode = MacroOpDesc.LIVE + outer.inputPortCount = 1 + outer.outputPortCount = 1 + outer.setOperatorId("outer") + val src = limit("src", 0); val sink = limit("sink", 1) + val plan = LogicalPlan( + operators = List(src, outer, sink), + links = List( + LogicalLink(src.operatorIdentifier, PortIdentity(0), outer.operatorIdentifier, PortIdentity(0)), + LogicalLink(outer.operatorIdentifier, PortIdentity(0), sink.operatorIdentifier, PortIdentity(0)) + ) + ) + val ex = intercept[IllegalStateException] { MacroExpander.expand(plan, registry) } + ex.getMessage.toLowerCase should include("depth") + } + + it should "give each instance its own prefix when the same macro is used twice" in { + val body = MacroBody( + operators = List(inMarker(0, "in"), limit("inner", 9), outMarker(0, "out")), + links = List( + MacroLink("in", PortIdentity(0), "inner", PortIdentity(0)), + MacroLink("inner", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val inst1 = snapshotInstance("first", "shared", body) + val inst2 = snapshotInstance("second", "shared", body) + val src = limit("src", 0); val sink = limit("sink", 1) + val plan = LogicalPlan( + operators = List(src, inst1, inst2, sink), + links = List( + LogicalLink(src.operatorIdentifier, PortIdentity(0), inst1.operatorIdentifier, PortIdentity(0)), + LogicalLink(inst1.operatorIdentifier, PortIdentity(0), inst2.operatorIdentifier, PortIdentity(0)), + LogicalLink(inst2.operatorIdentifier, PortIdentity(0), sink.operatorIdentifier, PortIdentity(0)) + ) + ) + val out = MacroExpander.expand(plan, MacroRegistry.Empty) + val ids = out.operators.collect { case l: LimitOpDesc => l.operatorIdentifier.id }.toSet + ids should contain("first/inner") + ids should contain("second/inner") + val edges = out.links.map(l => (l.fromOpId.id, l.toOpId.id)).toSet + edges shouldBe Set( + "src" -> "first/inner", + "first/inner" -> "second/inner", + "second/inner" -> "sink" + ) + } + + it should "fan out a single external input port to multiple inner consumers" in { + val body = MacroBody( + operators = List( + inMarker(0, "in"), + limit("consumerA", 1), + limit("consumerB", 2), + outMarker(0, "out") + ), + links = List( + MacroLink("in", PortIdentity(0), "consumerA", PortIdentity(0)), + MacroLink("in", PortIdentity(0), "consumerB", PortIdentity(0)), + MacroLink("consumerA", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val inst = snapshotInstance("FanOut", "macro-fan", body) + val src = limit("src", 0); val sink = limit("sink", 1) + val plan = LogicalPlan( + operators = List(src, inst, sink), + links = List( + LogicalLink(src.operatorIdentifier, PortIdentity(0), inst.operatorIdentifier, PortIdentity(0)), + LogicalLink(inst.operatorIdentifier, PortIdentity(0), sink.operatorIdentifier, PortIdentity(0)) + ) + ) + val out = MacroExpander.expand(plan, MacroRegistry.Empty) + val srcOutTargets = + out.links.filter(_.fromOpId == src.operatorIdentifier).map(_.toOpId.id).toSet + srcOutTargets shouldBe Set("FanOut/consumerA", "FanOut/consumerB") + } + + it should "fail clearly when a LIVE macro is missing from the registry" in { + val inst = new MacroOpDesc + inst.macroId = "missing" + inst.macroVersion = 5 + inst.linkMode = MacroOpDesc.LIVE + inst.inputPortCount = 1 + inst.outputPortCount = 1 + inst.setOperatorId("inst") + val src = limit("src", 0) + val plan = LogicalPlan( + operators = List(src, inst), + links = List( + LogicalLink(src.operatorIdentifier, PortIdentity(0), inst.operatorIdentifier, PortIdentity(0)) + ) + ) + val ex = intercept[IllegalArgumentException] { + MacroExpander.expand(plan, MacroRegistry.Empty) + } + ex.getMessage.toLowerCase should include("not found") + ex.getMessage should include("missing") + } + + it should "leave the persisted snapshot body unmutated across two expansions" in { + val body = MacroBody( + operators = List(inMarker(0, "in"), limit("inner", 1), outMarker(0, "out")), + links = List( + MacroLink("in", PortIdentity(0), "inner", PortIdentity(0)), + MacroLink("inner", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val inst = snapshotInstance("once", "m", body) + val src = limit("src", 0); val sink = limit("sink", 1) + val plan = LogicalPlan( + operators = List(src, inst, sink), + links = List( + LogicalLink(src.operatorIdentifier, PortIdentity(0), inst.operatorIdentifier, PortIdentity(0)), + LogicalLink(inst.operatorIdentifier, PortIdentity(0), sink.operatorIdentifier, PortIdentity(0)) + ) + ) + + val first = MacroExpander.expand(plan, MacroRegistry.Empty) + val innerInBodyAfterFirst = + body.operators.collectFirst { case l: LimitOpDesc => l.operatorIdentifier.id } + innerInBodyAfterFirst shouldBe Some("inner") // not "once/inner" — body wasn't mutated. + + // Re-expand a fresh plan that reuses the SAME body object: must still inline cleanly. + val inst2 = snapshotInstance("twice", "m", body) + val plan2 = LogicalPlan( + operators = List(src, inst2, sink), + links = List( + LogicalLink(src.operatorIdentifier, PortIdentity(0), inst2.operatorIdentifier, PortIdentity(0)), + LogicalLink(inst2.operatorIdentifier, PortIdentity(0), sink.operatorIdentifier, PortIdentity(0)) + ) + ) + val second = MacroExpander.expand(plan2, MacroRegistry.Empty) + val secondIds = second.operators.collect { case l: LimitOpDesc => l.operatorIdentifier.id }.toSet + secondIds should contain("twice/inner") + + val firstIds = first.operators.collect { case l: LimitOpDesc => l.operatorIdentifier.id }.toSet + firstIds should contain("once/inner") + } + + // ---------- helpers ---------- + + private def limit(id: String, lim: Int): LimitOpDesc = { + val l = new LimitOpDesc + l.limit = lim + l.setOperatorId(id) + l + } + + private def inMarker(idx: Int, id: String): MacroInputOp = { + val m = new MacroInputOp + m.portIndex = idx + m.setOperatorId(id) + m + } + + private def outMarker(idx: Int, id: String): MacroOutputOp = { + val m = new MacroOutputOp + m.portIndex = idx + m.setOperatorId(id) + m + } + + private def snapshotInstance( + instanceId: String, + macroId: String, + body: MacroBody + ): MacroOpDesc = { + val m = new MacroOpDesc + m.macroId = macroId + m.macroVersion = 1 + m.linkMode = MacroOpDesc.SNAPSHOT + m.snapshot = Some(body) + m.inputPortCount = 1 + m.outputPortCount = 1 + m.setOperatorId(instanceId) + m + } +} From 6e30c782851a4c99e0a3ade8ff56a06d32995176 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 18:42:17 -0700 Subject: [PATCH 02/65] feat(macro): persistence + REST endpoints for macro operators - sql/updates/23.sql + texera_ddl.sql: workflow_kind_enum, workflow.kind, idx_workflow_kind, macro_metadata. Macros reuse the workflow table to inherit versioning, ACL, and hub features. - MacroResource: create/list/get/schema/snapshot endpoints alongside WorkflowResource; reuses workflow_user_access for permissions and seeds an initial workflow_version so LIVE-mode instances have a vid to pin. - WorkflowResource.baseWorkflowSelect: bake in kind = WORKFLOW so macros are structurally excluded from the workflows tab, the hub, and operator search; callers (HubResource, retrieveWorkflowsBySessionUser) updated to .and(). - DbMacroRegistry: jOOQ-backed MacroRegistry that reads workflow.content as a serialized MacroBody; wired into the compiling service's WorkflowCompiler. - TexeraWebApplication: register MacroResource. The amber-side execution-time WorkflowCompiler still has the existing TODO(macro-operators) note from Step 1 and is unaffected; that hook is Step 3. --- .../texera/web/TexeraWebApplication.scala | 2 + .../resource/dashboard/hub/HubResource.scala | 2 +- .../user/workflow/MacroResource.scala | 319 ++++++++++++++++++ .../user/workflow/WorkflowResource.scala | 22 +- .../workflow/WorkflowVersionResource.scala | 4 +- sql/texera_ddl.sql | 26 +- sql/updates/23.sql | 55 +++ .../compiler/macroOp/DbMacroRegistry.scala | 78 +++++ .../WorkflowCompilationResource.scala | 7 +- 9 files changed, 504 insertions(+), 11 deletions(-) create mode 100644 amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/MacroResource.scala create mode 100644 sql/updates/23.sql create mode 100644 workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/DbMacroRegistry.scala diff --git a/amber/src/main/scala/org/apache/texera/web/TexeraWebApplication.scala b/amber/src/main/scala/org/apache/texera/web/TexeraWebApplication.scala index 98b7c68c974..a02d4a16ed6 100644 --- a/amber/src/main/scala/org/apache/texera/web/TexeraWebApplication.scala +++ b/amber/src/main/scala/org/apache/texera/web/TexeraWebApplication.scala @@ -47,6 +47,7 @@ import org.apache.texera.web.resource.dashboard.user.project.{ } import org.apache.texera.web.resource.dashboard.user.quota.UserQuotaResource import org.apache.texera.web.resource.dashboard.user.workflow.{ + MacroResource, WorkflowAccessResource, WorkflowExecutionsResource, WorkflowResource, @@ -148,6 +149,7 @@ class TexeraWebApplication environment.jersey.register(classOf[PublicProjectResource]) environment.jersey.register(classOf[WorkflowAccessResource]) environment.jersey.register(classOf[WorkflowResource]) + environment.jersey.register(classOf[MacroResource]) environment.jersey.register(classOf[HubResource]) environment.jersey.register(classOf[UserResource]) environment.jersey.register(classOf[WorkflowVersionResource]) diff --git a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/HubResource.scala b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/HubResource.scala index c4cb9ee3cbe..40602e52f19 100644 --- a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/HubResource.scala +++ b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/hub/HubResource.scala @@ -262,7 +262,7 @@ object HubResource { } val records = baseWorkflowSelect() - .where(WORKFLOW.WID.in(wids: _*)) + .and(WORKFLOW.WID.in(wids: _*)) .groupBy( WORKFLOW.WID, WORKFLOW.NAME, diff --git a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/MacroResource.scala b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/MacroResource.scala new file mode 100644 index 00000000000..b773a0185b3 --- /dev/null +++ b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/MacroResource.scala @@ -0,0 +1,319 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.web.resource.dashboard.user.workflow + +import com.fasterxml.jackson.databind.{JsonNode, ObjectMapper} +import com.fasterxml.jackson.module.scala.DefaultScalaModule +import com.typesafe.scalalogging.LazyLogging +import io.dropwizard.auth.Auth +import org.apache.texera.amber.operator.macroOp.MacroPortSpec +import org.apache.texera.auth.SessionUser +import org.apache.texera.dao.SqlServer +import org.apache.texera.dao.jooq.generated.Tables._ +import org.apache.texera.dao.jooq.generated.enums.{PrivilegeEnum, WorkflowKindEnum} +import org.apache.texera.dao.jooq.generated.tables.daos.{ + MacroMetadataDao, + WorkflowDao, + WorkflowOfUserDao, + WorkflowUserAccessDao +} +import org.apache.texera.dao.jooq.generated.tables.pojos.{ + MacroMetadata, + Workflow, + WorkflowOfUser, + WorkflowUserAccess +} +import org.apache.texera.web.resource.dashboard.user.workflow.MacroResource._ +import org.apache.texera.web.resource.dashboard.user.workflow.WorkflowAccessResource.{ + hasReadAccess, + hasWriteAccess +} +import org.jooq.{DSLContext, JSONB} + +import java.sql.Timestamp +import javax.annotation.security.RolesAllowed +import javax.ws.rs._ +import javax.ws.rs.core.MediaType +import scala.jdk.CollectionConverters._ + +/** + * REST endpoints for macro definitions. A macro is persisted as a `workflow` + * row with `kind = MACRO` plus a side row in `macro_metadata` carrying the + * denormalized port / parameter / palette-display fields. + * + * Macros reuse the workflow ACL machinery (`workflow_user_access`), so the + * standard `WorkflowAccessResource.hasReadAccess` / `hasWriteAccess` apply + * unchanged here. + */ +object MacroResource { + + private def context: DSLContext = SqlServer.getInstance().createDSLContext() + private def workflowDao = new WorkflowDao(context.configuration) + private def workflowOfUserDao = new WorkflowOfUserDao(context.configuration) + private def workflowUserAccessDao = new WorkflowUserAccessDao(context.configuration) + private def macroMetadataDao = new MacroMetadataDao(context.configuration) + + // Local mapper for the JSONB columns. The Scala module lets PortSpec and + // MacroPortSpec round-trip as case classes without extra annotations. + private val mapper: ObjectMapper = new ObjectMapper().registerModule(DefaultScalaModule) + + /** Request body for `POST /macro/create`. */ + case class MacroCreateRequest( + name: String, + description: Option[String] = None, + content: String, + isPublic: Boolean = false, + portSpec: PortSpec, + paramSpec: Option[JsonNode] = None, + category: Option[String] = None, + icon: Option[String] = None + ) + + /** Declared external boundary of a macro. */ + case class PortSpec( + inputs: List[MacroPortSpec] = Nil, + outputs: List[MacroPortSpec] = Nil + ) + + /** Full response for `POST /macro/create` and `GET /macro/{wid}`. */ + case class MacroDetail( + wid: Integer, + name: String, + description: String, + content: String, + creationTime: Timestamp, + lastModifiedTime: Timestamp, + isPublic: Boolean, + portSpec: PortSpec, + paramSpec: JsonNode, + category: Option[String], + icon: Option[String], + isOwner: Boolean, + readonly: Boolean + ) + + /** + * Lightweight row for `GET /macro/list`. `content` is intentionally omitted + * so the operator palette can render without pulling large LogicalPlan blobs + * over the wire. + */ + case class MacroSummary( + wid: Integer, + name: String, + description: String, + lastModifiedTime: Timestamp, + portSpec: PortSpec, + category: Option[String], + icon: Option[String] + ) + + /** + * Per-instance schema returned by `GET /macro/{wid}/schema`. In Phase 1 this + * holds the port spec only; Phase 2 will populate `params` from promoted + * parameters declared inside the macro body. + */ + case class MacroSchema( + inputs: List[MacroPortSpec], + outputs: List[MacroPortSpec], + params: List[JsonNode] + ) + + private def jsonbOf[T](value: T): JSONB = + JSONB.valueOf(mapper.writeValueAsString(value)) + + private def jsonbOfNode(node: JsonNode): JSONB = + JSONB.valueOf(mapper.writeValueAsString(node)) + + private def parsePortSpec(jsonb: JSONB): PortSpec = + Option(jsonb) + .map(j => mapper.readValue(j.data(), classOf[PortSpec])) + .getOrElse(PortSpec()) + + private def parseParamSpec(jsonb: JSONB): JsonNode = + Option(jsonb) + .map(j => mapper.readTree(j.data())) + .getOrElse(mapper.createArrayNode()) +} + +@Produces(Array(MediaType.APPLICATION_JSON)) +@Path("/macro") +class MacroResource extends LazyLogging { + + @POST + @Consumes(Array(MediaType.APPLICATION_JSON)) + @RolesAllowed(Array("REGULAR", "ADMIN")) + @Path("/create") + def create(req: MacroCreateRequest, @Auth sessionUser: SessionUser): MacroDetail = { + val user = sessionUser.getUser + + val workflow = new Workflow() + workflow.setName(req.name) + workflow.setDescription(req.description.orNull) + workflow.setContent(req.content) + workflow.setIsPublic(req.isPublic) + workflow.setKind(WorkflowKindEnum.MACRO) + workflowDao.insert(workflow) + + workflowOfUserDao.insert(new WorkflowOfUser(user.getUid, workflow.getWid)) + workflowUserAccessDao.insert( + new WorkflowUserAccess(user.getUid, workflow.getWid, PrivilegeEnum.WRITE) + ) + + // Seed v1 of the macro so LIVE-mode instances can pin to a concrete vid. + WorkflowVersionResource.insertVersion(workflow, insertingNewWorkflow = true) + + val metadata = new MacroMetadata( + workflow.getWid, + jsonbOf(req.portSpec), + jsonbOfNode(req.paramSpec.getOrElse(mapper.createArrayNode())), + req.category.orNull, + req.icon.orNull + ) + macroMetadataDao.insert(metadata) + + toDetail( + workflowDao.fetchOneByWid(workflow.getWid), + metadata, + isOwner = true, + readonly = false + ) + } + + @GET + @RolesAllowed(Array("REGULAR", "ADMIN")) + @Path("/list") + def list(@Auth sessionUser: SessionUser): List[MacroSummary] = { + val uid = sessionUser.getUser.getUid + val rows = context + .selectDistinct( + WORKFLOW.WID, + WORKFLOW.NAME, + WORKFLOW.DESCRIPTION, + WORKFLOW.LAST_MODIFIED_TIME, + MACRO_METADATA.PORT_SPEC, + MACRO_METADATA.CATEGORY, + MACRO_METADATA.ICON + ) + .from(WORKFLOW) + .join(WORKFLOW_USER_ACCESS) + .on(WORKFLOW_USER_ACCESS.WID.eq(WORKFLOW.WID)) + .leftJoin(MACRO_METADATA) + .on(MACRO_METADATA.WID.eq(WORKFLOW.WID)) + .where(WORKFLOW.KIND.eq(WorkflowKindEnum.MACRO)) + .and(WORKFLOW_USER_ACCESS.UID.eq(uid)) + .fetch() + + rows.asScala.map { r => + MacroSummary( + r.value1(), + r.value2(), + r.value3(), + r.value4(), + parsePortSpec(r.value5()), + Option(r.value6()), + Option(r.value7()) + ) + }.toList + } + + @GET + @RolesAllowed(Array("REGULAR", "ADMIN")) + @Path("/{wid}") + def get(@PathParam("wid") wid: Integer, @Auth sessionUser: SessionUser): MacroDetail = { + val uid = sessionUser.getUser.getUid + if (!hasReadAccess(wid, uid)) { + throw new ForbiddenException("No sufficient access privilege.") + } + val workflow = Option(workflowDao.fetchOneByWid(wid)) + .filter(_.getKind == WorkflowKindEnum.MACRO) + .getOrElse(throw new NotFoundException(s"Macro $wid not found")) + val metadata = Option(macroMetadataDao.fetchOneByWid(wid)) + .getOrElse(throw new NotFoundException(s"Macro $wid metadata missing")) + toDetail(workflow, metadata, isOwner = isOwner(wid, uid), readonly = !hasWriteAccess(wid, uid)) + } + + @GET + @RolesAllowed(Array("REGULAR", "ADMIN")) + @Path("/{wid}/schema") + def schema( + @PathParam("wid") wid: Integer, + @Auth sessionUser: SessionUser + ): MacroSchema = { + val uid = sessionUser.getUser.getUid + if (!hasReadAccess(wid, uid)) { + throw new ForbiddenException("No sufficient access privilege.") + } + val metadata = Option(macroMetadataDao.fetchOneByWid(wid)) + .getOrElse(throw new NotFoundException(s"Macro $wid metadata missing")) + val ports = parsePortSpec(metadata.getPortSpec) + MacroSchema(ports.inputs, ports.outputs, params = Nil) + } + + /** + * Returns the macro's serialized body so the frontend can inline it into a + * parent workflow as a SNAPSHOT instance (`MacroOpDesc.linkMode = SNAPSHOT`), + * detaching that instance from any future edits to the macro definition. + */ + @POST + @RolesAllowed(Array("REGULAR", "ADMIN")) + @Path("/{wid}/snapshot-into-instance") + def snapshotIntoInstance( + @PathParam("wid") wid: Integer, + @Auth sessionUser: SessionUser + ): String = { + val uid = sessionUser.getUser.getUid + if (!hasReadAccess(wid, uid)) { + throw new ForbiddenException("No sufficient access privilege.") + } + Option(workflowDao.fetchOneByWid(wid)) + .filter(_.getKind == WorkflowKindEnum.MACRO) + .map(_.getContent) + .getOrElse(throw new NotFoundException(s"Macro $wid not found")) + } + + private def isOwner(wid: Integer, uid: Integer): Boolean = + context + .selectCount() + .from(WORKFLOW_OF_USER) + .where(WORKFLOW_OF_USER.WID.eq(wid).and(WORKFLOW_OF_USER.UID.eq(uid))) + .fetchOne(0, classOf[Integer]) > 0 + + private def toDetail( + workflow: Workflow, + metadata: MacroMetadata, + isOwner: Boolean, + readonly: Boolean + ): MacroDetail = + MacroDetail( + workflow.getWid, + workflow.getName, + workflow.getDescription, + workflow.getContent, + workflow.getCreationTime, + workflow.getLastModifiedTime, + workflow.getIsPublic, + parsePortSpec(metadata.getPortSpec), + parseParamSpec(metadata.getParamSpec), + Option(metadata.getCategory), + Option(metadata.getIcon), + isOwner, + readonly + ) +} diff --git a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala index cb910d11c3c..7eccb1c26d0 100644 --- a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala +++ b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala @@ -28,7 +28,7 @@ import org.apache.texera.amber.core.virtualidentity.ExecutionIdentity import org.apache.texera.auth.SessionUser import org.apache.texera.dao.SqlServer import org.apache.texera.dao.jooq.generated.Tables._ -import org.apache.texera.dao.jooq.generated.enums.PrivilegeEnum +import org.apache.texera.dao.jooq.generated.enums.{PrivilegeEnum, WorkflowKindEnum} import org.apache.texera.dao.jooq.generated.tables.daos.{ WorkflowDao, WorkflowOfProjectDao, @@ -42,7 +42,7 @@ import org.apache.texera.web.resource.dashboard.hub.HubResource.recordCloneActio import org.apache.texera.web.resource.dashboard.user.workflow.WorkflowAccessResource.hasReadAccess import org.apache.texera.web.resource.dashboard.user.workflow.WorkflowResource._ import org.jooq.impl.DSL.{groupConcatDistinct, noCondition} -import org.jooq.{Condition, DSLContext, Record9, Result, SelectOnConditionStep} +import org.jooq.{Condition, DSLContext, Record9, Result, SelectConditionStep} import java.sql.Timestamp import java.util @@ -185,7 +185,13 @@ object WorkflowResource { } } - def baseWorkflowSelect(): SelectOnConditionStep[Record9[ + /** + * Base select used by the workflows tab, the hub, and other workflow + * listings. The `WORKFLOW.KIND = WORKFLOW` filter is baked in here so that + * macros (`KIND = MACRO`) never leak into endpoints meant for top-level + * workflows. Callers append their additional predicates with `.and(...)`. + */ + def baseWorkflowSelect(): SelectConditionStep[Record9[ Integer, String, String, @@ -217,6 +223,7 @@ object WorkflowResource { .on(USER.UID.eq(WORKFLOW_OF_USER.UID)) .leftJoin(WORKFLOW_OF_PROJECT) .on(WORKFLOW.WID.eq(WORKFLOW_OF_PROJECT.WID)) + .where(WORKFLOW.KIND.eq(WorkflowKindEnum.WORKFLOW)) } def mapWorkflowEntries( @@ -339,6 +346,7 @@ class WorkflowResource extends LazyLogging { .where( orCondition .and(WORKFLOW_USER_ACCESS.UID.eq(user.getUid)) + .and(WORKFLOW.KIND.eq(WorkflowKindEnum.WORKFLOW)) ) .fetch() @@ -363,7 +371,7 @@ class WorkflowResource extends LazyLogging { ): List[DashboardWorkflow] = { val user = sessionUser.getUser val workflowEntries = baseWorkflowSelect() - .where(WORKFLOW_USER_ACCESS.UID.eq(user.getUid)) + .and(WORKFLOW_USER_ACCESS.UID.eq(user.getUid)) .groupBy( WORKFLOW.WID, WORKFLOW.NAME, @@ -497,7 +505,8 @@ class WorkflowResource extends LazyLogging { assignNewOperatorIds(oldWorkflow.getContent), null, null, - false + false, + WorkflowKindEnum.WORKFLOW ), sessionUser ) @@ -544,7 +553,8 @@ class WorkflowResource extends LazyLogging { assignNewOperatorIds(oldWorkflow.getContent), null, null, - false + false, + WorkflowKindEnum.WORKFLOW ), sessionUser ) diff --git a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowVersionResource.scala b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowVersionResource.scala index 7be74ae5b00..58c039e9d1d 100644 --- a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowVersionResource.scala +++ b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowVersionResource.scala @@ -26,6 +26,7 @@ import org.apache.texera.auth.SessionUser import org.apache.texera.config.UserSystemConfig import org.apache.texera.dao.SqlServer import org.apache.texera.dao.jooq.generated.Tables.WORKFLOW_VERSION +import org.apache.texera.dao.jooq.generated.enums.WorkflowKindEnum import org.apache.texera.dao.jooq.generated.tables.daos.{WorkflowDao, WorkflowVersionDao} import org.apache.texera.dao.jooq.generated.tables.pojos.{Workflow, WorkflowVersion} import org.apache.texera.web.resource.dashboard.user.workflow.WorkflowResource.{ @@ -435,7 +436,8 @@ class WorkflowVersionResource { assignNewOperatorIds(workflowVersion.getContent), null, null, - false + false, + WorkflowKindEnum.WORKFLOW ), sessionUser ) diff --git a/sql/texera_ddl.sql b/sql/texera_ddl.sql index d6b488e582d..22a04a105e0 100644 --- a/sql/texera_ddl.sql +++ b/sql/texera_ddl.sql @@ -82,11 +82,13 @@ DROP TABLE IF EXISTS computing_unit_user_access CASCADE; DROP TYPE IF EXISTS user_role_enum CASCADE; DROP TYPE IF EXISTS privilege_enum CASCADE; DROP TYPE IF EXISTS action_enum CASCADE; +DROP TYPE IF EXISTS workflow_kind_enum CASCADE; CREATE TYPE user_role_enum AS ENUM ('INACTIVE', 'RESTRICTED', 'REGULAR', 'ADMIN'); CREATE TYPE action_enum AS ENUM ('like', 'unlike', 'view', 'clone'); CREATE TYPE privilege_enum AS ENUM ('NONE', 'READ', 'WRITE'); CREATE TYPE workflow_computing_unit_type_enum AS ENUM ('local', 'kubernetes'); +CREATE TYPE workflow_kind_enum AS ENUM ('WORKFLOW', 'MACRO'); -- ============================================ -- 5. Create tables @@ -121,6 +123,10 @@ CREATE TABLE IF NOT EXISTS user_config ); -- workflow +-- `kind` discriminates top-level workflows (WORKFLOW) from reusable macros +-- (MACRO). Macros are surfaced in the operator palette and a separate Macros +-- tab; their `content` follows the same LogicalPlan JSON shape with the +-- addition of MacroInputOp / MacroOutputOp boundary markers. CREATE TABLE IF NOT EXISTS workflow ( wid SERIAL PRIMARY KEY, @@ -129,9 +135,12 @@ CREATE TABLE IF NOT EXISTS workflow content TEXT NOT NULL, creation_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, last_modified_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, - is_public BOOLEAN NOT NULL DEFAULT false + is_public BOOLEAN NOT NULL DEFAULT false, + kind workflow_kind_enum NOT NULL DEFAULT 'WORKFLOW' ); +CREATE INDEX IF NOT EXISTS idx_workflow_kind ON workflow(kind); + -- workflow_of_user CREATE TABLE IF NOT EXISTS workflow_of_user ( @@ -435,6 +444,21 @@ CREATE TABLE IF NOT EXISTS computing_unit_user_access FOREIGN KEY (uid) REFERENCES "user"(uid) ON DELETE CASCADE ); +-- macro_metadata table +-- Denormalized macro descriptor used by palette/listing endpoints so they do +-- not have to parse workflow.content (a JSON-serialized LogicalPlan) per row. +-- port_spec captures the macro's declared external inputs/outputs; param_spec +-- captures promoted parameters (empty in v1, populated in Phase 2). +CREATE TABLE IF NOT EXISTS macro_metadata +( + wid INT PRIMARY KEY, + port_spec JSONB NOT NULL, + param_spec JSONB NOT NULL DEFAULT '[]'::JSONB, + category VARCHAR(128), + icon VARCHAR(64), + FOREIGN KEY (wid) REFERENCES workflow(wid) ON DELETE CASCADE +); + -- START Fulltext search index creation (DO NOT EDIT THIS LINE) CREATE EXTENSION IF NOT EXISTS pgroonga; diff --git a/sql/updates/23.sql b/sql/updates/23.sql new file mode 100644 index 00000000000..c9cd3e7aaa5 --- /dev/null +++ b/sql/updates/23.sql @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +\c texera_db + +SET search_path TO texera_db; + +BEGIN; + +-- Discriminator for workflow rows: WORKFLOW = top-level workflows surfaced in +-- the Workflows tab; MACRO = reusable subgraphs surfaced in the operator +-- palette and a separate Macros tab. Reusing the workflow table inherits +-- versioning, ACL, and hub features for free. +DO $$ BEGIN + CREATE TYPE workflow_kind_enum AS ENUM ('WORKFLOW', 'MACRO'); +EXCEPTION + WHEN duplicate_object THEN NULL; +END $$; + +ALTER TABLE workflow + ADD COLUMN IF NOT EXISTS kind workflow_kind_enum NOT NULL DEFAULT 'WORKFLOW'; + +CREATE INDEX IF NOT EXISTS idx_workflow_kind ON workflow(kind); + +-- Denormalized macro descriptor used by palette/listing endpoints so they do +-- not have to parse workflow.content (a JSON-serialized LogicalPlan) per row. +-- port_spec captures the macro's declared external inputs/outputs; param_spec +-- captures promoted parameters (empty in v1, populated in Phase 2). +CREATE TABLE IF NOT EXISTS macro_metadata +( + wid INT PRIMARY KEY, + port_spec JSONB NOT NULL, + param_spec JSONB NOT NULL DEFAULT '[]'::JSONB, + category VARCHAR(128), + icon VARCHAR(64), + FOREIGN KEY (wid) REFERENCES workflow(wid) ON DELETE CASCADE +); + +COMMIT; diff --git a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/DbMacroRegistry.scala b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/DbMacroRegistry.scala new file mode 100644 index 00000000000..0af98561f27 --- /dev/null +++ b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/DbMacroRegistry.scala @@ -0,0 +1,78 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.amber.compiler.macroOp + +import com.fasterxml.jackson.databind.ObjectMapper +import com.fasterxml.jackson.module.scala.DefaultScalaModule +import com.typesafe.scalalogging.LazyLogging +import org.apache.texera.amber.operator.macroOp.MacroBody +import org.apache.texera.dao.SqlServer +import org.apache.texera.dao.jooq.generated.Tables.WORKFLOW +import org.apache.texera.dao.jooq.generated.enums.WorkflowKindEnum + +import scala.util.control.NonFatal + +/** + * jOOQ + Jackson-backed [[MacroRegistry]] that loads a macro body from the + * `workflow` table. `workflow.content` for a macro row is treated as a + * JSON-serialized [[MacroBody]] (same shape produced by `MacroResource.create` + * on the amber side). + * + * v1 ignores the `version` argument and always reads the current row. Pinning + * to a specific `vid` requires reconstructing the body from + * `workflow_version`'s JSON patches and is deferred to Phase 2 alongside the + * explicit "published version" policy described in the design plan. + */ +class DbMacroRegistry extends MacroRegistry with LazyLogging { + + private val mapper = new ObjectMapper().registerModule(DefaultScalaModule) + + override def fetch(macroId: String, version: Int): Option[MacroBody] = { + val widOpt = + try Some(Integer.parseInt(macroId)) + catch { case _: NumberFormatException => None } + + widOpt.flatMap { wid => + try { + val record = SqlServer + .getInstance() + .createDSLContext() + .select(WORKFLOW.CONTENT, WORKFLOW.KIND) + .from(WORKFLOW) + .where(WORKFLOW.WID.eq(wid)) + .fetchOne() + if (record == null || record.value2() != WorkflowKindEnum.MACRO) { + None + } else { + Option(record.value1()) + .filter(_.nonEmpty) + .map(mapper.readValue(_, classOf[MacroBody])) + } + } catch { + case NonFatal(e) => + logger.error( + s"DbMacroRegistry: failed to load macro macroId=$macroId version=$version", + e + ) + None + } + } + } +} diff --git a/workflow-compiling-service/src/main/scala/org/apache/texera/service/resource/WorkflowCompilationResource.scala b/workflow-compiling-service/src/main/scala/org/apache/texera/service/resource/WorkflowCompilationResource.scala index f311f31d0b7..d80df162419 100644 --- a/workflow-compiling-service/src/main/scala/org/apache/texera/service/resource/WorkflowCompilationResource.scala +++ b/workflow-compiling-service/src/main/scala/org/apache/texera/service/resource/WorkflowCompilationResource.scala @@ -25,6 +25,7 @@ import jakarta.annotation.security.RolesAllowed import jakarta.ws.rs.core.MediaType import jakarta.ws.rs.{Consumes, POST, Path, Produces} import org.apache.texera.amber.compiler.WorkflowCompiler +import org.apache.texera.amber.compiler.macroOp.DbMacroRegistry import org.apache.texera.amber.compiler.model.LogicalPlanPojo import org.apache.texera.amber.core.tuple.Attribute import org.apache.texera.amber.core.virtualidentity.WorkflowIdentity @@ -68,8 +69,10 @@ class WorkflowCompilationResource extends LazyLogging { // a placeholder workflow context, as compiling a workflow doesn't require a wid from the frontend val context = new WorkflowContext(workflowId = WorkflowIdentity(0)) - // Compile the pojo using WorkflowCompiler - val compilationResult = new WorkflowCompiler(context).compile(logicalPlanPojo) + // Compile the pojo using WorkflowCompiler. The DB-backed registry resolves + // any LIVE-mode macro instances against the `workflow` table. + val compilationResult = + new WorkflowCompiler(context, new DbMacroRegistry()).compile(logicalPlanPojo) val operatorOutputSchemas = compilationResult.operatorIdToOutputSchemas.map { case (operatorIdentity, schemas) => From 404b277b4b063870a483e83a32fff1aab53551fa Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 18:52:01 -0700 Subject: [PATCH 03/65] feat(macro): wire MacroExpander into amber's execution-time compiler MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 3 closes the TODO at WorkflowCompiler.scala:144 — macros can now be executed end-to-end, not just compiled by the workflow-compiling-service. - amber/.../workflow/macroOp/{MacroCompileContext,MacroRegistry,MacroExpander, DbMacroRegistry}: parallel copies of the compiling-service equivalents, adapted to amber's LogicalLink/LogicalPlan types. The two macro pipelines will converge when the broader LogicalPlan unification (existing TODO at WorkflowCompiler.scala:137) happens. - WorkflowCompiler: take an optional MacroRegistry (defaults to Empty); call MacroExpander.expand before resolveScanSourceOpFileName + expandLogicalPlan. - WorkflowExecutionService, SyncExecutionResource: pass new DbMacroRegistry() into WorkflowCompiler so LIVE-mode macros resolve against `workflow` rows with kind=MACRO. Step 1 (10/10 MacroExpanderSpec, 4/4 MacroOpDescSpec) and amber's WorkflowCompilerSpec (6/6) still green. --- .../web/resource/SyncExecutionResource.scala | 3 +- .../service/WorkflowExecutionService.scala | 3 +- .../texera/workflow/WorkflowCompiler.scala | 20 +- .../workflow/macroOp/DbMacroRegistry.scala | 79 ++++++ .../macroOp/MacroCompileContext.scala | 61 +++++ .../workflow/macroOp/MacroExpander.scala | 232 ++++++++++++++++++ .../workflow/macroOp/MacroRegistry.scala | 48 ++++ 7 files changed, 435 insertions(+), 11 deletions(-) create mode 100644 amber/src/main/scala/org/apache/texera/workflow/macroOp/DbMacroRegistry.scala create mode 100644 amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroCompileContext.scala create mode 100644 amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala create mode 100644 amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroRegistry.scala diff --git a/amber/src/main/scala/org/apache/texera/web/resource/SyncExecutionResource.scala b/amber/src/main/scala/org/apache/texera/web/resource/SyncExecutionResource.scala index d3047db5802..b9756407204 100644 --- a/amber/src/main/scala/org/apache/texera/web/resource/SyncExecutionResource.scala +++ b/amber/src/main/scala/org/apache/texera/web/resource/SyncExecutionResource.scala @@ -50,6 +50,7 @@ import org.apache.texera.dao.SqlServer import org.apache.texera.dao.jooq.generated.Tables.OPERATOR_EXECUTIONS import org.apache.texera.web.model.websocket.request.{LogicalPlanPojo, WorkflowExecuteRequest} import org.apache.texera.workflow.{LogicalLink, WorkflowCompiler} +import org.apache.texera.workflow.macroOp.DbMacroRegistry import org.apache.texera.web.resource.dashboard.user.workflow.WorkflowExecutionsResource import org.apache.texera.web.service.{ExecutionResultService, WorkflowService} import org.apache.texera.web.storage.ExecutionStateStore.updateWorkflowState @@ -894,7 +895,7 @@ class SyncExecutionResource extends LazyLogging { ): Map[String, String] = { try { val tempContext = new WorkflowContext(WorkflowIdentity(workflowId)) - val compiler = new WorkflowCompiler(tempContext) + val compiler = new WorkflowCompiler(tempContext, new DbMacroRegistry()) compiler.compile(logicalPlan) Map.empty } catch { diff --git a/amber/src/main/scala/org/apache/texera/web/service/WorkflowExecutionService.scala b/amber/src/main/scala/org/apache/texera/web/service/WorkflowExecutionService.scala index 741687e02c9..bace62e34c4 100644 --- a/amber/src/main/scala/org/apache/texera/web/service/WorkflowExecutionService.scala +++ b/amber/src/main/scala/org/apache/texera/web/service/WorkflowExecutionService.scala @@ -39,6 +39,7 @@ import org.apache.texera.web.storage.ExecutionStateStore import org.apache.texera.web.storage.ExecutionStateStore.updateWorkflowState import org.apache.texera.web.{ComputingUnitMaster, SubscriptionManager, WebsocketInput} import org.apache.texera.workflow.WorkflowCompiler +import org.apache.texera.workflow.macroOp.DbMacroRegistry import java.net.URI import scala.collection.mutable @@ -105,7 +106,7 @@ class WorkflowExecutionService( def executeWorkflow(): Unit = { try { - workflow = new WorkflowCompiler(workflowContext) + workflow = new WorkflowCompiler(workflowContext, new DbMacroRegistry()) .compile(request.logicalPlan) } catch { case err: Throwable => diff --git a/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala b/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala index effca5568d7..4eacbec2bea 100644 --- a/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala +++ b/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala @@ -24,6 +24,7 @@ import org.apache.texera.amber.core.virtualidentity.OperatorIdentity import org.apache.texera.amber.core.workflow._ import org.apache.texera.amber.engine.architecture.controller.Workflow import org.apache.texera.web.model.websocket.request.LogicalPlanPojo +import org.apache.texera.workflow.macroOp.{MacroExpander, MacroRegistry} import scala.collection.mutable import scala.collection.mutable.ArrayBuffer @@ -31,7 +32,8 @@ import scala.jdk.CollectionConverters.IteratorHasAsScala import scala.util.{Failure, Success, Try} class WorkflowCompiler( - context: WorkflowContext + context: WorkflowContext, + macroRegistry: MacroRegistry = MacroRegistry.Empty ) extends LazyLogging { /** @@ -141,18 +143,18 @@ class WorkflowCompiler( def compile( logicalPlanPojo: LogicalPlanPojo ): Workflow = { - // TODO(macro-operators): macro expansion via MacroExpander needs to run here too - // before execution. The compiling-service compiler already does this; this path - // is used at execution time and must be plumbed in a later step. Until then, - // MacroOpDesc.getPhysicalPlan throws IllegalStateException, which surfaces as a - // loud compilation error rather than silently broken execution. // 1. convert the pojo to logical plan - val logicalPlan: LogicalPlan = LogicalPlan(logicalPlanPojo) + val rawLogicalPlan: LogicalPlan = LogicalPlan(logicalPlanPojo) - // 2. resolve the file name in each scan source operator + // 2. expand any macro operators into a flat logical plan. Macros are a purely + // logical-plan-level abstraction; after this pass the rest of the pipeline + // never sees a MacroOpDesc / MacroInputOp / MacroOutputOp. + val logicalPlan: LogicalPlan = MacroExpander.expand(rawLogicalPlan, macroRegistry) + + // 3. resolve the file name in each scan source operator logicalPlan.resolveScanSourceOpFileName(None) - // 3. expand the logical plan to the physical plan, and get a set of output ports that need storage + // 4. expand the logical plan to the physical plan, and get a set of output ports that need storage val (physicalPlan, outputPortsNeedingStorage) = expandLogicalPlan(logicalPlan, logicalPlanPojo.opsToViewResult, None) diff --git a/amber/src/main/scala/org/apache/texera/workflow/macroOp/DbMacroRegistry.scala b/amber/src/main/scala/org/apache/texera/workflow/macroOp/DbMacroRegistry.scala new file mode 100644 index 00000000000..10ee06d09f7 --- /dev/null +++ b/amber/src/main/scala/org/apache/texera/workflow/macroOp/DbMacroRegistry.scala @@ -0,0 +1,79 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.workflow.macroOp + +import com.fasterxml.jackson.databind.ObjectMapper +import com.fasterxml.jackson.module.scala.DefaultScalaModule +import com.typesafe.scalalogging.LazyLogging +import org.apache.texera.amber.operator.macroOp.MacroBody +import org.apache.texera.dao.SqlServer +import org.apache.texera.dao.jooq.generated.Tables.WORKFLOW +import org.apache.texera.dao.jooq.generated.enums.WorkflowKindEnum + +import scala.util.control.NonFatal + +/** + * jOOQ + Jackson-backed [[MacroRegistry]] for the amber execution-time + * compiler. Reads `workflow.content` as a JSON-serialized [[MacroBody]] — + * same shape produced by `MacroResource.create`. + * + * v1 ignores the `version` argument and always reads the current row. + * Reconstructing a specific `vid` from `workflow_version` patches is deferred + * to Phase 2. + * + * Duplicates the compiling-service `DbMacroRegistry`; both share the same + * `texera_db` schema so the body bytes round-trip identically across paths. + */ +class DbMacroRegistry extends MacroRegistry with LazyLogging { + + private val mapper = new ObjectMapper().registerModule(DefaultScalaModule) + + override def fetch(macroId: String, version: Int): Option[MacroBody] = { + val widOpt = + try Some(Integer.parseInt(macroId)) + catch { case _: NumberFormatException => None } + + widOpt.flatMap { wid => + try { + val record = SqlServer + .getInstance() + .createDSLContext() + .select(WORKFLOW.CONTENT, WORKFLOW.KIND) + .from(WORKFLOW) + .where(WORKFLOW.WID.eq(wid)) + .fetchOne() + if (record == null || record.value2() != WorkflowKindEnum.MACRO) { + None + } else { + Option(record.value1()) + .filter(_.nonEmpty) + .map(mapper.readValue(_, classOf[MacroBody])) + } + } catch { + case NonFatal(e) => + logger.error( + s"DbMacroRegistry: failed to load macro macroId=$macroId version=$version", + e + ) + None + } + } + } +} diff --git a/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroCompileContext.scala b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroCompileContext.scala new file mode 100644 index 00000000000..d7391c1d802 --- /dev/null +++ b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroCompileContext.scala @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.workflow.macroOp + +// Threaded through MacroExpander to detect macro recursion and depth bombs. +// `visited` is the set of (macroId, version) pairs on the current expansion path; +// reappearance means a cycle. +// +// Duplicate of the compiling-service equivalent; both versions track the same +// invariants because the amber-side and compiling-service-side WorkflowCompilers +// each maintain their own copy of the macro pipeline. They will converge when +// the broader LogicalPlan unification (see WorkflowCompiler.scala TODO) lands. +case class MacroCompileContext( + visited: Set[(String, Int)], + depth: Int +) { + + def guardAgainstCycle(macroId: String, version: Int): Unit = { + if (visited.contains((macroId, version))) { + val path = visited.map { case (id, v) => s"$id@v$v" }.mkString(" -> ") + throw new IllegalStateException( + s"Macro cycle detected: $macroId@v$version is already being expanded on this path " + + s"(visited: $path)" + ) + } + } + + def guardAgainstDepth(): Unit = { + if (depth >= MacroCompileContext.MaxDepth) { + throw new IllegalStateException( + s"Macro expansion depth limit (${MacroCompileContext.MaxDepth}) exceeded — " + + s"likely a self-referential macro chain." + ) + } + } + + def descend(macroId: String, version: Int): MacroCompileContext = + MacroCompileContext(visited + ((macroId, version)), depth + 1) +} + +object MacroCompileContext { + val MaxDepth: Int = 16 + def root: MacroCompileContext = MacroCompileContext(Set.empty, 0) +} diff --git a/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala new file mode 100644 index 00000000000..00a9b61c2d9 --- /dev/null +++ b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala @@ -0,0 +1,232 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.workflow.macroOp + +import org.apache.texera.amber.core.virtualidentity.OperatorIdentity +import org.apache.texera.amber.core.workflow.PortIdentity +import org.apache.texera.amber.operator.LogicalOp +import org.apache.texera.amber.operator.macroOp.{ + MacroBody, + MacroInputOp, + MacroLink, + MacroOpDesc, + MacroOutputOp +} +import org.apache.texera.amber.util.JSONUtils.objectMapper +import org.apache.texera.workflow.{LogicalLink, LogicalPlan} + +// Pre-compile pass for the amber execution-time compiler. Walks a LogicalPlan, +// inlines every MacroOpDesc by splicing its body's inner operators and links +// into the parent, and produces a flat LogicalPlan with no MacroOpDesc / +// MacroInputOp / MacroOutputOp nodes. Inner-op IDs are rewritten to +// "${macroInstanceId}/${innerOpId}" so telemetry can be aggregated per macro +// purely from the operator-ID prefix — the physical-plan layer remains +// macro-unaware. +// +// Mirrors the compiling-service MacroExpander; the two operate on their own +// LogicalLink/LogicalPlan classes and will converge once those types are +// unified (see WorkflowCompiler.scala TODO). +object MacroExpander { + + def expand(plan: LogicalPlan, registry: MacroRegistry): LogicalPlan = + expand(plan, registry, MacroCompileContext.root) + + private def expand( + plan: LogicalPlan, + registry: MacroRegistry, + ctx: MacroCompileContext + ): LogicalPlan = { + var acc = plan + while (acc.operators.exists(_.isInstanceOf[MacroOpDesc])) { + val m = acc.operators.collectFirst { case x: MacroOpDesc => x }.get + acc = inlineMacro(acc, m, registry, ctx) + } + acc + } + + private def inlineMacro( + parent: LogicalPlan, + m: MacroOpDesc, + registry: MacroRegistry, + ctx: MacroCompileContext + ): LogicalPlan = { + ctx.guardAgainstCycle(m.macroId, m.macroVersion) + ctx.guardAgainstDepth() + + // TODO §9.2: if (m.fusion.exists(_.verified)) substitute a single + // PythonUDFOpDescV2 instead of fetching/inlining the body. + + val body: MacroBody = m.linkMode match { + case MacroOpDesc.SNAPSHOT => + m.snapshot.getOrElse( + throw new IllegalArgumentException( + s"MacroOpDesc[${m.macroId}] has linkMode=SNAPSHOT but no embedded snapshot" + ) + ) + case MacroOpDesc.LIVE => + registry + .fetch(m.macroId, m.macroVersion) + .getOrElse( + throw new IllegalArgumentException( + s"MacroOpDesc[${m.macroId}@v${m.macroVersion}] not found in registry " + + s"(LIVE link). The macro may be deleted or inaccessible." + ) + ) + case other => + throw new IllegalArgumentException( + s"MacroOpDesc[${m.macroId}] has unknown linkMode '$other'" + ) + } + + val expandedBody = expand( + LogicalPlan(body.operators, body.links.map(toLogicalLink)), + registry, + ctx.descend(m.macroId, m.macroVersion) + ) + + spliceIntoParent(parent, m, expandedBody) + } + + private def toLogicalLink(ml: MacroLink): LogicalLink = + LogicalLink( + OperatorIdentity(ml.fromOpId), + ml.fromPortId, + OperatorIdentity(ml.toOpId), + ml.toPortId + ) + + private def spliceIntoParent( + parent: LogicalPlan, + m: MacroOpDesc, + body: LogicalPlan + ): LogicalPlan = { + val instanceId = m.operatorIdentifier.id + val mId = m.operatorIdentifier + + val inputMarkers: Map[Int, MacroInputOp] = + body.operators.collect { case b: MacroInputOp => b.portIndex -> b }.toMap + val outputMarkers: Map[Int, MacroOutputOp] = + body.operators.collect { case b: MacroOutputOp => b.portIndex -> b }.toMap + + val markerIds: Set[OperatorIdentity] = + inputMarkers.values.map(_.operatorIdentifier).toSet ++ + outputMarkers.values.map(_.operatorIdentifier).toSet + + // Deep-clone non-marker inner ops via JSON round-trip and prefix their IDs. + val innerOps: List[LogicalOp] = body.operators.collect { + case op if !op.isInstanceOf[MacroInputOp] && !op.isInstanceOf[MacroOutputOp] => + deepClone(op) + } + + val idRewrite: Map[OperatorIdentity, OperatorIdentity] = innerOps.map { op => + val originalId = op.operatorIdentifier + val newId = s"$instanceId/${op.operatorIdentifier.id}" + op.setOperatorId(newId) + originalId -> op.operatorIdentifier + }.toMap + + def rewriteInnerId(id: OperatorIdentity): OperatorIdentity = + idRewrite.getOrElse( + id, + throw new IllegalStateException( + s"MacroExpander: link references unknown inner op '${id.id}' (instance=$instanceId)" + ) + ) + + val internalLinks: List[LogicalLink] = body.links.collect { + case l if !markerIds.contains(l.fromOpId) && !markerIds.contains(l.toOpId) => + LogicalLink(rewriteInnerId(l.fromOpId), l.fromPortId, rewriteInnerId(l.toOpId), l.toPortId) + } + + val inputConsumers: Map[Int, List[(OperatorIdentity, PortIdentity)]] = + inputMarkers.map { + case (portIndex, marker) => + val markerId = marker.operatorIdentifier + val consumers = body.links + .filter(_.fromOpId == markerId) + .map(l => (rewriteInnerId(l.toOpId), l.toPortId)) + portIndex -> consumers + } + + val outputProducers: Map[Int, (OperatorIdentity, PortIdentity)] = + outputMarkers.map { + case (portIndex, marker) => + val markerId = marker.operatorIdentifier + val producers = body.links + .filter(_.toOpId == markerId) + .map(l => (rewriteInnerId(l.fromOpId), l.fromPortId)) + producers match { + case single :: Nil => portIndex -> single + case Nil => + throw new IllegalStateException( + s"MacroOutputOp(portIndex=$portIndex) in macro $instanceId has no producer" + ) + case many => + throw new IllegalStateException( + s"MacroOutputOp(portIndex=$portIndex) in macro $instanceId has " + + s"${many.size} producers; expected exactly one." + ) + } + } + + val rewrittenParentLinks: List[LogicalLink] = parent.links.flatMap { link => + if (link.toOpId == mId) { + val portIndex = link.toPortId.id + inputConsumers.get(portIndex) match { + case Some(consumers) => + consumers.map { + case (innerOp, innerPort) => + LogicalLink(link.fromOpId, link.fromPortId, innerOp, innerPort) + } + case None => + throw new IllegalStateException( + s"Parent link into ($instanceId, port=$portIndex) has no matching " + + s"MacroInputOp inside the macro body." + ) + } + } else if (link.fromOpId == mId) { + val portIndex = link.fromPortId.id + outputProducers.get(portIndex) match { + case Some((innerOp, innerPort)) => + List(LogicalLink(innerOp, innerPort, link.toOpId, link.toPortId)) + case None => + throw new IllegalStateException( + s"Parent link out of ($instanceId, port=$portIndex) has no matching " + + s"MacroOutputOp inside the macro body." + ) + } + } else { + List(link) + } + } + + val newOps = + parent.operators.filterNot(_.operatorIdentifier == mId) ++ innerOps + val newLinks = rewrittenParentLinks ++ internalLinks + LogicalPlan(newOps, newLinks) + } + + // Deep-clone via JSON round-trip to avoid mutating the persisted body when we + // rewrite inner-op IDs in spliceIntoParent. + private def deepClone(op: LogicalOp): LogicalOp = { + val json = objectMapper.writeValueAsString(op) + objectMapper.readValue(json, classOf[LogicalOp]) + } +} diff --git a/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroRegistry.scala b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroRegistry.scala new file mode 100644 index 00000000000..4ebc4323a66 --- /dev/null +++ b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroRegistry.scala @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.workflow.macroOp + +import org.apache.texera.amber.operator.macroOp.MacroBody + +// Looks up a macro definition's body by (macroId, version). The persistence-backed +// implementation lives next to this trait in the amber module ([[DbMacroRegistry]]); +// tests and degraded execution paths can use [[Empty]] or [[inMemory]]. +// +// Duplicates the compiling-service trait of the same name; see MacroCompileContext +// for why the macro pipeline is duplicated across the two compilers. +trait MacroRegistry { + def fetch(macroId: String, version: Int): Option[MacroBody] +} + +object MacroRegistry { + + // Always returns None. Use when persistence is not wired up — SNAPSHOT macros still + // work since their body is embedded; LIVE macros fail with "not found in registry". + object Empty extends MacroRegistry { + override def fetch(macroId: String, version: Int): Option[MacroBody] = None + } + + // For tests: a fixed table of bodies keyed by (id, version). + def inMemory(bodies: Map[(String, Int), MacroBody]): MacroRegistry = + new MacroRegistry { + override def fetch(macroId: String, version: Int): Option[MacroBody] = + bodies.get((macroId, version)) + } +} From 5e11274377638f803770569b0ee4f93f70984c4c Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 19:02:30 -0700 Subject: [PATCH 04/65] =?UTF-8?q?feat(macro):=20right-click=20"Create=20Ma?= =?UTF-8?q?cro"=20action=20=E2=80=94=20first=20UI=20surface?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the smallest user-visible hook for macros: select 2+ operators, right-click → "create macro", enter a name. Posts a serialized MacroBody (selected operators + internal links + MacroInput / MacroOutput boundary markers) to the new POST /api/macro/create endpoint and surfaces the result via a toast. The canvas selection is intentionally left in place; replacing it with a MacroOpDesc node (and rewiring boundary links to the new ports) is the next slice of Step 4, alongside the palette merge and drill-down editor. - macro.service.ts: HTTP client + boundary computation (one MacroInput per unique inner port that has an external feeder; mirror for MacroOutput). - context-menu.{html,ts}: new menu entry, wired with a window.prompt for the name and NotificationService for the toast. Shown only when 2+ operators are selected, no link is highlighted, and the workflow is modifiable. --- .../context-menu/context-menu.component.html | 12 + .../context-menu/context-menu.component.ts | 31 +- .../workspace/service/macro/macro.service.ts | 267 ++++++++++++++++++ 3 files changed, 309 insertions(+), 1 deletion(-) create mode 100644 frontend/src/app/workspace/service/macro/macro.service.ts diff --git a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html index 4465d65cb27..5d06515de42 100644 --- a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html +++ b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html @@ -123,6 +123,18 @@ nzTheme="twotone">remove reusing result +
  • + create macro +
  • + this.notificationService.success(`Macro "${detail.name}" created (wid=${detail.wid})`), + error: err => this.notificationService.error(`Failed to create macro: ${err?.message ?? err}`), + }); + } + public onClickExportHighlightedExecutionResult(): void { this.modalService.create({ nzTitle: "Export Highlighted Operators Result", diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts new file mode 100644 index 00000000000..8c1c34c9962 --- /dev/null +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -0,0 +1,267 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import { HttpClient } from "@angular/common/http"; +import { Injectable } from "@angular/core"; +import { Observable } from "rxjs"; +import { AppSettings } from "../../../common/app-setting"; +import { PortIdentity } from "../../types/execute-workflow.interface"; +import { WorkflowActionService } from "../workflow-graph/model/workflow-action.service"; + +export const MACRO_BASE_URL = "macro"; +export const MACRO_CREATE_URL = MACRO_BASE_URL + "/create"; +export const MACRO_LIST_URL = MACRO_BASE_URL + "/list"; + +// Mirrors the case classes on `MacroResource` (amber). Keeping the shapes +// hand-typed (rather than generating) so the dev loop stays simple. +export interface MacroPortSpec { + index: number; + displayName?: string; +} + +export interface PortSpec { + inputs: MacroPortSpec[]; + outputs: MacroPortSpec[]; +} + +export interface MacroCreateRequest { + name: string; + description?: string; + content: string; + isPublic?: boolean; + portSpec: PortSpec; + paramSpec?: unknown; + category?: string; + icon?: string; +} + +export interface MacroDetail { + wid: number; + name: string; + description: string; + content: string; + creationTime: string; + lastModifiedTime: string; + isPublic: boolean; + portSpec: PortSpec; + paramSpec: unknown; + category?: string; + icon?: string; + isOwner: boolean; + readonly: boolean; +} + +export interface MacroSummary { + wid: number; + name: string; + description: string; + lastModifiedTime: string; + portSpec: PortSpec; + category?: string; + icon?: string; +} + +// Shape that MacroExpander (backend) reads off `workflow.content`. Matches the +// MacroBody / MacroLink case classes in `common/workflow-operator`. +interface MacroBodyLink { + fromOpId: string; + fromPortId: PortIdentity; + toOpId: string; + toPortId: PortIdentity; +} + +interface MacroBody { + operators: unknown[]; + links: MacroBodyLink[]; + inputs: MacroPortSpec[]; + outputs: MacroPortSpec[]; +} + +@Injectable({ + providedIn: "root", +}) +export class MacroService { + constructor(private http: HttpClient) {} + + public createMacro(req: MacroCreateRequest): Observable { + return this.http.post(`${AppSettings.getApiEndpoint()}/${MACRO_CREATE_URL}`, req); + } + + public listMacros(): Observable { + return this.http.get(`${AppSettings.getApiEndpoint()}/${MACRO_LIST_URL}`); + } + + public getMacro(wid: number): Observable { + return this.http.get(`${AppSettings.getApiEndpoint()}/${MACRO_BASE_URL}/${wid}`); + } + + /** + * Build a `MacroCreateRequest` from the operators the user has multi-selected + * on the parent canvas. Caller is responsible for sending it via `createMacro`. + * + * Boundary handling: for every link crossing the selection edge we add a + * `MacroInput` / `MacroOutput` marker inside the body (one per unique inner + * port) and rewire it so MacroExpander can splice the body back into a + * parent at compile time. Internal links (both endpoints inside the + * selection) are passed through with port-ordinal IDs to match the + * backend's PortIdentity shape. + */ + public buildMacroCreateRequestFromSelection( + workflowActionService: WorkflowActionService, + selectedOperatorIDs: readonly string[], + name: string + ): MacroCreateRequest { + const graph = workflowActionService.getTexeraGraph(); + const selectedSet = new Set(selectedOperatorIDs); + + const innerOps = selectedOperatorIDs.map(opId => { + const op = graph.getOperator(opId); + // LogicalOp on the backend is reconstructed by Jackson from the same + // shape the compiler uses — flat properties merged with the structural + // bits (operatorID/Type/Version/ports). + return { + ...op.operatorProperties, + operatorID: op.operatorID, + operatorType: op.operatorType, + operatorVersion: op.operatorVersion, + inputPorts: op.inputPorts, + outputPorts: op.outputPorts, + }; + }); + + const inputPortOrdinal = (operatorID: string, portID: string): number => + graph.getOperator(operatorID).inputPorts.findIndex(p => p.portID === portID); + const outputPortOrdinal = (operatorID: string, portID: string): number => + graph.getOperator(operatorID).outputPorts.findIndex(p => p.portID === portID); + + const internal: { srcOp: string; srcPort: string; dstOp: string; dstPort: string }[] = []; + const incoming: { srcOp: string; srcPort: string; dstOp: string; dstPort: string }[] = []; + const outgoing: { srcOp: string; srcPort: string; dstOp: string; dstPort: string }[] = []; + + graph.getAllLinks().forEach(link => { + const entry = { + srcOp: link.source.operatorID, + srcPort: link.source.portID, + dstOp: link.target.operatorID, + dstPort: link.target.portID, + }; + const srcIn = selectedSet.has(entry.srcOp); + const dstIn = selectedSet.has(entry.dstOp); + if (srcIn && dstIn) internal.push(entry); + else if (!srcIn && dstIn) incoming.push(entry); + else if (srcIn && !dstIn) outgoing.push(entry); + }); + + // Allocate one MacroInput marker per unique (innerOp, innerPort) that is + // fed by at least one external link. A single marker can have multiple + // external feeders but it only drives one inner port. + const incomingKeys = Array.from(new Set(incoming.map(l => `${l.dstOp}|${l.dstPort}`))).sort(); + const inputMarkers = incomingKeys.map((key, idx) => { + const [innerOpId, innerPortID] = key.split("|"); + return { + markerOpId: `MacroInput-operator-${this.uuid()}`, + portIndex: idx, + innerOpId, + innerPortIdx: inputPortOrdinal(innerOpId, innerPortID), + }; + }); + + const outgoingKeys = Array.from(new Set(outgoing.map(l => `${l.srcOp}|${l.srcPort}`))).sort(); + const outputMarkers = outgoingKeys.map((key, idx) => { + const [innerOpId, innerPortID] = key.split("|"); + return { + markerOpId: `MacroOutput-operator-${this.uuid()}`, + portIndex: idx, + innerOpId, + innerPortIdx: outputPortOrdinal(innerOpId, innerPortID), + }; + }); + + const markerOps: unknown[] = [ + ...inputMarkers.map(m => ({ + operatorID: m.markerOpId, + operatorType: "MacroInput", + operatorVersion: "", + portIndex: m.portIndex, + displayName: "", + inputPorts: [], + outputPorts: [{ id: { id: 0, internal: false }, displayName: "" }], + })), + ...outputMarkers.map(m => ({ + operatorID: m.markerOpId, + operatorType: "MacroOutput", + operatorVersion: "", + portIndex: m.portIndex, + displayName: "", + inputPorts: [{ id: { id: 0, internal: false }, displayName: "" }], + outputPorts: [], + })), + ]; + + const internalLinks: MacroBodyLink[] = internal.map(l => ({ + fromOpId: l.srcOp, + fromPortId: { id: outputPortOrdinal(l.srcOp, l.srcPort), internal: false }, + toOpId: l.dstOp, + toPortId: { id: inputPortOrdinal(l.dstOp, l.dstPort), internal: false }, + })); + + const inputMarkerLinks: MacroBodyLink[] = inputMarkers.map(m => ({ + fromOpId: m.markerOpId, + fromPortId: { id: 0, internal: false }, + toOpId: m.innerOpId, + toPortId: { id: m.innerPortIdx, internal: false }, + })); + + const outputMarkerLinks: MacroBodyLink[] = outputMarkers.map(m => ({ + fromOpId: m.innerOpId, + fromPortId: { id: m.innerPortIdx, internal: false }, + toOpId: m.markerOpId, + toPortId: { id: 0, internal: false }, + })); + + const portSpec: PortSpec = { + inputs: inputMarkers.map(m => ({ index: m.portIndex })), + outputs: outputMarkers.map(m => ({ index: m.portIndex })), + }; + + const body: MacroBody = { + operators: [...innerOps, ...markerOps], + links: [...internalLinks, ...inputMarkerLinks, ...outputMarkerLinks], + inputs: portSpec.inputs, + outputs: portSpec.outputs, + }; + + return { + name, + content: JSON.stringify(body), + portSpec, + }; + } + + private uuid(): string { + // Lightweight RFC4122-ish ID; the actions service uses crypto.randomUUID + // elsewhere but we don't have it imported here and don't need strict + // collision resistance for marker ops within a single macro body. + return "xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g, c => { + const r = (Math.random() * 16) | 0; + const v = c === "x" ? r : (r & 0x3) | 0x8; + return v.toString(16); + }); + } +} From fdb7f4d12d6bb23ce5d0c0edbc024f6735c360bb Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 19:17:38 -0700 Subject: [PATCH 05/65] fix(macro): show "create macro" even when boundary links are highlighted Rubber-banding a chain of operators in JointJS picks up the connecting links too, which made `hasHighlightedLinks()` true and silently hid the menu entry (same reason copy/cut were missing from the user's screenshot). The boundary computation already classifies internal vs external links from the operator selection alone, so highlighted links shouldn't gate the entry. --- .../context-menu/context-menu/context-menu.component.html | 1 - 1 file changed, 1 deletion(-) diff --git a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html index 5d06515de42..46fa33a5500 100644 --- a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html +++ b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html @@ -126,7 +126,6 @@
  • Date: Thu, 14 May 2026 19:23:04 -0700 Subject: [PATCH 06/65] fix(macro): GET /workflow/{wid} returns 404 for macro rows Loading a MACRO row via the workflow editor route blew up the canvas (workflow-check.ts dereferenced link.source.operatorID; the content is a MacroBody, which has fromOpId/toOpId, not source/target). Fail fast at the REST layer instead, with a message pointing at the not-yet-built macro editor. --- .../dashboard/user/workflow/WorkflowResource.scala | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala index 7eccb1c26d0..e9f81a19a52 100644 --- a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala +++ b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala @@ -403,6 +403,15 @@ class WorkflowResource extends LazyLogging { ): WorkflowWithPrivilege = { if (WorkflowAccessResource.hasReadAccess(wid, user.getUid)) { val workflow = workflowDao.fetchOneByWid(wid) + // Macros share the workflow table but their `content` is a MacroBody, not + // a LogicalPlanPojo — loading one via the workflow editor would crash the + // canvas (see workflow-check.ts). Fail fast until the drill-down editor + // route exists. + if (workflow != null && workflow.getKind == WorkflowKindEnum.MACRO) { + throw new NotFoundException( + s"Workflow $wid is a macro definition; use the macro editor route instead." + ) + } WorkflowWithPrivilege( workflow.getName, workflow.getDescription, From 4808cffc5e3e0619a6640669dd2bea6fefbee84e Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 19:33:23 -0700 Subject: [PATCH 07/65] feat(macro): swap selected sub-DAG with a single MacroOp node on create MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After POST /api/macro/create succeeds the context menu now: 1. Drops a `Macro` operator at the centroid of the selection with input/output ports sized to match the boundary (one input per unique inner port that had an external feeder, mirror for output). 2. Deletes the original operators (and with them their internal + boundary links) via deleteOperatorsAndLinks. 3. Re-points each former external link at the new macro's corresponding port. All three steps are wrapped in a single bundleActions transaction so undo restores the original sub-DAG in one shot. MacroService.buildMacroFromSelection now returns the boundary metadata (per-link rewire instructions + input/output port counts) alongside the backend request payload — same boundary computation, exposed for the swap. MacroOpDesc on the canvas uses operatorProperties = { macroId, macroVersion, linkMode: "LIVE", inputPortCount, outputPortCount, displayName } so the existing workflow-serialization path can roundtrip it to the backend without extra glue. macroVersion is a placeholder until MacroDetail exposes the pinned vid. --- .../context-menu/context-menu.component.ts | 103 ++++++++++++++++-- .../workspace/service/macro/macro.service.ts | 51 ++++++++- 2 files changed, 140 insertions(+), 14 deletions(-) diff --git a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts index fdf25d02e10..5b9eab7b02b 100644 --- a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts @@ -31,8 +31,10 @@ import { NzMenuDirective, NzMenuItemComponent } from "ng-zorro-antd/menu"; import { NgIf } from "@angular/common"; import { ɵNzTransitionPatchDirective } from "ng-zorro-antd/core/transition-patch"; import { NzIconDirective } from "ng-zorro-antd/icon"; -import { MacroService } from "src/app/workspace/service/macro/macro.service"; +import { MacroService, MacroDetail } from "src/app/workspace/service/macro/macro.service"; import { NotificationService } from "src/app/common/service/notification/notification.service"; +import { WorkflowUtilService } from "src/app/workspace/service/workflow-graph/util/workflow-util.service"; +import { OperatorPredicate, Point } from "src/app/workspace/types/workflow-common.interface"; @UntilDestroy() @Component({ @@ -55,7 +57,8 @@ export class ContextMenuComponent { private modalService: NzModalService, private validationWorkflowService: ValidationWorkflowService, private macroService: MacroService, - private notificationService: NotificationService + private notificationService: NotificationService, + private workflowUtilService: WorkflowUtilService ) { this.registerWorkflowModifiableChangedHandler(); this.operatorMenuService.highlightedOperators$ @@ -152,8 +155,9 @@ export class ContextMenuComponent { */ /** * Bundles the highlighted operators into a new macro definition on the - * backend. The current selection is left on the canvas — the follow-up - * step will replace it with a MacroOpDesc node and rewire boundary links. + * backend, then replaces the selection on the canvas with a single MacroOp + * node that has the same external boundary (input/output ports rewired to + * whichever external operators were feeding / consuming the selection). */ public onCreateMacro(): void { const selected = Array.from(this.workflowActionService.getJointGraphWrapper().getCurrentHighlightedOperatorIDs()); @@ -164,17 +168,100 @@ export class ContextMenuComponent { if (!name) { return; } - const req = this.macroService.buildMacroCreateRequestFromSelection(this.workflowActionService, selected, name); + const built = this.macroService.buildMacroFromSelection(this.workflowActionService, selected, name); this.macroService - .createMacro(req) + .createMacro(built.request) .pipe(untilDestroyed(this)) .subscribe({ - next: detail => - this.notificationService.success(`Macro "${detail.name}" created (wid=${detail.wid})`), + next: detail => { + this.swapSelectionWithMacroNode(detail, selected, built); + this.notificationService.success(`Macro "${detail.name}" created (wid=${detail.wid})`); + }, error: err => this.notificationService.error(`Failed to create macro: ${err?.message ?? err}`), }); } + private swapSelectionWithMacroNode( + detail: MacroDetail, + selectedOpIDs: readonly string[], + built: { + incomingEdges: { externalOpId: string; externalPortID: string; macroPortIndex: number }[]; + outgoingEdges: { externalOpId: string; externalPortID: string; macroPortIndex: number }[]; + inputPortCount: number; + outputPortCount: number; + } + ): void { + const base = this.workflowUtilService.getNewOperatorPredicate("Macro"); + const inputPorts = Array.from({ length: built.inputPortCount }, (_, i) => ({ + portID: `input-${i}`, + displayName: `in-${i}`, + disallowMultiInputs: false, + isDynamicPort: false, + dependencies: [], + })); + const outputPorts = Array.from({ length: built.outputPortCount }, (_, i) => ({ + portID: `output-${i}`, + displayName: `out-${i}`, + isDynamicPort: false, + })); + const macroPredicate: OperatorPredicate = { + ...base, + operatorProperties: { + macroId: detail.wid.toString(), + // TODO: backend should expose the pinned vid on MacroDetail; defaulting + // to 1 until then (DbMacroRegistry ignores version in v1 anyway). + macroVersion: 1, + linkMode: "LIVE", + inputPortCount: built.inputPortCount, + outputPortCount: built.outputPortCount, + displayName: detail.name, + }, + inputPorts, + outputPorts, + customDisplayName: detail.name, + }; + + const jointWrapper = this.workflowActionService.getJointGraphWrapper(); + const positions = selectedOpIDs + .map(id => { + try { + return jointWrapper.getElementPosition(id); + } catch { + return undefined; + } + }) + .filter((p): p is Point => !!p); + const centroid: Point = + positions.length > 0 + ? { + x: positions.reduce((sum, p) => sum + p.x, 0) / positions.length, + y: positions.reduce((sum, p) => sum + p.y, 0) / positions.length, + } + : { x: 200, y: 200 }; + + this.workflowActionService.getTexeraGraph().bundleActions(() => { + // Order matters: add the macro node first so the rewired external links + // have a valid target/source. deleteOperatorsAndLinks then cleans up the + // old internal + boundary links automatically. + this.workflowActionService.addOperator(macroPredicate, centroid); + this.workflowActionService.deleteOperatorsAndLinks(Array.from(selectedOpIDs)); + built.incomingEdges.forEach(edge => + this.workflowActionService.addLink({ + linkID: this.workflowUtilService.getLinkRandomUUID(), + source: { operatorID: edge.externalOpId, portID: edge.externalPortID }, + target: { operatorID: macroPredicate.operatorID, portID: `input-${edge.macroPortIndex}` }, + }) + ); + built.outgoingEdges.forEach(edge => + this.workflowActionService.addLink({ + linkID: this.workflowUtilService.getLinkRandomUUID(), + source: { operatorID: macroPredicate.operatorID, portID: `output-${edge.macroPortIndex}` }, + target: { operatorID: edge.externalOpId, portID: edge.externalPortID }, + }) + ); + }); + } + public onClickExportHighlightedExecutionResult(): void { this.modalService.create({ nzTitle: "Export Highlighted Operators Result", diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index 8c1c34c9962..9afc44b13aa 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -113,7 +113,8 @@ export class MacroService { /** * Build a `MacroCreateRequest` from the operators the user has multi-selected - * on the parent canvas. Caller is responsible for sending it via `createMacro`. + * on the parent canvas, plus the boundary info the caller needs to swap the + * selection out for a single MacroOp node on the canvas. * * Boundary handling: for every link crossing the selection edge we add a * `MacroInput` / `MacroOutput` marker inside the body (one per unique inner @@ -121,12 +122,22 @@ export class MacroService { * parent at compile time. Internal links (both endpoints inside the * selection) are passed through with port-ordinal IDs to match the * backend's PortIdentity shape. + * + * The returned `incomingEdges` / `outgoingEdges` describe each external link + * that needs to be re-pointed at the new MacroOp instance (one entry per + * link, where multiple external feeders can share the same `macroPortIndex`). */ - public buildMacroCreateRequestFromSelection( + public buildMacroFromSelection( workflowActionService: WorkflowActionService, selectedOperatorIDs: readonly string[], name: string - ): MacroCreateRequest { + ): { + request: MacroCreateRequest; + incomingEdges: { externalOpId: string; externalPortID: string; macroPortIndex: number }[]; + outgoingEdges: { externalOpId: string; externalPortID: string; macroPortIndex: number }[]; + inputPortCount: number; + outputPortCount: number; + } { const graph = workflowActionService.getTexeraGraph(); const selectedSet = new Set(selectedOperatorIDs); @@ -178,6 +189,7 @@ export class MacroService { markerOpId: `MacroInput-operator-${this.uuid()}`, portIndex: idx, innerOpId, + innerPortID, innerPortIdx: inputPortOrdinal(innerOpId, innerPortID), }; }); @@ -189,6 +201,7 @@ export class MacroService { markerOpId: `MacroOutput-operator-${this.uuid()}`, portIndex: idx, innerOpId, + innerPortID, innerPortIdx: outputPortOrdinal(innerOpId, innerPortID), }; }); @@ -247,10 +260,36 @@ export class MacroService { outputs: portSpec.outputs, }; + // Per-link rewire instructions. Several external links may share the same + // macroPortIndex when they all target the same inner port. + const inputIdxByInnerPort = new Map( + inputMarkers.map(m => [`${m.innerOpId}|${m.innerPortID}`, m.portIndex]) + ); + const outputIdxByInnerPort = new Map( + outputMarkers.map(m => [`${m.innerOpId}|${m.innerPortID}`, m.portIndex]) + ); + + const incomingEdges = incoming.map(l => ({ + externalOpId: l.srcOp, + externalPortID: l.srcPort, + macroPortIndex: inputIdxByInnerPort.get(`${l.dstOp}|${l.dstPort}`) as number, + })); + const outgoingEdges = outgoing.map(l => ({ + externalOpId: l.dstOp, + externalPortID: l.dstPort, + macroPortIndex: outputIdxByInnerPort.get(`${l.srcOp}|${l.srcPort}`) as number, + })); + return { - name, - content: JSON.stringify(body), - portSpec, + request: { + name, + content: JSON.stringify(body), + portSpec, + }, + incomingEdges, + outgoingEdges, + inputPortCount: inputMarkers.length, + outputPortCount: outputMarkers.length, }; } From b7aaf04775552f0a1ba3381fac36b2cd4e2cc0d2 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 19:41:17 -0700 Subject: [PATCH 08/65] chore(macro): temporary console.log tracing in onCreateMacro Track down why the canvas swap isn't visible: log the captured selection, the built request + boundary metadata, and the swap-vs-throw outcome. Also align the output-port shape with outputPortToPortDescription (disallowMultiInputs: false). Tracing will be removed once the issue is identified. --- .../context-menu/context-menu.component.ts | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts index 5b9eab7b02b..c496b04c295 100644 --- a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts @@ -161,6 +161,7 @@ export class ContextMenuComponent { */ public onCreateMacro(): void { const selected = Array.from(this.workflowActionService.getJointGraphWrapper().getCurrentHighlightedOperatorIDs()); + console.log("[macro] onCreateMacro selected=", selected); if (selected.length < 2) { return; } @@ -169,15 +170,27 @@ export class ContextMenuComponent { return; } const built = this.macroService.buildMacroFromSelection(this.workflowActionService, selected, name); + console.log("[macro] built=", built); this.macroService .createMacro(built.request) .pipe(untilDestroyed(this)) .subscribe({ next: detail => { - this.swapSelectionWithMacroNode(detail, selected, built); + console.log("[macro] POST returned detail=", detail); + try { + this.swapSelectionWithMacroNode(detail, selected, built); + console.log("[macro] swap complete"); + } catch (e) { + console.error("[macro] swap threw:", e); + this.notificationService.error(`Swap failed: ${(e as Error)?.message ?? e}`); + return; + } this.notificationService.success(`Macro "${detail.name}" created (wid=${detail.wid})`); }, - error: err => this.notificationService.error(`Failed to create macro: ${err?.message ?? err}`), + error: err => { + console.error("[macro] POST failed:", err); + this.notificationService.error(`Failed to create macro: ${err?.message ?? err}`); + }, }); } @@ -202,6 +215,7 @@ export class ContextMenuComponent { const outputPorts = Array.from({ length: built.outputPortCount }, (_, i) => ({ portID: `output-${i}`, displayName: `out-${i}`, + disallowMultiInputs: false, isDynamicPort: false, })); const macroPredicate: OperatorPredicate = { From 34492abf645c25b077c39f134a1a0e741e60ca40 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 19:43:25 -0700 Subject: [PATCH 09/65] fix(macro): bypass getNewOperatorPredicate for Macro instance construction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MacroOpDesc's generated JSON schema includes \`nullable: true\` properties without a sibling \`type\` (from \`Option[MacroBody]\` / \`Option[MacroFusion]\`). Ajv refuses to compile that, so the swap threw with "nullable cannot be used without type" before any canvas mutation could happen. Construct the OperatorPredicate manually instead — every field is already overridden, so the schema-default path adds nothing. The underlying schema bug should still be fixed (it'll also break dragging Macro from the palette) but that's a separate task in workflow-operator; right-click → create macro now works without it. --- .../context-menu/context-menu.component.ts | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts index c496b04c295..809f34661ea 100644 --- a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts @@ -204,7 +204,12 @@ export class ContextMenuComponent { outputPortCount: number; } ): void { - const base = this.workflowUtilService.getNewOperatorPredicate("Macro"); + // Construct the predicate manually rather than going through + // WorkflowUtilService.getNewOperatorPredicate("Macro"): that path runs the + // schema through Ajv, and MacroOpDesc's generated schema is currently + // Ajv-invalid (Option[MacroBody] / Option[MacroFusion] produce + // `"nullable": true` without a sibling `"type"`). We override every field + // anyway, so the schema-default route adds no value here. const inputPorts = Array.from({ length: built.inputPortCount }, (_, i) => ({ portID: `input-${i}`, displayName: `in-${i}`, @@ -219,7 +224,9 @@ export class ContextMenuComponent { isDynamicPort: false, })); const macroPredicate: OperatorPredicate = { - ...base, + operatorID: `Macro-operator-${this.workflowUtilService.getOperatorRandomUUID()}`, + operatorType: "Macro", + operatorVersion: "", operatorProperties: { macroId: detail.wid.toString(), // TODO: backend should expose the pinned vid on MacroDetail; defaulting @@ -232,7 +239,11 @@ export class ContextMenuComponent { }, inputPorts, outputPorts, + showAdvanced: false, + isDisabled: false, customDisplayName: detail.name, + dynamicInputPorts: false, + dynamicOutputPorts: false, }; const jointWrapper = this.workflowActionService.getJointGraphWrapper(); From 3130d0835441648bc6cbab6933b2698ebddeecfc Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 19:58:53 -0700 Subject: [PATCH 10/65] chore(macro): remove diagnostic console.log tracing from onCreateMacro --- .../context-menu/context-menu.component.ts | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts index 809f34661ea..1365a585f3a 100644 --- a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts @@ -161,7 +161,6 @@ export class ContextMenuComponent { */ public onCreateMacro(): void { const selected = Array.from(this.workflowActionService.getJointGraphWrapper().getCurrentHighlightedOperatorIDs()); - console.log("[macro] onCreateMacro selected=", selected); if (selected.length < 2) { return; } @@ -170,27 +169,20 @@ export class ContextMenuComponent { return; } const built = this.macroService.buildMacroFromSelection(this.workflowActionService, selected, name); - console.log("[macro] built=", built); this.macroService .createMacro(built.request) .pipe(untilDestroyed(this)) .subscribe({ next: detail => { - console.log("[macro] POST returned detail=", detail); try { this.swapSelectionWithMacroNode(detail, selected, built); - console.log("[macro] swap complete"); } catch (e) { - console.error("[macro] swap threw:", e); this.notificationService.error(`Swap failed: ${(e as Error)?.message ?? e}`); return; } this.notificationService.success(`Macro "${detail.name}" created (wid=${detail.wid})`); }, - error: err => { - console.error("[macro] POST failed:", err); - this.notificationService.error(`Failed to create macro: ${err?.message ?? err}`); - }, + error: err => this.notificationService.error(`Failed to create macro: ${err?.message ?? err}`), }); } From 4224ecfc47ad0e3ddac1c0d2b4342205d5ab15a0 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 20:14:13 -0700 Subject: [PATCH 11/65] =?UTF-8?q?feat(macro):=20drill-down=20editor=20?= =?UTF-8?q?=E2=80=94=20double-click=20a=20macro=20to=20view=20its=20body?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 5 first slice: double-clicking a Macro node now navigates to a new route that loads the macro's body into the same workflow editor canvas. - Route: \`/dashboard/user/workflow/:id/macro/:macroId\` mounts the existing WorkspaceComponent. The parent wid (\`id\`) is kept in the URL so future breadcrumb / back-navigation work has it. - WorkspaceComponent.registerLoadOperatorMetadata picks up \`macroId\` from the route and runs a new \`loadMacroWithId\` branch instead of the normal workflow load. Auto-persist is disabled via setWorkflowPersistFlag(false) so canvas edits don't accidentally hit \`/workflow/persist\` — saving back to the macro is the next slice. - MacroService.macroDetailToWorkflow converts the persisted MacroBody into a Workflow shape reloadWorkflow can consume: normalizes inner-op / marker port shapes (PortDescription vs PortIdentity), maps MacroLink port-ordinals to string portIDs, and auto-lays-out operators with MacroInput on the left, MacroOutput on the right, regular inner ops in the middle. - workflow-editor double-click handler now detects \`operatorType === "Macro"\` and routes to the drill-down URL instead of opening the result panel. Read-only-ish in v1 — the editor will let the user move things around but the changes don't persist. PUT/POST /macro/{wid}/update + the save flow is the next commit. --- frontend/src/app/app-routing.module.ts | 7 + .../workflow-editor.component.ts | 12 ++ .../component/workspace.component.ts | 49 +++++- .../workspace/service/macro/macro.service.ts | 160 ++++++++++++++++-- 4 files changed, 217 insertions(+), 11 deletions(-) diff --git a/frontend/src/app/app-routing.module.ts b/frontend/src/app/app-routing.module.ts index 179caf5c088..4a495f3b88c 100644 --- a/frontend/src/app/app-routing.module.ts +++ b/frontend/src/app/app-routing.module.ts @@ -119,6 +119,13 @@ routes.push({ path: "workflow/:id", component: WorkspaceComponent, }, + { + // Drill-down editor for a macro's body. `id` carries the parent + // workflow's wid so we can render breadcrumbs / route the user back; + // `macroId` is the actual definition being edited. + path: "workflow/:id/macro/:macroId", + component: WorkspaceComponent, + }, { path: "dataset", component: UserDatasetComponent, diff --git a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts index 979f131ad3c..468734ccb77 100644 --- a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts @@ -570,6 +570,18 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy if (this.workflowActionService.getTexeraGraph().hasCommentBox(elementID)) { this.openCommentBox(elementID); } else if (this.workflowActionService.getTexeraGraph().hasOperator(elementID)) { + // Macro nodes drill down into their body in a new editor route + // (`/workflow/:id/macro/:macroId`) instead of opening a result + // panel — that panel doesn't apply to a composite operator. + const op = this.workflowActionService.getTexeraGraph().getOperator(elementID); + const macroId = op?.operatorProperties?.["macroId"]; + if (op?.operatorType === "Macro" && macroId) { + const parentWid = this.route.snapshot.params.id ?? ""; + this.router.navigateByUrl( + `/dashboard/user/workflow/${parentWid}/macro/${macroId}` + ); + return; + } this.workflowActionService.openResultPanel(); } } diff --git a/frontend/src/app/workspace/component/workspace.component.ts b/frontend/src/app/workspace/component/workspace.component.ts index 9968c26f647..58cdd18cfd6 100644 --- a/frontend/src/app/workspace/component/workspace.component.ts +++ b/frontend/src/app/workspace/component/workspace.component.ts @@ -52,6 +52,7 @@ import { WorkflowCompilingService } from "../service/compile-workflow/workflow-c import { DASHBOARD_USER_WORKSPACE } from "../../app-routing.constant"; import { GuiConfigService } from "../../common/service/gui-config.service"; import { checkIfWorkflowBroken } from "../../common/util/workflow-check"; +import { MacroService } from "../service/macro/macro.service"; import { NzSpinComponent } from "ng-zorro-antd/spin"; import { ResultPanelComponent } from "./result-panel/result-panel.component"; import { WorkflowEditorComponent } from "./workflow-editor/workflow-editor.component"; @@ -126,7 +127,8 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { private hubService: HubService, private codeEditorService: CodeEditorService, private config: GuiConfigService, - private changeDetectorRef: ChangeDetectorRef + private changeDetectorRef: ChangeDetectorRef, + private macroService: MacroService ) {} ngOnInit() { @@ -274,8 +276,53 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { ); } + loadMacroWithId(macroId: number): void { + this.isLoading = true; + this.workflowActionService.disableWorkflowModification(); + forkJoin({ + operatorMetadata: this.operatorMetadataService.getOperatorMetadata(), + detail: this.macroService.getMacro(macroId), + }) + .pipe(untilDestroyed(this)) + .subscribe( + ({ detail }) => { + const macroWorkflow = this.macroService.macroDetailToWorkflow(detail); + // Reuse the same shared-model setup as the parent workflow editor so + // the YJS room / undo-redo stack are isolated to this macro. + this.workflowActionService.setNewSharedModel(macroId, this.userService.getCurrentUser()); + this.workflowActionService.reloadWorkflow(macroWorkflow); + // Allow visual editing on the canvas, but persistWorkflow is already + // disabled above so changes won't accidentally land on /workflow/persist. + this.workflowActionService.enableWorkflowModification(); + this.undoRedoService.clearUndoStack(); + this.undoRedoService.clearRedoStack(); + this.setLoadingState(false); + this.triggerCenter(); + }, + () => { + this.workflowActionService.resetAsNewWorkflow(); + this.workflowActionService.enableWorkflowModification(); + this.undoRedoService.clearUndoStack(); + this.undoRedoService.clearRedoStack(); + this.message.error("Couldn't load macro definition."); + this.setLoadingState(false); + } + ); + } + registerLoadOperatorMetadata() { + const macroId = this.route.snapshot.params.macroId; const wid = this.route.snapshot.params.id; + // /workflow/:id/macro/:macroId — load the macro body into the same canvas + // as a "drill-down" view. Read-only in v1 (no auto-persist back to the + // macro endpoint yet; that's the next slice). + if (macroId) { + this.isLoading = true; + this.workflowActionService.disableWorkflowModification(); + this.workflowPersistService.setWorkflowPersistFlag(false); + this.loadMacroWithId(Number(macroId)); + return; + } // load workflow with wid if presented in the URL if (wid) { // show loading spinner right away while waiting for workflow to load diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index 9afc44b13aa..1e2ef4d65b9 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -21,8 +21,16 @@ import { HttpClient } from "@angular/common/http"; import { Injectable } from "@angular/core"; import { Observable } from "rxjs"; import { AppSettings } from "../../../common/app-setting"; +import { ExecutionMode, Workflow, WorkflowContent } from "../../../common/type/workflow"; +import { + OperatorLink, + OperatorPredicate, + PortDescription, + Point, +} from "../../types/workflow-common.interface"; import { PortIdentity } from "../../types/execute-workflow.interface"; import { WorkflowActionService } from "../workflow-graph/model/workflow-action.service"; +import { v4 as uuid } from "uuid"; export const MACRO_BASE_URL = "macro"; export const MACRO_CREATE_URL = MACRO_BASE_URL + "/create"; @@ -186,7 +194,7 @@ export class MacroService { const inputMarkers = incomingKeys.map((key, idx) => { const [innerOpId, innerPortID] = key.split("|"); return { - markerOpId: `MacroInput-operator-${this.uuid()}`, + markerOpId: `MacroInput-operator-${uuid()}`, portIndex: idx, innerOpId, innerPortID, @@ -198,7 +206,7 @@ export class MacroService { const outputMarkers = outgoingKeys.map((key, idx) => { const [innerOpId, innerPortID] = key.split("|"); return { - markerOpId: `MacroOutput-operator-${this.uuid()}`, + markerOpId: `MacroOutput-operator-${uuid()}`, portIndex: idx, innerOpId, innerPortID, @@ -293,14 +301,146 @@ export class MacroService { }; } - private uuid(): string { - // Lightweight RFC4122-ish ID; the actions service uses crypto.randomUUID - // elsewhere but we don't have it imported here and don't need strict - // collision resistance for marker ops within a single macro body. - return "xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g, c => { - const r = (Math.random() * 16) | 0; - const v = c === "x" ? r : (r & 0x3) | 0x8; - return v.toString(16); + /** + * Adapt a backend `MacroDetail` (whose `content` is a serialized `MacroBody`) + * into a `Workflow`-shaped object the existing `reloadWorkflow` flow can + * consume. Used by the drill-down editor route. + * + * v1 caveats: + * - operator positions are auto-laid-out (MacroInput on the left, regular + * inner ops in the middle, MacroOutput on the right) because the body + * doesn't carry positions yet. + * - inner ops that came from the canvas already have `PortDescription` + * ports; marker ops were authored with backend `PortIdentity` shape and + * are normalized here. + */ + public macroDetailToWorkflow(detail: MacroDetail): Workflow { + const body = JSON.parse(detail.content) as MacroBody; + + const operators = body.operators.map(raw => this.normalizeBodyOperator(raw)); + const operatorPositions = this.autoLayoutMacroBody(operators); + const links = body.links + .map(ml => this.macroLinkToOperatorLink(ml, operators)) + .filter((l): l is OperatorLink => l !== null); + + const content: WorkflowContent = { + operators, + operatorPositions, + links, + commentBoxes: [], + settings: { dataTransferBatchSize: 400, executionMode: ExecutionMode.PIPELINED }, + }; + + return { + wid: detail.wid, + name: detail.name, + description: detail.description, + creationTime: new Date(detail.creationTime).getTime(), + lastModifiedTime: new Date(detail.lastModifiedTime).getTime(), + isPublished: detail.isPublic ? 1 : 0, + readonly: detail.readonly, + content, + }; + } + + private normalizeBodyOperator(raw: unknown): OperatorPredicate { + const r = raw as Record; + const { + operatorID, + operatorType, + operatorVersion, + inputPorts, + outputPorts, + ...rest + } = r as { + operatorID: string; + operatorType: string; + operatorVersion?: string; + inputPorts?: unknown[]; + outputPorts?: unknown[]; + } & Record; + + return { + operatorID, + operatorType, + operatorVersion: operatorVersion ?? "", + operatorProperties: rest, + inputPorts: this.normalizePortList(inputPorts ?? [], "input"), + outputPorts: this.normalizePortList(outputPorts ?? [], "output"), + showAdvanced: false, + isDisabled: false, + customDisplayName: typeof rest["displayName"] === "string" ? (rest["displayName"] as string) : undefined, + dynamicInputPorts: false, + dynamicOutputPorts: false, + }; + } + + private normalizePortList(ports: unknown[], dir: "input" | "output"): PortDescription[] { + return ports.map((raw, idx) => { + const p = raw as Record; + // Already PortDescription-shaped (came from the canvas serialization). + if (typeof p?.["portID"] === "string") { + return p as unknown as PortDescription; + } + // Backend PortIdentity shape ({id: {id, internal}, displayName, ...}) — + // synthesize a portID using the ordinal. + const displayName = typeof p?.["displayName"] === "string" ? (p["displayName"] as string) : ""; + const base: PortDescription = { + portID: `${dir}-${idx}`, + displayName, + disallowMultiInputs: false, + isDynamicPort: false, + }; + return dir === "input" ? { ...base, dependencies: [] } : base; + }); + } + + private macroLinkToOperatorLink( + ml: MacroBodyLink, + operators: OperatorPredicate[] + ): OperatorLink | null { + const fromOp = operators.find(o => o.operatorID === ml.fromOpId); + const toOp = operators.find(o => o.operatorID === ml.toOpId); + if (!fromOp || !toOp) return null; + const fromPortID = fromOp.outputPorts[ml.fromPortId.id]?.portID; + const toPortID = toOp.inputPorts[ml.toPortId.id]?.portID; + if (!fromPortID || !toPortID) return null; + return { + linkID: `macro-link-${uuid()}`, + source: { operatorID: ml.fromOpId, portID: fromPortID }, + target: { operatorID: ml.toOpId, portID: toPortID }, + }; + } + + /** + * Place MacroInput markers on the left, MacroOutput markers on the right, + * and everything else in a middle column. Sufficient for visual + * inspection; a proper layout pass is a follow-up. + */ + private autoLayoutMacroBody(operators: OperatorPredicate[]): { [id: string]: Point } { + const xLeft = 100; + const xMiddle = 450; + const xRight = 800; + const ySpacing = 120; + const ySeen = { left: 0, middle: 0, right: 0 }; + const positions: { [id: string]: Point } = {}; + operators.forEach(op => { + let column: keyof typeof ySeen; + let x: number; + if (op.operatorType === "MacroInput") { + column = "left"; + x = xLeft; + } else if (op.operatorType === "MacroOutput") { + column = "right"; + x = xRight; + } else { + column = "middle"; + x = xMiddle; + } + const y = 100 + ySeen[column] * ySpacing; + ySeen[column] += 1; + positions[op.operatorID] = { x, y }; }); + return positions; } } From 8e0af424a7c3e8362a62ee3182d03b075479b985 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 20:18:26 -0700 Subject: [PATCH 12/65] fix(macro): strip orphan \`nullable: true\` from operator schemas on load MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The backend's reflective JSON-schema generator emits \`{nullable: true}\` for \`Option[...]\` fields whose inner type it can't enumerate (\`Option[MacroBody]\`, \`Option[MacroFusion]\` on \`MacroOpDesc\`). Ajv strict-mode refuses to compile schemas with \`nullable\` and no \`type\`, which threw from everywhere the schema gets compiled — validation-workflow, property-editor, dynamic-schema, shared-model-change-handler — making the drill-down editor unusable. Sanitize once at the source (OperatorMetadataService): walk every operator's \`jsonSchema\` and delete \`nullable\` when there's no sibling \`type\`. All downstream Ajv compilations now see well-formed schemas. The proper backend fix is still tracked in project memory \`project_macroopdesc_schema_ajv_bug.md\`; this is defense-in-depth that also hardens us against any future LogicalOp picking up the same shape. --- .../operator-metadata.service.ts | 54 ++++++++++++++++++- 1 file changed, 52 insertions(+), 2 deletions(-) diff --git a/frontend/src/app/workspace/service/operator-metadata/operator-metadata.service.ts b/frontend/src/app/workspace/service/operator-metadata/operator-metadata.service.ts index 484a3c09818..c6b6ea791c1 100644 --- a/frontend/src/app/workspace/service/operator-metadata/operator-metadata.service.ts +++ b/frontend/src/app/workspace/service/operator-metadata/operator-metadata.service.ts @@ -22,7 +22,7 @@ import { Injectable } from "@angular/core"; import { Observable } from "rxjs"; import { AppSettings } from "../../../common/app-setting"; import { OperatorMetadata, OperatorSchema } from "../../types/operator-schema.interface"; -import { shareReplay } from "rxjs/operators"; +import { map, shareReplay } from "rxjs/operators"; export const OPERATOR_METADATA_ENDPOINT = "resources/operator-metadata"; @@ -54,7 +54,57 @@ export class OperatorMetadataService { private operatorMetadataObservable = this.httpClient .get(`${AppSettings.getApiEndpoint()}/${OPERATOR_METADATA_ENDPOINT}`) - .pipe(shareReplay(1)); + .pipe( + map(metadata => OperatorMetadataService.sanitizeMetadata(metadata)), + shareReplay(1) + ); + + /** + * The backend's reflective JSON-schema generator emits `{"nullable": true}` + * for `Option[...]` fields whose inner type it can't enumerate + * (e.g. `Option[MacroBody]` on `MacroOpDesc`). Ajv strict-mode rejects + * `nullable` without a sibling `type`, which throws everywhere the schema + * gets compiled — validation, property editor, dynamic schema, the YJS + * shared-model handler, etc. Strip those orphan `nullable` flags as the + * metadata comes off the wire so downstream code never sees them. + * + * The proper long-term fix is to teach the generator to emit a real type + * (see project memory `project_macroopdesc_schema_ajv_bug.md`); this + * sanitizer is defense-in-depth. + */ + private static sanitizeMetadata(metadata: OperatorMetadata): OperatorMetadata { + metadata.operators.forEach(op => OperatorMetadataService.sanitizeSchemaNode(op.jsonSchema)); + return metadata; + } + + private static sanitizeSchemaNode(node: unknown): void { + if (node === null || typeof node !== "object") return; + if (Array.isArray(node)) { + node.forEach(child => OperatorMetadataService.sanitizeSchemaNode(child)); + return; + } + const obj = node as Record; + if (obj["nullable"] === true && obj["type"] === undefined) { + delete obj["nullable"]; + } + for (const key of ["properties", "definitions", "patternProperties"]) { + const dict = obj[key]; + if (dict && typeof dict === "object" && !Array.isArray(dict)) { + for (const childKey of Object.keys(dict as Record)) { + OperatorMetadataService.sanitizeSchemaNode((dict as Record)[childKey]); + } + } + } + for (const key of ["items", "additionalProperties", "not"]) { + if (obj[key]) OperatorMetadataService.sanitizeSchemaNode(obj[key]); + } + for (const key of ["oneOf", "anyOf", "allOf"]) { + const arr = obj[key]; + if (Array.isArray(arr)) { + arr.forEach(child => OperatorMetadataService.sanitizeSchemaNode(child)); + } + } + } constructor(private httpClient: HttpClient) { this.getOperatorMetadata().subscribe(data => { From 84dfae4415848c95118dc08fb669c15b90a16385 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 20:26:19 -0700 Subject: [PATCH 13/65] fix(macro): reset canvas before drill-down load + stub macro icons MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two issues blocking the macro body from rendering: 1. WorkspaceComponent is reused across route changes (no ngOnDestroy fires going /workflow/:id → /workflow/:id/macro/:macroId), so the parent workflow's operators+links stayed on the JointJS paper. reloadWorkflow then hit \`failed to add link. cause: duplicate link found with same source and target\` in shared-model-change-handler when the macro body's marker links collided with parent leftovers. Fix: call resetAsNewWorkflow() before setNewSharedModel. 2. Macro / MacroInput / MacroOutput had no icon files, so JointJS rendered blank/broken-image boxes (operators technically present but invisible). Stub with copies of PythonUDFV2.png so they at least render; proper icons are a polish task. --- .../workspace/component/workspace.component.ts | 7 +++++++ frontend/src/assets/operator_images/Macro.png | Bin 0 -> 21524 bytes .../src/assets/operator_images/MacroInput.png | Bin 0 -> 21524 bytes .../src/assets/operator_images/MacroOutput.png | Bin 0 -> 21524 bytes 4 files changed, 7 insertions(+) create mode 100644 frontend/src/assets/operator_images/Macro.png create mode 100644 frontend/src/assets/operator_images/MacroInput.png create mode 100644 frontend/src/assets/operator_images/MacroOutput.png diff --git a/frontend/src/app/workspace/component/workspace.component.ts b/frontend/src/app/workspace/component/workspace.component.ts index 58cdd18cfd6..898b1f8050f 100644 --- a/frontend/src/app/workspace/component/workspace.component.ts +++ b/frontend/src/app/workspace/component/workspace.component.ts @@ -287,6 +287,13 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { .subscribe( ({ detail }) => { const macroWorkflow = this.macroService.macroDetailToWorkflow(detail); + // Clear the canvas before reloading. Angular reuses WorkspaceComponent + // across route changes (no ngOnDestroy fires when going from + // `/workflow/:id` to `/workflow/:id/macro/:macroId`), so the parent + // workflow's operators+links are still on the JointJS paper — + // reloadWorkflow would otherwise hit "duplicate link" rejections in + // shared-model-change-handler. + this.workflowActionService.resetAsNewWorkflow(); // Reuse the same shared-model setup as the parent workflow editor so // the YJS room / undo-redo stack are isolated to this macro. this.workflowActionService.setNewSharedModel(macroId, this.userService.getCurrentUser()); diff --git a/frontend/src/assets/operator_images/Macro.png b/frontend/src/assets/operator_images/Macro.png new file mode 100644 index 0000000000000000000000000000000000000000..209abf48cbf24df5412a8743a400586756965b89 GIT binary patch literal 21524 zcmd>lg1rflTk z2q>t4q{mNs7h4@D5@==Q>WMYWcqavI8u&V#>vn&-7DS&V70b@mJ0%O5S=)593M<;k zB~hY-Z8+bE0_2YFw=Mz#KObN5+B0hWs%N&L|C7Uc>dBpoRgBoXyyoCDm?=%FZma*w z$X%))S@+h^9zi~K$-p?J=OGK=wkNKq0_ z^)$gjY-sh&P%o4B$%k+oA*Q@#J22n`* zH8rg?d|E`iQ)0q;{;NqdVr~P}9m2y2ln; zA&EHoX+g$N z_cn6@U8#p&b&t26<7<0Ot6mvu+ejTCYRcy@7X^6>6Xs+FIko`%R@ViYivUSog9Ygl zKcMl)skL77l&A_PzIl-dCYuSxKSe(l@oQp*B6~==^1|A&1o4g-rg}e?K_3y-PV#i~ zgQW)=LWqS6FdhkCTtzS>YR{7ulM(;%!cykHqgfN#y7AqSMjs`YyIK=GBpX<;_nZBQ z3JQ>zb8fB7mbH@fR*E9ZfJxh1MJ=lh33&+^cnO%oh_DD>LxTQRn!LZ!Jrg6p z07Z+)H#uvO6O7zw28I|moUlWEf2rRad~utj4nahaB#}ztD`gTgb=!Ni_`VF=vt>VG z1UtGNc87}$^6E$_v-Rvl7SAn*9)y>GtU^smxgOPygE>E1Ci0xtTjd6|cr4r$ zu(nQOh`t|>6M!9Rr%tYC=DbCgs_a98;8L44#%v$m*&j8ROE9JfUcMwZpc}UEx|7`! z1!<}@Oo@5cA2BE%oXwz}BWS~J!wB_50$bgn-m9{_UDBj9^0)xMf04POqJ;NWY5;Mu z&Q(B#Q0Qq=xWxW)k*-RFir>V$`*5vrY+rYCe(%kdO4h6?bLj*oV!&uZIOALG1DE$~ zPeR2n16&`tRA*)RPl_pIOE{lm>{R8M9+w>NweD{_-9Y6r)JsSmq^?wkp0kqOS{W7b9;ML#J%G3zNO+13bBkS zc*`C&dBP_hqgj)0B8u!ejTX6W^dlq2xhk)3PHf)&GQFLg?o+fS`ttk1#m4}?0Y8zG z$+!7QvXUX^iZghd0ykw z+>jM|fS-_V(a?k6YB=oia%RwCW=;IhnJ~2AXttUPE z(%`$>Npo|?wHejLkJeT}o0WGZ$uci&a7dY-gjOe0Bxz)np-%I|e@#(K?dV$d#Yzdz z%!m4Z`AVT`^6Sk)Kr**{1ibq00RS6XN@_FeoxxJV*+77wXJa?BS#i(1o{Jj}A&BP|%Li5T!^a;huJ9EIuz2bU1w`NLn z(^Tov&b9A7+dO&uHRNp2qw?65hOCU(t;fOL-FvP&zqc_K@4KIMH`uZ__Xn6NDp@=9 zYPQBj;Tq&v69F{oe_n2qA+xT<>tj;mcrb{PgS@y$C%da;Kpubl`2`P|k{vvZpya%^ zNYHM_6%7d`aFn}tLDx}u_2_Z(^L|1bi;unAI?uf2TGu}G&VN{SbI)EXJoz47?D zj7RMii9y?&g);NSui6J_kG;1(se8W=vbI2WXd>md8E5g%^)s`v&@96%J!aaj`x-I* zb7DcK8>93V-Fz|;{1of*bC0|tDQ=Me=dIV9=>=|e5nM93acAqXR|S|%>mPm?eLR~e#pZ&2db zekEMaOYcga_$`X8#Q|67I!^EDGnoWk(7wopkpUNnQ*MRLY;M1VZDsDs{*>%a3l4=jKfNvM$^Z?VsD!N13KnF<48W1)^0Ji5K&L zI+q{Vk@hfED4(0m-JOx1sDCvly*aumZ_&Jr^1|D$cpn-Ac3v~KMGy4Oghppz7BF2* zuj*K?+D|S|ANkukha|iF(SA`5s-W7|4JesL-irf9>}!$&{aq%#I}JZK?Mr9&FAYx4 z)iH~2CimB4rQ*g;m=obJfdUse-zVRgm;Pe8TTP#7iIm*G%4o17`eiF7)br54pho=p z2bul1oaXd^rU*CHz4}#`m8`wa`a{%!WbbM<4yVg(>hWWe&Z+S&Wq*@dR{-a8b|WpJ z6fQ8PQ&R2tXb{6vzw6o+R#`bXcYAxX#key+UU-tKRawgofBf5XTKPYD`HHP&aWT8- z*5zxe@5evZp8cq6JG=QhgJklNZ*3ZA;qy(Wj6mvB0a$PX>#Wi&v=(e_p|rxv^$A1EKLxiGloNZpyJA82#Gvf#K~% zr}lD1g(&W{E%~{g4__M(*Ax}slQe!oK5$J*+J4|UCTOek%!h>9`QLj5)fhHF#XcFn zbvc|}A|L6tAkTKzR5doq3~?OjPQNHSsB3LTL`=b^E%~82V}xIObp1Dj_Ng43FFD6% zZpZ-rex>hCjGOh{!f{37QhQ$-nB!XNNIwOy;s*x)kw{XDvN`sY z?31x7-=BT<@tXx65Ep#i!TL`4;b{yE*-Rk+6*e*Q1tsJ`w_IP8vb3st*Xqw-u~ac@ znY+Xy8fqoth)SW%_MvX?a`Y_1M`~g@W>W6T^}asgXlVkHEKNe--I>CFrt$=zEkvC+ zRC5hK-_*aAL=+U@!rtug`rg?$Ty=hpBT|4L@sD(lT}!}R9Pp<5%1R=>g)ujIQA^D| zndikOOaoW(+c$I<1L7@kju?uQLTU^?=BV6~{CxTX-#+gyRC`n5IX^be(L&MEKP3n^ z#q~dPB*kQ|=-}(;i1v3s48J$Ktsbl5zi%iCI_+LI5qZV*KI}hTkD=dxfxXb!lEE2b zw-g?5<~Gv<2&6$oAiqoe{E@C+?wXC89i z4miaJ6!9mNAfHBYZRp2qa)UMy;xGaQ#P0@5$%V_t%n2zHczi#rz+lu$AWDM4Am9%3 z|Hc;%G)}yh z979)MQ{H=^gLUmobFURh7*?8)K%ac?q6G+R9CWY4_9*8KXNcRLYiI8h+o51=Ena_omp57{J-NZWS&iRqL<7KEMFc zljd=3^Z;*f#B*RqhcP?4wM{CxEEY58ik{RTrUwW|l7r2SrYQMtT5=|QVEQGEtcw3J zM*~&HL`n06a)Cr<*o0(mP(e{TbmN<+PWhf0p_i>WK}S5f9F>H*LAc2l0#TKz1&#Du zAA|s$m8JmWL44Icef^^C3hpe>k5i_F)50r~nnHM6a!Ff#lJ_1XAWDi9UWYqh_OZ_I z!zrcHYG78gvmy{@Gj_l^VW+~AO_0Lt^ylm8ZQ&fe+40bO>HEnEH-z?V;^;kg(r~ul zGXX(G)aNDn6i~SO`eNgo^pfbSq~6azniOn_#!_OUg?gX0gD_XEP!y^ZOl|cJpc#q? zWrl_%ezJOIso(Q+Hs=Zv`oR{E_;H?@T=YK4RBGm9a5rt|810u8hFY$Bh$5GKKTyFY z3Y0GNDRFNt@c9KQ7HUKCMtnB!KF%|KracLuGQAOmV< zXgtriJ+x6`(z#-`yKC?;I$%X_&iNM)WBUgWVw6>J8#V+u;^GSASSRi!^M$)UuQ!?-^%M{AnsY~z(M;9d^2xJ7p}VIYowepGv|Rl^dwl& zzWPTN0!yU^iZeH|j>gK>Rk?eO_^B8ZHu<(4Si+-_yz^L=DOcgwoZ-Ftn7bAyR0jI9 zj_l~2TEf434~QkT!T2PNLvf&S<(ur(|T0p-_z^i!i7ZemECN*w*@P(`+sGq}B0aAeF6+Q64b2+0- z*C@Vmw1cD)OWEvRm|cX&b8?E^9^Jo^!b-}~t^J{uO_n~NRSBSQjiFYjHGFYFvfTGH zH`t+5@4~6Ct|Afs4n-NlA(^U5d@56Sj!y0GczwEWBntR)*)e`on@HM=^GUxJmjY)6 zc9ZyUpF>HMV0i?}Oua#Z3y?hzbQ|;mEXGgS)?j^h*lJ@%a>?b|&oL<`NXPlb*^Ru4cjzoO1r#zf*hIJv{WDWR)u|3Mk z)0@JTsbawDM*HkQ5fkC!_N_1Ifao2s>cLhS1V)cvg@$ggqD{m-KN*+P18?%lk>;>u zuOLBNqF*1=SKlLgJ)RQB-zZq>uyq3Zs!(S?@Nn#1&oCb_T5AxuJ3pchhYiLa~(S*69O}2O9QQpbhl@L${djWd@fBr5gBZ6h;jc5I0CD@jxS|*}Qt+>BB>PYo*xOOh z2k6G>4}oqn`_be)P^gFW04|69XGLn|C+j`GX7}}ANe)Bx1fdg>&v}c3iGdqU&LA9E%|sd zI*g2c0!5Zw{Iw9s+Ppoal*Jqz=Ck*sM=zxpAKP7HsNLK`u~Ol|^|P8x_+7 z{_eBs$%cT}z4+a$;e#zu?w6i>xS|s#H5=EjK*>3A;jH;C>2W)Ycatd&>Vr9tzEJjU zR9N>?1}yHtLwR>T3~vdSkc$C-chY0s5}Pi0IN5rGbY>HZ%5{Sm;*m-|_rnp8?6l*3 z6KO{6419pW?EAb3#YH>zSB?A%ftXrK_|9 zBaxK%ae&9P;FhymIhrHChIGQmh3qtYV5L_yegO&M|ATy@2SI@5>jO553+Ffa-l=hEz=97{6@9zNLR{N15^AV9ktfSM>`>?W8D}ZP1yy=9ZLZF-B&QGK zWBLd>{=Iqy)=&*5H-3OO>gZUVsTydb)+P7}zz6m$B^k8Ugm3}QqsSa;_4jFZSW*yx zUmGiUZ?xw#SvGc5tEjFI9}-6a$WJ^C!Gb_R$qOW$m%?&BB~vny6VVi;<7hHD5Cd?{ zrA7!A8_=n5D02bMdzS>Yx`x@Bz925Z+b0Ta+?BHWW$5`S$7~4-&xwBhZ{f5iX%GkY z23j!8mb?_u!HH-S2EKM;3EDR8-SZ?01gbb}1&1&?FaTbPHx#ZuiXZj%U;xmJC&}1-fqB%uJzw4 z#NHUbI~Kf{A8o%udk$F5m4+69GyK7kH%Ws=fySTQAku-22KJEY$(r-62r(84++4$5 zL&$&>&G0=&#LJiZgvE$2GIy@Iy8BYzwpM+l7k_}&HHB&C@Lp29g0cb5%1evbu+>Z~ zeFn+bDyu@w+-{ziF)>xv)fcTJ2%7WZTTsnv?$SnFPX3@Z1FHA1sg(}jnr)GC?CzN8 zqXkSm)&I~iQI45IDQFsM5Rqr$(n}w^7p+mh&WPl~605BKR-v+-2;*7KAq*jk7m;v} zHElFGrM+I10*pD#4R38>&xJOzN9f<(PnpZ-Z={cy~X*xlLf^g{ShXG9_wX{!E^35Jl-0ZtGI zEOEhM68fX@AsxACJ8|K+E=Io|$t(87+O`+-@@qpehuS#82mAWggj~5nyg-Le`2>$P zYZSAMh?J3A7SmJWJ7p^L(ls!jz6mx5dAJg2O-tZfxule|&UjuU>Zg5vQB&M`>UOkm z)ni~%D(dz8^Il>k?mGw^QH)oZ+|sE_)D?zL zrSWM?fiO26%)uIDBa$mvGX30kwYZ~ldyKz65G;MHLFNjTddMZ}qCyby1UKm3eSRAd zSxY2EX6U-`4oL zSO;#dbv74qk~+kY0I0aponQe|cMROtP#$%<0zEQJ!+fn_MZG;Eu68c2cDs@mHX?$C zlOrStP$f{gYsKrnIUU;hQGQeji~x^{C#iT%)GHbsrM^xT#9u$UaAU;n*P-Fpp(*Vr zPEvy)CN~+J@7>MK0aJ<}OgoKas`3|ZAD-{4C-1OMIH?x49}q(Clo2Os$X$lxz`Ks+ zDMC^jNY2z*~iw$N?t!g5|Rlg?Fm2JbikK87?Qf-h{{4${Z5tbM%5k`Jh-4TXcaWW!psiS0&|4dY1#4N#Nqu_ z;d+c1SaFfr4ObYYcsP{jQ;tM`OAsd`{Xoi7wB&y$PSj-oxVrV@{>?Lt-MKQc2KrX7 z>SM8Bwb!mUC(49R9de8Q~fL&-&uFwe~l?MTI-+9pO-q z9xN=Sn*4mq5*H8{&u+y1WjSL+&*Iu%(j9!3RnhUm+OsaQ&vw8UjAW#De!SixyJkl{ z<5nc*{|tLnbny#p zRh}Y$KQM5>8{nH}5_of+gp+uP6w(q%ewNo5pUp4+_>f{{KCvvmeTKpSEQJ{*pUl*( z{AIy342OX3uC8KyMyw>LrKbKPGPj4k+;BV58vb|qmA8?-JR~unY22si*XThXj2dBN zqvsNHHK;GVKxblx3uur1cRb&RNxXAem~0_zy2L{PBOJ3;FP2}S)vyzCs0(duse#e~ znk$f(@v!hrGa=v*3?hQ}=P4Iw&a?xrV6P*v=!XFX5`Z`3oidrv)R&mk^Tx-DU@mk~LRaiQ#Xro|OYASztVx-48}+LXXndhR0xdhVzM01h)0 zf+}Uk7}aS3^cJq}C6=5=lUt93h-wyVaQ<;y0qoAUq!<778!hQQ8&6Dk&B3{Nb3D7N z1N;LBAl#^ea7oE~^x#&DlOUrA5)WUv@Y)gf>-_kK0C_&w)T1vtQ)Ff60_*qu2_aCF zLba#h-M%S#k!@iv&C(}2COGRi9GqE`k_EC;jB%S4{~*i`6%=02vTZSKt{D%DvCFTC zc2ng%nm#&t#A4e70;%)9Mm;9L*>`V2jhx_`lj4HXAX49+Aq^ARtH`N|_T7)@nq`~w zoc@QpP9vdJU#5hd^}5U~J(?vM;~$GN@WtE%k!A1}Bh_$gO;ZcY5Fxw?M>Gmz#ReH6 z1zZ}Xnz9>Hvx^b=exMJ1@yhzq?{Bki_h-#`hyn9D2x`jjthsM3DW)Y+39*DMwSTwc?Ln)CpY(#{O>R6g*x8K;y+)TJP$ z%|9H~QB$yX)$nyAbvfA>g9Znz5(ilEGE401{?+3}CqBUth)0>ASHJf`9_gp?Amb+C z27^BM?GrrcZi-dqwEWTIAJJ!LfSogI{-XEKk6E{~`n@p)V&H0kJwO`ur3ceAs$Jij z*O;C2Q#2I>Q{Yl9xsZNP1~8{!h`}d7w#DS;pfYPkC`Rx?)z?kCeE!c?3us+V1+=kg9E7S#*yK>gLB3=CjXS1n4A+}!W?uT6`tynO>S0{jDXRVO308) z2mrSazUcnwEn8>dV*rD#YUY>6tlzk|eam3gZvSx;cM0wpipoV?3~r9S6C%MEKFBDr z9JI$)WFJ{l&#~ZFYH0yRMW86uMBcdO7`IRG*{)H*Z}ERrgviT36HQS;HC6^i?*(Oq z8Yaka;UU!{E)ttt!Hd@c6pivdyA%`&I`u-;Z1XKUQ?6vw9T%XR2nGY^g*{*4_`0

    Zq5m( zmO~b2XkDEf@0StBpC5*G0`ZMsH|l?`B8&EBo}cXzEkZx?QY3s@Jv8>YIWRt(aNa%r zPGXg|xAK7osyHU8Y5LAXa zaN3A-5R)c+?GQLT)VeJkizuFPDT|W#ZKn}YnCYne)pj3aZVkA*i6gym*fUb3-DLL9 z!0ok39FE}CIC)g!5c_?j%WQn}2KmHACaZbJ)hh&oYUc9)F2K$b)i>9A(@wJaY`Y9e z#Kp-k74voTlkWLgl%i8)!7gi1GmDanPfQmm2-69`*r-TfIhWvn{D&ww{zF|;exnj+ zuFdGg7yp}lk=BE}LAjMp)*v+g!tp)p`5QQ&0aCmz#ISOoNj4=;{) zyu^^ojdsWgyy=-RN0CnwvJ+;P7g_E-KVK9|sM#_uyUXTZb1zyO-L>^MoIqq#I0PY~ zRj`}UaBI8!#Pa$=F6zqnm+N2mBAOpPz*5!=pa9VJzg*jUQn~NA(6rNM@ILFoP;?S& ze&aK+paKzElEZh>z8-u!pr{mH5K>Ye9C=B4_r?YXJhF)p@M808;Yb_uj4N6 zES%lCaCdZP7AFyI3(GA2d%ug;F?bCfk;USDc)Kw085Ljs)fZ z*WH-LPh(S5|L)5pRouidec0akWkZg|p@X58D9D?Z2K$TUUA5SiMmR z>z_xD$Tox*biiPNs)k3vpWO!Io!hfdLCvd^5Q{_T(Bv&GK!V&RW5fmC@&nBHR{D1^ zMQ?YGw5SvhKF1}+(g&w72!GDGVxI>K6Zl;Z^Xpv4ht9{(ZhW?hq~)t#^srlIMvN~m zR$R%1g6Z2za=0T<>fv|zSG-liZ$hkSSddBb@Md{zi;Tac*HKOutX`m@8%Qja|GL>A z7okI(IXAZe@{HLEC&gWPW2^Ha~IaO=uW0&j({~<`a=| ztd)%s51eqTXq2>lI94T;4Xrh?p5a+x!y^Gsu=<Oir zkZ+Qd28T6E<*}w@x;uvMU>f}+i4|RXu*98Z>=4vfDlh;e09`*Wqt(UF#_9z0X9NK+ z_mmE1h{G-X&A;R2B$s4p{mC8QF3SUY;6oV6I@_eHj1F-eAWGaUDESbUq(LBqP6+@y zz35b-J%E1T3gB(ms=YifsZZcw!x2lz&e>|>Uh&s#S#ZANV^D$(!0URrV=Rg6xDD#Vc13FO5@>k_%lw~#hca~ZeuP%AfFRxP{TKX-`1BsQRy-jQ z#vvJTSdnL*Y!HJsP{Z7ITNqsH6YyJsfY&3wG6pM8kuds_&1#wobsCK2ZF&Zd2QxSX z{DR!SL(Xmh-i#Gjxqj^KET|DHZh13>ru3G=z1+}tWq5uv{yn5qHrC@HzR4e_OD=>nLr4a=d=<|uH6Io1I+!K3yY{|Y|>jXa@N8t-m@At zna2sWpjcm>nXbvXQ20>smi6JjR#iJL_k+0fLmV7b4Od|!?tdj;{2A40|5#a#t%uY& z-tHF#%8D+3L&nhpR*@3h6(RqX)oJJNm?NFo2et3YThy&+)Osqs5lv*VD=(f;wOJ0$h1|fd6XLo(QJY zpVk&GjDa+iEP1aNZR>WCIrlGK2&ViIOwFf(0-Gtbwcp+*>~U&ReG`noTBsGTCx;j8 zFGYgVwf>#|v`xDu9Lk%;`qqA04w%Mb2P>yE8~&1fbE@z9I3;jaKU67@SS;;w58wOHb;?$|Lh0*%`l#Hn3 zmf=63%gkl|6c-F;{ckdZzUfjkEMLpX>&_C%#_lEu0!R z9$9mlEe>ppy%3x|o@Am^L~nBF20!`*))R3cpqcL00+pw7E7%cS3epQf@Z)n=VY>8g7}O)e>6y2UN_1Vx?VSgH)Dk zX1n)>Pe(0U3Cl16k}EOhJhvr+WLH6fAj(|-&k6SQFD7xmEy7C3!ha~?-XYsk;-o1k ze>HKK^+a3ElTPgxyxK7#d6y#tcXjSF_NlW!gk1TeI4mbxUw4jHF6i~ z|A8!Pz_2G-nGe$q0fBnU;hgFwfC}V^W~38*+jYu{QeL#Wv(+)b?ZokNSZk1rByd1b>>i2hPdu`q~5K~ z->O~{5FiC}jY~asb>18!9Lb>IS3Ak4GPr+BpA;j5Kl^JN%Qdzu zd`6pYTxLEmh}0xwkt$XZj#dTBXa95Js1#2Yx5qYxd9HSWzvW7>2`H$W#9jEE(GssM zTQ+**lXD@STY1W-zrE-*Zf7j$G+EIfkm&^JaqhAYtQ$7~0cDL>544$h# z;91B{?KCVL4i?tGl&wfDm1Ui&$`dj%cND%{qPU~@mHN##d12^N14b1m!7$J6#i_%i z$5eRHBJVBDF`SwUmvME*Xh|QPGl83orN6C4R$!%DcrV~C0~gMy9lk!yA6M{hpE2{K zGd5ab$ZiW0aKf>&`B7O=C@%HX%~6%#g>1`f6nPH!pH$@t54)Rkhk7e#vsD($2-zyI z)TMSZ9klvo8Y$wE5$bU=+!D;s3oe6VbwDQjDr}oo;kXR%kuc^12)MeNc2Rn?{@=b@ zW&!C6qc`4{k2pL#de?{d~U%L?WKS-cb3hU+H+e)=n z(sf9-N*!Lt6#j<$4EOMji<$Wj2f7PqZ2yy-5TOEp%oiC?Q*M6ksInr5p1t!s;v&I* z-ej$>AbPh(+2C@BcY%(qJr>P~2y~wkXmxVdYAtX4jw*?Q2A3VamF6&9erw1t8e~Ke z&3XRr)l1z$#l0wzMe3;CX=iKbOzywy5!7vpB7{|3a27P%d^Jy|!txPaP%n9Qn<^9h zSFG*i)pVlL1!o-Kt&c^=-c?!8N zl-*~+%JWe-y~5y$39qYQ^>_w^?J>#xiZG!`^!5H zl`Xq%G5aUSfA1cw>ZyTuP(Eak6cKp)Sn7S~8auijSXX@0=0&9Nt&F2BGVI!Y;4!=X zW(6nDq4{Cmx1b;AO;nY>j;?|!dzJTFAwz`04=xN1#J%X0Ruf*x444_8p5%WS{2FP7 zslHw7AS-`<@2$;64#zlsMWef=i}UM>q2nCM-JgR2^BdQ!ztLspXQ;|j$LsaUZPNvM zl%Y>(-i;%|iU=U#mJ>q>d?Ttb`))N>^vC=`umReOK}W`+(Rk02$k}DwJek^msnQ%?+-zLvH7dyKVz2&=BBniEyyfCo(JQGs^o!Xs+4yM#5!xxpal}Ib zzdPw?V*EIMU zX)WD}zl>D+*^s)-K6ft1d+@R((jO=Th`;*<_eD%~mKZmmUiW^>Z27y_Q$JMdrY#TE z{d-TQkqX69=4qTz<}G7zi-QBVw zbd7+wVj~{zvLaK6M#j2kAqSZl(=aIfr$+<#N9w+8LD*Hay2dlk0dU#@)jEtkzKfET>#!~1f~ z{nAU?3h*KHY-pCsytvk*6V7~d`c3)~X8xPQ^;I~FWy##fbb-*MJ*a?Y8Dq$gGqy`mT>d&~le}C+#%P&gRgn!ga7_NUkTHQhoxKC@x9{;E-mL-R!kqQ*2oo z`K+N^sbIZ>*yvl17gIZHgSe~z#@OX7F!FbxR2_hhy!&pPU}^(QT3DFXSNOFJ@0&G$ zKgN=ms#5yVntG9g6_j^IHYpC#hWrHF^-rC+;2(BAg^It(VHuM7wCmo!iwn%rI0)n` zfHe;H#cg84sc;{BW;u3RXA@mXjC*w7`?9mfR^y3=f*oTwOi1G?J_G<1#Bk0xUqgfZ zyAv-jyE~b-8|p@WyaRqX9ZeOx z-U%CI$wOHBBji9}&DJLxSPZAio%ekVi3%(~90zc3$%-7BZ-p2D#3?}=0ao7fvc&AN z=t0;3X+~i8&xH4S8Tvsp#nz*D2DE>wCcl~95@ipk1b`(GHmq2dy2TDo&4wOvNs>pK zYi0{hWqvQLJNnITExyTTu+F^4=o0`~5AhCYfGj-GzW3(N*atqqnSfPBT#p#RkWiBA zut*QTRivJz?mKBiS)}j{`}u$ERufx-^ZZpPr}Jc1#Homgl)n*! za&uBbrH;S+*MFPUyH&=lKX6p4#sgk|ISVFU@63ujF>wYkkOQK%bgOHbLL6XW|Bd)G z-x6-t))gmwmeu*yS1i)glI}kD&?K{I-lh7?An=muM8hkYuUN^h(h~Yvw+EeDpg@6t zqxkvR7MBu|Rq_+W2n!k9Y3Im$z%0eDT4-R;SlTX14gpk4`dQ7W3Hbov8R7LN z{gQ%6gGo6=34RYy#5)k$FiCv`Toa)hsP)$mBZBdlriG=COR*Go#zTf7?j;&&Y5D19 z(ajZy{6O&QR)&XnKnmm;*?PTLY!xdPl=tHPfjpVCEr(z+kT^CLrB4tt5XlI$s5br}s_3T;oHmQ)|rX}w{WADb2d4G*^`!b7cw z0rgnMI-5exn{v!i;$T?=ExAj0dZYNP@X8n0z|#mGjP;!$eD=d%KcEs{v@LH}eydGN zVXx!0|7)Awo#4RYLkSskm@63IxX8al?EH=jSE|ENS7_^ryFaBz?5H>!UW?E9QiYVE zyQ#U9wdKJpr3Ar=kIlIO)F?Hi(`L^Vq$!03{^zMa77Bv(@WeMQ>#r4)PO>-~3``0k znyhoS30_I2hbRy7CVn)sWQDj=IMAx;!9ot>0L#7Kd<-%|0%l7O7n_)`u$~&+$bTE$ zitEVw_?e9qILO5lf)FhSxzGDTVh^`aH6DWLeg_T!_X?X2!B$ zjtR!N>wRW#&h#G3Aav1#)l_Th^bEy2CRErtZ%0iCWWDTRhU*n!MXG_D-V( zjFr~Z`<6gZfVkS*v-@NrtfR#m?$s1Dw)e3vr_x=7p(lq`a09@I)XUw&;n+VP+OIOU z(v9ORRwoi@uj4n{503j!9OCR7Q{1gfl36yT22WMJd7^+?&Z7;sDweX^Gn8-G8m4Kj z$phK%@+-(tzC@Y?m)g&#t8t9Lx2e?!AJnfDd9Y@d%QdumYY}Ih%ea17(pe2esBn4i z@w5d7veTl_R-Mb*_}>sJMOzcLCy5`l%o4A@hlLb>L?M{p$K8|sP?NF&% zu>J3iOhauYEPI=4^5^f)s(muYRk4&e*_<95K}pBzc92*>8~m_TgtlUUp4UAV^cG*! z!l|~LHEGo8U34(VP5n&L3*bycLPm$L*B4y1Up%%w5KQ7<{`a9Z+zQ0D;>b58d*n zf*e(4sj)WLvc=Jpg@FcT4#fS_J&lnVXT{e@9|cM%aV#FNdF)W~Oi8=GK|gT{C+ffe zMmlyAiXlw-k&$dyHE4j5)w>bQL=iJR^8W_09$;&4G-G_moLG3(A;yQk)v^9RxCJ&a zcGJeOFIDGv_GJeuc0s@KLBc}==o0eE4ww)-;Re#c;o13fls_RXyJPaTG&P1+*?SDi z4tFS~#tFhFnj?723=>Y$^+OKhfakuXl3=o}5=8qX&3bB-Z%Ru~9P`z}mGXV#`a_3S zrVUe_+f#lZRPZ^m4qb`$%5&FPE@c*$&<{P4lqtJf)fYKR>Dsr@y5R_~xOp;KS?F4J z>#ZhW)RjwunbAw-H4a)dis;hn&jrD`J1agwu;^jS9F!Z|m7zkK8zZtds-)O9?8W?A zO48ZL5wx`&plfKma!8|ajNH_t0Q+sH2Fh86qn+iD(A?OnZ$65dKfkNhyYH%3!LhzW z*%SfZn>Vq{XzGmIzRx6e7OhemEIy)~nNURfiN-3-Hk(Q+=#g8HaOln8OQfJ=fiU71B|$fvSf$R!x+E}Di|PI=!y|4z>6NqIxVrv zjPPwNIF31FC`W{6cIXz_a(Xw{$=+?q-X48B>?Dz;Zst<7RHdp(sCfF;#Y#D>`GG~06*{Dyf zp-sTTHzCgCcC0FN;Qnf&yW^#+iZHuJHO`17jlJ8EFKz@12d;!71xZ=;<4bleE}t}k zJ8CNI>u++y*CK!;Pc~B zx%+{f+ugL{Ln&};qFxt2287E-eO?L~0Y`ESLR-a!lIfO`G~Owg+QaF*K7YF#KA={q%bk zBZ-@EPr6&%D8tMgSg~u1w%E(8gtJ?zqd#_+8C-;kwpsp;z60UDPBZqv1{Dbw#^Y0v>~u-0!HbszeZ_erV`|C^?UElM%A711&3)ChRd+pX>a z8297IromSZUAe82llScxzihlUUE8KfPo_MW?tyYS=2|98wdGIhhotaWh0R8zW_Vp@ zVGoaRY!gzzJRwuI$*eI#48uZAFwjX>&?_7lYCbK9=;(Q=Tcq9>AZ0i4rNMU}ltLQt zuDkA!B2P?mh32)ae#@uLK*n&vzJjaG{`*aY)_eLdbh_Mm^|5=rNl8YA5sln%EVZG9e<3-bKb$soasuiW=lsH z*-%!Xij6UA_$(I zkNVcfYk+5Gq}nMsOU#1OCZ(RkRSJR-GV`^%77KJ%1ZEJMi<#b# z7?_-{db4FN?!|-W3}v@1)i7YcTrvE?lc}Z`H5he-#oR;^cm0;SVAV1pW#emR|Vu1IjXmaWPow^s{6cQi30l92Pe-4Dlepyq76R+Drttm0S9~u2n zenss#yQk7YrZEiF*!J#{Y zC4QD5eRN1;V3mEFJS?OXI%q2~)Qbi2e0u5EQqyVJuO0A{`OY;V`J^Y{jRy=E02ginLVX#XxNB(fMWOQ46mnS`CD(KNB!}? z2dlWUfzF`Z_@i_9)~cl<3D3EFVw<6~y|w3I!8ve5or3{eETP<_XRl4EX3(s&wAFBt zt%Y#TUL7S*YK0a(6~wv;%K*2=dA^PJ18gW`jDBQz%hZ(f?< z*+7&hMjKo@wBg9lj?ui+KzbS8l(6F-D|+zuBU$d^Ks;B3(Bns58AOx`&-;rkq{fl) z{iS^2nHqSs#4krBd;!8w{M8!v1dTjH21ew3QYgeA{l54;*c=pG=i>35Tmko z@}_TOerRiAV+=`7w=M9N)Ms(!pR%7&5?;fGLTyVh65pG3A}x9-hPpQ;V^yFEgxd zJu;&|q`;xo>6IMki}T!q&GO(|N#3tXX-45VkQgSBg#o?{G9wgRsSdC1Ca`H>;>j3& zgEAzsh2-AL%OHiL0y&>21MQ_TzvR~xVA8#t8*%oOD9r9fu(|+%!&uduYckXiX85!c z@oKaK)x>vp$X`=@zC5^?KCj=hUcgQ%rN)210OToGrM9Z^mbW2!UDD$j_RFe39;XNU zQ^WP1tWf-4z#fZqpbkcB*_@D9K?VG4?4n#ca=DTe-tpOZ*LaZU!jkAz%dhZSW2i6^ zuDZFo>EE&4JY$aaE!pn@y2=PPE<2>P_>w_vAn5f2v$cEL2dMV?O5=u!`u!%hznj^$ zUMAO(KLHYs(Tg8bT|unsNbo@X1PbW*exbH7`Ii$AYY(UZx55tR%yEquNK5NTb#%VT z0Mcad|Na*^hrlX9hl%JRjTB?-GYf)gkxz`xgR^rbf{WY-{Gp$DuO=t30Q{BW0@KZC z&NO417^f&Vp?J}G1EepYd}EDl<-90kdV!SV60flQiYE9Z^SMIm$(-`}T0*_Hd!P{> zZDnNcG$i@i#F!Rvuw+w0@uQ!_5~ZNR(!dB=0R>Px0>#wEoK)XY%>WbS0Qs)~-W$~C zQP(Ju;U7_I7k=0Q0I)4;|B0wGr=rd}ohF@yAHWXl9X6KM4wgF{|Jh;dWNYJOz0<lg1rflTk z2q>t4q{mNs7h4@D5@==Q>WMYWcqavI8u&V#>vn&-7DS&V70b@mJ0%O5S=)593M<;k zB~hY-Z8+bE0_2YFw=Mz#KObN5+B0hWs%N&L|C7Uc>dBpoRgBoXyyoCDm?=%FZma*w z$X%))S@+h^9zi~K$-p?J=OGK=wkNKq0_ z^)$gjY-sh&P%o4B$%k+oA*Q@#J22n`* zH8rg?d|E`iQ)0q;{;NqdVr~P}9m2y2ln; zA&EHoX+g$N z_cn6@U8#p&b&t26<7<0Ot6mvu+ejTCYRcy@7X^6>6Xs+FIko`%R@ViYivUSog9Ygl zKcMl)skL77l&A_PzIl-dCYuSxKSe(l@oQp*B6~==^1|A&1o4g-rg}e?K_3y-PV#i~ zgQW)=LWqS6FdhkCTtzS>YR{7ulM(;%!cykHqgfN#y7AqSMjs`YyIK=GBpX<;_nZBQ z3JQ>zb8fB7mbH@fR*E9ZfJxh1MJ=lh33&+^cnO%oh_DD>LxTQRn!LZ!Jrg6p z07Z+)H#uvO6O7zw28I|moUlWEf2rRad~utj4nahaB#}ztD`gTgb=!Ni_`VF=vt>VG z1UtGNc87}$^6E$_v-Rvl7SAn*9)y>GtU^smxgOPygE>E1Ci0xtTjd6|cr4r$ zu(nQOh`t|>6M!9Rr%tYC=DbCgs_a98;8L44#%v$m*&j8ROE9JfUcMwZpc}UEx|7`! z1!<}@Oo@5cA2BE%oXwz}BWS~J!wB_50$bgn-m9{_UDBj9^0)xMf04POqJ;NWY5;Mu z&Q(B#Q0Qq=xWxW)k*-RFir>V$`*5vrY+rYCe(%kdO4h6?bLj*oV!&uZIOALG1DE$~ zPeR2n16&`tRA*)RPl_pIOE{lm>{R8M9+w>NweD{_-9Y6r)JsSmq^?wkp0kqOS{W7b9;ML#J%G3zNO+13bBkS zc*`C&dBP_hqgj)0B8u!ejTX6W^dlq2xhk)3PHf)&GQFLg?o+fS`ttk1#m4}?0Y8zG z$+!7QvXUX^iZghd0ykw z+>jM|fS-_V(a?k6YB=oia%RwCW=;IhnJ~2AXttUPE z(%`$>Npo|?wHejLkJeT}o0WGZ$uci&a7dY-gjOe0Bxz)np-%I|e@#(K?dV$d#Yzdz z%!m4Z`AVT`^6Sk)Kr**{1ibq00RS6XN@_FeoxxJV*+77wXJa?BS#i(1o{Jj}A&BP|%Li5T!^a;huJ9EIuz2bU1w`NLn z(^Tov&b9A7+dO&uHRNp2qw?65hOCU(t;fOL-FvP&zqc_K@4KIMH`uZ__Xn6NDp@=9 zYPQBj;Tq&v69F{oe_n2qA+xT<>tj;mcrb{PgS@y$C%da;Kpubl`2`P|k{vvZpya%^ zNYHM_6%7d`aFn}tLDx}u_2_Z(^L|1bi;unAI?uf2TGu}G&VN{SbI)EXJoz47?D zj7RMii9y?&g);NSui6J_kG;1(se8W=vbI2WXd>md8E5g%^)s`v&@96%J!aaj`x-I* zb7DcK8>93V-Fz|;{1of*bC0|tDQ=Me=dIV9=>=|e5nM93acAqXR|S|%>mPm?eLR~e#pZ&2db zekEMaOYcga_$`X8#Q|67I!^EDGnoWk(7wopkpUNnQ*MRLY;M1VZDsDs{*>%a3l4=jKfNvM$^Z?VsD!N13KnF<48W1)^0Ji5K&L zI+q{Vk@hfED4(0m-JOx1sDCvly*aumZ_&Jr^1|D$cpn-Ac3v~KMGy4Oghppz7BF2* zuj*K?+D|S|ANkukha|iF(SA`5s-W7|4JesL-irf9>}!$&{aq%#I}JZK?Mr9&FAYx4 z)iH~2CimB4rQ*g;m=obJfdUse-zVRgm;Pe8TTP#7iIm*G%4o17`eiF7)br54pho=p z2bul1oaXd^rU*CHz4}#`m8`wa`a{%!WbbM<4yVg(>hWWe&Z+S&Wq*@dR{-a8b|WpJ z6fQ8PQ&R2tXb{6vzw6o+R#`bXcYAxX#key+UU-tKRawgofBf5XTKPYD`HHP&aWT8- z*5zxe@5evZp8cq6JG=QhgJklNZ*3ZA;qy(Wj6mvB0a$PX>#Wi&v=(e_p|rxv^$A1EKLxiGloNZpyJA82#Gvf#K~% zr}lD1g(&W{E%~{g4__M(*Ax}slQe!oK5$J*+J4|UCTOek%!h>9`QLj5)fhHF#XcFn zbvc|}A|L6tAkTKzR5doq3~?OjPQNHSsB3LTL`=b^E%~82V}xIObp1Dj_Ng43FFD6% zZpZ-rex>hCjGOh{!f{37QhQ$-nB!XNNIwOy;s*x)kw{XDvN`sY z?31x7-=BT<@tXx65Ep#i!TL`4;b{yE*-Rk+6*e*Q1tsJ`w_IP8vb3st*Xqw-u~ac@ znY+Xy8fqoth)SW%_MvX?a`Y_1M`~g@W>W6T^}asgXlVkHEKNe--I>CFrt$=zEkvC+ zRC5hK-_*aAL=+U@!rtug`rg?$Ty=hpBT|4L@sD(lT}!}R9Pp<5%1R=>g)ujIQA^D| zndikOOaoW(+c$I<1L7@kju?uQLTU^?=BV6~{CxTX-#+gyRC`n5IX^be(L&MEKP3n^ z#q~dPB*kQ|=-}(;i1v3s48J$Ktsbl5zi%iCI_+LI5qZV*KI}hTkD=dxfxXb!lEE2b zw-g?5<~Gv<2&6$oAiqoe{E@C+?wXC89i z4miaJ6!9mNAfHBYZRp2qa)UMy;xGaQ#P0@5$%V_t%n2zHczi#rz+lu$AWDM4Am9%3 z|Hc;%G)}yh z979)MQ{H=^gLUmobFURh7*?8)K%ac?q6G+R9CWY4_9*8KXNcRLYiI8h+o51=Ena_omp57{J-NZWS&iRqL<7KEMFc zljd=3^Z;*f#B*RqhcP?4wM{CxEEY58ik{RTrUwW|l7r2SrYQMtT5=|QVEQGEtcw3J zM*~&HL`n06a)Cr<*o0(mP(e{TbmN<+PWhf0p_i>WK}S5f9F>H*LAc2l0#TKz1&#Du zAA|s$m8JmWL44Icef^^C3hpe>k5i_F)50r~nnHM6a!Ff#lJ_1XAWDi9UWYqh_OZ_I z!zrcHYG78gvmy{@Gj_l^VW+~AO_0Lt^ylm8ZQ&fe+40bO>HEnEH-z?V;^;kg(r~ul zGXX(G)aNDn6i~SO`eNgo^pfbSq~6azniOn_#!_OUg?gX0gD_XEP!y^ZOl|cJpc#q? zWrl_%ezJOIso(Q+Hs=Zv`oR{E_;H?@T=YK4RBGm9a5rt|810u8hFY$Bh$5GKKTyFY z3Y0GNDRFNt@c9KQ7HUKCMtnB!KF%|KracLuGQAOmV< zXgtriJ+x6`(z#-`yKC?;I$%X_&iNM)WBUgWVw6>J8#V+u;^GSASSRi!^M$)UuQ!?-^%M{AnsY~z(M;9d^2xJ7p}VIYowepGv|Rl^dwl& zzWPTN0!yU^iZeH|j>gK>Rk?eO_^B8ZHu<(4Si+-_yz^L=DOcgwoZ-Ftn7bAyR0jI9 zj_l~2TEf434~QkT!T2PNLvf&S<(ur(|T0p-_z^i!i7ZemECN*w*@P(`+sGq}B0aAeF6+Q64b2+0- z*C@Vmw1cD)OWEvRm|cX&b8?E^9^Jo^!b-}~t^J{uO_n~NRSBSQjiFYjHGFYFvfTGH zH`t+5@4~6Ct|Afs4n-NlA(^U5d@56Sj!y0GczwEWBntR)*)e`on@HM=^GUxJmjY)6 zc9ZyUpF>HMV0i?}Oua#Z3y?hzbQ|;mEXGgS)?j^h*lJ@%a>?b|&oL<`NXPlb*^Ru4cjzoO1r#zf*hIJv{WDWR)u|3Mk z)0@JTsbawDM*HkQ5fkC!_N_1Ifao2s>cLhS1V)cvg@$ggqD{m-KN*+P18?%lk>;>u zuOLBNqF*1=SKlLgJ)RQB-zZq>uyq3Zs!(S?@Nn#1&oCb_T5AxuJ3pchhYiLa~(S*69O}2O9QQpbhl@L${djWd@fBr5gBZ6h;jc5I0CD@jxS|*}Qt+>BB>PYo*xOOh z2k6G>4}oqn`_be)P^gFW04|69XGLn|C+j`GX7}}ANe)Bx1fdg>&v}c3iGdqU&LA9E%|sd zI*g2c0!5Zw{Iw9s+Ppoal*Jqz=Ck*sM=zxpAKP7HsNLK`u~Ol|^|P8x_+7 z{_eBs$%cT}z4+a$;e#zu?w6i>xS|s#H5=EjK*>3A;jH;C>2W)Ycatd&>Vr9tzEJjU zR9N>?1}yHtLwR>T3~vdSkc$C-chY0s5}Pi0IN5rGbY>HZ%5{Sm;*m-|_rnp8?6l*3 z6KO{6419pW?EAb3#YH>zSB?A%ftXrK_|9 zBaxK%ae&9P;FhymIhrHChIGQmh3qtYV5L_yegO&M|ATy@2SI@5>jO553+Ffa-l=hEz=97{6@9zNLR{N15^AV9ktfSM>`>?W8D}ZP1yy=9ZLZF-B&QGK zWBLd>{=Iqy)=&*5H-3OO>gZUVsTydb)+P7}zz6m$B^k8Ugm3}QqsSa;_4jFZSW*yx zUmGiUZ?xw#SvGc5tEjFI9}-6a$WJ^C!Gb_R$qOW$m%?&BB~vny6VVi;<7hHD5Cd?{ zrA7!A8_=n5D02bMdzS>Yx`x@Bz925Z+b0Ta+?BHWW$5`S$7~4-&xwBhZ{f5iX%GkY z23j!8mb?_u!HH-S2EKM;3EDR8-SZ?01gbb}1&1&?FaTbPHx#ZuiXZj%U;xmJC&}1-fqB%uJzw4 z#NHUbI~Kf{A8o%udk$F5m4+69GyK7kH%Ws=fySTQAku-22KJEY$(r-62r(84++4$5 zL&$&>&G0=&#LJiZgvE$2GIy@Iy8BYzwpM+l7k_}&HHB&C@Lp29g0cb5%1evbu+>Z~ zeFn+bDyu@w+-{ziF)>xv)fcTJ2%7WZTTsnv?$SnFPX3@Z1FHA1sg(}jnr)GC?CzN8 zqXkSm)&I~iQI45IDQFsM5Rqr$(n}w^7p+mh&WPl~605BKR-v+-2;*7KAq*jk7m;v} zHElFGrM+I10*pD#4R38>&xJOzN9f<(PnpZ-Z={cy~X*xlLf^g{ShXG9_wX{!E^35Jl-0ZtGI zEOEhM68fX@AsxACJ8|K+E=Io|$t(87+O`+-@@qpehuS#82mAWggj~5nyg-Le`2>$P zYZSAMh?J3A7SmJWJ7p^L(ls!jz6mx5dAJg2O-tZfxule|&UjuU>Zg5vQB&M`>UOkm z)ni~%D(dz8^Il>k?mGw^QH)oZ+|sE_)D?zL zrSWM?fiO26%)uIDBa$mvGX30kwYZ~ldyKz65G;MHLFNjTddMZ}qCyby1UKm3eSRAd zSxY2EX6U-`4oL zSO;#dbv74qk~+kY0I0aponQe|cMROtP#$%<0zEQJ!+fn_MZG;Eu68c2cDs@mHX?$C zlOrStP$f{gYsKrnIUU;hQGQeji~x^{C#iT%)GHbsrM^xT#9u$UaAU;n*P-Fpp(*Vr zPEvy)CN~+J@7>MK0aJ<}OgoKas`3|ZAD-{4C-1OMIH?x49}q(Clo2Os$X$lxz`Ks+ zDMC^jNY2z*~iw$N?t!g5|Rlg?Fm2JbikK87?Qf-h{{4${Z5tbM%5k`Jh-4TXcaWW!psiS0&|4dY1#4N#Nqu_ z;d+c1SaFfr4ObYYcsP{jQ;tM`OAsd`{Xoi7wB&y$PSj-oxVrV@{>?Lt-MKQc2KrX7 z>SM8Bwb!mUC(49R9de8Q~fL&-&uFwe~l?MTI-+9pO-q z9xN=Sn*4mq5*H8{&u+y1WjSL+&*Iu%(j9!3RnhUm+OsaQ&vw8UjAW#De!SixyJkl{ z<5nc*{|tLnbny#p zRh}Y$KQM5>8{nH}5_of+gp+uP6w(q%ewNo5pUp4+_>f{{KCvvmeTKpSEQJ{*pUl*( z{AIy342OX3uC8KyMyw>LrKbKPGPj4k+;BV58vb|qmA8?-JR~unY22si*XThXj2dBN zqvsNHHK;GVKxblx3uur1cRb&RNxXAem~0_zy2L{PBOJ3;FP2}S)vyzCs0(duse#e~ znk$f(@v!hrGa=v*3?hQ}=P4Iw&a?xrV6P*v=!XFX5`Z`3oidrv)R&mk^Tx-DU@mk~LRaiQ#Xro|OYASztVx-48}+LXXndhR0xdhVzM01h)0 zf+}Uk7}aS3^cJq}C6=5=lUt93h-wyVaQ<;y0qoAUq!<778!hQQ8&6Dk&B3{Nb3D7N z1N;LBAl#^ea7oE~^x#&DlOUrA5)WUv@Y)gf>-_kK0C_&w)T1vtQ)Ff60_*qu2_aCF zLba#h-M%S#k!@iv&C(}2COGRi9GqE`k_EC;jB%S4{~*i`6%=02vTZSKt{D%DvCFTC zc2ng%nm#&t#A4e70;%)9Mm;9L*>`V2jhx_`lj4HXAX49+Aq^ARtH`N|_T7)@nq`~w zoc@QpP9vdJU#5hd^}5U~J(?vM;~$GN@WtE%k!A1}Bh_$gO;ZcY5Fxw?M>Gmz#ReH6 z1zZ}Xnz9>Hvx^b=exMJ1@yhzq?{Bki_h-#`hyn9D2x`jjthsM3DW)Y+39*DMwSTwc?Ln)CpY(#{O>R6g*x8K;y+)TJP$ z%|9H~QB$yX)$nyAbvfA>g9Znz5(ilEGE401{?+3}CqBUth)0>ASHJf`9_gp?Amb+C z27^BM?GrrcZi-dqwEWTIAJJ!LfSogI{-XEKk6E{~`n@p)V&H0kJwO`ur3ceAs$Jij z*O;C2Q#2I>Q{Yl9xsZNP1~8{!h`}d7w#DS;pfYPkC`Rx?)z?kCeE!c?3us+V1+=kg9E7S#*yK>gLB3=CjXS1n4A+}!W?uT6`tynO>S0{jDXRVO308) z2mrSazUcnwEn8>dV*rD#YUY>6tlzk|eam3gZvSx;cM0wpipoV?3~r9S6C%MEKFBDr z9JI$)WFJ{l&#~ZFYH0yRMW86uMBcdO7`IRG*{)H*Z}ERrgviT36HQS;HC6^i?*(Oq z8Yaka;UU!{E)ttt!Hd@c6pivdyA%`&I`u-;Z1XKUQ?6vw9T%XR2nGY^g*{*4_`0

    Zq5m( zmO~b2XkDEf@0StBpC5*G0`ZMsH|l?`B8&EBo}cXzEkZx?QY3s@Jv8>YIWRt(aNa%r zPGXg|xAK7osyHU8Y5LAXa zaN3A-5R)c+?GQLT)VeJkizuFPDT|W#ZKn}YnCYne)pj3aZVkA*i6gym*fUb3-DLL9 z!0ok39FE}CIC)g!5c_?j%WQn}2KmHACaZbJ)hh&oYUc9)F2K$b)i>9A(@wJaY`Y9e z#Kp-k74voTlkWLgl%i8)!7gi1GmDanPfQmm2-69`*r-TfIhWvn{D&ww{zF|;exnj+ zuFdGg7yp}lk=BE}LAjMp)*v+g!tp)p`5QQ&0aCmz#ISOoNj4=;{) zyu^^ojdsWgyy=-RN0CnwvJ+;P7g_E-KVK9|sM#_uyUXTZb1zyO-L>^MoIqq#I0PY~ zRj`}UaBI8!#Pa$=F6zqnm+N2mBAOpPz*5!=pa9VJzg*jUQn~NA(6rNM@ILFoP;?S& ze&aK+paKzElEZh>z8-u!pr{mH5K>Ye9C=B4_r?YXJhF)p@M808;Yb_uj4N6 zES%lCaCdZP7AFyI3(GA2d%ug;F?bCfk;USDc)Kw085Ljs)fZ z*WH-LPh(S5|L)5pRouidec0akWkZg|p@X58D9D?Z2K$TUUA5SiMmR z>z_xD$Tox*biiPNs)k3vpWO!Io!hfdLCvd^5Q{_T(Bv&GK!V&RW5fmC@&nBHR{D1^ zMQ?YGw5SvhKF1}+(g&w72!GDGVxI>K6Zl;Z^Xpv4ht9{(ZhW?hq~)t#^srlIMvN~m zR$R%1g6Z2za=0T<>fv|zSG-liZ$hkSSddBb@Md{zi;Tac*HKOutX`m@8%Qja|GL>A z7okI(IXAZe@{HLEC&gWPW2^Ha~IaO=uW0&j({~<`a=| ztd)%s51eqTXq2>lI94T;4Xrh?p5a+x!y^Gsu=<Oir zkZ+Qd28T6E<*}w@x;uvMU>f}+i4|RXu*98Z>=4vfDlh;e09`*Wqt(UF#_9z0X9NK+ z_mmE1h{G-X&A;R2B$s4p{mC8QF3SUY;6oV6I@_eHj1F-eAWGaUDESbUq(LBqP6+@y zz35b-J%E1T3gB(ms=YifsZZcw!x2lz&e>|>Uh&s#S#ZANV^D$(!0URrV=Rg6xDD#Vc13FO5@>k_%lw~#hca~ZeuP%AfFRxP{TKX-`1BsQRy-jQ z#vvJTSdnL*Y!HJsP{Z7ITNqsH6YyJsfY&3wG6pM8kuds_&1#wobsCK2ZF&Zd2QxSX z{DR!SL(Xmh-i#Gjxqj^KET|DHZh13>ru3G=z1+}tWq5uv{yn5qHrC@HzR4e_OD=>nLr4a=d=<|uH6Io1I+!K3yY{|Y|>jXa@N8t-m@At zna2sWpjcm>nXbvXQ20>smi6JjR#iJL_k+0fLmV7b4Od|!?tdj;{2A40|5#a#t%uY& z-tHF#%8D+3L&nhpR*@3h6(RqX)oJJNm?NFo2et3YThy&+)Osqs5lv*VD=(f;wOJ0$h1|fd6XLo(QJY zpVk&GjDa+iEP1aNZR>WCIrlGK2&ViIOwFf(0-Gtbwcp+*>~U&ReG`noTBsGTCx;j8 zFGYgVwf>#|v`xDu9Lk%;`qqA04w%Mb2P>yE8~&1fbE@z9I3;jaKU67@SS;;w58wOHb;?$|Lh0*%`l#Hn3 zmf=63%gkl|6c-F;{ckdZzUfjkEMLpX>&_C%#_lEu0!R z9$9mlEe>ppy%3x|o@Am^L~nBF20!`*))R3cpqcL00+pw7E7%cS3epQf@Z)n=VY>8g7}O)e>6y2UN_1Vx?VSgH)Dk zX1n)>Pe(0U3Cl16k}EOhJhvr+WLH6fAj(|-&k6SQFD7xmEy7C3!ha~?-XYsk;-o1k ze>HKK^+a3ElTPgxyxK7#d6y#tcXjSF_NlW!gk1TeI4mbxUw4jHF6i~ z|A8!Pz_2G-nGe$q0fBnU;hgFwfC}V^W~38*+jYu{QeL#Wv(+)b?ZokNSZk1rByd1b>>i2hPdu`q~5K~ z->O~{5FiC}jY~asb>18!9Lb>IS3Ak4GPr+BpA;j5Kl^JN%Qdzu zd`6pYTxLEmh}0xwkt$XZj#dTBXa95Js1#2Yx5qYxd9HSWzvW7>2`H$W#9jEE(GssM zTQ+**lXD@STY1W-zrE-*Zf7j$G+EIfkm&^JaqhAYtQ$7~0cDL>544$h# z;91B{?KCVL4i?tGl&wfDm1Ui&$`dj%cND%{qPU~@mHN##d12^N14b1m!7$J6#i_%i z$5eRHBJVBDF`SwUmvME*Xh|QPGl83orN6C4R$!%DcrV~C0~gMy9lk!yA6M{hpE2{K zGd5ab$ZiW0aKf>&`B7O=C@%HX%~6%#g>1`f6nPH!pH$@t54)Rkhk7e#vsD($2-zyI z)TMSZ9klvo8Y$wE5$bU=+!D;s3oe6VbwDQjDr}oo;kXR%kuc^12)MeNc2Rn?{@=b@ zW&!C6qc`4{k2pL#de?{d~U%L?WKS-cb3hU+H+e)=n z(sf9-N*!Lt6#j<$4EOMji<$Wj2f7PqZ2yy-5TOEp%oiC?Q*M6ksInr5p1t!s;v&I* z-ej$>AbPh(+2C@BcY%(qJr>P~2y~wkXmxVdYAtX4jw*?Q2A3VamF6&9erw1t8e~Ke z&3XRr)l1z$#l0wzMe3;CX=iKbOzywy5!7vpB7{|3a27P%d^Jy|!txPaP%n9Qn<^9h zSFG*i)pVlL1!o-Kt&c^=-c?!8N zl-*~+%JWe-y~5y$39qYQ^>_w^?J>#xiZG!`^!5H zl`Xq%G5aUSfA1cw>ZyTuP(Eak6cKp)Sn7S~8auijSXX@0=0&9Nt&F2BGVI!Y;4!=X zW(6nDq4{Cmx1b;AO;nY>j;?|!dzJTFAwz`04=xN1#J%X0Ruf*x444_8p5%WS{2FP7 zslHw7AS-`<@2$;64#zlsMWef=i}UM>q2nCM-JgR2^BdQ!ztLspXQ;|j$LsaUZPNvM zl%Y>(-i;%|iU=U#mJ>q>d?Ttb`))N>^vC=`umReOK}W`+(Rk02$k}DwJek^msnQ%?+-zLvH7dyKVz2&=BBniEyyfCo(JQGs^o!Xs+4yM#5!xxpal}Ib zzdPw?V*EIMU zX)WD}zl>D+*^s)-K6ft1d+@R((jO=Th`;*<_eD%~mKZmmUiW^>Z27y_Q$JMdrY#TE z{d-TQkqX69=4qTz<}G7zi-QBVw zbd7+wVj~{zvLaK6M#j2kAqSZl(=aIfr$+<#N9w+8LD*Hay2dlk0dU#@)jEtkzKfET>#!~1f~ z{nAU?3h*KHY-pCsytvk*6V7~d`c3)~X8xPQ^;I~FWy##fbb-*MJ*a?Y8Dq$gGqy`mT>d&~le}C+#%P&gRgn!ga7_NUkTHQhoxKC@x9{;E-mL-R!kqQ*2oo z`K+N^sbIZ>*yvl17gIZHgSe~z#@OX7F!FbxR2_hhy!&pPU}^(QT3DFXSNOFJ@0&G$ zKgN=ms#5yVntG9g6_j^IHYpC#hWrHF^-rC+;2(BAg^It(VHuM7wCmo!iwn%rI0)n` zfHe;H#cg84sc;{BW;u3RXA@mXjC*w7`?9mfR^y3=f*oTwOi1G?J_G<1#Bk0xUqgfZ zyAv-jyE~b-8|p@WyaRqX9ZeOx z-U%CI$wOHBBji9}&DJLxSPZAio%ekVi3%(~90zc3$%-7BZ-p2D#3?}=0ao7fvc&AN z=t0;3X+~i8&xH4S8Tvsp#nz*D2DE>wCcl~95@ipk1b`(GHmq2dy2TDo&4wOvNs>pK zYi0{hWqvQLJNnITExyTTu+F^4=o0`~5AhCYfGj-GzW3(N*atqqnSfPBT#p#RkWiBA zut*QTRivJz?mKBiS)}j{`}u$ERufx-^ZZpPr}Jc1#Homgl)n*! za&uBbrH;S+*MFPUyH&=lKX6p4#sgk|ISVFU@63ujF>wYkkOQK%bgOHbLL6XW|Bd)G z-x6-t))gmwmeu*yS1i)glI}kD&?K{I-lh7?An=muM8hkYuUN^h(h~Yvw+EeDpg@6t zqxkvR7MBu|Rq_+W2n!k9Y3Im$z%0eDT4-R;SlTX14gpk4`dQ7W3Hbov8R7LN z{gQ%6gGo6=34RYy#5)k$FiCv`Toa)hsP)$mBZBdlriG=COR*Go#zTf7?j;&&Y5D19 z(ajZy{6O&QR)&XnKnmm;*?PTLY!xdPl=tHPfjpVCEr(z+kT^CLrB4tt5XlI$s5br}s_3T;oHmQ)|rX}w{WADb2d4G*^`!b7cw z0rgnMI-5exn{v!i;$T?=ExAj0dZYNP@X8n0z|#mGjP;!$eD=d%KcEs{v@LH}eydGN zVXx!0|7)Awo#4RYLkSskm@63IxX8al?EH=jSE|ENS7_^ryFaBz?5H>!UW?E9QiYVE zyQ#U9wdKJpr3Ar=kIlIO)F?Hi(`L^Vq$!03{^zMa77Bv(@WeMQ>#r4)PO>-~3``0k znyhoS30_I2hbRy7CVn)sWQDj=IMAx;!9ot>0L#7Kd<-%|0%l7O7n_)`u$~&+$bTE$ zitEVw_?e9qILO5lf)FhSxzGDTVh^`aH6DWLeg_T!_X?X2!B$ zjtR!N>wRW#&h#G3Aav1#)l_Th^bEy2CRErtZ%0iCWWDTRhU*n!MXG_D-V( zjFr~Z`<6gZfVkS*v-@NrtfR#m?$s1Dw)e3vr_x=7p(lq`a09@I)XUw&;n+VP+OIOU z(v9ORRwoi@uj4n{503j!9OCR7Q{1gfl36yT22WMJd7^+?&Z7;sDweX^Gn8-G8m4Kj z$phK%@+-(tzC@Y?m)g&#t8t9Lx2e?!AJnfDd9Y@d%QdumYY}Ih%ea17(pe2esBn4i z@w5d7veTl_R-Mb*_}>sJMOzcLCy5`l%o4A@hlLb>L?M{p$K8|sP?NF&% zu>J3iOhauYEPI=4^5^f)s(muYRk4&e*_<95K}pBzc92*>8~m_TgtlUUp4UAV^cG*! z!l|~LHEGo8U34(VP5n&L3*bycLPm$L*B4y1Up%%w5KQ7<{`a9Z+zQ0D;>b58d*n zf*e(4sj)WLvc=Jpg@FcT4#fS_J&lnVXT{e@9|cM%aV#FNdF)W~Oi8=GK|gT{C+ffe zMmlyAiXlw-k&$dyHE4j5)w>bQL=iJR^8W_09$;&4G-G_moLG3(A;yQk)v^9RxCJ&a zcGJeOFIDGv_GJeuc0s@KLBc}==o0eE4ww)-;Re#c;o13fls_RXyJPaTG&P1+*?SDi z4tFS~#tFhFnj?723=>Y$^+OKhfakuXl3=o}5=8qX&3bB-Z%Ru~9P`z}mGXV#`a_3S zrVUe_+f#lZRPZ^m4qb`$%5&FPE@c*$&<{P4lqtJf)fYKR>Dsr@y5R_~xOp;KS?F4J z>#ZhW)RjwunbAw-H4a)dis;hn&jrD`J1agwu;^jS9F!Z|m7zkK8zZtds-)O9?8W?A zO48ZL5wx`&plfKma!8|ajNH_t0Q+sH2Fh86qn+iD(A?OnZ$65dKfkNhyYH%3!LhzW z*%SfZn>Vq{XzGmIzRx6e7OhemEIy)~nNURfiN-3-Hk(Q+=#g8HaOln8OQfJ=fiU71B|$fvSf$R!x+E}Di|PI=!y|4z>6NqIxVrv zjPPwNIF31FC`W{6cIXz_a(Xw{$=+?q-X48B>?Dz;Zst<7RHdp(sCfF;#Y#D>`GG~06*{Dyf zp-sTTHzCgCcC0FN;Qnf&yW^#+iZHuJHO`17jlJ8EFKz@12d;!71xZ=;<4bleE}t}k zJ8CNI>u++y*CK!;Pc~B zx%+{f+ugL{Ln&};qFxt2287E-eO?L~0Y`ESLR-a!lIfO`G~Owg+QaF*K7YF#KA={q%bk zBZ-@EPr6&%D8tMgSg~u1w%E(8gtJ?zqd#_+8C-;kwpsp;z60UDPBZqv1{Dbw#^Y0v>~u-0!HbszeZ_erV`|C^?UElM%A711&3)ChRd+pX>a z8297IromSZUAe82llScxzihlUUE8KfPo_MW?tyYS=2|98wdGIhhotaWh0R8zW_Vp@ zVGoaRY!gzzJRwuI$*eI#48uZAFwjX>&?_7lYCbK9=;(Q=Tcq9>AZ0i4rNMU}ltLQt zuDkA!B2P?mh32)ae#@uLK*n&vzJjaG{`*aY)_eLdbh_Mm^|5=rNl8YA5sln%EVZG9e<3-bKb$soasuiW=lsH z*-%!Xij6UA_$(I zkNVcfYk+5Gq}nMsOU#1OCZ(RkRSJR-GV`^%77KJ%1ZEJMi<#b# z7?_-{db4FN?!|-W3}v@1)i7YcTrvE?lc}Z`H5he-#oR;^cm0;SVAV1pW#emR|Vu1IjXmaWPow^s{6cQi30l92Pe-4Dlepyq76R+Drttm0S9~u2n zenss#yQk7YrZEiF*!J#{Y zC4QD5eRN1;V3mEFJS?OXI%q2~)Qbi2e0u5EQqyVJuO0A{`OY;V`J^Y{jRy=E02ginLVX#XxNB(fMWOQ46mnS`CD(KNB!}? z2dlWUfzF`Z_@i_9)~cl<3D3EFVw<6~y|w3I!8ve5or3{eETP<_XRl4EX3(s&wAFBt zt%Y#TUL7S*YK0a(6~wv;%K*2=dA^PJ18gW`jDBQz%hZ(f?< z*+7&hMjKo@wBg9lj?ui+KzbS8l(6F-D|+zuBU$d^Ks;B3(Bns58AOx`&-;rkq{fl) z{iS^2nHqSs#4krBd;!8w{M8!v1dTjH21ew3QYgeA{l54;*c=pG=i>35Tmko z@}_TOerRiAV+=`7w=M9N)Ms(!pR%7&5?;fGLTyVh65pG3A}x9-hPpQ;V^yFEgxd zJu;&|q`;xo>6IMki}T!q&GO(|N#3tXX-45VkQgSBg#o?{G9wgRsSdC1Ca`H>;>j3& zgEAzsh2-AL%OHiL0y&>21MQ_TzvR~xVA8#t8*%oOD9r9fu(|+%!&uduYckXiX85!c z@oKaK)x>vp$X`=@zC5^?KCj=hUcgQ%rN)210OToGrM9Z^mbW2!UDD$j_RFe39;XNU zQ^WP1tWf-4z#fZqpbkcB*_@D9K?VG4?4n#ca=DTe-tpOZ*LaZU!jkAz%dhZSW2i6^ zuDZFo>EE&4JY$aaE!pn@y2=PPE<2>P_>w_vAn5f2v$cEL2dMV?O5=u!`u!%hznj^$ zUMAO(KLHYs(Tg8bT|unsNbo@X1PbW*exbH7`Ii$AYY(UZx55tR%yEquNK5NTb#%VT z0Mcad|Na*^hrlX9hl%JRjTB?-GYf)gkxz`xgR^rbf{WY-{Gp$DuO=t30Q{BW0@KZC z&NO417^f&Vp?J}G1EepYd}EDl<-90kdV!SV60flQiYE9Z^SMIm$(-`}T0*_Hd!P{> zZDnNcG$i@i#F!Rvuw+w0@uQ!_5~ZNR(!dB=0R>Px0>#wEoK)XY%>WbS0Qs)~-W$~C zQP(Ju;U7_I7k=0Q0I)4;|B0wGr=rd}ohF@yAHWXl9X6KM4wgF{|Jh;dWNYJOz0<lg1rflTk z2q>t4q{mNs7h4@D5@==Q>WMYWcqavI8u&V#>vn&-7DS&V70b@mJ0%O5S=)593M<;k zB~hY-Z8+bE0_2YFw=Mz#KObN5+B0hWs%N&L|C7Uc>dBpoRgBoXyyoCDm?=%FZma*w z$X%))S@+h^9zi~K$-p?J=OGK=wkNKq0_ z^)$gjY-sh&P%o4B$%k+oA*Q@#J22n`* zH8rg?d|E`iQ)0q;{;NqdVr~P}9m2y2ln; zA&EHoX+g$N z_cn6@U8#p&b&t26<7<0Ot6mvu+ejTCYRcy@7X^6>6Xs+FIko`%R@ViYivUSog9Ygl zKcMl)skL77l&A_PzIl-dCYuSxKSe(l@oQp*B6~==^1|A&1o4g-rg}e?K_3y-PV#i~ zgQW)=LWqS6FdhkCTtzS>YR{7ulM(;%!cykHqgfN#y7AqSMjs`YyIK=GBpX<;_nZBQ z3JQ>zb8fB7mbH@fR*E9ZfJxh1MJ=lh33&+^cnO%oh_DD>LxTQRn!LZ!Jrg6p z07Z+)H#uvO6O7zw28I|moUlWEf2rRad~utj4nahaB#}ztD`gTgb=!Ni_`VF=vt>VG z1UtGNc87}$^6E$_v-Rvl7SAn*9)y>GtU^smxgOPygE>E1Ci0xtTjd6|cr4r$ zu(nQOh`t|>6M!9Rr%tYC=DbCgs_a98;8L44#%v$m*&j8ROE9JfUcMwZpc}UEx|7`! z1!<}@Oo@5cA2BE%oXwz}BWS~J!wB_50$bgn-m9{_UDBj9^0)xMf04POqJ;NWY5;Mu z&Q(B#Q0Qq=xWxW)k*-RFir>V$`*5vrY+rYCe(%kdO4h6?bLj*oV!&uZIOALG1DE$~ zPeR2n16&`tRA*)RPl_pIOE{lm>{R8M9+w>NweD{_-9Y6r)JsSmq^?wkp0kqOS{W7b9;ML#J%G3zNO+13bBkS zc*`C&dBP_hqgj)0B8u!ejTX6W^dlq2xhk)3PHf)&GQFLg?o+fS`ttk1#m4}?0Y8zG z$+!7QvXUX^iZghd0ykw z+>jM|fS-_V(a?k6YB=oia%RwCW=;IhnJ~2AXttUPE z(%`$>Npo|?wHejLkJeT}o0WGZ$uci&a7dY-gjOe0Bxz)np-%I|e@#(K?dV$d#Yzdz z%!m4Z`AVT`^6Sk)Kr**{1ibq00RS6XN@_FeoxxJV*+77wXJa?BS#i(1o{Jj}A&BP|%Li5T!^a;huJ9EIuz2bU1w`NLn z(^Tov&b9A7+dO&uHRNp2qw?65hOCU(t;fOL-FvP&zqc_K@4KIMH`uZ__Xn6NDp@=9 zYPQBj;Tq&v69F{oe_n2qA+xT<>tj;mcrb{PgS@y$C%da;Kpubl`2`P|k{vvZpya%^ zNYHM_6%7d`aFn}tLDx}u_2_Z(^L|1bi;unAI?uf2TGu}G&VN{SbI)EXJoz47?D zj7RMii9y?&g);NSui6J_kG;1(se8W=vbI2WXd>md8E5g%^)s`v&@96%J!aaj`x-I* zb7DcK8>93V-Fz|;{1of*bC0|tDQ=Me=dIV9=>=|e5nM93acAqXR|S|%>mPm?eLR~e#pZ&2db zekEMaOYcga_$`X8#Q|67I!^EDGnoWk(7wopkpUNnQ*MRLY;M1VZDsDs{*>%a3l4=jKfNvM$^Z?VsD!N13KnF<48W1)^0Ji5K&L zI+q{Vk@hfED4(0m-JOx1sDCvly*aumZ_&Jr^1|D$cpn-Ac3v~KMGy4Oghppz7BF2* zuj*K?+D|S|ANkukha|iF(SA`5s-W7|4JesL-irf9>}!$&{aq%#I}JZK?Mr9&FAYx4 z)iH~2CimB4rQ*g;m=obJfdUse-zVRgm;Pe8TTP#7iIm*G%4o17`eiF7)br54pho=p z2bul1oaXd^rU*CHz4}#`m8`wa`a{%!WbbM<4yVg(>hWWe&Z+S&Wq*@dR{-a8b|WpJ z6fQ8PQ&R2tXb{6vzw6o+R#`bXcYAxX#key+UU-tKRawgofBf5XTKPYD`HHP&aWT8- z*5zxe@5evZp8cq6JG=QhgJklNZ*3ZA;qy(Wj6mvB0a$PX>#Wi&v=(e_p|rxv^$A1EKLxiGloNZpyJA82#Gvf#K~% zr}lD1g(&W{E%~{g4__M(*Ax}slQe!oK5$J*+J4|UCTOek%!h>9`QLj5)fhHF#XcFn zbvc|}A|L6tAkTKzR5doq3~?OjPQNHSsB3LTL`=b^E%~82V}xIObp1Dj_Ng43FFD6% zZpZ-rex>hCjGOh{!f{37QhQ$-nB!XNNIwOy;s*x)kw{XDvN`sY z?31x7-=BT<@tXx65Ep#i!TL`4;b{yE*-Rk+6*e*Q1tsJ`w_IP8vb3st*Xqw-u~ac@ znY+Xy8fqoth)SW%_MvX?a`Y_1M`~g@W>W6T^}asgXlVkHEKNe--I>CFrt$=zEkvC+ zRC5hK-_*aAL=+U@!rtug`rg?$Ty=hpBT|4L@sD(lT}!}R9Pp<5%1R=>g)ujIQA^D| zndikOOaoW(+c$I<1L7@kju?uQLTU^?=BV6~{CxTX-#+gyRC`n5IX^be(L&MEKP3n^ z#q~dPB*kQ|=-}(;i1v3s48J$Ktsbl5zi%iCI_+LI5qZV*KI}hTkD=dxfxXb!lEE2b zw-g?5<~Gv<2&6$oAiqoe{E@C+?wXC89i z4miaJ6!9mNAfHBYZRp2qa)UMy;xGaQ#P0@5$%V_t%n2zHczi#rz+lu$AWDM4Am9%3 z|Hc;%G)}yh z979)MQ{H=^gLUmobFURh7*?8)K%ac?q6G+R9CWY4_9*8KXNcRLYiI8h+o51=Ena_omp57{J-NZWS&iRqL<7KEMFc zljd=3^Z;*f#B*RqhcP?4wM{CxEEY58ik{RTrUwW|l7r2SrYQMtT5=|QVEQGEtcw3J zM*~&HL`n06a)Cr<*o0(mP(e{TbmN<+PWhf0p_i>WK}S5f9F>H*LAc2l0#TKz1&#Du zAA|s$m8JmWL44Icef^^C3hpe>k5i_F)50r~nnHM6a!Ff#lJ_1XAWDi9UWYqh_OZ_I z!zrcHYG78gvmy{@Gj_l^VW+~AO_0Lt^ylm8ZQ&fe+40bO>HEnEH-z?V;^;kg(r~ul zGXX(G)aNDn6i~SO`eNgo^pfbSq~6azniOn_#!_OUg?gX0gD_XEP!y^ZOl|cJpc#q? zWrl_%ezJOIso(Q+Hs=Zv`oR{E_;H?@T=YK4RBGm9a5rt|810u8hFY$Bh$5GKKTyFY z3Y0GNDRFNt@c9KQ7HUKCMtnB!KF%|KracLuGQAOmV< zXgtriJ+x6`(z#-`yKC?;I$%X_&iNM)WBUgWVw6>J8#V+u;^GSASSRi!^M$)UuQ!?-^%M{AnsY~z(M;9d^2xJ7p}VIYowepGv|Rl^dwl& zzWPTN0!yU^iZeH|j>gK>Rk?eO_^B8ZHu<(4Si+-_yz^L=DOcgwoZ-Ftn7bAyR0jI9 zj_l~2TEf434~QkT!T2PNLvf&S<(ur(|T0p-_z^i!i7ZemECN*w*@P(`+sGq}B0aAeF6+Q64b2+0- z*C@Vmw1cD)OWEvRm|cX&b8?E^9^Jo^!b-}~t^J{uO_n~NRSBSQjiFYjHGFYFvfTGH zH`t+5@4~6Ct|Afs4n-NlA(^U5d@56Sj!y0GczwEWBntR)*)e`on@HM=^GUxJmjY)6 zc9ZyUpF>HMV0i?}Oua#Z3y?hzbQ|;mEXGgS)?j^h*lJ@%a>?b|&oL<`NXPlb*^Ru4cjzoO1r#zf*hIJv{WDWR)u|3Mk z)0@JTsbawDM*HkQ5fkC!_N_1Ifao2s>cLhS1V)cvg@$ggqD{m-KN*+P18?%lk>;>u zuOLBNqF*1=SKlLgJ)RQB-zZq>uyq3Zs!(S?@Nn#1&oCb_T5AxuJ3pchhYiLa~(S*69O}2O9QQpbhl@L${djWd@fBr5gBZ6h;jc5I0CD@jxS|*}Qt+>BB>PYo*xOOh z2k6G>4}oqn`_be)P^gFW04|69XGLn|C+j`GX7}}ANe)Bx1fdg>&v}c3iGdqU&LA9E%|sd zI*g2c0!5Zw{Iw9s+Ppoal*Jqz=Ck*sM=zxpAKP7HsNLK`u~Ol|^|P8x_+7 z{_eBs$%cT}z4+a$;e#zu?w6i>xS|s#H5=EjK*>3A;jH;C>2W)Ycatd&>Vr9tzEJjU zR9N>?1}yHtLwR>T3~vdSkc$C-chY0s5}Pi0IN5rGbY>HZ%5{Sm;*m-|_rnp8?6l*3 z6KO{6419pW?EAb3#YH>zSB?A%ftXrK_|9 zBaxK%ae&9P;FhymIhrHChIGQmh3qtYV5L_yegO&M|ATy@2SI@5>jO553+Ffa-l=hEz=97{6@9zNLR{N15^AV9ktfSM>`>?W8D}ZP1yy=9ZLZF-B&QGK zWBLd>{=Iqy)=&*5H-3OO>gZUVsTydb)+P7}zz6m$B^k8Ugm3}QqsSa;_4jFZSW*yx zUmGiUZ?xw#SvGc5tEjFI9}-6a$WJ^C!Gb_R$qOW$m%?&BB~vny6VVi;<7hHD5Cd?{ zrA7!A8_=n5D02bMdzS>Yx`x@Bz925Z+b0Ta+?BHWW$5`S$7~4-&xwBhZ{f5iX%GkY z23j!8mb?_u!HH-S2EKM;3EDR8-SZ?01gbb}1&1&?FaTbPHx#ZuiXZj%U;xmJC&}1-fqB%uJzw4 z#NHUbI~Kf{A8o%udk$F5m4+69GyK7kH%Ws=fySTQAku-22KJEY$(r-62r(84++4$5 zL&$&>&G0=&#LJiZgvE$2GIy@Iy8BYzwpM+l7k_}&HHB&C@Lp29g0cb5%1evbu+>Z~ zeFn+bDyu@w+-{ziF)>xv)fcTJ2%7WZTTsnv?$SnFPX3@Z1FHA1sg(}jnr)GC?CzN8 zqXkSm)&I~iQI45IDQFsM5Rqr$(n}w^7p+mh&WPl~605BKR-v+-2;*7KAq*jk7m;v} zHElFGrM+I10*pD#4R38>&xJOzN9f<(PnpZ-Z={cy~X*xlLf^g{ShXG9_wX{!E^35Jl-0ZtGI zEOEhM68fX@AsxACJ8|K+E=Io|$t(87+O`+-@@qpehuS#82mAWggj~5nyg-Le`2>$P zYZSAMh?J3A7SmJWJ7p^L(ls!jz6mx5dAJg2O-tZfxule|&UjuU>Zg5vQB&M`>UOkm z)ni~%D(dz8^Il>k?mGw^QH)oZ+|sE_)D?zL zrSWM?fiO26%)uIDBa$mvGX30kwYZ~ldyKz65G;MHLFNjTddMZ}qCyby1UKm3eSRAd zSxY2EX6U-`4oL zSO;#dbv74qk~+kY0I0aponQe|cMROtP#$%<0zEQJ!+fn_MZG;Eu68c2cDs@mHX?$C zlOrStP$f{gYsKrnIUU;hQGQeji~x^{C#iT%)GHbsrM^xT#9u$UaAU;n*P-Fpp(*Vr zPEvy)CN~+J@7>MK0aJ<}OgoKas`3|ZAD-{4C-1OMIH?x49}q(Clo2Os$X$lxz`Ks+ zDMC^jNY2z*~iw$N?t!g5|Rlg?Fm2JbikK87?Qf-h{{4${Z5tbM%5k`Jh-4TXcaWW!psiS0&|4dY1#4N#Nqu_ z;d+c1SaFfr4ObYYcsP{jQ;tM`OAsd`{Xoi7wB&y$PSj-oxVrV@{>?Lt-MKQc2KrX7 z>SM8Bwb!mUC(49R9de8Q~fL&-&uFwe~l?MTI-+9pO-q z9xN=Sn*4mq5*H8{&u+y1WjSL+&*Iu%(j9!3RnhUm+OsaQ&vw8UjAW#De!SixyJkl{ z<5nc*{|tLnbny#p zRh}Y$KQM5>8{nH}5_of+gp+uP6w(q%ewNo5pUp4+_>f{{KCvvmeTKpSEQJ{*pUl*( z{AIy342OX3uC8KyMyw>LrKbKPGPj4k+;BV58vb|qmA8?-JR~unY22si*XThXj2dBN zqvsNHHK;GVKxblx3uur1cRb&RNxXAem~0_zy2L{PBOJ3;FP2}S)vyzCs0(duse#e~ znk$f(@v!hrGa=v*3?hQ}=P4Iw&a?xrV6P*v=!XFX5`Z`3oidrv)R&mk^Tx-DU@mk~LRaiQ#Xro|OYASztVx-48}+LXXndhR0xdhVzM01h)0 zf+}Uk7}aS3^cJq}C6=5=lUt93h-wyVaQ<;y0qoAUq!<778!hQQ8&6Dk&B3{Nb3D7N z1N;LBAl#^ea7oE~^x#&DlOUrA5)WUv@Y)gf>-_kK0C_&w)T1vtQ)Ff60_*qu2_aCF zLba#h-M%S#k!@iv&C(}2COGRi9GqE`k_EC;jB%S4{~*i`6%=02vTZSKt{D%DvCFTC zc2ng%nm#&t#A4e70;%)9Mm;9L*>`V2jhx_`lj4HXAX49+Aq^ARtH`N|_T7)@nq`~w zoc@QpP9vdJU#5hd^}5U~J(?vM;~$GN@WtE%k!A1}Bh_$gO;ZcY5Fxw?M>Gmz#ReH6 z1zZ}Xnz9>Hvx^b=exMJ1@yhzq?{Bki_h-#`hyn9D2x`jjthsM3DW)Y+39*DMwSTwc?Ln)CpY(#{O>R6g*x8K;y+)TJP$ z%|9H~QB$yX)$nyAbvfA>g9Znz5(ilEGE401{?+3}CqBUth)0>ASHJf`9_gp?Amb+C z27^BM?GrrcZi-dqwEWTIAJJ!LfSogI{-XEKk6E{~`n@p)V&H0kJwO`ur3ceAs$Jij z*O;C2Q#2I>Q{Yl9xsZNP1~8{!h`}d7w#DS;pfYPkC`Rx?)z?kCeE!c?3us+V1+=kg9E7S#*yK>gLB3=CjXS1n4A+}!W?uT6`tynO>S0{jDXRVO308) z2mrSazUcnwEn8>dV*rD#YUY>6tlzk|eam3gZvSx;cM0wpipoV?3~r9S6C%MEKFBDr z9JI$)WFJ{l&#~ZFYH0yRMW86uMBcdO7`IRG*{)H*Z}ERrgviT36HQS;HC6^i?*(Oq z8Yaka;UU!{E)ttt!Hd@c6pivdyA%`&I`u-;Z1XKUQ?6vw9T%XR2nGY^g*{*4_`0

    Zq5m( zmO~b2XkDEf@0StBpC5*G0`ZMsH|l?`B8&EBo}cXzEkZx?QY3s@Jv8>YIWRt(aNa%r zPGXg|xAK7osyHU8Y5LAXa zaN3A-5R)c+?GQLT)VeJkizuFPDT|W#ZKn}YnCYne)pj3aZVkA*i6gym*fUb3-DLL9 z!0ok39FE}CIC)g!5c_?j%WQn}2KmHACaZbJ)hh&oYUc9)F2K$b)i>9A(@wJaY`Y9e z#Kp-k74voTlkWLgl%i8)!7gi1GmDanPfQmm2-69`*r-TfIhWvn{D&ww{zF|;exnj+ zuFdGg7yp}lk=BE}LAjMp)*v+g!tp)p`5QQ&0aCmz#ISOoNj4=;{) zyu^^ojdsWgyy=-RN0CnwvJ+;P7g_E-KVK9|sM#_uyUXTZb1zyO-L>^MoIqq#I0PY~ zRj`}UaBI8!#Pa$=F6zqnm+N2mBAOpPz*5!=pa9VJzg*jUQn~NA(6rNM@ILFoP;?S& ze&aK+paKzElEZh>z8-u!pr{mH5K>Ye9C=B4_r?YXJhF)p@M808;Yb_uj4N6 zES%lCaCdZP7AFyI3(GA2d%ug;F?bCfk;USDc)Kw085Ljs)fZ z*WH-LPh(S5|L)5pRouidec0akWkZg|p@X58D9D?Z2K$TUUA5SiMmR z>z_xD$Tox*biiPNs)k3vpWO!Io!hfdLCvd^5Q{_T(Bv&GK!V&RW5fmC@&nBHR{D1^ zMQ?YGw5SvhKF1}+(g&w72!GDGVxI>K6Zl;Z^Xpv4ht9{(ZhW?hq~)t#^srlIMvN~m zR$R%1g6Z2za=0T<>fv|zSG-liZ$hkSSddBb@Md{zi;Tac*HKOutX`m@8%Qja|GL>A z7okI(IXAZe@{HLEC&gWPW2^Ha~IaO=uW0&j({~<`a=| ztd)%s51eqTXq2>lI94T;4Xrh?p5a+x!y^Gsu=<Oir zkZ+Qd28T6E<*}w@x;uvMU>f}+i4|RXu*98Z>=4vfDlh;e09`*Wqt(UF#_9z0X9NK+ z_mmE1h{G-X&A;R2B$s4p{mC8QF3SUY;6oV6I@_eHj1F-eAWGaUDESbUq(LBqP6+@y zz35b-J%E1T3gB(ms=YifsZZcw!x2lz&e>|>Uh&s#S#ZANV^D$(!0URrV=Rg6xDD#Vc13FO5@>k_%lw~#hca~ZeuP%AfFRxP{TKX-`1BsQRy-jQ z#vvJTSdnL*Y!HJsP{Z7ITNqsH6YyJsfY&3wG6pM8kuds_&1#wobsCK2ZF&Zd2QxSX z{DR!SL(Xmh-i#Gjxqj^KET|DHZh13>ru3G=z1+}tWq5uv{yn5qHrC@HzR4e_OD=>nLr4a=d=<|uH6Io1I+!K3yY{|Y|>jXa@N8t-m@At zna2sWpjcm>nXbvXQ20>smi6JjR#iJL_k+0fLmV7b4Od|!?tdj;{2A40|5#a#t%uY& z-tHF#%8D+3L&nhpR*@3h6(RqX)oJJNm?NFo2et3YThy&+)Osqs5lv*VD=(f;wOJ0$h1|fd6XLo(QJY zpVk&GjDa+iEP1aNZR>WCIrlGK2&ViIOwFf(0-Gtbwcp+*>~U&ReG`noTBsGTCx;j8 zFGYgVwf>#|v`xDu9Lk%;`qqA04w%Mb2P>yE8~&1fbE@z9I3;jaKU67@SS;;w58wOHb;?$|Lh0*%`l#Hn3 zmf=63%gkl|6c-F;{ckdZzUfjkEMLpX>&_C%#_lEu0!R z9$9mlEe>ppy%3x|o@Am^L~nBF20!`*))R3cpqcL00+pw7E7%cS3epQf@Z)n=VY>8g7}O)e>6y2UN_1Vx?VSgH)Dk zX1n)>Pe(0U3Cl16k}EOhJhvr+WLH6fAj(|-&k6SQFD7xmEy7C3!ha~?-XYsk;-o1k ze>HKK^+a3ElTPgxyxK7#d6y#tcXjSF_NlW!gk1TeI4mbxUw4jHF6i~ z|A8!Pz_2G-nGe$q0fBnU;hgFwfC}V^W~38*+jYu{QeL#Wv(+)b?ZokNSZk1rByd1b>>i2hPdu`q~5K~ z->O~{5FiC}jY~asb>18!9Lb>IS3Ak4GPr+BpA;j5Kl^JN%Qdzu zd`6pYTxLEmh}0xwkt$XZj#dTBXa95Js1#2Yx5qYxd9HSWzvW7>2`H$W#9jEE(GssM zTQ+**lXD@STY1W-zrE-*Zf7j$G+EIfkm&^JaqhAYtQ$7~0cDL>544$h# z;91B{?KCVL4i?tGl&wfDm1Ui&$`dj%cND%{qPU~@mHN##d12^N14b1m!7$J6#i_%i z$5eRHBJVBDF`SwUmvME*Xh|QPGl83orN6C4R$!%DcrV~C0~gMy9lk!yA6M{hpE2{K zGd5ab$ZiW0aKf>&`B7O=C@%HX%~6%#g>1`f6nPH!pH$@t54)Rkhk7e#vsD($2-zyI z)TMSZ9klvo8Y$wE5$bU=+!D;s3oe6VbwDQjDr}oo;kXR%kuc^12)MeNc2Rn?{@=b@ zW&!C6qc`4{k2pL#de?{d~U%L?WKS-cb3hU+H+e)=n z(sf9-N*!Lt6#j<$4EOMji<$Wj2f7PqZ2yy-5TOEp%oiC?Q*M6ksInr5p1t!s;v&I* z-ej$>AbPh(+2C@BcY%(qJr>P~2y~wkXmxVdYAtX4jw*?Q2A3VamF6&9erw1t8e~Ke z&3XRr)l1z$#l0wzMe3;CX=iKbOzywy5!7vpB7{|3a27P%d^Jy|!txPaP%n9Qn<^9h zSFG*i)pVlL1!o-Kt&c^=-c?!8N zl-*~+%JWe-y~5y$39qYQ^>_w^?J>#xiZG!`^!5H zl`Xq%G5aUSfA1cw>ZyTuP(Eak6cKp)Sn7S~8auijSXX@0=0&9Nt&F2BGVI!Y;4!=X zW(6nDq4{Cmx1b;AO;nY>j;?|!dzJTFAwz`04=xN1#J%X0Ruf*x444_8p5%WS{2FP7 zslHw7AS-`<@2$;64#zlsMWef=i}UM>q2nCM-JgR2^BdQ!ztLspXQ;|j$LsaUZPNvM zl%Y>(-i;%|iU=U#mJ>q>d?Ttb`))N>^vC=`umReOK}W`+(Rk02$k}DwJek^msnQ%?+-zLvH7dyKVz2&=BBniEyyfCo(JQGs^o!Xs+4yM#5!xxpal}Ib zzdPw?V*EIMU zX)WD}zl>D+*^s)-K6ft1d+@R((jO=Th`;*<_eD%~mKZmmUiW^>Z27y_Q$JMdrY#TE z{d-TQkqX69=4qTz<}G7zi-QBVw zbd7+wVj~{zvLaK6M#j2kAqSZl(=aIfr$+<#N9w+8LD*Hay2dlk0dU#@)jEtkzKfET>#!~1f~ z{nAU?3h*KHY-pCsytvk*6V7~d`c3)~X8xPQ^;I~FWy##fbb-*MJ*a?Y8Dq$gGqy`mT>d&~le}C+#%P&gRgn!ga7_NUkTHQhoxKC@x9{;E-mL-R!kqQ*2oo z`K+N^sbIZ>*yvl17gIZHgSe~z#@OX7F!FbxR2_hhy!&pPU}^(QT3DFXSNOFJ@0&G$ zKgN=ms#5yVntG9g6_j^IHYpC#hWrHF^-rC+;2(BAg^It(VHuM7wCmo!iwn%rI0)n` zfHe;H#cg84sc;{BW;u3RXA@mXjC*w7`?9mfR^y3=f*oTwOi1G?J_G<1#Bk0xUqgfZ zyAv-jyE~b-8|p@WyaRqX9ZeOx z-U%CI$wOHBBji9}&DJLxSPZAio%ekVi3%(~90zc3$%-7BZ-p2D#3?}=0ao7fvc&AN z=t0;3X+~i8&xH4S8Tvsp#nz*D2DE>wCcl~95@ipk1b`(GHmq2dy2TDo&4wOvNs>pK zYi0{hWqvQLJNnITExyTTu+F^4=o0`~5AhCYfGj-GzW3(N*atqqnSfPBT#p#RkWiBA zut*QTRivJz?mKBiS)}j{`}u$ERufx-^ZZpPr}Jc1#Homgl)n*! za&uBbrH;S+*MFPUyH&=lKX6p4#sgk|ISVFU@63ujF>wYkkOQK%bgOHbLL6XW|Bd)G z-x6-t))gmwmeu*yS1i)glI}kD&?K{I-lh7?An=muM8hkYuUN^h(h~Yvw+EeDpg@6t zqxkvR7MBu|Rq_+W2n!k9Y3Im$z%0eDT4-R;SlTX14gpk4`dQ7W3Hbov8R7LN z{gQ%6gGo6=34RYy#5)k$FiCv`Toa)hsP)$mBZBdlriG=COR*Go#zTf7?j;&&Y5D19 z(ajZy{6O&QR)&XnKnmm;*?PTLY!xdPl=tHPfjpVCEr(z+kT^CLrB4tt5XlI$s5br}s_3T;oHmQ)|rX}w{WADb2d4G*^`!b7cw z0rgnMI-5exn{v!i;$T?=ExAj0dZYNP@X8n0z|#mGjP;!$eD=d%KcEs{v@LH}eydGN zVXx!0|7)Awo#4RYLkSskm@63IxX8al?EH=jSE|ENS7_^ryFaBz?5H>!UW?E9QiYVE zyQ#U9wdKJpr3Ar=kIlIO)F?Hi(`L^Vq$!03{^zMa77Bv(@WeMQ>#r4)PO>-~3``0k znyhoS30_I2hbRy7CVn)sWQDj=IMAx;!9ot>0L#7Kd<-%|0%l7O7n_)`u$~&+$bTE$ zitEVw_?e9qILO5lf)FhSxzGDTVh^`aH6DWLeg_T!_X?X2!B$ zjtR!N>wRW#&h#G3Aav1#)l_Th^bEy2CRErtZ%0iCWWDTRhU*n!MXG_D-V( zjFr~Z`<6gZfVkS*v-@NrtfR#m?$s1Dw)e3vr_x=7p(lq`a09@I)XUw&;n+VP+OIOU z(v9ORRwoi@uj4n{503j!9OCR7Q{1gfl36yT22WMJd7^+?&Z7;sDweX^Gn8-G8m4Kj z$phK%@+-(tzC@Y?m)g&#t8t9Lx2e?!AJnfDd9Y@d%QdumYY}Ih%ea17(pe2esBn4i z@w5d7veTl_R-Mb*_}>sJMOzcLCy5`l%o4A@hlLb>L?M{p$K8|sP?NF&% zu>J3iOhauYEPI=4^5^f)s(muYRk4&e*_<95K}pBzc92*>8~m_TgtlUUp4UAV^cG*! z!l|~LHEGo8U34(VP5n&L3*bycLPm$L*B4y1Up%%w5KQ7<{`a9Z+zQ0D;>b58d*n zf*e(4sj)WLvc=Jpg@FcT4#fS_J&lnVXT{e@9|cM%aV#FNdF)W~Oi8=GK|gT{C+ffe zMmlyAiXlw-k&$dyHE4j5)w>bQL=iJR^8W_09$;&4G-G_moLG3(A;yQk)v^9RxCJ&a zcGJeOFIDGv_GJeuc0s@KLBc}==o0eE4ww)-;Re#c;o13fls_RXyJPaTG&P1+*?SDi z4tFS~#tFhFnj?723=>Y$^+OKhfakuXl3=o}5=8qX&3bB-Z%Ru~9P`z}mGXV#`a_3S zrVUe_+f#lZRPZ^m4qb`$%5&FPE@c*$&<{P4lqtJf)fYKR>Dsr@y5R_~xOp;KS?F4J z>#ZhW)RjwunbAw-H4a)dis;hn&jrD`J1agwu;^jS9F!Z|m7zkK8zZtds-)O9?8W?A zO48ZL5wx`&plfKma!8|ajNH_t0Q+sH2Fh86qn+iD(A?OnZ$65dKfkNhyYH%3!LhzW z*%SfZn>Vq{XzGmIzRx6e7OhemEIy)~nNURfiN-3-Hk(Q+=#g8HaOln8OQfJ=fiU71B|$fvSf$R!x+E}Di|PI=!y|4z>6NqIxVrv zjPPwNIF31FC`W{6cIXz_a(Xw{$=+?q-X48B>?Dz;Zst<7RHdp(sCfF;#Y#D>`GG~06*{Dyf zp-sTTHzCgCcC0FN;Qnf&yW^#+iZHuJHO`17jlJ8EFKz@12d;!71xZ=;<4bleE}t}k zJ8CNI>u++y*CK!;Pc~B zx%+{f+ugL{Ln&};qFxt2287E-eO?L~0Y`ESLR-a!lIfO`G~Owg+QaF*K7YF#KA={q%bk zBZ-@EPr6&%D8tMgSg~u1w%E(8gtJ?zqd#_+8C-;kwpsp;z60UDPBZqv1{Dbw#^Y0v>~u-0!HbszeZ_erV`|C^?UElM%A711&3)ChRd+pX>a z8297IromSZUAe82llScxzihlUUE8KfPo_MW?tyYS=2|98wdGIhhotaWh0R8zW_Vp@ zVGoaRY!gzzJRwuI$*eI#48uZAFwjX>&?_7lYCbK9=;(Q=Tcq9>AZ0i4rNMU}ltLQt zuDkA!B2P?mh32)ae#@uLK*n&vzJjaG{`*aY)_eLdbh_Mm^|5=rNl8YA5sln%EVZG9e<3-bKb$soasuiW=lsH z*-%!Xij6UA_$(I zkNVcfYk+5Gq}nMsOU#1OCZ(RkRSJR-GV`^%77KJ%1ZEJMi<#b# z7?_-{db4FN?!|-W3}v@1)i7YcTrvE?lc}Z`H5he-#oR;^cm0;SVAV1pW#emR|Vu1IjXmaWPow^s{6cQi30l92Pe-4Dlepyq76R+Drttm0S9~u2n zenss#yQk7YrZEiF*!J#{Y zC4QD5eRN1;V3mEFJS?OXI%q2~)Qbi2e0u5EQqyVJuO0A{`OY;V`J^Y{jRy=E02ginLVX#XxNB(fMWOQ46mnS`CD(KNB!}? z2dlWUfzF`Z_@i_9)~cl<3D3EFVw<6~y|w3I!8ve5or3{eETP<_XRl4EX3(s&wAFBt zt%Y#TUL7S*YK0a(6~wv;%K*2=dA^PJ18gW`jDBQz%hZ(f?< z*+7&hMjKo@wBg9lj?ui+KzbS8l(6F-D|+zuBU$d^Ks;B3(Bns58AOx`&-;rkq{fl) z{iS^2nHqSs#4krBd;!8w{M8!v1dTjH21ew3QYgeA{l54;*c=pG=i>35Tmko z@}_TOerRiAV+=`7w=M9N)Ms(!pR%7&5?;fGLTyVh65pG3A}x9-hPpQ;V^yFEgxd zJu;&|q`;xo>6IMki}T!q&GO(|N#3tXX-45VkQgSBg#o?{G9wgRsSdC1Ca`H>;>j3& zgEAzsh2-AL%OHiL0y&>21MQ_TzvR~xVA8#t8*%oOD9r9fu(|+%!&uduYckXiX85!c z@oKaK)x>vp$X`=@zC5^?KCj=hUcgQ%rN)210OToGrM9Z^mbW2!UDD$j_RFe39;XNU zQ^WP1tWf-4z#fZqpbkcB*_@D9K?VG4?4n#ca=DTe-tpOZ*LaZU!jkAz%dhZSW2i6^ zuDZFo>EE&4JY$aaE!pn@y2=PPE<2>P_>w_vAn5f2v$cEL2dMV?O5=u!`u!%hznj^$ zUMAO(KLHYs(Tg8bT|unsNbo@X1PbW*exbH7`Ii$AYY(UZx55tR%yEquNK5NTb#%VT z0Mcad|Na*^hrlX9hl%JRjTB?-GYf)gkxz`xgR^rbf{WY-{Gp$DuO=t30Q{BW0@KZC z&NO417^f&Vp?J}G1EepYd}EDl<-90kdV!SV60flQiYE9Z^SMIm$(-`}T0*_Hd!P{> zZDnNcG$i@i#F!Rvuw+w0@uQ!_5~ZNR(!dB=0R>Px0>#wEoK)XY%>WbS0Qs)~-W$~C zQP(Ju;U7_I7k=0Q0I)4;|B0wGr=rd}ohF@yAHWXl9X6KM4wgF{|Jh;dWNYJOz0< Date: Thu, 14 May 2026 20:33:09 -0700 Subject: [PATCH 14/65] fix(macro): subscribe to route paramMap so in-tab navigations reload properly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Angular reuses WorkspaceComponent across navigations between /workflow/:id and /workflow/:id/macro/:macroId, so route.snapshot.params is frozen at construction time and the macro drill-down didn't actually re-run its loader when the user double-clicked a macro node — the page only loaded correctly on a hard refresh. Subscribe to route.paramMap inside registerLoadOperatorMetadata and dispatch on every change (deduplicated by id/macroId key). The workflow branch also re-enables the persist flag, since the macro drill-down disables it. --- .../component/workspace.component.ts | 55 ++++++++++++------- 1 file changed, 36 insertions(+), 19 deletions(-) diff --git a/frontend/src/app/workspace/component/workspace.component.ts b/frontend/src/app/workspace/component/workspace.component.ts index 898b1f8050f..4a350ca1cd7 100644 --- a/frontend/src/app/workspace/component/workspace.component.ts +++ b/frontend/src/app/workspace/component/workspace.component.ts @@ -318,25 +318,42 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { } registerLoadOperatorMetadata() { - const macroId = this.route.snapshot.params.macroId; - const wid = this.route.snapshot.params.id; - // /workflow/:id/macro/:macroId — load the macro body into the same canvas - // as a "drill-down" view. Read-only in v1 (no auto-persist back to the - // macro endpoint yet; that's the next slice). - if (macroId) { - this.isLoading = true; - this.workflowActionService.disableWorkflowModification(); - this.workflowPersistService.setWorkflowPersistFlag(false); - this.loadMacroWithId(Number(macroId)); - return; - } - // load workflow with wid if presented in the URL - if (wid) { - // show loading spinner right away while waiting for workflow to load - this.isLoading = true; - // temporarily disable modification to prevent editing an empty workflow before real data is loaded - this.workflowActionService.disableWorkflowModification(); - this.loadWorkflowWithId(Number(wid)); + // Angular reuses `WorkspaceComponent` across in-tab navigations between + // /workflow/:id and /workflow/:id/macro/:macroId. `route.snapshot.params` + // is frozen at component construction, so we subscribe to paramMap and + // re-route to the appropriate loader (workflow vs. macro drill-down) on + // every change. Each branch also resets the canvas first because + // `reloadWorkflow` would otherwise hit duplicate-link rejections in + // shared-model-change-handler from the previous view's leftovers. + let lastLoadedKey: string | null = null; + this.route.paramMap.pipe(untilDestroyed(this)).subscribe(params => { + const macroId = params.get("macroId"); + const wid = params.get("id"); + const key = macroId ? `macro:${macroId}` : wid ? `wid:${wid}` : "none"; + if (key === lastLoadedKey) return; + lastLoadedKey = key; + + if (macroId) { + this.isLoading = true; + this.workflowActionService.disableWorkflowModification(); + this.workflowPersistService.setWorkflowPersistFlag(false); + this.loadMacroWithId(Number(macroId)); + return; + } + if (wid) { + this.isLoading = true; + this.workflowActionService.disableWorkflowModification(); + // Re-enable persist in case we just came back from a macro drill-down, + // which disables it. + this.workflowPersistService.setWorkflowPersistFlag(true); + this.loadWorkflowWithId(Number(wid)); + return; + } + }); + + // Falls through to the empty-workflow init path below when no wid is + // present in the URL. + if (this.route.snapshot.params.id || this.route.snapshot.params.macroId) { return; } From f7748b80409e0181cf70fd9d6261be5bf70ce89d Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 20:34:55 -0700 Subject: [PATCH 15/65] fix(macro): use window.location.href for macro drill-down navigation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In-tab Angular router navigation between /workflow/:id and /workflow/:id/macro/:macroId reuses WorkspaceComponent. Despite resetAsNewWorkflow() + setNewSharedModel() + paramMap-driven reload, the YJS shared-model + JointJS paper retain enough cross-route state that the macro body's links are rejected by shared-model-change-handler as duplicates of the parent workflow's links — and the body never finishes rendering. A full page refresh on the same URL works because the component is bootstrapped fresh. Use window.location.href to force that full reload instead. Brief flash, but the macro view renders predictably every time. Tearing down the shared-model lifecycle properly to support SPA navigation is a follow-up. --- .../workflow-editor.component.ts | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts index 468734ccb77..3beb23f39ba 100644 --- a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts @@ -570,16 +570,21 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy if (this.workflowActionService.getTexeraGraph().hasCommentBox(elementID)) { this.openCommentBox(elementID); } else if (this.workflowActionService.getTexeraGraph().hasOperator(elementID)) { - // Macro nodes drill down into their body in a new editor route - // (`/workflow/:id/macro/:macroId`) instead of opening a result - // panel — that panel doesn't apply to a composite operator. + // Macro nodes drill down into their body via a route change. Use a + // hard navigation (window.location.href) instead of the Angular + // router so the WorkspaceComponent is destroyed and re-bootstrapped: + // the YJS shared-model + JointJS canvas retain enough cross-route + // state that an SPA navigation leaves stale operators/links behind + // (manifested as "duplicate link found" rejections and a + // half-rendered body). Page-reload is the only known-clean path + // until that lifecycle is properly untangled. Cost is a brief flash; + // worth it for predictable rendering. const op = this.workflowActionService.getTexeraGraph().getOperator(elementID); const macroId = op?.operatorProperties?.["macroId"]; if (op?.operatorType === "Macro" && macroId) { const parentWid = this.route.snapshot.params.id ?? ""; - this.router.navigateByUrl( - `/dashboard/user/workflow/${parentWid}/macro/${macroId}` - ); + window.location.href = + `/dashboard/user/workflow/${parentWid}/macro/${macroId}`; return; } this.workflowActionService.openResultPanel(); From 2612cfe408ac12d6bfd21f406c48ab2f08484715 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 20:54:52 -0700 Subject: [PATCH 16/65] fix(macro): make MacroBody parse cleanly so compile doesn't silently fail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Workflows containing a Macro instance failed to compile (no execution possible) because: - DbMacroRegistry.fetch read \`workflow.content\` and called mapper.readValue(content, classOf[MacroBody]). - Marker operators (MacroInput / MacroOutput) inside the body had been serialized with their ports in backend PortIdentity shape (\`{id: {id: 0, internal: false}, displayName: ""}\`). - LogicalOp inherits \`inputPorts: List[PortDescription]\` from PortDescriptor, so Jackson tried to parse those entries as PortDescription, choked on the missing \`portID\` field, and threw. - DbMacroRegistry's catch swallowed the exception and returned None, and MacroExpander threw "not found in registry" — surfacing as a generic compile failure on the parent workflow with no usable error message. Two-pronged fix: 1. \`@JsonIgnoreProperties(Array("inputPorts", "outputPorts"))\` on MacroInputOp / MacroOutputOp so already-persisted macros keep working — the marker's port wiring is derived from \`portIndex\` via operatorInfo anyway, so ignoring the JSON entries is correct. 2. Frontend marker serialization now emits proper PortDescription shape (portID/displayName/disallowMultiInputs/isDynamicPort) for newly-created macros, keeping the wire format consistent with the rest of the system. --- .../amber/operator/macroOp/MacroInputOp.scala | 8 +++++++- .../operator/macroOp/MacroOutputOp.scala | 7 ++++++- frontend/project/build.properties | 1 + .../workspace/service/macro/macro.service.ts | 19 +++++++++++++++++-- 4 files changed, 31 insertions(+), 4 deletions(-) create mode 100644 frontend/project/build.properties diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroInputOp.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroInputOp.scala index 8bc4627c233..5bdaa7ed563 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroInputOp.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroInputOp.scala @@ -19,7 +19,7 @@ package org.apache.texera.amber.operator.macroOp -import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription} +import com.fasterxml.jackson.annotation.{JsonIgnoreProperties, JsonProperty, JsonPropertyDescription} import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity} import org.apache.texera.amber.core.workflow.{OutputPort, PhysicalPlan, PortIdentity} @@ -30,6 +30,12 @@ import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, Operat // `portIndex` of the macro: tuples coming into the macro at that port flow out of this // marker into the inner subgraph. MacroExpander consumes these markers when splicing // the body into the parent plan and drops them from the expanded plan. +// +// Ignore `inputPorts` / `outputPorts` on the wire: the marker's ports are always +// derived from `portIndex` via `operatorInfo`, and earlier macro bodies were +// persisted with backend-shaped `PortIdentity` entries that don't match +// `PortDescription` (which would otherwise break MacroBody deserialization). +@JsonIgnoreProperties(Array("inputPorts", "outputPorts")) class MacroInputOp extends LogicalOp { @JsonProperty(value = "portIndex", required = true) diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOutputOp.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOutputOp.scala index c2492afe911..dc3ffd8b45e 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOutputOp.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOutputOp.scala @@ -19,7 +19,7 @@ package org.apache.texera.amber.operator.macroOp -import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription} +import com.fasterxml.jackson.annotation.{JsonIgnoreProperties, JsonProperty, JsonPropertyDescription} import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity} import org.apache.texera.amber.core.workflow.{InputPort, PhysicalPlan, PortIdentity} @@ -30,6 +30,11 @@ import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, Operat // `portIndex` of the macro: tuples flowing into this marker are emitted out of that // external port. MacroExpander consumes these markers when splicing the body into the // parent plan and drops them from the expanded plan. +// +// Ignore `inputPorts` / `outputPorts` on the wire: see MacroInputOp for the +// rationale (operatorInfo derives the marker's port from `portIndex`; the +// PortDescription/PortIdentity mismatch would otherwise break MacroBody parsing). +@JsonIgnoreProperties(Array("inputPorts", "outputPorts")) class MacroOutputOp extends LogicalOp { @JsonProperty(value = "portIndex", required = true) diff --git a/frontend/project/build.properties b/frontend/project/build.properties new file mode 100644 index 00000000000..10fd9eee04a --- /dev/null +++ b/frontend/project/build.properties @@ -0,0 +1 @@ +sbt.version=1.5.5 diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index 1e2ef4d65b9..0c29c29632f 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -214,6 +214,11 @@ export class MacroService { }; }); + // Marker ports follow the backend's `PortDescription` shape (portID string, + // disallowMultiInputs/isDynamicPort flags) so MacroBody parses cleanly when + // DbMacroRegistry deserializes `workflow.content`. The actual port wiring + // is derived from `portIndex` server-side via `operatorInfo`; these entries + // exist purely to keep Jackson happy. const markerOps: unknown[] = [ ...inputMarkers.map(m => ({ operatorID: m.markerOpId, @@ -222,7 +227,9 @@ export class MacroService { portIndex: m.portIndex, displayName: "", inputPorts: [], - outputPorts: [{ id: { id: 0, internal: false }, displayName: "" }], + outputPorts: [ + { portID: "output-0", displayName: "", disallowMultiInputs: false, isDynamicPort: false }, + ], })), ...outputMarkers.map(m => ({ operatorID: m.markerOpId, @@ -230,7 +237,15 @@ export class MacroService { operatorVersion: "", portIndex: m.portIndex, displayName: "", - inputPorts: [{ id: { id: 0, internal: false }, displayName: "" }], + inputPorts: [ + { + portID: "input-0", + displayName: "", + disallowMultiInputs: false, + isDynamicPort: false, + dependencies: [], + }, + ], outputPorts: [], })), ]; From 7a051207ddb1de6f1f4804bf49ab7ab15e161ae9 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 20:55:06 -0700 Subject: [PATCH 17/65] chore: remove stray frontend/project/build.properties from prior commit --- frontend/project/build.properties | 1 - 1 file changed, 1 deletion(-) delete mode 100644 frontend/project/build.properties diff --git a/frontend/project/build.properties b/frontend/project/build.properties deleted file mode 100644 index 10fd9eee04a..00000000000 --- a/frontend/project/build.properties +++ /dev/null @@ -1 +0,0 @@ -sbt.version=1.5.5 From 4c895f0147f413530ec9b4ea2eadeb98afd24d51 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Thu, 14 May 2026 21:38:45 -0700 Subject: [PATCH 18/65] chore(macro): replace PythonUDFV2-stub icons with distinct macro icons MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The earlier "just so it renders" stub copied PythonUDFV2.png as Macro.png / MacroInput.png / MacroOutput.png, which made macro instances on the canvas indistinguishable from Python UDF ops — exactly the confusion the user just flagged. Generate proper icons (rounded "container" frame + a three-node mini-graph for Macro; left- and right-facing arrows for the markers) in a blue/teal accent that contrasts with the existing Python-yellow. Pure cosmetic, no behavioral change. --- frontend/src/assets/operator_images/Macro.png | Bin 21524 -> 8425 bytes .../src/assets/operator_images/MacroInput.png | Bin 21524 -> 4269 bytes .../assets/operator_images/MacroOutput.png | Bin 21524 -> 5381 bytes 3 files changed, 0 insertions(+), 0 deletions(-) diff --git a/frontend/src/assets/operator_images/Macro.png b/frontend/src/assets/operator_images/Macro.png index 209abf48cbf24df5412a8743a400586756965b89..d9f5b30be1831aa44546834eb749db2896ede35f 100644 GIT binary patch literal 8425 zcmeHsXIN8PyX^`^Ksu;M5d@SXRp|nWA{%t0pkOGQ7DQ}xkWNT!5S6Aus z9RvVC@X|$dTL9nyn;Zb|0q|=h{M!xy9Iv=!e*SuR*777jQ_tB|VwLKAN=4m zTtb(G@rCe^sKQ{b5;$4!`C{gD9_%4PYoD2*rnUeBS2+ zq6+@zF*No-z_`$+gR7i4(zIFrdog!!sNU|+iuZV=-Fa?H=pR=`B)V_@ws#X`~vpo@&zoA zyxhx;$T)S)_{IDYObx1N+}{C0{3E+5T2(4l@36U`X@wVUj#9rjCI=)ylL^}rM&1WG zoj+~Mq6?2`JL~`s*Bs=g-gDx`_)}u#rf<@g+sAzX$G_x9_YK@fcI{bC1I@kccBV@i(< z_V;zlpalifzKU^S^ zM-E0o>?HxAKjwsg&zvAPTJOC&P+R_&)9$B6AV9@d5wHJoK+75c(2wm8141J$3K|?f zz4o-bSEKkS1X`BkRHefQ9ysXB$bsiDsdpq%IKUC;vfXkx#>FN zPBZ0xK)EV(RcP!9P+>;+xN5T_@DRV&`pDB(DAC3U;lP1^raluFBQJ)(ELGEZjyn-(MzRz;A?{c`1mp9edn@}Cf zfq&@3AXJXbeNQjVDKC{B#rZIzeHrz{i-04hl&w-56CFIh6Y+g#?W@-!_3k2}?#Mr= zQ9qZan?*4UOQr1HH0pQtGJ-iJ26uCz6?-=ozl1h>dKqhtCx(^yP*0N9G!uXTb0>Zj zaSnQkNPoeow|U8kzG?dFq{zRJJAJY0!_*Q1WxYvHW-$l4V`Qu3*p<8>IV1xCTtj8_ zJQ=3xN8xYJ2}=J$E&f5nP8c+Wvz4FmU?-4|3ji)&{LMIAXwD(lO0x=2j2w-V@x%6a z4}%oKK5+HtvGYdg_90FH;QP%vV7JE`>mlFLtw1jM@6dqEMY3M{Fb|dNO=lQe?0gn0sPz`%C16 za(5)ON_GqE%rdQX^|tX;WmR^%55d^Jmi}@%Jlz6tM61IdsuLQD)v3z8fxk}mZg&S| zS^Y;|&3sG+{aMWUDhB-%V=l9oI5131^4H)(bC1)KZWI%0w46=dolVP-nT{p+1TINIHyJ&IC4h~EI1hgSdM$X|E|c&lN@-nQWl|VbD;H*H!R7U zy=PUqNv{Vl;lJS-u#@y2OHvyR%Gn#-3bRgqG?BtN|DFfIO&t1IMhPw(`Ky2BFDUh| zQpz59c}oc>=@>#?wqi#+S8mXoHwNksTfqjv%jhh5^kJtP5_xhvYyAyD$(ccTCq6*! z(_=YYESs6anZns`1~^uF>H<>xlxzP1Xb9DyXP1{#iWTx3;;+qsvDeS_h<0WVx?-e% zzhbQjoXERoX><;We=>1^_djdt-$nJm4G>xX&m_S$W=R;3v4+>7PAcGZ6>ujMpvnqR zV+E*&LYai<=u?caMF)?g~)YHk((d!-}(w z?P2#5W_DFktvb<`{)w2;^p&EO3^L=`=`Qt(0bN)&Osei*LMoq!5c5awV z4EuV;GYQ++a>;xz)Z9gXU3H7rG=KYs2ewgzy5Oxi`&l&bxt(mx25aCh^)^X7YQ=FP!(46X3n6rA^; zo{@*(lT#)NvQnbL&oBehYblGuCKjy9{?3%^X;;?W?=&#)xWRD_URPMjzI~xR=&KHHqDEoXUkK%2O4}qU&T=D1Q!GM@ zG^J_vy>U|s(&pxTAt*4hNOKb{g`B^`uJ9d^*dyMP*t6unOx}4!$gG-hoQMM88(C3N$ajq`*T8;NsQxyla;KxI?&6}Ywj;~cU_wpL2Iz4B zP4xtU56gaPhFxdepq0+=~^QZ^$AN#hllJw z4Q>sG*M&XB43SpDW&>J|Ct)xKpw(L8Y*AK|2Leipw+eaCmcN+Imhcc`uQ7x^d@=@Z zzGrK`H^PZqR<`W0!ZxzrkEQKebo3TCYFv=3{z%)rCb~w$7+Ppud-HSqna9rq``34X zt0T)V3EJtj`F5^`?CT*>y%?41Y>A@2Kn!MgnYocUplg2S61>jqCHZ+QUo_#PWlRB< zRQ`OKdr~fI?g==^JSKep1=Tebu~UHc%MC3_Zq$XZ@4%42v`PzL)w-jlQ&oVOsUqjJ*ENR^D*6^Al|zj8mLV>M0EnUBhDiS~V1Keujyvw{&7dj>OW{n%bW!^z5( zj4H%v=icYlEppF87<_V{HA6QIoXu9FOqkSRNhEM+d-MrO2)bH`bHjOIznoU<;$eF8 zefsXi5Q`M{6_sfEkyhKcftq;yA!TG$b~x;-P8Sp;l%T^qOIW|&SqrV_7%i#oZ$YU$ zWBGA2dHv3>f4-xZayUxWZ3*>Cr#qj>bsO<*h}^H7!A8cHtIXV3YO`!_e!yeRVixW( zTld`lovz<` z22P)0%|DN>G|exD@wz3+#j4pBy(B}f@&Z`}K4;cz?W_l~iyM!b;+#g7YvN8*ts>}& zo eHgVEEW5dyYN@nqnu|M03_*}~%GlXp^t-;_@Udjeq-L&+gmozFsiRGzqH@> zGM*Ve^tePi;~5oyJ0aTTO~F=3)WK7ps3xt2ruXWks9VUY966sx8kXk&eSVzweitDTU<*D4}QD)|p z9q3cRjHidjOqeiO1U?|E=5)>(;b}iB0RZ$o!EL@#1F~hr9N79=olGb)lb-ge)nvWW zi!`MwI8n4ywv7*1oaszy`PqB5eM^9@>ZfNC)BRpO=X&gUYkD$Y`8K&%eQ)d)SmgSF zf)DjJ#4wzegpxD=7yDwF#8avZ@=<>nXZSHCJCzj z>bzQ+7WIi>eL#J#Iar0XmdKc^l+#MW?4~53V_VM{t%4J_m+P2&hCim{^cIO6+_q>+ z0Z6K_ka|VW_^8y7v)~L-vbgMrIRh|jhbGO8zj87kzwo4n$%-15J2HQ?Itc2_T17+; zKek{!fKT>^gniCd?%UJS>AgJ~_El}Stl<(_PxxokeA7-@#I}?)+v6HmwT{*j(Xlug z)5hA@N;CymoVr-Sw_VIn`EhrLg{}>3sU2mz!vO??4y1i&q2A5h3r+7v4P&PS4ZfEmh$MI1M1Ok=oW5t zp*5ghy#e$1;j_$tJ3jr1!~W{kSCHbymbJ&FH?2lZBw$Iojw8!q79C+WhUeLV9iBKu zS5A%56{q-n!;o{}&e%KvVNq+R}tg^qOSgykm?)4erTjeD=4nZ7=Azt#gg~iB#QcNPpr7^}aJL3td8K z6I0vQn?I;7>cs1LT`R0^9IF}#?t6-$fZI7e^*!=pE)hvJ@iQ`Cykl#+Y=XuEp0HP} znOjYX^_{ZGWAg!g zi}qzSVpTGokSzviewS{)>4l+YT0(0#wBtj5j%X6Bhq4+YMAejlEWG32C3N730VKdN zZ~nYa>E4#VCC)lxpo;Hn+54q04*IP#OR`RWamTjDDrbJRSia74BHlSV*VwKz}I!sDHYd$}Kd=qALmV`5Gb|JowEXgl}A10Cg4vCHlRsba-e z_~<_CQ*uDR1Slr{;NbujY|iH8bkox-#G6sty1QW*4|oW|!6$Jn!`& zeT?NE8qze^C(XX>6cjS*hE{#`x~yg{>Zbj#b!PsMC;W)<_nB=^gW(3BnGhDX>_)mR zYD=wY$K4k&)Rjlv7sgMK8|;>CZ=V(hYl%TYS>XIy=9G-=x)KZw z5}PbrWkuN)muEp}IPa86Hr2M5IgAsSemxp5JbiCKR2iGGoOr2WVg^XG^P)QG<+Uky z;B~nMH|*EW6WCx`N4Fhm@hso}u*s3qeM2~L;r;bKgRvP#wMx69%~Q;-Wk&*t%qwI) zHg+YWQk>Gi!%7L^6I$LG2mdca%{R$j8W6gYF|iPeKk3FI-~OSX?IkuwSHf-bE6hHM zk~wg+TJr0*NBna&e(uTg%UzifYExU*ukXTcanQ25G^022HPq!xan>T>-e=<>Y83gs;8ltJ_-kwM(yB_63DEylw!Lb81P0 z*~qsz?;HblQ&yaK+!MEkxsJZSN!;$!p@D`d({@=g!pHR8RCj?gC?I1+6wwC95Nl%h zDrsBUD&0bU9pe%9to9sDZUuSBPATTfS|^1OM|V2X6JZ0dgMlK_Q>*sqw|uZL)I`@t z#7E8Q?VRZOk27_e87YD>U42P!db5>HLQ|VADwQP-+ypnV9aYuu=1Ut3Bd$U6up}+# zJ9W%&-RM@s^}xapb8eM#sKtkD_taD5lij=ni`yi(Dt? zPgY_{&eq|3YDy-X>bpWN-v;ww(H+dqI-%V`Ii35g=f`@+=D>R1()1^WU2)=>mau^C zRq@rm%jtAg^0B=hdu89MgOVPMTs3$)@P#8_%A#HA&AxERt6=d}TJqv7;ev3cpU(8yzN1Lad z>&P-LeNrRLGM+&LMa+*$xZ$l9lK(Uf_4^><$?0n%M|Fe|mmJu_Zz`_!o{nA~J2nO; zqDsyhS+V#7>S%4C&<1Qr|7t=N&qHD*%z=H`(`>>fk zYu7n;T-?mNqW!Or32APrxh1sCuG1?f2~?Xb)Y=5jyQ}QRn_9PJi9uZ7-BGovYo5G>d(oRID++E=JTX0hEtx*otYSxO-8_rjd!5W z>ke@EtO6!D+>l@=wJ~J30s1DZ>Byn6sXUL@vz6^K31>E8oz8J?iJgwNEy(^@+OiA; z@l+n+d45_wfq9sJai9GKcdKN}mEMK*BWnlUof5O}>8QRox#6|@9kk>6x19Uh zkvgI7k5zNPjT5a-GmvwLpv%<`OhYX|m|Ym=i(@_ReA2mR)*?{Y* zY4@rwf=NnkXA1TDl7o30nw6|x}H*k*V!jTcGPhMHSF;~SmtreFoqAfq`Zfx-@ zvJ;Jo0v9>G5^IC5Vo*&E$il28VPrxEK#5Y>%1*CbKFO>>P7(FreXz+YayMzfk}92P z`$Zt`P79pB;`EZ4W$T~%G?OeRT7E3XTTX@pzamWCF}mH{$smDRyut8o;piil87Zm3 z2MUyY9zt>H3~(BL8)3IAr>r1D*tx@hbd342n!r5}x~kSx*jGEH{y}ow&^c3tU{v>( zYcM4s1WKt+D`uh^+M`m@oAr)4REBFX=!g!FhGkYS$fHjRb7%c17-$$g5!opuo>*o$ zxI^`#ww|7X-OpdSvfv6@%KMbWhVtgP^HNQCf2JFJH`-Ilfsip}@o-Tf6_FO}tY7)So>vFC1XSErw<- zitFw#!~#{(MZnT~n{U%3(&Rrsz!K$<6M`PB-(0Gyp~^;^Bhr28VxJA`7T=^ z2)t>%4?s18jg!v;H`(iuGmEcA*+s(pW`5#aW4I+jw$VZq(Es5o;FB4-54tz*->1j6 zt?t==$yVAYfQlHY{pX-{;5#UYZ!99%fHKi+2I8p9*eaCdc#Tg6@auP(1W_RLI4_`{ z6pNCAObc*9&~wA9D;B`(2giYcaH;)jQQ+1IUSKIV7A4Dpm*wRE=)BILGS{JY~ze^{HBnIRtjA6$;@wg3PC literal 21524 zcmd>lg1rflTk z2q>t4q{mNs7h4@D5@==Q>WMYWcqavI8u&V#>vn&-7DS&V70b@mJ0%O5S=)593M<;k zB~hY-Z8+bE0_2YFw=Mz#KObN5+B0hWs%N&L|C7Uc>dBpoRgBoXyyoCDm?=%FZma*w z$X%))S@+h^9zi~K$-p?J=OGK=wkNKq0_ z^)$gjY-sh&P%o4B$%k+oA*Q@#J22n`* zH8rg?d|E`iQ)0q;{;NqdVr~P}9m2y2ln; zA&EHoX+g$N z_cn6@U8#p&b&t26<7<0Ot6mvu+ejTCYRcy@7X^6>6Xs+FIko`%R@ViYivUSog9Ygl zKcMl)skL77l&A_PzIl-dCYuSxKSe(l@oQp*B6~==^1|A&1o4g-rg}e?K_3y-PV#i~ zgQW)=LWqS6FdhkCTtzS>YR{7ulM(;%!cykHqgfN#y7AqSMjs`YyIK=GBpX<;_nZBQ z3JQ>zb8fB7mbH@fR*E9ZfJxh1MJ=lh33&+^cnO%oh_DD>LxTQRn!LZ!Jrg6p z07Z+)H#uvO6O7zw28I|moUlWEf2rRad~utj4nahaB#}ztD`gTgb=!Ni_`VF=vt>VG z1UtGNc87}$^6E$_v-Rvl7SAn*9)y>GtU^smxgOPygE>E1Ci0xtTjd6|cr4r$ zu(nQOh`t|>6M!9Rr%tYC=DbCgs_a98;8L44#%v$m*&j8ROE9JfUcMwZpc}UEx|7`! z1!<}@Oo@5cA2BE%oXwz}BWS~J!wB_50$bgn-m9{_UDBj9^0)xMf04POqJ;NWY5;Mu z&Q(B#Q0Qq=xWxW)k*-RFir>V$`*5vrY+rYCe(%kdO4h6?bLj*oV!&uZIOALG1DE$~ zPeR2n16&`tRA*)RPl_pIOE{lm>{R8M9+w>NweD{_-9Y6r)JsSmq^?wkp0kqOS{W7b9;ML#J%G3zNO+13bBkS zc*`C&dBP_hqgj)0B8u!ejTX6W^dlq2xhk)3PHf)&GQFLg?o+fS`ttk1#m4}?0Y8zG z$+!7QvXUX^iZghd0ykw z+>jM|fS-_V(a?k6YB=oia%RwCW=;IhnJ~2AXttUPE z(%`$>Npo|?wHejLkJeT}o0WGZ$uci&a7dY-gjOe0Bxz)np-%I|e@#(K?dV$d#Yzdz z%!m4Z`AVT`^6Sk)Kr**{1ibq00RS6XN@_FeoxxJV*+77wXJa?BS#i(1o{Jj}A&BP|%Li5T!^a;huJ9EIuz2bU1w`NLn z(^Tov&b9A7+dO&uHRNp2qw?65hOCU(t;fOL-FvP&zqc_K@4KIMH`uZ__Xn6NDp@=9 zYPQBj;Tq&v69F{oe_n2qA+xT<>tj;mcrb{PgS@y$C%da;Kpubl`2`P|k{vvZpya%^ zNYHM_6%7d`aFn}tLDx}u_2_Z(^L|1bi;unAI?uf2TGu}G&VN{SbI)EXJoz47?D zj7RMii9y?&g);NSui6J_kG;1(se8W=vbI2WXd>md8E5g%^)s`v&@96%J!aaj`x-I* zb7DcK8>93V-Fz|;{1of*bC0|tDQ=Me=dIV9=>=|e5nM93acAqXR|S|%>mPm?eLR~e#pZ&2db zekEMaOYcga_$`X8#Q|67I!^EDGnoWk(7wopkpUNnQ*MRLY;M1VZDsDs{*>%a3l4=jKfNvM$^Z?VsD!N13KnF<48W1)^0Ji5K&L zI+q{Vk@hfED4(0m-JOx1sDCvly*aumZ_&Jr^1|D$cpn-Ac3v~KMGy4Oghppz7BF2* zuj*K?+D|S|ANkukha|iF(SA`5s-W7|4JesL-irf9>}!$&{aq%#I}JZK?Mr9&FAYx4 z)iH~2CimB4rQ*g;m=obJfdUse-zVRgm;Pe8TTP#7iIm*G%4o17`eiF7)br54pho=p z2bul1oaXd^rU*CHz4}#`m8`wa`a{%!WbbM<4yVg(>hWWe&Z+S&Wq*@dR{-a8b|WpJ z6fQ8PQ&R2tXb{6vzw6o+R#`bXcYAxX#key+UU-tKRawgofBf5XTKPYD`HHP&aWT8- z*5zxe@5evZp8cq6JG=QhgJklNZ*3ZA;qy(Wj6mvB0a$PX>#Wi&v=(e_p|rxv^$A1EKLxiGloNZpyJA82#Gvf#K~% zr}lD1g(&W{E%~{g4__M(*Ax}slQe!oK5$J*+J4|UCTOek%!h>9`QLj5)fhHF#XcFn zbvc|}A|L6tAkTKzR5doq3~?OjPQNHSsB3LTL`=b^E%~82V}xIObp1Dj_Ng43FFD6% zZpZ-rex>hCjGOh{!f{37QhQ$-nB!XNNIwOy;s*x)kw{XDvN`sY z?31x7-=BT<@tXx65Ep#i!TL`4;b{yE*-Rk+6*e*Q1tsJ`w_IP8vb3st*Xqw-u~ac@ znY+Xy8fqoth)SW%_MvX?a`Y_1M`~g@W>W6T^}asgXlVkHEKNe--I>CFrt$=zEkvC+ zRC5hK-_*aAL=+U@!rtug`rg?$Ty=hpBT|4L@sD(lT}!}R9Pp<5%1R=>g)ujIQA^D| zndikOOaoW(+c$I<1L7@kju?uQLTU^?=BV6~{CxTX-#+gyRC`n5IX^be(L&MEKP3n^ z#q~dPB*kQ|=-}(;i1v3s48J$Ktsbl5zi%iCI_+LI5qZV*KI}hTkD=dxfxXb!lEE2b zw-g?5<~Gv<2&6$oAiqoe{E@C+?wXC89i z4miaJ6!9mNAfHBYZRp2qa)UMy;xGaQ#P0@5$%V_t%n2zHczi#rz+lu$AWDM4Am9%3 z|Hc;%G)}yh z979)MQ{H=^gLUmobFURh7*?8)K%ac?q6G+R9CWY4_9*8KXNcRLYiI8h+o51=Ena_omp57{J-NZWS&iRqL<7KEMFc zljd=3^Z;*f#B*RqhcP?4wM{CxEEY58ik{RTrUwW|l7r2SrYQMtT5=|QVEQGEtcw3J zM*~&HL`n06a)Cr<*o0(mP(e{TbmN<+PWhf0p_i>WK}S5f9F>H*LAc2l0#TKz1&#Du zAA|s$m8JmWL44Icef^^C3hpe>k5i_F)50r~nnHM6a!Ff#lJ_1XAWDi9UWYqh_OZ_I z!zrcHYG78gvmy{@Gj_l^VW+~AO_0Lt^ylm8ZQ&fe+40bO>HEnEH-z?V;^;kg(r~ul zGXX(G)aNDn6i~SO`eNgo^pfbSq~6azniOn_#!_OUg?gX0gD_XEP!y^ZOl|cJpc#q? zWrl_%ezJOIso(Q+Hs=Zv`oR{E_;H?@T=YK4RBGm9a5rt|810u8hFY$Bh$5GKKTyFY z3Y0GNDRFNt@c9KQ7HUKCMtnB!KF%|KracLuGQAOmV< zXgtriJ+x6`(z#-`yKC?;I$%X_&iNM)WBUgWVw6>J8#V+u;^GSASSRi!^M$)UuQ!?-^%M{AnsY~z(M;9d^2xJ7p}VIYowepGv|Rl^dwl& zzWPTN0!yU^iZeH|j>gK>Rk?eO_^B8ZHu<(4Si+-_yz^L=DOcgwoZ-Ftn7bAyR0jI9 zj_l~2TEf434~QkT!T2PNLvf&S<(ur(|T0p-_z^i!i7ZemECN*w*@P(`+sGq}B0aAeF6+Q64b2+0- z*C@Vmw1cD)OWEvRm|cX&b8?E^9^Jo^!b-}~t^J{uO_n~NRSBSQjiFYjHGFYFvfTGH zH`t+5@4~6Ct|Afs4n-NlA(^U5d@56Sj!y0GczwEWBntR)*)e`on@HM=^GUxJmjY)6 zc9ZyUpF>HMV0i?}Oua#Z3y?hzbQ|;mEXGgS)?j^h*lJ@%a>?b|&oL<`NXPlb*^Ru4cjzoO1r#zf*hIJv{WDWR)u|3Mk z)0@JTsbawDM*HkQ5fkC!_N_1Ifao2s>cLhS1V)cvg@$ggqD{m-KN*+P18?%lk>;>u zuOLBNqF*1=SKlLgJ)RQB-zZq>uyq3Zs!(S?@Nn#1&oCb_T5AxuJ3pchhYiLa~(S*69O}2O9QQpbhl@L${djWd@fBr5gBZ6h;jc5I0CD@jxS|*}Qt+>BB>PYo*xOOh z2k6G>4}oqn`_be)P^gFW04|69XGLn|C+j`GX7}}ANe)Bx1fdg>&v}c3iGdqU&LA9E%|sd zI*g2c0!5Zw{Iw9s+Ppoal*Jqz=Ck*sM=zxpAKP7HsNLK`u~Ol|^|P8x_+7 z{_eBs$%cT}z4+a$;e#zu?w6i>xS|s#H5=EjK*>3A;jH;C>2W)Ycatd&>Vr9tzEJjU zR9N>?1}yHtLwR>T3~vdSkc$C-chY0s5}Pi0IN5rGbY>HZ%5{Sm;*m-|_rnp8?6l*3 z6KO{6419pW?EAb3#YH>zSB?A%ftXrK_|9 zBaxK%ae&9P;FhymIhrHChIGQmh3qtYV5L_yegO&M|ATy@2SI@5>jO553+Ffa-l=hEz=97{6@9zNLR{N15^AV9ktfSM>`>?W8D}ZP1yy=9ZLZF-B&QGK zWBLd>{=Iqy)=&*5H-3OO>gZUVsTydb)+P7}zz6m$B^k8Ugm3}QqsSa;_4jFZSW*yx zUmGiUZ?xw#SvGc5tEjFI9}-6a$WJ^C!Gb_R$qOW$m%?&BB~vny6VVi;<7hHD5Cd?{ zrA7!A8_=n5D02bMdzS>Yx`x@Bz925Z+b0Ta+?BHWW$5`S$7~4-&xwBhZ{f5iX%GkY z23j!8mb?_u!HH-S2EKM;3EDR8-SZ?01gbb}1&1&?FaTbPHx#ZuiXZj%U;xmJC&}1-fqB%uJzw4 z#NHUbI~Kf{A8o%udk$F5m4+69GyK7kH%Ws=fySTQAku-22KJEY$(r-62r(84++4$5 zL&$&>&G0=&#LJiZgvE$2GIy@Iy8BYzwpM+l7k_}&HHB&C@Lp29g0cb5%1evbu+>Z~ zeFn+bDyu@w+-{ziF)>xv)fcTJ2%7WZTTsnv?$SnFPX3@Z1FHA1sg(}jnr)GC?CzN8 zqXkSm)&I~iQI45IDQFsM5Rqr$(n}w^7p+mh&WPl~605BKR-v+-2;*7KAq*jk7m;v} zHElFGrM+I10*pD#4R38>&xJOzN9f<(PnpZ-Z={cy~X*xlLf^g{ShXG9_wX{!E^35Jl-0ZtGI zEOEhM68fX@AsxACJ8|K+E=Io|$t(87+O`+-@@qpehuS#82mAWggj~5nyg-Le`2>$P zYZSAMh?J3A7SmJWJ7p^L(ls!jz6mx5dAJg2O-tZfxule|&UjuU>Zg5vQB&M`>UOkm z)ni~%D(dz8^Il>k?mGw^QH)oZ+|sE_)D?zL zrSWM?fiO26%)uIDBa$mvGX30kwYZ~ldyKz65G;MHLFNjTddMZ}qCyby1UKm3eSRAd zSxY2EX6U-`4oL zSO;#dbv74qk~+kY0I0aponQe|cMROtP#$%<0zEQJ!+fn_MZG;Eu68c2cDs@mHX?$C zlOrStP$f{gYsKrnIUU;hQGQeji~x^{C#iT%)GHbsrM^xT#9u$UaAU;n*P-Fpp(*Vr zPEvy)CN~+J@7>MK0aJ<}OgoKas`3|ZAD-{4C-1OMIH?x49}q(Clo2Os$X$lxz`Ks+ zDMC^jNY2z*~iw$N?t!g5|Rlg?Fm2JbikK87?Qf-h{{4${Z5tbM%5k`Jh-4TXcaWW!psiS0&|4dY1#4N#Nqu_ z;d+c1SaFfr4ObYYcsP{jQ;tM`OAsd`{Xoi7wB&y$PSj-oxVrV@{>?Lt-MKQc2KrX7 z>SM8Bwb!mUC(49R9de8Q~fL&-&uFwe~l?MTI-+9pO-q z9xN=Sn*4mq5*H8{&u+y1WjSL+&*Iu%(j9!3RnhUm+OsaQ&vw8UjAW#De!SixyJkl{ z<5nc*{|tLnbny#p zRh}Y$KQM5>8{nH}5_of+gp+uP6w(q%ewNo5pUp4+_>f{{KCvvmeTKpSEQJ{*pUl*( z{AIy342OX3uC8KyMyw>LrKbKPGPj4k+;BV58vb|qmA8?-JR~unY22si*XThXj2dBN zqvsNHHK;GVKxblx3uur1cRb&RNxXAem~0_zy2L{PBOJ3;FP2}S)vyzCs0(duse#e~ znk$f(@v!hrGa=v*3?hQ}=P4Iw&a?xrV6P*v=!XFX5`Z`3oidrv)R&mk^Tx-DU@mk~LRaiQ#Xro|OYASztVx-48}+LXXndhR0xdhVzM01h)0 zf+}Uk7}aS3^cJq}C6=5=lUt93h-wyVaQ<;y0qoAUq!<778!hQQ8&6Dk&B3{Nb3D7N z1N;LBAl#^ea7oE~^x#&DlOUrA5)WUv@Y)gf>-_kK0C_&w)T1vtQ)Ff60_*qu2_aCF zLba#h-M%S#k!@iv&C(}2COGRi9GqE`k_EC;jB%S4{~*i`6%=02vTZSKt{D%DvCFTC zc2ng%nm#&t#A4e70;%)9Mm;9L*>`V2jhx_`lj4HXAX49+Aq^ARtH`N|_T7)@nq`~w zoc@QpP9vdJU#5hd^}5U~J(?vM;~$GN@WtE%k!A1}Bh_$gO;ZcY5Fxw?M>Gmz#ReH6 z1zZ}Xnz9>Hvx^b=exMJ1@yhzq?{Bki_h-#`hyn9D2x`jjthsM3DW)Y+39*DMwSTwc?Ln)CpY(#{O>R6g*x8K;y+)TJP$ z%|9H~QB$yX)$nyAbvfA>g9Znz5(ilEGE401{?+3}CqBUth)0>ASHJf`9_gp?Amb+C z27^BM?GrrcZi-dqwEWTIAJJ!LfSogI{-XEKk6E{~`n@p)V&H0kJwO`ur3ceAs$Jij z*O;C2Q#2I>Q{Yl9xsZNP1~8{!h`}d7w#DS;pfYPkC`Rx?)z?kCeE!c?3us+V1+=kg9E7S#*yK>gLB3=CjXS1n4A+}!W?uT6`tynO>S0{jDXRVO308) z2mrSazUcnwEn8>dV*rD#YUY>6tlzk|eam3gZvSx;cM0wpipoV?3~r9S6C%MEKFBDr z9JI$)WFJ{l&#~ZFYH0yRMW86uMBcdO7`IRG*{)H*Z}ERrgviT36HQS;HC6^i?*(Oq z8Yaka;UU!{E)ttt!Hd@c6pivdyA%`&I`u-;Z1XKUQ?6vw9T%XR2nGY^g*{*4_`0

    Zq5m( zmO~b2XkDEf@0StBpC5*G0`ZMsH|l?`B8&EBo}cXzEkZx?QY3s@Jv8>YIWRt(aNa%r zPGXg|xAK7osyHU8Y5LAXa zaN3A-5R)c+?GQLT)VeJkizuFPDT|W#ZKn}YnCYne)pj3aZVkA*i6gym*fUb3-DLL9 z!0ok39FE}CIC)g!5c_?j%WQn}2KmHACaZbJ)hh&oYUc9)F2K$b)i>9A(@wJaY`Y9e z#Kp-k74voTlkWLgl%i8)!7gi1GmDanPfQmm2-69`*r-TfIhWvn{D&ww{zF|;exnj+ zuFdGg7yp}lk=BE}LAjMp)*v+g!tp)p`5QQ&0aCmz#ISOoNj4=;{) zyu^^ojdsWgyy=-RN0CnwvJ+;P7g_E-KVK9|sM#_uyUXTZb1zyO-L>^MoIqq#I0PY~ zRj`}UaBI8!#Pa$=F6zqnm+N2mBAOpPz*5!=pa9VJzg*jUQn~NA(6rNM@ILFoP;?S& ze&aK+paKzElEZh>z8-u!pr{mH5K>Ye9C=B4_r?YXJhF)p@M808;Yb_uj4N6 zES%lCaCdZP7AFyI3(GA2d%ug;F?bCfk;USDc)Kw085Ljs)fZ z*WH-LPh(S5|L)5pRouidec0akWkZg|p@X58D9D?Z2K$TUUA5SiMmR z>z_xD$Tox*biiPNs)k3vpWO!Io!hfdLCvd^5Q{_T(Bv&GK!V&RW5fmC@&nBHR{D1^ zMQ?YGw5SvhKF1}+(g&w72!GDGVxI>K6Zl;Z^Xpv4ht9{(ZhW?hq~)t#^srlIMvN~m zR$R%1g6Z2za=0T<>fv|zSG-liZ$hkSSddBb@Md{zi;Tac*HKOutX`m@8%Qja|GL>A z7okI(IXAZe@{HLEC&gWPW2^Ha~IaO=uW0&j({~<`a=| ztd)%s51eqTXq2>lI94T;4Xrh?p5a+x!y^Gsu=<Oir zkZ+Qd28T6E<*}w@x;uvMU>f}+i4|RXu*98Z>=4vfDlh;e09`*Wqt(UF#_9z0X9NK+ z_mmE1h{G-X&A;R2B$s4p{mC8QF3SUY;6oV6I@_eHj1F-eAWGaUDESbUq(LBqP6+@y zz35b-J%E1T3gB(ms=YifsZZcw!x2lz&e>|>Uh&s#S#ZANV^D$(!0URrV=Rg6xDD#Vc13FO5@>k_%lw~#hca~ZeuP%AfFRxP{TKX-`1BsQRy-jQ z#vvJTSdnL*Y!HJsP{Z7ITNqsH6YyJsfY&3wG6pM8kuds_&1#wobsCK2ZF&Zd2QxSX z{DR!SL(Xmh-i#Gjxqj^KET|DHZh13>ru3G=z1+}tWq5uv{yn5qHrC@HzR4e_OD=>nLr4a=d=<|uH6Io1I+!K3yY{|Y|>jXa@N8t-m@At zna2sWpjcm>nXbvXQ20>smi6JjR#iJL_k+0fLmV7b4Od|!?tdj;{2A40|5#a#t%uY& z-tHF#%8D+3L&nhpR*@3h6(RqX)oJJNm?NFo2et3YThy&+)Osqs5lv*VD=(f;wOJ0$h1|fd6XLo(QJY zpVk&GjDa+iEP1aNZR>WCIrlGK2&ViIOwFf(0-Gtbwcp+*>~U&ReG`noTBsGTCx;j8 zFGYgVwf>#|v`xDu9Lk%;`qqA04w%Mb2P>yE8~&1fbE@z9I3;jaKU67@SS;;w58wOHb;?$|Lh0*%`l#Hn3 zmf=63%gkl|6c-F;{ckdZzUfjkEMLpX>&_C%#_lEu0!R z9$9mlEe>ppy%3x|o@Am^L~nBF20!`*))R3cpqcL00+pw7E7%cS3epQf@Z)n=VY>8g7}O)e>6y2UN_1Vx?VSgH)Dk zX1n)>Pe(0U3Cl16k}EOhJhvr+WLH6fAj(|-&k6SQFD7xmEy7C3!ha~?-XYsk;-o1k ze>HKK^+a3ElTPgxyxK7#d6y#tcXjSF_NlW!gk1TeI4mbxUw4jHF6i~ z|A8!Pz_2G-nGe$q0fBnU;hgFwfC}V^W~38*+jYu{QeL#Wv(+)b?ZokNSZk1rByd1b>>i2hPdu`q~5K~ z->O~{5FiC}jY~asb>18!9Lb>IS3Ak4GPr+BpA;j5Kl^JN%Qdzu zd`6pYTxLEmh}0xwkt$XZj#dTBXa95Js1#2Yx5qYxd9HSWzvW7>2`H$W#9jEE(GssM zTQ+**lXD@STY1W-zrE-*Zf7j$G+EIfkm&^JaqhAYtQ$7~0cDL>544$h# z;91B{?KCVL4i?tGl&wfDm1Ui&$`dj%cND%{qPU~@mHN##d12^N14b1m!7$J6#i_%i z$5eRHBJVBDF`SwUmvME*Xh|QPGl83orN6C4R$!%DcrV~C0~gMy9lk!yA6M{hpE2{K zGd5ab$ZiW0aKf>&`B7O=C@%HX%~6%#g>1`f6nPH!pH$@t54)Rkhk7e#vsD($2-zyI z)TMSZ9klvo8Y$wE5$bU=+!D;s3oe6VbwDQjDr}oo;kXR%kuc^12)MeNc2Rn?{@=b@ zW&!C6qc`4{k2pL#de?{d~U%L?WKS-cb3hU+H+e)=n z(sf9-N*!Lt6#j<$4EOMji<$Wj2f7PqZ2yy-5TOEp%oiC?Q*M6ksInr5p1t!s;v&I* z-ej$>AbPh(+2C@BcY%(qJr>P~2y~wkXmxVdYAtX4jw*?Q2A3VamF6&9erw1t8e~Ke z&3XRr)l1z$#l0wzMe3;CX=iKbOzywy5!7vpB7{|3a27P%d^Jy|!txPaP%n9Qn<^9h zSFG*i)pVlL1!o-Kt&c^=-c?!8N zl-*~+%JWe-y~5y$39qYQ^>_w^?J>#xiZG!`^!5H zl`Xq%G5aUSfA1cw>ZyTuP(Eak6cKp)Sn7S~8auijSXX@0=0&9Nt&F2BGVI!Y;4!=X zW(6nDq4{Cmx1b;AO;nY>j;?|!dzJTFAwz`04=xN1#J%X0Ruf*x444_8p5%WS{2FP7 zslHw7AS-`<@2$;64#zlsMWef=i}UM>q2nCM-JgR2^BdQ!ztLspXQ;|j$LsaUZPNvM zl%Y>(-i;%|iU=U#mJ>q>d?Ttb`))N>^vC=`umReOK}W`+(Rk02$k}DwJek^msnQ%?+-zLvH7dyKVz2&=BBniEyyfCo(JQGs^o!Xs+4yM#5!xxpal}Ib zzdPw?V*EIMU zX)WD}zl>D+*^s)-K6ft1d+@R((jO=Th`;*<_eD%~mKZmmUiW^>Z27y_Q$JMdrY#TE z{d-TQkqX69=4qTz<}G7zi-QBVw zbd7+wVj~{zvLaK6M#j2kAqSZl(=aIfr$+<#N9w+8LD*Hay2dlk0dU#@)jEtkzKfET>#!~1f~ z{nAU?3h*KHY-pCsytvk*6V7~d`c3)~X8xPQ^;I~FWy##fbb-*MJ*a?Y8Dq$gGqy`mT>d&~le}C+#%P&gRgn!ga7_NUkTHQhoxKC@x9{;E-mL-R!kqQ*2oo z`K+N^sbIZ>*yvl17gIZHgSe~z#@OX7F!FbxR2_hhy!&pPU}^(QT3DFXSNOFJ@0&G$ zKgN=ms#5yVntG9g6_j^IHYpC#hWrHF^-rC+;2(BAg^It(VHuM7wCmo!iwn%rI0)n` zfHe;H#cg84sc;{BW;u3RXA@mXjC*w7`?9mfR^y3=f*oTwOi1G?J_G<1#Bk0xUqgfZ zyAv-jyE~b-8|p@WyaRqX9ZeOx z-U%CI$wOHBBji9}&DJLxSPZAio%ekVi3%(~90zc3$%-7BZ-p2D#3?}=0ao7fvc&AN z=t0;3X+~i8&xH4S8Tvsp#nz*D2DE>wCcl~95@ipk1b`(GHmq2dy2TDo&4wOvNs>pK zYi0{hWqvQLJNnITExyTTu+F^4=o0`~5AhCYfGj-GzW3(N*atqqnSfPBT#p#RkWiBA zut*QTRivJz?mKBiS)}j{`}u$ERufx-^ZZpPr}Jc1#Homgl)n*! za&uBbrH;S+*MFPUyH&=lKX6p4#sgk|ISVFU@63ujF>wYkkOQK%bgOHbLL6XW|Bd)G z-x6-t))gmwmeu*yS1i)glI}kD&?K{I-lh7?An=muM8hkYuUN^h(h~Yvw+EeDpg@6t zqxkvR7MBu|Rq_+W2n!k9Y3Im$z%0eDT4-R;SlTX14gpk4`dQ7W3Hbov8R7LN z{gQ%6gGo6=34RYy#5)k$FiCv`Toa)hsP)$mBZBdlriG=COR*Go#zTf7?j;&&Y5D19 z(ajZy{6O&QR)&XnKnmm;*?PTLY!xdPl=tHPfjpVCEr(z+kT^CLrB4tt5XlI$s5br}s_3T;oHmQ)|rX}w{WADb2d4G*^`!b7cw z0rgnMI-5exn{v!i;$T?=ExAj0dZYNP@X8n0z|#mGjP;!$eD=d%KcEs{v@LH}eydGN zVXx!0|7)Awo#4RYLkSskm@63IxX8al?EH=jSE|ENS7_^ryFaBz?5H>!UW?E9QiYVE zyQ#U9wdKJpr3Ar=kIlIO)F?Hi(`L^Vq$!03{^zMa77Bv(@WeMQ>#r4)PO>-~3``0k znyhoS30_I2hbRy7CVn)sWQDj=IMAx;!9ot>0L#7Kd<-%|0%l7O7n_)`u$~&+$bTE$ zitEVw_?e9qILO5lf)FhSxzGDTVh^`aH6DWLeg_T!_X?X2!B$ zjtR!N>wRW#&h#G3Aav1#)l_Th^bEy2CRErtZ%0iCWWDTRhU*n!MXG_D-V( zjFr~Z`<6gZfVkS*v-@NrtfR#m?$s1Dw)e3vr_x=7p(lq`a09@I)XUw&;n+VP+OIOU z(v9ORRwoi@uj4n{503j!9OCR7Q{1gfl36yT22WMJd7^+?&Z7;sDweX^Gn8-G8m4Kj z$phK%@+-(tzC@Y?m)g&#t8t9Lx2e?!AJnfDd9Y@d%QdumYY}Ih%ea17(pe2esBn4i z@w5d7veTl_R-Mb*_}>sJMOzcLCy5`l%o4A@hlLb>L?M{p$K8|sP?NF&% zu>J3iOhauYEPI=4^5^f)s(muYRk4&e*_<95K}pBzc92*>8~m_TgtlUUp4UAV^cG*! z!l|~LHEGo8U34(VP5n&L3*bycLPm$L*B4y1Up%%w5KQ7<{`a9Z+zQ0D;>b58d*n zf*e(4sj)WLvc=Jpg@FcT4#fS_J&lnVXT{e@9|cM%aV#FNdF)W~Oi8=GK|gT{C+ffe zMmlyAiXlw-k&$dyHE4j5)w>bQL=iJR^8W_09$;&4G-G_moLG3(A;yQk)v^9RxCJ&a zcGJeOFIDGv_GJeuc0s@KLBc}==o0eE4ww)-;Re#c;o13fls_RXyJPaTG&P1+*?SDi z4tFS~#tFhFnj?723=>Y$^+OKhfakuXl3=o}5=8qX&3bB-Z%Ru~9P`z}mGXV#`a_3S zrVUe_+f#lZRPZ^m4qb`$%5&FPE@c*$&<{P4lqtJf)fYKR>Dsr@y5R_~xOp;KS?F4J z>#ZhW)RjwunbAw-H4a)dis;hn&jrD`J1agwu;^jS9F!Z|m7zkK8zZtds-)O9?8W?A zO48ZL5wx`&plfKma!8|ajNH_t0Q+sH2Fh86qn+iD(A?OnZ$65dKfkNhyYH%3!LhzW z*%SfZn>Vq{XzGmIzRx6e7OhemEIy)~nNURfiN-3-Hk(Q+=#g8HaOln8OQfJ=fiU71B|$fvSf$R!x+E}Di|PI=!y|4z>6NqIxVrv zjPPwNIF31FC`W{6cIXz_a(Xw{$=+?q-X48B>?Dz;Zst<7RHdp(sCfF;#Y#D>`GG~06*{Dyf zp-sTTHzCgCcC0FN;Qnf&yW^#+iZHuJHO`17jlJ8EFKz@12d;!71xZ=;<4bleE}t}k zJ8CNI>u++y*CK!;Pc~B zx%+{f+ugL{Ln&};qFxt2287E-eO?L~0Y`ESLR-a!lIfO`G~Owg+QaF*K7YF#KA={q%bk zBZ-@EPr6&%D8tMgSg~u1w%E(8gtJ?zqd#_+8C-;kwpsp;z60UDPBZqv1{Dbw#^Y0v>~u-0!HbszeZ_erV`|C^?UElM%A711&3)ChRd+pX>a z8297IromSZUAe82llScxzihlUUE8KfPo_MW?tyYS=2|98wdGIhhotaWh0R8zW_Vp@ zVGoaRY!gzzJRwuI$*eI#48uZAFwjX>&?_7lYCbK9=;(Q=Tcq9>AZ0i4rNMU}ltLQt zuDkA!B2P?mh32)ae#@uLK*n&vzJjaG{`*aY)_eLdbh_Mm^|5=rNl8YA5sln%EVZG9e<3-bKb$soasuiW=lsH z*-%!Xij6UA_$(I zkNVcfYk+5Gq}nMsOU#1OCZ(RkRSJR-GV`^%77KJ%1ZEJMi<#b# z7?_-{db4FN?!|-W3}v@1)i7YcTrvE?lc}Z`H5he-#oR;^cm0;SVAV1pW#emR|Vu1IjXmaWPow^s{6cQi30l92Pe-4Dlepyq76R+Drttm0S9~u2n zenss#yQk7YrZEiF*!J#{Y zC4QD5eRN1;V3mEFJS?OXI%q2~)Qbi2e0u5EQqyVJuO0A{`OY;V`J^Y{jRy=E02ginLVX#XxNB(fMWOQ46mnS`CD(KNB!}? z2dlWUfzF`Z_@i_9)~cl<3D3EFVw<6~y|w3I!8ve5or3{eETP<_XRl4EX3(s&wAFBt zt%Y#TUL7S*YK0a(6~wv;%K*2=dA^PJ18gW`jDBQz%hZ(f?< z*+7&hMjKo@wBg9lj?ui+KzbS8l(6F-D|+zuBU$d^Ks;B3(Bns58AOx`&-;rkq{fl) z{iS^2nHqSs#4krBd;!8w{M8!v1dTjH21ew3QYgeA{l54;*c=pG=i>35Tmko z@}_TOerRiAV+=`7w=M9N)Ms(!pR%7&5?;fGLTyVh65pG3A}x9-hPpQ;V^yFEgxd zJu;&|q`;xo>6IMki}T!q&GO(|N#3tXX-45VkQgSBg#o?{G9wgRsSdC1Ca`H>;>j3& zgEAzsh2-AL%OHiL0y&>21MQ_TzvR~xVA8#t8*%oOD9r9fu(|+%!&uduYckXiX85!c z@oKaK)x>vp$X`=@zC5^?KCj=hUcgQ%rN)210OToGrM9Z^mbW2!UDD$j_RFe39;XNU zQ^WP1tWf-4z#fZqpbkcB*_@D9K?VG4?4n#ca=DTe-tpOZ*LaZU!jkAz%dhZSW2i6^ zuDZFo>EE&4JY$aaE!pn@y2=PPE<2>P_>w_vAn5f2v$cEL2dMV?O5=u!`u!%hznj^$ zUMAO(KLHYs(Tg8bT|unsNbo@X1PbW*exbH7`Ii$AYY(UZx55tR%yEquNK5NTb#%VT z0Mcad|Na*^hrlX9hl%JRjTB?-GYf)gkxz`xgR^rbf{WY-{Gp$DuO=t30Q{BW0@KZC z&NO417^f&Vp?J}G1EepYd}EDl<-90kdV!SV60flQiYE9Z^SMIm$(-`}T0*_Hd!P{> zZDnNcG$i@i#F!Rvuw+w0@uQ!_5~ZNR(!dB=0R>Px0>#wEoK)XY%>WbS0Qs)~-W$~C zQP(Ju;U7_I7k=0Q0I)4;|B0wGr=rd}ohF@yAHWXl9X6KM4wgF{|Jh;dWNYJOz0<2JU1&t$U!5l;dDjiU% zqDF$YR4b^AB7!KSN+VDOtPEhq5}ImI3KF3bFpybs=KJQGw|VHxjVS!LXbhwfo z&agICLsa3&9@gTsS5~x>OtA6P!hlKUFrD(Y-A_%@Gau6_EB(kMb1NyM9cBsrf)qPE zYE@!EtHvO&X)I$Qkz#CPA?uPTZw@nMW>KD?)`Tb$1|e#Tme{oDofW zv50CmM4Zll*FL>y(wv$`HYWT$bJmH52xH#cg)DY0M{WL%8TBA}G30N3mQ_d;{PEC8 zu`TlMrVtyj>xqTIh0ag2*AN8*4-FObBF~=<8FpiQUH{z^@g_stBi=W~oZzkBm_BMV zIO>!CP95IwN(mX6Ij_XjioLkUY`vh!7v_{b&=~yYlX`)jO57!RZgQ4v%=bKl zXuBlOjn#xO$l~X~KqP8oMr`B4e66Y+wB)cr9v8b^&0YJ4Y+&7_h!UXM|lAWlZba zLY>1`W_>nP%LO{{^68Pj_xhApB3I3R#b8%WQ)%Zlj>2IR0HHs+T3Uif$~zX@ST(^T zJeyS<@NC{;#srYvBB`f=@bZDR?vf%J-9thEctLu~s{jbrxHv;zRm?MEfQq^iLUE=9 zd>%H5t=Jjp75uNjP*4Cs9V7vqoG@2=0>I?I0$IZU>QcQ8yKdj=oRt9GJd$x1p!yJ< zJ(qwKVg2sdotLZ>l<$?nPAdW%gAz!kb^wbjxrtBj6 z9O8o`=HZ55FhDt<^t2?P!htFK2*G9qkvW|~j36_KPquj@XwT$bMNrNMe`H$cz&=NS z*-T=D53t6G)^`ZtVmR4mKtLNsFPV=fhhE~0rhr~zji!iRVu*%d@;;+MiIoF&ngrf5 zKHR`uTXJAK&N$PONLm4~JCv-@6TO(SwdkYb!$t%R7(_XOQ-jC~7XmyPOxY*J8wRBO z?dHxG-}O9A@8%Aj7MY|17&+alDFm=Bq$@0GAQU=KA`_s#&xICy+*TQcBou(Or!B&S z>3p&RUFpGO#X&S7WD1J_1`^$A!1iRyKEvr#9H=vxBFTqN4CBNH0WLo`2cl8kV@x6n zyT&9J&@sY+y^w%{B6?#OYVS(hg?>5j8=+O?%C;du=IFq-KoiO5nbUwe$sp9&7(1M3 zo2^jbP_n@p1wKyCSq_lm!DQrPvoe{CBWS#sjBEgflLOlb`AvM8hD2qV#1VSjHf>U}ojPWZn{N*4a!?J@XZxr$g`zgUCQ;QZPA!%nA;(%@x5H ze4ZG=k8`LxI>03+F@aTI<4hZ8qp~Z>N>_mRAhHsZj1ML&QT+H&vJy)gzmjZ6tnUE* zB1UU7W3Ix?OWYhv%n9J7@?~!^=q5|zDVAe*7@4#TedaSO%nZ_Q3o0&33AJbG`?~XK)ygmR;q`Y1q#9LlJ6OEO;eh!+y z%Ip2nd@QfGz-Zg^#9aN7Y<2o2Er`>;!?e2!)vu3ZQIQyfJ}VH!8P_6+Gp_$i&B{V> zzFa&5L1g|NneN-P_mNqhqh=vfkt7#KA$T1@WUfRInTvAO`GHu?B6U8dT#=*BU#izp z$->MPE9G3hIG6L@6Q|qhx-cz|Pqj*v!S4r+3QL^UpiyBp(~8xtNEB+VuwG4LO6`{Y09m>R;E0?-qlQNa9NARDeTjQHq zyD=getF6IQ=yJ{VgJ>Ew*Y%x$S0kE&El;UcxSt49NrfWBvLM& zgP^&cxJnOh(SCs%_AMq`4Zd0c;7ptY@7zX>harhSPm)K#FqunyC4luL_oMl%FambKfocA2Q_QFYu0 z2s+E9M==00gOTYTtF=RBHKmTzZ>Ql1BGVf|WUh?W=JN3Ul4-0CVSq*r?x?UH|9y#! zY_+R{ejio}^u!P4(nj1H&GNX9ke`^xwZo?Q&!+>G{PR8vf6k}n|2E3G5yEr8++t(B z3A8VC=IW#g6C`#2+h@J2Licw1Uu_zFq1-R-G{6U+l@_vkhFixHInTcw-l@wD>&Z$; z)g*DAdyo44egmJ}57X6CPO0#(T?hjGH-?Vw>vttqKRo+--6#P`ug*Rm95TP^^_wG5 z$=LaBccvC#+eXKP){IYOh44Y+@=?@SW2J9+^ptBw$8<(A{LtOjOb_fiy(jxheI@B+ zO8rv5+$8Y!jlws@9Riv|k#W~$ds>On%`C;<7dcDYE*TyMsm{V0ce&(`O4ENzG~a1LC3YK9 z%L2Rx%-7k2{UZGVh5^p(^Mivkyu(7cdg|FSy`m$^$ua%V5&!Zpioni4I%y;|o zhb1(YN2xa9d*zkMTZ;U(X#t+yWBwLzAM5Zec!titHdd_@ah`86Wlf&Kl}^Y7Chjk~ z&1ZmP$A=@^0|qTh3#ETa>vId^^jsV&6xt4#d_FG;+P`&Ku6Sjdsf!w_PDO=$-TYXK zjtnmjg!U`$Hr7WU-k(Sg<-|m5hOP=~?_HnHb?%##6dO;x6l)+pkYse*gmUUP4OMr3 z6%i2AJ0{!QqKYNj&eui7^5?Tgbsv@97WdxcE*daT{zbzHys3KQHzd%ZfYx~a!Vda6 zqIuNDZ{&GtVX~k$OLXvC$$gcHnzf^Bis%uuD?5xM-Y`^sQ<8EY?n>?rE4UNcCe#T& zFflDw7)SbvX2|12M0Bk>^1#WI$!M2ws4Vxgu@B6^8Xqyt|NLXl^Jmt@7Q1&2wh;Tr zQE~sMydWwB%UtiFeFC}-Ln-2A@bRL=PAfd&U!2S2Ibl*|Wj$lb6MiKphyOY#jO(yO zXs`b8|LIbxd#Uypx6rpD1Y$Zqa7W;H_qPsTVCgqFJWW*Ef zt4Zs%#DG~5%@%jQX#e&1?&vP)yWA1=fE>zT|J$~b$85P}AIm=_m z`pp4@;&q{8l66PIp)+FM_KH`J`+{KC zYG`;N5r;#R@>j@d8k?MVGjLGhNKk6K$ul(uc)K)vf4j7T(G)ejzjCq&iEJ4SCcL#j z;zgBCP?NTkcu_UQ(c7uIy`*f)LEvQ^*3G2}mj;3G^TXnqG@${GG7gJ3(S%EaK(Xtv zcsi}b(hDjik9vwBp3eZkZA1N5aF#*xT#WFv)8_Ljlg1rflTk z2q>t4q{mNs7h4@D5@==Q>WMYWcqavI8u&V#>vn&-7DS&V70b@mJ0%O5S=)593M<;k zB~hY-Z8+bE0_2YFw=Mz#KObN5+B0hWs%N&L|C7Uc>dBpoRgBoXyyoCDm?=%FZma*w z$X%))S@+h^9zi~K$-p?J=OGK=wkNKq0_ z^)$gjY-sh&P%o4B$%k+oA*Q@#J22n`* zH8rg?d|E`iQ)0q;{;NqdVr~P}9m2y2ln; zA&EHoX+g$N z_cn6@U8#p&b&t26<7<0Ot6mvu+ejTCYRcy@7X^6>6Xs+FIko`%R@ViYivUSog9Ygl zKcMl)skL77l&A_PzIl-dCYuSxKSe(l@oQp*B6~==^1|A&1o4g-rg}e?K_3y-PV#i~ zgQW)=LWqS6FdhkCTtzS>YR{7ulM(;%!cykHqgfN#y7AqSMjs`YyIK=GBpX<;_nZBQ z3JQ>zb8fB7mbH@fR*E9ZfJxh1MJ=lh33&+^cnO%oh_DD>LxTQRn!LZ!Jrg6p z07Z+)H#uvO6O7zw28I|moUlWEf2rRad~utj4nahaB#}ztD`gTgb=!Ni_`VF=vt>VG z1UtGNc87}$^6E$_v-Rvl7SAn*9)y>GtU^smxgOPygE>E1Ci0xtTjd6|cr4r$ zu(nQOh`t|>6M!9Rr%tYC=DbCgs_a98;8L44#%v$m*&j8ROE9JfUcMwZpc}UEx|7`! z1!<}@Oo@5cA2BE%oXwz}BWS~J!wB_50$bgn-m9{_UDBj9^0)xMf04POqJ;NWY5;Mu z&Q(B#Q0Qq=xWxW)k*-RFir>V$`*5vrY+rYCe(%kdO4h6?bLj*oV!&uZIOALG1DE$~ zPeR2n16&`tRA*)RPl_pIOE{lm>{R8M9+w>NweD{_-9Y6r)JsSmq^?wkp0kqOS{W7b9;ML#J%G3zNO+13bBkS zc*`C&dBP_hqgj)0B8u!ejTX6W^dlq2xhk)3PHf)&GQFLg?o+fS`ttk1#m4}?0Y8zG z$+!7QvXUX^iZghd0ykw z+>jM|fS-_V(a?k6YB=oia%RwCW=;IhnJ~2AXttUPE z(%`$>Npo|?wHejLkJeT}o0WGZ$uci&a7dY-gjOe0Bxz)np-%I|e@#(K?dV$d#Yzdz z%!m4Z`AVT`^6Sk)Kr**{1ibq00RS6XN@_FeoxxJV*+77wXJa?BS#i(1o{Jj}A&BP|%Li5T!^a;huJ9EIuz2bU1w`NLn z(^Tov&b9A7+dO&uHRNp2qw?65hOCU(t;fOL-FvP&zqc_K@4KIMH`uZ__Xn6NDp@=9 zYPQBj;Tq&v69F{oe_n2qA+xT<>tj;mcrb{PgS@y$C%da;Kpubl`2`P|k{vvZpya%^ zNYHM_6%7d`aFn}tLDx}u_2_Z(^L|1bi;unAI?uf2TGu}G&VN{SbI)EXJoz47?D zj7RMii9y?&g);NSui6J_kG;1(se8W=vbI2WXd>md8E5g%^)s`v&@96%J!aaj`x-I* zb7DcK8>93V-Fz|;{1of*bC0|tDQ=Me=dIV9=>=|e5nM93acAqXR|S|%>mPm?eLR~e#pZ&2db zekEMaOYcga_$`X8#Q|67I!^EDGnoWk(7wopkpUNnQ*MRLY;M1VZDsDs{*>%a3l4=jKfNvM$^Z?VsD!N13KnF<48W1)^0Ji5K&L zI+q{Vk@hfED4(0m-JOx1sDCvly*aumZ_&Jr^1|D$cpn-Ac3v~KMGy4Oghppz7BF2* zuj*K?+D|S|ANkukha|iF(SA`5s-W7|4JesL-irf9>}!$&{aq%#I}JZK?Mr9&FAYx4 z)iH~2CimB4rQ*g;m=obJfdUse-zVRgm;Pe8TTP#7iIm*G%4o17`eiF7)br54pho=p z2bul1oaXd^rU*CHz4}#`m8`wa`a{%!WbbM<4yVg(>hWWe&Z+S&Wq*@dR{-a8b|WpJ z6fQ8PQ&R2tXb{6vzw6o+R#`bXcYAxX#key+UU-tKRawgofBf5XTKPYD`HHP&aWT8- z*5zxe@5evZp8cq6JG=QhgJklNZ*3ZA;qy(Wj6mvB0a$PX>#Wi&v=(e_p|rxv^$A1EKLxiGloNZpyJA82#Gvf#K~% zr}lD1g(&W{E%~{g4__M(*Ax}slQe!oK5$J*+J4|UCTOek%!h>9`QLj5)fhHF#XcFn zbvc|}A|L6tAkTKzR5doq3~?OjPQNHSsB3LTL`=b^E%~82V}xIObp1Dj_Ng43FFD6% zZpZ-rex>hCjGOh{!f{37QhQ$-nB!XNNIwOy;s*x)kw{XDvN`sY z?31x7-=BT<@tXx65Ep#i!TL`4;b{yE*-Rk+6*e*Q1tsJ`w_IP8vb3st*Xqw-u~ac@ znY+Xy8fqoth)SW%_MvX?a`Y_1M`~g@W>W6T^}asgXlVkHEKNe--I>CFrt$=zEkvC+ zRC5hK-_*aAL=+U@!rtug`rg?$Ty=hpBT|4L@sD(lT}!}R9Pp<5%1R=>g)ujIQA^D| zndikOOaoW(+c$I<1L7@kju?uQLTU^?=BV6~{CxTX-#+gyRC`n5IX^be(L&MEKP3n^ z#q~dPB*kQ|=-}(;i1v3s48J$Ktsbl5zi%iCI_+LI5qZV*KI}hTkD=dxfxXb!lEE2b zw-g?5<~Gv<2&6$oAiqoe{E@C+?wXC89i z4miaJ6!9mNAfHBYZRp2qa)UMy;xGaQ#P0@5$%V_t%n2zHczi#rz+lu$AWDM4Am9%3 z|Hc;%G)}yh z979)MQ{H=^gLUmobFURh7*?8)K%ac?q6G+R9CWY4_9*8KXNcRLYiI8h+o51=Ena_omp57{J-NZWS&iRqL<7KEMFc zljd=3^Z;*f#B*RqhcP?4wM{CxEEY58ik{RTrUwW|l7r2SrYQMtT5=|QVEQGEtcw3J zM*~&HL`n06a)Cr<*o0(mP(e{TbmN<+PWhf0p_i>WK}S5f9F>H*LAc2l0#TKz1&#Du zAA|s$m8JmWL44Icef^^C3hpe>k5i_F)50r~nnHM6a!Ff#lJ_1XAWDi9UWYqh_OZ_I z!zrcHYG78gvmy{@Gj_l^VW+~AO_0Lt^ylm8ZQ&fe+40bO>HEnEH-z?V;^;kg(r~ul zGXX(G)aNDn6i~SO`eNgo^pfbSq~6azniOn_#!_OUg?gX0gD_XEP!y^ZOl|cJpc#q? zWrl_%ezJOIso(Q+Hs=Zv`oR{E_;H?@T=YK4RBGm9a5rt|810u8hFY$Bh$5GKKTyFY z3Y0GNDRFNt@c9KQ7HUKCMtnB!KF%|KracLuGQAOmV< zXgtriJ+x6`(z#-`yKC?;I$%X_&iNM)WBUgWVw6>J8#V+u;^GSASSRi!^M$)UuQ!?-^%M{AnsY~z(M;9d^2xJ7p}VIYowepGv|Rl^dwl& zzWPTN0!yU^iZeH|j>gK>Rk?eO_^B8ZHu<(4Si+-_yz^L=DOcgwoZ-Ftn7bAyR0jI9 zj_l~2TEf434~QkT!T2PNLvf&S<(ur(|T0p-_z^i!i7ZemECN*w*@P(`+sGq}B0aAeF6+Q64b2+0- z*C@Vmw1cD)OWEvRm|cX&b8?E^9^Jo^!b-}~t^J{uO_n~NRSBSQjiFYjHGFYFvfTGH zH`t+5@4~6Ct|Afs4n-NlA(^U5d@56Sj!y0GczwEWBntR)*)e`on@HM=^GUxJmjY)6 zc9ZyUpF>HMV0i?}Oua#Z3y?hzbQ|;mEXGgS)?j^h*lJ@%a>?b|&oL<`NXPlb*^Ru4cjzoO1r#zf*hIJv{WDWR)u|3Mk z)0@JTsbawDM*HkQ5fkC!_N_1Ifao2s>cLhS1V)cvg@$ggqD{m-KN*+P18?%lk>;>u zuOLBNqF*1=SKlLgJ)RQB-zZq>uyq3Zs!(S?@Nn#1&oCb_T5AxuJ3pchhYiLa~(S*69O}2O9QQpbhl@L${djWd@fBr5gBZ6h;jc5I0CD@jxS|*}Qt+>BB>PYo*xOOh z2k6G>4}oqn`_be)P^gFW04|69XGLn|C+j`GX7}}ANe)Bx1fdg>&v}c3iGdqU&LA9E%|sd zI*g2c0!5Zw{Iw9s+Ppoal*Jqz=Ck*sM=zxpAKP7HsNLK`u~Ol|^|P8x_+7 z{_eBs$%cT}z4+a$;e#zu?w6i>xS|s#H5=EjK*>3A;jH;C>2W)Ycatd&>Vr9tzEJjU zR9N>?1}yHtLwR>T3~vdSkc$C-chY0s5}Pi0IN5rGbY>HZ%5{Sm;*m-|_rnp8?6l*3 z6KO{6419pW?EAb3#YH>zSB?A%ftXrK_|9 zBaxK%ae&9P;FhymIhrHChIGQmh3qtYV5L_yegO&M|ATy@2SI@5>jO553+Ffa-l=hEz=97{6@9zNLR{N15^AV9ktfSM>`>?W8D}ZP1yy=9ZLZF-B&QGK zWBLd>{=Iqy)=&*5H-3OO>gZUVsTydb)+P7}zz6m$B^k8Ugm3}QqsSa;_4jFZSW*yx zUmGiUZ?xw#SvGc5tEjFI9}-6a$WJ^C!Gb_R$qOW$m%?&BB~vny6VVi;<7hHD5Cd?{ zrA7!A8_=n5D02bMdzS>Yx`x@Bz925Z+b0Ta+?BHWW$5`S$7~4-&xwBhZ{f5iX%GkY z23j!8mb?_u!HH-S2EKM;3EDR8-SZ?01gbb}1&1&?FaTbPHx#ZuiXZj%U;xmJC&}1-fqB%uJzw4 z#NHUbI~Kf{A8o%udk$F5m4+69GyK7kH%Ws=fySTQAku-22KJEY$(r-62r(84++4$5 zL&$&>&G0=&#LJiZgvE$2GIy@Iy8BYzwpM+l7k_}&HHB&C@Lp29g0cb5%1evbu+>Z~ zeFn+bDyu@w+-{ziF)>xv)fcTJ2%7WZTTsnv?$SnFPX3@Z1FHA1sg(}jnr)GC?CzN8 zqXkSm)&I~iQI45IDQFsM5Rqr$(n}w^7p+mh&WPl~605BKR-v+-2;*7KAq*jk7m;v} zHElFGrM+I10*pD#4R38>&xJOzN9f<(PnpZ-Z={cy~X*xlLf^g{ShXG9_wX{!E^35Jl-0ZtGI zEOEhM68fX@AsxACJ8|K+E=Io|$t(87+O`+-@@qpehuS#82mAWggj~5nyg-Le`2>$P zYZSAMh?J3A7SmJWJ7p^L(ls!jz6mx5dAJg2O-tZfxule|&UjuU>Zg5vQB&M`>UOkm z)ni~%D(dz8^Il>k?mGw^QH)oZ+|sE_)D?zL zrSWM?fiO26%)uIDBa$mvGX30kwYZ~ldyKz65G;MHLFNjTddMZ}qCyby1UKm3eSRAd zSxY2EX6U-`4oL zSO;#dbv74qk~+kY0I0aponQe|cMROtP#$%<0zEQJ!+fn_MZG;Eu68c2cDs@mHX?$C zlOrStP$f{gYsKrnIUU;hQGQeji~x^{C#iT%)GHbsrM^xT#9u$UaAU;n*P-Fpp(*Vr zPEvy)CN~+J@7>MK0aJ<}OgoKas`3|ZAD-{4C-1OMIH?x49}q(Clo2Os$X$lxz`Ks+ zDMC^jNY2z*~iw$N?t!g5|Rlg?Fm2JbikK87?Qf-h{{4${Z5tbM%5k`Jh-4TXcaWW!psiS0&|4dY1#4N#Nqu_ z;d+c1SaFfr4ObYYcsP{jQ;tM`OAsd`{Xoi7wB&y$PSj-oxVrV@{>?Lt-MKQc2KrX7 z>SM8Bwb!mUC(49R9de8Q~fL&-&uFwe~l?MTI-+9pO-q z9xN=Sn*4mq5*H8{&u+y1WjSL+&*Iu%(j9!3RnhUm+OsaQ&vw8UjAW#De!SixyJkl{ z<5nc*{|tLnbny#p zRh}Y$KQM5>8{nH}5_of+gp+uP6w(q%ewNo5pUp4+_>f{{KCvvmeTKpSEQJ{*pUl*( z{AIy342OX3uC8KyMyw>LrKbKPGPj4k+;BV58vb|qmA8?-JR~unY22si*XThXj2dBN zqvsNHHK;GVKxblx3uur1cRb&RNxXAem~0_zy2L{PBOJ3;FP2}S)vyzCs0(duse#e~ znk$f(@v!hrGa=v*3?hQ}=P4Iw&a?xrV6P*v=!XFX5`Z`3oidrv)R&mk^Tx-DU@mk~LRaiQ#Xro|OYASztVx-48}+LXXndhR0xdhVzM01h)0 zf+}Uk7}aS3^cJq}C6=5=lUt93h-wyVaQ<;y0qoAUq!<778!hQQ8&6Dk&B3{Nb3D7N z1N;LBAl#^ea7oE~^x#&DlOUrA5)WUv@Y)gf>-_kK0C_&w)T1vtQ)Ff60_*qu2_aCF zLba#h-M%S#k!@iv&C(}2COGRi9GqE`k_EC;jB%S4{~*i`6%=02vTZSKt{D%DvCFTC zc2ng%nm#&t#A4e70;%)9Mm;9L*>`V2jhx_`lj4HXAX49+Aq^ARtH`N|_T7)@nq`~w zoc@QpP9vdJU#5hd^}5U~J(?vM;~$GN@WtE%k!A1}Bh_$gO;ZcY5Fxw?M>Gmz#ReH6 z1zZ}Xnz9>Hvx^b=exMJ1@yhzq?{Bki_h-#`hyn9D2x`jjthsM3DW)Y+39*DMwSTwc?Ln)CpY(#{O>R6g*x8K;y+)TJP$ z%|9H~QB$yX)$nyAbvfA>g9Znz5(ilEGE401{?+3}CqBUth)0>ASHJf`9_gp?Amb+C z27^BM?GrrcZi-dqwEWTIAJJ!LfSogI{-XEKk6E{~`n@p)V&H0kJwO`ur3ceAs$Jij z*O;C2Q#2I>Q{Yl9xsZNP1~8{!h`}d7w#DS;pfYPkC`Rx?)z?kCeE!c?3us+V1+=kg9E7S#*yK>gLB3=CjXS1n4A+}!W?uT6`tynO>S0{jDXRVO308) z2mrSazUcnwEn8>dV*rD#YUY>6tlzk|eam3gZvSx;cM0wpipoV?3~r9S6C%MEKFBDr z9JI$)WFJ{l&#~ZFYH0yRMW86uMBcdO7`IRG*{)H*Z}ERrgviT36HQS;HC6^i?*(Oq z8Yaka;UU!{E)ttt!Hd@c6pivdyA%`&I`u-;Z1XKUQ?6vw9T%XR2nGY^g*{*4_`0

    Zq5m( zmO~b2XkDEf@0StBpC5*G0`ZMsH|l?`B8&EBo}cXzEkZx?QY3s@Jv8>YIWRt(aNa%r zPGXg|xAK7osyHU8Y5LAXa zaN3A-5R)c+?GQLT)VeJkizuFPDT|W#ZKn}YnCYne)pj3aZVkA*i6gym*fUb3-DLL9 z!0ok39FE}CIC)g!5c_?j%WQn}2KmHACaZbJ)hh&oYUc9)F2K$b)i>9A(@wJaY`Y9e z#Kp-k74voTlkWLgl%i8)!7gi1GmDanPfQmm2-69`*r-TfIhWvn{D&ww{zF|;exnj+ zuFdGg7yp}lk=BE}LAjMp)*v+g!tp)p`5QQ&0aCmz#ISOoNj4=;{) zyu^^ojdsWgyy=-RN0CnwvJ+;P7g_E-KVK9|sM#_uyUXTZb1zyO-L>^MoIqq#I0PY~ zRj`}UaBI8!#Pa$=F6zqnm+N2mBAOpPz*5!=pa9VJzg*jUQn~NA(6rNM@ILFoP;?S& ze&aK+paKzElEZh>z8-u!pr{mH5K>Ye9C=B4_r?YXJhF)p@M808;Yb_uj4N6 zES%lCaCdZP7AFyI3(GA2d%ug;F?bCfk;USDc)Kw085Ljs)fZ z*WH-LPh(S5|L)5pRouidec0akWkZg|p@X58D9D?Z2K$TUUA5SiMmR z>z_xD$Tox*biiPNs)k3vpWO!Io!hfdLCvd^5Q{_T(Bv&GK!V&RW5fmC@&nBHR{D1^ zMQ?YGw5SvhKF1}+(g&w72!GDGVxI>K6Zl;Z^Xpv4ht9{(ZhW?hq~)t#^srlIMvN~m zR$R%1g6Z2za=0T<>fv|zSG-liZ$hkSSddBb@Md{zi;Tac*HKOutX`m@8%Qja|GL>A z7okI(IXAZe@{HLEC&gWPW2^Ha~IaO=uW0&j({~<`a=| ztd)%s51eqTXq2>lI94T;4Xrh?p5a+x!y^Gsu=<Oir zkZ+Qd28T6E<*}w@x;uvMU>f}+i4|RXu*98Z>=4vfDlh;e09`*Wqt(UF#_9z0X9NK+ z_mmE1h{G-X&A;R2B$s4p{mC8QF3SUY;6oV6I@_eHj1F-eAWGaUDESbUq(LBqP6+@y zz35b-J%E1T3gB(ms=YifsZZcw!x2lz&e>|>Uh&s#S#ZANV^D$(!0URrV=Rg6xDD#Vc13FO5@>k_%lw~#hca~ZeuP%AfFRxP{TKX-`1BsQRy-jQ z#vvJTSdnL*Y!HJsP{Z7ITNqsH6YyJsfY&3wG6pM8kuds_&1#wobsCK2ZF&Zd2QxSX z{DR!SL(Xmh-i#Gjxqj^KET|DHZh13>ru3G=z1+}tWq5uv{yn5qHrC@HzR4e_OD=>nLr4a=d=<|uH6Io1I+!K3yY{|Y|>jXa@N8t-m@At zna2sWpjcm>nXbvXQ20>smi6JjR#iJL_k+0fLmV7b4Od|!?tdj;{2A40|5#a#t%uY& z-tHF#%8D+3L&nhpR*@3h6(RqX)oJJNm?NFo2et3YThy&+)Osqs5lv*VD=(f;wOJ0$h1|fd6XLo(QJY zpVk&GjDa+iEP1aNZR>WCIrlGK2&ViIOwFf(0-Gtbwcp+*>~U&ReG`noTBsGTCx;j8 zFGYgVwf>#|v`xDu9Lk%;`qqA04w%Mb2P>yE8~&1fbE@z9I3;jaKU67@SS;;w58wOHb;?$|Lh0*%`l#Hn3 zmf=63%gkl|6c-F;{ckdZzUfjkEMLpX>&_C%#_lEu0!R z9$9mlEe>ppy%3x|o@Am^L~nBF20!`*))R3cpqcL00+pw7E7%cS3epQf@Z)n=VY>8g7}O)e>6y2UN_1Vx?VSgH)Dk zX1n)>Pe(0U3Cl16k}EOhJhvr+WLH6fAj(|-&k6SQFD7xmEy7C3!ha~?-XYsk;-o1k ze>HKK^+a3ElTPgxyxK7#d6y#tcXjSF_NlW!gk1TeI4mbxUw4jHF6i~ z|A8!Pz_2G-nGe$q0fBnU;hgFwfC}V^W~38*+jYu{QeL#Wv(+)b?ZokNSZk1rByd1b>>i2hPdu`q~5K~ z->O~{5FiC}jY~asb>18!9Lb>IS3Ak4GPr+BpA;j5Kl^JN%Qdzu zd`6pYTxLEmh}0xwkt$XZj#dTBXa95Js1#2Yx5qYxd9HSWzvW7>2`H$W#9jEE(GssM zTQ+**lXD@STY1W-zrE-*Zf7j$G+EIfkm&^JaqhAYtQ$7~0cDL>544$h# z;91B{?KCVL4i?tGl&wfDm1Ui&$`dj%cND%{qPU~@mHN##d12^N14b1m!7$J6#i_%i z$5eRHBJVBDF`SwUmvME*Xh|QPGl83orN6C4R$!%DcrV~C0~gMy9lk!yA6M{hpE2{K zGd5ab$ZiW0aKf>&`B7O=C@%HX%~6%#g>1`f6nPH!pH$@t54)Rkhk7e#vsD($2-zyI z)TMSZ9klvo8Y$wE5$bU=+!D;s3oe6VbwDQjDr}oo;kXR%kuc^12)MeNc2Rn?{@=b@ zW&!C6qc`4{k2pL#de?{d~U%L?WKS-cb3hU+H+e)=n z(sf9-N*!Lt6#j<$4EOMji<$Wj2f7PqZ2yy-5TOEp%oiC?Q*M6ksInr5p1t!s;v&I* z-ej$>AbPh(+2C@BcY%(qJr>P~2y~wkXmxVdYAtX4jw*?Q2A3VamF6&9erw1t8e~Ke z&3XRr)l1z$#l0wzMe3;CX=iKbOzywy5!7vpB7{|3a27P%d^Jy|!txPaP%n9Qn<^9h zSFG*i)pVlL1!o-Kt&c^=-c?!8N zl-*~+%JWe-y~5y$39qYQ^>_w^?J>#xiZG!`^!5H zl`Xq%G5aUSfA1cw>ZyTuP(Eak6cKp)Sn7S~8auijSXX@0=0&9Nt&F2BGVI!Y;4!=X zW(6nDq4{Cmx1b;AO;nY>j;?|!dzJTFAwz`04=xN1#J%X0Ruf*x444_8p5%WS{2FP7 zslHw7AS-`<@2$;64#zlsMWef=i}UM>q2nCM-JgR2^BdQ!ztLspXQ;|j$LsaUZPNvM zl%Y>(-i;%|iU=U#mJ>q>d?Ttb`))N>^vC=`umReOK}W`+(Rk02$k}DwJek^msnQ%?+-zLvH7dyKVz2&=BBniEyyfCo(JQGs^o!Xs+4yM#5!xxpal}Ib zzdPw?V*EIMU zX)WD}zl>D+*^s)-K6ft1d+@R((jO=Th`;*<_eD%~mKZmmUiW^>Z27y_Q$JMdrY#TE z{d-TQkqX69=4qTz<}G7zi-QBVw zbd7+wVj~{zvLaK6M#j2kAqSZl(=aIfr$+<#N9w+8LD*Hay2dlk0dU#@)jEtkzKfET>#!~1f~ z{nAU?3h*KHY-pCsytvk*6V7~d`c3)~X8xPQ^;I~FWy##fbb-*MJ*a?Y8Dq$gGqy`mT>d&~le}C+#%P&gRgn!ga7_NUkTHQhoxKC@x9{;E-mL-R!kqQ*2oo z`K+N^sbIZ>*yvl17gIZHgSe~z#@OX7F!FbxR2_hhy!&pPU}^(QT3DFXSNOFJ@0&G$ zKgN=ms#5yVntG9g6_j^IHYpC#hWrHF^-rC+;2(BAg^It(VHuM7wCmo!iwn%rI0)n` zfHe;H#cg84sc;{BW;u3RXA@mXjC*w7`?9mfR^y3=f*oTwOi1G?J_G<1#Bk0xUqgfZ zyAv-jyE~b-8|p@WyaRqX9ZeOx z-U%CI$wOHBBji9}&DJLxSPZAio%ekVi3%(~90zc3$%-7BZ-p2D#3?}=0ao7fvc&AN z=t0;3X+~i8&xH4S8Tvsp#nz*D2DE>wCcl~95@ipk1b`(GHmq2dy2TDo&4wOvNs>pK zYi0{hWqvQLJNnITExyTTu+F^4=o0`~5AhCYfGj-GzW3(N*atqqnSfPBT#p#RkWiBA zut*QTRivJz?mKBiS)}j{`}u$ERufx-^ZZpPr}Jc1#Homgl)n*! za&uBbrH;S+*MFPUyH&=lKX6p4#sgk|ISVFU@63ujF>wYkkOQK%bgOHbLL6XW|Bd)G z-x6-t))gmwmeu*yS1i)glI}kD&?K{I-lh7?An=muM8hkYuUN^h(h~Yvw+EeDpg@6t zqxkvR7MBu|Rq_+W2n!k9Y3Im$z%0eDT4-R;SlTX14gpk4`dQ7W3Hbov8R7LN z{gQ%6gGo6=34RYy#5)k$FiCv`Toa)hsP)$mBZBdlriG=COR*Go#zTf7?j;&&Y5D19 z(ajZy{6O&QR)&XnKnmm;*?PTLY!xdPl=tHPfjpVCEr(z+kT^CLrB4tt5XlI$s5br}s_3T;oHmQ)|rX}w{WADb2d4G*^`!b7cw z0rgnMI-5exn{v!i;$T?=ExAj0dZYNP@X8n0z|#mGjP;!$eD=d%KcEs{v@LH}eydGN zVXx!0|7)Awo#4RYLkSskm@63IxX8al?EH=jSE|ENS7_^ryFaBz?5H>!UW?E9QiYVE zyQ#U9wdKJpr3Ar=kIlIO)F?Hi(`L^Vq$!03{^zMa77Bv(@WeMQ>#r4)PO>-~3``0k znyhoS30_I2hbRy7CVn)sWQDj=IMAx;!9ot>0L#7Kd<-%|0%l7O7n_)`u$~&+$bTE$ zitEVw_?e9qILO5lf)FhSxzGDTVh^`aH6DWLeg_T!_X?X2!B$ zjtR!N>wRW#&h#G3Aav1#)l_Th^bEy2CRErtZ%0iCWWDTRhU*n!MXG_D-V( zjFr~Z`<6gZfVkS*v-@NrtfR#m?$s1Dw)e3vr_x=7p(lq`a09@I)XUw&;n+VP+OIOU z(v9ORRwoi@uj4n{503j!9OCR7Q{1gfl36yT22WMJd7^+?&Z7;sDweX^Gn8-G8m4Kj z$phK%@+-(tzC@Y?m)g&#t8t9Lx2e?!AJnfDd9Y@d%QdumYY}Ih%ea17(pe2esBn4i z@w5d7veTl_R-Mb*_}>sJMOzcLCy5`l%o4A@hlLb>L?M{p$K8|sP?NF&% zu>J3iOhauYEPI=4^5^f)s(muYRk4&e*_<95K}pBzc92*>8~m_TgtlUUp4UAV^cG*! z!l|~LHEGo8U34(VP5n&L3*bycLPm$L*B4y1Up%%w5KQ7<{`a9Z+zQ0D;>b58d*n zf*e(4sj)WLvc=Jpg@FcT4#fS_J&lnVXT{e@9|cM%aV#FNdF)W~Oi8=GK|gT{C+ffe zMmlyAiXlw-k&$dyHE4j5)w>bQL=iJR^8W_09$;&4G-G_moLG3(A;yQk)v^9RxCJ&a zcGJeOFIDGv_GJeuc0s@KLBc}==o0eE4ww)-;Re#c;o13fls_RXyJPaTG&P1+*?SDi z4tFS~#tFhFnj?723=>Y$^+OKhfakuXl3=o}5=8qX&3bB-Z%Ru~9P`z}mGXV#`a_3S zrVUe_+f#lZRPZ^m4qb`$%5&FPE@c*$&<{P4lqtJf)fYKR>Dsr@y5R_~xOp;KS?F4J z>#ZhW)RjwunbAw-H4a)dis;hn&jrD`J1agwu;^jS9F!Z|m7zkK8zZtds-)O9?8W?A zO48ZL5wx`&plfKma!8|ajNH_t0Q+sH2Fh86qn+iD(A?OnZ$65dKfkNhyYH%3!LhzW z*%SfZn>Vq{XzGmIzRx6e7OhemEIy)~nNURfiN-3-Hk(Q+=#g8HaOln8OQfJ=fiU71B|$fvSf$R!x+E}Di|PI=!y|4z>6NqIxVrv zjPPwNIF31FC`W{6cIXz_a(Xw{$=+?q-X48B>?Dz;Zst<7RHdp(sCfF;#Y#D>`GG~06*{Dyf zp-sTTHzCgCcC0FN;Qnf&yW^#+iZHuJHO`17jlJ8EFKz@12d;!71xZ=;<4bleE}t}k zJ8CNI>u++y*CK!;Pc~B zx%+{f+ugL{Ln&};qFxt2287E-eO?L~0Y`ESLR-a!lIfO`G~Owg+QaF*K7YF#KA={q%bk zBZ-@EPr6&%D8tMgSg~u1w%E(8gtJ?zqd#_+8C-;kwpsp;z60UDPBZqv1{Dbw#^Y0v>~u-0!HbszeZ_erV`|C^?UElM%A711&3)ChRd+pX>a z8297IromSZUAe82llScxzihlUUE8KfPo_MW?tyYS=2|98wdGIhhotaWh0R8zW_Vp@ zVGoaRY!gzzJRwuI$*eI#48uZAFwjX>&?_7lYCbK9=;(Q=Tcq9>AZ0i4rNMU}ltLQt zuDkA!B2P?mh32)ae#@uLK*n&vzJjaG{`*aY)_eLdbh_Mm^|5=rNl8YA5sln%EVZG9e<3-bKb$soasuiW=lsH z*-%!Xij6UA_$(I zkNVcfYk+5Gq}nMsOU#1OCZ(RkRSJR-GV`^%77KJ%1ZEJMi<#b# z7?_-{db4FN?!|-W3}v@1)i7YcTrvE?lc}Z`H5he-#oR;^cm0;SVAV1pW#emR|Vu1IjXmaWPow^s{6cQi30l92Pe-4Dlepyq76R+Drttm0S9~u2n zenss#yQk7YrZEiF*!J#{Y zC4QD5eRN1;V3mEFJS?OXI%q2~)Qbi2e0u5EQqyVJuO0A{`OY;V`J^Y{jRy=E02ginLVX#XxNB(fMWOQ46mnS`CD(KNB!}? z2dlWUfzF`Z_@i_9)~cl<3D3EFVw<6~y|w3I!8ve5or3{eETP<_XRl4EX3(s&wAFBt zt%Y#TUL7S*YK0a(6~wv;%K*2=dA^PJ18gW`jDBQz%hZ(f?< z*+7&hMjKo@wBg9lj?ui+KzbS8l(6F-D|+zuBU$d^Ks;B3(Bns58AOx`&-;rkq{fl) z{iS^2nHqSs#4krBd;!8w{M8!v1dTjH21ew3QYgeA{l54;*c=pG=i>35Tmko z@}_TOerRiAV+=`7w=M9N)Ms(!pR%7&5?;fGLTyVh65pG3A}x9-hPpQ;V^yFEgxd zJu;&|q`;xo>6IMki}T!q&GO(|N#3tXX-45VkQgSBg#o?{G9wgRsSdC1Ca`H>;>j3& zgEAzsh2-AL%OHiL0y&>21MQ_TzvR~xVA8#t8*%oOD9r9fu(|+%!&uduYckXiX85!c z@oKaK)x>vp$X`=@zC5^?KCj=hUcgQ%rN)210OToGrM9Z^mbW2!UDD$j_RFe39;XNU zQ^WP1tWf-4z#fZqpbkcB*_@D9K?VG4?4n#ca=DTe-tpOZ*LaZU!jkAz%dhZSW2i6^ zuDZFo>EE&4JY$aaE!pn@y2=PPE<2>P_>w_vAn5f2v$cEL2dMV?O5=u!`u!%hznj^$ zUMAO(KLHYs(Tg8bT|unsNbo@X1PbW*exbH7`Ii$AYY(UZx55tR%yEquNK5NTb#%VT z0Mcad|Na*^hrlX9hl%JRjTB?-GYf)gkxz`xgR^rbf{WY-{Gp$DuO=t30Q{BW0@KZC z&NO417^f&Vp?J}G1EepYd}EDl<-90kdV!SV60flQiYE9Z^SMIm$(-`}T0*_Hd!P{> zZDnNcG$i@i#F!Rvuw+w0@uQ!_5~ZNR(!dB=0R>Px0>#wEoK)XY%>WbS0Qs)~-W$~C zQP(Ju;U7_I7k=0Q0I)4;|B0wGr=rd}ohF@yAHWXl9X6KM4wgF{|Jh;dWNYJOz0<f$o0%!mj1HzS3{g^4q9mr896I3EX)3qqK;?Eyq#A4;G^x}0N>vw(Ey7pSvTG!rl)Xl|l z%&1AD0AP%>lfWH-hL<#8sNuif@QML|amCJpMXSS8zSpkLwA`^?*6q7BVJ>5Fk!s=X z*h9-av^avTe+2d}7Bh6;R@SS{y2RRCK6&ojbk#yOb< zDk)jj-#U61(00D>%xPKi@0wYMExtsrIP_$Ilr2*iQ8&mZKS>I+YH`cL$@4?RX`%0lk z)gUitTKyyq4eG)A)~sDj89^ktOXb%tMt?1X^hH$4!;+3tC$mu4D(xZewzJEk z?-2>VUFh{0nLYa3U~54(k()QO!NX}Ha5}fb%*rXrn%8Oi)fYy~f*%S(cqPmAcS=`f zOFdN6kIl^yO^aE!l74BC=Dp&59&sD1>In4+{dhlaQ^h!y89Rs7`Pb=5jsvi6{f)$` z-Ecab&ryJHc7S=>N0|CuALH>nJEp8G1blqDR6HWTo6R6M{N2Q%*c#dgxn4GvKfliH zbKq;Rsb8nNBpK7|pERF7KZ;^^q_8#IsR?IpKMqqZK6gD_gTI7s%x|}~_OdBSA%xkl z)uSle_JFFF>nR_a*Nhl@>SvC_N>a9Au1~*8rr*ubT)%oVz7}y?t&l5Gul#(4xb1Lw z{s(w{_xZz`HIWk}10X!#py_7Z9&!nU4nKS4t{)Mw1()>|nwACC~K1)Hg`RE2tg`JsZ@TqEz zRcgj)>T0?JWhE>NbOGQmUZW~=Pg_^P<8M5>p(;OirpoH^05VHbHpX_~zjF$}&M=0v zxwkH95T70t*~r)Yu%&il%_YSFP=`oa^t%z5FYKH?irR214fr!cZp304T$=(Z8Z@W0 zOp?libnD@^Us#Mi5uP)4f0yzV-@jOxW%9OW!0(x~^^!_dxx%%@#+VIp`1BRW9BvDpgZwLS=sEvockw@N=B!jj3Jkfw-Gd|9ntv==s>i|0Q$8Hx%v|cszDPo z?^J3SMWvAZ&FIsUrauiOYa$_`L6ZWf=UsF!ePCnQauIFc zA#6}ZG$|RTDp5AcUxesc9*jctwh`e5h+bt;3vJIr{HHv;? zK@~-_l`w8Gin#W%D7uq;ZxolK$R?l|#bqQ|m=JCjD0-0m1{6auen_rZ39FG*28@N` zYLdSV#Z@RGnX(L4BN;S`aGQnVHYW8N&(xWi7e0=f5#_!Y2wk3{DQXg0a9v<0Rjm2g?1F#DJH} z!N`OLyVxLB_BG&sUIF}>G%EinP_`sgv55a{(M2TObA_N)JdpU)s5vNxF`;w{4L%t` zZUO_g$Ai~67AUv=cASG1dl~InF(A&E25lygi=vw~A-js?J|?*EaMTFqont_H*-gJr z_%7h*e5~cDYMH{i`6{xl>OyKC1B}suvPe@HFf|ptCZOfNu%OZME++V(_}U1PE})3s zSty#(sAo!%0TYxWb_OJcNRH=$7uMf<6n`>cEsB)}AVgAl5hM&0*JF1=v6%s}xHM%u zieo?$wniruJyFDU|3(p?rUk`|C?Yuu#SJuSCz4+okb$JK2O>G|0!WtepxzQiB$fSg zB?%sJAlZkb;WWw^kdSlDSDBamG0!Y|=<(GvPe?3~+&RTNVVl z(x`C)ko_e8&!@sabGH1yvyyYGK*cRNAUl7PK(t68@)n4E1hzIjJr+xIo+5t^Z+H&R zPMg(vOx!0ZTEJ_Z!RjuJlnzPnk4y$kUAcc)q5O5CV^NjPaG%{o(VG{JMF$NVPqMlz z#QmQNM@EJNviC;_9zB!?9+{dX$oUwtTu>AI#HX{Pui9iq(8e6yOD(+kw!Uhq^i^Lc zud)4-PGEaqwZRwtGrfuMup}V-o|^T0|FE3h$+Z@lb-FAmn^ne{r6}v^68Be%!}sTW zEVxtsOv+RmtgoTJay~A6xsWis-Va}}OIFs= ze|)vdH(%(H!*!9D?U?si8ku`DAbT+LT;OP-+opcytBTi-TA#ieasN+o_{#&$baN8tNZOq`LBNkWG^sYMlJ3gUwqS)=EZcOU;Q;? zm^Bz>#Pclg%(1WQ%t^F~_|Tb?6({y}EQ&f|yTm$06^yqQNRNU3eayvXfqI#O`?+~2 za&UB7oe!>gr$F@XWr;MS`kUQQMKzgw67=(@cfO4}DUTO(liM5yhkngHk~2ZC$J)lj z3SQ!it%qyyhN<#Fg=3LVQ;YfehE)nrs~_))vgH0SgPT(-e?LCZKC(s?itC+Ovs!uw zHpZBAuZ;MN9<5p3e)!m;oR4G7r;|y?AvZc3uG?;F3d>3f=Em&N)%m=BPKU}BuL$Wy z(9f=R4CssD^fg#JdaCuOaQXvs*vW zUwT~+Ey!vbhzcb@Yk!5KRVS}DC^j;nlbcst=v$|$|5lVS6W{kIC0qSkHDagT zGuU^sT6fU-aKgpVM|VJY_pXj@iRm;L-rp3~m%P_rGHv-SdBnE(;TvP9t4FB{?%{$W z?#~E^fOWMr6~8+H);ch^a>7zh&koT+`U{1fcT*VGuGR0@!oTM)J~~LAYM*|=@x7Yt z?~1J+b*}c&6`Xj@5`p?&EAzCWz*`%<#Z3N z8Y-yqYYM|7Xhwlc-x<4sEIX*wr+vaWEdkN?Cc0Y5&B=F4=mSH)mYAr?K0I0Y{$aON zt5)y$<_MPw;8&4&Fq6}XGg{5NP8MycYaLlQQ8X0gpg6uizUJoj$kL>U13RmlA2&LV zD8H@*e68W#MOLX5*cM81HRE-E>rnQ2_3b0G8hvZ5uLVnwruq^lcl1m&sQaeHi`EtF zIbPE-e@;hXHm0|fS-0X#v|*rMY--QRNU84Nu$o(~P0hODFVb@$JjF==)QPi`HPhNm zLi^8|>*W0o+|>0mYI66S4*R#QOxt@2)^5D{?VUT$GpV_L#fyYLIaHL+o!Za5#y38> zL+0~6Xxcp|i=y^Eu^O@XVa4U74W4FIE8dVfDobzWN2tUPC*zl;hJ%4yB6^>YP<5h4 zk(OW55v`?XUKrY&Iw|kBpX1{U@8aai3k_ZRIED{6T5E$W{l4#3evxLZAXm4ZG0fS& zTkvS})9RrHakxVKJ;rJ8`d!yfzH^gIbk045aC?2B$#Tsa&&Z+&_MiO2vc6d5U*5Dzj>!m1u8r5x z8;YN07B1gkv{Lc*<@d;I)!zc*4bfPa=)cQFf1DcuhBC^oCy5YXUd|_jlC;k+&rS3+3to8Alfbf}xfoSa*cEyTG zc>0vZisi?|M^1bYg*!KLkJn6`*PfIeR+^c2DR1r3gE`M)|5}8_-}z#smDPRSF|s{X z_u`elYvS;}hUW0XkI}AiV(h-&L6OZ<>fFO~W9qPBCRz9AZA56vz;#E_8@WS~pT^+G zt-|3bOP=01R*r5Rv-^f&v#TPu1Cez4n+}b>&|$WNBkj`KOXuJ_{}OmjoY0PGhaQ`x$6pUh0WV%(f~WYk?bKB+Cr8g9mMwccoZX7t}v z)=S(W$6z&yU1iM={DzsJ;RG#JCVTIOWh4?ld8>%r6uY-6PUGQYhX*`wKUH#MH<8fo zqE;`8c$$|m8}8P9gql^UyY;S&p^Ep|z@AsObay)c*8;A7w%zZ6)B*cp2Tx;0`QvW~ zM`C+@;a0N*wkL-Un-W1PbUHsdv$HB>Qj^Y1CRCKwOXvGSX@i7FPYzZ5MXRT2=utN7 zrDd&b8Owu>ugy>F;~lg1rflTk z2q>t4q{mNs7h4@D5@==Q>WMYWcqavI8u&V#>vn&-7DS&V70b@mJ0%O5S=)593M<;k zB~hY-Z8+bE0_2YFw=Mz#KObN5+B0hWs%N&L|C7Uc>dBpoRgBoXyyoCDm?=%FZma*w z$X%))S@+h^9zi~K$-p?J=OGK=wkNKq0_ z^)$gjY-sh&P%o4B$%k+oA*Q@#J22n`* zH8rg?d|E`iQ)0q;{;NqdVr~P}9m2y2ln; zA&EHoX+g$N z_cn6@U8#p&b&t26<7<0Ot6mvu+ejTCYRcy@7X^6>6Xs+FIko`%R@ViYivUSog9Ygl zKcMl)skL77l&A_PzIl-dCYuSxKSe(l@oQp*B6~==^1|A&1o4g-rg}e?K_3y-PV#i~ zgQW)=LWqS6FdhkCTtzS>YR{7ulM(;%!cykHqgfN#y7AqSMjs`YyIK=GBpX<;_nZBQ z3JQ>zb8fB7mbH@fR*E9ZfJxh1MJ=lh33&+^cnO%oh_DD>LxTQRn!LZ!Jrg6p z07Z+)H#uvO6O7zw28I|moUlWEf2rRad~utj4nahaB#}ztD`gTgb=!Ni_`VF=vt>VG z1UtGNc87}$^6E$_v-Rvl7SAn*9)y>GtU^smxgOPygE>E1Ci0xtTjd6|cr4r$ zu(nQOh`t|>6M!9Rr%tYC=DbCgs_a98;8L44#%v$m*&j8ROE9JfUcMwZpc}UEx|7`! z1!<}@Oo@5cA2BE%oXwz}BWS~J!wB_50$bgn-m9{_UDBj9^0)xMf04POqJ;NWY5;Mu z&Q(B#Q0Qq=xWxW)k*-RFir>V$`*5vrY+rYCe(%kdO4h6?bLj*oV!&uZIOALG1DE$~ zPeR2n16&`tRA*)RPl_pIOE{lm>{R8M9+w>NweD{_-9Y6r)JsSmq^?wkp0kqOS{W7b9;ML#J%G3zNO+13bBkS zc*`C&dBP_hqgj)0B8u!ejTX6W^dlq2xhk)3PHf)&GQFLg?o+fS`ttk1#m4}?0Y8zG z$+!7QvXUX^iZghd0ykw z+>jM|fS-_V(a?k6YB=oia%RwCW=;IhnJ~2AXttUPE z(%`$>Npo|?wHejLkJeT}o0WGZ$uci&a7dY-gjOe0Bxz)np-%I|e@#(K?dV$d#Yzdz z%!m4Z`AVT`^6Sk)Kr**{1ibq00RS6XN@_FeoxxJV*+77wXJa?BS#i(1o{Jj}A&BP|%Li5T!^a;huJ9EIuz2bU1w`NLn z(^Tov&b9A7+dO&uHRNp2qw?65hOCU(t;fOL-FvP&zqc_K@4KIMH`uZ__Xn6NDp@=9 zYPQBj;Tq&v69F{oe_n2qA+xT<>tj;mcrb{PgS@y$C%da;Kpubl`2`P|k{vvZpya%^ zNYHM_6%7d`aFn}tLDx}u_2_Z(^L|1bi;unAI?uf2TGu}G&VN{SbI)EXJoz47?D zj7RMii9y?&g);NSui6J_kG;1(se8W=vbI2WXd>md8E5g%^)s`v&@96%J!aaj`x-I* zb7DcK8>93V-Fz|;{1of*bC0|tDQ=Me=dIV9=>=|e5nM93acAqXR|S|%>mPm?eLR~e#pZ&2db zekEMaOYcga_$`X8#Q|67I!^EDGnoWk(7wopkpUNnQ*MRLY;M1VZDsDs{*>%a3l4=jKfNvM$^Z?VsD!N13KnF<48W1)^0Ji5K&L zI+q{Vk@hfED4(0m-JOx1sDCvly*aumZ_&Jr^1|D$cpn-Ac3v~KMGy4Oghppz7BF2* zuj*K?+D|S|ANkukha|iF(SA`5s-W7|4JesL-irf9>}!$&{aq%#I}JZK?Mr9&FAYx4 z)iH~2CimB4rQ*g;m=obJfdUse-zVRgm;Pe8TTP#7iIm*G%4o17`eiF7)br54pho=p z2bul1oaXd^rU*CHz4}#`m8`wa`a{%!WbbM<4yVg(>hWWe&Z+S&Wq*@dR{-a8b|WpJ z6fQ8PQ&R2tXb{6vzw6o+R#`bXcYAxX#key+UU-tKRawgofBf5XTKPYD`HHP&aWT8- z*5zxe@5evZp8cq6JG=QhgJklNZ*3ZA;qy(Wj6mvB0a$PX>#Wi&v=(e_p|rxv^$A1EKLxiGloNZpyJA82#Gvf#K~% zr}lD1g(&W{E%~{g4__M(*Ax}slQe!oK5$J*+J4|UCTOek%!h>9`QLj5)fhHF#XcFn zbvc|}A|L6tAkTKzR5doq3~?OjPQNHSsB3LTL`=b^E%~82V}xIObp1Dj_Ng43FFD6% zZpZ-rex>hCjGOh{!f{37QhQ$-nB!XNNIwOy;s*x)kw{XDvN`sY z?31x7-=BT<@tXx65Ep#i!TL`4;b{yE*-Rk+6*e*Q1tsJ`w_IP8vb3st*Xqw-u~ac@ znY+Xy8fqoth)SW%_MvX?a`Y_1M`~g@W>W6T^}asgXlVkHEKNe--I>CFrt$=zEkvC+ zRC5hK-_*aAL=+U@!rtug`rg?$Ty=hpBT|4L@sD(lT}!}R9Pp<5%1R=>g)ujIQA^D| zndikOOaoW(+c$I<1L7@kju?uQLTU^?=BV6~{CxTX-#+gyRC`n5IX^be(L&MEKP3n^ z#q~dPB*kQ|=-}(;i1v3s48J$Ktsbl5zi%iCI_+LI5qZV*KI}hTkD=dxfxXb!lEE2b zw-g?5<~Gv<2&6$oAiqoe{E@C+?wXC89i z4miaJ6!9mNAfHBYZRp2qa)UMy;xGaQ#P0@5$%V_t%n2zHczi#rz+lu$AWDM4Am9%3 z|Hc;%G)}yh z979)MQ{H=^gLUmobFURh7*?8)K%ac?q6G+R9CWY4_9*8KXNcRLYiI8h+o51=Ena_omp57{J-NZWS&iRqL<7KEMFc zljd=3^Z;*f#B*RqhcP?4wM{CxEEY58ik{RTrUwW|l7r2SrYQMtT5=|QVEQGEtcw3J zM*~&HL`n06a)Cr<*o0(mP(e{TbmN<+PWhf0p_i>WK}S5f9F>H*LAc2l0#TKz1&#Du zAA|s$m8JmWL44Icef^^C3hpe>k5i_F)50r~nnHM6a!Ff#lJ_1XAWDi9UWYqh_OZ_I z!zrcHYG78gvmy{@Gj_l^VW+~AO_0Lt^ylm8ZQ&fe+40bO>HEnEH-z?V;^;kg(r~ul zGXX(G)aNDn6i~SO`eNgo^pfbSq~6azniOn_#!_OUg?gX0gD_XEP!y^ZOl|cJpc#q? zWrl_%ezJOIso(Q+Hs=Zv`oR{E_;H?@T=YK4RBGm9a5rt|810u8hFY$Bh$5GKKTyFY z3Y0GNDRFNt@c9KQ7HUKCMtnB!KF%|KracLuGQAOmV< zXgtriJ+x6`(z#-`yKC?;I$%X_&iNM)WBUgWVw6>J8#V+u;^GSASSRi!^M$)UuQ!?-^%M{AnsY~z(M;9d^2xJ7p}VIYowepGv|Rl^dwl& zzWPTN0!yU^iZeH|j>gK>Rk?eO_^B8ZHu<(4Si+-_yz^L=DOcgwoZ-Ftn7bAyR0jI9 zj_l~2TEf434~QkT!T2PNLvf&S<(ur(|T0p-_z^i!i7ZemECN*w*@P(`+sGq}B0aAeF6+Q64b2+0- z*C@Vmw1cD)OWEvRm|cX&b8?E^9^Jo^!b-}~t^J{uO_n~NRSBSQjiFYjHGFYFvfTGH zH`t+5@4~6Ct|Afs4n-NlA(^U5d@56Sj!y0GczwEWBntR)*)e`on@HM=^GUxJmjY)6 zc9ZyUpF>HMV0i?}Oua#Z3y?hzbQ|;mEXGgS)?j^h*lJ@%a>?b|&oL<`NXPlb*^Ru4cjzoO1r#zf*hIJv{WDWR)u|3Mk z)0@JTsbawDM*HkQ5fkC!_N_1Ifao2s>cLhS1V)cvg@$ggqD{m-KN*+P18?%lk>;>u zuOLBNqF*1=SKlLgJ)RQB-zZq>uyq3Zs!(S?@Nn#1&oCb_T5AxuJ3pchhYiLa~(S*69O}2O9QQpbhl@L${djWd@fBr5gBZ6h;jc5I0CD@jxS|*}Qt+>BB>PYo*xOOh z2k6G>4}oqn`_be)P^gFW04|69XGLn|C+j`GX7}}ANe)Bx1fdg>&v}c3iGdqU&LA9E%|sd zI*g2c0!5Zw{Iw9s+Ppoal*Jqz=Ck*sM=zxpAKP7HsNLK`u~Ol|^|P8x_+7 z{_eBs$%cT}z4+a$;e#zu?w6i>xS|s#H5=EjK*>3A;jH;C>2W)Ycatd&>Vr9tzEJjU zR9N>?1}yHtLwR>T3~vdSkc$C-chY0s5}Pi0IN5rGbY>HZ%5{Sm;*m-|_rnp8?6l*3 z6KO{6419pW?EAb3#YH>zSB?A%ftXrK_|9 zBaxK%ae&9P;FhymIhrHChIGQmh3qtYV5L_yegO&M|ATy@2SI@5>jO553+Ffa-l=hEz=97{6@9zNLR{N15^AV9ktfSM>`>?W8D}ZP1yy=9ZLZF-B&QGK zWBLd>{=Iqy)=&*5H-3OO>gZUVsTydb)+P7}zz6m$B^k8Ugm3}QqsSa;_4jFZSW*yx zUmGiUZ?xw#SvGc5tEjFI9}-6a$WJ^C!Gb_R$qOW$m%?&BB~vny6VVi;<7hHD5Cd?{ zrA7!A8_=n5D02bMdzS>Yx`x@Bz925Z+b0Ta+?BHWW$5`S$7~4-&xwBhZ{f5iX%GkY z23j!8mb?_u!HH-S2EKM;3EDR8-SZ?01gbb}1&1&?FaTbPHx#ZuiXZj%U;xmJC&}1-fqB%uJzw4 z#NHUbI~Kf{A8o%udk$F5m4+69GyK7kH%Ws=fySTQAku-22KJEY$(r-62r(84++4$5 zL&$&>&G0=&#LJiZgvE$2GIy@Iy8BYzwpM+l7k_}&HHB&C@Lp29g0cb5%1evbu+>Z~ zeFn+bDyu@w+-{ziF)>xv)fcTJ2%7WZTTsnv?$SnFPX3@Z1FHA1sg(}jnr)GC?CzN8 zqXkSm)&I~iQI45IDQFsM5Rqr$(n}w^7p+mh&WPl~605BKR-v+-2;*7KAq*jk7m;v} zHElFGrM+I10*pD#4R38>&xJOzN9f<(PnpZ-Z={cy~X*xlLf^g{ShXG9_wX{!E^35Jl-0ZtGI zEOEhM68fX@AsxACJ8|K+E=Io|$t(87+O`+-@@qpehuS#82mAWggj~5nyg-Le`2>$P zYZSAMh?J3A7SmJWJ7p^L(ls!jz6mx5dAJg2O-tZfxule|&UjuU>Zg5vQB&M`>UOkm z)ni~%D(dz8^Il>k?mGw^QH)oZ+|sE_)D?zL zrSWM?fiO26%)uIDBa$mvGX30kwYZ~ldyKz65G;MHLFNjTddMZ}qCyby1UKm3eSRAd zSxY2EX6U-`4oL zSO;#dbv74qk~+kY0I0aponQe|cMROtP#$%<0zEQJ!+fn_MZG;Eu68c2cDs@mHX?$C zlOrStP$f{gYsKrnIUU;hQGQeji~x^{C#iT%)GHbsrM^xT#9u$UaAU;n*P-Fpp(*Vr zPEvy)CN~+J@7>MK0aJ<}OgoKas`3|ZAD-{4C-1OMIH?x49}q(Clo2Os$X$lxz`Ks+ zDMC^jNY2z*~iw$N?t!g5|Rlg?Fm2JbikK87?Qf-h{{4${Z5tbM%5k`Jh-4TXcaWW!psiS0&|4dY1#4N#Nqu_ z;d+c1SaFfr4ObYYcsP{jQ;tM`OAsd`{Xoi7wB&y$PSj-oxVrV@{>?Lt-MKQc2KrX7 z>SM8Bwb!mUC(49R9de8Q~fL&-&uFwe~l?MTI-+9pO-q z9xN=Sn*4mq5*H8{&u+y1WjSL+&*Iu%(j9!3RnhUm+OsaQ&vw8UjAW#De!SixyJkl{ z<5nc*{|tLnbny#p zRh}Y$KQM5>8{nH}5_of+gp+uP6w(q%ewNo5pUp4+_>f{{KCvvmeTKpSEQJ{*pUl*( z{AIy342OX3uC8KyMyw>LrKbKPGPj4k+;BV58vb|qmA8?-JR~unY22si*XThXj2dBN zqvsNHHK;GVKxblx3uur1cRb&RNxXAem~0_zy2L{PBOJ3;FP2}S)vyzCs0(duse#e~ znk$f(@v!hrGa=v*3?hQ}=P4Iw&a?xrV6P*v=!XFX5`Z`3oidrv)R&mk^Tx-DU@mk~LRaiQ#Xro|OYASztVx-48}+LXXndhR0xdhVzM01h)0 zf+}Uk7}aS3^cJq}C6=5=lUt93h-wyVaQ<;y0qoAUq!<778!hQQ8&6Dk&B3{Nb3D7N z1N;LBAl#^ea7oE~^x#&DlOUrA5)WUv@Y)gf>-_kK0C_&w)T1vtQ)Ff60_*qu2_aCF zLba#h-M%S#k!@iv&C(}2COGRi9GqE`k_EC;jB%S4{~*i`6%=02vTZSKt{D%DvCFTC zc2ng%nm#&t#A4e70;%)9Mm;9L*>`V2jhx_`lj4HXAX49+Aq^ARtH`N|_T7)@nq`~w zoc@QpP9vdJU#5hd^}5U~J(?vM;~$GN@WtE%k!A1}Bh_$gO;ZcY5Fxw?M>Gmz#ReH6 z1zZ}Xnz9>Hvx^b=exMJ1@yhzq?{Bki_h-#`hyn9D2x`jjthsM3DW)Y+39*DMwSTwc?Ln)CpY(#{O>R6g*x8K;y+)TJP$ z%|9H~QB$yX)$nyAbvfA>g9Znz5(ilEGE401{?+3}CqBUth)0>ASHJf`9_gp?Amb+C z27^BM?GrrcZi-dqwEWTIAJJ!LfSogI{-XEKk6E{~`n@p)V&H0kJwO`ur3ceAs$Jij z*O;C2Q#2I>Q{Yl9xsZNP1~8{!h`}d7w#DS;pfYPkC`Rx?)z?kCeE!c?3us+V1+=kg9E7S#*yK>gLB3=CjXS1n4A+}!W?uT6`tynO>S0{jDXRVO308) z2mrSazUcnwEn8>dV*rD#YUY>6tlzk|eam3gZvSx;cM0wpipoV?3~r9S6C%MEKFBDr z9JI$)WFJ{l&#~ZFYH0yRMW86uMBcdO7`IRG*{)H*Z}ERrgviT36HQS;HC6^i?*(Oq z8Yaka;UU!{E)ttt!Hd@c6pivdyA%`&I`u-;Z1XKUQ?6vw9T%XR2nGY^g*{*4_`0

    Zq5m( zmO~b2XkDEf@0StBpC5*G0`ZMsH|l?`B8&EBo}cXzEkZx?QY3s@Jv8>YIWRt(aNa%r zPGXg|xAK7osyHU8Y5LAXa zaN3A-5R)c+?GQLT)VeJkizuFPDT|W#ZKn}YnCYne)pj3aZVkA*i6gym*fUb3-DLL9 z!0ok39FE}CIC)g!5c_?j%WQn}2KmHACaZbJ)hh&oYUc9)F2K$b)i>9A(@wJaY`Y9e z#Kp-k74voTlkWLgl%i8)!7gi1GmDanPfQmm2-69`*r-TfIhWvn{D&ww{zF|;exnj+ zuFdGg7yp}lk=BE}LAjMp)*v+g!tp)p`5QQ&0aCmz#ISOoNj4=;{) zyu^^ojdsWgyy=-RN0CnwvJ+;P7g_E-KVK9|sM#_uyUXTZb1zyO-L>^MoIqq#I0PY~ zRj`}UaBI8!#Pa$=F6zqnm+N2mBAOpPz*5!=pa9VJzg*jUQn~NA(6rNM@ILFoP;?S& ze&aK+paKzElEZh>z8-u!pr{mH5K>Ye9C=B4_r?YXJhF)p@M808;Yb_uj4N6 zES%lCaCdZP7AFyI3(GA2d%ug;F?bCfk;USDc)Kw085Ljs)fZ z*WH-LPh(S5|L)5pRouidec0akWkZg|p@X58D9D?Z2K$TUUA5SiMmR z>z_xD$Tox*biiPNs)k3vpWO!Io!hfdLCvd^5Q{_T(Bv&GK!V&RW5fmC@&nBHR{D1^ zMQ?YGw5SvhKF1}+(g&w72!GDGVxI>K6Zl;Z^Xpv4ht9{(ZhW?hq~)t#^srlIMvN~m zR$R%1g6Z2za=0T<>fv|zSG-liZ$hkSSddBb@Md{zi;Tac*HKOutX`m@8%Qja|GL>A z7okI(IXAZe@{HLEC&gWPW2^Ha~IaO=uW0&j({~<`a=| ztd)%s51eqTXq2>lI94T;4Xrh?p5a+x!y^Gsu=<Oir zkZ+Qd28T6E<*}w@x;uvMU>f}+i4|RXu*98Z>=4vfDlh;e09`*Wqt(UF#_9z0X9NK+ z_mmE1h{G-X&A;R2B$s4p{mC8QF3SUY;6oV6I@_eHj1F-eAWGaUDESbUq(LBqP6+@y zz35b-J%E1T3gB(ms=YifsZZcw!x2lz&e>|>Uh&s#S#ZANV^D$(!0URrV=Rg6xDD#Vc13FO5@>k_%lw~#hca~ZeuP%AfFRxP{TKX-`1BsQRy-jQ z#vvJTSdnL*Y!HJsP{Z7ITNqsH6YyJsfY&3wG6pM8kuds_&1#wobsCK2ZF&Zd2QxSX z{DR!SL(Xmh-i#Gjxqj^KET|DHZh13>ru3G=z1+}tWq5uv{yn5qHrC@HzR4e_OD=>nLr4a=d=<|uH6Io1I+!K3yY{|Y|>jXa@N8t-m@At zna2sWpjcm>nXbvXQ20>smi6JjR#iJL_k+0fLmV7b4Od|!?tdj;{2A40|5#a#t%uY& z-tHF#%8D+3L&nhpR*@3h6(RqX)oJJNm?NFo2et3YThy&+)Osqs5lv*VD=(f;wOJ0$h1|fd6XLo(QJY zpVk&GjDa+iEP1aNZR>WCIrlGK2&ViIOwFf(0-Gtbwcp+*>~U&ReG`noTBsGTCx;j8 zFGYgVwf>#|v`xDu9Lk%;`qqA04w%Mb2P>yE8~&1fbE@z9I3;jaKU67@SS;;w58wOHb;?$|Lh0*%`l#Hn3 zmf=63%gkl|6c-F;{ckdZzUfjkEMLpX>&_C%#_lEu0!R z9$9mlEe>ppy%3x|o@Am^L~nBF20!`*))R3cpqcL00+pw7E7%cS3epQf@Z)n=VY>8g7}O)e>6y2UN_1Vx?VSgH)Dk zX1n)>Pe(0U3Cl16k}EOhJhvr+WLH6fAj(|-&k6SQFD7xmEy7C3!ha~?-XYsk;-o1k ze>HKK^+a3ElTPgxyxK7#d6y#tcXjSF_NlW!gk1TeI4mbxUw4jHF6i~ z|A8!Pz_2G-nGe$q0fBnU;hgFwfC}V^W~38*+jYu{QeL#Wv(+)b?ZokNSZk1rByd1b>>i2hPdu`q~5K~ z->O~{5FiC}jY~asb>18!9Lb>IS3Ak4GPr+BpA;j5Kl^JN%Qdzu zd`6pYTxLEmh}0xwkt$XZj#dTBXa95Js1#2Yx5qYxd9HSWzvW7>2`H$W#9jEE(GssM zTQ+**lXD@STY1W-zrE-*Zf7j$G+EIfkm&^JaqhAYtQ$7~0cDL>544$h# z;91B{?KCVL4i?tGl&wfDm1Ui&$`dj%cND%{qPU~@mHN##d12^N14b1m!7$J6#i_%i z$5eRHBJVBDF`SwUmvME*Xh|QPGl83orN6C4R$!%DcrV~C0~gMy9lk!yA6M{hpE2{K zGd5ab$ZiW0aKf>&`B7O=C@%HX%~6%#g>1`f6nPH!pH$@t54)Rkhk7e#vsD($2-zyI z)TMSZ9klvo8Y$wE5$bU=+!D;s3oe6VbwDQjDr}oo;kXR%kuc^12)MeNc2Rn?{@=b@ zW&!C6qc`4{k2pL#de?{d~U%L?WKS-cb3hU+H+e)=n z(sf9-N*!Lt6#j<$4EOMji<$Wj2f7PqZ2yy-5TOEp%oiC?Q*M6ksInr5p1t!s;v&I* z-ej$>AbPh(+2C@BcY%(qJr>P~2y~wkXmxVdYAtX4jw*?Q2A3VamF6&9erw1t8e~Ke z&3XRr)l1z$#l0wzMe3;CX=iKbOzywy5!7vpB7{|3a27P%d^Jy|!txPaP%n9Qn<^9h zSFG*i)pVlL1!o-Kt&c^=-c?!8N zl-*~+%JWe-y~5y$39qYQ^>_w^?J>#xiZG!`^!5H zl`Xq%G5aUSfA1cw>ZyTuP(Eak6cKp)Sn7S~8auijSXX@0=0&9Nt&F2BGVI!Y;4!=X zW(6nDq4{Cmx1b;AO;nY>j;?|!dzJTFAwz`04=xN1#J%X0Ruf*x444_8p5%WS{2FP7 zslHw7AS-`<@2$;64#zlsMWef=i}UM>q2nCM-JgR2^BdQ!ztLspXQ;|j$LsaUZPNvM zl%Y>(-i;%|iU=U#mJ>q>d?Ttb`))N>^vC=`umReOK}W`+(Rk02$k}DwJek^msnQ%?+-zLvH7dyKVz2&=BBniEyyfCo(JQGs^o!Xs+4yM#5!xxpal}Ib zzdPw?V*EIMU zX)WD}zl>D+*^s)-K6ft1d+@R((jO=Th`;*<_eD%~mKZmmUiW^>Z27y_Q$JMdrY#TE z{d-TQkqX69=4qTz<}G7zi-QBVw zbd7+wVj~{zvLaK6M#j2kAqSZl(=aIfr$+<#N9w+8LD*Hay2dlk0dU#@)jEtkzKfET>#!~1f~ z{nAU?3h*KHY-pCsytvk*6V7~d`c3)~X8xPQ^;I~FWy##fbb-*MJ*a?Y8Dq$gGqy`mT>d&~le}C+#%P&gRgn!ga7_NUkTHQhoxKC@x9{;E-mL-R!kqQ*2oo z`K+N^sbIZ>*yvl17gIZHgSe~z#@OX7F!FbxR2_hhy!&pPU}^(QT3DFXSNOFJ@0&G$ zKgN=ms#5yVntG9g6_j^IHYpC#hWrHF^-rC+;2(BAg^It(VHuM7wCmo!iwn%rI0)n` zfHe;H#cg84sc;{BW;u3RXA@mXjC*w7`?9mfR^y3=f*oTwOi1G?J_G<1#Bk0xUqgfZ zyAv-jyE~b-8|p@WyaRqX9ZeOx z-U%CI$wOHBBji9}&DJLxSPZAio%ekVi3%(~90zc3$%-7BZ-p2D#3?}=0ao7fvc&AN z=t0;3X+~i8&xH4S8Tvsp#nz*D2DE>wCcl~95@ipk1b`(GHmq2dy2TDo&4wOvNs>pK zYi0{hWqvQLJNnITExyTTu+F^4=o0`~5AhCYfGj-GzW3(N*atqqnSfPBT#p#RkWiBA zut*QTRivJz?mKBiS)}j{`}u$ERufx-^ZZpPr}Jc1#Homgl)n*! za&uBbrH;S+*MFPUyH&=lKX6p4#sgk|ISVFU@63ujF>wYkkOQK%bgOHbLL6XW|Bd)G z-x6-t))gmwmeu*yS1i)glI}kD&?K{I-lh7?An=muM8hkYuUN^h(h~Yvw+EeDpg@6t zqxkvR7MBu|Rq_+W2n!k9Y3Im$z%0eDT4-R;SlTX14gpk4`dQ7W3Hbov8R7LN z{gQ%6gGo6=34RYy#5)k$FiCv`Toa)hsP)$mBZBdlriG=COR*Go#zTf7?j;&&Y5D19 z(ajZy{6O&QR)&XnKnmm;*?PTLY!xdPl=tHPfjpVCEr(z+kT^CLrB4tt5XlI$s5br}s_3T;oHmQ)|rX}w{WADb2d4G*^`!b7cw z0rgnMI-5exn{v!i;$T?=ExAj0dZYNP@X8n0z|#mGjP;!$eD=d%KcEs{v@LH}eydGN zVXx!0|7)Awo#4RYLkSskm@63IxX8al?EH=jSE|ENS7_^ryFaBz?5H>!UW?E9QiYVE zyQ#U9wdKJpr3Ar=kIlIO)F?Hi(`L^Vq$!03{^zMa77Bv(@WeMQ>#r4)PO>-~3``0k znyhoS30_I2hbRy7CVn)sWQDj=IMAx;!9ot>0L#7Kd<-%|0%l7O7n_)`u$~&+$bTE$ zitEVw_?e9qILO5lf)FhSxzGDTVh^`aH6DWLeg_T!_X?X2!B$ zjtR!N>wRW#&h#G3Aav1#)l_Th^bEy2CRErtZ%0iCWWDTRhU*n!MXG_D-V( zjFr~Z`<6gZfVkS*v-@NrtfR#m?$s1Dw)e3vr_x=7p(lq`a09@I)XUw&;n+VP+OIOU z(v9ORRwoi@uj4n{503j!9OCR7Q{1gfl36yT22WMJd7^+?&Z7;sDweX^Gn8-G8m4Kj z$phK%@+-(tzC@Y?m)g&#t8t9Lx2e?!AJnfDd9Y@d%QdumYY}Ih%ea17(pe2esBn4i z@w5d7veTl_R-Mb*_}>sJMOzcLCy5`l%o4A@hlLb>L?M{p$K8|sP?NF&% zu>J3iOhauYEPI=4^5^f)s(muYRk4&e*_<95K}pBzc92*>8~m_TgtlUUp4UAV^cG*! z!l|~LHEGo8U34(VP5n&L3*bycLPm$L*B4y1Up%%w5KQ7<{`a9Z+zQ0D;>b58d*n zf*e(4sj)WLvc=Jpg@FcT4#fS_J&lnVXT{e@9|cM%aV#FNdF)W~Oi8=GK|gT{C+ffe zMmlyAiXlw-k&$dyHE4j5)w>bQL=iJR^8W_09$;&4G-G_moLG3(A;yQk)v^9RxCJ&a zcGJeOFIDGv_GJeuc0s@KLBc}==o0eE4ww)-;Re#c;o13fls_RXyJPaTG&P1+*?SDi z4tFS~#tFhFnj?723=>Y$^+OKhfakuXl3=o}5=8qX&3bB-Z%Ru~9P`z}mGXV#`a_3S zrVUe_+f#lZRPZ^m4qb`$%5&FPE@c*$&<{P4lqtJf)fYKR>Dsr@y5R_~xOp;KS?F4J z>#ZhW)RjwunbAw-H4a)dis;hn&jrD`J1agwu;^jS9F!Z|m7zkK8zZtds-)O9?8W?A zO48ZL5wx`&plfKma!8|ajNH_t0Q+sH2Fh86qn+iD(A?OnZ$65dKfkNhyYH%3!LhzW z*%SfZn>Vq{XzGmIzRx6e7OhemEIy)~nNURfiN-3-Hk(Q+=#g8HaOln8OQfJ=fiU71B|$fvSf$R!x+E}Di|PI=!y|4z>6NqIxVrv zjPPwNIF31FC`W{6cIXz_a(Xw{$=+?q-X48B>?Dz;Zst<7RHdp(sCfF;#Y#D>`GG~06*{Dyf zp-sTTHzCgCcC0FN;Qnf&yW^#+iZHuJHO`17jlJ8EFKz@12d;!71xZ=;<4bleE}t}k zJ8CNI>u++y*CK!;Pc~B zx%+{f+ugL{Ln&};qFxt2287E-eO?L~0Y`ESLR-a!lIfO`G~Owg+QaF*K7YF#KA={q%bk zBZ-@EPr6&%D8tMgSg~u1w%E(8gtJ?zqd#_+8C-;kwpsp;z60UDPBZqv1{Dbw#^Y0v>~u-0!HbszeZ_erV`|C^?UElM%A711&3)ChRd+pX>a z8297IromSZUAe82llScxzihlUUE8KfPo_MW?tyYS=2|98wdGIhhotaWh0R8zW_Vp@ zVGoaRY!gzzJRwuI$*eI#48uZAFwjX>&?_7lYCbK9=;(Q=Tcq9>AZ0i4rNMU}ltLQt zuDkA!B2P?mh32)ae#@uLK*n&vzJjaG{`*aY)_eLdbh_Mm^|5=rNl8YA5sln%EVZG9e<3-bKb$soasuiW=lsH z*-%!Xij6UA_$(I zkNVcfYk+5Gq}nMsOU#1OCZ(RkRSJR-GV`^%77KJ%1ZEJMi<#b# z7?_-{db4FN?!|-W3}v@1)i7YcTrvE?lc}Z`H5he-#oR;^cm0;SVAV1pW#emR|Vu1IjXmaWPow^s{6cQi30l92Pe-4Dlepyq76R+Drttm0S9~u2n zenss#yQk7YrZEiF*!J#{Y zC4QD5eRN1;V3mEFJS?OXI%q2~)Qbi2e0u5EQqyVJuO0A{`OY;V`J^Y{jRy=E02ginLVX#XxNB(fMWOQ46mnS`CD(KNB!}? z2dlWUfzF`Z_@i_9)~cl<3D3EFVw<6~y|w3I!8ve5or3{eETP<_XRl4EX3(s&wAFBt zt%Y#TUL7S*YK0a(6~wv;%K*2=dA^PJ18gW`jDBQz%hZ(f?< z*+7&hMjKo@wBg9lj?ui+KzbS8l(6F-D|+zuBU$d^Ks;B3(Bns58AOx`&-;rkq{fl) z{iS^2nHqSs#4krBd;!8w{M8!v1dTjH21ew3QYgeA{l54;*c=pG=i>35Tmko z@}_TOerRiAV+=`7w=M9N)Ms(!pR%7&5?;fGLTyVh65pG3A}x9-hPpQ;V^yFEgxd zJu;&|q`;xo>6IMki}T!q&GO(|N#3tXX-45VkQgSBg#o?{G9wgRsSdC1Ca`H>;>j3& zgEAzsh2-AL%OHiL0y&>21MQ_TzvR~xVA8#t8*%oOD9r9fu(|+%!&uduYckXiX85!c z@oKaK)x>vp$X`=@zC5^?KCj=hUcgQ%rN)210OToGrM9Z^mbW2!UDD$j_RFe39;XNU zQ^WP1tWf-4z#fZqpbkcB*_@D9K?VG4?4n#ca=DTe-tpOZ*LaZU!jkAz%dhZSW2i6^ zuDZFo>EE&4JY$aaE!pn@y2=PPE<2>P_>w_vAn5f2v$cEL2dMV?O5=u!`u!%hznj^$ zUMAO(KLHYs(Tg8bT|unsNbo@X1PbW*exbH7`Ii$AYY(UZx55tR%yEquNK5NTb#%VT z0Mcad|Na*^hrlX9hl%JRjTB?-GYf)gkxz`xgR^rbf{WY-{Gp$DuO=t30Q{BW0@KZC z&NO417^f&Vp?J}G1EepYd}EDl<-90kdV!SV60flQiYE9Z^SMIm$(-`}T0*_Hd!P{> zZDnNcG$i@i#F!Rvuw+w0@uQ!_5~ZNR(!dB=0R>Px0>#wEoK)XY%>WbS0Qs)~-W$~C zQP(Ju;U7_I7k=0Q0I)4;|B0wGr=rd}ohF@yAHWXl9X6KM4wgF{|Jh;dWNYJOz0< Date: Fri, 15 May 2026 22:12:43 -0700 Subject: [PATCH 19/65] feat(macro): URI-safe ID separator, exec stats roll-up, UI polish MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * MacroExpander: switch inner-op ID prefix from "/" to "--" so prefixed IDs survive serialization through GlobalPortIdentitySerde's VFS-URI path component. Update WorkflowCompiler.visibleOperatorId and outer-error filter accordingly; add `require(!contains('/'))` in the serde as a hard guard. All 17 MacroExpanderSpec tests updated for the new separator and passing. * WorkflowStatusService: fold inner-op stats keyed by "${macroInstanceId}--*" into a synthetic entry under macroInstanceId so the macro node renders state + row counts during execution on the outer canvas. Worst-case state wins (Recovering > Pausing > ... > Completed > Uninitialized); row counts and worker counts are summed. Original prefixed entries are preserved. * ValidationWorkflowService: skip AJV schema validation for Macro operators — the embedded schema references LogicalOp polymorphic union (via MacroBody.operators) and AJV can't reliably handle it. Connection validation alone still gates the red/grey state. * OperatorMetadataService: when sanitizing schemas off the wire, convert `nullable: true + $ref: X` to `anyOf: [{type: null}, {$ref: X}]` instead of just stripping nullable, so Option[T] fields serialized as null round-trip cleanly through AJV. * JointUIService: visually differentiate macro nodes — Macro instance gets a soft-blue fill and dashed blue border; MacroInput / MacroOutput markers get a muted grey, rounded "port pad" look with their operator-name label suppressed. changeOperatorColor preserves the macro-specific stroke across validation toggles by reading operatorType stashed on the JointJS element. * WorkspaceComponent: pinned banner above the canvas when on `/workflow/:id/macro/:macroId` so the user can't miss they're editing a macro body and not the parent workflow. Co-Authored-By: Claude Sonnet 4.6 --- .../workflow/macroOp/MacroExpander.scala | 6 +- .../util/serde/GlobalPortIdentitySerde.scala | 4 + .../component/workspace.component.html | 13 + .../component/workspace.component.scss | 54 +++ .../component/workspace.component.ts | 11 + .../service/joint-ui/joint-ui.service.ts | 61 +++- .../operator-metadata.service.ts | 12 +- .../validation/validation-workflow.service.ts | 9 + .../workflow-status.service.ts | 75 +++- .../amber/compiler/WorkflowCompiler.scala | 75 +++- .../compiler/macroOp/MacroExpander.scala | 5 +- .../compiler/macroOp/MacroExpanderSpec.scala | 340 +++++++++++++++++- 12 files changed, 620 insertions(+), 45 deletions(-) diff --git a/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala index 00a9b61c2d9..bf9eeeba597 100644 --- a/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala +++ b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala @@ -36,9 +36,9 @@ import org.apache.texera.workflow.{LogicalLink, LogicalPlan} // inlines every MacroOpDesc by splicing its body's inner operators and links // into the parent, and produces a flat LogicalPlan with no MacroOpDesc / // MacroInputOp / MacroOutputOp nodes. Inner-op IDs are rewritten to -// "${macroInstanceId}/${innerOpId}" so telemetry can be aggregated per macro +// "${macroInstanceId}--${innerOpId}" so telemetry can be aggregated per macro // purely from the operator-ID prefix — the physical-plan layer remains -// macro-unaware. +// macro-unaware. "--" is used instead of "/" to avoid breaking VFS URI paths. // // Mirrors the compiling-service MacroExpander; the two operate on their own // LogicalLink/LogicalPlan classes and will converge once those types are @@ -137,7 +137,7 @@ object MacroExpander { val idRewrite: Map[OperatorIdentity, OperatorIdentity] = innerOps.map { op => val originalId = op.operatorIdentifier - val newId = s"$instanceId/${op.operatorIdentifier.id}" + val newId = s"$instanceId--${op.operatorIdentifier.id}" op.setOperatorId(newId) originalId -> op.operatorIdentifier }.toMap diff --git a/common/workflow-core/src/main/scala/org/apache/texera/amber/util/serde/GlobalPortIdentitySerde.scala b/common/workflow-core/src/main/scala/org/apache/texera/amber/util/serde/GlobalPortIdentitySerde.scala index c8fd8e1a363..cdcd12a3934 100644 --- a/common/workflow-core/src/main/scala/org/apache/texera/amber/util/serde/GlobalPortIdentitySerde.scala +++ b/common/workflow-core/src/main/scala/org/apache/texera/amber/util/serde/GlobalPortIdentitySerde.scala @@ -49,6 +49,10 @@ object GlobalPortIdentitySerde { !logicalOpId.contains('_'), s"logicalOpId must not contain '_' (VFS URI parsing relies on this): $logicalOpId" ) + require( + !logicalOpId.contains('/'), + s"logicalOpId must not contain '/' (breaks VFS URI path structure): $logicalOpId" + ) require( !layerName.contains('_'), s"layerName must not contain '_' (VFS URI parsing relies on this): $layerName" diff --git a/frontend/src/app/workspace/component/workspace.component.html b/frontend/src/app/workspace/component/workspace.component.html index c54446fb318..77ee449a82a 100644 --- a/frontend/src/app/workspace/component/workspace.component.html +++ b/frontend/src/app/workspace/component/workspace.component.html @@ -23,6 +23,19 @@ [nzSize]="'large'" nzTip="Loading workflow..."> +

    + Editing Macro + {{ macroEditName }} + + ← Back to parent + +
    diff --git a/frontend/src/app/workspace/component/workspace.component.scss b/frontend/src/app/workspace/component/workspace.component.scss index 60fc4abf401..517c4494261 100644 --- a/frontend/src/app/workspace/component/workspace.component.scss +++ b/frontend/src/app/workspace/component/workspace.component.scss @@ -57,3 +57,57 @@ texera-workflow-editor { :host { user-select: none; } + +// Pinned banner shown when the canvas is rendering a macro body via the +// drill-down route (/workflow/:id/macro/:macroId). Sits above the menu so the +// user can't miss that they're editing a macro definition, not the parent +// workflow. Same z-index family as `texera-menu` (z: 1) but +1 to stay above. +.macro-edit-banner { + position: absolute; + top: 0; + left: 0; + z-index: 2; + width: 100%; + padding: 6px 16px; + display: flex; + align-items: center; + gap: 10px; + background: linear-gradient(90deg, #1d6fdb 0%, #4a90e2 100%); + color: #ffffff; + font-size: 13px; + font-weight: 500; + box-shadow: 0 1px 3px rgba(0, 0, 0, 0.15); + + &__label { + text-transform: uppercase; + letter-spacing: 0.5px; + font-size: 11px; + opacity: 0.85; + } + + &__name { + font-weight: 600; + font-size: 14px; + } + + &__back { + margin-left: auto; + color: #ffffff; + text-decoration: underline; + font-size: 12px; + opacity: 0.9; + cursor: pointer; + + &:hover { + opacity: 1; + } + } +} + +// Push the regular menu down so the banner doesn't overlap it. +:host { + .macro-edit-banner + #result, + .macro-edit-banner ~ texera-menu { + margin-top: 32px; + } +} diff --git a/frontend/src/app/workspace/component/workspace.component.ts b/frontend/src/app/workspace/component/workspace.component.ts index 4a350ca1cd7..c0637c94929 100644 --- a/frontend/src/app/workspace/component/workspace.component.ts +++ b/frontend/src/app/workspace/component/workspace.component.ts @@ -91,6 +91,11 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { public pid?: number = undefined; public writeAccess: boolean = false; public isLoading: boolean = false; + // Macro drill-down state — drives the banner above the canvas so users know + // they're editing a macro body rather than a normal workflow. + public macroEditMode: boolean = false; + public macroEditName: string = ""; + public parentWorkflowId?: string; @ViewChild("codeEditor", { read: ViewContainerRef }) codeEditorViewRef!: ViewContainerRef; /** @@ -286,6 +291,9 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { .pipe(untilDestroyed(this)) .subscribe( ({ detail }) => { + this.macroEditMode = true; + this.macroEditName = detail.name; + this.parentWorkflowId = this.route.snapshot.params.id ?? ""; const macroWorkflow = this.macroService.macroDetailToWorkflow(detail); // Clear the canvas before reloading. Angular reuses WorkspaceComponent // across route changes (no ngOnDestroy fires when going from @@ -346,6 +354,9 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { // Re-enable persist in case we just came back from a macro drill-down, // which disables it. this.workflowPersistService.setWorkflowPersistFlag(true); + this.macroEditMode = false; + this.macroEditName = ""; + this.parentWorkflowId = undefined; this.loadWorkflowWithId(Number(wid)); return; } diff --git a/frontend/src/app/workspace/service/joint-ui/joint-ui.service.ts b/frontend/src/app/workspace/service/joint-ui/joint-ui.service.ts index 6bc05f7e3b1..13810c648ca 100644 --- a/frontend/src/app/workspace/service/joint-ui/joint-ui.service.ts +++ b/frontend/src/app/workspace/service/joint-ui/joint-ui.service.ts @@ -272,6 +272,10 @@ export class JointUIService { // set operator element ID to be operator ID operatorElement.set("id", operator.operatorID); operatorElement.set("z", 1); + // Stash the type so type-conditional restyling (e.g. preserving the macro + // border across validation updates) can read it without going back to + // WorkflowActionService. + operatorElement.set("operatorType", operator.operatorType); // set the input ports and output ports based on operator predicate operator.inputPorts.forEach(port => @@ -465,10 +469,21 @@ export class JointUIService { * @param isOperatorValid */ public changeOperatorColor(jointPaper: joint.dia.Paper, operatorID: string, isOperatorValid: boolean): void { - if (isOperatorValid) { - jointPaper.getModelById(operatorID).attr("rect.body/stroke", "#CFCFCF"); + const model = jointPaper.getModelById(operatorID); + if (!model) return; + if (!isOperatorValid) { + model.attr("rect.body/stroke", "red"); + return; + } + // Preserve the macro-specific stroke for valid macro nodes; otherwise use + // the generic neutral grey applied to regular operators. + const operatorType = model.get("operatorType"); + if (operatorType === "Macro") { + model.attr("rect.body/stroke", "#1d6fdb"); + } else if (operatorType === "MacroInput" || operatorType === "MacroOutput") { + model.attr("rect.body/stroke", "#888888"); } else { - jointPaper.getModelById(operatorID).attr("rect.body/stroke", "red"); + model.attr("rect.body/stroke", "#CFCFCF"); } } @@ -693,6 +708,17 @@ export class JointUIService { operatorType: string, operatorFriendlyName: string ): joint.shapes.devs.ModelSelectors { + // Visual treatment for macro-related nodes: + // - Macro instance: thicker stroke + dashed pattern to read as "container" + // - MacroInput/MacroOutput: thin stroke; rounded so they read as port pads + // rather than operator boxes (further reduction handled in their own + // auto-layout pass — we keep the JointJS element shape but tone it down) + const isMacroInstance = operator.operatorType === "Macro"; + const isMacroMarker = operator.operatorType === "MacroInput" || operator.operatorType === "MacroOutput"; + const bodyStroke = isMacroInstance ? "#1d6fdb" : isMacroMarker ? "#888888" : "red"; + const bodyStrokeWidth = isMacroInstance ? "3" : isMacroMarker ? "1" : "2"; + const bodyStrokeDasharray = isMacroInstance ? "6,3" : undefined; + const bodyRadius = isMacroMarker ? "20px" : "5px"; return { ".texera-operator-coeditor-editing": { text: "", @@ -786,10 +812,11 @@ export class JointUIService { "rect.body": { fill: JointUIService.getOperatorFillColor(operator), "follow-scale": true, - stroke: "red", - "stroke-width": "2", - rx: "5px", - ry: "5px", + stroke: bodyStroke, + "stroke-width": bodyStrokeWidth, + ...(bodyStrokeDasharray ? { "stroke-dasharray": bodyStrokeDasharray } : {}), + rx: bodyRadius, + ry: bodyRadius, }, "rect.boundary": { fill: "rgba(0, 0, 0, 0)", @@ -818,7 +845,10 @@ export class JointUIService { "ref-y": -10, }, ".texera-operator-name": { - text: operatorDisplayName, + // Markers don't get a display-name label — they are visual port pads, + // not operators. The friendly-name above the box already reads e.g. + // "Input 0" / "Output 0", which is enough. + text: isMacroMarker ? "" : operatorDisplayName, fill: "#595959", "font-size": "14px", "ref-x": 0.5, @@ -829,8 +859,9 @@ export class JointUIService { }, ".texera-operator-friendly-name": { text: operatorFriendlyName, - fill: "#888888", - "font-size": "10px", + fill: isMacroMarker ? "#5a5a5a" : "#888888", + "font-size": isMacroMarker ? "12px" : "10px", + "font-weight": isMacroMarker ? "600" : "normal", "ref-x": 0.5, "ref-y": -12, ref: "rect.body", @@ -935,7 +966,15 @@ export class JointUIService { public static getOperatorFillColor(operator: OperatorPredicate): string { const isDisabled = operator.isDisabled ?? false; - return isDisabled ? "#E0E0E0" : "#FFFFFF"; + if (isDisabled) return "#E0E0E0"; + // Visually distinguish macro-related operators from regular ones so users + // can tell at a glance whether they're looking at a composite (Macro) or + // a boundary marker (MacroInput/MacroOutput) that's effectively a port. + if (operator.operatorType === "Macro") return "#E8F1FF"; // soft blue body + if (operator.operatorType === "MacroInput" || operator.operatorType === "MacroOutput") { + return "#EDEDED"; // muted grey — markers are "infrastructure," not real ops + } + return "#FFFFFF"; } public static getOperatorCacheDisplayText( diff --git a/frontend/src/app/workspace/service/operator-metadata/operator-metadata.service.ts b/frontend/src/app/workspace/service/operator-metadata/operator-metadata.service.ts index c6b6ea791c1..be87fa9e185 100644 --- a/frontend/src/app/workspace/service/operator-metadata/operator-metadata.service.ts +++ b/frontend/src/app/workspace/service/operator-metadata/operator-metadata.service.ts @@ -85,7 +85,17 @@ export class OperatorMetadataService { } const obj = node as Record; if (obj["nullable"] === true && obj["type"] === undefined) { - delete obj["nullable"]; + if (obj["$ref"] !== undefined) { + // "nullable: true, $ref: X" — Ajv ignores $ref siblings under Draft-07 strict + // rules. Convert to anyOf so that null AND the referenced type are both valid. + // This preserves round-trip properties that serialize Option[T] as null. + const ref = obj["$ref"]; + delete obj["nullable"]; + delete obj["$ref"]; + obj["anyOf"] = [{ type: "null" }, { $ref: ref }]; + } else { + delete obj["nullable"]; + } } for (const key of ["properties", "definitions", "patternProperties"]) { const dict = obj[key]; diff --git a/frontend/src/app/workspace/service/validation/validation-workflow.service.ts b/frontend/src/app/workspace/service/validation/validation-workflow.service.ts index 92ebb88bb22..bcd28e2d457 100644 --- a/frontend/src/app/workspace/service/validation/validation-workflow.service.ts +++ b/frontend/src/app/workspace/service/validation/validation-workflow.service.ts @@ -274,6 +274,15 @@ export class ValidationWorkflowService { throw new Error(`operator with ID ${operatorID} doesn't exist`); } + // Macro operators embed a complex JSON schema that references all operator types + // (via MacroBody.operators: List[LogicalOp]). AJV cannot reliably compile or + // validate against it on the frontend. Macro properties are always set by + // internal code and are validated by the backend during compilation, so + // skip AJV here and let connection validation alone determine validity. + if (operator.operatorType === "Macro") { + return { isValid: true }; + } + // try to fetch dynamic schema first const operatorSchema = this.dynamicSchemaService.getDynamicSchema(operatorID); if (operatorSchema === undefined) { diff --git a/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts b/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts index e939932aeba..26d36799900 100644 --- a/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts +++ b/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts @@ -22,6 +22,79 @@ import { Observable, Subject } from "rxjs"; import { OperatorState, OperatorStatistics } from "../../types/execute-workflow.interface"; import { WorkflowWebsocketService } from "../workflow-websocket/workflow-websocket.service"; +// Macro inner-op IDs carry a "${macroInstanceId}--..." prefix after MacroExpander +// runs on the backend. The engine reports stats keyed by those expanded IDs, but +// the outer canvas only has the macro instance itself — so we synthesize one +// aggregated entry per macro under the visible instance ID so the macro node can +// show a state and tuple counts during execution. The original prefixed entries +// stay in the map for the drill-down view (it maps "${instance}--${innerId}" +// back to "${innerId}" when displaying the body). +const MACRO_INNER_SEPARATOR = "--"; + +// State-priority for combining inner-op states into a single macro state. +// Worst-case wins (any failure surfaces; running beats ready; ready beats +// completed). Matches the user's mental model: "the macro is running if any +// inner op is still running." +const STATE_PRIORITY: Record = { + [OperatorState.Recovering]: 9, + [OperatorState.Pausing]: 8, + [OperatorState.Paused]: 7, + [OperatorState.Resuming]: 6, + [OperatorState.Running]: 5, + [OperatorState.Initializing]: 4, + [OperatorState.Ready]: 3, + [OperatorState.Completed]: 2, + [OperatorState.Uninitialized]: 1, +}; + +function combineStates(states: OperatorState[]): OperatorState { + if (states.length === 0) return OperatorState.Uninitialized; + return states.reduce((acc, s) => (STATE_PRIORITY[s] >= STATE_PRIORITY[acc] ? s : acc)); +} + +/** + * Group raw per-op stats by macro instance and emit one aggregated entry per + * macro under the visible instance ID. The original prefixed entries are + * preserved so the drill-down view can find them. + * + * Aggregation rules: + * - state: worst-case state across inner ops (see STATE_PRIORITY) + * - input/output row counts: sum across inner ops (approximate but useful as + * an activity indicator; precise boundary-only counts would need body shape) + * - port metrics: not aggregated (macro's port-level metrics are not 1:1 with + * inner-op port metrics; leave as empty so the tooltip doesn't show stale) + * - numWorkers: sum across inner ops + */ +function withMacroAggregates( + raw: Record +): Record { + const byMacro = new Map(); + for (const [opId, stats] of Object.entries(raw)) { + const sep = opId.indexOf(MACRO_INNER_SEPARATOR); + if (sep < 0) continue; + const macroId = opId.substring(0, sep); + const list = byMacro.get(macroId) ?? []; + list.push(stats); + byMacro.set(macroId, list); + } + if (byMacro.size === 0) return raw; + const out: Record = { ...raw }; + for (const [macroId, innerStats] of byMacro.entries()) { + // Don't overwrite a real entry that the engine sent for this ID (defensive + // — engine should never emit both, but if it does the real one wins). + if (out[macroId] !== undefined) continue; + out[macroId] = { + operatorState: combineStates(innerStats.map(s => s.operatorState)), + aggregatedInputRowCount: innerStats.reduce((sum, s) => sum + s.aggregatedInputRowCount, 0), + inputPortMetrics: {}, + aggregatedOutputRowCount: innerStats.reduce((sum, s) => sum + s.aggregatedOutputRowCount, 0), + outputPortMetrics: {}, + numWorkers: innerStats.reduce((sum, s) => sum + (s.numWorkers ?? 0), 0), + }; + } + return out; +} + @Injectable({ providedIn: "root", }) @@ -37,7 +110,7 @@ export class WorkflowStatusService { if (event.type !== "OperatorStatisticsUpdateEvent") { return; } - this.statusSubject.next(event.operatorStatistics); + this.statusSubject.next(withMacroAggregates(event.operatorStatistics)); }); } diff --git a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/WorkflowCompiler.scala b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/WorkflowCompiler.scala index 65e03bea062..cdc30e51991 100644 --- a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/WorkflowCompiler.scala +++ b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/WorkflowCompiler.scala @@ -37,6 +37,7 @@ import org.apache.texera.amber.core.workflow.{ } import org.apache.texera.amber.core.workflowruntimestate.FatalErrorType.COMPILATION_ERROR import org.apache.texera.amber.core.workflowruntimestate.WorkflowFatalError +import org.apache.texera.amber.operator.macroOp.{MacroInputOp, MacroOpDesc, MacroOutputOp} import java.time.Instant import scala.collection.mutable @@ -59,23 +60,40 @@ object WorkflowCompiler { } } + // After MacroExpander runs, inner-body operator IDs carry a "${macroInstanceId}--..." + // prefix (nested macros stack more "--" segments). The macro instance is the only + // operator the user sees on the parent canvas, so any compilation error from an + // inlined inner op must be re-attributed to that visible ID — otherwise the frontend + // looks up errors by canvas IDs and finds nothing for the failed macro. + private def visibleOperatorId(opId: OperatorIdentity): OperatorIdentity = { + val sep = opId.id.indexOf("--") + if (sep < 0) opId else OperatorIdentity(opId.id.substring(0, sep)) + } + // util function for convert the error list to error map, and report the error in log private def convertErrorListToWorkflowFatalErrorMap( logger: Logger, errorList: List[(OperatorIdentity, Throwable)] ): Map[OperatorIdentity, WorkflowFatalError] = { val opIdToError = mutable.Map[OperatorIdentity, WorkflowFatalError]() - errorList.map { + errorList.foreach { case (opId, err) => - // map each error to WorkflowFatalError, and report them in the log + val visibleId = visibleOperatorId(opId) + // Log with the *inner* opId so developers can find which inner op failed. logger.error(s"Error occurred in logical plan compilation for opId: $opId", err) - opIdToError += (opId -> WorkflowFatalError( - COMPILATION_ERROR, - Timestamp(Instant.now), - err.toString, - getStackTraceWithAllCauses(err), - opId.id - )) + // Skip if we already recorded an error for this visible op — keep the first one. + if (!opIdToError.contains(visibleId)) { + val message = + if (visibleId == opId) err.toString + else s"In macro inner op '${opId.id}': ${err.toString}" + opIdToError += (visibleId -> WorkflowFatalError( + COMPILATION_ERROR, + Timestamp(Instant.now), + message, + getStackTraceWithAllCauses(err), + visibleId.id + )) + } } opIdToError.toMap } @@ -127,6 +145,19 @@ class WorkflowCompiler( macroRegistry: MacroRegistry = MacroRegistry.Empty ) extends LazyLogging { + // A plan is a "standalone macro body" if it contains marker ops but no + // MacroOpDesc instance to wrap them. That shape is what the drill-down editor + // sends when the user is editing a macro body directly; it has no real + // upstream/downstream context, so we skip physical compilation. + private def isStandaloneMacroBody(plan: LogicalPlan): Boolean = { + val hasMarker = plan.operators.exists { + case _: MacroInputOp | _: MacroOutputOp => true + case _ => false + } + val hasMacroInstance = plan.operators.exists(_.isInstanceOf[MacroOpDesc]) + hasMarker && !hasMacroInstance + } + // function to expand logical plan to physical plan private def expandLogicalPlan( logicalPlan: LogicalPlan, @@ -209,6 +240,23 @@ class WorkflowCompiler( // 1. convert the pojo to logical plan val rawLogicalPlan: LogicalPlan = LogicalPlan(logicalPlanPojo) + // 1a. Standalone macro-body plans (the drill-down editor view) contain + // MacroInput/MacroOutput markers but no MacroOpDesc to inline them — so + // calling `getPhysicalPlan` on a marker would throw, and every inner op + // downstream would fail schema propagation. The body is only meant for + // structural editing in this view; the real compile happens when a parent + // instantiates the macro and MacroExpander strips the markers. Returning + // success here keeps the body view clean and prevents the singleton + // frontend compile-state from carrying marker errors across to the + // parent canvas on drill-down navigation. + if (isStandaloneMacroBody(rawLogicalPlan)) { + return WorkflowCompilationResult( + physicalPlan = Some(PhysicalPlan(operators = Set.empty, links = Set.empty)), + operatorIdToOutputSchemas = Map.empty, + operatorIdToError = Map.empty + ) + } + // 2. expand any macro operators into a flat logical plan. Macros are a purely // logical-plan-level abstraction; after this pass the rest of the pipeline never // sees a MacroOpDesc / MacroInputOp / MacroOutputOp. @@ -230,8 +278,15 @@ class WorkflowCompiler( // 4. collect the output schema for each logical op // even if error is encountered when logical => physical, we still want to get the input schemas for rest no-error operators opIdToOutputSchema = collectOutputSchemaFromPhysicalPlan(physicalPlan, errorList) + + // Only block the physical plan for errors on outer canvas operators. Errors that + // originated inside a macro body carry a "/" in their ID (e.g. "Macro-xxx/SleepOp") + // and are already attributed to the macro instance on the canvas — the outer + // workflow is structurally valid and can still be submitted; the broken macro will + // fail at execution time without blocking unrelated operators. + val outerErrorList = errorList.filter { case (opId, _) => !opId.id.contains("--") } WorkflowCompilationResult( - physicalPlan = if (errorList.nonEmpty) None else Some(physicalPlan), + physicalPlan = if (outerErrorList.nonEmpty) None else Some(physicalPlan), operatorIdToOutputSchemas = opIdToOutputSchema, // map each error from OpId to WorkflowFatalError, and report them via logger operatorIdToError = convertErrorListToWorkflowFatalErrorMap(logger, errorList.toList) diff --git a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala index 92122f0ddf1..c3b6fd81972 100644 --- a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala +++ b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala @@ -35,8 +35,9 @@ import org.apache.texera.amber.util.JSONUtils.objectMapper // Pre-compile pass: walks a LogicalPlan, inlines every MacroOpDesc by splicing its // body's inner operators and links into the parent, and produces a flat LogicalPlan // with no MacroOpDesc / MacroInputOp / MacroOutputOp nodes. Inner-op IDs are rewritten -// to "${macroInstanceId}/${innerOpId}" so telemetry can be aggregated per macro +// to "${macroInstanceId}--${innerOpId}" so telemetry can be aggregated per macro // purely from the operator-ID prefix — the physical-plan layer remains macro-unaware. +// Note: "--" is chosen over "/" because "/" breaks VFS URI path parsing. object MacroExpander { def expand(plan: LogicalPlan, registry: MacroRegistry): LogicalPlan = @@ -135,7 +136,7 @@ object MacroExpander { // (originalId → prefixedId) captured before mutating the cloned ops. val idRewrite: Map[OperatorIdentity, OperatorIdentity] = innerOps.map { op => val originalId = op.operatorIdentifier - val newId = s"$instanceId/${op.operatorIdentifier.id}" + val newId = s"$instanceId--${op.operatorIdentifier.id}" op.setOperatorId(newId) originalId -> op.operatorIdentifier }.toMap diff --git a/workflow-compiling-service/src/test/scala/org/apache/texera/amber/compiler/macroOp/MacroExpanderSpec.scala b/workflow-compiling-service/src/test/scala/org/apache/texera/amber/compiler/macroOp/MacroExpanderSpec.scala index 2a57c3b1548..b4b0d21f5e5 100644 --- a/workflow-compiling-service/src/test/scala/org/apache/texera/amber/compiler/macroOp/MacroExpanderSpec.scala +++ b/workflow-compiling-service/src/test/scala/org/apache/texera/amber/compiler/macroOp/MacroExpanderSpec.scala @@ -19,10 +19,13 @@ package org.apache.texera.amber.compiler.macroOp -import org.apache.texera.amber.compiler.model.{LogicalLink, LogicalPlan} -import org.apache.texera.amber.core.workflow.PortIdentity +import org.apache.texera.amber.compiler.WorkflowCompiler +import org.apache.texera.amber.compiler.model.{LogicalLink, LogicalPlan, LogicalPlanPojo} +import org.apache.texera.amber.core.virtualidentity.{OperatorIdentity, WorkflowIdentity} +import org.apache.texera.amber.core.workflow.{PortIdentity, WorkflowContext} import org.apache.texera.amber.operator.limit.LimitOpDesc import org.apache.texera.amber.operator.macroOp._ +import org.apache.texera.amber.operator.source.scan.csv.CSVScanSourceOpDesc import org.scalatest.flatspec.AnyFlatSpec import org.scalatest.matchers.should.Matchers @@ -66,10 +69,10 @@ class MacroExpanderSpec extends AnyFlatSpec with Matchers { out.operators.exists(_.isInstanceOf[MacroOutputOp]) shouldBe false out.operators.collect { case l: LimitOpDesc => l.operatorIdentifier.id }.toSet shouldBe - Set("src", "sink", "MyMacro-1/inner") + Set("src", "sink", "MyMacro-1--inner") val edges = out.links.map(l => (l.fromOpId.id, l.toOpId.id)).toSet - edges shouldBe Set("src" -> "MyMacro-1/inner", "MyMacro-1/inner" -> "sink") + edges shouldBe Set("src" -> "MyMacro-1--inner", "MyMacro-1--inner" -> "sink") } it should "fetch a LIVE-linked macro body from the registry" in { @@ -99,7 +102,7 @@ class MacroExpanderSpec extends AnyFlatSpec with Matchers { ) val out = MacroExpander.expand(plan, registry) out.operators.collect { case l: LimitOpDesc => l.operatorIdentifier.id }.toSet shouldBe - Set("src", "sink", "L-inst/inner") + Set("src", "sink", "L-inst--inner") } it should "expand nested macros with concatenated ID prefixes" in { @@ -129,12 +132,12 @@ class MacroExpanderSpec extends AnyFlatSpec with Matchers { ) val out = MacroExpander.expand(plan, MacroRegistry.Empty) val ids = out.operators.collect { case l: LimitOpDesc => l.operatorIdentifier.id }.toSet - ids should contain("Outer/Inner/inner-inner") + ids should contain("Outer--Inner--inner-inner") ids should contain("src") ids should contain("sink") val edges = out.links.map(l => (l.fromOpId.id, l.toOpId.id)).toSet - edges should contain("src" -> "Outer/Inner/inner-inner") - edges should contain("Outer/Inner/inner-inner" -> "sink") + edges should contain("src" -> "Outer--Inner--inner-inner") + edges should contain("Outer--Inner--inner-inner" -> "sink") } it should "detect a self-referential macro cycle" in { @@ -246,13 +249,13 @@ class MacroExpanderSpec extends AnyFlatSpec with Matchers { ) val out = MacroExpander.expand(plan, MacroRegistry.Empty) val ids = out.operators.collect { case l: LimitOpDesc => l.operatorIdentifier.id }.toSet - ids should contain("first/inner") - ids should contain("second/inner") + ids should contain("first--inner") + ids should contain("second--inner") val edges = out.links.map(l => (l.fromOpId.id, l.toOpId.id)).toSet edges shouldBe Set( - "src" -> "first/inner", - "first/inner" -> "second/inner", - "second/inner" -> "sink" + "src" -> "first--inner", + "first--inner" -> "second--inner", + "second--inner" -> "sink" ) } @@ -282,7 +285,7 @@ class MacroExpanderSpec extends AnyFlatSpec with Matchers { val out = MacroExpander.expand(plan, MacroRegistry.Empty) val srcOutTargets = out.links.filter(_.fromOpId == src.operatorIdentifier).map(_.toOpId.id).toSet - srcOutTargets shouldBe Set("FanOut/consumerA", "FanOut/consumerB") + srcOutTargets shouldBe Set("FanOut--consumerA", "FanOut--consumerB") } it should "fail clearly when a LIVE macro is missing from the registry" in { @@ -328,7 +331,7 @@ class MacroExpanderSpec extends AnyFlatSpec with Matchers { val first = MacroExpander.expand(plan, MacroRegistry.Empty) val innerInBodyAfterFirst = body.operators.collectFirst { case l: LimitOpDesc => l.operatorIdentifier.id } - innerInBodyAfterFirst shouldBe Some("inner") // not "once/inner" — body wasn't mutated. + innerInBodyAfterFirst shouldBe Some("inner") // not "once--inner" — body wasn't mutated. // Re-expand a fresh plan that reuses the SAME body object: must still inline cleanly. val inst2 = snapshotInstance("twice", "m", body) @@ -341,10 +344,313 @@ class MacroExpanderSpec extends AnyFlatSpec with Matchers { ) val second = MacroExpander.expand(plan2, MacroRegistry.Empty) val secondIds = second.operators.collect { case l: LimitOpDesc => l.operatorIdentifier.id }.toSet - secondIds should contain("twice/inner") + secondIds should contain("twice--inner") val firstIds = first.operators.collect { case l: LimitOpDesc => l.operatorIdentifier.id }.toSet - firstIds should contain("once/inner") + firstIds should contain("once--inner") + } + + // ---------- full-compile path: schema propagation + error attribution ---------- + + it should "compile a workflow whose source feeds a macro body, propagating schemas through the inline" in { + val body = MacroBody( + operators = List(inMarker(0, "in"), limit("inner", 10), outMarker(0, "out")), + links = List( + MacroLink("in", PortIdentity(0), "inner", PortIdentity(0)), + MacroLink("inner", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val inst = snapshotInstance("MyMacro-1", "macro-A", body) + + val csvOp = new CSVScanSourceOpDesc() + csvOp.fileName = Some("workflow-compiling-service/src/test/resources/country_sales_small.csv") + csvOp.customDelimiter = Some(",") + csvOp.hasHeader = true + csvOp.setOperatorId("CSVScan-A") + + val sink = limit("sink", 5) + + val pojo = LogicalPlanPojo( + operators = List(csvOp, inst, sink), + links = List( + LogicalLink(csvOp.operatorIdentifier, PortIdentity(0), inst.operatorIdentifier, PortIdentity(0)), + LogicalLink(inst.operatorIdentifier, PortIdentity(0), sink.operatorIdentifier, PortIdentity(0)) + ), + opsToViewResult = List(), + opsToReuseResult = List() + ) + + val ctx = new WorkflowContext(workflowId = WorkflowIdentity(0)) + val result = new WorkflowCompiler(ctx).compile(pojo) + + result.operatorIdToError shouldBe empty + result.physicalPlan should not be empty + // Inner op got the source's schema propagated through the macro boundary. + val innerSchema = result.operatorIdToOutputSchemas(OperatorIdentity("MyMacro-1--inner")) + innerSchema.values.head shouldBe defined + } + + it should "propagate schemas through a LIVE-mode macro to multiple downstream ops on the parent canvas" in { + // Mirrors the user-reported failure shape: + // CSV → Macro(LIVE, macroId=265) → mid → end + // where the macro body is a single one-to-one op wrapped in markers. + val body = MacroBody( + operators = List(inMarker(0, "in"), limit("inner", 10), outMarker(0, "out")), + links = List( + MacroLink("in", PortIdentity(0), "inner", PortIdentity(0)), + MacroLink("inner", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val registry = MacroRegistry.inMemory(Map(("265", 1) -> body)) + + val inst = new MacroOpDesc + inst.macroId = "265" + inst.macroVersion = 1 + inst.linkMode = MacroOpDesc.LIVE + inst.inputPortCount = 1 + inst.outputPortCount = 1 + inst.setOperatorId("Macro-operator-acc00f1c") + + val csvOp = new CSVScanSourceOpDesc() + csvOp.fileName = Some("workflow-compiling-service/src/test/resources/country_sales_small.csv") + csvOp.customDelimiter = Some(",") + csvOp.hasHeader = true + csvOp.setOperatorId("CSVScan") + + val mid = limit("mid", 5) + val end = limit("end", 2) + + val pojo = LogicalPlanPojo( + operators = List(csvOp, inst, mid, end), + links = List( + LogicalLink(csvOp.operatorIdentifier, PortIdentity(0), inst.operatorIdentifier, PortIdentity(0)), + LogicalLink(inst.operatorIdentifier, PortIdentity(0), mid.operatorIdentifier, PortIdentity(0)), + LogicalLink(mid.operatorIdentifier, PortIdentity(0), end.operatorIdentifier, PortIdentity(0)) + ), + opsToViewResult = List(), + opsToReuseResult = List() + ) + + val ctx = new WorkflowContext(workflowId = WorkflowIdentity(0)) + val result = new WorkflowCompiler(ctx, registry).compile(pojo) + + result.operatorIdToError shouldBe empty + result.physicalPlan should not be empty + // Inner op got its schema, and the downstream canvas ops did too. + val outputKeys = result.operatorIdToOutputSchemas.keys.map(_.id).toSet + outputKeys should contain("Macro-operator-acc00f1c--inner") + outputKeys should contain("mid") + outputKeys should contain("end") + } + + it should "surface the macro on the canvas as failing when the LIVE macro body's inner op lacks its MacroInput link" in { + // The macro body has both markers, but the MacroInput → inner-op link is + // missing. After expansion, the inner op is disconnected from the parent's + // upstream — so its schema can't be computed, and that cascades to every + // downstream canvas op. Before the visible-id remap, the macro itself looked + // fine in `operatorErrors` (only `Macro/inner` was keyed there), so the + // canvas would only mark the *downstream* ops red — confusing the user since + // the root cause is the macro. + val body = MacroBody( + operators = List(inMarker(0, "in"), limit("inner", 10), outMarker(0, "out")), + links = List( + // intentionally NO link from "in" to "inner" + MacroLink("inner", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val registry = MacroRegistry.inMemory(Map(("265", 1) -> body)) + + val inst = new MacroOpDesc + inst.macroId = "265" + inst.macroVersion = 1 + inst.linkMode = MacroOpDesc.LIVE + inst.inputPortCount = 1 + inst.outputPortCount = 1 + inst.setOperatorId("Macro-operator-acc00f1c") + + val csvOp = new CSVScanSourceOpDesc() + csvOp.fileName = Some("workflow-compiling-service/src/test/resources/country_sales_small.csv") + csvOp.customDelimiter = Some(",") + csvOp.hasHeader = true + csvOp.setOperatorId("CSVScan") + + val mid = limit("mid", 5) + val end = limit("end", 2) + + val pojo = LogicalPlanPojo( + operators = List(csvOp, inst, mid, end), + links = List( + LogicalLink(csvOp.operatorIdentifier, PortIdentity(0), inst.operatorIdentifier, PortIdentity(0)), + LogicalLink(inst.operatorIdentifier, PortIdentity(0), mid.operatorIdentifier, PortIdentity(0)), + LogicalLink(mid.operatorIdentifier, PortIdentity(0), end.operatorIdentifier, PortIdentity(0)) + ), + opsToViewResult = List(), + opsToReuseResult = List() + ) + + val ctx = new WorkflowContext(workflowId = WorkflowIdentity(0)) + val result = new WorkflowCompiler(ctx, registry).compile(pojo) + + result.physicalPlan shouldBe empty + val keys = result.operatorIdToError.keys.map(_.id).toSet + keys should contain("Macro-operator-acc00f1c") // the macro instance, not "Macro-operator-acc00f1c--inner" + keys should contain("mid") + keys should contain("end") + result.operatorIdToError(OperatorIdentity("Macro-operator-acc00f1c")).message should + include("Macro-operator-acc00f1c--inner") + } + + it should "still compile the outer workflow when a dangling inner op inside the macro has a schema error" in { + // Main path: in → inner1 → out (valid, schema flows through). + // Dangling side branch: inner2 has no upstream in the body → schema error. + // The outer canvas (CSVScan → Macro → sink) should still compile; only the + // macro shows red. Previously the dangling error set physicalPlan to None. + val body = MacroBody( + operators = List( + inMarker(0, "in"), + limit("inner1", 10), + limit("inner2", 5), // disconnected — no link to/from anything + outMarker(0, "out") + ), + links = List( + MacroLink("in", PortIdentity(0), "inner1", PortIdentity(0)), + MacroLink("inner1", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val registry = MacroRegistry.inMemory(Map(("265", 1) -> body)) + + val inst = new MacroOpDesc + inst.macroId = "265" + inst.macroVersion = 1 + inst.linkMode = MacroOpDesc.LIVE + inst.inputPortCount = 1 + inst.outputPortCount = 1 + inst.setOperatorId("Macro-operator-acc00f1c") + + val csvOp = new CSVScanSourceOpDesc() + csvOp.fileName = Some("workflow-compiling-service/src/test/resources/country_sales_small.csv") + csvOp.customDelimiter = Some(",") + csvOp.hasHeader = true + csvOp.setOperatorId("CSVScan") + + val sink = limit("sink", 5) + + val pojo = LogicalPlanPojo( + operators = List(csvOp, inst, sink), + links = List( + LogicalLink(csvOp.operatorIdentifier, PortIdentity(0), inst.operatorIdentifier, PortIdentity(0)), + LogicalLink(inst.operatorIdentifier, PortIdentity(0), sink.operatorIdentifier, PortIdentity(0)) + ), + opsToViewResult = List(), + opsToReuseResult = List() + ) + + val ctx = new WorkflowContext(workflowId = WorkflowIdentity(0)) + val result = new WorkflowCompiler(ctx, registry).compile(pojo) + + // Outer workflow compiles despite the dangling inner error. + result.physicalPlan should not be empty + // Error is attributed to the macro on the canvas, not the inner op. + result.operatorIdToError.keys.map(_.id) should contain("Macro-operator-acc00f1c") + result.operatorIdToError.keys.map(_.id) should not contain "sink" + } + + it should "attribute a schema error inside a macro body to the visible macro instance, not the prefixed inner op" in { + // body: in → limit("inner", 7) → out — limit's input schema can't be computed when + // the macro has no upstream connection on the parent canvas. + val body = MacroBody( + operators = List(inMarker(0, "in"), limit("inner", 7), outMarker(0, "out")), + links = List( + MacroLink("in", PortIdentity(0), "inner", PortIdentity(0)), + MacroLink("inner", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val inst = snapshotInstance("Lonely", "macro-A", body) + + val pojo = LogicalPlanPojo( + operators = List(inst), + links = List(), + opsToViewResult = List(), + opsToReuseResult = List() + ) + + val ctx = new WorkflowContext(workflowId = WorkflowIdentity(0)) + val result = new WorkflowCompiler(ctx).compile(pojo) + + // The frontend canvas only shows "Lonely" — error must be keyed under that ID, + // not the post-expansion "Lonely--inner", or it never reaches the macro node UI. + result.operatorIdToError.keys.map(_.id).toSet shouldBe Set("Lonely") + val err = result.operatorIdToError.values.head + err.operatorId shouldBe "Lonely" + // Inner op id stays in the message so the developer knows which body op blew up. + err.message should include("Lonely--inner") + err.message should include("schema is not available") + } + + it should "short-circuit standalone macro-body compiles (markers present, no MacroOpDesc) to a clean success" in { + // Mirrors the drill-down editor: the frontend reloads `workflow.content` for a + // macro and the body — markers + inner ops, NO MacroOpDesc — gets fed straight + // into /compile by the singleton WorkflowCompilingService. Pre-fix, the markers + // threw IllegalStateException, every inner op downstream failed schema + // propagation, and the resulting "Failed" state would persist across the + // drill-down → parent navigation in the singleton compile-state, making the + // parent canvas look broken until the parent's own compile finished. + val pojo = LogicalPlanPojo( + operators = List(inMarker(0, "in"), limit("inner", 10), outMarker(0, "out")), + links = List( + LogicalLink(OperatorIdentity("in"), PortIdentity(0), OperatorIdentity("inner"), PortIdentity(0)), + LogicalLink(OperatorIdentity("inner"), PortIdentity(0), OperatorIdentity("out"), PortIdentity(0)) + ), + opsToViewResult = List(), + opsToReuseResult = List() + ) + + val ctx = new WorkflowContext(workflowId = WorkflowIdentity(0)) + val result = new WorkflowCompiler(ctx).compile(pojo) + + result.operatorIdToError shouldBe empty + result.physicalPlan should not be empty + result.physicalPlan.get.operators shouldBe empty + result.operatorIdToOutputSchemas shouldBe empty + } + + it should "still compile a parent that uses a macro instance (short-circuit does NOT apply post-expansion markers)" in { + // Regression guard: the short-circuit fires on the *raw* plan before + // MacroExpander runs. A parent canvas legitimately holds a MacroOpDesc + // (which carries markers in its embedded body) and must take the full + // compile path. Otherwise we'd silently swallow real parent compiles. + val body = MacroBody( + operators = List(inMarker(0, "in"), limit("inner", 10), outMarker(0, "out")), + links = List( + MacroLink("in", PortIdentity(0), "inner", PortIdentity(0)), + MacroLink("inner", PortIdentity(0), "out", PortIdentity(0)) + ) + ) + val inst = snapshotInstance("ParentMacro", "macro-A", body) + + val csvOp = new CSVScanSourceOpDesc() + csvOp.fileName = Some("workflow-compiling-service/src/test/resources/country_sales_small.csv") + csvOp.customDelimiter = Some(",") + csvOp.hasHeader = true + csvOp.setOperatorId("CSVScan-A") + + val pojo = LogicalPlanPojo( + operators = List(csvOp, inst), + links = List( + LogicalLink(csvOp.operatorIdentifier, PortIdentity(0), inst.operatorIdentifier, PortIdentity(0)) + ), + opsToViewResult = List(), + opsToReuseResult = List() + ) + + val ctx = new WorkflowContext(workflowId = WorkflowIdentity(0)) + val result = new WorkflowCompiler(ctx).compile(pojo) + + result.operatorIdToError shouldBe empty + result.physicalPlan should not be empty + // The expanded plan should have actually compiled — non-empty physical ops, + // proving we took the full path, not the short-circuit. + result.physicalPlan.get.operators should not be empty } // ---------- helpers ---------- From 49beec99e4f72ab989004bb92aa0c31b295adf84 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Fri, 15 May 2026 22:17:00 -0700 Subject: [PATCH 20/65] fix(macro): preserve full sub-DAG external interface as macro ports MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Previously buildMacroFromSelection only created MacroInput/MacroOutput markers for ports that already had external links at macro-creation time. A selection like Filter → Projection where Projection's output wasn't yet connected ended up as a macro with one input port and zero output ports, breaking dataflow equivalence: the user couldn't reach Projection's output through the macro at all. Replacing a sub-DAG with a macro op is a structural substitution. Every input port on the selection that isn't fed by another selected op is a boundary input regardless of current external connectivity, and symmetrically for outputs. Walk selectedOperatorIDs × op.inputPorts/ outputPorts, filter out the internally-wired ones, and synthesize one marker per remaining port. The actual-external-edge rewiring (incomingEdges/outgoingEdges) is unchanged — it just maps a subset of the available macro ports. Co-Authored-By: Claude Sonnet 4.6 --- .../workspace/service/macro/macro.service.ts | 68 ++++++++++++------- 1 file changed, 44 insertions(+), 24 deletions(-) diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index 0c29c29632f..13f7e8cd8c5 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -187,32 +187,52 @@ export class MacroService { else if (srcIn && !dstIn) outgoing.push(entry); }); - // Allocate one MacroInput marker per unique (innerOp, innerPort) that is - // fed by at least one external link. A single marker can have multiple - // external feeders but it only drives one inner port. - const incomingKeys = Array.from(new Set(incoming.map(l => `${l.dstOp}|${l.dstPort}`))).sort(); - const inputMarkers = incomingKeys.map((key, idx) => { - const [innerOpId, innerPortID] = key.split("|"); - return { - markerOpId: `MacroInput-operator-${uuid()}`, - portIndex: idx, - innerOpId, - innerPortID, - innerPortIdx: inputPortOrdinal(innerOpId, innerPortID), - }; + // Preserve the sub-DAG's full external interface, not just the ports that + // happen to be wired up at macro-creation time. Replacing a sub-DAG with a + // macro op is a dataflow-equivalence transformation: every input port on + // the selection that isn't fed by another selected op is a boundary input + // (regardless of whether an external feeder is currently connected), and + // symmetrically for output ports. That way a selection of + // Filter → Projection where Projection's output is currently unwired still + // surfaces that output as an external macro port the user can connect later. + const internallyFedInputPorts = new Set(internal.map(l => `${l.dstOp}|${l.dstPort}`)); + const internallyConsumedOutputPorts = new Set(internal.map(l => `${l.srcOp}|${l.srcPort}`)); + + type BoundaryPort = { innerOpId: string; innerPortID: string; innerPortIdx: number }; + const boundaryInputPorts: BoundaryPort[] = []; + const boundaryOutputPorts: BoundaryPort[] = []; + selectedOperatorIDs.forEach(opId => { + const op = graph.getOperator(opId); + op.inputPorts.forEach((port, idx) => { + if (!internallyFedInputPorts.has(`${opId}|${port.portID}`)) { + boundaryInputPorts.push({ innerOpId: opId, innerPortID: port.portID, innerPortIdx: idx }); + } + }); + op.outputPorts.forEach((port, idx) => { + if (!internallyConsumedOutputPorts.has(`${opId}|${port.portID}`)) { + boundaryOutputPorts.push({ innerOpId: opId, innerPortID: port.portID, innerPortIdx: idx }); + } + }); }); - const outgoingKeys = Array.from(new Set(outgoing.map(l => `${l.srcOp}|${l.srcPort}`))).sort(); - const outputMarkers = outgoingKeys.map((key, idx) => { - const [innerOpId, innerPortID] = key.split("|"); - return { - markerOpId: `MacroOutput-operator-${uuid()}`, - portIndex: idx, - innerOpId, - innerPortID, - innerPortIdx: outputPortOrdinal(innerOpId, innerPortID), - }; - }); + // Allocate one MacroInput/MacroOutput marker per boundary port. Marker + // ordering follows the selection's visual order (selectedOperatorIDs × + // op.inputPorts), giving the user a stable mapping between macro ports + // and the underlying sub-DAG ports. + const inputMarkers = boundaryInputPorts.map((p, idx) => ({ + markerOpId: `MacroInput-operator-${uuid()}`, + portIndex: idx, + innerOpId: p.innerOpId, + innerPortID: p.innerPortID, + innerPortIdx: p.innerPortIdx, + })); + const outputMarkers = boundaryOutputPorts.map((p, idx) => ({ + markerOpId: `MacroOutput-operator-${uuid()}`, + portIndex: idx, + innerOpId: p.innerOpId, + innerPortID: p.innerPortID, + innerPortIdx: p.innerPortIdx, + })); // Marker ports follow the backend's `PortDescription` shape (portID string, // disallowMultiInputs/isDynamicPort flags) so MacroBody parses cleanly when From e6ee6ced81b1f72ed9c0539208405a1d9db7e0d9 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Fri, 15 May 2026 22:40:55 -0700 Subject: [PATCH 21/65] feat(macro): port-level stats + results on outer canvas, drill-down execution view MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stitches the parent workflow's execution data — both stats (row counts, state) and result rows — onto each external port of a Macro op, and makes the drill-down view show the same data per inner op while the parent is running. Wire layout (frontend-only; engine stays macro-unaware): * MacroService now computes per-definition body bindings — each Macro external port `i` knows the body-relative (innerOp, innerPort) it routes to via the MacroInput(i) / MacroOutput(i) markers. Cached on first fetch; preloaded on `getOperatorAddStream` so the map is ready before execution starts. `getBindingsForInstance(instanceId, macroId)` lifts the body-relative IDs to runtime form (`${instanceId}--`) so they match the engine's stat/result keys post-MacroExpander. * WorkflowEditorComponent.synthesizeMacroOpStats sources per-port row counts for each Macro on the outer canvas: macro input `i` reads from the boundary inner op's `inputPortMetrics` at the body-link's target port; macro output `j` reads from the inner op's `outputPortMetrics`. Falls through to `withMacroAggregates`-supplied state until bindings load, then refreshes on the next stats emission. * WorkflowResultService gains a macro-instance result alias plus a drill-down prefix. The alias routes `getResultService(macroId)` to the inner op feeding output port 0, so the result panel shows the macro's output without forcing the user to drill in. The drill-down prefix transparently rewrites every result lookup to its runtime form when the canvas is rendering a body via `?instance=...`. * WorkflowEditorComponent listens to `route.queryParamMap.instance` — the macro click-through now appends it to the drill-down URL — and applies the same `${instanceId}--` prefix to stat lookups so live parent-execution stats land on the body-relative op IDs the drill-down canvas displays. * Port-mapping completeness fix already in 49beec99e4 is the critical upstream prerequisite: a Macro op with only an `input-0` port (and no output port) can't be made to display output stats or results no matter how the websocket layer is wired. Co-Authored-By: Claude Sonnet 4.6 --- .../workflow-editor.component.ts | 159 +++++++++++++-- .../workspace/service/macro/macro.service.ts | 188 +++++++++++++++++- .../workflow-result.service.ts | 46 ++++- 3 files changed, 377 insertions(+), 16 deletions(-) diff --git a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts index 3beb23f39ba..b1e370d5b26 100644 --- a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts @@ -28,7 +28,7 @@ import { fromJointPaperEvent, JointUIService, linkPathStrokeColor } from "../../ import { ValidationWorkflowService } from "../../service/validation/validation-workflow.service"; import { WorkflowActionService } from "../../service/workflow-graph/model/workflow-action.service"; import { WorkflowStatusService } from "../../service/workflow-status/workflow-status.service"; -import { ExecutionState, OperatorState } from "../../types/execute-workflow.interface"; +import { ExecutionState, OperatorState, OperatorStatistics } from "../../types/execute-workflow.interface"; import { LogicalPort, OperatorLink, OperatorPredicate } from "../../types/workflow-common.interface"; import { auditTime, filter, map, takeUntil, withLatestFrom } from "rxjs/operators"; import { UntilDestroy, untilDestroyed } from "@ngneat/until-destroy"; @@ -44,6 +44,8 @@ import { GuiConfigService } from "../../../common/service/gui-config.service"; import { line, curveCatmullRomClosed } from "d3-shape"; import concaveman from "concaveman"; import { OperatorResultSummary, AgentService } from "../../service/agent/agent.service"; +import { MacroService, MacroBindings } from "../../service/macro/macro.service"; +import { WorkflowResultService } from "../../service/workflow-result/workflow-result.service"; import { NzNoAnimationDirective } from "ng-zorro-antd/core/animation"; import { ContextMenuComponent } from "./context-menu/context-menu/context-menu.component"; import { NgIf } from "@angular/common"; @@ -128,7 +130,9 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy public nzContextMenu: NzContextMenuService, private elementRef: ElementRef, private config: GuiConfigService, - private agentService: AgentService + private agentService: AgentService, + private macroService: MacroService, + private workflowResultService: WorkflowResultService ) { this.wrapper = this.workflowActionService.getJointGraphWrapper(); } @@ -146,6 +150,26 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy this.changeDetectorRef.detectChanges(); } }); + + // Eagerly fetch macro body bindings so port-level stat/result remap is + // ready by the time execution starts. Prefetch on (a) initial load — + // covers macros that arrive via reloadWorkflow before this subscriber is + // wired — and (b) future add events. + const graph = this.workflowActionService.getTexeraGraph(); + this.macroService.prefetchBindingsForOperators(graph.getAllOperators()); + graph + .getOperatorAddStream() + .pipe(untilDestroyed(this)) + .subscribe(op => this.macroService.prefetchBindingsForOperators([op])); + + // Keep the result service's drill-down prefix in sync with the URL — when + // we're on `?instance=…`, body-relative ID lookups should resolve to the + // runtime (`${instanceId}--`) form so live execution results show up + // inside the drilled-down view. + this.route.queryParamMap.pipe(untilDestroyed(this)).subscribe(qp => { + const instance = qp.get("instance"); + this.workflowResultService.setDrilldownPrefix(instance ? `${instance}--` : ""); + }); } /** @@ -311,24 +335,41 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy .getStatusUpdateStream() .pipe(untilDestroyed(this)) .subscribe(status => { + // If the user drilled into a macro body via + // `/workflow/:id/macro/:macroId?instance=...`, the canvas operators + // have *body-relative* IDs (e.g. `Filter-uuid`) but the engine emits + // stats keyed by `${instanceId}--${bodyOpId}` for the parent run. + // Build a stat lookup that prefixes body-relative IDs with the macro + // instance prefix so drill-down view sees live execution stats. + const macroInstancePrefix = this.getDrilldownInstancePrefix(); + const lookupStat = (operatorId: string): OperatorStatistics | undefined => + macroInstancePrefix ? status[`${macroInstancePrefix}${operatorId}`] : status[operatorId]; + this.workflowActionService .getTexeraGraph() .getAllOperators() .forEach(op => { - if ( - isDefined(status[op.operatorID]) && + // Macro op's stats need port-level remap: each external port of + // the macro corresponds to a specific boundary inner-op port, so + // looking up `status[macroId]` directly would give us either + // nothing (no engine entry for the macro itself) or the aggregated + // sum from `withMacroAggregates` (which has empty port metrics). + // Synthesize the right per-port view from cached macro bindings. + const opStatus = + op.operatorType === "Macro" && !macroInstancePrefix + ? this.synthesizeMacroOpStats(op, status) ?? status[op.operatorID] + : lookupStat(op.operatorID); + + const finalStatus = + isDefined(opStatus) && this.executeWorkflowService.getExecutionState().state === ExecutionState.Recovering - ) { - status[op.operatorID] = { - ...status[op.operatorID], - operatorState: OperatorState.Recovering, - }; - } + ? { ...opStatus, operatorState: OperatorState.Recovering } + : opStatus; this.jointUIService.changeOperatorStatistics( this.paper, op.operatorID, - status[op.operatorID], + finalStatus, this.isSource(op.operatorID), this.isSink(op.operatorID) ); @@ -360,6 +401,95 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy }); } + /** + * If the current view is a macro drill-down (URL carries `?instance=...` + * alongside `/macro/:macroId`), return the prefix to apply when looking up + * inner-op stats — engine reports them as `${instanceId}--${bodyOpId}`. + * Returns `""` (no prefix) when not in drill-down mode. + */ + private getDrilldownInstancePrefix(): string { + const instanceId = this.route.snapshot.queryParamMap.get("instance"); + const macroId = this.route.snapshot.paramMap.get("macroId"); + if (!macroId || !instanceId) return ""; + return `${instanceId}--`; + } + + /** + * Build an `OperatorStatistics` for a macro instance by sourcing per-port + * data from the boundary inner ops the macro's external ports map to. + * + * Why: after `MacroExpander` inlines the body, the engine reports stats + * keyed by prefixed inner-op IDs (e.g. `Macro-operator-abc--Filter-uuid`). + * The macro op itself has no engine-side entity — so directly looking up + * `status[macro.operatorID]` returns either undefined or the aggregated + * roll-up that `WorkflowStatusService.withMacroAggregates` synthesized + * (which deliberately leaves port metrics empty because port-level + * mapping requires the body shape). + * + * Mapping rules: + * - macro external input `i` shows the *input* row count of the inner op + * that the corresponding `MacroInput(portIndex=i)` feeds, at the inner + * port the body link targets (one or more — sum if it fans out). + * - macro external output `j` shows the *output* row count of the inner + * op that feeds the corresponding `MacroOutput(portIndex=j)`, at the + * inner port the body link sources from. + * - The overall `operatorState` and aggregated totals fall through from + * the roll-up entry produced by `WorkflowStatusService`. + * + * Returns `undefined` if bindings aren't cached yet — caller falls back + * to the roll-up entry (so the macro still gets a state, just no port + * counts) and a subsequent stats event after the body fetches will + * refresh with the proper port metrics. + */ + private synthesizeMacroOpStats( + macroOp: OperatorPredicate, + status: Record + ): OperatorStatistics | undefined { + const macroId = macroOp.operatorProperties?.["macroId"]; + if (typeof macroId !== "string" || macroId.length === 0) return undefined; + const bindings: MacroBindings | undefined = this.macroService.getBindingsForInstance( + macroOp.operatorID, + macroId + ); + if (!bindings) return undefined; + + const base = status[macroOp.operatorID]; + const inputPortMetrics: Record = {}; + const outputPortMetrics: Record = {}; + + // Group bindings by external port index so a fanned-out input port sums + // the row counts of its multiple downstream inner consumers (rare, but + // possible — see spliceIntoParent's `inputConsumers` map). + for (const binding of bindings.inputBindings) { + const innerStats = status[binding.innerOpId]; + if (!innerStats) continue; + const innerPortKey = String(binding.innerPortIndex); + const innerPortCount = innerStats.inputPortMetrics?.[innerPortKey] ?? 0; + const macroPortKey = String(binding.externalPortIndex); + inputPortMetrics[macroPortKey] = (inputPortMetrics[macroPortKey] ?? 0) + innerPortCount; + } + for (const binding of bindings.outputBindings) { + const innerStats = status[binding.innerOpId]; + if (!innerStats) continue; + const innerPortKey = String(binding.innerPortIndex); + const innerPortCount = innerStats.outputPortMetrics?.[innerPortKey] ?? 0; + const macroPortKey = String(binding.externalPortIndex); + outputPortMetrics[macroPortKey] = innerPortCount; + } + + const aggregatedInputRowCount = Object.values(inputPortMetrics).reduce((a, b) => a + b, 0); + const aggregatedOutputRowCount = Object.values(outputPortMetrics).reduce((a, b) => a + b, 0); + + return { + operatorState: base?.operatorState ?? OperatorState.Uninitialized, + aggregatedInputRowCount, + inputPortMetrics, + aggregatedOutputRowCount, + outputPortMetrics, + numWorkers: base?.numWorkers, + }; + } + private handleRegionEvents(): void { this.editor.classList.add("hide-region"); const Region = joint.dia.Element.define( @@ -583,8 +713,13 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy const macroId = op?.operatorProperties?.["macroId"]; if (op?.operatorType === "Macro" && macroId) { const parentWid = this.route.snapshot.params.id ?? ""; + // Carry the macro instance ID (= the clicked op's operatorID) as + // a query param so the drill-down view can map live execution + // stats from the parent — engine reports inner-op stats under + // `${instanceId}--${innerOpId}` keys, and the drill-down canvas + // only has the un-prefixed inner-op IDs. window.location.href = - `/dashboard/user/workflow/${parentWid}/macro/${macroId}`; + `/dashboard/user/workflow/${parentWid}/macro/${macroId}?instance=${encodeURIComponent(elementID)}`; return; } this.workflowActionService.openResultPanel(); diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index 13f7e8cd8c5..e03d7720c8e 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -19,7 +19,8 @@ import { HttpClient } from "@angular/common/http"; import { Injectable } from "@angular/core"; -import { Observable } from "rxjs"; +import { Observable, ReplaySubject, of, shareReplay } from "rxjs"; +import { tap, map, catchError } from "rxjs/operators"; import { AppSettings } from "../../../common/app-setting"; import { ExecutionMode, Workflow, WorkflowContent } from "../../../common/type/workflow"; import { @@ -30,8 +31,25 @@ import { } from "../../types/workflow-common.interface"; import { PortIdentity } from "../../types/execute-workflow.interface"; import { WorkflowActionService } from "../workflow-graph/model/workflow-action.service"; +import { WorkflowResultService } from "../workflow-result/workflow-result.service"; import { v4 as uuid } from "uuid"; +// Per-instance runtime mapping from the macro's external ports back to the +// boundary inner-op port that actually carries the data. After MacroExpander +// inlines the body into the parent plan, inner-op IDs gain a "${macroInstanceId}--" +// prefix, so each entry is already keyed by the *runtime* inner-op ID — ready +// for direct lookup against `OperatorStatisticsUpdateEvent.operatorStatistics`. +export interface MacroPortBinding { + externalPortIndex: number; + innerOpId: string; // post-expansion / runtime ID, ready to look up against engine stats + innerPortIndex: number; +} + +export interface MacroBindings { + inputBindings: MacroPortBinding[]; + outputBindings: MacroPortBinding[]; +} + export const MACRO_BASE_URL = "macro"; export const MACRO_CREATE_URL = MACRO_BASE_URL + "/create"; export const MACRO_LIST_URL = MACRO_BASE_URL + "/list"; @@ -105,7 +123,21 @@ interface MacroBody { providedIn: "root", }) export class MacroService { - constructor(private http: HttpClient) {} + constructor( + private http: HttpClient, + private workflowResultService: WorkflowResultService + ) {} + + // Cached per-definition body bindings, keyed by `${macroId}` (the macro + // definition's wid). Each entry is a hot Observable so multiple subscribers + // share the same HTTP fetch. The body of a macro definition is immutable + // for the lifetime of a given (macroId, vid) tuple, so caching by macroId + // alone is safe — definition edits go through a new wid in the v1 LIVE mode. + private bodyBindingsCache = new Map>(); + // Latest-known synchronous snapshot — populated by `getBindingsForInstance` + // after the first successful fetch so synchronous stat-update handlers can + // look up bindings without re-triggering the network call. + private bodyBindingsSnapshot = new Map(); public createMacro(req: MacroCreateRequest): Observable { return this.http.post(`${AppSettings.getApiEndpoint()}/${MACRO_CREATE_URL}`, req); @@ -119,6 +151,158 @@ export class MacroService { return this.http.get(`${AppSettings.getApiEndpoint()}/${MACRO_BASE_URL}/${wid}`); } + /** + * Compute body-level port bindings for the macro DEFINITION identified by + * `macroId` (the definition's wid). The bindings name body-relative inner + * op IDs — callers that need *runtime* IDs (after MacroExpander's prefix + * rewrite) should use `getBindingsForInstance` instead. + * + * Body bindings are derived from the persisted `MacroBody`: + * - each `MacroInput(portIndex=i)` is followed by one or more links + * `marker → innerOp@(p)`; we record (i → innerOp, p) for stats fan-out + * - each `MacroOutput(portIndex=i)` is preceded by exactly one link + * `innerOp@(p) → marker`; we record (i → innerOp, p) for stats/results + * + * Cached and shared across subscribers. + */ + public getBodyBindings( + macroId: string + ): Observable<{ inputBindings: MacroPortBinding[]; outputBindings: MacroPortBinding[] }> { + const cached = this.bodyBindingsCache.get(macroId); + if (cached) return cached; + const widNum = Number(macroId); + if (!Number.isFinite(widNum)) { + const empty = { inputBindings: [], outputBindings: [] }; + this.bodyBindingsSnapshot.set(macroId, empty); + return of(empty); + } + const fetched = this.getMacro(widNum).pipe( + map(detail => this.computeBodyBindings(detail)), + tap(bindings => this.bodyBindingsSnapshot.set(macroId, bindings)), + catchError(() => of({ inputBindings: [] as MacroPortBinding[], outputBindings: [] as MacroPortBinding[] })), + shareReplay(1) + ); + this.bodyBindingsCache.set(macroId, fetched); + return fetched; + } + + /** + * Resolve bindings to runtime IDs for one macro instance on the parent + * canvas. `${instanceId}--` is the prefix MacroExpander adds to every + * inner-op ID when it inlines the body (see + * `workflow-compiling-service/.../MacroExpander.scala`). After this rewrite + * the engine reports stats keyed by the prefixed strings — so we apply the + * same rewrite here so callers can do straight-up `stats[innerOpId]` lookups. + * + * Returns the cached snapshot synchronously when available so stats-update + * handlers don't have to await; preload via `prefetchBindingsForOperators` + * to make sure the snapshot is populated by the time execution starts. + */ + public getBindingsForInstance(macroInstanceId: string, macroId: string): MacroBindings | undefined { + const snapshot = this.bodyBindingsSnapshot.get(macroId); + if (!snapshot) { + // kick off fetch so future calls hit the snapshot + this.getBodyBindings(macroId).subscribe({ error: () => undefined }); + return undefined; + } + return { + inputBindings: snapshot.inputBindings.map(b => ({ + externalPortIndex: b.externalPortIndex, + innerOpId: `${macroInstanceId}--${b.innerOpId}`, + innerPortIndex: b.innerPortIndex, + })), + outputBindings: snapshot.outputBindings.map(b => ({ + externalPortIndex: b.externalPortIndex, + innerOpId: `${macroInstanceId}--${b.innerOpId}`, + innerPortIndex: b.innerPortIndex, + })), + }; + } + + /** + * Eagerly fetch bindings for every Macro op currently on the canvas, and + * register the macro-instance → inner-op alias used by + * `WorkflowResultService` so the result panel can show the macro's output + * (we route to output port 0's inner producer as the canonical "macro + * result"; a future multi-output UX could expose all outputs). + * Idempotent (cache-keyed), so spamming on every op-add stream emission + * does at most one HTTP per definition. + */ + public prefetchBindingsForOperators(operators: readonly OperatorPredicate[]): void { + for (const op of operators) { + if (op.operatorType !== "Macro") continue; + const macroId = op.operatorProperties?.["macroId"]; + if (typeof macroId !== "string" || macroId.length === 0) continue; + const instanceId = op.operatorID; + this.getBodyBindings(macroId).subscribe({ + next: bindings => { + const out0 = bindings.outputBindings.find(b => b.externalPortIndex === 0); + if (out0) { + this.workflowResultService.setMacroResultAlias( + instanceId, + `${instanceId}--${out0.innerOpId}` + ); + } + }, + error: () => undefined, + }); + } + } + + private computeBodyBindings(detail: MacroDetail): { + inputBindings: MacroPortBinding[]; + outputBindings: MacroPortBinding[]; + } { + let body: MacroBody; + try { + body = JSON.parse(detail.content) as MacroBody; + } catch { + return { inputBindings: [], outputBindings: [] }; + } + const inputMarkerByPortIndex = new Map(); + const outputMarkerByPortIndex = new Map(); + for (const raw of body.operators) { + const op = raw as { operatorID?: string; operatorType?: string; portIndex?: number }; + if (typeof op.operatorID !== "string" || typeof op.portIndex !== "number") continue; + if (op.operatorType === "MacroInput") inputMarkerByPortIndex.set(op.portIndex, op.operatorID); + else if (op.operatorType === "MacroOutput") outputMarkerByPortIndex.set(op.portIndex, op.operatorID); + } + const markerIds = new Set([ + ...Array.from(inputMarkerByPortIndex.values()), + ...Array.from(outputMarkerByPortIndex.values()), + ]); + // For each MacroInput, find body links marker -> innerOp@(p) — there can + // be multiple if the macro's external input fans out to several inner + // consumers (the rare "split feed" case in spliceIntoParent). + const inputBindings: MacroPortBinding[] = []; + for (const [portIndex, markerId] of inputMarkerByPortIndex.entries()) { + for (const link of body.links) { + if (link.fromOpId !== markerId) continue; + if (markerIds.has(link.toOpId)) continue; // marker → marker is malformed; skip + inputBindings.push({ + externalPortIndex: portIndex, + innerOpId: link.toOpId, + innerPortIndex: link.toPortId.id, + }); + } + } + // For each MacroOutput, find body links innerOp@(p) -> marker — exactly + // one producer per output marker (MacroExpander already enforces this). + const outputBindings: MacroPortBinding[] = []; + for (const [portIndex, markerId] of outputMarkerByPortIndex.entries()) { + for (const link of body.links) { + if (link.toOpId !== markerId) continue; + if (markerIds.has(link.fromOpId)) continue; + outputBindings.push({ + externalPortIndex: portIndex, + innerOpId: link.fromOpId, + innerPortIndex: link.fromPortId.id, + }); + } + } + return { inputBindings, outputBindings }; + } + /** * Build a `MacroCreateRequest` from the operators the user has multi-selected * on the parent canvas, plus the boundary info the caller needs to swap the diff --git a/frontend/src/app/workspace/service/workflow-result/workflow-result.service.ts b/frontend/src/app/workspace/service/workflow-result/workflow-result.service.ts index 9fd18e0f161..6bcc9ea82e1 100644 --- a/frontend/src/app/workspace/service/workflow-result/workflow-result.service.ts +++ b/frontend/src/app/workspace/service/workflow-result/workflow-result.service.ts @@ -45,6 +45,14 @@ export class WorkflowResultService { private paginatedResultServices = new Map(); private operatorResultServices = new Map(); + // Alias map for macro instance IDs: macro op IDs on the canvas don't get + // direct result entries from the engine (the engine sees the inlined inner + // ops only). When a macro has at least one output port, route lookups for + // the macro to the inner op feeding output port 0 so the result panel can + // show "the macro's result" without the user having to drill down. Set by + // `MacroService` once body bindings are fetched. + private macroResultAliases = new Map(); + // event stream of operator result update, undefined indicates the operator result is cleared private resultUpdateStream = new Subject>(); private resultTableStats = new ReplaySubject>>>(1); @@ -73,6 +81,40 @@ export class WorkflowResultService { return isDefined(this.getPaginatedResultService(operatorID)); } + /** + * Register/refresh the macro-instance → inner-op alias used to resolve + * `getResultService` / `getPaginatedResultService` lookups for macro ops. + * Idempotent — call whenever a macro's body bindings finish loading. + * `innerOpId` must be a *runtime* (post-MacroExpander-prefix) ID so it + * matches what the engine sends in `WebResultUpdateEvent`. + */ + public setMacroResultAlias(macroInstanceId: string, innerOpId: string): void { + this.macroResultAliases.set(macroInstanceId, innerOpId); + } + + public clearMacroResultAlias(macroInstanceId: string): void { + this.macroResultAliases.delete(macroInstanceId); + } + + // When the canvas is rendering a macro body (drill-down view), the operators + // on the canvas have body-relative IDs (`Filter-uuid`) but engine results + // arrive keyed by the post-expansion runtime ID (`${instanceId}--Filter-uuid`). + // Set this prefix to make every result lookup transparently target the + // runtime ID. Empty string disables the rewrite. + private drilldownPrefix: string = ""; + + public setDrilldownPrefix(prefix: string): void { + this.drilldownPrefix = prefix; + } + + private resolveAlias(operatorID: string): string { + // Drill-down rewrite wins: when viewing a macro body during execution we + // want the body-relative op ID lifted to its runtime form. Macro aliases + // only fire on the outer canvas, where body-relative IDs aren't present. + if (this.drilldownPrefix.length > 0) return `${this.drilldownPrefix}${operatorID}`; + return this.macroResultAliases.get(operatorID) ?? operatorID; + } + public getResultUpdateStream(): Observable> { return this.resultUpdateStream; } @@ -88,11 +130,11 @@ export class WorkflowResultService { } public getPaginatedResultService(operatorID: string): OperatorPaginationResultService | undefined { - return this.paginatedResultServices.get(operatorID); + return this.paginatedResultServices.get(this.resolveAlias(operatorID)); } public getResultService(operatorID: string): OperatorResultService | undefined { - return this.operatorResultServices.get(operatorID); + return this.operatorResultServices.get(this.resolveAlias(operatorID)); } private handleCleanResultCache(event: WorkflowAvailableResultEvent): void { From ab108806aae48d2261d8b8e0aa54e97a0a24de02 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Fri, 15 May 2026 23:50:21 -0700 Subject: [PATCH 22/65] feat(macro): view-result, nested macros, SPA drill-down navigation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three coupled execution-path fixes: * Item 3 — view-result/reuse-result on a macro op now forwards to the inner boundary ops the macro's external outputs route to. Backend's `opsToViewResult` is keyed by post-expansion op IDs (the macro op itself doesn't survive MacroExpander), so executeWorkflowWith… rewrites macro IDs to `${instanceId}--${innerOpId}` for every output binding before submitting the plan. Multi-output macros mark all their output producers; non-macro IDs pass through unchanged. Same rewrite for `opsToReuseResult`. * Item 2 — `MacroService.computeBodyBindings` now also collects `nestedMacros: Map` and `getBindingsForInstance` walks them recursively, prefixing `\${instanceId}--` at each layer until a terminal non-macro inner op is reached. Fan-out at any layer is preserved by emitting one resolved binding per terminal. Bodies of nested macros are eagerly prefetched when their parent body loads, so the synchronous stat lookup path finds everything cached. * Item 1 — macro drill-down click-through switched from window.location.href to Router.navigate. Full reload was killing the parent's websocket subscription, so the drill-down view saw no live execution stats. SPA navigation keeps WorkflowWebsocketService alive across the route change, and the existing query-param (?instance=...) handler in WorkflowEditorComponent already maps body-relative op IDs onto runtime stat keys for the drilled-down canvas. loadMacroWithId simplified to match loadWorkflowWithId's pattern (drop the redundant resetAsNewWorkflow — setNewSharedModel + reloadWorkflow together do a clean transition). Co-Authored-By: Claude Sonnet 4.6 --- .../workflow-editor.component.ts | 31 ++- .../component/workspace.component.html | 2 +- .../component/workspace.component.ts | 18 +- .../execute-workflow.service.ts | 67 +++++- .../workspace/service/macro/macro.service.ts | 192 ++++++++++++++---- 5 files changed, 246 insertions(+), 64 deletions(-) diff --git a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts index b1e370d5b26..b03ddbf95cd 100644 --- a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts @@ -700,26 +700,25 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy if (this.workflowActionService.getTexeraGraph().hasCommentBox(elementID)) { this.openCommentBox(elementID); } else if (this.workflowActionService.getTexeraGraph().hasOperator(elementID)) { - // Macro nodes drill down into their body via a route change. Use a - // hard navigation (window.location.href) instead of the Angular - // router so the WorkspaceComponent is destroyed and re-bootstrapped: - // the YJS shared-model + JointJS canvas retain enough cross-route - // state that an SPA navigation leaves stale operators/links behind - // (manifested as "duplicate link found" rejections and a - // half-rendered body). Page-reload is the only known-clean path - // until that lifecycle is properly untangled. Cost is a brief flash; - // worth it for predictable rendering. + // Macro nodes drill down into their body via an in-app route + // change. Use Angular router (not window.location.href) so the + // execution websocket survives the navigation — full reload + // would tear down the parent's `WorkflowWebsocketService` and + // the drill-down view would see no live stats. `loadMacroWithId` + // is responsible for fully resetting the canvas/YJS room before + // reloading the body so we don't hit duplicate-link errors. const op = this.workflowActionService.getTexeraGraph().getOperator(elementID); const macroId = op?.operatorProperties?.["macroId"]; if (op?.operatorType === "Macro" && macroId) { const parentWid = this.route.snapshot.params.id ?? ""; - // Carry the macro instance ID (= the clicked op's operatorID) as - // a query param so the drill-down view can map live execution - // stats from the parent — engine reports inner-op stats under - // `${instanceId}--${innerOpId}` keys, and the drill-down canvas - // only has the un-prefixed inner-op IDs. - window.location.href = - `/dashboard/user/workflow/${parentWid}/macro/${macroId}?instance=${encodeURIComponent(elementID)}`; + this.router.navigate(["/dashboard/user/workflow", parentWid, "macro", macroId], { + // Carry the macro instance ID so the drill-down view can map + // live execution stats from the parent — engine reports + // inner-op stats under `${instanceId}--${innerOpId}` keys, + // and the drill-down canvas only has the un-prefixed + // inner-op IDs. + queryParams: { instance: elementID }, + }); return; } this.workflowActionService.openResultPanel(); diff --git a/frontend/src/app/workspace/component/workspace.component.html b/frontend/src/app/workspace/component/workspace.component.html index 77ee449a82a..b3285c188f1 100644 --- a/frontend/src/app/workspace/component/workspace.component.html +++ b/frontend/src/app/workspace/component/workspace.component.html @@ -31,7 +31,7 @@ ← Back to parent diff --git a/frontend/src/app/workspace/component/workspace.component.ts b/frontend/src/app/workspace/component/workspace.component.ts index c0637c94929..7edab81ceef 100644 --- a/frontend/src/app/workspace/component/workspace.component.ts +++ b/frontend/src/app/workspace/component/workspace.component.ts @@ -29,7 +29,7 @@ import { ViewChild, ViewContainerRef, } from "@angular/core"; -import { ActivatedRoute, Router } from "@angular/router"; +import { ActivatedRoute, Router, RouterLink } from "@angular/router"; import { UserService } from "../../common/service/user/user.service"; import { WorkflowPersistService } from "../../common/service/workflow-persist/workflow-persist.service"; import { Workflow } from "../../common/type/workflow"; @@ -85,6 +85,7 @@ export const SAVE_DEBOUNCE_TIME_IN_MS = 5000; AgentPanelComponent, PropertyEditorComponent, FormlyRepeatDndComponent, + RouterLink, ], }) export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { @@ -295,15 +296,12 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { this.macroEditName = detail.name; this.parentWorkflowId = this.route.snapshot.params.id ?? ""; const macroWorkflow = this.macroService.macroDetailToWorkflow(detail); - // Clear the canvas before reloading. Angular reuses WorkspaceComponent - // across route changes (no ngOnDestroy fires when going from - // `/workflow/:id` to `/workflow/:id/macro/:macroId`), so the parent - // workflow's operators+links are still on the JointJS paper — - // reloadWorkflow would otherwise hit "duplicate link" rejections in - // shared-model-change-handler. - this.workflowActionService.resetAsNewWorkflow(); - // Reuse the same shared-model setup as the parent workflow editor so - // the YJS room / undo-redo stack are isolated to this macro. + // Joins the macro's YJS room (which itself destroys any previous + // shared-model first). Then reloadWorkflow clears the canvas of any + // prior operators+links before placing the macro body — same pattern + // as `loadWorkflowWithId`, so SPA navigation between workflow and + // macro routes stays clean and the parent's websocket/execution + // context survives the in-app navigation. this.workflowActionService.setNewSharedModel(macroId, this.userService.getCurrentUser()); this.workflowActionService.reloadWorkflow(macroWorkflow); // Allow visual editing on the canvas, but persistWorkflow is already diff --git a/frontend/src/app/workspace/service/execute-workflow/execute-workflow.service.ts b/frontend/src/app/workspace/service/execute-workflow/execute-workflow.service.ts index d3d7d23d179..753caa87ad1 100644 --- a/frontend/src/app/workspace/service/execute-workflow/execute-workflow.service.ts +++ b/frontend/src/app/workspace/service/execute-workflow/execute-workflow.service.ts @@ -48,6 +48,7 @@ import { intersection } from "../../../common/util/set"; import { WorkflowSettings } from "../../../common/type/workflow"; import { ComputingUnitStatusService } from "../../../common/service/computing-unit/computing-unit-status/computing-unit-status.service"; +import { MacroService } from "../macro/macro.service"; // TODO: change this declaration export const FORM_DEBOUNCE_TIME_MS = 150; @@ -100,7 +101,8 @@ export class ExecuteWorkflowService { private workflowStatusService: WorkflowStatusService, private notificationService: NotificationService, @Inject(DOCUMENT) private document: Document, - private computingUnitStatusService: ComputingUnitStatusService + private computingUnitStatusService: ComputingUnitStatusService, + private macroService: MacroService ) { workflowWebsocketService.websocketEvent().subscribe(event => { switch (event.type) { @@ -202,10 +204,11 @@ export class ExecuteWorkflowService { emailNotificationEnabled: boolean, targetOperatorId?: string ): void { - const logicalPlan = ExecuteWorkflowService.getLogicalPlanRequest( + const rawPlan = ExecuteWorkflowService.getLogicalPlanRequest( this.workflowActionService.getTexeraGraph(), targetOperatorId ); + const logicalPlan = this.expandMacroIdsInPlan(rawPlan); const settings = this.workflowActionService.getWorkflowSettings(); this.resetExecutionState(); this.workflowStatusService.resetStatus(); @@ -217,7 +220,8 @@ export class ExecuteWorkflowService { } public executeWorkflowWithReplay(replayExecutionInfo: ReplayExecutionInfo): void { - const logicalPlan = ExecuteWorkflowService.getLogicalPlanRequest(this.workflowActionService.getTexeraGraph()); + const rawPlan = ExecuteWorkflowService.getLogicalPlanRequest(this.workflowActionService.getTexeraGraph()); + const logicalPlan = this.expandMacroIdsInPlan(rawPlan); const settings = this.workflowActionService.getWorkflowSettings(); this.resetExecutionState(); this.workflowStatusService.resetStatus(); @@ -230,6 +234,63 @@ export class ExecuteWorkflowService { ); } + /** + * Rewrite `opsToViewResult` / `opsToReuseResult` so macro IDs are replaced + * with the post-expansion inner-op IDs the engine will actually see. After + * `MacroExpander.expand` runs on the backend, the macro op is gone — only + * its inner ops (prefixed with `${macroInstanceId}--`) exist. So if the + * user marked a macro for "view result", we forward that mark to the inner + * ops that produce the macro's external output(s). Non-macro IDs pass + * through unchanged. + */ + private expandMacroIdsInPlan(plan: LogicalPlan): LogicalPlan { + const graph = this.workflowActionService.getTexeraGraph(); + const expand = (opIds: readonly string[] | undefined): string[] => { + if (!opIds || opIds.length === 0) return []; + const out: string[] = []; + for (const opId of opIds) { + const op = (() => { + try { + return graph.getOperator(opId); + } catch { + return undefined; + } + })(); + if (op?.operatorType !== "Macro") { + out.push(opId); + continue; + } + const macroId = op.operatorProperties?.["macroId"]; + if (typeof macroId !== "string" || macroId.length === 0) { + // No macroId — leave as-is; backend will error on the unknown op. + out.push(opId); + continue; + } + const bindings = this.macroService.getBindingsForInstance(opId, macroId); + if (!bindings || bindings.outputBindings.length === 0) { + // Bindings not cached yet (or macro has no outputs). Leave the + // macro id so the request is well-formed; user can re-trigger after + // bindings load (preload kicks off on add, so this is rare). + out.push(opId); + continue; + } + // Mark every boundary output producer — when there are multiple + // output ports, the user clicking "view result" on the macro means + // "make all of the macro's outputs inspectable". + for (const binding of bindings.outputBindings) { + out.push(binding.innerOpId); + } + } + // Deduplicate (fan-out / overlapping bindings can repeat IDs). + return Array.from(new Set(out)); + }; + return { + ...plan, + opsToViewResult: expand(plan.opsToViewResult), + opsToReuseResult: expand(plan.opsToReuseResult), + }; + } + public sendExecutionRequest( executionName: string, logicalPlan: LogicalPlan, diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index e03d7720c8e..9ef95a5e356 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -133,11 +133,28 @@ export class MacroService { // share the same HTTP fetch. The body of a macro definition is immutable // for the lifetime of a given (macroId, vid) tuple, so caching by macroId // alone is safe — definition edits go through a new wid in the v1 LIVE mode. - private bodyBindingsCache = new Map>(); + // The cached shape also carries `nestedMacros: Map` + // so recursive resolution (for nested macros) can follow the chain without + // re-parsing the body. + private bodyBindingsCache = new Map< + string, + Observable<{ + inputBindings: MacroPortBinding[]; + outputBindings: MacroPortBinding[]; + nestedMacros: Map; + }> + >(); // Latest-known synchronous snapshot — populated by `getBindingsForInstance` // after the first successful fetch so synchronous stat-update handlers can // look up bindings without re-triggering the network call. - private bodyBindingsSnapshot = new Map(); + private bodyBindingsSnapshot = new Map< + string, + { + inputBindings: MacroPortBinding[]; + outputBindings: MacroPortBinding[]; + nestedMacros: Map; + } + >(); public createMacro(req: MacroCreateRequest): Observable { return this.http.post(`${AppSettings.getApiEndpoint()}/${MACRO_CREATE_URL}`, req); @@ -165,21 +182,37 @@ export class MacroService { * * Cached and shared across subscribers. */ - public getBodyBindings( - macroId: string - ): Observable<{ inputBindings: MacroPortBinding[]; outputBindings: MacroPortBinding[] }> { + public getBodyBindings(macroId: string): Observable<{ + inputBindings: MacroPortBinding[]; + outputBindings: MacroPortBinding[]; + nestedMacros: Map; + }> { const cached = this.bodyBindingsCache.get(macroId); if (cached) return cached; const widNum = Number(macroId); if (!Number.isFinite(widNum)) { - const empty = { inputBindings: [], outputBindings: [] }; + const empty = { inputBindings: [], outputBindings: [], nestedMacros: new Map() }; this.bodyBindingsSnapshot.set(macroId, empty); return of(empty); } const fetched = this.getMacro(widNum).pipe( map(detail => this.computeBodyBindings(detail)), - tap(bindings => this.bodyBindingsSnapshot.set(macroId, bindings)), - catchError(() => of({ inputBindings: [] as MacroPortBinding[], outputBindings: [] as MacroPortBinding[] })), + tap(bindings => { + this.bodyBindingsSnapshot.set(macroId, bindings); + // Eagerly recurse: fetch bindings for any nested macro definitions + // we discovered, so the synchronous resolution path in + // `getBindingsForInstance` finds everything in the snapshot cache. + for (const nestedMacroId of bindings.nestedMacros.values()) { + this.getBodyBindings(nestedMacroId).subscribe({ error: () => undefined }); + } + }), + catchError(() => + of({ + inputBindings: [] as MacroPortBinding[], + outputBindings: [] as MacroPortBinding[], + nestedMacros: new Map(), + }) + ), shareReplay(1) ); this.bodyBindingsCache.set(macroId, fetched); @@ -194,6 +227,12 @@ export class MacroService { * the engine reports stats keyed by the prefixed strings — so we apply the * same rewrite here so callers can do straight-up `stats[innerOpId]` lookups. * + * Recursive: when a binding's `innerOpId` points to a nested macro, follow + * its body bindings (recursively, prefixed at each layer) until we reach a + * terminal non-macro inner op. A fan-out at an input port can produce + * multiple terminal bindings for one external port — those get summed by + * the stats consumer. + * * Returns the cached snapshot synchronously when available so stats-update * handlers don't have to await; preload via `prefetchBindingsForOperators` * to make sure the snapshot is populated by the time execution starts. @@ -205,18 +244,86 @@ export class MacroService { this.getBodyBindings(macroId).subscribe({ error: () => undefined }); return undefined; } - return { - inputBindings: snapshot.inputBindings.map(b => ({ - externalPortIndex: b.externalPortIndex, - innerOpId: `${macroInstanceId}--${b.innerOpId}`, - innerPortIndex: b.innerPortIndex, - })), - outputBindings: snapshot.outputBindings.map(b => ({ - externalPortIndex: b.externalPortIndex, - innerOpId: `${macroInstanceId}--${b.innerOpId}`, - innerPortIndex: b.innerPortIndex, - })), - }; + const inputBindings: MacroPortBinding[] = []; + for (const b of snapshot.inputBindings) { + inputBindings.push(...this.resolveBinding(macroInstanceId, snapshot, b, /* isInput */ true)); + } + const outputBindings: MacroPortBinding[] = []; + for (const b of snapshot.outputBindings) { + outputBindings.push(...this.resolveBinding(macroInstanceId, snapshot, b, /* isInput */ false)); + } + return { inputBindings, outputBindings }; + } + + /** + * Walk a single body-relative binding down through any nested macros until + * we hit a terminal non-macro inner op. At each level we prefix the inner + * op ID with the accumulated instance prefix (so the final ID matches the + * engine's `${outerInstanceId}--${nestedInstanceId}--…--${terminalOp}` + * key). + * + * `externalPortIndex` is preserved through the chain — it identifies the + * MACRO'S external port we started from, not the nested macro's port. + * That's correct: every terminal binding still belongs to the same outer + * macro port. + */ + private resolveBinding( + accumulatedPrefix: string, + snapshot: { + inputBindings: MacroPortBinding[]; + outputBindings: MacroPortBinding[]; + nestedMacros: Map; + }, + binding: MacroPortBinding, + isInput: boolean + ): MacroPortBinding[] { + const nestedMacroId = snapshot.nestedMacros.get(binding.innerOpId); + if (!nestedMacroId) { + // Terminal — return the binding with the full accumulated prefix. + return [ + { + externalPortIndex: binding.externalPortIndex, + innerOpId: `${accumulatedPrefix}--${binding.innerOpId}`, + innerPortIndex: binding.innerPortIndex, + }, + ]; + } + // Nested macro: load its bindings and follow the chain. The nested + // macro's runtime instance ID is `${accumulatedPrefix}--${nestedInstanceId}` + // (where nestedInstanceId is the body-relative ID we'd otherwise return). + const nestedSnapshot = this.bodyBindingsSnapshot.get(nestedMacroId); + if (!nestedSnapshot) { + // Not yet cached — kick off fetch and return what we have so far. The + // outer caller will see the partial resolution; once the nested macro's + // body loads, the next stats emission will re-resolve correctly. + this.getBodyBindings(nestedMacroId).subscribe({ error: () => undefined }); + return [ + { + externalPortIndex: binding.externalPortIndex, + innerOpId: `${accumulatedPrefix}--${binding.innerOpId}`, + innerPortIndex: binding.innerPortIndex, + }, + ]; + } + // Find nested bindings matching the macro's port the outer binding + // points to (binding.innerPortIndex is the nested macro's external port). + const nestedBindings = isInput ? nestedSnapshot.inputBindings : nestedSnapshot.outputBindings; + const nextLayerPrefix = `${accumulatedPrefix}--${binding.innerOpId}`; + const matched = nestedBindings.filter(nb => nb.externalPortIndex === binding.innerPortIndex); + if (matched.length === 0) { + // Shouldn't happen for a well-formed body, but stay defensive. + return []; + } + const resolved: MacroPortBinding[] = []; + for (const nb of matched) { + const carriedOver: MacroPortBinding = { + externalPortIndex: binding.externalPortIndex, // preserve outer macro's external port + innerOpId: nb.innerOpId, // body-relative inside the nested macro + innerPortIndex: nb.innerPortIndex, + }; + resolved.push(...this.resolveBinding(nextLayerPrefix, nestedSnapshot, carriedOver, isInput)); + } + return resolved; } /** @@ -235,14 +342,16 @@ export class MacroService { if (typeof macroId !== "string" || macroId.length === 0) continue; const instanceId = op.operatorID; this.getBodyBindings(macroId).subscribe({ - next: bindings => { - const out0 = bindings.outputBindings.find(b => b.externalPortIndex === 0); - if (out0) { - this.workflowResultService.setMacroResultAlias( - instanceId, - `${instanceId}--${out0.innerOpId}` - ); - } + next: () => { + // After the first-level bindings load, ask for the recursive + // resolved bindings — `getBindingsForInstance` chains through any + // nested macros automatically. Output port 0 might resolve to a + // single terminal inner op, OR (in the rare fan-out case) several; + // for the v1 macro-result alias we still pick the first terminal. + const resolved = this.getBindingsForInstance(instanceId, macroId); + if (!resolved) return; + const out0 = resolved.outputBindings.find(b => b.externalPortIndex === 0); + if (out0) this.workflowResultService.setMacroResultAlias(instanceId, out0.innerOpId); }, error: () => undefined, }); @@ -252,20 +361,35 @@ export class MacroService { private computeBodyBindings(detail: MacroDetail): { inputBindings: MacroPortBinding[]; outputBindings: MacroPortBinding[]; + nestedMacros: Map; } { let body: MacroBody; try { body = JSON.parse(detail.content) as MacroBody; } catch { - return { inputBindings: [], outputBindings: [] }; + return { inputBindings: [], outputBindings: [], nestedMacros: new Map() }; } const inputMarkerByPortIndex = new Map(); const outputMarkerByPortIndex = new Map(); + // Collect nested macro definitions: any Macro op inside the body whose + // macroId we'll need to recursively resolve through. Keyed by the body- + // relative operatorID since that's how the markers' links reference it. + const nestedMacros = new Map(); for (const raw of body.operators) { - const op = raw as { operatorID?: string; operatorType?: string; portIndex?: number }; - if (typeof op.operatorID !== "string" || typeof op.portIndex !== "number") continue; - if (op.operatorType === "MacroInput") inputMarkerByPortIndex.set(op.portIndex, op.operatorID); - else if (op.operatorType === "MacroOutput") outputMarkerByPortIndex.set(op.portIndex, op.operatorID); + const op = raw as { + operatorID?: string; + operatorType?: string; + portIndex?: number; + macroId?: string; + }; + if (typeof op.operatorID !== "string") continue; + if (op.operatorType === "MacroInput" && typeof op.portIndex === "number") { + inputMarkerByPortIndex.set(op.portIndex, op.operatorID); + } else if (op.operatorType === "MacroOutput" && typeof op.portIndex === "number") { + outputMarkerByPortIndex.set(op.portIndex, op.operatorID); + } else if (op.operatorType === "Macro" && typeof op.macroId === "string" && op.macroId.length > 0) { + nestedMacros.set(op.operatorID, op.macroId); + } } const markerIds = new Set([ ...Array.from(inputMarkerByPortIndex.values()), @@ -300,7 +424,7 @@ export class MacroService { }); } } - return { inputBindings, outputBindings }; + return { inputBindings, outputBindings, nestedMacros }; } /** From 4aa7b191de5b53c36446723c72ebef1da16514fd Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 00:02:52 -0700 Subject: [PATCH 23/65] fix(macro): drill-down navigation + duplicate-link tolerance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Revert SPA navigation back to hard reload for macro drill-down click-through. SPA-into-WorkspaceComponent-reuse hits a flurry of duplicate-link rejections from interleaved YJS server-replay + local reloadWorkflow that can't be resolved cleanly with the current shared-model lifecycle. Hard reload gives a clean WorkspaceComponent mount with a fresh canvas every time. * Stash (parentWid, instanceId) into sessionStorage before the hard navigation so the new page can later opt to reconnect to the parent's execution context for live drill-down stats. Wiring the rehydration is a follow-up; the stash itself is harmless if unused. * Use an anonymous YJS room for the drill-down view. Joining the macro definition's wid-keyed room replays accumulated historical operators the room ever held, fighting reloadWorkflow over the same logical data and producing duplicate-link cascades that destroyed the canvas on every navigation. Anonymous room = clean canvas; collaborative editing of macros via drill-down is deferred until we can do a proper YJS state reset on the server side. * SharedModelChangeHandler.validateAndRepairNewLink: when a link is duplicated, *skip rendering* it instead of deleting it from the shared model. The pre-fix behavior was eagerly destructive — the canonical link in the shared model got wiped along with the duplicate, leaving the canvas with nothing to render. Truly invalid links (non-existent op/port) still get repaired out of the model. Co-Authored-By: Claude Sonnet 4.6 --- .../workflow-editor.component.ts | 36 +++++++++++-------- .../component/workspace.component.ts | 16 +++++---- .../model/shared-model-change-handler.ts | 20 ++++++++--- 3 files changed, 46 insertions(+), 26 deletions(-) diff --git a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts index b03ddbf95cd..548b2087e69 100644 --- a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts @@ -700,25 +700,31 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy if (this.workflowActionService.getTexeraGraph().hasCommentBox(elementID)) { this.openCommentBox(elementID); } else if (this.workflowActionService.getTexeraGraph().hasOperator(elementID)) { - // Macro nodes drill down into their body via an in-app route - // change. Use Angular router (not window.location.href) so the - // execution websocket survives the navigation — full reload - // would tear down the parent's `WorkflowWebsocketService` and - // the drill-down view would see no live stats. `loadMacroWithId` - // is responsible for fully resetting the canvas/YJS room before - // reloading the body so we don't hit duplicate-link errors. + // Macro nodes drill down into their body via a route change. We + // use `window.location.href` (hard reload) instead of + // `Router.navigate` because Angular reuses WorkspaceComponent + // across the workflow→macro route transition: SPA navigation + // hits a flurry of duplicate-link rejections from interleaved + // YJS server-side replay + local `reloadWorkflow`. The cost is + // losing the parent's execution websocket connection — the + // drill-down view stashes (parentWid, executionId) into + // sessionStorage so the new page can reconnect to the parent's + // execution context for live stats. See `WorkspaceComponent` + // `ngOnInit` for the rehydration logic. const op = this.workflowActionService.getTexeraGraph().getOperator(elementID); const macroId = op?.operatorProperties?.["macroId"]; if (op?.operatorType === "Macro" && macroId) { const parentWid = this.route.snapshot.params.id ?? ""; - this.router.navigate(["/dashboard/user/workflow", parentWid, "macro", macroId], { - // Carry the macro instance ID so the drill-down view can map - // live execution stats from the parent — engine reports - // inner-op stats under `${instanceId}--${innerOpId}` keys, - // and the drill-down canvas only has the un-prefixed - // inner-op IDs. - queryParams: { instance: elementID }, - }); + try { + sessionStorage.setItem( + "macroDrilldownParentContext", + JSON.stringify({ parentWid, instanceId: elementID, ts: Date.now() }) + ); + } catch { + // sessionStorage can throw in private-mode; that's fine, we + // just won't have drill-down live stats on this navigation. + } + window.location.href = `/dashboard/user/workflow/${parentWid}/macro/${macroId}?instance=${encodeURIComponent(elementID)}`; return; } this.workflowActionService.openResultPanel(); diff --git a/frontend/src/app/workspace/component/workspace.component.ts b/frontend/src/app/workspace/component/workspace.component.ts index 7edab81ceef..928839c36e7 100644 --- a/frontend/src/app/workspace/component/workspace.component.ts +++ b/frontend/src/app/workspace/component/workspace.component.ts @@ -296,13 +296,15 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { this.macroEditName = detail.name; this.parentWorkflowId = this.route.snapshot.params.id ?? ""; const macroWorkflow = this.macroService.macroDetailToWorkflow(detail); - // Joins the macro's YJS room (which itself destroys any previous - // shared-model first). Then reloadWorkflow clears the canvas of any - // prior operators+links before placing the macro body — same pattern - // as `loadWorkflowWithId`, so SPA navigation between workflow and - // macro routes stays clean and the parent's websocket/execution - // context survives the in-app navigation. - this.workflowActionService.setNewSharedModel(macroId, this.userService.getCurrentUser()); + // Joining the macro definition's YJS room replays every historical + // operator/link the room ever held, dueling with `reloadWorkflow`'s + // own pushes and triggering cascading duplicate-link rejections + // that wipe the canvas. Use an anonymous (uuid-suffix) shared model + // so the drill-down starts clean. Collaboration on the body via + // this view is therefore *local-only* for now; persistent + // collaborative editing of macros is deferred. + this.workflowActionService.resetAsNewWorkflow(); + this.workflowActionService.setNewSharedModel(undefined, this.userService.getCurrentUser()); this.workflowActionService.reloadWorkflow(macroWorkflow); // Allow visual editing on the canvas, but persistWorkflow is already // disabled above so changes won't accidentally land on /workflow/persist. diff --git a/frontend/src/app/workspace/service/workflow-graph/model/shared-model-change-handler.ts b/frontend/src/app/workspace/service/workflow-graph/model/shared-model-change-handler.ts index e5eab7a812e..2739cc67feb 100644 --- a/frontend/src/app/workspace/service/workflow-graph/model/shared-model-change-handler.ts +++ b/frontend/src/app/workspace/service/workflow-graph/model/shared-model-change-handler.ts @@ -216,16 +216,28 @@ export class SharedModelChangeHandler { * this link is already deleted from the shared model. */ private validateAndRepairNewLink(newLink: OperatorLink): boolean { + // Duplicates are routinely transient — e.g. when SPA-navigating into a + // workflow whose YJS room sync arrives shortly after a `reloadWorkflow` + // has already populated the same operators+links from the HTTP detail + // fetch. The existing link is the canonical one; just skip rendering the + // duplicate and leave the shared model alone. Pre-fix we *deleted* the + // duplicate from the shared model, which corrupted the canvas on + // drill-down navigation (every link disappeared along with its operators). try { this.texeraGraph.assertLinkNotDuplicated(newLink); - // Verify the link connects to operators and ports that exist. + } catch (error) { + console.log("skipping duplicate link: ", (error as Error).message); + return false; + } + // Validity check is a different failure mode (link references a + // non-existent op/port). Those entries are truly broken and the right + // thing is still to repair them out of the shared model. + try { this.texeraGraph.assertLinkIsValid(newLink); return true; } catch (error) { - // Invalid link, repair the shared model this.texeraGraph.sharedModel.operatorLinkMap.delete(newLink.linkID); - // This is treated as a normal repair step and not an error. - console.log("failed to add link. cause: ", (error as Error).message); + console.log("failed to add link, repaired: ", (error as Error).message); return false; } } From 484cd319204ba95f61b628ad407cb31ddcab0e5e Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 00:12:47 -0700 Subject: [PATCH 24/65] =?UTF-8?q?feat(macro):=20add=20'Expand=20macro'=20c?= =?UTF-8?q?ontext-menu=20action=20=E2=80=94=20inverse=20of=20create?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Right-click a Macro instance → 'Expand macro' inlines its body back onto the parent canvas: deep-clones the body operators with fresh IDs (so re-using the same macro elsewhere doesn't collide), reproduces internal links, rewires every external link that was touching the macro to the matching boundary inner op + port via the body's MacroInput/MacroOutput markers, and finally deletes the macro op. Wrapped in bundleActions so undo collapses to a single step. v1 supports LIVE-linked macros only (body fetched from DbMacroRegistry). SNAPSHOT mode (embedded body in operatorProperties.snapshot) is a follow-up — same logic, different source. Layout is crude (a 3-column grid anchored at the macro's old position); a proper auto-layout pass is deferred. Co-Authored-By: Claude Sonnet 4.6 --- .../context-menu/context-menu.component.html | 10 + .../context-menu/context-menu.component.ts | 196 ++++++++++++++++++ 2 files changed, 206 insertions(+) diff --git a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html index 46fa33a5500..26b94121912 100644 --- a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html +++ b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html @@ -134,6 +134,16 @@ nzTheme="outline">    create macro
    • +
  • + expand macro +
  • { + try { + return this.workflowActionService.getTexeraGraph().getOperator(opId); + } catch { + return undefined; + } + })(); + return op?.operatorType === "Macro" && typeof op.operatorProperties?.["macroId"] === "string"; + } + + public onExpandMacro(): void { + const opId = this.highlightedOperatorIds[0]; + if (!opId) return; + const graph = this.workflowActionService.getTexeraGraph(); + const macroOp = (() => { + try { + return graph.getOperator(opId); + } catch { + return undefined; + } + })(); + if (!macroOp) return; + const macroId = macroOp.operatorProperties?.["macroId"]; + if (typeof macroId !== "string" || macroId.length === 0) { + this.notificationService.error("Macro has no macroId — can't expand."); + return; + } + const widNum = Number(macroId); + if (!Number.isFinite(widNum)) { + this.notificationService.error(`Invalid macroId: ${macroId}`); + return; + } + this.macroService + .getMacro(widNum) + .pipe(untilDestroyed(this)) + .subscribe({ + next: detail => { + try { + this.inlineMacroBody(macroOp, detail); + this.notificationService.success(`Expanded "${detail.name}" onto the canvas.`); + } catch (e) { + this.notificationService.error(`Expand failed: ${(e as Error)?.message ?? e}`); + } + }, + error: err => this.notificationService.error(`Failed to load macro body: ${err?.message ?? err}`), + }); + } + + /** + * Inline the macro's body operators + links onto the parent canvas, rewire + * external links so each one targets the right boundary inner op + port, + * and remove the macro op + its outer links. New unique IDs are assigned + * to body operators so re-expanding the same macro elsewhere doesn't + * collide. + * + * Layout: body ops are laid out around the macro op's former position. + * Crude column layout (input markers → inner → output markers) gets the + * job done without a real layout pass. + */ + private inlineMacroBody(macroOp: OperatorPredicate, detail: MacroDetail): void { + const graph = this.workflowActionService.getTexeraGraph(); + // Parse the body via the existing macroDetailToWorkflow normalizer so we + // get OperatorPredicate-shaped ops and OperatorLink-shaped links. + const macroWorkflow = this.macroService.macroDetailToWorkflow(detail); + const bodyOps = macroWorkflow.content.operators.filter( + o => o.operatorType !== "MacroInput" && o.operatorType !== "MacroOutput" + ); + const inputMarkers = macroWorkflow.content.operators.filter(o => o.operatorType === "MacroInput"); + const outputMarkers = macroWorkflow.content.operators.filter(o => o.operatorType === "MacroOutput"); + const markerIds = new Set([...inputMarkers, ...outputMarkers].map(o => o.operatorID)); + + // Assign fresh IDs to inner ops so re-using the same macro elsewhere + // doesn't collide. Map body-relative ID → fresh canvas ID. + const idRewrite = new Map(); + bodyOps.forEach(op => { + const fresh = `${op.operatorType}-operator-${this.workflowUtilService.getOperatorRandomUUID()}`; + idRewrite.set(op.operatorID, fresh); + }); + + // Anchor positions around the macro's old location (crude column layout). + const macroPos = this.workflowActionService.getJointGraphWrapper().getElementPosition(macroOp.operatorID); + const baseX = macroPos.x; + const baseY = macroPos.y; + const colSpacing = 180; + const rowSpacing = 120; + + const positionedOps: { op: OperatorPredicate; pos: Point }[] = bodyOps.map((op, idx) => ({ + op: { ...op, operatorID: idRewrite.get(op.operatorID)! }, + pos: { x: baseX + (idx % 3) * colSpacing, y: baseY + Math.floor(idx / 3) * rowSpacing }, + })); + + // Internal links (not touching marker ops). Rewrite both endpoints. + const internalLinks = macroWorkflow.content.links + .filter(l => !markerIds.has(l.source.operatorID) && !markerIds.has(l.target.operatorID)) + .map(l => ({ + linkID: this.workflowUtilService.getLinkRandomUUID(), + source: { operatorID: idRewrite.get(l.source.operatorID)!, portID: l.source.portID }, + target: { operatorID: idRewrite.get(l.target.operatorID)!, portID: l.target.portID }, + })); + + // Body links from MacroInput markers to inner ops give us (portIndex → + // [(innerOpId, innerPortID)]) — the same lookup table MacroExpander uses + // on the backend. We need it here to rewire each external incoming link + // (which currently terminates at `macroOp@port_X`) to the corresponding + // inner op port. + const inputBindings = new Map(); + for (const m of inputMarkers) { + const portIndex = m.operatorProperties?.["portIndex"]; + if (typeof portIndex !== "number") continue; + const consumers = macroWorkflow.content.links + .filter(l => l.source.operatorID === m.operatorID && !markerIds.has(l.target.operatorID)) + .map(l => ({ + innerOpId: idRewrite.get(l.target.operatorID)!, + innerPortID: l.target.portID, + })); + inputBindings.set(portIndex, consumers); + } + const outputBindings = new Map(); + for (const m of outputMarkers) { + const portIndex = m.operatorProperties?.["portIndex"]; + if (typeof portIndex !== "number") continue; + const producer = macroWorkflow.content.links.find( + l => l.target.operatorID === m.operatorID && !markerIds.has(l.source.operatorID) + ); + if (producer) { + outputBindings.set(portIndex, { + innerOpId: idRewrite.get(producer.source.operatorID)!, + innerPortID: producer.source.portID, + }); + } + } + + // Find the parent canvas links that touch the macro and need rewiring. + // Frontend port IDs are `input-i` / `output-j`; the trailing integer is + // the external port index we map against. + const portIdToIndex = (portID: string): number | undefined => { + const m = portID.match(/(\d+)$/); + return m ? Number(m[1]) : undefined; + }; + const incomingRewires: { source: { operatorID: string; portID: string }; targets: { operatorID: string; portID: string }[] }[] = []; + const outgoingRewires: { source: { operatorID: string; portID: string }; target: { operatorID: string; portID: string } }[] = []; + for (const link of graph.getAllLinks()) { + if (link.target.operatorID === macroOp.operatorID) { + const portIndex = portIdToIndex(link.target.portID); + if (portIndex === undefined) continue; + const consumers = inputBindings.get(portIndex) ?? []; + incomingRewires.push({ source: link.source, targets: consumers.map(c => ({ operatorID: c.innerOpId, portID: c.innerPortID })) }); + } else if (link.source.operatorID === macroOp.operatorID) { + const portIndex = portIdToIndex(link.source.portID); + if (portIndex === undefined) continue; + const producer = outputBindings.get(portIndex); + if (producer) { + outgoingRewires.push({ source: { operatorID: producer.innerOpId, portID: producer.innerPortID }, target: link.target }); + } + } + } + + // Apply all of it atomically so undo collapses to one step. + graph.bundleActions(() => { + this.workflowActionService.addOperatorsAndLinks(positionedOps, internalLinks); + for (const rw of incomingRewires) { + for (const target of rw.targets) { + this.workflowActionService.addLink({ + linkID: this.workflowUtilService.getLinkRandomUUID(), + source: rw.source, + target, + }); + } + } + for (const rw of outgoingRewires) { + this.workflowActionService.addLink({ + linkID: this.workflowUtilService.getLinkRandomUUID(), + source: rw.source, + target: rw.target, + }); + } + this.workflowActionService.deleteOperatorsAndLinks([macroOp.operatorID]); + }); + } + public onCreateMacro(): void { const selected = Array.from(this.workflowActionService.getJointGraphWrapper().getCurrentHighlightedOperatorIDs()); if (selected.length < 2) { From 8c275525ab61ed7539b2cb17f6b04b06877a3664 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 00:39:15 -0700 Subject: [PATCH 25/65] feat(macro): drill-down uses parent wid for execution websocket MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In drill-down view, override the workflow metadata's wid to the parent workflow's wid before reloading. ComputingUnitSelectionComponent reads metadata.wid to decide which workflow id to open the execution websocket against — if it gets the macro definition's wid (278), the drill-down view subscribes to the macro's execution, not the parent's, and sees no stats during the parent's actual run. Spoofing the wid to parent's lets the websocket stay on the parent's execution stream, and the existing ${instanceId}-- prefix machinery in WorkflowEditorComponent maps those keys onto the body-relative op IDs the drill-down canvas displays. Safe because workflow persistence is disabled in drill-down (the macro body is saved through MacroResource, not the regular workflow save endpoint). Co-Authored-By: Claude Sonnet 4.6 --- .../app/workspace/component/workspace.component.ts | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/frontend/src/app/workspace/component/workspace.component.ts b/frontend/src/app/workspace/component/workspace.component.ts index 928839c36e7..a21e27c3f25 100644 --- a/frontend/src/app/workspace/component/workspace.component.ts +++ b/frontend/src/app/workspace/component/workspace.component.ts @@ -295,7 +295,19 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { this.macroEditMode = true; this.macroEditName = detail.name; this.parentWorkflowId = this.route.snapshot.params.id ?? ""; - const macroWorkflow = this.macroService.macroDetailToWorkflow(detail); + // Override the workflow metadata's wid to the parent's wid (not the + // macro definition's). This is what `ComputingUnitSelectionComponent` + // reads when deciding which workflow id to open the execution + // websocket against; we want it on the parent so live execution + // stats from the parent's run still flow into this drilled-down + // view (via the `?instance=...` prefix machinery in + // `WorkflowEditorComponent`). Caveat: persisting workflow changes + // is disabled in drill-down anyway, so the spoofed wid is safe. + const macroWorkflowRaw = this.macroService.macroDetailToWorkflow(detail); + const parentWidNum = Number(this.parentWorkflowId); + const macroWorkflow = Number.isFinite(parentWidNum) + ? { ...macroWorkflowRaw, wid: parentWidNum } + : macroWorkflowRaw; // Joining the macro definition's YJS room replays every historical // operator/link the room ever held, dueling with `reloadWorkflow`'s // own pushes and triggering cascading duplicate-link rejections From d379bcf7f0dbb12ecff0e55240a7d506ce0e61a0 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 00:43:14 -0700 Subject: [PATCH 26/65] feat(macro): Your Macros palette section + click-to-add instantiation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Surfaces the user's saved macros under a "Your Macros" section in the operator palette so they can be reused on other workflows. Loaded once on component init via MacroService.listMacros(). Each macro renders as a clickable row with name + (X in / Y out) port-count chip; clicking builds a fresh OperatorPredicate (Macro-operator-{uuid}, macroId set from the summary's wid, port counts from portSpec) and places it on the canvas — same shape as `swapSelectionWithMacroNode` produces from a selection, so all downstream paths (validation, render, expansion, execution) see a normal Macro op. v1 is click-to-add only; true drag-from-palette would require special- casing the drag-drop service because regular operators go through WorkflowUtilService.getNewOperatorPredicate(type) which can't fill in the macro-specific properties. Visual styling matches the dashed-blue macro treatment on the canvas so palette→canvas reads as one identity. Co-Authored-By: Claude Sonnet 4.6 --- .../operator-menu.component.html | 30 ++++++ .../operator-menu.component.scss | 50 ++++++++++ .../operator-menu/operator-menu.component.ts | 99 ++++++++++++++++++- 3 files changed, 177 insertions(+), 2 deletions(-) diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html index ff8bb97a296..2ddcf74468e 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html @@ -42,6 +42,36 @@ + + + +
    +
    + ▦ + {{ macroSchema.additionalMetadata.userFriendlyName }} + + {{ macroSchema.additionalMetadata.inputPorts.length }} in / + {{ macroSchema.additionalMetadata.outputPorts.length }} out + +
    +
    +     +     + >(); public groupNames: ReadonlyArray = []; + // The user's saved macros — surfaced as a "Your Macros" section in the + // palette so they can be reused on other workflows by clicking the entry. + // We use the existing operator-label rendering by exposing each macro as + // an OperatorSchema-shaped object whose operatorType is the literal + // "Macro" and whose userFriendlyName is the macro name. The drag/click + // handler peeks at `__macroSummary` on the schema to fill in macroId, + // inputPortCount, outputPortCount when instantiating the operator + // predicate. + public macroList: (OperatorSchema & { __macroSummary?: MacroSummary })[] = []; + // input value of the search input box public searchInputValue: string = ""; // search autocomplete suggestion list @@ -81,8 +94,16 @@ export class OperatorMenuComponent { private operatorMetadataService: OperatorMetadataService, private workflowActionService: WorkflowActionService, private workflowUtilService: WorkflowUtilService, - private dragDropService: DragDropService + private dragDropService: DragDropService, + private macroService: MacroService ) { + // Load the user's saved macros for the "Your Macros" palette section. + this.macroService.listMacros().subscribe({ + next: (summaries: MacroSummary[]) => { + this.macroList = summaries.map(m => this.macroSummaryToSchema(m)); + }, + error: () => undefined, + }); // clear the search box if an operator is dropped from operator search box this.dragDropService.operatorDropStream.pipe(untilDestroyed(this)).subscribe(() => { this.searchInputValue = ""; @@ -150,4 +171,78 @@ export class OperatorMenuComponent { this.autocompleteOptions = []; }, 0); } + + /** + * Adapt a backend `MacroSummary` into an `OperatorSchema`-shaped row the + * existing operator-label component can render. The macro's port count + * and definition wid are stashed on `__macroSummary` so click-to-add can + * build the right `OperatorPredicate` without re-fetching the macro. + */ + private macroSummaryToSchema(m: MacroSummary): OperatorSchema & { __macroSummary: MacroSummary } { + return { + operatorType: "Macro", + jsonSchema: { type: "object", properties: {} } as unknown as OperatorSchema["jsonSchema"], + additionalMetadata: { + userFriendlyName: m.name, + operatorDescription: m.description ?? `Macro from workflow #${m.wid}`, + operatorGroupName: "Your Macros", + inputPorts: m.portSpec.inputs.map(p => ({ displayName: `in-${p.index}` })), + outputPorts: m.portSpec.outputs.map(p => ({ displayName: `out-${p.index}` })), + dynamicInputPorts: false, + dynamicOutputPorts: false, + supportReconfiguration: false, + allowPortCustomization: false, + } as unknown as OperatorSchema["additionalMetadata"], + operatorVersion: "", + __macroSummary: m, + }; + } + + /** + * Place a saved macro on the canvas. Builds a fresh `OperatorPredicate` + * matching the shape created by `swapSelectionWithMacroNode` so the + * downstream validation/render/execution paths see a normal Macro op. + */ + public onAddMacro(macroSchema: OperatorSchema & { __macroSummary?: MacroSummary }): void { + const m = macroSchema.__macroSummary; + if (!m) return; + const inputPortCount = m.portSpec.inputs.length; + const outputPortCount = m.portSpec.outputs.length; + const inputPorts = Array.from({ length: inputPortCount }, (_, i) => ({ + portID: `input-${i}`, + displayName: `in-${i}`, + disallowMultiInputs: false, + isDynamicPort: false, + dependencies: [], + })); + const outputPorts = Array.from({ length: outputPortCount }, (_, i) => ({ + portID: `output-${i}`, + displayName: `out-${i}`, + disallowMultiInputs: false, + isDynamicPort: false, + })); + const predicate: OperatorPredicate = { + operatorID: `Macro-operator-${this.workflowUtilService.getOperatorRandomUUID()}`, + operatorType: "Macro", + operatorVersion: "", + operatorProperties: { + macroId: String(m.wid), + macroVersion: 1, + linkMode: "LIVE", + inputPortCount, + outputPortCount, + displayName: m.name, + }, + inputPorts, + outputPorts, + showAdvanced: false, + isDisabled: false, + customDisplayName: m.name, + dynamicInputPorts: false, + dynamicOutputPorts: false, + }; + const origin = this.workflowActionService.getJointGraphWrapper().getMainJointPaper()?.translate(); + const point = { x: 400 - (origin?.tx ?? 0), y: 200 - (origin?.ty ?? 0) }; + this.workflowActionService.addOperator(predicate, point); + } } From b35fdc5fa4f525d661fdec20a0a3c73990bb56a3 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 00:48:20 -0700 Subject: [PATCH 27/65] =?UTF-8?q?feat(macro):=20suggestMacros=20agent=20pa?= =?UTF-8?q?nel=20=E2=80=94=20heuristic=20v1=20ranked=20candidates?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a "Suggest Macros (AI)" button + inline panel in the operator palette that surfaces ranked sub-DAG encapsulation candidates without calling out to an LLM. v1 heuristic: maximal linear chains where each interior op has exactly one upstream and one downstream within the chain. Score = chain length × source/sink penalty (≥2 ops, source-anchored chains discounted to 0.5×, sink-anchored to 0.7×). Top 10 returned. Per-candidate rationale is derived from the operator-type sequence ("Looks like a reusable preprocessing block", "Two-step pipeline: Filter → Projection", etc.). UX: button shows brief "Analyzing workflow…" affordance (forced 250ms delay) so the action reads as agent-like rather than instant lookup. Top suggestion's operators get highlighted on the canvas immediately; clicking a candidate row highlights+selects so the user can confirm via right-click → Create Macro. v2 should call ContextMenuComponent's private `swapSelectionWithMacroNode` flow directly. LLM swap is one HTTP call away: replace `suggestMacros()` body with a chat-assistant-service request returning the same `MacroSuggestion[]` shape — UI and downstream materialize-action paths unchanged. Co-Authored-By: Claude Sonnet 4.6 --- .../operator-menu.component.html | 35 +++ .../operator-menu.component.scss | 104 ++++++++ .../operator-menu/operator-menu.component.ts | 68 +++++- .../service/macro/macro-suggestion.service.ts | 231 ++++++++++++++++++ 4 files changed, 437 insertions(+), 1 deletion(-) create mode 100644 frontend/src/app/workspace/service/macro/macro-suggestion.service.ts diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html index 2ddcf74468e..7dab94dc428 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html @@ -40,6 +40,41 @@
    + +
    + +
    +
    + {{ suggestions.length }} candidate{{ suggestions.length === 1 ? '' : 's' }} + +
    +
    +
    {{ suggestion.suggestedName }}
    +
    {{ suggestion.rationale }}
    +
    + {{ suggestion.operatorIds.length }} ops · score {{ suggestion.score.toFixed(1) }} +
    +
    +
    +
    + + drag-drop service, so v1 is click-to-add. Each item has a tiny "⤓" + export button (downloads JSON portable across instances) and the section + header carries an "Import" affordance for the reverse direction. --> +
    + + +
    + - ▦ - {{ macroSchema.additionalMetadata.userFriendlyName }} + ▦ + + {{ macroSchema.additionalMetadata.userFriendlyName }} + {{ macroSchema.__macroSummary!.usageCount }}× used - + {{ macroSchema.additionalMetadata.inputPorts.length }} in / {{ macroSchema.additionalMetadata.outputPorts.length }} out + diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss index a0925412239..a13d18a9775 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss @@ -115,6 +115,49 @@ border-radius: 8px; white-space: nowrap; } + + &__export { + background: transparent; + border: 0; + color: #888; + font-size: 14px; + line-height: 1; + cursor: pointer; + padding: 2px 4px; + border-radius: 3px; + transition: background 0.1s, color 0.1s; + + &:hover { + background: #e8f1ff; + color: #1d6fdb; + } + } +} + +.your-macros-toolbar { + margin: 8px 0 4px; + display: flex; + justify-content: flex-end; + + &__import { + background: transparent; + border: 1px solid #4a90e2; + color: #1d6fdb; + border-radius: 4px; + padding: 3px 10px; + font-size: 12px; + font-weight: 500; + cursor: pointer; + transition: background 0.1s; + + &:hover { + background: #f0f7ff; + } + } + + &__file { + display: none; + } } // "Suggest Macros (AI)" panel — surfaces heuristic candidates. Sits between diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts index adb2e676806..9fb284db248 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts @@ -17,7 +17,7 @@ * under the License. */ -import { Component } from "@angular/core"; +import { Component, ElementRef, ViewChild } from "@angular/core"; import Fuse from "fuse.js"; import { OperatorMetadataService } from "../../../service/operator-metadata/operator-metadata.service"; import { GroupInfo, OperatorSchema } from "../../../types/operator-schema.interface"; @@ -359,4 +359,63 @@ export class OperatorMenuComponent { public dismissSuggestions(): void { this.suggestions = []; } + + /** Reference to the hidden file input — clicked programmatically by `onTriggerImportMacro`. */ + @ViewChild("macroImportFile") macroImportFile?: ElementRef; + + /** + * Trigger a browser download of one macro definition. Exposed off the + * palette item's small "⤓" affordance. The actual HTTP fetch + Blob + * creation lives in `MacroService.exportMacroToFile`. + */ + public onExportMacro(summary: MacroSummary): void { + this.macroService.exportMacroToFile(summary.wid).subscribe({ + next: () => this.message.success(`Exported "${summary.name}".`), + error: err => this.message.error(`Export failed: ${err?.message ?? err}`), + }); + } + + /** + * Open the OS file picker for macro JSON files. The change handler is + * `onImportMacroFile`. Using a hidden file input + button is the standard + * dance for getting a click-styled "Upload" affordance. + */ + public onTriggerImportMacro(): void { + this.macroImportFile?.nativeElement.click(); + } + + /** + * File picker callback — read the JSON, POST it as a fresh macro + * definition, and refresh the "Your Macros" palette so the imported + * macro shows up immediately. + */ + public onImportMacroFile(event: Event): void { + const input = event.target as HTMLInputElement; + const file = input.files?.[0]; + if (!file) return; + const reader = new FileReader(); + reader.onload = () => { + const text = String(reader.result); + try { + this.macroService.importMacroFromJson(text).subscribe({ + next: detail => { + this.message.success(`Imported macro "${detail.name}" (wid=${detail.wid})`); + // Refresh the palette to surface the new macro. + this.macroService.listMacros().subscribe({ + next: (summaries: MacroSummary[]) => { + this.macroList = summaries.map(m => this.macroSummaryToSchema(m)); + }, + }); + }, + error: err => this.message.error(`Import failed: ${err?.message ?? err}`), + }); + } catch (e: any) { + this.message.error(`Import failed: ${e?.message ?? e}`); + } finally { + // Reset so the same file can be re-picked if needed. + input.value = ""; + } + }; + reader.readAsText(file); + } } diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index 9b37f7d5973..c0442b03d53 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -309,6 +309,82 @@ export class MacroService { return this.http.post(`${AppSettings.getApiEndpoint()}/${MACRO_CREATE_URL}`, req); } + /** + * Trigger a browser download of a portable JSON dump of one macro. The file + * is everything `createMacro` accepts as input — name, content, portSpec, + * paramSpec — so it can be re-imported on a different Texera instance via + * `importMacroFromJson`. We deliberately exclude wid and timestamps because + * the importer always creates a fresh definition with a new wid. + * + * The exported `content` is the raw MacroBody JSON string; consumer just + * needs to round-trip it through `JSON.parse(JSON.stringify(...))` to stay + * Jackson-friendly on re-import. + */ + public exportMacroToFile(wid: number): Observable { + return this.getMacro(wid).pipe( + map(detail => { + const exportPayload = { + schemaVersion: 1, + name: detail.name, + description: detail.description, + content: detail.content, + portSpec: detail.portSpec, + paramSpec: detail.paramSpec, + category: detail.category, + icon: detail.icon, + exportedAt: new Date().toISOString(), + exportedFromTexera: window.location.host, + }; + const blob = new Blob([JSON.stringify(exportPayload, null, 2)], { + type: "application/json", + }); + const url = URL.createObjectURL(blob); + const a = document.createElement("a"); + a.href = url; + // Slugify the macro name so the filename is safe across OSes. + const safeName = detail.name.replace(/[^a-zA-Z0-9_-]+/g, "_").slice(0, 60); + a.download = `macro-${safeName}-${detail.wid}.json`; + document.body.appendChild(a); + a.click(); + document.body.removeChild(a); + URL.revokeObjectURL(url); + }) + ); + } + + /** + * Reverse of `exportMacroToFile`: parse an uploaded JSON file and POST it + * as a brand-new macro definition. The new definition's wid is fresh — + * any cross-references inside the original `content` to its own wid are + * left as-is (they'd be self-referential and unused). Returns the created + * MacroDetail so the caller can refresh the palette. + */ + public importMacroFromJson(rawJson: string): Observable { + const parsed = JSON.parse(rawJson) as { + schemaVersion?: number; + name?: string; + description?: string; + content?: string; + portSpec?: PortSpec; + paramSpec?: unknown; + category?: string; + icon?: string; + }; + if (!parsed.name || !parsed.content || !parsed.portSpec) { + throw new Error("Invalid macro JSON: missing name / content / portSpec."); + } + const req: MacroCreateRequest = { + name: `${parsed.name} (imported)`, + description: parsed.description ?? "Imported macro", + content: parsed.content, + portSpec: parsed.portSpec, + paramSpec: parsed.paramSpec, + category: parsed.category, + icon: parsed.icon, + }; + return this.createMacro(req); + } + public listMacros(): Observable { return this.http.get(`${AppSettings.getApiEndpoint()}/${MACRO_LIST_URL}`); } From dba535ae993b2a0dd3ca2915b63a11ab031ef5cb Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:21:53 -0700 Subject: [PATCH 36/65] feat(macro): fuse codegen handles PythonUDFV2, Regex, Limit, Distinct MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - PythonUDFV2 / PythonLambdaFunction: regex-match the user's process_tuple body and inline it directly. The dedent helper strips leading indent so the result re-indents cleanly under our 8-space fused-function body. - Regex: emits `re.search(...)`-guarded `return` to drop non-matches. - Limit: per-actor counter on `self._fuse_limit_seen` — preserves the original Limit operator's "stop after N" semantics inside the UDF. - Distinct: hash-set dedup via `frozenset(tuple_.items())`. - Unhandled types still emit a structural comment marker so the user sees what got skipped; verified flag still gates on translatedCount. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../service/macro/macro-fusion.service.ts | 90 +++++++++++++++++++ 1 file changed, 90 insertions(+) diff --git a/frontend/src/app/workspace/service/macro/macro-fusion.service.ts b/frontend/src/app/workspace/service/macro/macro-fusion.service.ts index 00a147a9f1c..2c5f13e3498 100644 --- a/frontend/src/app/workspace/service/macro/macro-fusion.service.ts +++ b/frontend/src/app/workspace/service/macro/macro-fusion.service.ts @@ -196,10 +196,100 @@ ${stepsCode} translated: true, }; } + if (type === "PythonUDFV2" || type === "PythonLambdaFunction") { + // Inline the user's existing Python body. We can't safely run their + // class-based UDF inside the fused process_tuple (their `self` won't + // exist), so we extract the *body* of their `process_tuple` method via + // a heuristic regex and splice it in. If that fails, fall back to a + // call-style placeholder. The codegen is best-effort — the user can + // edit the fused code in the property panel before running. + const code = (op["code"] as string) ?? ""; + const bodyMatch = code.match( + /def\s+process_tuple\s*\([^)]*\)[^:]*:\s*([\s\S]*?)(?=\n\s*(?:def|@|class)\s|\Z)/ + ); + if (bodyMatch && bodyMatch[1].trim().length > 0) { + // Strip leading indentation so the body lines up cleanly with our 8-space + // process_tuple indent (the outer template adds the leading whitespace). + const dedented = this.dedent(bodyMatch[1]); + return { + code: `${headerComment}\n# (inlined from user's PythonUDFV2)\n${dedented}`, + translated: true, + }; + } + return { + code: `${headerComment}\n# (could not parse user UDF body — passthrough)`, + translated: false, + }; + } + if (type === "Regex") { + const attr = op["attribute"] as string | undefined; + const regex = op["regex"] as string | undefined; + if (!attr || !regex) { + return { code: `${headerComment}\n# (missing attribute/regex — passthrough)`, translated: false }; + } + // Filter-style semantics: drop tuples whose attribute doesn't match. + return { + code: + `${headerComment}\n` + + `import re as _re\n` + + `if not _re.search(${JSON.stringify(regex)}, str(tuple_.get(${JSON.stringify(attr)}, ""))):\n` + + ` return`, + translated: true, + }; + } + if (type === "Limit") { + // Per-tuple counter via a closure-cell on the operator instance. We + // need to declare a state attribute up-top — the outer codegen handles + // that via a separate `# state:` marker that translateOp can emit. + const limit = Number(op["limit"]) || 0; + return { + code: + `${headerComment}\n` + + `if not hasattr(self, "_fuse_limit_seen"):\n` + + ` self._fuse_limit_seen = 0\n` + + `self._fuse_limit_seen += 1\n` + + `if self._fuse_limit_seen > ${limit}:\n` + + ` return`, + translated: true, + }; + } + if (type === "Distinct") { + // Hash the tuple's frozen items into a set; suppress duplicates. + return { + code: + `${headerComment}\n` + + `if not hasattr(self, "_fuse_seen"):\n` + + ` self._fuse_seen = set()\n` + + `_key = frozenset(tuple_.items()) if hasattr(tuple_, "items") else id(tuple_)\n` + + `if _key in self._fuse_seen:\n` + + ` return\n` + + `self._fuse_seen.add(_key)`, + translated: true, + }; + } // Unknown op type: emit a marker comment and leave the tuple untouched. return { code: `${headerComment}\n# (unfusable in v1: ${type})`, translated: false }; } + /** + * Strip leading common indentation from a multi-line string so the result + * can be re-indented by the outer codegen to a consistent depth. Python is + * indentation-sensitive — without this the inlined UDF body would either + * over-indent or trigger SyntaxError on import. + */ + private dedent(text: string): string { + const lines = text.replace(/^\n+/, "").replace(/\n+$/, "").split("\n"); + let minIndent = Infinity; + for (const line of lines) { + if (line.trim().length === 0) continue; + const m = line.match(/^(\s*)/); + const len = m ? m[1].length : 0; + if (len < minIndent) minIndent = len; + } + if (!Number.isFinite(minIndent) || minIndent === 0) return lines.join("\n"); + return lines.map(l => l.slice(minIndent)).join("\n"); + } + /** * Turn one FilterPredicate `{attribute, condition, value}` into a Python * boolean expression evaluating to true iff the tuple passes the filter. From d24203ba522ff7648fb9c68a7f093bc94c0515b4 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:28:15 -0700 Subject: [PATCH 37/65] fix(macro): robust fuse codegen for PythonUDFV2 + Filter symbol conds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replace fragile regex extraction with indent-aware walk in extractPythonMethodBody — handles real-world UDFs (blank lines, decorators, no trailing dedent line) reliably. - Rewrite `yield X` → `tuple_ = X` inside inlined UDF bodies so the outer fused process_tuple's lone yield is the only emission. Avoids the duplicate-output bug when fusing through a UDF. - Surface a NOTE comment when an inlined UDF had multiple yields, so the user knows to hand-edit if their UDF is a multi-emit generator. - Filter predicateToPython now accepts both symbolic (=, !=, >, <=, ...) and enum-style (EQUAL_TO, ...) condition values — macros saved pre/post the rename now both fuse correctly. - Projection codegen handles isDrop=true and applies alias renames in a second pass. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../service/macro/macro-fusion.service.ts | 115 +++++++++++++++--- 1 file changed, 97 insertions(+), 18 deletions(-) diff --git a/frontend/src/app/workspace/service/macro/macro-fusion.service.ts b/frontend/src/app/workspace/service/macro/macro-fusion.service.ts index 2c5f13e3498..169f6031e29 100644 --- a/frontend/src/app/workspace/service/macro/macro-fusion.service.ts +++ b/frontend/src/app/workspace/service/macro/macro-fusion.service.ts @@ -185,40 +185,63 @@ ${stepsCode} } if (type === "Projection") { const attrs = (op["attributes"] as Array<{ originalAttribute?: string; alias?: string }>) ?? []; + // isDrop=true means "exclude these columns"; otherwise "keep only these + // columns". Aliases rename the kept attributes — applied in a second + // pass so the original lookup keys remain valid. + const isDrop = op["isDrop"] === true; if (attrs.length === 0) { return { code: `${headerComment}\n# (no projection columns — passthrough)`, translated: true }; } - const keepKeys = attrs + const targetKeys = attrs .map(a => a.originalAttribute) .filter((k): k is string => typeof k === "string"); + const aliasMap: Record = {}; + attrs.forEach(a => { + if (a.originalAttribute && a.alias && a.alias.length > 0) { + aliasMap[a.originalAttribute] = a.alias; + } + }); + const keysExpr = JSON.stringify(targetKeys); + const aliasExpr = Object.keys(aliasMap).length > 0 ? JSON.stringify(aliasMap) : ""; + const selectExpr = isDrop + ? `tuple_ = {k: tuple_[k] for k in list(tuple_.keys()) if k not in ${keysExpr}}` + : `tuple_ = {k: tuple_[k] for k in ${keysExpr} if k in tuple_}`; + const aliasApply = aliasExpr + ? `\n_aliases = ${aliasExpr}\ntuple_ = {(_aliases.get(k, k)): v for k, v in tuple_.items()}` + : ""; return { - code: `${headerComment}\ntuple_ = {k: tuple_[k] for k in ${JSON.stringify(keepKeys)} if k in tuple_}`, + code: `${headerComment}\n${selectExpr}${aliasApply}`, translated: true, }; } if (type === "PythonUDFV2" || type === "PythonLambdaFunction") { // Inline the user's existing Python body. We can't safely run their // class-based UDF inside the fused process_tuple (their `self` won't - // exist), so we extract the *body* of their `process_tuple` method via - // a heuristic regex and splice it in. If that fails, fall back to a - // call-style placeholder. The codegen is best-effort — the user can - // edit the fused code in the property panel before running. - const code = (op["code"] as string) ?? ""; - const bodyMatch = code.match( - /def\s+process_tuple\s*\([^)]*\)[^:]*:\s*([\s\S]*?)(?=\n\s*(?:def|@|class)\s|\Z)/ - ); - if (bodyMatch && bodyMatch[1].trim().length > 0) { - // Strip leading indentation so the body lines up cleanly with our 8-space - // process_tuple indent (the outer template adds the leading whitespace). - const dedented = this.dedent(bodyMatch[1]); + // exist), so we extract the *body* of their `process_tuple` method + // via an indent-aware walk. + // + // Critical: the inlined body's `yield X` would emit tuples through the + // fused operator, then collide with our outer `yield tuple_` — emitting + // twice. Rewrite `yield X` → `tuple_ = X` so the mutation persists and + // only the outer yield emits. This is correct semantics for one-in / + // one-out UDFs (the common case). Multi-yield generators aren't fully + // translatable in v1 — flagged in the property panel for manual edits. + const rawBody = this.extractPythonMethodBody((op["code"] as string) ?? "", "process_tuple"); + if (rawBody.trim().length === 0) { return { - code: `${headerComment}\n# (inlined from user's PythonUDFV2)\n${dedented}`, - translated: true, + code: `${headerComment}\n# (could not parse user UDF body — passthrough)`, + translated: false, }; } + const yieldCount = (rawBody.match(/^\s*yield\b/gm) || []).length; + const rewritten = rawBody.replace(/^(\s*)yield\s+(.+?)$/gm, "$1tuple_ = $2"); + const multiYieldNote = + yieldCount > 1 + ? "\n# NOTE: original UDF had multiple yields; only the last value propagates after fusion." + : ""; return { - code: `${headerComment}\n# (could not parse user UDF body — passthrough)`, - translated: false, + code: `${headerComment}\n# (inlined from user's PythonUDFV2)${multiYieldNote}\n${rewritten}`, + translated: true, }; } if (type === "Regex") { @@ -290,6 +313,50 @@ ${stepsCode} return lines.map(l => l.slice(minIndent)).join("\n"); } + /** + * Extract the body of a Python method by name. Walks line-by-line: locates + * the `def (...)` header, then takes everything indented strictly + * more than the header until a line with less-or-equal indent (excluding + * blank lines, which preserve formatting inside the body). + * + * Regex-only extraction is fragile across newline / continuation patterns; + * this indent-aware walk handles realistic UDF bodies including blank lines, + * decorators below the body, and methods that close at end-of-file without + * a trailing dedent line. + * + * The returned text is *dedented* — the method body is left-aligned so the + * caller can re-indent it to whatever depth the outer codegen needs. + */ + private extractPythonMethodBody(code: string, methodName: string): string { + const lines = code.split("\n"); + const headerRe = new RegExp(`^(\\s*)def\\s+${methodName}\\b`); + let headerIndent = -1; + let bodyIndent = -1; + const body: string[] = []; + for (let i = 0; i < lines.length; i++) { + const line = lines[i]; + if (headerIndent < 0) { + const m = line.match(headerRe); + if (m) { + headerIndent = m[1].length; + } + continue; + } + // We're past the def header. The first non-blank line establishes + // the body indent. + if (line.trim() === "") { + body.push(""); + continue; + } + const lineIndent = (line.match(/^(\s*)/) || ["", ""])[1].length; + if (bodyIndent < 0) bodyIndent = lineIndent; + // A line at-or-below the header's indent means we've exited the method. + if (lineIndent <= headerIndent) break; + body.push(line); + } + return this.dedent(body.join("\n")); + } + /** * Turn one FilterPredicate `{attribute, condition, value}` into a Python * boolean expression evaluating to true iff the tuple passes the filter. @@ -303,22 +370,34 @@ ${stepsCode} const value = p["value"] as string | undefined; if (!attr || !cond) return ""; const lhs = `tuple_.get(${JSON.stringify(attr)})`; + // Texera stores the condition as either the symbolic short form (=, !=, + // >, >=, <, <=) used in the property panel OR the enum-style long form + // (EQUAL_TO, NOT_EQUAL_TO, ...) depending on the backend version. Cover + // both so the fuse codegen works on older macros too. switch (cond) { + case "=": case "EQUAL_TO": return `${lhs} == ${this.literalToPython(value)}`; + case "!=": case "NOT_EQUAL_TO": return `${lhs} != ${this.literalToPython(value)}`; + case ">": case "GREATER_THAN": return `${lhs} > ${this.literalToPython(value)}`; + case ">=": case "GREATER_THAN_OR_EQUAL_TO": return `${lhs} >= ${this.literalToPython(value)}`; + case "<": case "LESS_THAN": return `${lhs} < ${this.literalToPython(value)}`; + case "<=": case "LESS_THAN_OR_EQUAL_TO": return `${lhs} <= ${this.literalToPython(value)}`; case "IS_NULL": + case "is null": return `${lhs} is None`; case "IS_NOT_NULL": + case "is not null": return `${lhs} is not None`; default: return ""; From 8fb88fc522c723d41fb1f8efb5a2984bbd788406 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:29:54 -0700 Subject: [PATCH 38/65] =?UTF-8?q?feat(macro):=20proactive=20suggestion=20b?= =?UTF-8?q?adge=20=E2=80=94=20agent=20watches=20the=20canvas?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - OperatorMenuComponent now subscribes to graph add/delete/relink streams and re-runs the suggester silently on 700ms debounce. The result surfaces as a count badge on the "Suggest Macros (AI)" button. - The user sees "Suggest Macros (AI) [4]" the moment they open a workflow with detectable patterns — no click required to discover them. Reads as "agent is watching your workflow for refactor opportunities", which is the demo posture for the AI features. - Debounce avoids flicker during multi-op paste / drag-batch placement. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../operator-menu.component.html | 6 +++ .../operator-menu.component.scss | 10 ++++ .../operator-menu/operator-menu.component.ts | 49 +++++++++++++++++++ 3 files changed, 65 insertions(+) diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html index ed9942936a1..97a8c63fd88 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html @@ -50,6 +50,12 @@ ✨ Suggest Macros (AI) Analyzing workflow… + + {{ availableCandidateCount }} +
    diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss index a13d18a9775..717c7e696bc 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss @@ -200,6 +200,16 @@ font-size: 14px; } + &__count-badge { + margin-left: auto; + padding: 1px 8px; + background: rgba(255, 255, 255, 0.25); + border-radius: 10px; + font-size: 11px; + font-weight: 700; + color: #fff; + } + &__panel { margin-top: 8px; background: #f6f8fc; diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts index 9fb284db248..61b88ea0599 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts @@ -80,6 +80,12 @@ export class OperatorMenuComponent { // collapsed. public suggestions: MacroSuggestion[] = []; public isSuggesting: boolean = false; + // Proactive count — how many candidates the heuristic would surface RIGHT + // NOW if the user clicked the button. Refreshed whenever the canvas changes + // (with a short debounce). Surfaced as a small chip on the Suggest button + // so the user sees "4 candidates found" without having to click. This is + // the "agent is watching your workflow" feel. + public availableCandidateCount: number = 0; // input value of the search input box public searchInputValue: string = ""; @@ -123,6 +129,49 @@ export class OperatorMenuComponent { .getWorkflowModificationEnabledStream() .pipe(untilDestroyed(this)) .subscribe(canModify => (this.canModify = canModify)); + // Proactive macro-suggestion watcher: every time the workflow graph + // changes (add/delete/relink), debounce 700ms then run the heuristic + // suggester silently and update `availableCandidateCount`. The UI badges + // the Suggest button so the user discovers patterns without clicking. + // 700ms is long enough that mid-drag operator placements don't trigger + // a flicker, short enough to feel responsive after a click settles. + const refreshSuggestionCount = () => { + try { + const graph = this.workflowActionService.getTexeraGraph(); + const list = this.macroSuggestionService.suggestMacros(graph); + this.availableCandidateCount = list.length; + } catch { + this.availableCandidateCount = 0; + } + }; + let debounceHandle: ReturnType | null = null; + const scheduleRefresh = () => { + if (debounceHandle) clearTimeout(debounceHandle); + debounceHandle = setTimeout(refreshSuggestionCount, 700); + }; + this.workflowActionService + .getTexeraGraph() + .getOperatorAddStream() + .pipe(untilDestroyed(this)) + .subscribe(scheduleRefresh); + this.workflowActionService + .getTexeraGraph() + .getOperatorDeleteStream() + .pipe(untilDestroyed(this)) + .subscribe(scheduleRefresh); + this.workflowActionService + .getTexeraGraph() + .getLinkAddStream() + .pipe(untilDestroyed(this)) + .subscribe(scheduleRefresh); + this.workflowActionService + .getTexeraGraph() + .getLinkDeleteStream() + .pipe(untilDestroyed(this)) + .subscribe(scheduleRefresh); + // Kick off an initial scan once the canvas has settled. + setTimeout(refreshSuggestionCount, 1200); + this.operatorMetadataService .getOperatorMetadata() .pipe(untilDestroyed(this)) From 59994ea07953578c1be7a6ad61a6d35d8e752070 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:34:32 -0700 Subject: [PATCH 39/65] fix(macro): set schema-propagation fields on fused PythonUDFOpDescV2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - substituteFused now sets retainInputColumns=true (for macros with inputs) and workers=1 on the synthesized PythonUDFOpDescV2. Without these, RegionExecutionCoordinator throws 'Schema is missing' because PythonUDFOpDescV2.getOutputSchema can't materialize one. - Mirror the fix in workflow-compiling-service/MacroExpander. - For input-less macros (sources/aggregators inside) the fused op gets retainInputColumns=false; users with such macros need to declare outputColumns manually in the fused code's property panel — a follow-up issue, not a hackathon blocker. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../texera/workflow/macroOp/MacroExpander.scala | 11 +++++++++++ .../texera/amber/compiler/macroOp/MacroExpander.scala | 7 +++++++ 2 files changed, 18 insertions(+) diff --git a/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala index 8c61adf3a21..939e9f98e92 100644 --- a/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala +++ b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala @@ -255,6 +255,17 @@ object MacroExpander { val instanceId = m.operatorIdentifier.id val fused = new PythonUDFOpDescV2() fused.code = fusion.code + // Schema propagation for the fused UDF: a fused macro that takes an input + // re-emits a tuple of the same shape (filter/projection/map operators + // mutate or drop the input dict but don't introduce new columns unless + // the user adds them in the fused code). retainInputColumns=true lets the + // engine carry the input schema through to the output without a hand- + // declared outputColumns list. workers=1 keeps the fused execution + // single-actor — the whole point of fusion is collapsing serialization + // hops, not parallelism. + fused.retainInputColumns = m.inputPortCount > 0 + fused.outputColumns = List.empty + fused.workers = 1 // Keep the macro op's external interface — same input/output port // counts so the upstream/downstream link wiring on the parent canvas // doesn't need to change. diff --git a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala index 7bc40f26394..f3100a39884 100644 --- a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala +++ b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala @@ -250,6 +250,13 @@ object MacroExpander { val fusion = m.fusion.get val fused = new PythonUDFOpDescV2() fused.code = fusion.code + // Schema propagation: see amber/.../MacroExpander.scala substituteFused + // for the same rationale. retainInputColumns lets the engine carry the + // input schema through to the output without a hand-declared + // outputColumns list; workers=1 keeps the fused execution single-actor. + fused.retainInputColumns = m.inputPortCount > 0 + fused.outputColumns = List.empty + fused.workers = 1 fused.inputPorts = (0 until m.inputPortCount).map { i => PortDescription( portID = s"input-$i", From 5dd6739d439efca069a4affbcbbe01357a65ad90 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:36:29 -0700 Subject: [PATCH 40/65] feat(macro): stale-instance detection + refresh-from-source action MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - MacroService caches lastModifiedTime per wid from every listMacros response; getLatestModifiedTime(wid) is the lookup helper. - Macro instances on parent canvases now stamp macroSyncedAt on the operatorProperties at create-time (both palette click-to-add and createMacroFromSelection). The timestamp records when this instance was last in-sync with its source definition. - Context menu adds "refresh macro (stale)" (gold sync icon), visible only when the macro's lastModifiedTime > macroSyncedAt. Clicking bumps the timestamp and drops any verified-fusion flag (since the body content has changed; old fusion verification no longer applies). - Scaffolding for the macro-versioning UX — surfaces meaningfully once drill-down body editing persists back to the source definition. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../operator-menu/operator-menu.component.ts | 7 +++ .../context-menu/context-menu.component.html | 11 ++++ .../context-menu/context-menu.component.ts | 61 +++++++++++++++++++ .../workspace/service/macro/macro.service.ts | 45 +++++++++++++- 4 files changed, 123 insertions(+), 1 deletion(-) diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts index 61b88ea0599..28abbba686d 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts @@ -291,6 +291,13 @@ export class OperatorMenuComponent { inputPortCount, outputPortCount, displayName: m.name, + // Mark this instance as in-sync-with the macro's CURRENT + // lastModifiedTime. If the macro is later edited, this stays put; + // the "refresh macro (stale)" context-menu item then surfaces. + macroSyncedAt: + typeof m.lastModifiedTime === "number" + ? m.lastModifiedTime + : new Date(m.lastModifiedTime as unknown as string).getTime(), }, inputPorts, outputPorts, diff --git a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html index 9dfb1dfa6a5..ec54c86db6d 100644 --- a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html +++ b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.html @@ -165,6 +165,17 @@ nzTheme="outline">unfuse (restore body)
    • +
  • + refresh macro (stale) +
  • { + try { + return this.workflowActionService.getTexeraGraph().getOperator(opId); + } catch { + return undefined; + } + })(); + if (op?.operatorType !== "Macro") return false; + const macroId = op.operatorProperties?.["macroId"]; + if (typeof macroId !== "string" || macroId.length === 0) return false; + // Only worth offering if we actually know of a newer-than-instance time. + const syncedAt = Number(op.operatorProperties?.["macroSyncedAt"] ?? 0); + const latest = this.macroService.getLatestModifiedTime(macroId); + return latest > 0 && latest > syncedAt; + } + + public onRefreshMacroInstance(): void { + const opId = this.highlightedOperatorIds[0]; + if (!opId) return; + const graph = this.workflowActionService.getTexeraGraph(); + const op = (() => { + try { + return graph.getOperator(opId); + } catch { + return undefined; + } + })(); + if (!op) return; + const macroId = op.operatorProperties?.["macroId"] as string; + const latest = this.macroService.getLatestModifiedTime(macroId); + const newProperties: Record = { ...op.operatorProperties }; + newProperties["macroSyncedAt"] = latest > 0 ? latest : Date.now(); + // The fusion's contract is "verified for THIS body's hash". When the + // body changes (the trigger for refresh), drop the verified flag so + // the next compile re-inlines the up-to-date body. The user can re- + // fuse against the new body if desired. + if (newProperties["fusion"]) { + delete newProperties["fusion"]; + const paper = this.workflowActionService.getJointGraphWrapper().getMainJointPaper(); + if (paper) this.jointUIService.refreshMacroFusionStyle(paper, opId, false); + } + this.workflowActionService.setOperatorProperty(opId, newProperties); + this.notificationService.info("Macro instance refreshed to latest definition."); + } + public onFuseMacro(): void { const opId = this.highlightedOperatorIds[0]; if (!opId) return; diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index c0442b03d53..ed302c5991c 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -231,6 +231,13 @@ export class MacroService { inputPortCount: built.inputPortCount, outputPortCount: built.outputPortCount, displayName: detail.name, + // Newly-created instance is in-sync with the definition we just + // POSTed; stamp the modify time so the staleness check in the + // context-menu sees this as fresh until the definition is edited. + macroSyncedAt: + typeof detail.lastModifiedTime === "number" + ? detail.lastModifiedTime + : new Date(detail.lastModifiedTime as unknown as string).getTime(), }, inputPorts, outputPorts, @@ -386,7 +393,43 @@ export class MacroService { } public listMacros(): Observable { - return this.http.get(`${AppSettings.getApiEndpoint()}/${MACRO_LIST_URL}`); + return this.http + .get(`${AppSettings.getApiEndpoint()}/${MACRO_LIST_URL}`) + .pipe( + tap(summaries => { + // Mirror into the latest-modified cache so canvas-side consumers can + // detect stale instances without re-fetching. lastModifiedTime is a + // string in transport (LDT JSON) but a number once Jackson serializes + // a Timestamp; coerce both into ms-since-epoch for easy compare. + for (const m of summaries) { + const tsRaw = m.lastModifiedTime as unknown; + const tsMs = + typeof tsRaw === "number" ? tsRaw : new Date(tsRaw as string).getTime(); + this.latestModifiedByWid.set(m.wid, tsMs); + } + }) + ); + } + + /** + * Map of `macroId → most recently seen lastModifiedTime` (epoch ms), + * populated by every `listMacros` response. Used by the "refresh macro + * instance" context-menu action to decide whether a canvas instance is + * stale, and to imprint the freshness timestamp when re-syncing. + */ + private latestModifiedByWid = new Map(); + + /** + * Lookup helper for callers (e.g. the JointUI service when it renders a + * Macro op) — returns the most recent lastModifiedTime we've seen for the + * given macro definition, in ms since epoch. Returns 0 if we haven't seen + * the macro yet (i.e. listMacros hasn't been called or the macro is + * inaccessible to the current user). + */ + public getLatestModifiedTime(macroId: number | string): number { + const wid = typeof macroId === "number" ? macroId : Number(macroId); + if (!Number.isFinite(wid)) return 0; + return this.latestModifiedByWid.get(wid) ?? 0; } public getMacro(wid: number): Observable { From d9207148b71a3f76e8b5b80e39fcebdadd8bb393 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:39:07 -0700 Subject: [PATCH 41/65] feat(macro): show estimated speedup on canvas next to FUSED badge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - MacroFusion payload now carries estimatedSpeedup (e.g. "2.5×"). - JointUI builds the canvas badge as "⚡ FUSED · 2.5×" when the speedup is set, falling back to plain "⚡ FUSED" for older fused instances. - refreshMacroFusionStyle accepts the speedup as an optional 4th arg so the post-fuse visual update reflects the new metric without a full re-render. - Reinforces the demo claim "fusing collapses N actors into 1 — here's the predicted speedup" without forcing the user to open the property panel. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../context-menu/context-menu.component.ts | 6 ++++-- .../service/joint-ui/joint-ui.service.ts | 17 +++++++++++++---- .../service/macro/macro-fusion.service.ts | 6 ++++++ 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts index b8cdd08ab14..645e0059220 100644 --- a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts @@ -528,9 +528,11 @@ export class ContextMenuComponent { fusion: this.macroFusionService.toFusionPayload(result), }; this.workflowActionService.setOperatorProperty(opId, newProperties); - // Update the visual immediately — solid gold stroke + ⚡FUSED badge. + // Update the visual immediately — solid gold stroke + ⚡FUSED badge, + // with the speedup metric appended so the perf claim is on-canvas. const paper = this.workflowActionService.getJointGraphWrapper().getMainJointPaper(); - if (paper) this.jointUIService.refreshMacroFusionStyle(paper, opId, true); + if (paper) + this.jointUIService.refreshMacroFusionStyle(paper, opId, true, result.estimatedSpeedup); this.notificationService.success( `Fused "${macroOp.customDisplayName ?? macroOp.operatorID}" — ${result.rationale}` ); diff --git a/frontend/src/app/workspace/service/joint-ui/joint-ui.service.ts b/frontend/src/app/workspace/service/joint-ui/joint-ui.service.ts index 017b8af0dcb..6a329bf34ad 100644 --- a/frontend/src/app/workspace/service/joint-ui/joint-ui.service.ts +++ b/frontend/src/app/workspace/service/joint-ui/joint-ui.service.ts @@ -479,14 +479,16 @@ export class JointUIService { public refreshMacroFusionStyle( jointPaper: joint.dia.Paper, operatorID: string, - isFused: boolean + isFused: boolean, + estimatedSpeedup?: string ): void { const model = jointPaper.getModelById(operatorID); if (!model) return; if (isFused) { model.attr("rect.body/stroke", "#d4a017"); model.attr("rect.body/stroke-dasharray", "none"); - model.attr(`.${operatorFusionBadgeClass}/text`, "⚡ FUSED"); + const badgeText = estimatedSpeedup ? `⚡ FUSED · ${estimatedSpeedup}` : "⚡ FUSED"; + model.attr(`.${operatorFusionBadgeClass}/text`, badgeText); model.attr(`.${operatorFusionBadgeClass}/visibility`, "visible"); } else { model.attr("rect.body/stroke", "#1d6fdb"); @@ -748,7 +750,7 @@ export class JointUIService { // body at compile time — so visually we want the node to read differently // from a normal (still-inlinable) macro. Solid gold stroke + ⚡FUSED badge. const fusion = (operator.operatorProperties as Record | undefined)?.["fusion"] as - | { verified?: boolean } + | { verified?: boolean; estimatedSpeedup?: string } | undefined; const isFusedMacro = isMacroInstance && fusion?.verified === true; const bodyStroke = isFusedMacro ? "#d4a017" : isMacroInstance ? "#1d6fdb" : isMacroMarker ? "#888888" : "red"; @@ -757,7 +759,14 @@ export class JointUIService { // "this node is now a single op, not a composite waiting to be inlined". const bodyStrokeDasharray = isMacroInstance && !isFusedMacro ? "6,3" : undefined; const bodyRadius = isMacroMarker ? "20px" : "5px"; - const fusionBadgeText = isFusedMacro ? "⚡ FUSED" : ""; + // Badge: "⚡ FUSED" alone, OR "⚡ FUSED · 2.5×" when we have a speedup + // estimate. The speedup is set by MacroFusionService when the fusion is + // first generated and persisted into operatorProperties.fusion. + const fusionBadgeText = isFusedMacro + ? fusion?.estimatedSpeedup + ? `⚡ FUSED · ${fusion.estimatedSpeedup}` + : "⚡ FUSED" + : ""; return { ".texera-operator-coeditor-editing": { text: "", diff --git a/frontend/src/app/workspace/service/macro/macro-fusion.service.ts b/frontend/src/app/workspace/service/macro/macro-fusion.service.ts index 169f6031e29..74b46e8b12c 100644 --- a/frontend/src/app/workspace/service/macro/macro-fusion.service.ts +++ b/frontend/src/app/workspace/service/macro/macro-fusion.service.ts @@ -30,6 +30,11 @@ export interface MacroFusion { verified: boolean; sampleSize: number; verifiedAt: number; + // Human-readable speedup estimate (e.g. "2.5×"). Rendered on the canvas + // next to the ⚡ FUSED badge so the user sees the perf claim at a glance. + // Optional — older fused instances created before this field existed will + // render as just "⚡ FUSED" until re-fused. + estimatedSpeedup?: string; } export interface FusionResult { @@ -93,6 +98,7 @@ export class MacroFusionService { verified: result.verified, sampleSize: result.sampleSize, verifiedAt: Date.now(), + estimatedSpeedup: result.estimatedSpeedup, }; } From be666d20d96f1813d22878e47719b869b9a3f6e8 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:40:09 -0700 Subject: [PATCH 42/65] feat(macro): pre-fill smart default name in Create Macro prompt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replace `macro-${timestamp}` default with a snake-cased name derived from the selected operator types (filter_projection_block, csvfilescan_regex_filter_block, etc.). - 4+-op selections get a "_pipeline" suffix; 3-op selections get "_block"; pairs use the bare type names. Same shape the AI panel's pattern suggestions produce — names stay consistent between the right-click and auto-materialize paths. - Falls back to the timestamp form if the selection can't be inspected. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../context-menu/context-menu.component.ts | 37 ++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts index 645e0059220..505aed86ce4 100644 --- a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts @@ -546,7 +546,12 @@ export class ContextMenuComponent { if (selected.length < 2) { return; } - const name = window.prompt("Macro name", `macro-${Date.now()}`); + // Pre-fill the prompt with a smart default derived from the selected + // operators' types so the user gets a readable name (e.g. + // "filter_projection_block") rather than a UNIX-time tag. Falls back to + // the legacy timestamp if no type info is available. + const defaultName = this.suggestedMacroNameForSelection(selected) || `macro-${Date.now()}`; + const name = window.prompt("Macro name", defaultName); if (!name) { return; } @@ -568,6 +573,36 @@ export class ContextMenuComponent { }); } + /** + * Suggest a snake-cased default name for a fresh macro from its selected + * operator types. "Filter→Projection" → "filter_projection_block"; + * a 4+-op chain gets a `_pipeline` suffix. Falls back to undefined if + * we can't read the types (caller should default to the timestamp form). + * + * Mirrors `MacroSuggestionService.suggestedNameForChain` so the names + * produced via right-click match the names suggested by the AI panel. + */ + private suggestedMacroNameForSelection(selectedIds: readonly string[]): string | undefined { + if (selectedIds.length === 0) return undefined; + const graph = this.workflowActionService.getTexeraGraph(); + const types: string[] = []; + for (const id of selectedIds) { + try { + types.push(graph.getOperator(id).operatorType); + } catch { + return undefined; + } + } + if (types.length === 0) return undefined; + const compact = types + .slice(0, Math.min(3, types.length)) + .map(t => t.replace(/OpDesc$|Op$/, "")) + .map(t => t.toLowerCase()) + .map(t => t.replace(/[^a-z0-9]/g, "")); + const suffix = types.length >= 4 ? "_pipeline" : types.length >= 3 ? "_block" : ""; + return compact.join("_") + suffix; + } + private swapSelectionWithMacroNode( detail: MacroDetail, selectedOpIDs: readonly string[], From 402f65fdfbef88a44aa8a200d232a3921dcd3746 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:40:52 -0700 Subject: [PATCH 43/65] feat(macro): hover-preview highlighting for suggestion rows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Hovering a row in the Suggest Macros panel flashes that suggestion's operators on the canvas (via JointGraphWrapper highlight). Unhover restores whatever the user had highlighted before. - Lets the user scan the candidate list and SEE which ops each suggestion covers without committing to materializing — replaces the old "only top suggestion is highlighted on Suggest click" flow with a more interactive browse. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../operator-menu.component.html | 2 ++ .../operator-menu/operator-menu.component.ts | 28 +++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html index 97a8c63fd88..f5e35bcbf40 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html @@ -72,6 +72,8 @@ class="suggest-macros__item" [class.suggest-macros__item--pattern]="suggestion.id.startsWith('pattern-')" (click)="onMaterializeSuggestion(suggestion)" + (mouseenter)="onSuggestionHover(suggestion)" + (mouseleave)="onSuggestionUnhover()" [title]="suggestion.rationale">
    ⟲ pattern diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts index 28abbba686d..6fe8202840a 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts @@ -416,6 +416,34 @@ export class OperatorMenuComponent { this.suggestions = []; } + /** + * Hovering a suggestion row should flash that suggestion's operators on + * the canvas as a visual preview. We highlight via JointGraphWrapper — + * same path that selection uses — so the canvas treatment matches the + * "selected" look. On unhover we restore whatever the user had highlighted + * before they started hovering (typically: nothing). + * + * Stash the prior highlight set in `preHoverHighlight` so unhover can + * cleanly undo without clobbering other UI state. + */ + private preHoverHighlight: string[] = []; + public onSuggestionHover(suggestion: MacroSuggestion): void { + const jw = this.workflowActionService.getJointGraphWrapper(); + this.preHoverHighlight = Array.from(jw.getCurrentHighlightedOperatorIDs()); + jw.unhighlightOperators(...this.preHoverHighlight); + jw.setMultiSelectMode(true); + jw.highlightOperators(...suggestion.operatorIds); + } + + public onSuggestionUnhover(): void { + const jw = this.workflowActionService.getJointGraphWrapper(); + jw.unhighlightOperators(...jw.getCurrentHighlightedOperatorIDs()); + if (this.preHoverHighlight.length > 0) { + jw.highlightOperators(...this.preHoverHighlight); + } + this.preHoverHighlight = []; + } + /** Reference to the hidden file input — clicked programmatically by `onTriggerImportMacro`. */ @ViewChild("macroImportFile") macroImportFile?: ElementRef; From f3dee71a091ef7be0f9a2f09f7bcb097fff0f789 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:44:37 -0700 Subject: [PATCH 44/65] feat(macro): one-click 'Fuse all macros' + Your Macros filter input MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Fuse all macros: gold gradient button appears on workflows with at least one un-fused Macro op. Click → forkJoin over every Macro, generate + stamp fusion + refresh canvas style in parallel. Per- macro failures don't abort the batch; the toast separates fused vs. skipped counts. Hides when there's nothing to do. - Your Macros section gains a search-by-name filter input when the user has more than 3 saved macros — speeds finding the right one in a large library. - The collapse header shows a total count (e.g. "Your Macros (14)"). - forkJoin + catchError keep the batch resilient: one bad macroId (e.g. a deleted definition) won't break the rest. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../operator-menu.component.html | 24 +++- .../operator-menu.component.scss | 46 ++++++++ .../operator-menu/operator-menu.component.ts | 103 ++++++++++++++++++ 3 files changed, 171 insertions(+), 2 deletions(-) diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html index f5e35bcbf40..ba95af57c6d 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html @@ -87,6 +87,20 @@
    + + + +
    +
    {{ group.category }}
    +
    + ▦ + + {{ macroSchema.additionalMetadata.userFriendlyName }} + + + {{ macroSchema.__macroSummary!.usageCount }}× used + + + {{ macroSchema.additionalMetadata.inputPorts.length }} in / + {{ macroSchema.additionalMetadata.outputPorts.length }} out + + +
    diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss index a6ffaa12f9d..b9abac33aec 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss @@ -165,6 +165,20 @@ } } +.macro-category { + margin: 4px 0 6px; + + &__header { + font-size: 10px; + font-weight: 700; + color: #888; + text-transform: uppercase; + letter-spacing: 0.5px; + margin: 6px 0 3px; + padding: 0 4px; + } +} + .your-macros-filter { width: 100%; padding: 4px 8px; diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts index 7f555bfa67f..bd600a62e07 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts @@ -89,6 +89,91 @@ export class OperatorMenuComponent { ); } + /** + * Cache for the per-macro auto-inferred category. Keyed by macroId (the + * wid as string). Populated lazily on demand and after each listMacros + * refresh so the palette can group by category without an extra HTTP call + * per macro on every render. Categories are derived from the macro body's + * operator-type composition — see `inferCategory()`. + */ + private macroCategoryCache = new Map(); + + /** + * Cheap heuristic that maps a macro's body to a category label. We pull + * the body once via `getMacro` and inspect the operator-type composition: + * - majority Aggregate/GroupBy/Sort → "aggregation" + * - majority Visualizer/Chart → "visualization" + * - majority PythonUDF/Lambda → "transformation" + * - else (Filter/Projection/Regex/...) → "preprocessing" + * + * Returns "uncategorized" if we can't load the body. The category is + * cached the first time we look up a given macroId. + */ + public categoryForMacro(macroSchema: OperatorSchema & { __macroSummary?: MacroSummary }): string { + const wid = macroSchema.__macroSummary?.wid; + if (!wid) return "uncategorized"; + const key = String(wid); + const cached = this.macroCategoryCache.get(key); + if (cached) return cached; + // Fire and forget the body fetch; once it lands, fill the cache so a + // future render uses the real category. + this.macroService.getMacro(wid).subscribe({ + next: detail => { + try { + const body = JSON.parse(detail.content) as { + operators?: Array<{ operatorType?: string }>; + }; + this.macroCategoryCache.set(key, this.inferCategory(body.operators ?? [])); + } catch { + this.macroCategoryCache.set(key, "uncategorized"); + } + }, + error: () => this.macroCategoryCache.set(key, "uncategorized"), + }); + return "loading…"; + } + + private inferCategory(operators: Array<{ operatorType?: string }>): string { + const inner = operators.filter( + o => o.operatorType !== "MacroInput" && o.operatorType !== "MacroOutput" + ); + if (inner.length === 0) return "uncategorized"; + const counts = { agg: 0, viz: 0, transform: 0, prep: 0 }; + for (const op of inner) { + const t = (op.operatorType ?? "").toLowerCase(); + if (/aggregate|groupby|sort|reduce|count/.test(t)) counts.agg++; + else if (/visualizer|chart|wordcloud|plot|piechart|barchart|linechart/.test(t)) counts.viz++; + else if (/python|lambda|udf/.test(t)) counts.transform++; + else counts.prep++; + } + const max = Math.max(counts.agg, counts.viz, counts.transform, counts.prep); + if (max === counts.viz && counts.viz > 0) return "visualization"; + if (max === counts.agg && counts.agg > 0) return "aggregation"; + if (max === counts.transform && counts.transform > 0) return "transformation"; + return "preprocessing"; + } + + /** + * Group the filtered macro list by auto-inferred category, returning a + * stable category-ordered array so the palette renders deterministically. + * Empty categories are omitted. Categories the user hasn't seen yet emit + * a single "loading…" bucket that swaps in once getMacro lands. + */ + public get groupedMacroList(): Array<{ category: string; macros: (OperatorSchema & { __macroSummary?: MacroSummary })[] }> { + const categoryOrder = ["preprocessing", "transformation", "aggregation", "visualization", "uncategorized", "loading…"]; + const grouped = new Map(); + for (const macro of this.filteredMacroList) { + const cat = this.categoryForMacro(macro); + if (!grouped.has(cat)) grouped.set(cat, []); + grouped.get(cat)!.push(macro); + } + const result: Array<{ category: string; macros: (OperatorSchema & { __macroSummary?: MacroSummary })[] }> = []; + for (const cat of categoryOrder) { + if (grouped.has(cat)) result.push({ category: cat, macros: grouped.get(cat)! }); + } + return result; + } + // Inline panel for "AI" macro suggestions. Populated on user click, then // cleared after a selection is materialized. Empty list means panel is // collapsed. From 44154d143659a2cf71ef193915f0241fefd3626b Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:49:18 -0700 Subject: [PATCH 46/65] feat(macro): show op-chain subtitle under each palette macro MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Each palette macro now renders its op-type chain as a small subtitle beneath the name (e.g. 'Filter→Projection' or 'Filter→Projection→ Limit +2' when the chain is longer than 3 ops). - Lazily fetched alongside the category cache from the same getMacro call, so adding the subtitle costs zero extra HTTP roundtrips beyond what categorization already does. - Gives at-a-glance context for what each macro does without the user having to hover/click — important once libraries grow past a few similarly-named macros. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../operator-menu.component.html | 10 +++++- .../operator-menu.component.scss | 9 +++++ .../operator-menu/operator-menu.component.ts | 36 +++++++++++++++++-- 3 files changed, 52 insertions(+), 3 deletions(-) diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html index 867c69ea769..ed1579cfc9d 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html @@ -147,8 +147,16 @@ [class.disabled]="!canModify" [title]="macroSchema.additionalMetadata.operatorDescription"> ▦ - + {{ macroSchema.additionalMetadata.userFriendlyName }} + + {{ subtitleForMacro(macroSchema) }} + (); + /** + * Cache for the per-macro op-type subtitle (e.g. "Filter→Projection→Limit" + * truncated to 3 ops with a "+N" suffix when longer). Populated alongside + * the category cache from the same `getMacro` fetch. + */ + private macroSubtitleCache = new Map(); /** * Cheap heuristic that maps a macro's body to a category label. We pull @@ -123,16 +129,42 @@ export class OperatorMenuComponent { const body = JSON.parse(detail.content) as { operators?: Array<{ operatorType?: string }>; }; - this.macroCategoryCache.set(key, this.inferCategory(body.operators ?? [])); + const ops = body.operators ?? []; + this.macroCategoryCache.set(key, this.inferCategory(ops)); + this.macroSubtitleCache.set(key, this.subtitleFromOps(ops)); } catch { this.macroCategoryCache.set(key, "uncategorized"); + this.macroSubtitleCache.set(key, ""); } }, - error: () => this.macroCategoryCache.set(key, "uncategorized"), + error: () => { + this.macroCategoryCache.set(key, "uncategorized"); + this.macroSubtitleCache.set(key, ""); + }, }); return "loading…"; } + /** + * Return a short op-type chain string for a macro, e.g. + * "Filter→Projection" or "Filter→Projection→Limit +2" for longer chains. + * Lazily populated alongside the category cache by `categoryForMacro`. + */ + public subtitleForMacro(macroSchema: OperatorSchema & { __macroSummary?: MacroSummary }): string { + const wid = macroSchema.__macroSummary?.wid; + if (!wid) return ""; + return this.macroSubtitleCache.get(String(wid)) ?? ""; + } + + private subtitleFromOps(operators: Array<{ operatorType?: string }>): string { + const inner = operators + .filter(o => o.operatorType !== "MacroInput" && o.operatorType !== "MacroOutput") + .map(o => o.operatorType ?? "?"); + if (inner.length === 0) return ""; + const head = inner.slice(0, 3).join("→"); + return inner.length > 3 ? `${head} +${inner.length - 3}` : head; + } + private inferCategory(operators: Array<{ operatorType?: string }>): string { const inner = operators.filter( o => o.operatorType !== "MacroInput" && o.operatorType !== "MacroOutput" From 81130273fc8def3b18df0027c54ed5a76643f054 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:50:45 -0700 Subject: [PATCH 47/65] feat(macro): richer rationales in suggestion panel MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Per-pattern rationale generators surface domain-specific hints ("Filter + project block", "Row-filter block", "Text-summary visualization", "Aggregate + project block", etc.) rather than the generic "preprocessing pipeline" pitch. - Each rationale also explains the *why* of extraction ("Encapsulating this protects downstream consumers from schema changes", "Reusing this pipeline keeps your analytics consistent across workflows", etc.) — gives demo viewers a sense of the agent's intent, not just its pattern detection. - Adds detection for visualization and join+reshape patterns. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../service/macro/macro-suggestion.service.ts | 49 +++++++++++++++++-- 1 file changed, 46 insertions(+), 3 deletions(-) diff --git a/frontend/src/app/workspace/service/macro/macro-suggestion.service.ts b/frontend/src/app/workspace/service/macro/macro-suggestion.service.ts index bde77b4cdee..5a4305360e4 100644 --- a/frontend/src/app/workspace/service/macro/macro-suggestion.service.ts +++ b/frontend/src/app/workspace/service/macro/macro-suggestion.service.ts @@ -298,12 +298,18 @@ export class MacroSuggestionService { return `Two-step pipeline: ${head} → ${tail}. Reusable as a unit.`; } if (this.looksLikePreprocessing(types)) { - return `Looks like a reusable preprocessing block (${chain.length} ops).`; + return `${this.preprocessingHint(types)} (${chain.length} ops). Encapsulating this protects downstream consumers from the schema changes.`; } if (this.looksLikeAggregation(types)) { - return `Looks like a reusable aggregation pipeline (${chain.length} ops).`; + return `${this.aggregationHint(types)} (${chain.length} ops). Reusing this pipeline keeps your analytics consistent across workflows.`; } - return `Linear ${chain.length}-step chain — good macro candidate.`; + if (this.looksLikeVisualization(types)) { + return `${this.visualizationHint(types)} (${chain.length} ops). Once captured, the same chart definition can be reused without recopying ops.`; + } + if (this.looksLikeJoinAndShape(types)) { + return `Join + reshape pipeline (${chain.length} ops). Encapsulating hides the join's key contract behind a single macro port.`; + } + return `Linear ${chain.length}-step chain — good macro candidate. Extracts the unit and frees the parent canvas of intermediate ops.`; } private looksLikePreprocessing(types: string[]): boolean { @@ -316,6 +322,43 @@ export class MacroSuggestionService { return /aggregate|group|sum|count|reduce/.test(lc); } + private looksLikeVisualization(types: string[]): boolean { + const lc = types.join(" ").toLowerCase(); + return /chart|plot|visualizer|wordcloud|piechart|barchart|linechart/.test(lc); + } + + private looksLikeJoinAndShape(types: string[]): boolean { + const lc = types.join(" ").toLowerCase(); + return /(hashjoin|cartesian|union).*(projection|filter|map)/.test(lc); + } + + /** + * Detailed rationale generators — slot in the user's actual op types so + * the suggestion reads as concrete advice ("Filter → Projection block") + * instead of a generic "preprocessing pipeline" pitch. + */ + private preprocessingHint(types: string[]): string { + const lc = types.join(" ").toLowerCase(); + if (lc.includes("filter") && lc.includes("projection")) return "Filter + project block"; + if (lc.includes("filter")) return "Row-filter block"; + if (lc.includes("projection")) return "Column-project block"; + return "Preprocessing block"; + } + + private aggregationHint(types: string[]): string { + const lc = types.join(" ").toLowerCase(); + if (lc.includes("aggregate") && lc.includes("projection")) return "Aggregate + project block"; + if (lc.includes("groupby") || lc.includes("aggregate")) return "Grouping/aggregation block"; + return "Reduction pipeline"; + } + + private visualizationHint(types: string[]): string { + const lc = types.join(" ").toLowerCase(); + if (lc.includes("wordcloud")) return "Text-summary visualization"; + if (lc.includes("piechart") || lc.includes("barchart") || lc.includes("linechart")) return "Categorical chart block"; + return "Visualization block"; + } + private suggestedNameForChain(chain: string[], ops: readonly OperatorPredicate[]): string { const types = chain.map(id => ops.find(o => o.operatorID === id)?.operatorType ?? "Op"); // Compact 2-3 of the type names into a snake-cased candidate name. From 174a0bc3f8a6999a18fa94dd8a57f2a36b3c0aba Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:52:17 -0700 Subject: [PATCH 48/65] feat(macro): auto-generate description at create time MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - buildMacroFromSelection now fills the description with a 1-line summary derived from the body's op chain and port shape, e.g. 'Filter → Projection (2 ops, 1 in / 1 out)' or 'CSVFileScan → PythonUDFV2 → Aggregate +3 (7 ops, 0 in / 1 out)'. - Removes empty descriptions from the dashboard / palette tooltip and gives the macro a self-documenting summary the user can edit later if they want. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../workspace/service/macro/macro.service.ts | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index ed302c5991c..9190a0cf5a2 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -899,9 +899,20 @@ export class MacroService { macroPortIndex: outputIdxByInnerPort.get(`${l.srcOp}|${l.srcPort}`) as number, })); + // Auto-generate a 1-line description so users don't get an empty + // description on the dashboard / palette tooltip. Format: + // "Filter → Projection block (2 ops, 1 in/1 out)". + const innerOpTypes = selectedOperatorIDs.map(opId => graph.getOperator(opId).operatorType); + const description = this.autoDescriptionForBody( + innerOpTypes, + inputMarkers.length, + outputMarkers.length + ); + return { request: { name, + description, content: JSON.stringify(body), portSpec, }, @@ -912,6 +923,24 @@ export class MacroService { }; } + /** + * Compose a one-line description for a freshly-created macro based on the + * operator-type composition of its body and its external port shape. The + * resulting string lands on the macro definition's `description` field and + * shows up in the palette tooltip + the dashboard macro browser. + */ + private autoDescriptionForBody( + innerOpTypes: readonly string[], + inputPortCount: number, + outputPortCount: number + ): string { + if (innerOpTypes.length === 0) return "Empty macro"; + const head = innerOpTypes.slice(0, 3).join(" → "); + const chain = innerOpTypes.length > 3 ? `${head} +${innerOpTypes.length - 3}` : head; + const portShape = `${inputPortCount} in / ${outputPortCount} out`; + return `${chain} (${innerOpTypes.length} ops, ${portShape})`; + } + /** * Adapt a backend `MacroDetail` (whose `content` is a serialized `MacroBody`) * into a `Workflow`-shaped object the existing `reloadWorkflow` flow can From 74da9211b624146604661a4c6d0354a13b65222c Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:53:20 -0700 Subject: [PATCH 49/65] feat(macro): export records nested-macro dependencies for portability - exportMacroToFile now scans the body content for any nested macroId references and records them in the export payload as dependsOnMacroWids: [wid, ...]. Future v2 import can fetch and recreate these on the target instance before the root, producing a self-contained transfer. - Even without v2 import, the record gives a clear signal at import time that the macro has dependencies the user needs to bring along. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../workspace/service/macro/macro.service.ts | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index 9190a0cf5a2..c50a9a9a168 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -323,6 +323,14 @@ export class MacroService { * `importMacroFromJson`. We deliberately exclude wid and timestamps because * the importer always creates a fresh definition with a new wid. * + * Transitive: if the macro's body references nested macros, those are + * fetched too and embedded as `nestedMacros[oldWid] = detailPayload`. The + * importer reconstructs them in dependency order before the root macro, + * stitching the new wids into the root's body so the import is fully self- + * contained. (Currently the importer creates the root only; transitive + * import is a v2 enhancement, but the export side records everything so + * a manual rebuild is possible.) + * * The exported `content` is the raw MacroBody JSON string; consumer just * needs to round-trip it through `JSON.parse(JSON.stringify(...))` to stay * Jackson-friendly on re-import. @@ -330,6 +338,10 @@ export class MacroService { public exportMacroToFile(wid: number): Observable { return this.getMacro(wid).pipe( map(detail => { + // Scan the body for nested macros so the export is honest about its + // dependencies. Each nested macroId is stored alongside the root; + // the importer can fetch+create them on the target instance. + const nestedWids = this.collectNestedMacroIds(detail.content); const exportPayload = { schemaVersion: 1, name: detail.name, @@ -341,6 +353,7 @@ export class MacroService { icon: detail.icon, exportedAt: new Date().toISOString(), exportedFromTexera: window.location.host, + dependsOnMacroWids: nestedWids, }; const blob = new Blob([JSON.stringify(exportPayload, null, 2)], { type: "application/json", @@ -359,6 +372,22 @@ export class MacroService { ); } + /** + * Scan a macro's content (JSON string) for nested macroId references. The + * scan is regex-based for speed and resilience — body shape may have + * additional fields we don't care about. Used by `exportMacroToFile` to + * record dependencies in the export payload. + */ + private collectNestedMacroIds(content: string): number[] { + const matches = content.match(/"macroId"\s*:\s*"(\d+)"/g) ?? []; + const wids = new Set(); + for (const m of matches) { + const numMatch = m.match(/(\d+)/); + if (numMatch) wids.add(Number(numMatch[1])); + } + return Array.from(wids); + } + /** * Reverse of `exportMacroToFile`: parse an uploaded JSON file and POST it * as a brand-new macro definition. The new definition's wid is fresh — From abb6940eaf06aa2581cd06fe1a2e585d9f77e251 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:54:47 -0700 Subject: [PATCH 50/65] =?UTF-8?q?feat(macro):=20transitive=20export/import?= =?UTF-8?q?=20bundles=20=E2=80=94=20fully=20self-contained=20transfer?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - exportBundleForMacro walks nested macroId references depth-first and packages every reachable definition into a bundleVersion=2 JSON. - Nested macros are emitted in dependency-first order so the importer can create them children-before-parents. - importMacroFromJson detects bundleVersion=2 and applies it: creates each nested macro on the target instance, builds an oldWid→newWid map, and rewrites the next body's macroId references to the new wids before creating it. The root is rewritten + created last and its MacroDetail is returned. - v1 single-macro JSON exports still parse via the bundleVersion-1 fallback path. - Makes the export/import truly portable across Texera instances even for macros with deep nested dependencies. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../workspace/service/macro/macro.service.ts | 265 +++++++++++++++--- 1 file changed, 221 insertions(+), 44 deletions(-) diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index c50a9a9a168..bc566789655 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -336,40 +336,129 @@ export class MacroService { * Jackson-friendly on re-import. */ public exportMacroToFile(wid: number): Observable { - return this.getMacro(wid).pipe( - map(detail => { - // Scan the body for nested macros so the export is honest about its - // dependencies. Each nested macroId is stored alongside the root; - // the importer can fetch+create them on the target instance. - const nestedWids = this.collectNestedMacroIds(detail.content); - const exportPayload = { - schemaVersion: 1, - name: detail.name, - description: detail.description, - content: detail.content, - portSpec: detail.portSpec, - paramSpec: detail.paramSpec, - category: detail.category, - icon: detail.icon, - exportedAt: new Date().toISOString(), - exportedFromTexera: window.location.host, - dependsOnMacroWids: nestedWids, - }; - const blob = new Blob([JSON.stringify(exportPayload, null, 2)], { - type: "application/json", + return new Observable(subscriber => { + this.exportBundleForMacro(wid).subscribe({ + next: bundle => { + const blob = new Blob([JSON.stringify(bundle, null, 2)], { + type: "application/json", + }); + const url = URL.createObjectURL(blob); + const a = document.createElement("a"); + a.href = url; + const safeName = bundle.name.replace(/[^a-zA-Z0-9_-]+/g, "_").slice(0, 60); + a.download = `macro-${safeName}-${wid}.json`; + document.body.appendChild(a); + a.click(); + document.body.removeChild(a); + URL.revokeObjectURL(url); + subscriber.next(); + subscriber.complete(); + }, + error: err => subscriber.error(err), + }); + }); + } + + /** + * Build the transitive export bundle for a macro: the root payload plus + * full definitions of every nested macro it references (and their nested + * macros, recursively). The result is self-contained — importable on a + * fresh Texera instance with no other prep — and structured to be applied + * dependency-first so each parent's body can be rewritten to reference + * the new wids of its children. + * + * The bundle has a `bundleVersion: 2` marker distinguishing it from the + * v1 single-macro export (`schemaVersion: 1`). Both shapes round-trip + * through `importMacroFromJson`. + */ + public exportBundleForMacro(rootWid: number): Observable<{ + bundleVersion: 2; + name: string; + description: string; + rootContent: string; + portSpec: PortSpec; + paramSpec: unknown; + category?: string; + icon?: string; + exportedAt: string; + exportedFromTexera: string; + nestedMacros: Array<{ + originalWid: number; + name: string; + description: string; + content: string; + portSpec: PortSpec; + paramSpec: unknown; + }>; + }> { + // Walk the dependency graph depth-first, collecting every reachable + // macro id starting from the root. Cycles can't happen for macros + // (MacroExpander guards against them) but we still guard with `seen`. + return new Observable(subscriber => { + const seen = new Set(); + const order: number[] = []; + const details = new Map(); + const visit = (w: number): Promise => + new Promise((resolve, reject) => { + if (seen.has(w)) return resolve(); + seen.add(w); + this.getMacro(w).subscribe({ + next: async d => { + details.set(w, d); + const nestedWids = this.collectNestedMacroIds(d.content); + for (const nw of nestedWids) { + try { + await visit(nw); + } catch (e) { + return reject(e); + } + } + order.push(w); + resolve(); + }, + error: reject, + }); }); - const url = URL.createObjectURL(blob); - const a = document.createElement("a"); - a.href = url; - // Slugify the macro name so the filename is safe across OSes. - const safeName = detail.name.replace(/[^a-zA-Z0-9_-]+/g, "_").slice(0, 60); - a.download = `macro-${safeName}-${detail.wid}.json`; - document.body.appendChild(a); - a.click(); - document.body.removeChild(a); - URL.revokeObjectURL(url); - }) - ); + visit(rootWid).then( + () => { + const root = details.get(rootWid); + if (!root) { + subscriber.error(new Error("Root macro fetch failed")); + return; + } + // Nested macros are everything in `order` except the root, in + // dependency-first order (children before their parents). + const nestedMacros = order + .filter(w => w !== rootWid) + .map(w => { + const d = details.get(w)!; + return { + originalWid: w, + name: d.name, + description: d.description, + content: d.content, + portSpec: d.portSpec, + paramSpec: d.paramSpec, + }; + }); + subscriber.next({ + bundleVersion: 2 as const, + name: root.name, + description: root.description, + rootContent: root.content, + portSpec: root.portSpec, + paramSpec: root.paramSpec, + category: root.category, + icon: root.icon, + exportedAt: new Date().toISOString(), + exportedFromTexera: window.location.host, + nestedMacros, + }); + subscriber.complete(); + }, + err => subscriber.error(err) + ); + }); } /** @@ -392,11 +481,21 @@ export class MacroService { * Reverse of `exportMacroToFile`: parse an uploaded JSON file and POST it * as a brand-new macro definition. The new definition's wid is fresh — * any cross-references inside the original `content` to its own wid are - * left as-is (they'd be self-referential and unused). Returns the created - * MacroDetail so the caller can refresh the palette. + * left as-is (they'd be self-referential and unused). + * + * Bundle support (v2): if the JSON has `bundleVersion: 2`, all nested + * macros are created first (in dependency order), then the root content + * is rewritten to point at the new wids, then the root is created. The + * caller still receives the root's MacroDetail — the nested macros land + * in the user's library silently. Schema v1 (single-macro JSON) still + * works for back-compat. */ public importMacroFromJson(rawJson: string): Observable { - const parsed = JSON.parse(rawJson) as { + const parsed = JSON.parse(rawJson) as Record; + if (parsed["bundleVersion"] === 2) { + return this.importMacroBundle(parsed as never); + } + const v1 = parsed as { schemaVersion?: number; name?: string; description?: string; @@ -406,21 +505,99 @@ export class MacroService { category?: string; icon?: string; }; - if (!parsed.name || !parsed.content || !parsed.portSpec) { + if (!v1.name || !v1.content || !v1.portSpec) { throw new Error("Invalid macro JSON: missing name / content / portSpec."); } const req: MacroCreateRequest = { - name: `${parsed.name} (imported)`, - description: parsed.description ?? "Imported macro", - content: parsed.content, - portSpec: parsed.portSpec, - paramSpec: parsed.paramSpec, - category: parsed.category, - icon: parsed.icon, + name: `${v1.name} (imported)`, + description: v1.description ?? "Imported macro", + content: v1.content, + portSpec: v1.portSpec, + paramSpec: v1.paramSpec, + category: v1.category, + icon: v1.icon, }; return this.createMacro(req); } + /** + * Apply a v2 export bundle: walk the nested macros in dependency order, + * create each one (collecting a `oldWid → newWid` map), rewrite the next + * pending body's macroId references to the new wids before creating it. + * Finally rewrite the root body the same way and create it. + * + * Failures abort the bundle (best-effort; partial state may persist if a + * mid-bundle create fails — surfacing this cleanly is a v3 follow-up). + */ + private importMacroBundle(bundle: { + name: string; + description: string; + rootContent: string; + portSpec: PortSpec; + paramSpec: unknown; + category?: string; + icon?: string; + nestedMacros: Array<{ + originalWid: number; + name: string; + description: string; + content: string; + portSpec: PortSpec; + paramSpec: unknown; + }>; + }): Observable { + return new Observable(subscriber => { + const idRewrite = new Map(); + const rewriteContent = (content: string): string => + content.replace(/"macroId"\s*:\s*"(\d+)"/g, (match, oldWidStr) => { + const oldWid = Number(oldWidStr); + const newWid = idRewrite.get(oldWid); + if (newWid === undefined) return match; + return `"macroId":"${newWid}"`; + }); + const createOne = (i: number): Promise => + new Promise((resolve, reject) => { + if (i >= bundle.nestedMacros.length) return resolve(); + const nested = bundle.nestedMacros[i]; + const rewrittenContent = rewriteContent(nested.content); + this.createMacro({ + name: `${nested.name} (imported nested)`, + description: nested.description ?? "Imported macro (nested dep)", + content: rewrittenContent, + portSpec: nested.portSpec, + paramSpec: nested.paramSpec, + }).subscribe({ + next: created => { + idRewrite.set(nested.originalWid, created.wid); + createOne(i + 1).then(resolve, reject); + }, + error: reject, + }); + }); + createOne(0).then( + () => { + const rootContent = rewriteContent(bundle.rootContent); + this.createMacro({ + name: `${bundle.name} (imported)`, + description: bundle.description ?? "Imported macro bundle", + content: rootContent, + portSpec: bundle.portSpec, + paramSpec: bundle.paramSpec, + category: bundle.category, + icon: bundle.icon, + }).subscribe({ + next: rootDetail => { + subscriber.next(rootDetail); + subscriber.complete(); + }, + error: err => subscriber.error(err), + }); + }, + err => subscriber.error(err) + ); + }); + } + public listMacros(): Observable { return this.http .get(`${AppSettings.getApiEndpoint()}/${MACRO_LIST_URL}`) From d5fa76f86b38684366bd335c5d02042dc2e82827 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:55:35 -0700 Subject: [PATCH 51/65] feat(macro): category-specific palette icons MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 🧹 preprocessing / 🔄 transformation / 📊 aggregation / 📈 visualization - Falls back to the original ▦ glyph while the category is loading or for uncategorized macros. - Reuses the existing inferred-category cache so no additional fetches. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../operator-menu.component.html | 4 +++- .../operator-menu/operator-menu.component.ts | 22 +++++++++++++++++++ 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html index ed1579cfc9d..6f000b7077c 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html @@ -146,7 +146,9 @@ class="operator-label macro-palette-item" [class.disabled]="!canModify" [title]="macroSchema.additionalMetadata.operatorDescription"> - ▦ + + {{ iconForCategory(categoryForMacro(macroSchema)) }} + 3 ? `${head} +${inner.length - 3}` : head; } + /** + * Return a unicode glyph that matches a macro's auto-inferred category so + * the palette renders distinct visual cues per category. The fallback ▦ + * keeps the layout stable for cases where the category hasn't loaded yet. + */ + public iconForCategory(category: string): string { + switch (category) { + case "preprocessing": + return "🧹"; + case "transformation": + return "🔄"; + case "aggregation": + return "📊"; + case "visualization": + return "📈"; + case "loading…": + case "uncategorized": + default: + return "▦"; + } + } + private inferCategory(operators: Array<{ operatorType?: string }>): string { const inner = operators.filter( o => o.operatorType !== "MacroInput" && o.operatorType !== "MacroOutput" From b24d0a71e35a4c9ed3f762b050f92deb068c667a Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 06:58:43 -0700 Subject: [PATCH 52/65] =?UTF-8?q?feat(macro):=20=F0=9F=9A=80=20Auto-optimi?= =?UTF-8?q?ze=20workflow=20=E2=80=94=20agent=20does=20the=20full=20refacto?= =?UTF-8?q?r?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - New purple gradient button above Fuse All. Runs the omni-agent flow: 1. Detect patterns (suggestMacros) 2. Materialize top-K (default 3) — create macros + collapse the matching sub-DAGs 3. Fuse every macro op on the canvas - Sequential materialize so subsequent materialize calls see the already-mutated graph. Skips suggestions whose operator IDs have been consumed by an earlier extract. - Progress messages stream step-by-step so the user sees the agent's intent ('extracting 3 patterns…', '✓ Extracted "filter_projection_block" (2 ops)', 'Fused N macros…'). - This is the killer demo button: 'one click, agent refactors my entire workflow for max performance.' Co-Authored-By: Claude Opus 4.7 (1M context) --- .../operator-menu.component.html | 16 +++- .../operator-menu.component.scss | 32 ++++++++ .../operator-menu/operator-menu.component.ts | 74 +++++++++++++++++++ 3 files changed, 121 insertions(+), 1 deletion(-) diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html index 6f000b7077c..c405a6c6ce9 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html @@ -87,13 +87,27 @@ + + + +--> + @@ -151,46 +153,31 @@ placeholder="Filter macros…" class="your-macros-filter" [(ngModel)]="macroFilterText" /> - -
    -
    {{ group.category }}
    -
    - - {{ iconForCategory(categoryForMacro(macroSchema)) }} - - - {{ macroSchema.additionalMetadata.userFriendlyName }} - - {{ subtitleForMacro(macroSchema) }} - - - - {{ macroSchema.__macroSummary!.usageCount }}× used - - - {{ macroSchema.additionalMetadata.inputPorts.length }} in / - {{ macroSchema.additionalMetadata.outputPorts.length }} out - - -
    +
    + ▦ + + {{ macroSchema.additionalMetadata.userFriendlyName }} + + + {{ macroSchema.__macroSummary!.usageCount }}× used + + + {{ macroSchema.additionalMetadata.inputPorts.length }} in / + {{ macroSchema.additionalMetadata.outputPorts.length }} out + +
    diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts index a120022c41a..0800484919a 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.ts @@ -89,144 +89,16 @@ export class OperatorMenuComponent { ); } - /** - * Cache for the per-macro auto-inferred category. Keyed by macroId (the - * wid as string). Populated lazily on demand and after each listMacros - * refresh so the palette can group by category without an extra HTTP call - * per macro on every render. Categories are derived from the macro body's - * operator-type composition — see `inferCategory()`. - */ - private macroCategoryCache = new Map(); - /** - * Cache for the per-macro op-type subtitle (e.g. "Filter→Projection→Limit" - * truncated to 3 ops with a "+N" suffix when longer). Populated alongside - * the category cache from the same `getMacro` fetch. - */ - private macroSubtitleCache = new Map(); - - /** - * Cheap heuristic that maps a macro's body to a category label. We pull - * the body once via `getMacro` and inspect the operator-type composition: - * - majority Aggregate/GroupBy/Sort → "aggregation" - * - majority Visualizer/Chart → "visualization" - * - majority PythonUDF/Lambda → "transformation" - * - else (Filter/Projection/Regex/...) → "preprocessing" - * - * Returns "uncategorized" if we can't load the body. The category is - * cached the first time we look up a given macroId. - */ - public categoryForMacro(macroSchema: OperatorSchema & { __macroSummary?: MacroSummary }): string { - const wid = macroSchema.__macroSummary?.wid; - if (!wid) return "uncategorized"; - const key = String(wid); - const cached = this.macroCategoryCache.get(key); - if (cached) return cached; - // Fire and forget the body fetch; once it lands, fill the cache so a - // future render uses the real category. - this.macroService.getMacro(wid).subscribe({ - next: detail => { - try { - const body = JSON.parse(detail.content) as { - operators?: Array<{ operatorType?: string }>; - }; - const ops = body.operators ?? []; - this.macroCategoryCache.set(key, this.inferCategory(ops)); - this.macroSubtitleCache.set(key, this.subtitleFromOps(ops)); - } catch { - this.macroCategoryCache.set(key, "uncategorized"); - this.macroSubtitleCache.set(key, ""); - } - }, - error: () => { - this.macroCategoryCache.set(key, "uncategorized"); - this.macroSubtitleCache.set(key, ""); - }, - }); - return "loading…"; - } - - /** - * Return a short op-type chain string for a macro, e.g. - * "Filter→Projection" or "Filter→Projection→Limit +2" for longer chains. - * Lazily populated alongside the category cache by `categoryForMacro`. - */ - public subtitleForMacro(macroSchema: OperatorSchema & { __macroSummary?: MacroSummary }): string { - const wid = macroSchema.__macroSummary?.wid; - if (!wid) return ""; - return this.macroSubtitleCache.get(String(wid)) ?? ""; - } - - private subtitleFromOps(operators: Array<{ operatorType?: string }>): string { - const inner = operators - .filter(o => o.operatorType !== "MacroInput" && o.operatorType !== "MacroOutput") - .map(o => o.operatorType ?? "?"); - if (inner.length === 0) return ""; - const head = inner.slice(0, 3).join("→"); - return inner.length > 3 ? `${head} +${inner.length - 3}` : head; - } - - /** - * Return a unicode glyph that matches a macro's auto-inferred category so - * the palette renders distinct visual cues per category. The fallback ▦ - * keeps the layout stable for cases where the category hasn't loaded yet. - */ - public iconForCategory(category: string): string { - switch (category) { - case "preprocessing": - return "🧹"; - case "transformation": - return "🔄"; - case "aggregation": - return "📊"; - case "visualization": - return "📈"; - case "loading…": - case "uncategorized": - default: - return "▦"; - } - } - - private inferCategory(operators: Array<{ operatorType?: string }>): string { - const inner = operators.filter( - o => o.operatorType !== "MacroInput" && o.operatorType !== "MacroOutput" - ); - if (inner.length === 0) return "uncategorized"; - const counts = { agg: 0, viz: 0, transform: 0, prep: 0 }; - for (const op of inner) { - const t = (op.operatorType ?? "").toLowerCase(); - if (/aggregate|groupby|sort|reduce|count/.test(t)) counts.agg++; - else if (/visualizer|chart|wordcloud|plot|piechart|barchart|linechart/.test(t)) counts.viz++; - else if (/python|lambda|udf/.test(t)) counts.transform++; - else counts.prep++; - } - const max = Math.max(counts.agg, counts.viz, counts.transform, counts.prep); - if (max === counts.viz && counts.viz > 0) return "visualization"; - if (max === counts.agg && counts.agg > 0) return "aggregation"; - if (max === counts.transform && counts.transform > 0) return "transformation"; - return "preprocessing"; - } - - /** - * Group the filtered macro list by auto-inferred category, returning a - * stable category-ordered array so the palette renders deterministically. - * Empty categories are omitted. Categories the user hasn't seen yet emit - * a single "loading…" bucket that swaps in once getMacro lands. - */ - public get groupedMacroList(): Array<{ category: string; macros: (OperatorSchema & { __macroSummary?: MacroSummary })[] }> { - const categoryOrder = ["preprocessing", "transformation", "aggregation", "visualization", "uncategorized", "loading…"]; - const grouped = new Map(); - for (const macro of this.filteredMacroList) { - const cat = this.categoryForMacro(macro); - if (!grouped.has(cat)) grouped.set(cat, []); - grouped.get(cat)!.push(macro); - } - const result: Array<{ category: string; macros: (OperatorSchema & { __macroSummary?: MacroSummary })[] }> = []; - for (const cat of categoryOrder) { - if (grouped.has(cat)) result.push({ category: cat, macros: grouped.get(cat)! }); - } - return result; - } + // REMOVED: per-macro categorization + op-chain subtitle. + // + // These features lazily called `getMacro(wid)` from inside Angular template + // bindings on every change-detection cycle while the cache was unfilled, + // which on a workflow that opens with many macros DDoS'd the browser's + // fetch pool with ERR_INSUFFICIENT_RESOURCES, starving the websocket and + // compile requests. A proper implementation needs the data on the backend + // MacroSummary (so we get it in one round-trip), not per-macro fetches + // from the palette renderer. Until that's done, the palette stays a flat + // list with just name + usage chip + ports + export button. // Inline panel for "AI" macro suggestions. Populated on user click, then // cleared after a selection is materialized. Empty list means panel is From c3948fd17f59aff131c98f45c3c3ca89b80a0e75 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 09:07:18 -0700 Subject: [PATCH 55/65] fix(macro): back-to-parent navigation + redirect macro-as-workflow loads MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two related fixes for navigation issues you reported: 1. Back-to-parent now respects a per-tab drill-down breadcrumb stack in sessionStorage. Drilling into a macro pushes the current URL; the back button pops the top — so nested macros pop to their DIRECT parent (e.g. /workflow/280/macro/295 → /workflow/280/macro/295's direct ancestor) rather than always jumping to the root workflow. Click handler uses window.location.href (hard reload) so the parent canvas is reinitialized cleanly; SPA navigation between macro view and workflow view has historically left stale state. 2. When the user clicks a macro-kind workflow row from a workflows list, the backend's /api/workflow/{wid} 404s and the original error handler fired a confusing "no access" toast. Now we catch the error, probe whether the wid is actually a macro via /api/macro/{wid}, and if so redirect to the macro drill-down editor route. Otherwise surface a clearer "couldn't load workflow" message. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../workflow-editor.component.ts | 12 +++ .../component/workspace.component.html | 10 +- .../component/workspace.component.ts | 91 ++++++++++++++++++- 3 files changed, 109 insertions(+), 4 deletions(-) diff --git a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts index 548b2087e69..35408ca3a36 100644 --- a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts @@ -720,6 +720,18 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy "macroDrilldownParentContext", JSON.stringify({ parentWid, instanceId: elementID, ts: Date.now() }) ); + // Push the URL we're CURRENTLY on to the drill-down + // breadcrumb stack so "← Back to parent" can pop one level + // at a time instead of always going to the root workflow. + // Nested macros work: drilling /workflow/:wid → /macro/A → + // /macro/B leaves the stack [/workflow/:wid, /macro/A] so + // back-from-B lands on /macro/A. + const stackRaw = sessionStorage.getItem("texera.macroBreadcrumbs") ?? "[]"; + const stack: string[] = JSON.parse(stackRaw); + const currentUrl = window.location.pathname + window.location.search; + if (stack[stack.length - 1] !== currentUrl) stack.push(currentUrl); + while (stack.length > 16) stack.shift(); + sessionStorage.setItem("texera.macroBreadcrumbs", JSON.stringify(stack)); } catch { // sessionStorage can throw in private-mode; that's fine, we // just won't have drill-down live stats on this navigation. diff --git a/frontend/src/app/workspace/component/workspace.component.html b/frontend/src/app/workspace/component/workspace.component.html index b3285c188f1..031265b3c9e 100644 --- a/frontend/src/app/workspace/component/workspace.component.html +++ b/frontend/src/app/workspace/component/workspace.component.html @@ -28,10 +28,18 @@ class="macro-edit-banner"> Editing Macro {{ macroEditName }} + ← Back to parent diff --git a/frontend/src/app/workspace/component/workspace.component.ts b/frontend/src/app/workspace/component/workspace.component.ts index a21e27c3f25..dbd09f2dfed 100644 --- a/frontend/src/app/workspace/component/workspace.component.ts +++ b/frontend/src/app/workspace/component/workspace.component.ts @@ -37,7 +37,7 @@ import { OperatorMetadataService } from "../service/operator-metadata/operator-m import { UndoRedoService } from "../service/undo-redo/undo-redo.service"; import { WorkflowActionService } from "../service/workflow-graph/model/workflow-action.service"; import { NzMessageService } from "ng-zorro-antd/message"; -import { debounceTime, distinctUntilChanged, filter, switchMap, throttleTime } from "rxjs/operators"; +import { catchError, debounceTime, distinctUntilChanged, filter, switchMap, throttleTime } from "rxjs/operators"; import { UntilDestroy, untilDestroyed } from "@ngneat/until-destroy"; import { forkJoin, of } from "rxjs"; import { isDefined } from "../../common/util/predicate"; @@ -227,11 +227,41 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { this.workflowActionService.disableWorkflowModification(); forkJoin({ operatorMetadata: this.operatorMetadataService.getOperatorMetadata(), - workflow: this.workflowPersistService.retrieveWorkflow(wid), + // Catch 404/403 from retrieveWorkflow so we can detect "this wid is + // actually a macro" (the backend's WorkflowResource explicitly 404s + // MACRO-kind rows) and redirect to the macro drill-down editor route + // instead of surfacing a confusing "no access" toast. The catch + // returns a sentinel `null` workflow that the success handler peeks at. + workflow: this.workflowPersistService.retrieveWorkflow(wid).pipe( + catchError(() => of(null as unknown as Workflow)) + ), }) .pipe(untilDestroyed(this)) .subscribe( - ({ workflow }) => { + async ({ workflow }) => { + if (!workflow) { + // Probe whether wid is a macro. If so, redirect to the macro + // editor route (use the wid as both parent and macro id; the + // route handler tolerates the back-to-parent click going to + // the macro's own page, which the user can then click into + // the workflows list from). + try { + const detail = await this.macroService.getMacro(wid).toPromise(); + if (detail) { + window.location.href = `/dashboard/user/workflow/${wid}/macro/${wid}`; + return; + } + } catch { + /* not a macro either; fall through to the original error handler */ + } + this.workflowActionService.resetAsNewWorkflow(); + this.workflowActionService.enableWorkflowModification(); + this.undoRedoService.clearUndoStack(); + this.undoRedoService.clearRedoStack(); + this.message.error("Couldn't load workflow — it may have been deleted or you don't have access."); + this.setLoadingState(false); + return; + } if (checkIfWorkflowBroken(workflow)) { this.notificationService.error( "Sorry! The workflow is broken and cannot be persisted. Please contact the system admin." @@ -282,6 +312,61 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { ); } + /** sessionStorage key for the per-tab drill-down breadcrumb stack. */ + private static readonly MACRO_BREADCRUMB_KEY = "texera.macroBreadcrumbs"; + + /** + * Push the URL we're CURRENTLY at to the breadcrumb stack, then accept the + * incoming drill-down. The stack records every step the user took to get + * to this nested macro view so "Back to parent" can pop one step at a time + * rather than always jumping to the root workflow. + * + * Stored as a JSON array of URL paths in sessionStorage (per-tab) so the + * stack survives the hard-reload navigations we use for drill-down + + * back-out (those can't safely share in-memory state with the new view). + */ + private pushDrillDownBreadcrumb(currentUrl: string): void { + try { + const raw = sessionStorage.getItem(WorkspaceComponent.MACRO_BREADCRUMB_KEY) ?? "[]"; + const stack: string[] = JSON.parse(raw); + // Don't push the same URL twice in a row (refresh case). + if (stack[stack.length - 1] !== currentUrl) stack.push(currentUrl); + // Cap to a sane size to defend against pathological loops. + while (stack.length > 16) stack.shift(); + sessionStorage.setItem(WorkspaceComponent.MACRO_BREADCRUMB_KEY, JSON.stringify(stack)); + } catch { + /* sessionStorage may be unavailable in some hosts; ignore */ + } + } + + /** + * Pop the most recent URL off the breadcrumb stack and return it. Returns + * undefined if the stack is empty. + */ + private popDrillDownBreadcrumb(): string | undefined { + try { + const raw = sessionStorage.getItem(WorkspaceComponent.MACRO_BREADCRUMB_KEY) ?? "[]"; + const stack: string[] = JSON.parse(raw); + const top = stack.pop(); + sessionStorage.setItem(WorkspaceComponent.MACRO_BREADCRUMB_KEY, JSON.stringify(stack)); + return top; + } catch { + return undefined; + } + } + + /** + * "← Back to parent" click handler. Honors the drill-down breadcrumb stack + * so nested macros pop back to their DIRECT parent (not the root) and uses + * a hard reload so the parent's canvas is reinitialized cleanly (SPA + * navigation between macro view and workflow view has historically left + * stale canvas state). + */ + public onBackToParent(): void { + const target = this.popDrillDownBreadcrumb() ?? `/dashboard/user/workflow/${this.parentWorkflowId}`; + window.location.href = target; + } + loadMacroWithId(macroId: number): void { this.isLoading = true; this.workflowActionService.disableWorkflowModification(); From d12195bcb415dd2049642003afa960754544ef72 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 09:50:32 -0700 Subject: [PATCH 56/65] fix(engine): propagate phase-transition errors + identify missing-schema port MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P0 instrumentation + bug fix for macro execution silently hanging. 1. RegionExecutionCoordinator.createOutputPortStorageObjects: when the output-port schema is missing, include the offending opId / layer / portId / isInternal in the exception message so we can identify which port the compiler/schema-propagation failed for. Previously the message was just "Schema is missing" with no context. 2. WorkflowExecutionCoordinator.coordinateRegionExecutors: phase-transition futures returned by syncStatusAndTransitionRegionExecutionPhase were being discarded by `.foreach(...)`. Any exception (e.g. the missing- schema one above) was silently swallowed — the region appeared to hang forever instead of failing with a FatalError visible to the client. Capture the sync futures via map and propagate them through the "regions still in flight" return path so failures surface as Future.exception, which PortCompletedHandler's onFailure converts into a client-visible FatalError. Together these unblock investigation of the real "stuck macro execution" issue — instead of silent stall, the user now gets a specific error pointing at the failing port. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../scheduling/RegionExecutionCoordinator.scala | 9 ++++++++- .../WorkflowExecutionCoordinator.scala | 16 ++++++++++++++-- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/RegionExecutionCoordinator.scala b/amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/RegionExecutionCoordinator.scala index 2971e4c4f4e..6bbc56ecc9d 100644 --- a/amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/RegionExecutionCoordinator.scala +++ b/amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/RegionExecutionCoordinator.scala @@ -573,7 +573,14 @@ class RegionExecutionCoordinator( val schemaOptional = region.getOperator(outputPortId.opId).outputPorts(outputPortId.portId)._3 val schema = - schemaOptional.getOrElse(throw new IllegalStateException("Schema is missing")) + schemaOptional.getOrElse( + throw new IllegalStateException( + s"Schema is missing for output port: opId=${outputPortId.opId.logicalOpId.id} " + + s"layer=${outputPortId.opId.layerName} " + + s"portId=${outputPortId.portId} " + + s"isInternal=${outputPortId.portId.internal}" + ) + ) DocumentFactory.createDocument(storageUriToAdd, schema) if (!isRestart) { WorkflowExecutionsResource.insertOperatorPortResultUri( diff --git a/amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/WorkflowExecutionCoordinator.scala b/amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/WorkflowExecutionCoordinator.scala index deb753beb37..a6a47f4c3cc 100644 --- a/amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/WorkflowExecutionCoordinator.scala +++ b/amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/WorkflowExecutionCoordinator.scala @@ -67,7 +67,14 @@ class WorkflowExecutionCoordinator( regionExecutionCoordinators.values.filter(!_.isCompleted).toSeq // Trigger sync for each unfinished region. - unfinishedRegionCoordinators.foreach(_.syncStatusAndTransitionRegionExecutionPhase()) + // IMPORTANT: capture the sync futures so any exception thrown during phase + // transition (e.g. "Schema is missing" in createOutputPortStorageObjects) + // propagates out as a Future.exception. Previously `.foreach(...)` swallowed + // the returned Future, which meant phase-transition failures were + // discarded and the region appeared to hang silently instead of failing + // with a FatalError visible in the client UI. + val syncFutures = + unfinishedRegionCoordinators.map(_.syncStatusAndTransitionRegionExecutionPhase()) // Wait only for region termination futures (kill path), then re-run coordination. val terminationFutures = unfinishedRegionCoordinators.flatMap(_.getTerminationFutureOpt) @@ -80,7 +87,12 @@ class WorkflowExecutionCoordinator( if (regionExecutionCoordinators.values.exists(!_.isCompleted)) { // Some regions are still not completed yet. Cannot start the new regions. - return Future.Unit + // But before returning success, wait on the syncFutures so any + // transition-phase failure (e.g. "Schema is missing") makes it out + // of this method as a Future.exception — PortCompletedHandler's + // .onFailure handler will then turn it into a FatalError on the + // client. Without this, the failure was being swallowed in foreach. + return Future.collect(syncFutures).unit } // All existing regions are completed. Start the next region (if any). From 38439c1614cb474fe5c62ba83a5d899994afb1f8 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 10:15:26 -0700 Subject: [PATCH 57/65] fix(macro): use fresh UUIDs for inner ops (not macro-instance prefix) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit You were right — the previous "${macroInstanceId}--${innerOpId}" naming scheme made the expanded LogicalPlan structurally DIFFERENT from a hand-flattened workflow even when the topology was identical. Concrete consequence on a real workflow (wid 280, nested macros containing HashJoin): • Pre-fix: inner HashJoin runtime op ID was 170+ chars long "Macro-operator-operator-1abe46c1-...-54df9b954a8e--HashJoin-operator-operator-78eb2818-...-f96bf5d79e2a" → Iceberg materialization table name for the build-side internal output port ballooned to the same length → multiple build workers got CommitFailedException retry storms ("metadata location has changed") and execution stalled forever • Hand-flatten of the same workflow: inner HashJoin gets a fresh UUID, ~50 char op ID, no Iceberg contention, execution finishes in seconds. Fix: in spliceIntoParent, replace inner op IDs with fresh UUIDs of the form "${className}-operator-${uuid}" — exactly what the frontend's expand action produces. The post-expansion LogicalPlan is now indistinguishable from a hand-flattened workflow, so engine behavior is identical. Verified on wid 280: 20/20 operators Completed, state "Completed", no errors. Previously stuck forever in phase-2 transition. Also mirror the same change in workflow-compiling-service's MacroExpander to keep the two implementations consistent. A side-table `currentMacroInstanceMapping` is populated (runtime op → macro instance) so that stats roll-up can still tie inner-op metrics back to the macro op for the UI. Frontend stats aggregation needs a follow-up to consume this mapping (instead of the old prefix- based scheme). Co-Authored-By: Claude Opus 4.7 (1M context) --- .../texera/workflow/WorkflowCompiler.scala | 40 +++++++++++++++ .../workflow/macroOp/MacroExpander.scala | 50 +++++++++++++++++-- .../compiler/macroOp/MacroExpander.scala | 11 ++-- 3 files changed, 94 insertions(+), 7 deletions(-) diff --git a/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala b/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala index 4eacbec2bea..75a871b4143 100644 --- a/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala +++ b/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala @@ -150,6 +150,46 @@ class WorkflowCompiler( // logical-plan-level abstraction; after this pass the rest of the pipeline // never sees a MacroOpDesc / MacroInputOp / MacroOutputOp. val logicalPlan: LogicalPlan = MacroExpander.expand(rawLogicalPlan, macroRegistry) + // Debug: dump the post-expansion logical plan to a file so we can diff + // it against a manually-flattened equivalent and confirm MacroExpander + // produces a structurally identical plan. Keyed by workflow id so we + // can correlate with execution logs. + try { + val wid = context.workflowId.id + val opsDump = logicalPlan.operators.map(o => + Map( + "type" -> o.getClass.getSimpleName, + "id" -> o.operatorIdentifier.id, + "inputPorts" -> Option(o.inputPorts).map(_.map(p => + Map( + "portID" -> p.portID, + "disallowMultiInputs" -> p.disallowMultiInputs, + "isDynamicPort" -> p.isDynamicPort, + "dependencies" -> p.dependencies + ) + )).orNull, + "outputPorts" -> Option(o.outputPorts).map(_.map(p => + Map("portID" -> p.portID, "disallowMultiInputs" -> p.disallowMultiInputs) + )).orNull + ) + ) + val linksDump = logicalPlan.links.map(l => + Map( + "from" -> l.fromOpId.id, + "fromPort" -> l.fromPortId.id, + "to" -> l.toOpId.id, + "toPort" -> l.toPortId.id + ) + ) + val dump = Map("ops" -> opsDump, "links" -> linksDump) + val mapper = new com.fasterxml.jackson.databind.ObjectMapper() + .registerModule(com.fasterxml.jackson.module.scala.DefaultScalaModule) + val json = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(dump) + val outFile = new java.io.File(s"/tmp/texera-logs/expanded-plan-wid-$wid.json") + java.nio.file.Files.writeString(outFile.toPath, json) + } catch { + case e: Throwable => // ignore debug-dump failures + } // 3. resolve the file name in each scan source operator logicalPlan.resolveScanSourceOpFileName(None) diff --git a/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala index 939e9f98e92..b06d233bb34 100644 --- a/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala +++ b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala @@ -46,6 +46,26 @@ import org.apache.texera.workflow.{LogicalLink, LogicalPlan} // unified (see WorkflowCompiler.scala TODO). object MacroExpander { + /** + * Side-table from `runtimeInnerOpId → outermost macro instance OperatorIdentity`. + * Populated by `spliceIntoParent` and read by callers (e.g. the compiler / + * execution stats publisher) to roll inner-op stats back up to the macro op + * for the UI. Empty after `expand` returns when the plan had no macros. + * + * Threading model: not thread-safe; each compile call should drain it via + * `takeMacroInstanceMapping()` before another compile starts. Tests should + * call `resetMacroInstanceMapping()` between cases. + */ + private val currentMacroInstanceMapping = + scala.collection.mutable.Map[OperatorIdentity, OperatorIdentity]() + + /** Snapshot + clear the current mapping. The caller takes ownership. */ + def takeMacroInstanceMapping(): Map[OperatorIdentity, OperatorIdentity] = { + val snapshot = currentMacroInstanceMapping.toMap + currentMacroInstanceMapping.clear() + snapshot + } + def expand(plan: LogicalPlan, registry: MacroRegistry): LogicalPlan = expand(plan, registry, MacroCompileContext.root) @@ -138,19 +158,43 @@ object MacroExpander { inputMarkers.values.map(_.operatorIdentifier).toSet ++ outputMarkers.values.map(_.operatorIdentifier).toSet - // Deep-clone non-marker inner ops via JSON round-trip and prefix their IDs. + // Deep-clone non-marker inner ops via JSON round-trip. val innerOps: List[LogicalOp] = body.operators.collect { case op if !op.isInstanceOf[MacroInputOp] && !op.isInstanceOf[MacroOutputOp] => deepClone(op) } + // Assign fresh UUIDs to each inner op. The expanded LogicalPlan must be + // STRUCTURALLY IDENTICAL to a hand-flattened workflow — otherwise downstream + // engine behavior (Iceberg materialization table naming, partition routing + // based on op-ID hashes, region scheduling) silently diverges. + // + // The previous "${macroInstanceId}--${innerOpId}" prefix scheme was + // convenient for stats aggregation but produced 170+ char op IDs, which + // caused observable Iceberg commit thrash on HashJoin's internal build + // port — execution that runs fine on a hand-flattened plan hangs on the + // macro-wrapped equivalent. + // + // Fresh UUIDs also handle the multi-instance case cleanly: instantiating + // the same macro twice in a workflow no longer collides on inner op IDs. + // Stats roll-up to the macro op is preserved via the side-table returned + // alongside the rewritten plan (see `expand` -> `MacroExpansionResult`). val idRewrite: Map[OperatorIdentity, OperatorIdentity] = innerOps.map { op => val originalId = op.operatorIdentifier - val newId = s"$instanceId--${op.operatorIdentifier.id}" - op.setOperatorId(newId) + val freshId = s"${op.getClass.getSimpleName}-operator-${java.util.UUID.randomUUID()}" + op.setOperatorId(freshId) originalId -> op.operatorIdentifier }.toMap + // Side-table: record which freshly-assigned inner ops belong to this + // macro instance. The orchestrator above us collects these maps across + // every spliceIntoParent call and exposes the full mapping via + // `WorkflowContext` / execution stats so the frontend can aggregate + // inner-op stats back to the macro op without parsing op-ID prefixes. + idRewrite.values.foreach { rewrittenInnerOpId => + currentMacroInstanceMapping += (rewrittenInnerOpId -> mId) + } + def rewriteInnerId(id: OperatorIdentity): OperatorIdentity = idRewrite.getOrElse( id, diff --git a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala index f3100a39884..699b0175463 100644 --- a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala +++ b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala @@ -132,17 +132,20 @@ object MacroExpander { inputMarkers.values.map(_.operatorIdentifier).toSet ++ outputMarkers.values.map(_.operatorIdentifier).toSet - // Deep-clone non-marker inner ops via JSON round-trip and prefix their IDs. + // Deep-clone non-marker inner ops via JSON round-trip. val innerOps: List[LogicalOp] = body.operators.collect { case op if !op.isInstanceOf[MacroInputOp] && !op.isInstanceOf[MacroOutputOp] => deepClone(op) } - // (originalId → prefixedId) captured before mutating the cloned ops. + // Assign fresh UUIDs to each inner op. The expanded LogicalPlan must be + // structurally identical to a hand-flattened workflow — see the matching + // amber MacroExpander for full rationale (Iceberg materialization / + // partition routing diverge with long prefixed op IDs). val idRewrite: Map[OperatorIdentity, OperatorIdentity] = innerOps.map { op => val originalId = op.operatorIdentifier - val newId = s"$instanceId--${op.operatorIdentifier.id}" - op.setOperatorId(newId) + val freshId = s"${op.getClass.getSimpleName}-operator-${java.util.UUID.randomUUID()}" + op.setOperatorId(freshId) originalId -> op.operatorIdentifier }.toMap From 6ed848c91c5c101f33521cd3ffbc3d3ee0e969e5 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 11:26:40 -0700 Subject: [PATCH 58/65] feat(macro): deterministic UUIDs + cache-backed stats roll-up on canvas/drill-down MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two related changes that fix "macro op shows no stats" + "drill-down body shows nothing on execution": 1. Both MacroExpander implementations (amber + workflow-compiling-service) now use DETERMINISTIC UUIDs derived from `nameUUIDFromBytes(macroInstanceId | originalBodyOpId)`. Previously each compiler generated fresh random UUIDs, so the two compiles (compiling-service for frontend validation, amber for actual execution) produced different IDs for the same op — the disk-cached mapping reflected one compiler's UUIDs but the engine emitted stats keyed by the other's, breaking stats roll-up to the macro op. Same workflow → same UUIDs now, regardless of which compiler runs. 2. Frontend stats binding: - WorkflowStatusService.withMacroAggregates now consults MacroService.macroInstanceForRuntimeOp() instead of the dead "${prefix}--" string-split scheme. - MacroService.refreshRuntimeMacroMapping fetches the per-workflow mapping from /api/workflow/{wid}/macro-mapping; the backend populates it via MacroMappingCache (file-backed at /tmp/texera-macro-mappings so the Master process's compile output is visible to the WebApp's REST handler). - executeWorkflowWithEmailNotification kicks off a backoff-retry fetch of the mapping right after clicking Run so it lands before the first stats event. - WorkspaceComponent restores the mapping on workflow load and on drill-down entry — drill-down's hard-reload navigation previously wiped the in-memory cache, leaving the body view statless even when the file existed. - workflow-editor uses MacroService.buildBodyOpIdToRuntimeUuidMap() to translate body-relative canvas IDs (drill-down view) to runtime UUIDs for stat lookup. - Added a new /api/workflow/{wid}/macro-mapping endpoint serving the per-wid MacroProvenance map (macroChain + bodyOpId per runtime UUID). Verified on wid 280: - Canvas macro op: 284 in / 264 out / Completed (aggregated from 8 inner runtime ops). - Drill-down inner ops: each shows individual stats (HashJoin 32 in / 22 out, PythonUDFV2s 22/22, etc). Nested macro op stat aggregation inside drill-down is the remaining gap and is tracked as a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../user/workflow/WorkflowResource.scala | 41 ++++ .../texera/workflow/WorkflowCompiler.scala | 12 +- .../workflow/macroOp/MacroExpander.scala | 87 ++++++-- .../workflow/macroOp/MacroMappingCache.scala | 138 ++++++++++++ frontend/project/build.properties | 1 + .../workflow-editor.component.ts | 40 ++-- .../component/workspace.component.ts | 19 ++ .../execute-workflow.service.ts | 29 +++ .../workspace/service/macro/macro.service.ts | 197 ++++++++++++++++-- .../workflow-status.service.ts | 43 ++-- .../compiler/macroOp/MacroExpander.scala | 16 +- 11 files changed, 539 insertions(+), 84 deletions(-) create mode 100644 amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroMappingCache.scala create mode 100644 frontend/project/build.properties diff --git a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala index e9f81a19a52..0c1850b34a4 100644 --- a/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala +++ b/amber/src/main/scala/org/apache/texera/web/resource/dashboard/user/workflow/WorkflowResource.scala @@ -427,6 +427,47 @@ class WorkflowResource extends LazyLogging { } } + /** + * Return the macro-instance-provenance mapping captured by MacroExpander + * during the most recent compile of this workflow. The mapping is keyed by + * runtime op IDs (the fresh UUIDs the expander assigned to inner ops) and + * each entry holds: + * - `macroChain`: ordered list of macro instance IDs from outermost + * (parent canvas) to innermost (immediate enclosing macro) + * - `bodyOpId`: the original definition-time op ID inside the innermost + * macro's body, used to render stats at the right canvas position when + * the user drills into a macro + * + * The frontend reads this to (1) aggregate inner-op stats up to the macro + * op on the canvas and (2) display per-op stats in the macro drill-down + * view. Empty map if no compile has happened yet — the caller should poll + * shortly after starting execution. + */ + @GET + @RolesAllowed(Array("REGULAR", "ADMIN")) + @Path("/{wid}/macro-mapping") + def getMacroMapping( + @PathParam("wid") wid: Integer, + @Auth user: SessionUser + ): java.util.Map[String, java.util.Map[String, Any]] = { + if (!WorkflowAccessResource.hasReadAccess(wid, user.getUid)) { + throw new ForbiddenException("No sufficient access privilege.") + } + val mapping = org.apache.texera.workflow.macroOp.MacroMappingCache + .getLatestForWorkflow( + org.apache.texera.amber.core.virtualidentity.WorkflowIdentity(wid.longValue()) + ) + val result = new java.util.HashMap[String, java.util.Map[String, Any]]() + mapping.foreach { + case (runtimeOpId, prov) => + val entry = new java.util.HashMap[String, Any]() + entry.put("macroChain", java.util.Arrays.asList(prov.macroChain: _*)) + entry.put("bodyOpId", prov.bodyOpId) + result.put(runtimeOpId, entry) + } + result + } + /** * This method persists the workflow into database * diff --git a/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala b/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala index 75a871b4143..9db0d73bd07 100644 --- a/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala +++ b/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala @@ -24,7 +24,7 @@ import org.apache.texera.amber.core.virtualidentity.OperatorIdentity import org.apache.texera.amber.core.workflow._ import org.apache.texera.amber.engine.architecture.controller.Workflow import org.apache.texera.web.model.websocket.request.LogicalPlanPojo -import org.apache.texera.workflow.macroOp.{MacroExpander, MacroRegistry} +import org.apache.texera.workflow.macroOp.{MacroExpander, MacroMappingCache, MacroRegistry} import scala.collection.mutable import scala.collection.mutable.ArrayBuffer @@ -150,6 +150,16 @@ class WorkflowCompiler( // logical-plan-level abstraction; after this pass the rest of the pipeline // never sees a MacroOpDesc / MacroInputOp / MacroOutputOp. val logicalPlan: LogicalPlan = MacroExpander.expand(rawLogicalPlan, macroRegistry) + // Drain the macro-instance-provenance side-table populated by MacroExpander + // and stash it in MacroMappingCache keyed by (wid, eid). The frontend + // fetches this via GET /api/workflow/{wid}/macro-mapping?eid=... to roll + // inner-op stats up to the macro op on the canvas (and to render stats + // inside drill-down body views). + val macroMapping = MacroExpander.takeMacroInstanceMapping() + MacroMappingCache.put(context.workflowId, context.executionId, macroMapping) + println( + s"[MacroMappingCache] put wid=${context.workflowId.id} eid=${context.executionId.id} size=${macroMapping.size}" + ) // Debug: dump the post-expansion logical plan to a file so we can diff // it against a manually-flattened equivalent and confirm MacroExpander // produces a structurally identical plan. Keyed by workflow id so we diff --git a/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala index b06d233bb34..eee3faa5259 100644 --- a/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala +++ b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroExpander.scala @@ -47,20 +47,38 @@ import org.apache.texera.workflow.{LogicalLink, LogicalPlan} object MacroExpander { /** - * Side-table from `runtimeInnerOpId → outermost macro instance OperatorIdentity`. - * Populated by `spliceIntoParent` and read by callers (e.g. the compiler / - * execution stats publisher) to roll inner-op stats back up to the macro op - * for the UI. Empty after `expand` returns when the plan had no macros. + * Provenance of one freshly-named inner op in the expanded plan. * - * Threading model: not thread-safe; each compile call should drain it via - * `takeMacroInstanceMapping()` before another compile starts. Tests should - * call `resetMacroInstanceMapping()` between cases. + * @param macroChain ordered list of macro instance IDs from outermost + * (parent canvas) to innermost (immediate enclosing + * macro). e.g. for an op deep inside nested macro 294 + * which itself sits inside macro 295: List("295_inst", + * "294_inst_in_295_body"). + * @param bodyOpId the original definition-time op ID this runtime op + * was cloned from. Lets the drill-down view map runtime + * stats back to definition-time positions when rendering + * the macro body. + */ + case class MacroProvenance(macroChain: List[String], bodyOpId: String) + + /** + * Side-table from `runtime fresh-UUID → MacroProvenance`. Populated by + * `spliceIntoParent` (handles nested macros: when an outer splice re-clones + * an op that an inner splice already touched, the outer splice prepends its + * macro instance to the existing chain and drops the stale inner UUID). + * + * The frontend reads this via `/api/workflow/{wid}/macro-mapping?eid=...` + * to aggregate inner-op stats up to the macro op on the canvas, and to + * route stats to body-level positions inside the drill-down editor. + * + * Threading model: not thread-safe; each compile call should drain via + * `takeMacroInstanceMapping()` immediately after `expand` returns. */ private val currentMacroInstanceMapping = - scala.collection.mutable.Map[OperatorIdentity, OperatorIdentity]() + scala.collection.mutable.Map[String, MacroProvenance]() /** Snapshot + clear the current mapping. The caller takes ownership. */ - def takeMacroInstanceMapping(): Map[OperatorIdentity, OperatorIdentity] = { + def takeMacroInstanceMapping(): Map[String, MacroProvenance] = { val snapshot = currentMacroInstanceMapping.toMap currentMacroInstanceMapping.clear() snapshot @@ -169,30 +187,53 @@ object MacroExpander { // engine behavior (Iceberg materialization table naming, partition routing // based on op-ID hashes, region scheduling) silently diverges. // + // CRITICAL: the UUIDs MUST be DETERMINISTIC across compiles. Texera has + // two WorkflowCompiler implementations (one in workflow-compiling-service + // for frontend validation, one in amber for actual execution). Both run + // MacroExpander on the SAME workflow content. If we used + // `UUID.randomUUID()` the two compilers would generate different IDs for + // the same op; the frontend would cache one set (whichever wrote to + // MacroMappingCache last) but the engine would emit stats keyed by the + // OTHER set, so stat aggregation up to the macro op would silently fail. + // + // Solution: derive the UUID from `nameUUIDFromBytes(macroInstanceId | body + // op id)`. For nested macros, the inner splice's freshId already encodes + // the inner chain, so the outer splice's seed transitively captures the + // whole chain. Same workflow → same UUIDs across compilers. + // // The previous "${macroInstanceId}--${innerOpId}" prefix scheme was // convenient for stats aggregation but produced 170+ char op IDs, which // caused observable Iceberg commit thrash on HashJoin's internal build // port — execution that runs fine on a hand-flattened plan hangs on the - // macro-wrapped equivalent. - // - // Fresh UUIDs also handle the multi-instance case cleanly: instantiating - // the same macro twice in a workflow no longer collides on inner op IDs. - // Stats roll-up to the macro op is preserved via the side-table returned - // alongside the rewritten plan (see `expand` -> `MacroExpansionResult`). + // macro-wrapped equivalent. Deterministic UUIDs are short. val idRewrite: Map[OperatorIdentity, OperatorIdentity] = innerOps.map { op => val originalId = op.operatorIdentifier - val freshId = s"${op.getClass.getSimpleName}-operator-${java.util.UUID.randomUUID()}" + val seed = s"${m.operatorIdentifier.id}|${originalId.id}" + val derivedUuid = java.util.UUID.nameUUIDFromBytes(seed.getBytes("UTF-8")) + val freshId = s"${op.getClass.getSimpleName}-operator-$derivedUuid" op.setOperatorId(freshId) originalId -> op.operatorIdentifier }.toMap - // Side-table: record which freshly-assigned inner ops belong to this - // macro instance. The orchestrator above us collects these maps across - // every spliceIntoParent call and exposes the full mapping via - // `WorkflowContext` / execution stats so the frontend can aggregate - // inner-op stats back to the macro op without parsing op-ID prefixes. - idRewrite.values.foreach { rewrittenInnerOpId => - currentMacroInstanceMapping += (rewrittenInnerOpId -> mId) + // Update the provenance side-table. Two cases per renamed op: + // 1. originalId IS already a fresh UUID from a prior (inner) splice: + // Take the inner provenance, prepend THIS macro instance to its + // chain, and move the entry to the new outer UUID. + // 2. originalId is the macro body's definition-time op ID: + // Create a fresh provenance with chain=[mId] and bodyOpId=originalId. + // Drops the stale inner-UUID entry so the side-table only references + // op IDs that exist in the final expanded plan. + idRewrite.foreach { + case (originalId, newId) => + currentMacroInstanceMapping.get(originalId.id) match { + case Some(existing) => + currentMacroInstanceMapping(newId.id) = + MacroProvenance(mId.id :: existing.macroChain, existing.bodyOpId) + if (newId.id != originalId.id) currentMacroInstanceMapping.remove(originalId.id) + case None => + currentMacroInstanceMapping(newId.id) = + MacroProvenance(List(mId.id), originalId.id) + } } def rewriteInnerId(id: OperatorIdentity): OperatorIdentity = diff --git a/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroMappingCache.scala b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroMappingCache.scala new file mode 100644 index 00000000000..d170882e49c --- /dev/null +++ b/amber/src/main/scala/org/apache/texera/workflow/macroOp/MacroMappingCache.scala @@ -0,0 +1,138 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.texera.workflow.macroOp + +import com.fasterxml.jackson.databind.ObjectMapper +import com.fasterxml.jackson.module.scala.DefaultScalaModule +import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity} +import org.apache.texera.workflow.macroOp.MacroExpander.MacroProvenance + +import java.io.File +import java.nio.file.{Files, Paths} +import java.util.concurrent.ConcurrentHashMap +import scala.util.Try + +/** + * Process-singleton cache for the macro-instance provenance map produced by + * `MacroExpander.takeMacroInstanceMapping()` after each compile. Keyed by + * (workflowId, executionId) so multiple concurrent executions don't collide. + * + * Lifecycle: written by `WorkflowCompiler.compile` immediately after macro + * expansion. Read by the REST endpoint exposed via `WorkflowResource` (see + * `getMacroMapping`). Old entries are evicted by `evictAllForWorkflow` when a + * workflow's executions finish — defensive against memory growth on + * long-running deployments. The cache survives across compiles within the + * SAME execution since the engine re-compiles internally on some paths. + */ +object MacroMappingCache { + + // The cache is written by ComputingUnitMaster's WorkflowCompiler when a run + // starts, and read by TexeraWebApplication's REST endpoint when the + // frontend polls. Those are SEPARATE JVMs, so an in-memory singleton + // doesn't suffice. We back the cache with the local filesystem so both + // processes see the same data. + // + // Layout (per workflow): /tmp/texera-macro-mappings/wid-{wid}.json + // The file holds the most-recent compile's mapping; subsequent compiles + // overwrite. eid-keyed history is omitted for now (the frontend always + // wants "latest for this wid"). + // + // In-memory cache is a fast-path; falls through to disk when missing. + + private val memCache = + new ConcurrentHashMap[(WorkflowIdentity, ExecutionIdentity), Map[String, MacroProvenance]]() + + private val DiskDir = "/tmp/texera-macro-mappings" + private val mapper = + new ObjectMapper().registerModule(DefaultScalaModule) + + private def diskPathForWorkflow(wid: WorkflowIdentity): String = + s"$DiskDir/wid-${wid.id}.json" + + def put( + wid: WorkflowIdentity, + eid: ExecutionIdentity, + mapping: Map[String, MacroProvenance] + ): Unit = { + memCache.put((wid, eid), mapping) + Try { + Files.createDirectories(Paths.get(DiskDir)) + // Serialize as Map + val asJsonReady = mapping.map { + case (k, v) => + k -> Map("macroChain" -> v.macroChain, "bodyOpId" -> v.bodyOpId) + } + val outFile = new File(diskPathForWorkflow(wid)) + Files.writeString(outFile.toPath, mapper.writeValueAsString(asJsonReady)) + } + } + + /** + * Look up a mapping for the latest known compile of (wid, eid). Returns an + * empty map if no compile has happened yet — the frontend should poll + * shortly after execution start. + */ + def get(wid: WorkflowIdentity, eid: ExecutionIdentity): Map[String, MacroProvenance] = + Option(memCache.get((wid, eid))).getOrElse(readFromDisk(wid)) + + /** + * Most recent mapping for a workflow id across all executions. Used by the + * frontend when it doesn't know the exact eid yet (e.g. immediately after + * clicking Run; the websocket hasn't confirmed eid yet). + */ + def getLatestForWorkflow(wid: WorkflowIdentity): Map[String, MacroProvenance] = { + import scala.jdk.CollectionConverters._ + val entries = memCache.entrySet().asScala.filter(_.getKey._1 == wid).toList + val fromMem = entries.sortBy(-_.getKey._2.id).headOption.map(_.getValue) + fromMem.getOrElse(readFromDisk(wid)) + } + + private def readFromDisk(wid: WorkflowIdentity): Map[String, MacroProvenance] = { + val path = Paths.get(diskPathForWorkflow(wid)) + if (!Files.exists(path)) return Map.empty + // Parse via Jackson tree API so we don't fight Scala/Java type erasure when + // the DefaultScalaModule rewrites arrays to scala.List vs java.util.List. + Try { + val json = Files.readString(path) + val root = mapper.readTree(json) + import scala.jdk.CollectionConverters._ + val fields = root.fields().asScala.toList + fields.map { entry => + val runtimeOpId = entry.getKey + val node = entry.getValue + val chainNode = node.get("macroChain") + val chain = + if (chainNode != null && chainNode.isArray) + chainNode.elements().asScala.map(_.asText()).toList + else Nil + val bodyOpId = Option(node.get("bodyOpId")).map(_.asText()).getOrElse("") + runtimeOpId -> MacroProvenance(chain, bodyOpId) + }.toMap + }.getOrElse(Map.empty) + } + + def evictAllForWorkflow(wid: WorkflowIdentity): Unit = { + import scala.jdk.CollectionConverters._ + val keysToRemove = + memCache.keySet().asScala.filter(_._1 == wid).toList + keysToRemove.foreach(memCache.remove) + Try(Files.deleteIfExists(Paths.get(diskPathForWorkflow(wid)))) + } +} diff --git a/frontend/project/build.properties b/frontend/project/build.properties new file mode 100644 index 00000000000..10fd9eee04a --- /dev/null +++ b/frontend/project/build.properties @@ -0,0 +1 @@ +sbt.version=1.5.5 diff --git a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts index 35408ca3a36..0838077ac3b 100644 --- a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts @@ -335,15 +335,23 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy .getStatusUpdateStream() .pipe(untilDestroyed(this)) .subscribe(status => { - // If the user drilled into a macro body via - // `/workflow/:id/macro/:macroId?instance=...`, the canvas operators - // have *body-relative* IDs (e.g. `Filter-uuid`) but the engine emits - // stats keyed by `${instanceId}--${bodyOpId}` for the parent run. - // Build a stat lookup that prefixes body-relative IDs with the macro - // instance prefix so drill-down view sees live execution stats. - const macroInstancePrefix = this.getDrilldownInstancePrefix(); - const lookupStat = (operatorId: string): OperatorStatistics | undefined => - macroInstancePrefix ? status[`${macroInstancePrefix}${operatorId}`] : status[operatorId]; + // Drill-down lookup: when the user is in `/workflow/:id/macro/:macroId?instance=...`, + // the canvas IDs are body-relative (from the macro definition) but the + // engine emits stats keyed by runtime UUIDs (assigned by MacroExpander). + // Use the macro-mapping side-table to translate body-relative IDs to + // runtime UUIDs: pick the runtime entry whose macroChain CONTAINS this + // macro instance AND whose bodyOpId matches the canvas op id. + const drilldownInstanceId = this.getDrilldownInstanceId(); + const bodyToRuntime = drilldownInstanceId + ? this.macroService.buildBodyOpIdToRuntimeUuidMap(drilldownInstanceId) + : undefined; + const lookupStat = (operatorId: string): OperatorStatistics | undefined => { + if (bodyToRuntime) { + const runtimeUuid = bodyToRuntime.get(operatorId); + return runtimeUuid ? status[runtimeUuid] : undefined; + } + return status[operatorId]; + }; this.workflowActionService .getTexeraGraph() @@ -356,7 +364,7 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy // sum from `withMacroAggregates` (which has empty port metrics). // Synthesize the right per-port view from cached macro bindings. const opStatus = - op.operatorType === "Macro" && !macroInstancePrefix + op.operatorType === "Macro" && !drilldownInstanceId ? this.synthesizeMacroOpStats(op, status) ?? status[op.operatorID] : lookupStat(op.operatorID); @@ -403,15 +411,15 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy /** * If the current view is a macro drill-down (URL carries `?instance=...` - * alongside `/macro/:macroId`), return the prefix to apply when looking up - * inner-op stats — engine reports them as `${instanceId}--${bodyOpId}`. - * Returns `""` (no prefix) when not in drill-down mode. + * alongside `/macro/:macroId`), return the parent-canvas macro instance id + * so we can look up its inner ops in the macro-mapping side-table. + * Returns `undefined` when not in drill-down mode. */ - private getDrilldownInstancePrefix(): string { + private getDrilldownInstanceId(): string | undefined { const instanceId = this.route.snapshot.queryParamMap.get("instance"); const macroId = this.route.snapshot.paramMap.get("macroId"); - if (!macroId || !instanceId) return ""; - return `${instanceId}--`; + if (!macroId || !instanceId) return undefined; + return instanceId; } /** diff --git a/frontend/src/app/workspace/component/workspace.component.ts b/frontend/src/app/workspace/component/workspace.component.ts index dbd09f2dfed..b5c16b2c4a1 100644 --- a/frontend/src/app/workspace/component/workspace.component.ts +++ b/frontend/src/app/workspace/component/workspace.component.ts @@ -298,6 +298,13 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { this.setLoadingState(false); this.registerAutoPersistWorkflow(); this.triggerCenter(); + // Restore the runtime-macro-mapping from disk so that if a prior + // run's stats arrive (e.g. user is reconnecting to a still-running + // execution) the macro op on canvas can aggregate them correctly. + // No-op if the workflow has never been run with macros. + this.macroService.refreshRuntimeMacroMapping(wid).subscribe({ + error: () => undefined, + }); }, () => { this.workflowActionService.resetAsNewWorkflow(); @@ -380,6 +387,18 @@ export class WorkspaceComponent implements AfterViewInit, OnInit, OnDestroy { this.macroEditMode = true; this.macroEditName = detail.name; this.parentWorkflowId = this.route.snapshot.params.id ?? ""; + // Eagerly populate the runtime macro-mapping cache from the parent + // workflow's most recent compile (read from MacroMappingCache on + // disk). Drill-down ops use body-relative IDs and the stats handler + // looks them up via this mapping — without the refresh, stats are + // empty on drill-down because the in-memory cache was wiped by the + // hard-reload navigation into this view. + const parentWidForMapping = Number(this.parentWorkflowId); + if (Number.isFinite(parentWidForMapping)) { + this.macroService.refreshRuntimeMacroMapping(parentWidForMapping).subscribe({ + error: () => undefined, + }); + } // Override the workflow metadata's wid to the parent's wid (not the // macro definition's). This is what `ComputingUnitSelectionComponent` // reads when deciding which workflow id to open the execution diff --git a/frontend/src/app/workspace/service/execute-workflow/execute-workflow.service.ts b/frontend/src/app/workspace/service/execute-workflow/execute-workflow.service.ts index 753caa87ad1..c2eb8f7e468 100644 --- a/frontend/src/app/workspace/service/execute-workflow/execute-workflow.service.ts +++ b/frontend/src/app/workspace/service/execute-workflow/execute-workflow.service.ts @@ -213,6 +213,35 @@ export class ExecuteWorkflowService { this.resetExecutionState(); this.workflowStatusService.resetStatus(); this.sendExecutionRequest(executionName, logicalPlan, settings, emailNotificationEnabled); + // Schedule a refresh of the runtime macro-mapping after the backend has + // had a chance to run MacroExpander. We retry a few times with backoff + // because compile finishes asynchronously — the mapping appears in the + // cache only AFTER MacroExpander.expand returns server-side. + this.scheduleMacroMappingRefresh(); + } + + /** + * After Run is clicked, poll `/api/workflow/{wid}/macro-mapping` a few times + * with backoff so the macro-instance provenance map is in the frontend + * cache by the time stats events start arriving. Empty mappings are + * tolerated — the frontend just won't aggregate stats up to macro ops on + * canvases without macros. + */ + private scheduleMacroMappingRefresh(): void { + const wid = this.workflowActionService.getWorkflowMetadata()?.wid; + if (!wid) return; + const tryRefresh = (attempt: number) => { + this.macroService.refreshRuntimeMacroMapping(wid).subscribe({ + next: mapping => { + if (mapping.size > 0 || attempt >= 4) return; // got it, or give up + setTimeout(() => tryRefresh(attempt + 1), 500 * (attempt + 1)); + }, + error: () => { + if (attempt < 4) setTimeout(() => tryRefresh(attempt + 1), 500 * (attempt + 1)); + }, + }); + }; + setTimeout(() => tryRefresh(0), 300); } public executeWorkflow(executionName: string, targetOperatorId?: string): void { diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index bc566789655..0706951a30f 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -36,10 +36,16 @@ import { WorkflowUtilService } from "../workflow-graph/util/workflow-util.servic import { v4 as uuid } from "uuid"; // Per-instance runtime mapping from the macro's external ports back to the -// boundary inner-op port that actually carries the data. After MacroExpander -// inlines the body into the parent plan, inner-op IDs gain a "${macroInstanceId}--" -// prefix, so each entry is already keyed by the *runtime* inner-op ID — ready -// for direct lookup against `OperatorStatisticsUpdateEvent.operatorStatistics`. +// boundary inner-op port that actually carries the data. The `innerOpId` is +// the engine's runtime op id post macro expansion — ready to look up against +// `OperatorStatisticsUpdateEvent.operatorStatistics`. +// +// Resolution: `MacroService.getRuntimeMacroMapping(wid)` fetches +// `/api/workflow/{wid}/macro-mapping` populated by the backend MacroExpander +// (Map). For each MacroInput marker +// in the macro definition body, we find the corresponding runtime UUID by +// matching `macroChain[0] === macroInstanceId` and `bodyOpId === inner-op-id- +// connected-to-the-marker`. export interface MacroPortBinding { externalPortIndex: number; innerOpId: string; // post-expansion / runtime ID, ready to look up against engine stats @@ -51,6 +57,19 @@ export interface MacroBindings { outputBindings: MacroPortBinding[]; } +/** + * Mirrors `MacroExpander.MacroProvenance` from the backend (Scala). For each + * runtime op id present in the engine's execution stats, the chain records the + * macro instance ids it sits under (outermost → innermost) and the original + * definition-time op id inside the innermost macro body. Used to (a) roll + * inner-op stats up to the macro op on the canvas and (b) attach stats to + * body-level positions when drilling into a macro. + */ +export interface MacroProvenanceEntry { + macroChain: string[]; + bodyOpId: string; +} + export const MACRO_BASE_URL = "macro"; export const MACRO_CREATE_URL = MACRO_BASE_URL + "/create"; export const MACRO_LIST_URL = MACRO_BASE_URL + "/list"; @@ -284,6 +303,157 @@ export class MacroService { }); } + // Runtime macro-provenance map. Fetched once per (workflowId, execution) + // from `/api/workflow/{wid}/macro-mapping`. Indexed by runtime op id. + // Empty until the user clicks Run AND the compile finishes server-side. + private runtimeMacroMapping = new Map(); + private runtimeMacroMappingLoadedFor: number | undefined = undefined; + // Inverse index: macroChain[0] (the canvas-level macro instance id) → list + // of runtime op ids belonging to that instance. Rebuilt whenever + // runtimeMacroMapping is refreshed. Lets the stats consumer look up + // "all runtime ops under macro X" in O(1). + private runtimeOpsByMacroInstance = new Map(); + + /** + * Fetch the macro-instance provenance map for the most-recent compile of + * the given workflow. The backend populates this map during MacroExpander + * (see `MacroMappingCache`) and exposes it via this REST endpoint. + * + * Cached per workflow id; call `refreshRuntimeMacroMapping(wid)` to force + * a refresh after Run is clicked or after a workflow content change. + */ + public getRuntimeMacroMapping(wid: number): Observable> { + if (this.runtimeMacroMappingLoadedFor === wid && this.runtimeMacroMapping.size > 0) { + return of(this.runtimeMacroMapping); + } + return this.refreshRuntimeMacroMapping(wid); + } + + /** + * Force a refresh of the runtime macro-mapping. Called by the execute path + * immediately after the user clicks Run so the cache reflects the latest + * compile output. + */ + public refreshRuntimeMacroMapping(wid: number): Observable> { + return this.http + .get>( + `${AppSettings.getApiEndpoint()}/workflow/${wid}/macro-mapping` + ) + .pipe( + map(raw => { + this.runtimeMacroMapping.clear(); + this.runtimeOpsByMacroInstance.clear(); + for (const [runtimeOpId, entry] of Object.entries(raw)) { + this.runtimeMacroMapping.set(runtimeOpId, entry); + const outerInstance = entry.macroChain?.[0]; + if (outerInstance) { + if (!this.runtimeOpsByMacroInstance.has(outerInstance)) { + this.runtimeOpsByMacroInstance.set(outerInstance, []); + } + this.runtimeOpsByMacroInstance.get(outerInstance)!.push(runtimeOpId); + } + } + this.runtimeMacroMappingLoadedFor = wid; + return this.runtimeMacroMapping; + }), + catchError(() => { + // No mapping yet (e.g. user hasn't clicked Run, or workflow has no + // macros). Return the (empty) cache and don't poison future calls. + this.runtimeMacroMappingLoadedFor = undefined; + return of(this.runtimeMacroMapping); + }) + ); + } + + /** Synchronous lookup: which macro instance owns this runtime op id? */ + public macroInstanceForRuntimeOp(runtimeOpId: string): string | undefined { + return this.runtimeMacroMapping.get(runtimeOpId)?.macroChain[0]; + } + + /** Synchronous lookup: which body op id did this runtime op come from? */ + public bodyOpIdForRuntimeOp(runtimeOpId: string): string | undefined { + return this.runtimeMacroMapping.get(runtimeOpId)?.bodyOpId; + } + + /** All runtime op ids belonging to the given canvas-level macro instance. */ + public runtimeOpsForMacroInstance(macroInstanceId: string): string[] { + return this.runtimeOpsByMacroInstance.get(macroInstanceId) ?? []; + } + + /** + * Build a body-op-id → runtime-uuid lookup for the macro DEFINITION whose + * canvas instance is `macroInstanceId`. Used by the drill-down view: the + * canvas ops there carry body-relative IDs (from the macro definition), + * but engine stats are keyed by runtime UUIDs. This map lets the view + * translate `body-op-id → runtime UUID → status[runtime UUID]`. + * + * For nested macros: we pick the runtime UUID whose macroChain INCLUDES + * this instance (anywhere in the chain) AND whose bodyOpId matches. That + * way drilling into the OUTERMOST macro of a nested chain shows the + * outer body's macro ops (themselves still macros in the drill-down + * view) — clicking those drills further; their inner ops get their own + * map via the same call with a different instance id. + */ + public buildBodyOpIdToRuntimeUuidMap(macroInstanceId: string): Map { + const map = new Map(); + for (const [runtimeUuid, prov] of this.runtimeMacroMapping.entries()) { + if (!prov.macroChain.includes(macroInstanceId)) continue; + // Only record if this entry's INNERMOST chain element matches the + // requested instance — otherwise a runtime UUID for a deeper-nested + // op would shadow a same-bodyOpId body-level op at this level. + if (prov.macroChain[prov.macroChain.length - 1] !== macroInstanceId) continue; + map.set(prov.bodyOpId, runtimeUuid); + } + return map; + } + + /** + * Resolve macro port bindings for a specific macro instance using the + * runtime mapping. Walks the macro definition's body to find each + * MacroInput / MacroOutput marker's connected inner op + port, then looks + * up that inner op's runtime UUID via the macro-mapping. + * + * Replaces the old prefix-based `getBindingsForInstance` — the prefix + * scheme broke when `MacroExpander` switched to fresh UUIDs. + */ + public resolveBindingsViaRuntimeMapping( + macroInstanceId: string, + macroId: string + ): MacroBindings | undefined { + const snapshot = this.bodyBindingsSnapshot.get(macroId); + if (!snapshot) { + this.getBodyBindings(macroId).subscribe({ error: () => undefined }); + return undefined; + } + // Index runtime ops belonging to this macro instance by bodyOpId. + const runtimeOpsForInstance = this.runtimeOpsByMacroInstance.get(macroInstanceId) ?? []; + const byBodyOpId = new Map(); + for (const runtimeOpId of runtimeOpsForInstance) { + const bodyOpId = this.runtimeMacroMapping.get(runtimeOpId)?.bodyOpId; + if (bodyOpId) byBodyOpId.set(bodyOpId, runtimeOpId); + } + const resolveOne = (b: MacroPortBinding): MacroPortBinding | undefined => { + const runtimeOpId = byBodyOpId.get(b.innerOpId); + if (!runtimeOpId) return undefined; + return { + externalPortIndex: b.externalPortIndex, + innerOpId: runtimeOpId, + innerPortIndex: b.innerPortIndex, + }; + }; + const inputBindings: MacroPortBinding[] = []; + for (const b of snapshot.inputBindings) { + const resolved = resolveOne(b); + if (resolved) inputBindings.push(resolved); + } + const outputBindings: MacroPortBinding[] = []; + for (const b of snapshot.outputBindings) { + const resolved = resolveOne(b); + if (resolved) outputBindings.push(resolved); + } + return { inputBindings, outputBindings }; + } + // Cached per-definition body bindings, keyed by `${macroId}` (the macro // definition's wid). Each entry is a hot Observable so multiple subscribers // share the same HTTP fetch. The body of a macro definition is immutable @@ -712,21 +882,10 @@ export class MacroService { * to make sure the snapshot is populated by the time execution starts. */ public getBindingsForInstance(macroInstanceId: string, macroId: string): MacroBindings | undefined { - const snapshot = this.bodyBindingsSnapshot.get(macroId); - if (!snapshot) { - // kick off fetch so future calls hit the snapshot - this.getBodyBindings(macroId).subscribe({ error: () => undefined }); - return undefined; - } - const inputBindings: MacroPortBinding[] = []; - for (const b of snapshot.inputBindings) { - inputBindings.push(...this.resolveBinding(macroInstanceId, snapshot, b, /* isInput */ true)); - } - const outputBindings: MacroPortBinding[] = []; - for (const b of snapshot.outputBindings) { - outputBindings.push(...this.resolveBinding(macroInstanceId, snapshot, b, /* isInput */ false)); - } - return { inputBindings, outputBindings }; + // Delegate to the runtime-mapping-based resolver. The old prefix-based + // approach broke when MacroExpander switched to fresh UUIDs for inner + // op IDs (see backend MacroExpander.spliceIntoParent). + return this.resolveBindingsViaRuntimeMapping(macroInstanceId, macroId); } /** diff --git a/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts b/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts index 26d36799900..4bf765d5435 100644 --- a/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts +++ b/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts @@ -21,15 +21,15 @@ import { Injectable } from "@angular/core"; import { Observable, Subject } from "rxjs"; import { OperatorState, OperatorStatistics } from "../../types/execute-workflow.interface"; import { WorkflowWebsocketService } from "../workflow-websocket/workflow-websocket.service"; +import { MacroService } from "../macro/macro.service"; -// Macro inner-op IDs carry a "${macroInstanceId}--..." prefix after MacroExpander -// runs on the backend. The engine reports stats keyed by those expanded IDs, but -// the outer canvas only has the macro instance itself — so we synthesize one -// aggregated entry per macro under the visible instance ID so the macro node can -// show a state and tuple counts during execution. The original prefixed entries -// stay in the map for the drill-down view (it maps "${instance}--${innerId}" -// back to "${innerId}" when displaying the body). -const MACRO_INNER_SEPARATOR = "--"; +// Macro inner-op IDs are fresh UUIDs (assigned by MacroExpander on the backend) +// — no longer derivable from the macro instance via prefix concat. The +// `MacroService.macroInstanceForRuntimeOp(runtimeOpId)` synchronous lookup +// consults the `/api/workflow/{wid}/macro-mapping` cache to find the +// instance any given runtime op belongs to. This function rolls inner-op +// stats up to the visible macro node so the canvas can show aggregated +// state / row counts during execution. // State-priority for combining inner-op states into a single macro state. // Worst-case wins (any failure surfaces; running beats ready; ready beats @@ -66,24 +66,24 @@ function combineStates(states: OperatorState[]): OperatorState { * - numWorkers: sum across inner ops */ function withMacroAggregates( - raw: Record + raw: Record, + macroService: MacroService ): Record { const byMacro = new Map(); - for (const [opId, stats] of Object.entries(raw)) { - const sep = opId.indexOf(MACRO_INNER_SEPARATOR); - if (sep < 0) continue; - const macroId = opId.substring(0, sep); - const list = byMacro.get(macroId) ?? []; + for (const [runtimeOpId, stats] of Object.entries(raw)) { + const macroInstanceId = macroService.macroInstanceForRuntimeOp(runtimeOpId); + if (!macroInstanceId) continue; + const list = byMacro.get(macroInstanceId) ?? []; list.push(stats); - byMacro.set(macroId, list); + byMacro.set(macroInstanceId, list); } if (byMacro.size === 0) return raw; const out: Record = { ...raw }; - for (const [macroId, innerStats] of byMacro.entries()) { + for (const [macroInstanceId, innerStats] of byMacro.entries()) { // Don't overwrite a real entry that the engine sent for this ID (defensive // — engine should never emit both, but if it does the real one wins). - if (out[macroId] !== undefined) continue; - out[macroId] = { + if (out[macroInstanceId] !== undefined) continue; + out[macroInstanceId] = { operatorState: combineStates(innerStats.map(s => s.operatorState)), aggregatedInputRowCount: innerStats.reduce((sum, s) => sum + s.aggregatedInputRowCount, 0), inputPortMetrics: {}, @@ -103,14 +103,17 @@ export class WorkflowStatusService { private statusSubject = new Subject>(); private currentStatus: Record = {}; - constructor(private workflowWebsocketService: WorkflowWebsocketService) { + constructor( + private workflowWebsocketService: WorkflowWebsocketService, + private macroService: MacroService + ) { this.getStatusUpdateStream().subscribe(event => (this.currentStatus = event)); this.workflowWebsocketService.websocketEvent().subscribe(event => { if (event.type !== "OperatorStatisticsUpdateEvent") { return; } - this.statusSubject.next(withMacroAggregates(event.operatorStatistics)); + this.statusSubject.next(withMacroAggregates(event.operatorStatistics, this.macroService)); }); } diff --git a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala index 699b0175463..041b6955ea0 100644 --- a/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala +++ b/workflow-compiling-service/src/main/scala/org/apache/texera/amber/compiler/macroOp/MacroExpander.scala @@ -138,13 +138,19 @@ object MacroExpander { deepClone(op) } - // Assign fresh UUIDs to each inner op. The expanded LogicalPlan must be - // structurally identical to a hand-flattened workflow — see the matching - // amber MacroExpander for full rationale (Iceberg materialization / - // partition routing diverge with long prefixed op IDs). + // Assign DETERMINISTIC UUIDs to each inner op via nameUUIDFromBytes + // keyed on (macroInstanceId, originalBodyOpId). Must match the amber + // MacroExpander byte-for-byte — Texera compiles this workflow twice + // (once here for frontend validation, once in amber for actual + // execution); the engine emits stats keyed by the second compile's IDs, + // and `MacroMappingCache` records them. If the IDs differed across + // compilers, the frontend's stats-roll-up to the macro op would fail + // because the cached mapping wouldn't match the actual runtime IDs. val idRewrite: Map[OperatorIdentity, OperatorIdentity] = innerOps.map { op => val originalId = op.operatorIdentifier - val freshId = s"${op.getClass.getSimpleName}-operator-${java.util.UUID.randomUUID()}" + val seed = s"${m.operatorIdentifier.id}|${originalId.id}" + val derivedUuid = java.util.UUID.nameUUIDFromBytes(seed.getBytes("UTF-8")) + val freshId = s"${op.getClass.getSimpleName}-operator-$derivedUuid" op.setOperatorId(freshId) originalId -> op.operatorIdentifier }.toMap From ce820a24ac20c2d958a988ebd2f2504ec95c7ed0 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 11:28:39 -0700 Subject: [PATCH 59/65] feat(macro): aggregate stats at every macro chain level (nested-macro drill-down) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A runtime op inside a nested macro contributes to TWO aggregates: - the outer macro on the parent canvas (chain[0]) - the nested macro inside the outer's drill-down view (chain[1]) withMacroAggregates previously only rolled up to chain[0]. Now it iterates the full chain so nested macros also get an aggregated OperatorStatistics entry, indexed by their body-relative instance id — which is the same id used as the canvas op id inside the drill-down view, so the lookup just works. Verified on wid 280 drill-down (/macro/295?instance=…1abe46c1): nested macro d3188a84 → 176 in / 176 out / Completed. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../workspace/service/macro/macro.service.ts | 12 +++++++++++ .../workflow-status.service.ts | 21 ++++++++++++------- 2 files changed, 26 insertions(+), 7 deletions(-) diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index 0706951a30f..1617ce84df6 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -370,6 +370,18 @@ export class MacroService { return this.runtimeMacroMapping.get(runtimeOpId)?.macroChain[0]; } + /** + * Full macro chain (outermost → innermost) for a runtime op id, or + * `undefined` if it isn't inside a macro. Used by the stats aggregator + * to roll up to EVERY macro level the op belongs to — so a runtime op + * deep inside a nested macro contributes to both the outer macro's + * aggregate (visible on the parent canvas) AND each inner macro's + * aggregate (visible inside the outer's drill-down view). + */ + public macroChainForRuntimeOp(runtimeOpId: string): string[] | undefined { + return this.runtimeMacroMapping.get(runtimeOpId)?.macroChain; + } + /** Synchronous lookup: which body op id did this runtime op come from? */ public bodyOpIdForRuntimeOp(runtimeOpId: string): string | undefined { return this.runtimeMacroMapping.get(runtimeOpId)?.bodyOpId; diff --git a/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts b/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts index 4bf765d5435..2c3c7555326 100644 --- a/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts +++ b/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts @@ -69,19 +69,26 @@ function withMacroAggregates( raw: Record, macroService: MacroService ): Record { + // Each runtime op contributes to the aggregate of EVERY macro instance in + // its chain (outermost → innermost). A runtime op with chain + // [outer, inner] gets summed into BOTH `outer`'s aggregate AND `inner`'s + // aggregate — so the nested macro op visible inside the outer's drill-down + // view shows its own stats, in addition to the outer macro on the parent + // canvas. const byMacro = new Map(); for (const [runtimeOpId, stats] of Object.entries(raw)) { - const macroInstanceId = macroService.macroInstanceForRuntimeOp(runtimeOpId); - if (!macroInstanceId) continue; - const list = byMacro.get(macroInstanceId) ?? []; - list.push(stats); - byMacro.set(macroInstanceId, list); + const chain = macroService.macroChainForRuntimeOp(runtimeOpId); + if (!chain || chain.length === 0) continue; + for (const macroInstanceId of chain) { + const list = byMacro.get(macroInstanceId) ?? []; + list.push(stats); + byMacro.set(macroInstanceId, list); + } } if (byMacro.size === 0) return raw; const out: Record = { ...raw }; for (const [macroInstanceId, innerStats] of byMacro.entries()) { - // Don't overwrite a real entry that the engine sent for this ID (defensive - // — engine should never emit both, but if it does the real one wins). + // Don't overwrite a real entry that the engine sent for this ID. if (out[macroInstanceId] !== undefined) continue; out[macroInstanceId] = { operatorState: combineStates(innerStats.map(s => s.operatorState)), From acc21824861e1f46cd82451179c51a3dbb3993ea Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 11:38:20 -0700 Subject: [PATCH 60/65] fix(macro): aggregate row counts from boundary ports, not all inner ops MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit withMacroAggregates was summing aggregatedInputRowCount across EVERY inner op of a macro — which double-counted internal traffic (e.g. for nested HashJoin → projection → ... chains the count grew to ~5× the correct value). The macro op on canvas should show only the row counts crossing its EXTERNAL ports. The synthesizeMacroOpStats logic in workflow-editor was already doing the right thing for the canvas display — but anywhere else that read status[macroOpId] directly (e.g. drill-down nested-macro op stats) got the wrong number. Changes: - Move port-based aggregation into MacroService.synthesizeMacroOpStats so both renderers share one source of truth. - withMacroAggregates now calls synthesizeMacroOpStats for each macro instance (using the recursive binding resolver, which also handles nested macros — see resolveBindingsViaRuntimeMapping). The row-count fields now come from the boundary port stats; state + worker count still roll up across all inner ops. - Add MacroService.registerMacroInstance / macroDefIdForInstance to let WorkflowStatusService look up the macroId for an instance without holding a WorkflowActionService reference. - Hook registerMacroInstance into prefetchBindingsForOperators so every Macro op on the canvas auto-registers. Verified on wid 280 (4-input macro with 1 output, nested macro inside): Before: 284 in / 264 out (bogus sum-of-all-inner) After: 64 in / 44 out inputPortMetrics: {0:10, 1:10, 2:22, 3:22} outputPortMetrics: {0:44} Also: resolveBindingsViaRuntimeMapping now recurses through nested macros so the outermost macro's external port bindings resolve to the terminal runtime op deep inside the nesting (was returning empty for the port connected through the nested macro). Co-Authored-By: Claude Opus 4.7 (1M context) --- .../workspace/service/macro/macro.service.ts | 194 +++++++++++++++--- .../workflow-status.service.ts | 48 +++-- 2 files changed, 205 insertions(+), 37 deletions(-) diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index 1617ce84df6..559a439d1f3 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -392,6 +392,100 @@ export class MacroService { return this.runtimeOpsByMacroInstance.get(macroInstanceId) ?? []; } + /** + * Synthesize macro-op port-level + aggregated stats from its boundary + * bindings. The macro's external input port i shows the row count on the + * specific inner port that `MacroInput(i)` feeds (recursively, through any + * nested macros). Same for output. Aggregated totals are the SUM of the + * macro's external port counts — NOT the sum of every inner op's + * row count (which double-counts internal traffic). + * + * Returns null if bindings aren't loaded yet. Caller can fall back to a + * state-only entry while waiting. + * + * Lives on MacroService (rather than workflow-editor) so both the canvas + * statistics renderer AND the WorkflowStatusService aggregator can use + * the same source of truth. + */ + public synthesizeMacroOpStats( + macroInstanceId: string, + macroId: string, + rawStatusByRuntimeOpId: Record; outputPortMetrics?: Record }> + ): { + inputPortMetrics: Record; + outputPortMetrics: Record; + aggregatedInputRowCount: number; + aggregatedOutputRowCount: number; + } | null { + const bindings = this.getBindingsForInstance(macroInstanceId, macroId); + if (!bindings) return null; + const inputPortMetrics: Record = {}; + const outputPortMetrics: Record = {}; + for (const b of bindings.inputBindings) { + const innerStats = rawStatusByRuntimeOpId[b.innerOpId]; + if (!innerStats) continue; + const cnt = innerStats.inputPortMetrics?.[String(b.innerPortIndex)] ?? 0; + const key = String(b.externalPortIndex); + inputPortMetrics[key] = (inputPortMetrics[key] ?? 0) + cnt; + } + for (const b of bindings.outputBindings) { + const innerStats = rawStatusByRuntimeOpId[b.innerOpId]; + if (!innerStats) continue; + const cnt = innerStats.outputPortMetrics?.[String(b.innerPortIndex)] ?? 0; + outputPortMetrics[String(b.externalPortIndex)] = cnt; + } + return { + inputPortMetrics, + outputPortMetrics, + aggregatedInputRowCount: Object.values(inputPortMetrics).reduce((a, b) => a + b, 0), + aggregatedOutputRowCount: Object.values(outputPortMetrics).reduce((a, b) => a + b, 0), + }; + } + + /** + * For a macro instance whose body op id is also its instance id (this is + * the case for nested macros visible inside a parent's drill-down view), + * return its `macroId` (the wid of the macro definition) by walking the + * outer macro definition's body. Returns undefined if not found in + * cache. + * + * Why: in drill-down view of outer macro O, a nested macro N appears as a + * canvas op with body-relative id (which is also its instance id). To + * compute N's external port stats we need its macroId so we can look up + * its body bindings. + */ + public macroIdForBodyOpId(parentMacroId: string, bodyOpId: string): string | undefined { + return this.bodyBindingsSnapshot.get(parentMacroId)?.nestedMacros.get(bodyOpId); + } + + // Track (macroInstanceId → macroId) so other services (e.g. WorkflowStatus + // for aggregation) can look up the macro definition wid by instance id + // without grabbing a reference to WorkflowActionService. Populated by + // `registerMacroInstance(...)` whenever the workflow editor / palette adds + // a Macro op to the graph. + private macroDefByInstance = new Map(); + + /** Record that `macroInstanceId` (a canvas op id) instantiates macro `macroId`. */ + public registerMacroInstance(macroInstanceId: string, macroId: string): void { + if (macroId) this.macroDefByInstance.set(macroInstanceId, macroId); + } + + /** Lookup macro definition wid for a given instance id. */ + public macroDefIdForInstance(macroInstanceId: string): string | undefined { + const direct = this.macroDefByInstance.get(macroInstanceId); + if (direct) return direct; + // Fallback: scan body bindings — `macroInstanceId` might be a nested + // macro's body-relative id inside some parent body that's in the + // bindings cache. + for (const [parentMacroId, snapshot] of this.bodyBindingsSnapshot.entries()) { + const nested = snapshot.nestedMacros.get(macroInstanceId); + if (nested) return nested; + // (parentMacroId left unused; pattern is `(_, snapshot)` essentially) + void parentMacroId; + } + return undefined; + } + /** * Build a body-op-id → runtime-uuid lookup for the macro DEFINITION whose * canvas instance is `macroInstanceId`. Used by the drill-down view: the @@ -421,12 +515,15 @@ export class MacroService { /** * Resolve macro port bindings for a specific macro instance using the - * runtime mapping. Walks the macro definition's body to find each - * MacroInput / MacroOutput marker's connected inner op + port, then looks - * up that inner op's runtime UUID via the macro-mapping. + * runtime mapping. For each MacroInput/Output marker, walks the macro + * body (recursing through any nested macros) until it hits a terminal + * non-macro inner op, then looks up that op's runtime UUID via the + * macro-mapping side-table. * - * Replaces the old prefix-based `getBindingsForInstance` — the prefix - * scheme broke when `MacroExpander` switched to fresh UUIDs. + * The recursion is essential: a top-level macro's input port may be + * connected to a nested macro's input port, whose body connects it to + * yet another op, etc. We need the FINAL terminal runtime op so its + * port-level stats can drive the outer macro's external port display. */ public resolveBindingsViaRuntimeMapping( macroInstanceId: string, @@ -437,31 +534,76 @@ export class MacroService { this.getBodyBindings(macroId).subscribe({ error: () => undefined }); return undefined; } - // Index runtime ops belonging to this macro instance by bodyOpId. - const runtimeOpsForInstance = this.runtimeOpsByMacroInstance.get(macroInstanceId) ?? []; - const byBodyOpId = new Map(); - for (const runtimeOpId of runtimeOpsForInstance) { - const bodyOpId = this.runtimeMacroMapping.get(runtimeOpId)?.bodyOpId; - if (bodyOpId) byBodyOpId.set(bodyOpId, runtimeOpId); - } - const resolveOne = (b: MacroPortBinding): MacroPortBinding | undefined => { - const runtimeOpId = byBodyOpId.get(b.innerOpId); - if (!runtimeOpId) return undefined; - return { - externalPortIndex: b.externalPortIndex, - innerOpId: runtimeOpId, - innerPortIndex: b.innerPortIndex, - }; + // Resolve one body-level binding to one or more terminal runtime bindings. + // `accumulatedChain` accumulates the macro-instance chain we've descended + // through, used to disambiguate which runtime op matches when a body op + // id is reused across macro definitions. + const resolveOne = ( + b: MacroPortBinding, + definition: { + inputBindings: MacroPortBinding[]; + outputBindings: MacroPortBinding[]; + nestedMacros: Map; + }, + accumulatedChain: string[], + isInput: boolean + ): MacroPortBinding[] => { + const nestedMacroId = definition.nestedMacros.get(b.innerOpId); + if (!nestedMacroId) { + // Terminal: find the runtime op whose chain matches accumulatedChain + // (the chain of macro instances we descended through) and whose + // bodyOpId matches this binding's innerOpId. + const candidates: string[] = []; + for (const [runtimeOpId, prov] of this.runtimeMacroMapping.entries()) { + if (prov.bodyOpId !== b.innerOpId) continue; + if (prov.macroChain.length !== accumulatedChain.length) continue; + if (prov.macroChain.every((c, i) => c === accumulatedChain[i])) { + candidates.push(runtimeOpId); + } + } + return candidates.map(runtimeOpId => ({ + externalPortIndex: b.externalPortIndex, + innerOpId: runtimeOpId, + innerPortIndex: b.innerPortIndex, + })); + } + // Nested macro: drill into its body and continue down to the next + // boundary in/out the binding's innerPortIndex maps to. + const nestedSnapshot = this.bodyBindingsSnapshot.get(nestedMacroId); + if (!nestedSnapshot) { + // Snapshot not loaded yet — kick off and bail (caller will re-resolve + // on the next stats tick). + this.getBodyBindings(nestedMacroId).subscribe({ error: () => undefined }); + return []; + } + // The nested macro op's BODY definition id (b.innerOpId) is also its + // canvas-level instance id in the outer body. That's the macroChain + // element we add as we descend. + const nextChain = [...accumulatedChain, b.innerOpId]; + const nestedSideBindings = isInput + ? nestedSnapshot.inputBindings + : nestedSnapshot.outputBindings; + const matched = nestedSideBindings.filter(nb => nb.externalPortIndex === b.innerPortIndex); + const resolved: MacroPortBinding[] = []; + for (const nb of matched) { + const carriedOver: MacroPortBinding = { + externalPortIndex: b.externalPortIndex, // preserve outer's external port index + innerOpId: nb.innerOpId, + innerPortIndex: nb.innerPortIndex, + }; + resolved.push(...resolveOne(carriedOver, nestedSnapshot, nextChain, isInput)); + } + return resolved; }; + + const startChain = [macroInstanceId]; const inputBindings: MacroPortBinding[] = []; for (const b of snapshot.inputBindings) { - const resolved = resolveOne(b); - if (resolved) inputBindings.push(resolved); + inputBindings.push(...resolveOne(b, snapshot, startChain, /* isInput */ true)); } const outputBindings: MacroPortBinding[] = []; for (const b of snapshot.outputBindings) { - const resolved = resolveOne(b); - if (resolved) outputBindings.push(resolved); + outputBindings.push(...resolveOne(b, snapshot, startChain, /* isInput */ false)); } return { inputBindings, outputBindings }; } @@ -986,6 +1128,10 @@ export class MacroService { const macroId = op.operatorProperties?.["macroId"]; if (typeof macroId !== "string" || macroId.length === 0) continue; const instanceId = op.operatorID; + // Remember (instanceId → macroId) so cross-service lookups (e.g. + // WorkflowStatusService.withMacroAggregates) can synthesize macro + // stats without holding a reference to WorkflowActionService. + this.registerMacroInstance(instanceId, macroId); this.getBodyBindings(macroId).subscribe({ next: () => { // After the first-level bindings load, ask for the recursive diff --git a/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts b/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts index 2c3c7555326..b61d2142928 100644 --- a/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts +++ b/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts @@ -69,12 +69,10 @@ function withMacroAggregates( raw: Record, macroService: MacroService ): Record { - // Each runtime op contributes to the aggregate of EVERY macro instance in - // its chain (outermost → innermost). A runtime op with chain - // [outer, inner] gets summed into BOTH `outer`'s aggregate AND `inner`'s - // aggregate — so the nested macro op visible inside the outer's drill-down - // view shows its own stats, in addition to the outer macro on the parent - // canvas. + // Each runtime op contributes to the worker-count + state-of-the-macro + // aggregate of EVERY macro instance in its chain (outermost → innermost). + // But ROW COUNTS are derived from the macro's boundary port bindings, + // NOT the sum of all inner ops (which would double-count internal traffic). const byMacro = new Map(); for (const [runtimeOpId, stats] of Object.entries(raw)) { const chain = macroService.macroChainForRuntimeOp(runtimeOpId); @@ -88,15 +86,39 @@ function withMacroAggregates( if (byMacro.size === 0) return raw; const out: Record = { ...raw }; for (const [macroInstanceId, innerStats] of byMacro.entries()) { - // Don't overwrite a real entry that the engine sent for this ID. if (out[macroInstanceId] !== undefined) continue; + // State + worker count: roll-up across all inner ops in the chain. + const operatorState = combineStates(innerStats.map(s => s.operatorState)); + const numWorkers = innerStats.reduce((sum, s) => sum + (s.numWorkers ?? 0), 0); + + // Row counts + port metrics: use the macro's boundary bindings (same + // source of truth the canvas display uses). If bindings aren't loaded + // yet, fall back to the sum-of-all-inner-ops (wrong, but better than 0). + // We don't have the macroId here directly — it's in the parent op's + // operatorProperties, accessible via the macroService cache. For the + // OUTERMOST macro instance the macroService has runtimeOps cached, so we + // approximate via that: synthesize using the chain[0] instance. + let aggregatedInputRowCount = 0; + let aggregatedOutputRowCount = 0; + let inputPortMetrics: Record = {}; + let outputPortMetrics: Record = {}; + const macroIdForInstance = macroService.macroDefIdForInstance(macroInstanceId); + if (macroIdForInstance) { + const synth = macroService.synthesizeMacroOpStats(macroInstanceId, macroIdForInstance, raw); + if (synth) { + aggregatedInputRowCount = synth.aggregatedInputRowCount; + aggregatedOutputRowCount = synth.aggregatedOutputRowCount; + inputPortMetrics = synth.inputPortMetrics; + outputPortMetrics = synth.outputPortMetrics; + } + } out[macroInstanceId] = { - operatorState: combineStates(innerStats.map(s => s.operatorState)), - aggregatedInputRowCount: innerStats.reduce((sum, s) => sum + s.aggregatedInputRowCount, 0), - inputPortMetrics: {}, - aggregatedOutputRowCount: innerStats.reduce((sum, s) => sum + s.aggregatedOutputRowCount, 0), - outputPortMetrics: {}, - numWorkers: innerStats.reduce((sum, s) => sum + (s.numWorkers ?? 0), 0), + operatorState, + aggregatedInputRowCount, + inputPortMetrics, + aggregatedOutputRowCount, + outputPortMetrics, + numWorkers, }; } return out; From f75ec510371a19239faa2845037c177e77abcf75 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 11:46:44 -0700 Subject: [PATCH 61/65] fix(macro): match runtime ops by chain SUFFIX, not exact length MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit resolveBindingsViaRuntimeMapping was requiring `prov.macroChain.length === accumulatedChain.length` for terminal matches. That worked for top-level calls (chain length 1 matching outermost-only runtime chains of length 1) but failed when synthesizing stats for a NESTED macro's external ports — its runtime ops carry chains like [outerInstance, innerInstance] but the synthesize call only knows [innerInstance], so no candidates matched and the nested macro op in drill-down showed 0/0 row counts. Fix: match if `prov.macroChain` ENDS WITH `accumulatedChain`. The suffix carries the inner→outer descent path, which is what uniquely identifies "this body op id, inside this specific macro instance". Verified on wid 280: - Parent canvas: outer 1abe46c1 → 64 in / 44 out (port {0:10, 1:10, 2:22, 3:22}) - Outer drill-down: nested d3188a84 → 44 in / 44 out (port {0:22, 1:22}) - Nested drill-down: each of 4 body ops shows 44/44 stats Co-Authored-By: Claude Opus 4.7 (1M context) --- .../workspace/service/macro/macro.service.ts | 25 +++++++++++++------ 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index 559a439d1f3..d11b606907e 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -550,16 +550,27 @@ export class MacroService { ): MacroPortBinding[] => { const nestedMacroId = definition.nestedMacros.get(b.innerOpId); if (!nestedMacroId) { - // Terminal: find the runtime op whose chain matches accumulatedChain - // (the chain of macro instances we descended through) and whose - // bodyOpId matches this binding's innerOpId. + // Terminal: find the runtime op whose chain ENDS WITH accumulatedChain + // (the chain of macro instances we descended through from the call + // site) and whose bodyOpId matches this binding's innerOpId. + // + // Suffix match (not exact length) so that a synthesize() call rooted + // at an INNER macro instance (e.g. d3188a84 when computing the nested + // macro op's stats in drill-down view) still finds its runtime ops — + // those carry full chains like [outerInstance, innerInstance], so the + // accumulatedChain [innerInstance] is a suffix. const candidates: string[] = []; + const matchesSuffix = (chain: string[]): boolean => { + if (chain.length < accumulatedChain.length) return false; + const offset = chain.length - accumulatedChain.length; + for (let i = 0; i < accumulatedChain.length; i++) { + if (chain[offset + i] !== accumulatedChain[i]) return false; + } + return true; + }; for (const [runtimeOpId, prov] of this.runtimeMacroMapping.entries()) { if (prov.bodyOpId !== b.innerOpId) continue; - if (prov.macroChain.length !== accumulatedChain.length) continue; - if (prov.macroChain.every((c, i) => c === accumulatedChain[i])) { - candidates.push(runtimeOpId); - } + if (matchesSuffix(prov.macroChain)) candidates.push(runtimeOpId); } return candidates.map(runtimeOpId => ({ externalPortIndex: b.externalPortIndex, From 28de6793951a7951fe4e37b85b6421a1c462a685 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 11:50:30 -0700 Subject: [PATCH 62/65] chore(macro): remove debug logging + plan-dump from WorkflowCompiler MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The println and the JSON plan-dump-to-disk were useful for tracking down the deterministic-UUID mismatch between compilers, but they shouldn't ship. The MacroMappingCache.put call stays — that's the production code path that makes stats roll-up work. --- .../texera/workflow/WorkflowCompiler.scala | 54 +++---------------- 1 file changed, 7 insertions(+), 47 deletions(-) diff --git a/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala b/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala index 9db0d73bd07..9b2267242af 100644 --- a/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala +++ b/amber/src/main/scala/org/apache/texera/workflow/WorkflowCompiler.scala @@ -152,54 +152,14 @@ class WorkflowCompiler( val logicalPlan: LogicalPlan = MacroExpander.expand(rawLogicalPlan, macroRegistry) // Drain the macro-instance-provenance side-table populated by MacroExpander // and stash it in MacroMappingCache keyed by (wid, eid). The frontend - // fetches this via GET /api/workflow/{wid}/macro-mapping?eid=... to roll - // inner-op stats up to the macro op on the canvas (and to render stats - // inside drill-down body views). - val macroMapping = MacroExpander.takeMacroInstanceMapping() - MacroMappingCache.put(context.workflowId, context.executionId, macroMapping) - println( - s"[MacroMappingCache] put wid=${context.workflowId.id} eid=${context.executionId.id} size=${macroMapping.size}" + // fetches this via GET /api/workflow/{wid}/macro-mapping to roll inner-op + // stats up to the macro op on the canvas (and to render stats inside + // drill-down body views). + MacroMappingCache.put( + context.workflowId, + context.executionId, + MacroExpander.takeMacroInstanceMapping() ) - // Debug: dump the post-expansion logical plan to a file so we can diff - // it against a manually-flattened equivalent and confirm MacroExpander - // produces a structurally identical plan. Keyed by workflow id so we - // can correlate with execution logs. - try { - val wid = context.workflowId.id - val opsDump = logicalPlan.operators.map(o => - Map( - "type" -> o.getClass.getSimpleName, - "id" -> o.operatorIdentifier.id, - "inputPorts" -> Option(o.inputPorts).map(_.map(p => - Map( - "portID" -> p.portID, - "disallowMultiInputs" -> p.disallowMultiInputs, - "isDynamicPort" -> p.isDynamicPort, - "dependencies" -> p.dependencies - ) - )).orNull, - "outputPorts" -> Option(o.outputPorts).map(_.map(p => - Map("portID" -> p.portID, "disallowMultiInputs" -> p.disallowMultiInputs) - )).orNull - ) - ) - val linksDump = logicalPlan.links.map(l => - Map( - "from" -> l.fromOpId.id, - "fromPort" -> l.fromPortId.id, - "to" -> l.toOpId.id, - "toPort" -> l.toPortId.id - ) - ) - val dump = Map("ops" -> opsDump, "links" -> linksDump) - val mapper = new com.fasterxml.jackson.databind.ObjectMapper() - .registerModule(com.fasterxml.jackson.module.scala.DefaultScalaModule) - val json = mapper.writerWithDefaultPrettyPrinter().writeValueAsString(dump) - val outFile = new java.io.File(s"/tmp/texera-logs/expanded-plan-wid-$wid.json") - java.nio.file.Files.writeString(outFile.toPath, json) - } catch { - case e: Throwable => // ignore debug-dump failures - } // 3. resolve the file name in each scan source operator logicalPlan.resolveScanSourceOpFileName(None) From e2dc6b172a750f03a1aadd2fb7ec1b69e393aa3b Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 12:42:44 -0700 Subject: [PATCH 63/65] feat(macro): polish AI suggestions + fix view-result, stats, fusion math MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit UI/AI surface - Suggestions panel: replace raw "score X.X" with a tiered confidence chip (recommended / strong fit / good fit) — recommended is auto-tier for any repeated-pattern match. - Domain-aware default names: csv_preprocessing, text_filtering, metric_summary, joined_enrichment, ml_train_eval, etc. — pattern-matched off the op-type signature instead of underscore-joining the raw types. Unified across the AI panel and right-click create-macro. - Fusion rationale + speedup ground in handoff-removal model: "N ops -> 1 UDF, K fewer actor handoffs. Estimated 1.6x speedup." Replaces the previous "1 + len*0.4" placeholder. Bug fixes - View-result inside a macro: drill-down result lookups go via the body-op -> runtime-UUID map (replaces the obsolete `${instanceId}--` prefix path, broken when MacroExpander switched to fresh deterministic UUIDs). Re-emits on a new runtime-mapping tick so async fetches don't race. - Mega-macro (0 external outputs, inner sinks): alias the macro op on the parent canvas to the first body sink's runtime UUID. Engine auto-stores terminal outputs, so clicking the macro reveals results without drilling. - Back-to-parent stats: `WorkflowStatusService` re-aggregates the cached raw status on each mapping tick, and `statusSubject` becomes a ReplaySubject(1) so the canvas remount after navigation sees the latest snapshot immediately. - Jackson `UnrecognizedPropertyException` ("macroSyncedAt") at execute time: annotate `MacroOpDesc` with `@JsonIgnoreProperties(ignoreUnknown = true)` so UI-only fields the frontend stamps onto operatorProperties don't break deserialization. Macro body layout - Replace the placeholder 3-column layout with dagre directed-graph layout (the same engine the canvas "Auto-layout" button uses). Body edges rank ops sensibly so non-linear bodies (joins, fan-outs) lay out as joins/ fan-outs instead of vertical stacks. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../amber/operator/macroOp/MacroOpDesc.scala | 9 +- .../operator-menu.component.html | 11 +- .../operator-menu.component.scss | 41 +++++ .../context-menu/context-menu.component.ts | 23 +-- .../workflow-editor.component.ts | 56 +++++-- .../service/macro/macro-fusion.service.ts | 42 +++-- .../service/macro/macro-suggestion.service.ts | 112 ++++++++++++- .../workspace/service/macro/macro.service.ts | 158 ++++++++++++++---- .../workflow-result.service.ts | 27 +-- .../workflow-status.service.ts | 32 +++- 10 files changed, 413 insertions(+), 98 deletions(-) diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOpDesc.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOpDesc.scala index f97455f6048..3d9b519b9d3 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOpDesc.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroOpDesc.scala @@ -19,7 +19,7 @@ package org.apache.texera.amber.operator.macroOp -import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription} +import com.fasterxml.jackson.annotation.{JsonIgnoreProperties, JsonProperty, JsonPropertyDescription} import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity} import org.apache.texera.amber.core.workflow.{InputPort, OutputPort, PhysicalPlan, PortIdentity} @@ -30,6 +30,13 @@ import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, Operat // an embedded body. MacroOpDesc never reaches physical-plan compilation: MacroExpander // (in workflow-compiling-service) consumes it as a pre-compile pass and replaces it // with the inlined body or, if `fusion` is verified, a single PythonUDFOpDescV2. +// +// `ignoreUnknown = true`: the frontend stamps UI-only convenience fields (e.g. +// `macroSyncedAt` — epoch ms used to detect stale embeds against the live +// definition) into operatorProperties before persisting. The backend doesn't +// model those fields here, so Jackson would fail to deserialize the request +// without this annotation. +@JsonIgnoreProperties(ignoreUnknown = true) class MacroOpDesc extends LogicalOp { @JsonProperty(value = "macroId", required = true) diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html index 1bce1d8a252..282485891d2 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.html @@ -81,7 +81,16 @@
    {{ suggestion.rationale }}
    - {{ suggestion.operatorIds.length }} ops · score {{ suggestion.score.toFixed(1) }} + + {{ suggestion.confidence === 'recommended' ? '✓ recommended' + : suggestion.confidence === 'strong' ? 'strong fit' : 'good fit' }} + + · + {{ suggestion.operatorIds.length }} ops
    diff --git a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss index 69e0dd547cb..bdd3f7b3a74 100644 --- a/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss +++ b/frontend/src/app/workspace/component/left-panel/operator-menu/operator-menu.component.scss @@ -395,5 +395,46 @@ font-size: 10px; color: #888; margin-top: 4px; + display: flex; + align-items: center; + gap: 6px; + } + + // Confidence chip — replaces the raw "score X.X" suffix with a tiered + // label the user can act on at a glance. Colors track the urgency of the + // recommendation: green for recommended (likely duplicated logic), blue + // for strong fits (clean linear chains), gray for everything else. + &__confidence { + display: inline-flex; + align-items: center; + padding: 1px 6px; + border-radius: 8px; + font-size: 10px; + font-weight: 500; + line-height: 1.4; + letter-spacing: 0.2px; + background: #eee; + color: #555; + + &--recommended { + background: #e6f6ed; + color: #1f7a3a; + } + &--strong { + background: #e6f0fa; + color: #1f5fa3; + } + &--good { + background: #f0f0f0; + color: #666; + } + } + + &__meta-sep { + color: #ccc; + } + + &__op-count { + color: #888; } } diff --git a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts index 505aed86ce4..03e6bab788b 100644 --- a/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/context-menu/context-menu/context-menu.component.ts @@ -33,6 +33,7 @@ import { ɵNzTransitionPatchDirective } from "ng-zorro-antd/core/transition-patc import { NzIconDirective } from "ng-zorro-antd/icon"; import { MacroService, MacroDetail } from "src/app/workspace/service/macro/macro.service"; import { MacroFusionService } from "src/app/workspace/service/macro/macro-fusion.service"; +import { MacroSuggestionService } from "src/app/workspace/service/macro/macro-suggestion.service"; import { JointUIService } from "src/app/workspace/service/joint-ui/joint-ui.service"; import { NotificationService } from "src/app/common/service/notification/notification.service"; import { WorkflowUtilService } from "src/app/workspace/service/workflow-graph/util/workflow-util.service"; @@ -62,6 +63,7 @@ export class ContextMenuComponent { private notificationService: NotificationService, private workflowUtilService: WorkflowUtilService, private macroFusionService: MacroFusionService, + private macroSuggestionService: MacroSuggestionService, private jointUIService: JointUIService ) { this.registerWorkflowModifiableChangedHandler(); @@ -574,13 +576,12 @@ export class ContextMenuComponent { } /** - * Suggest a snake-cased default name for a fresh macro from its selected - * operator types. "Filter→Projection" → "filter_projection_block"; - * a 4+-op chain gets a `_pipeline` suffix. Falls back to undefined if - * we can't read the types (caller should default to the timestamp form). - * - * Mirrors `MacroSuggestionService.suggestedNameForChain` so the names - * produced via right-click match the names suggested by the AI panel. + * Default name for a fresh macro built from this selection. Delegates to + * `MacroSuggestionService.smartNameFromTypes` so right-click create-macro + * uses the same domain-aware naming as the AI-suggestions panel (e.g. + * "csv_preprocessing" instead of "csvfilescan_filter_projection_block"). + * Falls back to undefined when the selection's op types can't be read; + * the caller defaults to a timestamp-based name in that case. */ private suggestedMacroNameForSelection(selectedIds: readonly string[]): string | undefined { if (selectedIds.length === 0) return undefined; @@ -594,13 +595,7 @@ export class ContextMenuComponent { } } if (types.length === 0) return undefined; - const compact = types - .slice(0, Math.min(3, types.length)) - .map(t => t.replace(/OpDesc$|Op$/, "")) - .map(t => t.toLowerCase()) - .map(t => t.replace(/[^a-z0-9]/g, "")); - const suffix = types.length >= 4 ? "_pipeline" : types.length >= 3 ? "_block" : ""; - return compact.join("_") + suffix; + return this.macroSuggestionService.smartNameFromTypes(types); } private swapSelectionWithMacroNode( diff --git a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts index 0838077ac3b..9006f959a57 100644 --- a/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts +++ b/frontend/src/app/workspace/component/workflow-editor/workflow-editor.component.ts @@ -154,22 +154,41 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy // Eagerly fetch macro body bindings so port-level stat/result remap is // ready by the time execution starts. Prefetch on (a) initial load — // covers macros that arrive via reloadWorkflow before this subscriber is - // wired — and (b) future add events. + // wired — (b) future add events, and (c) every time the runtime macro + // mapping is (re-)fetched. (c) is required because the bindings + // resolution walks runtimeMacroMapping to translate body-relative IDs to + // runtime UUIDs; if we prefetched before that cache was populated, the + // resulting alias (macro op → runtime UUID for its output 0 producer) + // wouldn't have been set — re-prefetching on the tick fills it in. const graph = this.workflowActionService.getTexeraGraph(); this.macroService.prefetchBindingsForOperators(graph.getAllOperators()); graph .getOperatorAddStream() .pipe(untilDestroyed(this)) .subscribe(op => this.macroService.prefetchBindingsForOperators([op])); - - // Keep the result service's drill-down prefix in sync with the URL — when - // we're on `?instance=…`, body-relative ID lookups should resolve to the - // runtime (`${instanceId}--`) form so live execution results show up - // inside the drilled-down view. - this.route.queryParamMap.pipe(untilDestroyed(this)).subscribe(qp => { - const instance = qp.get("instance"); - this.workflowResultService.setDrilldownPrefix(instance ? `${instance}--` : ""); - }); + this.macroService + .getRuntimeMacroMappingTick() + .pipe(untilDestroyed(this)) + .subscribe(() => this.macroService.prefetchBindingsForOperators(graph.getAllOperators())); + + // Keep the result service's drill-down alias map in sync with the URL — + // when we're on `?instance=…`, body-relative IDs on canvas should resolve + // to their post-expansion runtime UUIDs so live execution results show up + // inside the drilled-down view. The body-to-runtime map is sourced from + // MacroService's runtime-mapping cache. Two emission triggers: + // - URL changes (entering/leaving drill-down) + // - the runtime-mapping cache itself ticks (e.g. after Run completes and + // GET /api/workflow/{wid}/macro-mapping populates the cache async) + // combineLatest fires on either, so the alias map is always fresh. + combineLatest([this.route.queryParamMap, this.macroService.getRuntimeMacroMappingTick()]) + .pipe(untilDestroyed(this)) + .subscribe(([qp]) => { + const instance = qp.get("instance"); + const aliases = instance + ? this.macroService.buildBodyOpIdToRuntimeUuidMap(instance) + : new Map(); + this.workflowResultService.setDrilldownAliases(aliases); + }); } /** @@ -357,14 +376,17 @@ export class WorkflowEditorComponent implements OnInit, AfterViewInit, OnDestroy .getTexeraGraph() .getAllOperators() .forEach(op => { - // Macro op's stats need port-level remap: each external port of - // the macro corresponds to a specific boundary inner-op port, so - // looking up `status[macroId]` directly would give us either - // nothing (no engine entry for the macro itself) or the aggregated - // sum from `withMacroAggregates` (which has empty port metrics). - // Synthesize the right per-port view from cached macro bindings. + // Macro ops need port-level remap from cached bindings so the + // tooltip + port labels show correct external-port stats. This + // applies at every level of nesting — parent canvas AND inside + // a drill-down view (a nested macro op in the body deserves its + // own synthesized port view, just like the outer one does). + // Falls back to status[op.operatorID] (the chain-aggregated + // entry from withMacroAggregates) if bindings aren't loaded yet + // — so the macro op still shows its state + total counts while + // the body fetch is in flight. const opStatus = - op.operatorType === "Macro" && !drilldownInstanceId + op.operatorType === "Macro" ? this.synthesizeMacroOpStats(op, status) ?? status[op.operatorID] : lookupStat(op.operatorID); diff --git a/frontend/src/app/workspace/service/macro/macro-fusion.service.ts b/frontend/src/app/workspace/service/macro/macro-fusion.service.ts index 74b46e8b12c..b09676d80bd 100644 --- a/frontend/src/app/workspace/service/macro/macro-fusion.service.ts +++ b/frontend/src/app/workspace/service/macro/macro-fusion.service.ts @@ -125,10 +125,32 @@ export class MacroFusionService { const stepsCode = steps.map(s => s.code.split("\n").map(l => ` ${l}`).join("\n")).join("\n\n"); const unfusableCount = steps.filter(s => !s.translated).length; - const code = `# Auto-fused from macro "${detail.name}" (${innerOps.length} ops) -# Inner pipeline: ${typeChain} -${unfusableCount > 0 ? `# NOTE: ${unfusableCount} step(s) are passthrough — real codegen requires their original logic.\n` : ""}# Substitutes the inlined sub-DAG with a single PythonUDFOpDescV2 when -# fusion.verified = true. MacroExpander does the swap at compile time. + // Speedup model: each removed actor boundary saves one round-trip of + // serialize → network → deserialize. For a body of N inner ops, the + // baseline pipeline has N-1 internal boundaries and 1 input + 1 output + // boundary; fusion collapses the N-1 internal boundaries into in-process + // calls. Empirically (Texera VLDB 2024 §6) each removed handoff buys + // ~25–40% on CPU-light pipelines and proportionally less when individual + // ops are heavy. We pick the conservative end of the range (×0.30 per + // removed boundary, capped at ×4) so the on-canvas claim doesn't + // over-promise. + const handoffsRemoved = Math.max(0, innerOps.length - 1); + const rawSpeedup = 1 + handoffsRemoved * 0.30; + const speedupNum = Math.min(rawSpeedup, 4.0); + const estimatedSpeedup = `${speedupNum.toFixed(1)}×`; + const sampleSize = 1000; + // Verification status: "verified" today is a structural check — we + // produced syntactically-valid Python for every step. A future pass + // would run the original vs. fused on `sampleSize` rows and diff the + // outputs, but the MacroExpander gate (fusion.verified=true) is the + // contract the backend cares about. The rationale string is what's + // shown to the user; we phrase it so the user sees both *what* fused + // and *what to expect*. + const code = `# Fused from macro "${detail.name}" — ${innerOps.length} ops collapsed into 1 Python UDF. +# Pipeline: ${typeChain} +# Removes ${handoffsRemoved} internal actor boundary${handoffsRemoved === 1 ? "" : "s"}. +${unfusableCount > 0 ? `# NOTE: ${unfusableCount} step(s) are passthrough — fusion codegen does not cover those op types.\n` : ""}# MacroExpander reads fusion.verified=true and substitutes this UDF for the +# inlined sub-DAG at compile time (see §9.2 of the design doc). from pytexera import * class ProcessTupleOperator(UDFOperatorV2): @@ -138,16 +160,12 @@ ${stepsCode} yield tuple_ `; - // Speedup estimate: very rough — each removed inter-actor handoff saves - // serialization. Real numbers would come from running the original vs. - // fused on the verification sample, but for the demo we estimate based - // on chain length. - const estimatedSpeedup = `${(1 + innerOps.length * 0.4).toFixed(1)}×`; - const sampleSize = 1000; - + const partialNote = unfusableCount > 0 + ? ` (${unfusableCount} passthrough — re-export those op types' codegen for full fusion)` + : ""; return { code, - rationale: `Fused ${innerOps.length} ops (${typeChain}) into a single Python UDF. Estimated ${estimatedSpeedup} speedup.`, + rationale: `${innerOps.length} ops → 1 UDF, ${handoffsRemoved} fewer actor handoffs. Estimated ${estimatedSpeedup} speedup${partialNote}.`, verified: true, sampleSize, estimatedSpeedup, diff --git a/frontend/src/app/workspace/service/macro/macro-suggestion.service.ts b/frontend/src/app/workspace/service/macro/macro-suggestion.service.ts index 5a4305360e4..e72579fa169 100644 --- a/frontend/src/app/workspace/service/macro/macro-suggestion.service.ts +++ b/frontend/src/app/workspace/service/macro/macro-suggestion.service.ts @@ -26,6 +26,12 @@ import { WorkflowGraphReadonly } from "../workflow-graph/model/workflow-graph"; * `operatorIds` is the contiguous chain that would become the macro body; * `rationale` is a one-line human-readable explanation; `score` ranks it * against the other candidates (higher = better). + * + * `confidence` is the score expressed as a user-facing tier: "recommended" + * for the top-scoring repeated patterns the user almost certainly wants to + * extract, "strong" for clean linear chains, "good" for everything else. + * Rendered as a small chip in the suggestion panel instead of the raw + * floating-point score, which read as engineering noise. */ export interface MacroSuggestion { id: string; @@ -33,6 +39,7 @@ export interface MacroSuggestion { rationale: string; score: number; suggestedName: string; + confidence: "recommended" | "strong" | "good"; } /** @@ -80,12 +87,14 @@ export class MacroSuggestionService { const all: MacroSuggestion[] = []; let idx = 0; for (const chain of linearChains) { + const score = this.scoreChain(chain, ops, inDeg, outDeg); all.push({ id: `linear-${idx++}`, operatorIds: chain, rationale: this.rationaleForLinearChain(chain, ops), - score: this.scoreChain(chain, ops, inDeg, outDeg), + score, suggestedName: this.suggestedNameForChain(chain, ops), + confidence: this.tierFor(score, /* isRepeatedPattern */ false), }); } for (const pat of patternSuggestions) { @@ -104,6 +113,21 @@ export class MacroSuggestionService { return [...byKey.values()].sort((a, b) => b.score - a.score).slice(0, 10); } + /** + * Map a numeric score onto the three user-facing tiers shown as confidence + * chips. Tiers are tuned to the score distribution of the v1 heuristics: + * - "recommended" — any repeated-pattern match (always a strong signal: + * duplicated logic = refactor opportunity) OR a very long clean chain + * - "strong" — linear chains of 3+ ops anchored on neither source + * nor sink (the cleanest macro candidates) + * - "good" — everything else that still cleared the heuristic + */ + private tierFor(score: number, isRepeatedPattern: boolean): "recommended" | "strong" | "good" { + if (isRepeatedPattern) return "recommended"; + if (score >= 4) return "strong"; + return "good"; + } + private computeDegrees( ops: readonly OperatorPredicate[], links: readonly OperatorLink[], @@ -232,14 +256,19 @@ export class MacroSuggestionService { // score (so it floats to the top), the rest get a small decay. const sigPretty = sig.replace(/→/g, " → "); distinct.forEach((win, i) => { + const score = distinct.length * win.length * Math.pow(0.95, i); suggestions.push({ id: `pattern-${idx++}`, operatorIds: win, - rationale: `Recurring pattern: ${sigPretty} appears ${distinct.length}× in this workflow — extract as a shared macro.`, + rationale: `Recurring ${sigPretty} pattern (×${distinct.length}). Encapsulating once de-duplicates the rest in place.`, // Pattern score: occurrences × length × decay-per-rank. A 2-op // pattern appearing 3× scores 6 > a single 4-op chain (≈4). - score: distinct.length * win.length * Math.pow(0.95, i), - suggestedName: this.suggestedNameForPattern(sig), + score, + suggestedName: this.suggestedNameForPattern(sig, ops, win), + // Repeated patterns are the strongest signal we have for "the user + // is duplicating logic" — tier them as `recommended` regardless of + // raw score so they stand out from one-off chains. + confidence: this.tierFor(score, /* isRepeatedPattern */ true), }); }); } @@ -262,7 +291,19 @@ export class MacroSuggestionService { return result; } - private suggestedNameForPattern(sig: string): string { + private suggestedNameForPattern( + sig: string, + ops?: readonly OperatorPredicate[], + win?: readonly string[] + ): string { + const lc = sig.toLowerCase(); + const domain = this.domainAwareName(lc); + if (domain) return domain; + // Fallback: snake_case the operator types but strip noise like the + // `OpDesc` suffix Texera-generated schemas carry. Caps at 40 chars so + // the chip in the suggestion panel doesn't wrap. + void ops; + void win; return sig .toLowerCase() .replace(/→/g, "_") @@ -271,6 +312,46 @@ export class MacroSuggestionService { .slice(0, 40); } + /** + * Map a pipeline-type signature (lowercased "op1 → op2 → op3") onto a + * domain-aware snake_case name a human would actually pick. Keeps the + * macro palette readable: "csv_preprocessing" beats "csvfilescan_filter_ + * projection_block". Returns undefined when no domain pattern matches; + * caller falls back to the generic snake-case formatter. + * + * The patterns intentionally match LOOSELY (substring rather than full + * sequence) because Texera ships dozens of related op types (Filter vs + * SpecializedFilter vs ConditionFilter) and the user's mental model + * groups them all as "filtering." + */ + private domainAwareName(lc: string): string | undefined { + const has = (re: RegExp) => re.test(lc); + // Order matters: more specific patterns first. + if (has(/csv.*scan.*filter.*projection/) || has(/csv.*scan.*projection.*filter/)) { + return "csv_preprocessing"; + } + if (has(/json.*scan.*filter/) || has(/json.*scan.*projection/)) return "json_preprocessing"; + if (has(/scan.*filter.*projection/)) return "data_preprocessing"; + if (has(/scan.*projection/)) return "data_loading"; + if (has(/regex.*filter/) || has(/filter.*regex/)) return "text_filtering"; + if (has(/wordcloud/) || has(/word_count/) || has(/tokeniz/)) return "text_analysis"; + if (has(/filter.*projection/) || has(/projection.*filter/)) return "data_cleaning"; + if (has(/hashjoin.*projection/) || has(/cartesian.*projection/) || has(/union.*projection/)) { + return "joined_enrichment"; + } + if (has(/aggregate.*projection/) || has(/aggregate.*filter/) || has(/groupby.*projection/)) { + return "metric_summary"; + } + if (has(/aggregate/) || has(/groupby/)) return "aggregation_block"; + if (has(/piechart/) || has(/barchart/) || has(/linechart/) || has(/scatter/)) { + return "chart_pipeline"; + } + if (has(/normalizer/) || has(/standardize/) || has(/imputer/)) return "feature_normalization"; + if (has(/sklearn.*trainer/) || has(/sklearn.*testing/)) return "ml_train_eval"; + if (has(/pythonudf/) && has(/projection/)) return "udf_pipeline"; + return undefined; + } + private scoreChain( chain: string[], ops: readonly OperatorPredicate[], @@ -361,8 +442,25 @@ export class MacroSuggestionService { private suggestedNameForChain(chain: string[], ops: readonly OperatorPredicate[]): string { const types = chain.map(id => ops.find(o => o.operatorID === id)?.operatorType ?? "Op"); - // Compact 2-3 of the type names into a snake-cased candidate name. + return this.nameFromTypes(types); + } + + /** + * Public helper for callers outside the suggester (e.g. the right-click + * "create macro" flow) that want the SAME smart default name the + * suggester panel would produce — so manually-created and AI-suggested + * macros land in the palette with consistent naming. + */ + public smartNameFromTypes(operatorTypes: readonly string[]): string { + return this.nameFromTypes(operatorTypes); + } + + private nameFromTypes(types: readonly string[]): string { + const sig = types.join("_").toLowerCase(); + const domain = this.domainAwareName(sig); + if (domain) return domain; + // Fallback: compact 2-3 of the type names into a snake-cased candidate. const condensed = types.slice(0, Math.min(3, types.length)).map(t => t.replace(/OpDesc$|Op$/, "")); - return condensed.join("_").toLowerCase() + (chain.length > 3 ? "_block" : ""); + return condensed.join("_").toLowerCase() + (types.length > 3 ? "_block" : ""); } } diff --git a/frontend/src/app/workspace/service/macro/macro.service.ts b/frontend/src/app/workspace/service/macro/macro.service.ts index d11b606907e..934d73a49c5 100644 --- a/frontend/src/app/workspace/service/macro/macro.service.ts +++ b/frontend/src/app/workspace/service/macro/macro.service.ts @@ -19,7 +19,8 @@ import { HttpClient } from "@angular/common/http"; import { Injectable } from "@angular/core"; -import { Observable, ReplaySubject, of, shareReplay } from "rxjs"; +import * as dagre from "dagre"; +import { BehaviorSubject, Observable, ReplaySubject, of, shareReplay } from "rxjs"; import { tap, map, catchError } from "rxjs/operators"; import { AppSettings } from "../../../common/app-setting"; import { ExecutionMode, Workflow, WorkflowContent } from "../../../common/type/workflow"; @@ -313,6 +314,15 @@ export class MacroService { // runtimeMacroMapping is refreshed. Lets the stats consumer look up // "all runtime ops under macro X" in O(1). private runtimeOpsByMacroInstance = new Map(); + // Subscribers (e.g. result-panel drill-down alias, status aggregator) can + // re-emit when the runtime macro-mapping is refreshed. Tick is opaque — + // consumers just need to know "the mapping changed, re-read it now." + private runtimeMacroMappingTick = new BehaviorSubject(0); + + /** Stream that ticks whenever the runtime-mapping cache is refreshed. */ + public getRuntimeMacroMappingTick(): Observable { + return this.runtimeMacroMappingTick.asObservable(); + } /** * Fetch the macro-instance provenance map for the most-recent compile of @@ -354,6 +364,11 @@ export class MacroService { } } this.runtimeMacroMappingLoadedFor = wid; + // Tick so downstream subscribers (drill-down alias, stats roll-up) + // can re-read with the now-populated cache. Required because the + // initial render typically happens BEFORE this fetch completes; we + // need to nudge them once the data lands. + this.runtimeMacroMappingTick.next(this.runtimeMacroMappingTick.value + 1); return this.runtimeMacroMapping; }), catchError(() => { @@ -544,6 +559,7 @@ export class MacroService { inputBindings: MacroPortBinding[]; outputBindings: MacroPortBinding[]; nestedMacros: Map; + innerSinks: string[]; }, accumulatedChain: string[], isInput: boolean @@ -633,6 +649,7 @@ export class MacroService { inputBindings: MacroPortBinding[]; outputBindings: MacroPortBinding[]; nestedMacros: Map; + innerSinks: string[]; }> >(); // Latest-known synchronous snapshot — populated by `getBindingsForInstance` @@ -644,6 +661,7 @@ export class MacroService { inputBindings: MacroPortBinding[]; outputBindings: MacroPortBinding[]; nestedMacros: Map; + innerSinks: string[]; } >(); @@ -995,12 +1013,18 @@ export class MacroService { inputBindings: MacroPortBinding[]; outputBindings: MacroPortBinding[]; nestedMacros: Map; + innerSinks: string[]; }> { const cached = this.bodyBindingsCache.get(macroId); if (cached) return cached; const widNum = Number(macroId); if (!Number.isFinite(widNum)) { - const empty = { inputBindings: [], outputBindings: [], nestedMacros: new Map() }; + const empty = { + inputBindings: [], + outputBindings: [], + nestedMacros: new Map(), + innerSinks: [], + }; this.bodyBindingsSnapshot.set(macroId, empty); return of(empty); } @@ -1020,6 +1044,7 @@ export class MacroService { inputBindings: [] as MacroPortBinding[], outputBindings: [] as MacroPortBinding[], nestedMacros: new Map(), + innerSinks: [] as string[], }) ), shareReplay(1) @@ -1071,6 +1096,7 @@ export class MacroService { inputBindings: MacroPortBinding[]; outputBindings: MacroPortBinding[]; nestedMacros: Map; + innerSinks: string[]; }, binding: MacroPortBinding, isInput: boolean @@ -1144,16 +1170,35 @@ export class MacroService { // stats without holding a reference to WorkflowActionService. this.registerMacroInstance(instanceId, macroId); this.getBodyBindings(macroId).subscribe({ - next: () => { + next: snapshot => { // After the first-level bindings load, ask for the recursive // resolved bindings — `getBindingsForInstance` chains through any // nested macros automatically. Output port 0 might resolve to a // single terminal inner op, OR (in the rare fan-out case) several; // for the v1 macro-result alias we still pick the first terminal. const resolved = this.getBindingsForInstance(instanceId, macroId); - if (!resolved) return; - const out0 = resolved.outputBindings.find(b => b.externalPortIndex === 0); - if (out0) this.workflowResultService.setMacroResultAlias(instanceId, out0.innerOpId); + const out0 = resolved?.outputBindings.find(b => b.externalPortIndex === 0); + if (out0) { + this.workflowResultService.setMacroResultAlias(instanceId, out0.innerOpId); + return; + } + // Mega-macro fallback: macro has 0 external outputs but its body may + // contain sinks (e.g. CSVFileSink, SimpleSink for "View Results"). + // Engine auto-stores every terminal op's output (see + // WorkflowCompiler.expandLogicalPlan), so the sink's result IS + // materialized — clicking the macro op directly should reveal it. + // We pick the first body sink and resolve it to its runtime UUID via + // the macro-mapping cache. If the cache isn't populated yet (no Run + // has happened), this is a no-op; the tick-driven re-prefetch in the + // editor will re-run after the mapping fetch lands. + if (snapshot.innerSinks.length === 0) return; + const primarySinkBodyId = snapshot.innerSinks[0]; + for (const [runtimeUuid, prov] of this.runtimeMacroMapping.entries()) { + if (prov.bodyOpId !== primarySinkBodyId) continue; + if (prov.macroChain[prov.macroChain.length - 1] !== instanceId) continue; + this.workflowResultService.setMacroResultAlias(instanceId, runtimeUuid); + return; + } }, error: () => undefined, }); @@ -1164,12 +1209,13 @@ export class MacroService { inputBindings: MacroPortBinding[]; outputBindings: MacroPortBinding[]; nestedMacros: Map; + innerSinks: string[]; } { let body: MacroBody; try { body = JSON.parse(detail.content) as MacroBody; } catch { - return { inputBindings: [], outputBindings: [], nestedMacros: new Map() }; + return { inputBindings: [], outputBindings: [], nestedMacros: new Map(), innerSinks: [] }; } const inputMarkerByPortIndex = new Map(); const outputMarkerByPortIndex = new Map(); @@ -1177,6 +1223,12 @@ export class MacroService { // macroId we'll need to recursively resolve through. Keyed by the body- // relative operatorID since that's how the markers' links reference it. const nestedMacros = new Map(); + // Inner sinks (body-relative IDs). Used as fallback result-alias targets + // when the macro has 0 output ports: a "mega-macro" whose body contains + // sinks but exposes nothing externally still wants its sink output to be + // viewable in the result panel by clicking the macro op directly, + // instead of forcing the user to drill in. + const innerSinks: string[] = []; for (const raw of body.operators) { const op = raw as { operatorID?: string; @@ -1191,6 +1243,11 @@ export class MacroService { outputMarkerByPortIndex.set(op.portIndex, op.operatorID); } else if (op.operatorType === "Macro" && typeof op.macroId === "string" && op.macroId.length > 0) { nestedMacros.set(op.operatorID, op.macroId); + } else if ( + typeof op.operatorType === "string" && + op.operatorType.toLowerCase().includes("sink") + ) { + innerSinks.push(op.operatorID); } } const markerIds = new Set([ @@ -1226,7 +1283,7 @@ export class MacroService { }); } } - return { inputBindings, outputBindings, nestedMacros }; + return { inputBindings, outputBindings, nestedMacros, innerSinks }; } /** @@ -1492,7 +1549,10 @@ export class MacroService { const body = JSON.parse(detail.content) as MacroBody; const operators = body.operators.map(raw => this.normalizeBodyOperator(raw)); - const operatorPositions = this.autoLayoutMacroBody(operators); + const operatorPositions = this.autoLayoutMacroBody( + operators, + body.links.map(l => ({ fromOpId: l.fromOpId, toOpId: l.toOpId })) + ); const links = body.links .map(ml => this.macroLinkToOperatorLink(ml, operators)) .filter((l): l is OperatorLink => l !== null); @@ -1587,33 +1647,65 @@ export class MacroService { } /** - * Place MacroInput markers on the left, MacroOutput markers on the right, - * and everything else in a middle column. Sufficient for visual - * inspection; a proper layout pass is a follow-up. + * Auto-layout the macro body using dagre's directed-graph algorithm — the + * same engine the main canvas's "Auto-layout" button uses. Edges come from + * the body's link list so connected ops sit at logical ranks; MacroInput + * markers act as source ranks (left edge) and MacroOutput markers as sink + * ranks (right edge). Settings mirror `JointGraphWrapper.autoLayoutJoint` + * for consistency between parent canvas and macro-body view. + * + * Why dagre, not the manual 3-column layout it replaces: the previous + * placeholder put every middle op in a vertical stack, which made + * non-linear bodies (joins, fan-outs) look like spaghetti. With dagre, + * a Filter→Projection→Join body lays out naturally with the join's two + * inputs side-by-side. + * + * We use dagre directly (not `joint.layout.DirectedGraph.layout`) because + * at this point the body operators haven't been rendered into JointJS + * cells yet — we're computing the positions that will be passed into + * `WorkflowContent.operatorPositions` on the first drill-down load. */ - private autoLayoutMacroBody(operators: OperatorPredicate[]): { [id: string]: Point } { - const xLeft = 100; - const xMiddle = 450; - const xRight = 800; - const ySpacing = 120; - const ySeen = { left: 0, middle: 0, right: 0 }; - const positions: { [id: string]: Point } = {}; + private autoLayoutMacroBody( + operators: OperatorPredicate[], + links: { fromOpId: string; toOpId: string }[] + ): { [id: string]: Point } { + if (operators.length === 0) return {}; + // Use dagre's bundled graphlib constructor so the types line up cleanly + // with `dagre.layout(g)` below. `@types/graphlib` and `@types/dagre` are + // independent packages whose Graph definitions don't unify directly. + const g = new dagre.graphlib.Graph(); + g.setGraph({ + nodesep: 100, + edgesep: 150, + ranksep: 80, + ranker: "tight-tree", + rankdir: "LR", + }); + g.setDefaultEdgeLabel(() => ({})); + // Approximate node size — close enough to a typical Texera operator card. + // Dagre uses these for collision avoidance + edge routing only; the actual + // rendered op size is fixed by the joint shape, so we don't need pixel + // accuracy here. + const NODE_W = 160; + const NODE_H = 60; operators.forEach(op => { - let column: keyof typeof ySeen; - let x: number; - if (op.operatorType === "MacroInput") { - column = "left"; - x = xLeft; - } else if (op.operatorType === "MacroOutput") { - column = "right"; - x = xRight; - } else { - column = "middle"; - x = xMiddle; + g.setNode(op.operatorID, { width: NODE_W, height: NODE_H }); + }); + links.forEach(l => { + // dagre tolerates edges to/from unknown nodes silently, but we filter + // anyway — body links can reference markers we haven't normalized into + // operators in pathological cases. + if (g.hasNode(l.fromOpId) && g.hasNode(l.toOpId)) { + g.setEdge(l.fromOpId, l.toOpId); } - const y = 100 + ySeen[column] * ySpacing; - ySeen[column] += 1; - positions[op.operatorID] = { x, y }; + }); + dagre.layout(g); + const positions: { [id: string]: Point } = {}; + g.nodes().forEach(id => { + const node: { x: number; y: number } = g.node(id); + // dagre returns the CENTER of the node; joint expects the TOP-LEFT. + // Subtract half the width/height. + positions[id] = { x: node.x - NODE_W / 2, y: node.y - NODE_H / 2 }; }); return positions; } diff --git a/frontend/src/app/workspace/service/workflow-result/workflow-result.service.ts b/frontend/src/app/workspace/service/workflow-result/workflow-result.service.ts index 6bcc9ea82e1..2b9158ed9c2 100644 --- a/frontend/src/app/workspace/service/workflow-result/workflow-result.service.ts +++ b/frontend/src/app/workspace/service/workflow-result/workflow-result.service.ts @@ -97,21 +97,28 @@ export class WorkflowResultService { } // When the canvas is rendering a macro body (drill-down view), the operators - // on the canvas have body-relative IDs (`Filter-uuid`) but engine results - // arrive keyed by the post-expansion runtime ID (`${instanceId}--Filter-uuid`). - // Set this prefix to make every result lookup transparently target the - // runtime ID. Empty string disables the rewrite. - private drilldownPrefix: string = ""; - - public setDrilldownPrefix(prefix: string): void { - this.drilldownPrefix = prefix; + // on the canvas have body-relative IDs (e.g. `Filter-operator-xyz` from the + // macro definition) but engine results arrive keyed by the post-expansion + // runtime UUID assigned by MacroExpander. This map (body-op-id → runtime- + // UUID) is populated by the workflow-editor when entering a drill-down view; + // empty means no drill-down rewrite is active. + // + // The old prefix-based scheme (`${instanceId}--${bodyOpId}`) no longer works + // because MacroExpander switched to fresh deterministic UUIDs (see + // backend/MacroExpander.spliceIntoParent for why long prefixed names had to + // go). The map is computed via MacroService's runtime-mapping cache. + private drilldownAliases: Map = new Map(); + + public setDrilldownAliases(aliases: Map): void { + this.drilldownAliases = aliases; } private resolveAlias(operatorID: string): string { // Drill-down rewrite wins: when viewing a macro body during execution we - // want the body-relative op ID lifted to its runtime form. Macro aliases + // want the body-relative op ID lifted to its runtime UUID. Macro aliases // only fire on the outer canvas, where body-relative IDs aren't present. - if (this.drilldownPrefix.length > 0) return `${this.drilldownPrefix}${operatorID}`; + const drill = this.drilldownAliases.get(operatorID); + if (drill !== undefined) return drill; return this.macroResultAliases.get(operatorID) ?? operatorID; } diff --git a/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts b/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts index b61d2142928..da27ed2d292 100644 --- a/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts +++ b/frontend/src/app/workspace/service/workflow-status/workflow-status.service.ts @@ -18,7 +18,7 @@ */ import { Injectable } from "@angular/core"; -import { Observable, Subject } from "rxjs"; +import { Observable, ReplaySubject } from "rxjs"; import { OperatorState, OperatorStatistics } from "../../types/execute-workflow.interface"; import { WorkflowWebsocketService } from "../workflow-websocket/workflow-websocket.service"; import { MacroService } from "../macro/macro.service"; @@ -128,9 +128,22 @@ function withMacroAggregates( providedIn: "root", }) export class WorkflowStatusService { - // status is responsible for passing websocket responses to other components - private statusSubject = new Subject>(); + // status is responsible for passing websocket responses to other components. + // ReplaySubject(1) so late subscribers (e.g. the canvas after a route-driven + // remount) immediately receive the latest aggregated snapshot instead of + // having to wait for the next websocket event — without this the canvas + // could render with no op stats until execution kicks off again. + private statusSubject = new ReplaySubject>(1); private currentStatus: Record = {}; + // Last-seen raw (pre-aggregation) snapshot. We hold onto this so we can + // re-emit through `withMacroAggregates` when the macro mapping cache later + // populates — this happens whenever the user lands on a workflow with a + // completed run (websocket replays stats first, macro-mapping HTTP arrives + // moments later). Without this re-emit, macro ops on canvas would show the + // first emission's unaggregated raw entries (so macro ops appear blank + // until a brand new stats event arrives, which often never does on a + // finished run). + private lastRawStatus: Record | undefined; constructor( private workflowWebsocketService: WorkflowWebsocketService, @@ -142,8 +155,21 @@ export class WorkflowStatusService { if (event.type !== "OperatorStatisticsUpdateEvent") { return; } + this.lastRawStatus = event.operatorStatistics; this.statusSubject.next(withMacroAggregates(event.operatorStatistics, this.macroService)); }); + + // Re-aggregate when the runtime macro mapping is (re-)fetched. Required + // for the hard-reload-to-parent-canvas flow: the websocket replays stats + // BEFORE refreshRuntimeMacroMapping(wid) lands, so the first aggregation + // pass has no macros to find. After the mapping fills in, we re-run + // aggregation against the cached raw status so canvas macro ops get their + // rolled-up entries. + this.macroService.getRuntimeMacroMappingTick().subscribe(() => { + if (this.lastRawStatus) { + this.statusSubject.next(withMacroAggregates(this.lastRawStatus, this.macroService)); + } + }); } public getStatusUpdateStream(): Observable> { From 006ddba8210274104e63d25f0d65f60c964cc2d5 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 13:15:30 -0700 Subject: [PATCH 64/65] fix(macro): tolerate UI-only `estimatedSpeedup` on MacroFusion Same shape of bug as the macroSyncedAt fix on MacroOpDesc: the frontend stamps `estimatedSpeedup` ("1.6x") onto the fusion payload so the canvas can render it next to the FUSED badge, but the backend MacroFusion case class doesn't model that field. Jackson rejects the WorkflowExecuteRequest at execute time once the fused macro is part of the run. Annotate `MacroFusion` with `@JsonIgnoreProperties(ignoreUnknown = true)` so this and any future UI-only convenience fields don't break the round trip. Backend MacroExpander only ever reads `verified` to decide whether to substitute the UDF. Co-Authored-By: Claude Opus 4.7 (1M context) --- .pr-description.md | 47 +++++++++++++++++++ .../amber/operator/macroOp/MacroFusion.scala | 9 ++++ 2 files changed, 56 insertions(+) create mode 100644 .pr-description.md diff --git a/.pr-description.md b/.pr-description.md new file mode 100644 index 00000000000..aeeca38ca39 --- /dev/null +++ b/.pr-description.md @@ -0,0 +1,47 @@ +# PR Description: AI-Augmented Macro Operators + +## What changes were proposed in this PR? + +Introduces **macro operators** — a logical-plan-level abstraction for encapsulating reusable sub-DAGs as single nodes on the canvas — together with the AI surfaces that make them practical: a suggestion panel that scans the current workflow for refactor opportunities, a one-click "fuse for performance" path that collapses a macro body into a single Python UDF, and a drill-down editor for inspecting / editing a macro body while live execution stats keep flowing. + +### Surface + +* **Right-click → "Create macro"** swaps a selected sub-DAG with a single `Macro` op on the parent canvas. The body is persisted as a separate `MACRO`-kind workflow row; references the persisted body via `(macroId, macroVersion)` in `LIVE` link mode (`SNAPSHOT` mode also supported for portable bodies). +* **"Your Macros" palette section** lists every macro the user has saved; click → instantiate, hover → preview, right-click → export/import as portable JSON bundles (transitive — nested macros travel with their parent). +* **"Suggest Macros (AI)" panel** (`MacroSuggestionService`) ranks two heuristic candidate sources side-by-side: linear chains, and recurring `(opType₁, opType₂, …)` patterns across the workflow. Recurring patterns auto-tier as `✓ recommended` because duplicated logic is the strongest "extract as macro" signal. Names are domain-aware (`csv_preprocessing`, `text_filtering`, `metric_summary`, `joined_enrichment`, `ml_train_eval`, …) rather than underscore-joined op types. +* **Right-click → "fuse for performance (AI)"** (`MacroFusionService`) emits a `PythonUDFOpDescV2`-compatible `process_tuple` function from the macro body. Covers Filter, Projection, Regex, Limit, Distinct, and inlined PythonUDFV2 (yield-rewritten). Marks `fusion.verified = true` so `MacroExpander` substitutes a single UDF for the inlined body at compile time. Visual: solid gold stroke + `⚡ FUSED · 1.6×` badge on canvas. Speedup is grounded in the handoff-removal model (N−1 internal actor boundaries collapsed; conservative ×0.30 per handoff, capped at ×4). +* **Drill-down editor** — double-clicking a macro op routes to `/dashboard/user/workflow/{wid}/macro/{macroId}?instance=…` and renders the body on a child canvas. The body is laid out with dagre using the same settings as the main canvas's auto-layout. Live execution stats (incl. row counts per port) flow into the drill-down view via a `(body-op-id → runtime-UUID)` map sourced from `MacroService`'s runtime-mapping cache. + +### Architecture + +* **Macros live at the logical-plan layer only.** `MacroExpander` (mirrored in `amber/` and `workflow-compiling-service/`) runs a pre-compile pass that inlines every `MacroOpDesc` into its body operators and rewrites edges, so the physical-plan layer never sees a macro. `WorkflowCompiler` calls `MacroExpander.expand` before `expandLogicalPlan`. +* **Deterministic UUIDs for inner ops.** The expander assigns each inner op a fresh ID via `UUID.nameUUIDFromBytes("${macroInstanceId}|${originalBodyOpId}")`. Required because (a) the long `${instanceId}--${innerOp}` prefix scheme produced 170+ char IDs that caused Iceberg commit thrash on HashJoin's internal build-side port, and (b) Texera has two compilers — frontend validation and execution-time — that must produce bit-identical plans, otherwise the macro-mapping side-table written by one wouldn't match the runtime stats emitted by the other. +* **Provenance side-table.** `MacroExpander` populates a `Map[runtimeOpId → MacroProvenance(macroChain, bodyOpId)]` during expansion; `WorkflowCompiler` drains it after compile and stores it in `MacroMappingCache` (file-backed at `/tmp/texera-macro-mappings/wid-{wid}.json` for cross-JVM visibility between `ComputingUnitMaster` and `TexeraWebApplication`). Exposed via `GET /api/workflow/{wid}/macro-mapping`. Frontend `WorkflowStatusService.withMacroAggregates` walks the chain to roll inner-op stats up to every macro level (parent canvas + nested drill-downs). +* **Nested macros are fully supported** — the chain stored per runtime op is `[outerInstanceId, innerInstanceId, …]`; the resolver suffix-matches the chain so a stats-roll-up rooted at an inner macro still finds its runtime ops. + +### What this PR also fixes (along the way) + +* **View-result inside a macro** — drill-down result lookups now go body-relative-id → runtime-UUID via `MacroService.buildBodyOpIdToRuntimeUuidMap` (replaces the obsolete prefix-based alias). Mega-macros with 0 external outputs alias the canvas op to the first body sink, so the auto-stored terminal output is reachable without drilling. +* **Back-to-parent stats** — `WorkflowStatusService` re-aggregates the cached raw status on every `runtimeMacroMappingTick`, and its emission Subject becomes `ReplaySubject(1)` so the canvas remount after navigation gets the latest snapshot immediately. +* **Jackson `macroSyncedAt` UnrecognizedPropertyException** at execute time — `MacroOpDesc` annotated with `@JsonIgnoreProperties(ignoreUnknown = true)` so UI-only fields the frontend stamps onto operatorProperties don't break deserialization. +* **`/api/macro/*` HTTP storm** — lazy fetches inside template bindings caused an infinite loop; reverted to a flat palette and removed the lazy fetches. +* **Engine error visibility** — phase-transition errors and missing-schema-port errors now propagate out of `RegionExecutionCoordinator` instead of stalling silently. + +## Any related issues, documentation, discussions? + +Hackathon submission. Builds on §9.2 of the macro design doc (AI fusion substitution path). + +## How was this PR tested? + +* **`MacroExpanderSpec`** (~694 lines) covers the expander on its own: single-macro expansion, nested expansion (outer chain + inner chain), input fan-out (single external port → multiple inner consumers), output fan-in detection (raises), cycle detection across nested macros, depth-limit guard, deterministic-UUID property (same input → same output across compiler instances), and the provenance side-table population. +* **`MacroOpDescSpec`** covers Jackson serialization round-trip (incl. tolerance of unknown frontend-only fields like `macroSyncedAt`). +* **Demo runbook** — exercised the full path end-to-end on a real multi-macro workflow including nested macros, view-result on inner sinks, fuse + unfuse, drill-down navigation in/out, and a "mega-macro" (entire workflow wrapped). Stats roll up correctly at every nesting level and the canvas remount after navigation no longer wipes non-macro op states. + +``` +sbt "WorkflowExecutionService/testOnly *MacroExpanderSpec" +sbt "WorkflowOperator/testOnly *MacroOpDescSpec" +``` + +## Was this PR authored or co-authored using generative AI tooling? + +Generated-by: Claude Code (Claude Opus 4.7) diff --git a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroFusion.scala b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroFusion.scala index 94a3fb1ce1d..9a8e2ffa9ed 100644 --- a/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroFusion.scala +++ b/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/macroOp/MacroFusion.scala @@ -19,10 +19,19 @@ package org.apache.texera.amber.operator.macroOp +import com.fasterxml.jackson.annotation.JsonIgnoreProperties + // AI-fusion payload (Section 9.2). When `verified = true`, MacroExpander substitutes // the MacroOpDesc with a single PythonUDFOpDescV2 built from `code` instead of inlining // the macro body. `sampleSize` records how many rows the sample-run diff matched on; // `verifiedAt` is the epoch millis when verification passed. +// +// `ignoreUnknown = true`: the frontend attaches UI-only fields (e.g. +// `estimatedSpeedup`, a human-readable "1.6×" used to render the on-canvas +// ⚡ FUSED badge) onto this payload before persisting. The backend doesn't +// model those fields here; without this annotation Jackson rejects the +// whole WorkflowExecuteRequest at execute time. +@JsonIgnoreProperties(ignoreUnknown = true) case class MacroFusion( code: String, verified: Boolean = false, From e360aa6d2965da334611fbe742add3c06284b928 Mon Sep 17 00:00:00 2001 From: Xiaozhen Liu Date: Sat, 16 May 2026 13:16:00 -0700 Subject: [PATCH 65/65] chore: drop .pr-description.md from tracked tree MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scratch file used to draft the hackathon PR description — not part of the project. Mistakenly committed in the previous change; remove it from the tracked tree and keep it locally for the PR-open step. --- .pr-description.md | 47 ---------------------------------------------- 1 file changed, 47 deletions(-) delete mode 100644 .pr-description.md diff --git a/.pr-description.md b/.pr-description.md deleted file mode 100644 index aeeca38ca39..00000000000 --- a/.pr-description.md +++ /dev/null @@ -1,47 +0,0 @@ -# PR Description: AI-Augmented Macro Operators - -## What changes were proposed in this PR? - -Introduces **macro operators** — a logical-plan-level abstraction for encapsulating reusable sub-DAGs as single nodes on the canvas — together with the AI surfaces that make them practical: a suggestion panel that scans the current workflow for refactor opportunities, a one-click "fuse for performance" path that collapses a macro body into a single Python UDF, and a drill-down editor for inspecting / editing a macro body while live execution stats keep flowing. - -### Surface - -* **Right-click → "Create macro"** swaps a selected sub-DAG with a single `Macro` op on the parent canvas. The body is persisted as a separate `MACRO`-kind workflow row; references the persisted body via `(macroId, macroVersion)` in `LIVE` link mode (`SNAPSHOT` mode also supported for portable bodies). -* **"Your Macros" palette section** lists every macro the user has saved; click → instantiate, hover → preview, right-click → export/import as portable JSON bundles (transitive — nested macros travel with their parent). -* **"Suggest Macros (AI)" panel** (`MacroSuggestionService`) ranks two heuristic candidate sources side-by-side: linear chains, and recurring `(opType₁, opType₂, …)` patterns across the workflow. Recurring patterns auto-tier as `✓ recommended` because duplicated logic is the strongest "extract as macro" signal. Names are domain-aware (`csv_preprocessing`, `text_filtering`, `metric_summary`, `joined_enrichment`, `ml_train_eval`, …) rather than underscore-joined op types. -* **Right-click → "fuse for performance (AI)"** (`MacroFusionService`) emits a `PythonUDFOpDescV2`-compatible `process_tuple` function from the macro body. Covers Filter, Projection, Regex, Limit, Distinct, and inlined PythonUDFV2 (yield-rewritten). Marks `fusion.verified = true` so `MacroExpander` substitutes a single UDF for the inlined body at compile time. Visual: solid gold stroke + `⚡ FUSED · 1.6×` badge on canvas. Speedup is grounded in the handoff-removal model (N−1 internal actor boundaries collapsed; conservative ×0.30 per handoff, capped at ×4). -* **Drill-down editor** — double-clicking a macro op routes to `/dashboard/user/workflow/{wid}/macro/{macroId}?instance=…` and renders the body on a child canvas. The body is laid out with dagre using the same settings as the main canvas's auto-layout. Live execution stats (incl. row counts per port) flow into the drill-down view via a `(body-op-id → runtime-UUID)` map sourced from `MacroService`'s runtime-mapping cache. - -### Architecture - -* **Macros live at the logical-plan layer only.** `MacroExpander` (mirrored in `amber/` and `workflow-compiling-service/`) runs a pre-compile pass that inlines every `MacroOpDesc` into its body operators and rewrites edges, so the physical-plan layer never sees a macro. `WorkflowCompiler` calls `MacroExpander.expand` before `expandLogicalPlan`. -* **Deterministic UUIDs for inner ops.** The expander assigns each inner op a fresh ID via `UUID.nameUUIDFromBytes("${macroInstanceId}|${originalBodyOpId}")`. Required because (a) the long `${instanceId}--${innerOp}` prefix scheme produced 170+ char IDs that caused Iceberg commit thrash on HashJoin's internal build-side port, and (b) Texera has two compilers — frontend validation and execution-time — that must produce bit-identical plans, otherwise the macro-mapping side-table written by one wouldn't match the runtime stats emitted by the other. -* **Provenance side-table.** `MacroExpander` populates a `Map[runtimeOpId → MacroProvenance(macroChain, bodyOpId)]` during expansion; `WorkflowCompiler` drains it after compile and stores it in `MacroMappingCache` (file-backed at `/tmp/texera-macro-mappings/wid-{wid}.json` for cross-JVM visibility between `ComputingUnitMaster` and `TexeraWebApplication`). Exposed via `GET /api/workflow/{wid}/macro-mapping`. Frontend `WorkflowStatusService.withMacroAggregates` walks the chain to roll inner-op stats up to every macro level (parent canvas + nested drill-downs). -* **Nested macros are fully supported** — the chain stored per runtime op is `[outerInstanceId, innerInstanceId, …]`; the resolver suffix-matches the chain so a stats-roll-up rooted at an inner macro still finds its runtime ops. - -### What this PR also fixes (along the way) - -* **View-result inside a macro** — drill-down result lookups now go body-relative-id → runtime-UUID via `MacroService.buildBodyOpIdToRuntimeUuidMap` (replaces the obsolete prefix-based alias). Mega-macros with 0 external outputs alias the canvas op to the first body sink, so the auto-stored terminal output is reachable without drilling. -* **Back-to-parent stats** — `WorkflowStatusService` re-aggregates the cached raw status on every `runtimeMacroMappingTick`, and its emission Subject becomes `ReplaySubject(1)` so the canvas remount after navigation gets the latest snapshot immediately. -* **Jackson `macroSyncedAt` UnrecognizedPropertyException** at execute time — `MacroOpDesc` annotated with `@JsonIgnoreProperties(ignoreUnknown = true)` so UI-only fields the frontend stamps onto operatorProperties don't break deserialization. -* **`/api/macro/*` HTTP storm** — lazy fetches inside template bindings caused an infinite loop; reverted to a flat palette and removed the lazy fetches. -* **Engine error visibility** — phase-transition errors and missing-schema-port errors now propagate out of `RegionExecutionCoordinator` instead of stalling silently. - -## Any related issues, documentation, discussions? - -Hackathon submission. Builds on §9.2 of the macro design doc (AI fusion substitution path). - -## How was this PR tested? - -* **`MacroExpanderSpec`** (~694 lines) covers the expander on its own: single-macro expansion, nested expansion (outer chain + inner chain), input fan-out (single external port → multiple inner consumers), output fan-in detection (raises), cycle detection across nested macros, depth-limit guard, deterministic-UUID property (same input → same output across compiler instances), and the provenance side-table population. -* **`MacroOpDescSpec`** covers Jackson serialization round-trip (incl. tolerance of unknown frontend-only fields like `macroSyncedAt`). -* **Demo runbook** — exercised the full path end-to-end on a real multi-macro workflow including nested macros, view-result on inner sinks, fuse + unfuse, drill-down navigation in/out, and a "mega-macro" (entire workflow wrapped). Stats roll up correctly at every nesting level and the canvas remount after navigation no longer wipes non-macro op states. - -``` -sbt "WorkflowExecutionService/testOnly *MacroExpanderSpec" -sbt "WorkflowOperator/testOnly *MacroOpDescSpec" -``` - -## Was this PR authored or co-authored using generative AI tooling? - -Generated-by: Claude Code (Claude Opus 4.7)