> For the complete documentation index, see [llms.txt](https://docs.umh.app/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.umh.app/usage/data-flows/stream-processor.md).

# Stream Processors

> **Prerequisite:** Understand [Data Flow concepts](https://docs.umh.app/usage/data-flows.md) and complete the [Getting Started guide](https://docs.umh.app/getting-started.md).

Stream processors transform data already inside the Unified Namespace, aggregating multiple device streams into business KPIs and metrics. They're the third type of data flow, specialized for transforming device models into business models.

## When to Use

Use stream processors for:

* Aggregating data from multiple devices into KPIs
* Creating business metrics from raw sensor data
* Calculating OEE, efficiency, or other derived values
* Reducing data volume for cloud transmission

Use bridges instead for:

* Getting data into the UNS initially
* Single device to model mapping
* Direct protocol connections

## UI Capabilities

| Action               | Available | Notes                                   |
| -------------------- | --------- | --------------------------------------- |
| Create processors    | ✅         | Visual configuration with topic browser |
| Select source topics | ✅         | Browse and multi-select                 |
| Map to model fields  | ✅         | Expression-based mapping                |
| View processors      | ✅         | Listed in Stream tab                    |
| Monitor throughput   | ✅         | Real-time message rates                 |
| Edit processors      | ✅         | Modify configuration                    |
| Delete processors    | ✅         | Remove when not needed                  |
| Code mode            | ✅         | Direct YAML editing                     |

## Creating a Stream Processor

Stream processors are created through the **Data Flows → Stream** tab:

![Stream Processors List](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-03fc326d1c5873a8f831005020af9d79b08bf9f3%2Fstream-processors.png?alt=media\&token=ac5329a0-b5cc-4c7f-9ad6-8fca1b05b65e)

### Step 1: General Configuration

Configure basic settings and location hierarchy:

![Stream Processor General Configuration](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-dfe614cb5747aaf26c5486ea829185d2e3af00b4%2Fstream-processors-add.png?alt=media\&token=909fe1c4-01d7-4618-a4ca-963fcb260f44)

* **Name**: Unique identifier for your processor
* **Instance**: The UMH Core instance to run on
* **Data Contract**: Select the output model (creates validated structure)
* **Location Levels**: Set where in the hierarchy to output data (Level 0-4)

### Step 2: Topic Selection

Select source topics and they'll automatically get variable names:

![Topic Selection and Variable Names](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-28a43a394aa74c4da5fccd22d44260f7efe554cf%2Fstream-processors-add-topic-expression.png?alt=media\&token=fc471521-ddbf-4edd-8628-e50f16dd52f4)

* Browse available topics in your UNS
* Select multiple sources with checkboxes
* Variables auto-generated (e.g., `_10` for first topic, `_10_2` for second)
* Use "Show Selected Only" to filter view
* Copy variable names for use in expressions

### Step 3: Expression Mapping

Map source variables to model fields using expressions:

![Expression Mapping](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-28a43a394aa74c4da5fccd22d44260f7efe554cf%2Fstream-processors-add-topic-expression.png?alt=media\&token=fc471521-ddbf-4edd-8628-e50f16dd52f4)

* Each field in your data model appears on the right
* Enter expressions using the source variables
* Simple passthrough: `source_alias`
* Calculations: `_10 + _10_2` or more complex formulas
* The expressions support standard mathematical operations

### Step 4: Code Mode (Advanced)

Switch to Code Mode to see or edit the generated YAML configuration:

![Code Mode Configuration](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-d642500a9352c4efd35819c18b1ed78f7fbec1ba%2Fstream-processors-add-code-mode.png?alt=media\&token=de9d7caf-ebdb-47f4-8fbb-78a218d1e19c)

The UI automatically generates YAML with:

* **sources**: Your selected topics with location\_path templates
* **mapping**: Field-to-expression mappings
* Comments explaining the structure

You can switch between UI and Code Mode anytime to fine-tune configurations.

**For detailed configuration instructions**, see the comprehensive [Stream Processors guide](https://docs.umh.app/usage/data-modeling/stream-processors.md) in Data Modeling.

## How It Works

Stream processors subscribe to multiple UNS topics and output to a single model-based topic:

```mermaid
flowchart LR
    A["Multiple Device Model Topics<br/>(e.g., _pump_v1, _cnc_v1)"] --> B["Stream Processor<br/>(aggregation)"] --> C["Business Model Topic<br/>(e.g., _maintenance_v1)"]
```

Example: Combining temperature and pressure from different sensors into a pump model:

* Input: `enterprise.site._raw.temp_sensor_1`, `enterprise.site._raw.pressure_gauge_2`
* Output: `enterprise.site._pump_v1` with calculated efficiency

The system handles:

* Consumer group management for offset tracking
* Dependency-based processing (waits for all required inputs)
* Guaranteed output structure via data models

## Configuration (YAML)

While the UI is the primary way to create stream processors, they're stored as YAML:

```yaml
streamProcessor:
  - name: pump_efficiency_calc
    desiredState: active
    streamProcessorServiceConfig:
      config:
        model:
          name: pump
          version: v1
        sources:
          inlet_temp: enterprise.site._raw.inlet_temp
          outlet_temp: enterprise.site._raw.outlet_temp
        mapping:
          efficiency: "(outlet_temp - inlet_temp) / inlet_temp * 100"
```

See the [Stream Processors configuration guide](https://docs.umh.app/usage/data-modeling/stream-processors.md) for detailed YAML structure.

## Key Differences from Other Data Flows

| Data Flow Type        | Purpose                       | Input Source                     | Output           |
| --------------------- | ----------------------------- | -------------------------------- | ---------------- |
| **Bridges**           | Get data INTO the UNS         | External devices (PLCs, sensors) | UNS topics       |
| **Stream Processors** | Transform data WITHIN the UNS | UNS topics                       | UNS model topics |
| **Stand-alone Flows** | Custom processing             | Any source                       | Any destination  |

## Metrics

The Management Console shows each stream processor's message counts. Open the stream processor and select **Metrics** on the Monitoring card. The view shows messages received and sent, errors and latency. Port 8080, umh-core's Prometheus endpoint, does not carry message counts. It carries the stream processor's state and error counts. [Metrics](https://docs.umh.app/usage/instances/agent/metrics.md) explains how to read them, which framework runs stream processors and what the labels and the state values mean.

Stream processors run on FSMv1 and use the standard [FSMv1 metrics](https://docs.umh.app/usage/instances/agent/metrics.md#fsmv1-metrics). For a stream processor named `oee-rollup`, the series carry these labels:

| Part                             | `component`                   | `instance`                                |
| -------------------------------- | ----------------------------- | ----------------------------------------- |
| The stream processor             | `stream_processor_instance`   | `oee-rollup`                              |
| The internal flow that runs it   | `dataflow_component_instance` | `dfc-streamprocessor-oee-rollup`          |
| The Benthos process of that flow | `benthos_instance`            | `dataflow-dfc-streamprocessor-oee-rollup` |

To alert when a stream processor has not been active for 5 minutes, use this rule with `for: 5m`. In Prometheus, write `exported_instance` for `instance`, or keep `instance` if the scrape job sets `honor_labels: true`:

```promql
umh_core_service_current_state{component="stream_processor_instance"} != 3 != 4 != 0
```

The rule does not fire for a stream processor that is stopped (0) or idle (4). A stopped one is one you stopped on purpose, and an idle one has no data to process.

When you delete a stream processor, umh-core keeps its last state series at -1 until the container restarts, so the rule keeps firing for it. Exclude the deleted name, for example by adding `exported_instance!="<deleted stream processor>"` to the selector, or restart the container.

The definitions are in [`pkg/metrics/metrics.go`](https://github.com/united-manufacturing-hub/united-manufacturing-hub/blob/staging/umh-core/pkg/metrics/metrics.go).

## Learn More

For comprehensive stream processor configuration and examples, see the [Stream Processors guide](https://docs.umh.app/usage/data-modeling/stream-processors.md) in Data Modeling documentation.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.umh.app/usage/data-flows/stream-processor.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
