Stream Processors

Prerequisite: Understand Data Flow concepts and complete the Getting Started guide.

Stream processors transform data already inside the Unified Namespace, aggregating multiple device streams into business KPIs and metrics. They're the third type of data flow, specialized for transforming device models into business models.

When to Use

Use stream processors for:

  • Aggregating data from multiple devices into KPIs

  • Creating business metrics from raw sensor data

  • Calculating OEE, efficiency, or other derived values

  • Reducing data volume for cloud transmission

Use bridges instead for:

  • Getting data into the UNS initially

  • Single device to model mapping

  • Direct protocol connections

UI Capabilities

Action
Available
Notes

Create processors

Visual configuration with topic browser

Select source topics

Browse and multi-select

Map to model fields

Expression-based mapping

View processors

Listed in Stream tab

Monitor throughput

Real-time message rates

Edit processors

Modify configuration

Delete processors

Remove when not needed

Code mode

Direct YAML editing

Creating a Stream Processor

Stream processors are created through the Data Flows → Stream tab:

Stream Processors List

Step 1: General Configuration

Configure basic settings and location hierarchy:

Stream Processor General Configuration
  • Name: Unique identifier for your processor

  • Instance: The UMH Core instance to run on

  • Data Contract: Select the output model (creates validated structure)

  • Location Levels: Set where in the hierarchy to output data (Level 0-4)

Step 2: Topic Selection

Select source topics and they'll automatically get variable names:

Topic Selection and Variable Names
  • Browse available topics in your UNS

  • Select multiple sources with checkboxes

  • Variables auto-generated (e.g., _10 for first topic, _10_2 for second)

  • Use "Show Selected Only" to filter view

  • Copy variable names for use in expressions

Step 3: Expression Mapping

Map source variables to model fields using expressions:

Expression Mapping
  • Each field in your data model appears on the right

  • Enter expressions using the source variables

  • Simple passthrough: source_alias

  • Calculations: _10 + _10_2 or more complex formulas

  • The expressions support standard mathematical operations

Step 4: Code Mode (Advanced)

Switch to Code Mode to see or edit the generated YAML configuration:

Code Mode Configuration

The UI automatically generates YAML with:

  • sources: Your selected topics with location_path templates

  • mapping: Field-to-expression mappings

  • Comments explaining the structure

You can switch between UI and Code Mode anytime to fine-tune configurations.

For detailed configuration instructions, see the comprehensive Stream Processors guide in Data Modeling.

How It Works

Stream processors subscribe to multiple UNS topics and output to a single model-based topic:

Multiple Device Model Topics → Stream Processor → Business Model Topic
(e.g., _pump_v1, _cnc_v1)        (aggregation)     (e.g., _maintenance_v1)

Example: Combining temperature and pressure from different sensors into a pump model:

  • Input: enterprise.site._raw.temp_sensor_1, enterprise.site._raw.pressure_gauge_2

  • Output: enterprise.site._pump_v1 with calculated efficiency

The system handles:

  • Consumer group management for offset tracking

  • Dependency-based processing (waits for all required inputs)

  • Guaranteed output structure via data models

Configuration (YAML)

While the UI is the primary way to create stream processors, they're stored as YAML:

streamprocessors:
  - name: pump_efficiency_calc
    model:
      name: pump
      version: v1
    sources:
      - topic: enterprise.site._raw.inlet_temp
      - topic: enterprise.site._raw.outlet_temp
    mapping:
      efficiency: "(outlet_temp - inlet_temp) / inlet_temp * 100"

See the Stream Processors configuration guide for detailed YAML structure.

Key Differences from Other Data Flows

Data Flow Type
Purpose
Input Source
Output

Bridges

Get data INTO the UNS

External devices (PLCs, sensors)

UNS topics

Stream Processors

Transform data WITHIN the UNS

UNS topics

UNS model topics

Stand-alone Flows

Custom processing

Any source

Any destination

Learn More

For comprehensive stream processor configuration and examples, see the Stream Processors guide in Data Modeling documentation.

Last updated