# Introduction

> **Looking for the old Kubernetes Helm stack?** See the [UMH Core vs UMH Classic FAQ](/umh-core-vs-classic-faq) to understand which edition fits your project and the current migration path.

## What is UMH Core?

UMH Core is a **single Docker container that turns any PC, VM, or edge gateway into an Industrial Data Hub**.

In manufacturing, every device speaks a different language - PLCs use OPC UA, sensors use Modbus, MES systems use REST APIs. Connecting them all creates a tangled mess of point-to-point integrations. Change one thing, break ten others.

UMH Core creates a **Unified Namespace (UNS)** - a central data backbone where all your industrial data lives in one organized, validated place. Instead of 100 devices talking to each other (creating 1000s of connections), they all publish to one place, and consumers subscribe to what they need.

### What's Inside

That one Docker container includes everything you need:

* **Redpanda** – an embedded, Kafka-compatible broker that buffers every message.
* **Benthos-UMH** – a stream-processor engine with 50+ industrial connectors.
* **Agent** – a Go service that reads `config.yaml`, launches pipelines, watches health, and phones home to the Management Console.
* **S6 Supervisor** – keeps every subprocess alive and starts them in the correct order.

### Why teams pick UMH Core

| Benefit         | What it means in practice                                                                                               |
| --------------- | ----------------------------------------------------------------------------------------------------------------------- |
| **Simple**      | PLCs, sensors, ERP/MES, cloud services all talk to **one Unified Namespace (UNS)** instead of point-to-point spaghetti. |
| **Lightweight** | Runs on almost everything                                                                                               |
| **No lock-in**  | 100 % open-source stack: Redpanda, Benthos, S6, and much more                                                           |

### Core Concepts You'll Learn

Through our getting-started guide, you'll understand:

* **Instance** – A running UMH Core container identified by its location path
* **Management Console** – Cloud UI for deploying and managing instances without touching YAML
* **Unified Namespace (UNS)** – The event-driven data backbone that eliminates point-to-point connections
* **Bridge** – The gateway for external data into the UNS (**the ONLY way data enters**)
* **Topic** – How data is addressed: `location.contract.virtual_path.tag_name`
* **Tag** – A time-series data point (like a PLC variable or sensor reading)
* **Virtual Path** – Folder organization within topics for grouping related data
* **Data Model** – Templates that define and validate data structure
* **Data Contract** – Validation rules (`_raw` = no validation, `_modelname_v1` = enforced structure)

Advanced concepts (after getting-started):

* **Stream Processor** – Transforms messages already inside the UNS (e.g., device models → business models)
* **Stand-alone Flow** – Point-to-point when UNS buffering isn't wanted
* **State Machines** – Component lifecycle management (active/idle/degraded states)

### How It Works

```
Your Factory Floor                    UMH Core                         Your Systems
──────────────────                    ────────                         ────────────
                                                        
PLCs (S7, Modbus)    ─┐                                          ┌─▶ Dashboards
Sensors (OPC UA)     ─┼─[Bridge]─▶ Unified Namespace ─[Bridge]───┼─▶ Cloud/MQTT
MES/ERP (REST)       ─┘              (organized data)            └─▶ Databases
```

1. **Bridges** connect your devices to the UNS (50+ protocols supported) - they're the ONLY entry point
2. **Data flows** into organized topics: `enterprise.site.area.line._contract.virtual_path.tag`
3. **Models** validate critical data (optional but recommended for production)
4. **Consumers** subscribe to the data they need
5. **Configure** via Management Console UI or directly edit YAML files

*Every message is buffered, validated, and organized - no data loss, guaranteed structure.*

## Getting Started

Sign up at [management.umh.app](https://management.umh.app) and deploy your first instance in 60 seconds through the UI.

**Prefer step-by-step learning?** Follow our progressive 4-step guide:

1. [**Install UMH Core**](/getting-started) - One Docker command (5 minutes)
2. [**Connect Your First Data**](/getting-started/1-connect-data) - Create a Bridge and see data flow (10 minutes)
3. [**Organize Your Data**](/getting-started/2-organize-data) - Scale from 1 to 1000s of tags automatically (15 minutes)
4. [**Validate Your Data**](/getting-started/3-validate-data) - Add quality control with Data Models (20 minutes)

By the end, you'll have production-ready data pipelines with validation, organization, and monitoring.

## Documentation Structure

* [**Usage Guides**](/usage) - Step-by-step implementation guides
  * [**Instances**](/usage/instances) - Managing and configuring UMH Core deployments
  * [**Unified Namespace**](/usage/unified-namespace) - Core messaging architecture
  * [**Data Flows**](/usage/data-flows) - Connect and process data streams
  * [**Data Modeling**](/usage/data-modeling) - Enterprise data structuring
  * [**Management Console**](/usage/management-console) - Cloud-based control center
* [**Production Deployment**](/production) - Scaling, security, and operations
* [**Reference Documentation**](/reference) - Complete API and configuration reference

## Learn More About UNS

For deeper understanding of the concepts behind UMH Core:

* [**The Unified Namespace Course Series**](https://learn.umh.app/featured/) - 4-chapter comprehensive course
  * [Chapter 1: OT Foundations](https://learn.umh.app/lesson/chapter-1-the-foundations-of-the-unified-namespace-in-operational-technology/) - Automation pyramid challenges
  * [Chapter 2: The Rise of UNS](https://learn.umh.app/lesson/chapter-2-the-rise-of-the-unified-namespace/) - Core architecture principles
  * [Chapter 3: IT Foundations](https://learn.umh.app/lesson/chapter-3-the-foundations-of-the-unified-namespace-in-information-technology/) - Modern IT patterns
* [**Industrial IoT Architecture**](https://learn.umh.app/blog/cloud-native-technologies-on-the-edge-in-manufacturing/) - Edge computing in manufacturing
* [**MQTT vs UNS Comparison**](https://learn.umh.app/blog/what-is-mqtt-why-most-mqtt-explanations-suck-and-our-attempt-to-fix-them/) - Why data contracts matter

## Community & Support

* [**Discord Community**](https://discord.gg/F9mqkZnm8U) - Get help and connect with other users
* [**GitHub Repository**](https://github.com/united-manufacturing-hub/united-manufacturing-hub) - Source code and issue tracking
* [**Management Console**](https://management.umh.app) - Web-based configuration and monitoring
* [**UMH Website**](https://www.umh.app/) - Company and product information


# Getting Started

**60 seconds — that's all it takes to launch UMH Core.**

No kubectl, no Kubernetes setup, just a single Docker container. Almost anything that runs Docker works — from your MacBook to enterprise edge boxes.

> **Note:** This getting-started guide follows the common PLC/sensor path (raw → device models). For ERP integration and other patterns, see the [Data Modeling documentation](/usage/data-modeling).

## System requirements

**Minimum:**

* 2 vCPU
* 4 GB RAM
* 40 GB SSD

**Architecture:**

* x86\_64 (Intel/AMD 64-bit)
* ARM64 (64-bit ARM, but NOT Raspberry Pi - Redpanda requires 48-bit virtual address space)

For detailed sizing recommendations, see the [Sizing Guide](/production/sizing-guide).

## Installation using the Management Console

1. **Sign up** at [management.umh.app](https://management.umh.app)
2. Click **"Add a new instance"**

![Instance setup screen](/files/bQ5yFCy3HCMFCcuyphzY)

3. **Configure your instance:**
   * **Name** (1): Give it a cool name (e.g., `my-first-umh`)
   * **Type** (2): Select "Core" from the dropdown
   * **Location** (3): Enter your location path:
     * **Level 0**: `your-company` (required - your enterprise name)
     * **Level 1**: `Your level 1 name` (optional - e.g., site or plant)
     * **Level 2-4**: Leave empty for now (optional)
   * Click **"Add Instance"** (4)
4. **Copy the Docker command** that appears
5. **Run it** on any machine with Docker installed

## 🎉 Success!

Your instance will appear as **"Online"** in the Management Console within seconds.

**Next step:** [Connect your first data source →](/getting-started/1-connect-data)

***

## Troubleshooting

**Volume permission errors** Named volumes automatically handle permissions for the container user (UID 1000). If you still see permission errors like `s6-svscan: warning: unable to stat benthos-dataflow-*`, try removing the container and recreating the volume:

```bash
docker rm -f umh-core && docker volume rm umh-core-data && docker volume create umh-core-data
```

Then re-run the `docker run` command.

For advanced users needing bind mounts (custom data locations), see the [Container Layout reference](/reference/container-layout#advanced-custom-data-location).

**Corporate firewall/proxy issues** If your corporate network intercepts TLS traffic, see the [Corporate Firewalls Guide](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/production/corporate-firewalls.md) to add your CA certificate or, as a last resort, set `allowInsecureTLS: true` in `config.yaml` or use `-e ALLOW_INSECURE_TLS=true` in your docker run command.

For proxy configuration, see the [Corporate Firewalls Guide](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/production/corporate-firewalls.md) or add these arguments to your docker run:

```bash
-e HTTP_PROXY=http://proxy.company.com:8080 \
-e HTTPS_PROXY=https://proxy.company.com:8080 \
-e NO_PROXY=localhost,127.0.0.1,.local
```

***

## Coming from UMH Classic?

See the [Migration Guide](/production/migration-from-classic) for step-by-step migration instructions, including data contract changes and configuration updates.

***

## Concepts Learned

* **UMH Core** - Single-container Industrial IoT platform
* **Management Console** - Cloud UI for managing instances
* **Instance** - A running UMH Core deployment
* **Location hierarchy** - Organizational structure (enterprise → site → area → line)
* **Docker container** - How UMH Core runs

## Ready to Connect Data?

**You've successfully installed UMH Core!** 🚀

**Next step:** [Connect your first data source →](/getting-started/1-connect-data)


# Step 2: Connect Data

Let's get some data flowing! We'll start with simulated data, then you can connect real devices later.

This guide walks you through connecting your first data source step by step. If you're looking for the complete bridge reference, see [Bridges](/usage/data-flows/bridges).

## Navigate to Data Flows

1. Click **"Data Flows"** in the left menu
2. Go to the "Bridges" tab. Here you find all of your bridges.
3. Click **"Add Bridge"**

![Data Flow](/files/leR3InfvWkqJfx6sWVxv)

## Choose How to Create the Bridge

The UI will ask you how to start

* **From Scratch**: Start building from one of our curated templates.
* **From Existing Bridge**: Copy the configuration of an existing bridge.

For this tutorial, pick **"From Scratch"**.

![Create a New Bridge](/files/5rFxY0vlLBvcMmkUJAtM)

## Select the `generate` Template

You'll land on a list of curated templates.

For this tutorial, type `generate` in the search bar and one template remains: **"Generic via Generate"** (Benthos `generate` input). Select it.

> 💡 **`generate`** is one of the inputs that Redpanda provides out of the box.
>
> [Learn more about the `generate` input here.](https://docs.redpanda.com/connect/components/inputs/generate/)

By default, this template produces `hello world` messages, so you don't need a real PLC to finish the tutorial.

![Select a Template](/files/ucrSJ3wWmoO1rZy0AN5k)

## Configure the Bridge - General Tab

The bridge editor has three tabs: **General**, **Read Flow**, and **Write Flow**.

On the **General** tab:

* **Name:** `my-first-generator-bridge`
* **Instance:** Select your UMH Core instance.
* **IP Address / Port:** The device to connect to. We have no real PLC for this tutorial, so use `localhost` and `8080`.
* **Location:** Pre-filled with your instance location. Leave it as is for now. In the screenshot we define a bridge for `enterprise.siteA`.

Once done, click **"Save & Deploy"**. A popup shows deployment progress and any errors or warnings. If everything goes well, you'll be redirected automatically.

![Bridge General Configuration: Entering IP, Port, Name & Instance.](/files/WabOo7gq3h8lchQ6lZ1i)

![Logs of a successful bridge deployment after clicking "Save & Deploy"](/files/v2PPEsq50PI97OwVbddr)

Back on the bridge, verify the connection succeeded. Here we see a latency of "0 ms" since we're connecting to localhost.

![Bridge Latency Good](/files/59BjKxhOpTEtV4DjzLzv)

If connecting to a real PLC fails, the latency indicator turns orange:

![Bridge Latency Bad](/files/PiMz1haBROUfZ7gP0At0)

## Configure the Bridge - Read Flow Tab

Because you started from the "Generic via Generate" template, the protocol is already set to **Generate** and pre-filled with a template generator configuration.

In Data Type, make sure that **Time-Series** is selected and that the **Activate Flow** toggle is enabled.

![Bridge Read Input](/files/aKxCprSshfW1JORnAJvT)

* **Input:** The template pre-fills a generator. For real protocols like "Modbus" or "Siemens S7" you'd see protocol-specific settings here. Keep the defaults: generate `hello world` every `1s`.

![Bridge Read Processing](/files/cVnLJf05qlhyztAaHP6S)

* **Processing:** The "Tag Processor" appears because we selected "Time Series" data type.

Click the **Always** expansion panel to expand the view. The expand shows the processor code.

There are three required fields:

* **location\_path:** Where the data goes (auto-filled from bridge location)
* **data\_contract:** Leave as `_raw` for now (no validation rules)
* **tag\_name:** Name your tag (data point) - we'll use `my_data`

> 💡 **What's a tag?** In industrial systems, a "tag" is a single data point - like a temperature sensor reading, motor speed, or valve position. Think of it as a variable that changes over time.

The **Always** section uses JavaScript to process messages. We're not modifying anything for now, just passing the data through.

> 💡 **Tip:** You can modify data here later (e.g., unit conversions renaming). If you don't know JavaScript, any LLM (ChatGPT, Claude) can help write the code.

![Bridge Read Output](/files/NhxHmVrXALS3GOoHGfXs)

The Output section is auto-generated - it sends data to your Unified Namespace.

Click **"Save & Deploy"**.

![Bridge Read Deployed](/files/gkvGO4VoJWrnud8AKlEy)

## 🎉 Success!

You should now see:

* **Status:** Active ✅
* **Throughput:** \~1 msg/sec

Your data is flowing!

## Step 3: View Your Data in the Topic Browser

![Topic Browser My Data](/files/qYf7K9xEzf2aODfbI0P0)

Click **"Topic Browser"** in the left menu. This shows all data in your Unified Namespace.

You'll see your data organized as a **topic**:

* `enterprise` → `siteA` → `_raw` → `my_data`

This is the full topic path: `enterprise.siteA._raw.my_data`

Exactly as we configured it! Click on `my_data` to see details.

* Topic Details shows your bridge configuration and data location
* Last Message shows the most recent `hello world` message with timestamp
* History shows a table of recent messages (would be a chart for numeric data)
* Metadata contains additional information about the data source (we'll use this later)

> 💡 **Only seeing one `hello world` row in your topics?**
>
> That's intended. Put simply, your data is processed by benthos-umh, which drops repeated, unchanged values. This is called **downsampling**.
>
> [Learn more about our benthos-umh downsampler here.](https://docs.umh.app/benthos-umh/processing/downsampler)

## Understanding What You Built

You just created a complete data pipeline:

```
Bridge → Processing → Unified Namespace → Topic Browser
```

**Key Concepts:**

* **Bridge:** The ONLY way data enters UMH (ensures quality and monitoring)
* **Location Path:** Organizes your data hierarchically (`enterprise.siteA`)
* **Data Contract:** Currently `_raw` (no validation rules)
* **Tag:** Your data point (`my_data`)
* **Topic:** Complete address in the UNS (`enterprise.siteA._raw.my_data`)

## What's Next?

**You have data flowing!** This is already production-ready for many use cases.

**Want to organize better?** → [Continue to Step 3: Organize Your Data](/getting-started/2-organize-data)

**Connect a real device?** Simply change the protocol from "Generate" to:

* **OPC UA** for modern PLCs
* **Modbus** for older equipment
* **MQTT Subscribe** for existing MQTT devices
* [See all 50+ supported protocols →](https://docs.umh.app/benthos-umh/input)

Real hardware uses the address-mapping table instead of the generator. The [Bridges reference](/usage/data-flows/bridges) covers that, along with logs, metrics, and debug logging.

## Concepts Learned

Building on the location path from Step 1, you now understand:

* **Unified Namespace (UNS)** - Event-driven data backbone that eliminates point-to-point connections ([learn more](/usage/unified-namespace))
* **Bridge** - Gateway for external data into the UNS (the only way data enters)
* **Bridge Template** - Pre-configured starting point for a bridge (e.g. "Generic via Generate")
* **Data Flows** - Configuration area for bridges and other data pipelines
* **Protocol** - Connection method (manufacturing protocols like OPC UA, Modbus, S7, or IT protocols like HTTP)
* **Connection** - IP/hostname and port settings with latency monitoring
* **Tag** - A time-series data point (like a PLC variable)
* **Tag Processor** - JavaScript-based transformation for time-series data
* **Topic** - Complete data address: `location_path.data_contract.tag_name` ([topic convention](/usage/unified-namespace/topic-convention))
* **data\_contract** - Data validation rules (`_raw` = no validation)
* **Topic Browser** - UI for viewing all data in the UNS
* **Downsampler** - Drops repeated unchanged values so the UNS stays clean
* **Throughput** - Tag updates per second for time-series data

***

**Pro tip:** Everything you just configured in the UI is stored as YAML. As you get comfortable, you can copy and modify these configurations for faster setup of similar devices.


# Step 3: Organize Data

> **Prerequisite:** You should have data flowing from [Step 2](/getting-started/1-connect-data). If not, go back and complete that first!

## Understanding Messages

When data flows through bridges, each message contains:

* **payload**: The actual data value
* **meta**: Information about the message (source, destination, tag names, etc.)

In the processing code, you'll work with these as `msg.payload` and `msg.meta`.

## The Challenge

Right now you have one measurement going to one place:

```
enterprise.siteA._raw.my_data
```

But real PLCs have:

* Hundreds of data points with various address formats
* Different data types (numbers, strings, booleans)
* Multiple machines on the same line

**Do you need to map each tag manually?** No! Let's connect a real PLC and see how automatic mapping works.

## Part 1: Connect a Real PLC (Or Simulate One)

### Create a New Bridge for a Real PLC

We'll use Siemens S7 as an example, but this works identically with OPC UA, Modbus, or any other protocol.

1. Go to **Data Flows** → **Add Bridge**
2. **Name:** `s7-plc`
3. **Protocol:** Select **Siemens S7** (or your protocol)
4. **Connection:**
   * **IP:** Enter your PLC's IP address (example: 192.168.1.100)
   * **Rack:** 0
   * **Slot:** 1

![Protocol Selection](/files/c0oTLooCPGvtJhXM8gS6)

### Configure What to Read

In the **Input** section, you'll see protocol-specific settings:

```yaml
s7comm:
    addresses:
        - DB1.DW20      # Example address
        - DB1.S30.10    # Another address
        - DB3.I270      # Third address
    tcpDevice: '{{ .IP }}'
    rack: 0
    slot: 1
```

![S7 Input Configuration](/files/QgVUoZP7FrAmOoVCENdt)

### Automatic Tag Mapping

In the **Processing** section, look at the **Always** code:

```javascript
// Set location from bridge config
msg.meta.location_path = "{{ .location_path }}";

// Use _raw (no validation - data passes through as-is)
msg.meta.data_contract = "_raw";

// Key concept: Protocol address becomes the tag name automatically
msg.meta.tag_name = msg.meta.s7_address;  // For OPC UA: msg.meta.opcua_tag_name

// Pass the value through
msg.payload = msg.payload;

return msg;
```

**Key line:** `msg.meta.tag_name = msg.meta.s7_address`

Each protocol provides its address in metadata:

* Siemens S7: `msg.meta.s7_address`
* OPC UA: `msg.meta.opcua_tag_name`
* Modbus: `msg.meta.modbus_address`

The addresses automatically become tags - no manual mapping needed!

### Template Variables

When you entered your PLC's IP address in the Connection settings, it automatically became available as `{{ .IP }}` in your configuration code. Template variables connect your Connection settings to the processing code:

* `{{ .IP }}` - The IP address you entered in Connection settings
* `{{ .PORT }}` - The port you entered in Connection settings
* `{{ .location_path }}` - The location from Bridge configuration

This makes configurations reusable across different sites - just change the Connection settings and the template variables update automatically.

Learn more: [Template Variables Reference](/reference/variables)

Click **Save & Deploy**.

### See Your PLC Data

In **Topic Browser**, you now see ALL your PLC data automatically organized:

```
enterprise.sksk._raw.DB1.DW20     [12345]
enterprise.sksk._raw.DB1.S30.10   ["Product ABC"]
enterprise.sksk._raw.DB3.I270     [789]
```

(These are S7 addresses, but OPC UA NodeIDs or Modbus registers would appear the same way)

**Result:** One bridge reads your entire PLC. Every address becomes a tag automatically.

![Topic Browser DB1.DW20](/files/W1eNNcA47Im8uueEHDMe)

## Part 2: Special Handling with Conditions

Some PLC tags need special treatment - scaling, unit conversion, or validation. Let's use **Conditions** to handle specific addresses differently.

### Add Your First Condition

1. Below the **Always** section, click **Add Condition**
2. **If Condition:** `msg.meta.s7_address == "DB1.DW20"` (use the appropriate metadata field for your protocol)
3. **Then Action:**

```javascript
// Scale the value and add engineering unit
msg.payload = parseFloat(msg.payload) * 1.0;
msg.meta.unit = "bar";  // This is pressure in bar
return msg;
```

### Add Another Condition for String Data

1. Click **Add Condition** again
2. **If Condition:** `msg.meta.s7_address == "DB1.S30.10"`
3. **Then Action:**

```javascript
// Clean up string data
msg.payload = msg.payload.trim();
msg.meta.unit = "Text";  // Mark as text data
return msg;
```

![Processing Conditions](/files/Nlsmc6hEmHIiDI7tueEU)

### What Conditions Do

* **Always section:** Runs for EVERY tag (sets basics)
* **Conditions:** Run ONLY for specific tags (special handling)

You can handle hundreds of tags with just a few conditions for the special cases.

### View the Enhanced Data

In **Topic Browser**, click on `DB1.DW20`. Notice:

* The value might be scaled
* **Metadata** now shows `unit: "bar"`
* Special handling applied ONLY to this tag

![Metadata View](/files/ScIxTOlFtbyNYxv1NqHt)

## Part 3: Organize with Virtual Folders

Right now all tags are at the root level. Let's organize them into logical folders using virtual paths.

### Add Organization with Conditions

1. **Add Condition** for production data
2. **If Condition:** `msg.meta.s7_address.startsWith("DB1")`
3. **Then Action:**

```javascript
// Group all DB1 data under "production" folder
msg.meta.virtual_path = "production";  // Creates organizational folder
return msg;
```

4. **Add Condition** for quality data
5. **If Condition:** `msg.meta.s7_address.startsWith("DB3")`
6. **Then Action:**

```javascript
// Group all DB3 data under "quality" folder
msg.meta.virtual_path = "quality";  // Creates organizational folder
return msg;
```

### The Result: Organized Folders

In **Topic Browser**, your data is now organized:

```
enterprise.sksk._raw.production.DB1.DW20     [12345]
enterprise.sksk._raw.production.DB1.S30.10   ["Product ABC"]
enterprise.sksk._raw.quality.DB3.I270        [789]
```

* All DB1 tags → production folder
* All DB3 tags → quality folder
* Automatically organized by data block

You organize entire data blocks with one condition, not individual tags.

## Part 4: Route to Different Machines

What if different data blocks belong to different machines? Let's route them!

### Dynamic Location Routing

Add conditions to route data blocks to different machines:

1. **Add Condition** for Machine 1
2. **If Condition:** `msg.meta.s7_address.startsWith("DB1")`
3. **Then Action:**

```javascript
// DB1 belongs to machine-1
msg.meta.location_path = "{{ .location_path }}.machine-1";
msg.meta.virtual_path = "sensors";
return msg;
```

4. **Add Condition** for Machine 2
5. **If Condition:** `msg.meta.s7_address.startsWith("DB3")`
6. **Then Action:**

```javascript
// DB3 belongs to machine-2
msg.meta.location_path = "{{ .location_path }}.machine-2";
msg.meta.virtual_path = "quality";
return msg;
```

### Result

**ONE bridge now routes to MULTIPLE machines:**

```
enterprise.sksk.machine-1._raw.sensors.DB1.DW20     [12345]
enterprise.sksk.machine-1._raw.sensors.DB1.S30.10   ["Product ABC"]
enterprise.sksk.machine-2._raw.quality.DB3.I270     [789]
```

## Understanding the Complete Picture

You now control every part of the topic:

```
umh.v1.enterprise.sksk.machine-1._raw.sensors.DB1.DW20
       └─ fixed ─┘└─ location_path ─┘     └virtual┘└tag┘
                                             path   name
```

* **location\_path**: WHERE the device is (can be dynamic)
* **virtual\_path**: HOW you organize (folders for grouping)
* **tag\_name**: WHAT you're reading (automatic from S7 address!)

## What You've Learned

✅ **Protocol addresses become tag names** - No manual mapping needed! ✅ **Conditions handle special cases** - Not every tag, just the exceptions ✅ **Folders organize automatically** - Group entire data blocks at once ✅ **Dynamic routing is simple** - Send different PLCs to different locations ✅ **Variables make it reusable** - `{{ .IP }}` works for any PLC

## Concepts Learned

Building on previous guides, you now understand:

* **Template variables** - Dynamic configuration using `{{ .IP }}`, `{{ .PORT }}`, `{{ .location_path }}`
* **Message structure** - msg.meta (metadata) and msg.payload (actual value)
* **Conditions** - If-then rules in Tag Processor for special handling
* **virtual\_path** - Additional folder organization within topics
* **Dynamic routing** - One bridge serving multiple locations
* **Automatic tag mapping** - Protocol addresses become tag names automatically
* **Metadata** - Additional context in msg.meta (units, sources, custom fields)

## What's Next?

You can now:

* Connect entire PLCs with one bridge
* Organize hundreds of tags with a few conditions
* Route to different locations dynamically

**But there's still a risk:** What if DB1.DW20 suddenly sends a string instead of a number? Or DB3.I270 goes out of range?

**Ready to add data validation?** → [Step 4: Validate Your Data](/getting-started/3-validate-data)


# Step 4: Validate Data

> **Prerequisite:** You should have multiple tags flowing from [Step 3](/getting-started/2-organize-data). If not, complete that first!

## The Problem

Right now, ALL your data goes into `_raw` - no validation:

* DB1.DW20 could suddenly send "hello" instead of a number
* Critical vibration data might arrive with wrong units
* Typos in tag names create duplicate data streams

**How do you ensure data quality?** Data Models!

## Part 1: Create Your First Data Model

Let's create a model for a CNC machine's vibration data.

### Add a Data Model

1. Go to **Data Models** → **Add Data Model**
2. **Instance:** Select your instance
3. **Name:** `cnc`
4. **Description:** "CNC machine vibration monitoring"

### Define the Structure

In the **Data Model Structure** section, add this YAML:

```yaml
vibration:
  x-axis:
    _payloadshape: timeseries-number
  y-axis:
    _payloadshape: timeseries-number
```

**What this means:**

* Your CNC model has a `vibration` folder
* Inside are two measurements: `x-axis` and `y-axis`
* Both must be numbers (enforced automatically!)

![Data Model Creation](/files/GCYvp07ucCdZlIuzMRMr)

Click **Save & Deploy**.

![Data Model List](/files/hnAxeszHJZiOEtF83ju0)

💡 **Behind the scenes:** The system automatically creates a data contract called `_cnc_v1`. This contract will enforce your model's structure. [Learn more about data contracts →](/usage/data-modeling/data-contracts)

## Part 2: Use Your Model in a Bridge

### Find Your Contract Name

1. Go to **Contracts** tab
2. Find `_cnc_v1` - this was auto-generated from your model
3. Note the structure it expects

![Contract View](/files/hm8a1sW1o6kDXOBkdQUe)

### Update Your Bridge

Go back to your S7 bridge from Step 3. In the **Always** section, change:

```javascript
// OLD: Everything goes to _raw
msg.meta.data_contract = "_raw";
```

To:

```javascript
// NEW: Everything goes to our validated model
msg.meta.data_contract = "_cnc_v1";
```

![Incorrect Processing Setup](/files/CA6LgOGZ2mod5F2CWzYa)

Click **Save & Deploy**.

## Part 3: Experience Validation (It Will Fail!)

![Deployment Failed](/files/cHuheBlPxLHfEYrALtbD)

**Your deployment fails!** Look at the error:

```
schema validation failed for message with topic 'umh.v1.enterprise.sksk._cnc_v1.DB1.DW20':
Valid virtual_paths are: [vibration.x-axis, vibration.y-axis].
Your virtual_path is: DB1.DW20
Fix: Update the bridge to set meta.virtual_path to 'vibration' and meta.tag_name to 'x-axis' or 'y-axis'
```

Data models ENFORCE structure. Your S7 address `DB1.DW20` doesn't match the expected paths `vibration.x-axis` or `vibration.y-axis`.

**Two new concepts here:**

1. Deployments can fail not only if the connection is bad (learned in Step 2)
2. But also if the bridge throws validation errors

## Part 4: Fix with Smart Routing

Instead of forcing ALL data into the model, let's be selective. Change your **Always** section back:

```javascript
// Most data stays unvalidated
msg.meta.data_contract = "_raw";
msg.meta.tag_name = msg.meta.s7_address;
return msg;
```

Now update your condition for DB1.DW20:

```javascript
// Special handling for DB1.DW20: Route to validated model
msg.payload = parseFloat(msg.payload) * 1.0;
msg.meta.data_contract = "_cnc_v1";  // Use validated model
msg.meta.virtual_path = "vibration";  // Required: matches model structure
msg.meta.tag_name = "x-axis";         // Required: matches model field
msg.meta.unit = "raw";
return msg;
```

![Correct Processing Setup](/files/ErJlzDQJKwSdjmW4jBGy)

**What this does:**

* DB1.DW20 → Validated as `vibration.x-axis` in CNC model
* All other tags → Continue to `_raw` (no validation)

Click **Save & Deploy**. Now it succeeds!

## Part 5: Success! View Your Validated Data

![Topic Browser with Validated Data](/files/yq4IIxBojS9FssMpOuh0)

In **Topic Browser**, you now see:

```
enterprise.sksk._cnc_v1.vibration.x-axis    [12345]  ✓ Validated
enterprise.sksk._raw.DB1.S30.10             ["text"]  (Unvalidated)
enterprise.sksk._raw.DB3.I270               [789]     (Unvalidated)
```

**How validation works:**

* The CNC model guarantees `x-axis` is always a number
* If someone sends text, it's rejected at the bridge
* Other data flows normally through `_raw`

## Concepts Learned

Building on previous guides, you now understand:

* **Data Models** - Reusable templates defining data structure
* **Payload Shapes** - Type validation (timeseries-number, etc.)
* **Schema Validation** - Automatic enforcement at the bridge
* **Auto-generated Contracts** - Models create contracts like `_cnc_v1`
* **Validation Errors** - Deployments fail if data doesn't match
* **Selective Validation** - Route specific data to validated models

## What's Next?

You now have a complete data pipeline with:

* ✅ Automatic tag discovery (Step 3)
* ✅ Smart organization (Step 3)
* ✅ Data validation (Step 4)

**Ready for production?** Check out:

* [Production Guide](/production) - Sizing, security, monitoring
* [Data Modeling Deep Dive](/usage/data-modeling) - Advanced models
* [Stream Processors](/usage/data-modeling/stream-processors) - One way to transform device data to business KPIs

***

**Congratulations!** You've mastered the fundamentals of UMH Core. Your data is now organized, validated, and production-ready. 🎉


# Usage

This section covers practical implementation of UMH Core, focusing on data flows and unified namespace concepts. Learn how to set up bridges, standalone flows, and stream processors to connect your industrial devices to the unified namespace. Includes detailed guides on data production, consumption, topic conventions, payload formats, real-time topic exploration via the Topic Browser, and storing history in the Historian.


# Instances

## What is an Instance?

An instance is a running UMH Core container - a single Docker container that hosts your entire Unified Namespace. Each instance is identified by its location path (like `enterprise.site.area.line`) which determines where data is organized in your industrial data infrastructure.

## Instance Overview Page

![Instance Overview](/files/cuQHkqPfWejU51z5k2N9)

The instance overview shows all your UMH instances at a glance:

* **Instance Name**: Your chosen identifier for the instance
* **Type**:
  * **Core**: Single container deployment (UMH Core)
  * **Classic**: Kubernetes-based deployment (legacy)
* **Version**: Container version identifier (e.g., `9e22396`)
* **Data Flows**: Number of configured bridges and stand-alone flows
* **Topics**: Total data points in your Unified Namespace
* **Latency**: Network response time in milliseconds
* **Throughput**: Messages per second flowing through the system

**Status indicators:**

* Green dot: Instance is online and reachable
* Gray dot: Instance is offline or unreachable

**Quick actions:** Click the context menu (⋮) on any instance for:

* Instance Details
* Config File
* Delete

## Instance Details Page

![Instance Details](/files/PrcreTNREh8FvF6sIb4K)

The instance details page provides comprehensive monitoring and management:

### Agent Panel

* **Name**: Instance identifier (e.g., `sk-core-hetzner`)
* **Location path**: Your organizational structure
  * Level 0: Enterprise (required)
  * Level 1-4: Site, Area, Line, etc. (optional)
* **Latency**: Connection health indicator (N/A when healthy)
* **Logs button**: Access S6 system logs for diagnostics

### Container Panel

Real-time resource monitoring:

* **CPU**: Usage percentage of available cores
* **Memory**: Usage of available RAM (GiB)
* **Disk**: Storage usage (GiB)
* **Architecture**: System architecture (e.g., `amd64`)
* **Hardware ID**: Unique container identifier

### Data Flows Panel

Bridge and flow status breakdown based on [state machines](/reference/state-machines):

* **Active**: Currently processing data (messages flowing)
* **Neutral**: Includes:
  * `idle`: Healthy but no data for 30+ seconds
  * `stopped`: Intentionally disabled
  * `starting`: Initialization in progress
  * Various transition states

Example states:

* `starting_dfc (port is open)`: Connection verified, waiting for Benthos initialization
* `starting: stopping: stopping`: Complex state transition in progress

**Bridges count**: Total configured with active/neutral breakdown

### Redpanda Panel

The message broker powering your [Unified Namespace](/usage/unified-namespace):

* **Incoming Throughput**: Data rate into the broker (KiB/s)
* **Outgoing Throughput**: Data rate from the broker (KiB/s)
* **Logs/Metrics buttons**: Direct access to Redpanda diagnostics

### Topic Browser Panel

* **Topics count**: Total number of data topics
* Quick access to the [Topic Browser](/usage/unified-namespace/topic-browser) for data exploration

### Release Panel

Version information for support and updates:

* **Version**: Full semantic version (e.g., `v0.43.4`)
* **Channel**: Release channel (`stable`, `beta`, etc.)
* **Software versions**: Individual component versions (S6 Overlay, Benthos, Redpanda)

## Next Steps

* **Configure your instance**: [Edit the config file](/usage/instances/config-file)
* **Connect devices**: [Create bridges](/usage/data-flows/bridges)
* **View your data**: [Topic Browser](/usage/unified-namespace/topic-browser)
* **Understand states**: [State Machines reference](/reference/state-machines)


# Config File

The configuration file editor allows you to directly modify your instance's `/data/config.yaml` file through the Management Console UI.

## Accessing the Config Editor

Two ways to open the configuration editor:

1. **From Instance Details**: Click the "Config File" button in the top-right corner
2. **From Instance Overview**: Click the context menu (⋮) → "Config File"

![Config File Editor](/files/g0nbJ5MVgpOKPXbLOzOK)

## ⚠️ Warning: Direct YAML Editing

**You are editing the config.yaml file directly.** Use this feature with caution:

* Invalid YAML syntax will break your instance
* Recovery requires SSH access to the container
* The agent applies changes via hot-reload - no restart needed

## Editor Controls

### Main Actions

* **Refresh**: Reload the current configuration from the instance
* **Save**: Apply your changes (the agent will hot-reload)

### Additional Options (⋮ menu)

![Local File Sync Options](/files/X2cuwPdZysgJ9L6bO1sk)

* **Download**: Save a local copy of the configuration
* **Local File Sync**:
  * **Open Local File**: Edit the config in your preferred IDE
  * **Create Local File**: Start a new local file for editing

Local file sync enables:

* Version control with Git
* IDE features (syntax highlighting, validation, autocomplete)
* AI assistant integration (Cursor, GitHub Copilot, Claude)

## Configuration Structure

The editor shows your complete configuration with main sections:

```yaml
templates:
  protocolConverter:
    # Reusable bridge templates

protocolConverter:
  - name: my-bridge
    # Bridge configurations (shown as "Bridges" in UI)

dataFlow:
  - name: my-flow
    # Stand-alone flow configurations
```

Key sections:

* `templates:` - Reusable configuration blocks for templating
* `protocolConverter:` - Bridge definitions (UI shows as "Bridges")
* `dataFlow:` - Stand-alone flow definitions
* `dataContracts:` - Data validation rules
* `dataModels:` - Data structure definitions

For complete documentation, see [Configuration Reference](/reference/configuration-reference).

## UI ↔ YAML Synchronization

UMH Core maintains two-way sync between UI and YAML:

**UI → YAML**: When you create bridges or data models in the UI, they appear here automatically

**YAML → UI**: After saving YAML changes, they're reflected in the UI immediately

This enables flexible workflows:

* **OT teams**: Use the UI for visual configuration
* **IT teams**: Use YAML for infrastructure-as-code
* **Learning**: Create in UI, study the generated YAML
* **Scaling**: Template in YAML for multiple similar devices

## Troubleshooting

**Syntax Errors**: The Save button will show errors if YAML is invalid. Common issues:

* Incorrect indentation (use spaces, not tabs)
* Missing colons after keys
* Unclosed quotes

**Changes Not Applied**: Ensure you clicked Save. Check the agent logs if changes don't take effect.

**Recovery from Bad Config**: If the instance breaks:

1. SSH into the container
2. Edit `/data/config.yaml` directly
3. The agent will auto-reload when valid

## Next Steps

* [Configuration Reference](/reference/configuration-reference) - Full YAML schema
* [Variables](/reference/variables) - Template variables and substitution
* [Bridges](/usage/data-flows/bridges) - Configure data connections


# Unified Namespace

> **Prerequisite:** Complete the [Getting Started guide](/getting-started) to see the UNS in action with real devices and data.

The Unified Namespace (UNS) is where ALL your industrial data lives in one organized, accessible place. Instead of hunting through 50 different systems to find a temperature reading, everything flows through one central hub.

## The Problem: Spaghetti Diagrams

Traditional manufacturing IT looks like this:

```
100 devices × 100 systems = 10,000 point-to-point connections
```

Every new dashboard means updating PLCs. Every new sensor means modifying databases. Every integration breaks when you upgrade. It's a maintenance nightmare that gets worse with scale.

## The Solution: One Central Hub

The UNS flips this architecture:

```
100 devices → 1 namespace ← 100 systems = 200 total connections
```

All data flows through one place with consistent structure, validation, and access patterns.

## How It Works

### Publish Regardless

Your PLC doesn't care if anyone is listening. It publishes "pump is running" to the UNS and moves on. When someone needs that data next week, it's already there. No reprogramming required.

### Structured Topics

Every piece of data has an address that answers three questions:

* **WHERE**: `enterprise.site.area.line` - the location path
* **WHAT**: `_pump_v1` - the data contract defining structure
* **WHICH**: `inlet_temperature` - the specific data point

Result: `umh.v1.enterprise.site.area.line._pump_v1.inlet_temperature`

### Bridges Handle All Data Flow

Data enters and exits the UNS exclusively through [bridges](/usage/data-flows/bridges). This ensures every message gets proper context, validation, and organization.

### From Device to Business

Start simple, add complexity as needed:

* **`_raw`** - Mirror device 1:1 for initial exploration and debugging
* **`_pump_v1` and similar** - Apply business names directly in bridges for production
* **`_maintenance_v1` and similar** - From stream processors (aggregating device models) OR directly from ERP/MES systems

The progression: Use `_raw` temporarily to understand your data, then apply device models directly in bridges for production. Business models can be created by aggregating device models via stream processors OR by connecting directly to business systems like ERP/MES.

## Documentation Structure

* [**Topic Convention**](/usage/unified-namespace/topic-convention) - How data is addressed in the namespace
* [**Payload Formats**](/usage/unified-namespace/payload-formats) - Time-series vs relational message structure
* [**Metadata and Tracing**](/usage/unified-namespace/metadata-and-tracing) - Original tags and data lineage
* [**Topic Browser**](/usage/unified-namespace/topic-browser) - Explore your namespace in real-time

## Next Steps

1. **Explore your data**: Open the [Topic Browser](/usage/unified-namespace/topic-browser)
2. **Connect devices**: Create [bridges](/usage/data-flows/bridges)
3. **Model your data**: Define [data models](/usage/data-modeling/data-models)
4. **Create KPIs**: Build [stream processors](/usage/data-modeling/stream-processors)


# Topic Convention

> **Prerequisite:** Understand the [UNS concept](/usage/unified-namespace) and see it in action via [Getting Started](/getting-started).

Every piece of data in the UNS has a unique address that tells you exactly where it came from and what it represents.

## See It In Action

Here's how real PLC data appears in the Topic Browser:

![Topic Browser showing DB1.DW20](/files/W1eNNcA47Im8uueEHDMe)

The topic `umh.v1.demo.DefaultArea.DefaultProductionLine.SIEMENS-S7._raw.DB1.DW20` tells us:

* **WHERE**: demo → DefaultArea → DefaultProductionLine → SIEMENS-S7
* **WHAT**: `_raw` (unprocessed device data)
* **WHICH**: DB1.DW20 (specific PLC register)

## The Pattern

```
umh.v1.<location_path>.<data_contract>[.<virtual_path>].<name>
```

### Breaking It Down

| Part            | Example                              | Purpose                      |
| --------------- | ------------------------------------ | ---------------------------- |
| `umh.v1`        | Always `umh.v1`                      | Version prefix               |
| `location_path` | `enterprise.site.area.line`          | Physical/logical hierarchy   |
| `data_contract` | `_raw` or `_pump_v1`                 | Data structure type          |
| `virtual_path`  | `motor.diagnostics`                  | Optional folder organization |
| `name`          | `temperature` or `work_order.create` | The data point or action     |

## Location Path - WHERE

The location path organizes your physical and logical structure:

* **Level 0 (Required)**: Enterprise - your company
* **Level 1**: Site - physical location
* **Level 2**: Area - department or zone
* **Level 3**: Line - production line or cell
* **Level 4**: Machine - specific equipment

You can use ISA-95, KKS, or any naming standard. The only rule: level 0 is mandatory.

Example: `umh.v1.acme.chicago.packaging.line1.filler._raw.speed`

## Data Contract - WHAT

The contract defines the data structure:

* **`_raw`**: Unvalidated device data for exploration
* **`_devicemodel_v1`**: Validated device models (e.g., `_pump_v1`, `_cnc_v1`)
* **`_businessmodel_v1`**: Business KPIs and aggregations (e.g., `_maintenance_v1`, `_production_v1`)

Contracts always start with underscore. They're your data's "type system."

## Virtual Path - Organization

Optional segments for grouping related data:

```
umh.v1.acme.plant._pump_v1.motor.diagnostics.vibration
                          └─────┬─────┘
                          Virtual path for organization
```

## Name - WHICH

The specific data point or action:

* **Time-series**: `temperature`, `pressure`, `running`
* **Relational**: `work_order.create`, `batch.complete`

## Next Steps

* [Understand payload formats](/usage/unified-namespace/payload-formats) for message structure
* [Connect devices with bridges](/usage/data-flows/bridges)
* [Explore with Topic Browser](/usage/unified-namespace/topic-browser)


# Payload Formats

UMH Core supports two payload formats: **Time-Series** for individual sensor readings and **Relational** for structured business data.

## Quick Reference

| Type            | Structure                             | Use Case                        | Example                       |
| --------------- | ------------------------------------- | ------------------------------- | ----------------------------- |
| **Time-Series** | `{"timestamp_ms": int, "value": any}` | Sensor readings, machine states | Temperature, pressure, status |
| **Relational**  | `{...any JSON object...}`             | Business records, work orders   | Batch data, quality reports   |

## Time-Series Data

Individual sensor values with timestamps. One value per message.

**Format:**

```json
{
  "timestamp_ms": 1733904005123,
  "value": 42.5
}
```

**Requirements:**

* Must have exactly `timestamp_ms` and `value` keys
* `timestamp_ms`: Integer or float without fraction
* `value`: number, boolean, or string (see [Data Model Type Definitions](/reference/data-model-type-definitions))
* No additional keys allowed
* Max size: 1MiB

**Examples:**

```json
// Temperature reading
{"timestamp_ms": 1733904005123, "value": 23.4}

// Machine state
{"timestamp_ms": 1733904005123, "value": "running"}

// Boolean sensor
{"timestamp_ms": 1733904005123, "value": true}
```

## Relational Data

Structured records with multiple fields. Used for business data and aggregated metrics.

**Format:**

```json
{
  "field1": "value1",
  "field2": 123,
  "nested": {
    "field3": true
  }
}
```

**Requirements:**

* Must be a JSON object (not array or primitive)
* Can have any keys and nested structure
* Max size: 1MiB

**Examples:**

```json
// Work order
{
  "order_id": "WO-2024-001",
  "product": "Widget-A",
  "quantity": 1000,
  "timestamp_ms": 1733904005123
}

// Quality inspection
{
  "batch_id": "B-2024-12-01",
  "measurements": {
    "length_mm": 100.2,
    "width_mm": 50.1,
    "weight_g": 250.5
  },
  "passed": true,
  "inspector": "John Doe"
}
```

## The "One Tag, One Topic" Rule

Each sensor value gets its own topic. This simplifies addressing, consumption, and debugging.

**Instead of:**

```
Topic: umh.v1.acme.plant._raw.weather
Payload: {"temperature": 23.4, "humidity": 42.1}
```

**Use:**

```
Topic: umh.v1.acme.plant._raw.weather.temperature
Payload: {"timestamp_ms": 1733904005123, "value": 23.4}

Topic: umh.v1.acme.plant._raw.weather.humidity
Payload: {"timestamp_ms": 1733904005123, "value": 42.1}
```

### Why This Matters

| Problem with Multi-Value Payloads | Real-World Impact                                                          |
| --------------------------------- | -------------------------------------------------------------------------- |
| **Timing issues**                 | Temperature arrives, humidity 100ms later. Do you cache? Wait? How long?   |
| **Merge windows**                 | Two PLC variables change almost simultaneously but arrive separately       |
| **Clock skew**                    | Reading 100 tags takes milliseconds - first may change before last is read |
| **Complex addressing**            | Need 3 dimensions: topic × tag name × JSON path                            |
| **Testing complexity**            | Each edge case (missing key, late field, partial failure) multiplies bugs  |

**Benefits of One Tag Per Topic:**

* **Zero merge logic** - Each value is complete when published
* **Simple subscriptions** - Subscribe only to tags you need
* **Clear debugging** - One topic = one data point
* **Schema simplicity** - Data contract just specifies time-series format

## Next Steps

* [Bridges](/usage/data-flows/bridges) - Send data to UNS
* [Stream Processors](/usage/data-modeling/stream-processors) - Transform time-series to relational
* [Data Models](/usage/data-modeling/data-models) - Define structure for both formats


# Metadata and Tracing

Every message in the UNS carries metadata that preserves its origin and transformation history. This enables troubleshooting and data lineage tracking.

## Understanding Metadata

When data flows through the UNS, metadata accumulates at each step:

```json
{
  "_incomingKeys": "s7_address",
  "_initialMetadata": "{\"s7_address\":\"DB1.DW20\"}",
  "bridged_by": "protocol-converter_pump-bridge",
  "data_contract": "_pump_v1",
  "data_contract_name": "_pump",
  "data_contract_version": "1",
  "kafka_msg_key": "umh.v1.enterprise.chicago.packaging.line-1.pump-01._pump_v1.inlet_temperature",
  "kafka_timestamp_ms": "1758290100065",
  "kafka_topic": "umh.messages",
  "location_path": "enterprise.chicago.packaging.line-1.pump-01",
  "s7_address": "DB1.DW20",
  "name": "inlet_temperature",
  "topic": "umh.v1.enterprise.chicago.packaging.line-1.pump-01._pump_v1.inlet_temperature",
  "umh_topic": "umh.v1.enterprise.chicago.packaging.line-1.pump-01._pump_v1.inlet_temperature"
}
```

## Tracing Data Flow

### Bridge Metadata

Every bridge adds metadata to identify the source:

```yaml
# In Topic Browser or when consuming:
bridged_by: "protocol-converter_pump-bridge"    # Which bridge created this
location_path: "enterprise.chicago.packaging"    # Where it came from
data_contract: "_pump_v1"                       # What model was applied
```

Note: The exact format of bridge metadata may vary by implementation.

## Accessing Metadata

### In Topic Browser

The Management Console shows metadata in the details panel:

1. Select any topic in the tree
2. View "Metadata" section in the right panel
3. See all headers including original tags

For details, see [Topic Browser documentation](/usage/unified-namespace/topic-browser).

**Further metadata**: Each input or processing step adds its own metadata. For complete field documentation, see:

* [Benthos-UMH Input Plugins](https://docs.umh.app/benthos-umh/input)

## Next Steps

* [**Topic Browser**](/usage/unified-namespace/topic-browser) - View metadata interactively
* [**Bridges**](/usage/data-flows/bridges) - How data enters and exits the UNS


# Topic Browser

Real-time exploration of your Unified Namespace data via Management Console or GraphQL API.

## Management Console

Access via `management.umh.app` → **Topic Browser** (not the Classic version).

### Interface Overview

* **Left panel**: Hierarchical topic tree following ISA-95 structure
* **Right panel**: Details for selected topic (metadata, values, history)
* **Auto-aggregation**: Combines data from all your UMH instances
* **Live updates**: Real-time data without manual refresh

### Key Features

**Topic Tree Navigation:**

* Expand/collapse hierarchy levels (Enterprise → Site → Area → Line → Equipment)
* See data contracts (`_raw`, `_pump_v1`) and virtual paths
* Search bar for filtering by topic name or metadata

**Topic Details Panel:** When selecting a topic, view:

* **Location**: Physical hierarchy path
* **Data Contract**: Applied schema (`_raw`, `_pump_v1`, etc.)
* **Current Value**: Latest data with timestamp
* **Metadata**: Including original device tags (see [Metadata and Tracing](/usage/unified-namespace/metadata-and-tracing))
* **History**: Last 100 values as chart (time-series) or table (boolean/string)

![Topic Browser showing DB1.DW20](/files/W1eNNcA47Im8uueEHDMe)

## GraphQL API

Programmatic access at `http://localhost:8090/graphql` (per instance).

{% hint style="info" %}
**Early Access.** We built the GraphQL API for developers who want to query the Unified Namespace programmatically. It's experimental, so it's off by default. See the instructions below and give it a try. Tell us what you think.
{% endhint %}

### Configuration

```yaml
internal:
  topicBrowser:
    desiredState: "active"  # Enable/disable

agent:
  graphql:
    enabled: true           # Enable GraphQL API (required!)
    port: 8090             # API port
    debug: false           # Set true for GraphiQL UI
```

### Basic Query

```graphql
{
  topics(filter: {
    text: "temperature",
    meta: [{ key: "data_contract", eq: "_pump_v1" }]
  }) {
    topic
    metadata {
      key
      value
    }
    lastEvent {
      ... on TimeSeriesEvent {
        producedAt
        numericValue
        scalarType
      }
    }
  }
}
```

### cURL Example

```bash
curl -X POST http://localhost:8090/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ topics(limit: 10) { topic } }"}'
```

## Common Use Cases

### Finding Specific Data

1. Use search bar for partial matches (e.g., "temp" finds all temperature topics)
2. Filter by metadata like data contract or location
3. Check metadata section for original device tags

### Verifying Data Flow

1. Select topic to see current value
2. Check "Produced At" timestamp for freshness
3. Review history chart for patterns
4. Inspect metadata for source bridge

### Debugging Issues

1. Verify topic exists in tree
2. Check metadata for `bridged_by` to identify source
3. Look for original tag names (e.g., `opcua_tag_name`, `s7_address`)
4. Compare timestamps to detect delays

## Performance Notes

* Topic Browser stores last 100 values per topic
* Extremely rapid updates are batched for display
* Large namespaces (>10,000 topics) may load progressively
* Each UMH instance has its own Topic Browser service

## Next Steps

* [**Metadata and Tracing**](/usage/unified-namespace/metadata-and-tracing) - Understanding metadata fields


# Data Flows

> **Prerequisite:** Complete the [Getting Started guide](/getting-started) to understand basic connections and data organization.

Data flows move and transform industrial data in UMH Core. Three types serve different purposes:

## Flow Types

### [Bridges](/usage/data-flows/bridges)

Move data into and out of the Unified Namespace with connection monitoring and automatic location path. Support for 50+ industrial protocols (OPC UA, Modbus, S7) plus IT systems. Write flows coming soon - use stand-alone flows meanwhile.

### [Stand-alone Flows](/usage/data-flows/stand-alone-flow)

Raw Benthos access for custom processing when bridges or stream processors aren't sufficient. Used as fallback for write flows, external integrations, and specialized transformations.

### [Stream Processors](/usage/data-flows/stream-processor)

Transform existing UNS data into different structures. Part of the [data modeling system](/usage/data-modeling/stream-processors) for aggregating device data into business views.

## Quick Start

Most users start with bridges to connect devices:

1. **Data Flows** → **Add Bridge**
2. Select your protocol (OPC UA, Modbus, S7)
3. Configure connection and location
4. Deploy to start data collection

## Learn More

* [Getting Started Guide](/getting-started) - Connect your first device
* [Data Modeling](/usage/data-modeling) - Structure and validate data
* [Unified Namespace](/usage/unified-namespace) - Understanding topics and payloads


# Bridges

> **Prerequisite:** Understand [Data Flow concepts](/usage/data-flows) and complete the [Getting Started guide](/getting-started).

Bridges move data into and out of the Unified Namespace, providing connection monitoring and automatic data organization. While read flows work with any protocol, write flows are not yet implemented - use [stand-alone flows](/usage/data-flows/stand-alone-flow) for writing to external systems.

> **New to UMH?** This page is the reference for setting up real bridges. If you just want to get mock data flowing and learn the basics first, start with [Step 2: Connect Your First Data Source](/getting-started/1-connect-data).

## UI Capabilities

| Action                      | Available | Notes                                                        |
| --------------------------- | --------- | ------------------------------------------------------------ |
| Create bridges              | ✅         | Visual protocol configuration                                |
| Create from existing bridge | ✅         | Copy a healthy bridge's configuration                        |
| Start from a template       | ✅         | 84 pre-configured templates, filtered by vendor and protocol |
| Select protocols            | ✅         | OPC UA, Modbus, S7, MQTT, 50+ more                           |
| Configure location path     | ✅         | ISA-95 levels (0-4)                                          |
| Map addresses to topics     | ✅         | Dropdown of data model fields for custom data contracts      |
| Monitor connection health   | ✅         | Real-time status                                             |
| View logs                   | ✅         | Live log streaming                                           |
| View metrics                | ✅         | Throughput monitoring                                        |
| Edit bridges                | ✅         | Modify and redeploy                                          |
| Delete bridges              | ✅         | Clean removal                                                |
| Advanced mode               | ✅         | Direct YAML editing                                          |

## Creating a Bridge (UI)

### Step 1: Navigate to Data Flows

Go to **Data Flows** and click **Add Bridge**:

![Data Flows Overview](/files/LsbyRB5rszI2bSYxiOUF)

### Step 2: Choose How to Start

![Create a New Bridge](/files/2x6WBFDuaVvmEFyYUaPp)

* **From Scratch**: start from a pre-configured protocol template, which is functional out of the box, or customize the template to fit your needs.
* **From Existing Bridge**: pick a healthy bridge from any instance and start with a copy of its configuration. Adapt the bridges name, connection, and addresses.

The remaining steps describe the from-scratch path. Starting from an existing bridge skips the template list and lands on the same screens with the fields already filled in.

### Step 3: Select a Template

![Select a Template](/files/eu90vKmAS23vXQJ5IikI)

Templates carry a working protocol configuration. Search by name, or narrow the list with the **Vendor** and **Protocol** filters.

Behind the templates we support **50+ industrial protocols** through [Benthos-UMH](https://docs.umh.app/benthos-umh/input) plus everything from [Redpanda Connect](https://docs.redpanda.com/redpanda-connect/components/inputs/about/):

Common industrial protocols:

* **Modbus**: For PLCs and RTUs
* **OPC UA**: For industrial servers
* **S7**: For Siemens PLCs
* **Ethernet/IP**: For Allen-Bradley
* **And many more**: See full list at [Benthos-UMH Inputs](https://docs.umh.app/benthos-umh/input)

### Step 4: Configure General Settings

Fill in the basic information and proceed with Step 5.

![General Information](/files/TjM6oECUnxyVI16Gzrtx)

**General Information:**

* **Name**: Unique identifier for your bridge
* **Instance**: Select your UMH instance
* **Level**: Location path (e.g., enterprise → site → area → line → machine)

![Connection Configuration](/files/3UIAhqDvMlV6YmF69HRt)

**Connection:**

* **IP Address**: Device IP (becomes `{{ .IP }}` variable)
* **Port**: Connection port (becomes `{{ .PORT }}` variable)

The protocol itself needs no selection here. It comes from the template you picked in Step 3, and the **Read** tab opens on that protocol's settings.

### Step 5: Configure Protocol Settings

Each protocol has specific settings. For example, S7:

![Siemens S7 Configuration](/files/bgoYToRnSiR8hRzoUNgP)

### Step 6: Map Addresses

Each row in the address mapping table becomes one UNS topic: the protocol address, followed by the data contract, virtual path, and tag name that build the topic.

![Address Mapping Table](/files/D6p8O7GvWblQ2M2vhYE6)

With `_historian` or `_raw`, Virtual Path and Tag Name are free-text fields. Selecting a custom data contract merges both columns into a single dropdown listing every field the contract's [data model](/usage/data-modeling/data-models) defines, so you pick a valid path instead of typing one. A value the model rejects, for example after switching contracts, stays in the row, is flagged red, and blocks deploy until you correct it. Contracts whose data model references other models keep free-text input.

### Step 7: Deploy

Click **Save & Deploy**. The bridge will start connecting to your device.

## Managing Bridges

### View Status

The main view shows all bridges with connection status and throughput:

![Bridges Overview](/files/LsbyRB5rszI2bSYxiOUF)

* **Green dot**: Connected and running
* **Yellow dot**: Starting or warning state
* **Grey dot**: Intermediate states (starting, stopping, or stuck in starting state)

### Context Menu

Right-click any bridge for quick actions:

![Bridge Context Menu](/files/jrt0fWTPYtUWSfUrPz0o)

### View Logs

Monitor bridge activity in real-time:

![Bridge Logs](/files/IEi0GxiSron0UwvCWnGH)

### View Metrics

Track performance and throughput:

![Bridge Metrics](/files/lPtOzNu04R8tIoXbIFM6)

### Enable Debug Logging

When troubleshooting connection or data issues, enable debug logging in your bridge configuration:

**Via YAML:**

```yaml
protocolConverter:
  - name: my-plc-bridge
    protocolConverterServiceConfig:
      debug_level: true  # Enable detailed logging
      config:
```

Debug logging provides:

* Detailed protocol communication logs
* Connection state transitions
* Data parsing details
* For OPC UA: Sets `OPC_DEBUG` environment variable for protocol-level debugging

**Remember to disable debug logging** after troubleshooting - it generates significantly more log data.

To view debug logs, go to the bridge and select **Logs**.

## Important: Data Types from Different Protocols

**Industrial protocols (OPC UA, Modbus, S7) only provide time-series data** - snapshots of values at points in time. They cannot directly provide relational records.

**IT protocols (MQTT, HTTP, Kafka) can provide relational data** - when you select "relational" as the data type in the UI, the bridge automatically uses [`nodered_js` processor](https://docs.umh.app/benthos-umh/processing/node-red-javascript-processor) to handle complex business records like work orders or batch reports.

For converting time-series to relational data, see [Stream Processors](/usage/data-flows/stream-processor#time-series-to-relational-challenges) or use [Stand-alone Flows](/usage/data-flows/stand-alone-flow) with state management.

## Configuration (YAML)

Advanced users can edit the YAML directly:

```yaml
protocolConverter:
  - name: my-plc-bridge
    desiredState: active
    protocolConverterServiceConfig:
      debug_level: false  # Direct child (matches standalone flows)
      location:
        2: "production-line"
        3: "plc-01"
      config:
        connection:
          nmap:
            target: "{{ .IP }}"
            port: "{{ .PORT }}"
        dataflowcomponent_read:
          benthos:
            input:
              s7comm:
                addresses:
                  - DB1.DW20
                  - DB3.I270
                tcpDevice: "{{ .IP }}"
                rack: 0
                slot: 1
            pipeline:
              processors:
                - tag_processor:
                    defaults: |
                      msg.meta.location_path = "{{ .location_path }}";
                      msg.meta.data_contract = "_raw";
                      msg.meta.tag_name = msg.meta.s7_address;
                      return msg;
            output:
              uns: {}
      variables:
        IP: "192.168.1.100"
        PORT: "102"
```

### Key Configuration Elements

**Location Path:**

```yaml
location:
  0: enterprise   # Optional: inherits from agent
  1: site        # Optional: inherits from agent
  2: area        # Required: your addition
  3: line        # Required: your addition
  4: cell        # Optional: further subdivision
```

**Template Variables:**

```yaml
variables:
  IP: "192.168.1.100"      # Available as {{ .IP }}
  PORT: "502"              # Available as {{ .PORT }}
  SCAN_RATE: "1s"          # Custom variable {{ .SCAN_RATE }}
```

**Connection Monitoring:**

```yaml
connection:
  nmap:
    target: "{{ .IP }}"    # Uses IP variable
    port: "{{ .PORT }}"    # Uses PORT variable
```

## Next Steps

* Try the [Getting Started guide](/getting-started) to connect your first device
* Learn about [Data Models](/usage/data-modeling/data-models) to structure your data
* Explore [Metadata and Tracing](/usage/unified-namespace/metadata-and-tracing) for debugging
* Use [Stand-alone Flows](/usage/data-flows/stand-alone-flow) for custom processing
* Configure [Stream Processors](/usage/data-flows/stream-processor) to transform data


# Stand-alone Flow

> **Prerequisite:** Understand [Data Flow concepts](/usage/data-flows) and [Bridges](/usage/data-flows/bridges).

Stand-alone flows provide raw Benthos access for custom data processing when bridges or stream processors aren't sufficient. They're the fallback mechanism for write flows, external integrations, and specialized transformations.

## When to Use

Use stand-alone flows for:

* Point-to-point data transformation
* External system integration (databases, APIs, files)
* Custom processing that doesn't fit bridge patterns

Use bridges instead for:

* Industrial device connections
* Data that needs UNS organization
* Connections requiring health monitoring

## UI Capabilities

| Action                | Available | Notes                       |
| --------------------- | --------- | --------------------------- |
| Create flows          | ✅         | Via UI with YAML editor     |
| View flows            | ✅         | Listed in Standalone tab    |
| Edit configuration    | ✅         | Direct YAML editing         |
| Control state         | ✅         | Active/Stopped toggle       |
| Use example templates | ✅         | Pre-built samples available |
| Monitor status        | ✅         | Throughput and state        |
| View logs             | ✅         | Real-time log streaming     |
| View metrics          | ✅         | Performance monitoring      |
| Delete flows          | ✅         | Via context menu            |

## Creating a Stand-alone Flow (UI)

### Step 1: Navigate to Data Flows

Go to **Data Flows** → **Standalone** tab and click **Add Standalone Flow**:

![Standalone Flows List](/files/OWsX3OnFz35FD8Mf4JPo)

### Step 2: Configure the Flow

The configuration interface has multiple sections:

![Standalone Flow Configuration](/files/jOMK0ZRsh7IwW585s9Rc)

**Key sections:**

* **General**: Name, instance, and state control (Active/Stopped)
* **Input (YAML)**: Configure data source (see [Benthos-UMH inputs](https://docs.umh.app/benthos-umh/input))
* **Processing (YAML)**: Transform data with processors
* **Output (YAML)**: Send to any destination (generic, not auto-generated like bridges)
* **Examples**: Pre-built templates including database bridges

### Step 3: Use Example Templates

Click the **Examples** tab to access pre-built configurations:

![Example Templates](/files/wG9J3XurujcvxOUQyFc9)

Available examples include:

* **mqtt\_to\_kafka\_historian\_bridge**: MQTT to Kafka with historian pattern
* **kafka\_to\_postgresql\_historian\_bridge**: Write to TimescaleDB (like Classic)
* **opcua\_to\_umh\_core**: OPC UA integration
* **generate\_to\_stdout**: Testing pattern

### Step 4: Control Flow State

Use the state dropdown to control execution:

![State Control](/files/9ffaXWFrOr40vVzwii94)

* **Active**: Flow is running
* **Stopped**: Temporarily disabled (useful for maintenance)

### Managing Stand-alone Flows

Right-click any flow for quick actions:

![Context Menu](/files/9DVyANxt7W6ww3sHZe61)

Options include:

* **Standalone Flow**: Edit configuration
* **Logs**: View real-time logs
* **Metrics**: Monitor performance
* **Delete**: Remove the flow

### Enable Debug Logging

For troubleshooting data processing issues in stand-alone flows, enable the `debug_level` configuration option:

**Via YAML:**

```yaml
dataFlow:
  - name: custom-processor
    desiredState: active
    dataFlowComponentConfig:
      debug_level: true  # Enable detailed logging (default: false)
      benthos:
        # ...
```

**What it controls:**

* Log level: DEBUG (when `true`) vs INFO (when `false`, default)
* Detailed operational logging for input/output operations
* Processing step execution details
* Data transformation trace information

**Important:** Debug logging generates significantly more log data and may impact performance. **Use only during active troubleshooting** and disable afterward.

View logs via right-click → **Logs**. For comprehensive debug\_level configuration reference, see [Configuration Reference](/reference/configuration-reference).

## Configuration (YAML)

Stand-alone flows use standard Benthos configuration:

```yaml
dataFlow:
  - name: custom-processor
    desiredState: active
    dataFlowComponentConfig:
      benthos:
        input:
          # Any Benthos input
        pipeline:
          processors:
            # Processing logic
        output:
          # Any Benthos output
```

## Migrating from Benthos

Existing Benthos configurations work with minimal changes:

```yaml
# Original Benthos config
input:
  mqtt:
    urls: ["tcp://broker:1883"]
output:
  stdout: {}

# As UMH Core stand-alone flow
dataFlow:
  - name: migrated-flow
    desiredState: active
    dataFlowComponentConfig:
      benthos:
        input:
          mqtt:
            urls: ["tcp://broker:1883"]
        output:
          stdout: {}
```

## Next Steps

* Learn about [Bridges](/usage/data-flows/bridges) for device connectivity
* Explore [Stream Processors](/usage/data-flows/stream-processor) for data transformation
* See [Benthos-UMH Documentation](https://docs.umh.app/benthos-umh) for all processors


# Stream Processors

> **Prerequisite:** Understand [Data Flow concepts](/usage/data-flows) and complete the [Getting Started guide](/getting-started).

Stream processors transform data already inside the Unified Namespace, aggregating multiple device streams into business KPIs and metrics. They're the third type of data flow, specialized for transforming device models into business models.

## When to Use

Use stream processors for:

* Aggregating data from multiple devices into KPIs
* Creating business metrics from raw sensor data
* Calculating OEE, efficiency, or other derived values
* Reducing data volume for cloud transmission

Use bridges instead for:

* Getting data into the UNS initially
* Single device to model mapping
* Direct protocol connections

## UI Capabilities

| Action               | Available | Notes                                   |
| -------------------- | --------- | --------------------------------------- |
| Create processors    | ✅         | Visual configuration with topic browser |
| Select source topics | ✅         | Browse and multi-select                 |
| Map to model fields  | ✅         | Expression-based mapping                |
| View processors      | ✅         | Listed in Stream tab                    |
| Monitor throughput   | ✅         | Real-time message rates                 |
| Edit processors      | ✅         | Modify configuration                    |
| Delete processors    | ✅         | Remove when not needed                  |
| Code mode            | ✅         | Direct YAML editing                     |

## Creating a Stream Processor

Stream processors are created through the **Data Flows → Stream** tab:

![Stream Processors List](/files/FatN1M4GE40ppwYCDit1)

### Step 1: General Configuration

Configure basic settings and location hierarchy:

![Stream Processor General Configuration](/files/lUTbEXCmmmHDOiJKBAXm)

* **Name**: Unique identifier for your processor
* **Instance**: The UMH Core instance to run on
* **Data Contract**: Select the output model (creates validated structure)
* **Location Levels**: Set where in the hierarchy to output data (Level 0-4)

### Step 2: Topic Selection

Select source topics and they'll automatically get variable names:

![Topic Selection and Variable Names](/files/NDApk2lthPEqjnatjNoo)

* Browse available topics in your UNS
* Select multiple sources with checkboxes
* Variables auto-generated (e.g., `_10` for first topic, `_10_2` for second)
* Use "Show Selected Only" to filter view
* Copy variable names for use in expressions

### Step 3: Expression Mapping

Map source variables to model fields using expressions:

![Expression Mapping](/files/NDApk2lthPEqjnatjNoo)

* Each field in your data model appears on the right
* Enter expressions using the source variables
* Simple passthrough: `source_alias`
* Calculations: `_10 + _10_2` or more complex formulas
* The expressions support standard mathematical operations

### Step 4: Code Mode (Advanced)

Switch to Code Mode to see or edit the generated YAML configuration:

![Code Mode Configuration](/files/nAc5OsipvoC6WbVTUeOs)

The UI automatically generates YAML with:

* **sources**: Your selected topics with location\_path templates
* **mapping**: Field-to-expression mappings
* Comments explaining the structure

You can switch between UI and Code Mode anytime to fine-tune configurations.

**For detailed configuration instructions**, see the comprehensive [Stream Processors guide](/usage/data-modeling/stream-processors) in Data Modeling.

## How It Works

Stream processors subscribe to multiple UNS topics and output to a single model-based topic:

```
Multiple Device Model Topics → Stream Processor → Business Model Topic
(e.g., _pump_v1, _cnc_v1)        (aggregation)     (e.g., _maintenance_v1)
```

Example: Combining temperature and pressure from different sensors into a pump model:

* Input: `enterprise.site._raw.temp_sensor_1`, `enterprise.site._raw.pressure_gauge_2`
* Output: `enterprise.site._pump_v1` with calculated efficiency

The system handles:

* Consumer group management for offset tracking
* Dependency-based processing (waits for all required inputs)
* Guaranteed output structure via data models

## Configuration (YAML)

While the UI is the primary way to create stream processors, they're stored as YAML:

```yaml
streamprocessors:
  - name: pump_efficiency_calc
    model:
      name: pump
      version: v1
    sources:
      - topic: enterprise.site._raw.inlet_temp
      - topic: enterprise.site._raw.outlet_temp
    mapping:
      efficiency: "(outlet_temp - inlet_temp) / inlet_temp * 100"
```

See the [Stream Processors configuration guide](/usage/data-modeling/stream-processors) for detailed YAML structure.

## Key Differences from Other Data Flows

| Data Flow Type        | Purpose                       | Input Source                     | Output           |
| --------------------- | ----------------------------- | -------------------------------- | ---------------- |
| **Bridges**           | Get data INTO the UNS         | External devices (PLCs, sensors) | UNS topics       |
| **Stream Processors** | Transform data WITHIN the UNS | UNS topics                       | UNS model topics |
| **Stand-alone Flows** | Custom processing             | Any source                       | Any destination  |

## Learn More

For comprehensive stream processor configuration and examples, see the [Stream Processors guide](/usage/data-modeling/stream-processors) in Data Modeling documentation.


# Data Modeling

Data modeling in UMH Core transforms device data into business-ready information through structured schemas and validation.

## The Component Chain

```
Payload Shapes → Data Models → Data Contracts → Data Flows
       ↓              ↓              ↓                ↓
  Value types     Structure      Enforcement    Execution
```

Each component builds on the previous:

* [**Payload Shapes**](/usage/data-modeling/payload-shapes) define what types of values are allowed
* [**Data Models**](/usage/data-modeling/data-models) use shapes to create hierarchical structure
* [**Data Contracts**](/usage/data-modeling/data-contracts) enforce models at runtime
* [**Data Flows**](/usage/data-flows) execute with or without contracts

## What You Can Model

In any UNS topic, data modeling controls specific portions:

```
umh.v1.enterprise.site.area.line._contract.virtual.path.name
       └───────── fixed ─────────┘         └── modeled ──┘
```

* **Fixed**: Location hierarchy comes from bridge configuration
* **Modeled**: Everything after the contract is defined by your data model
  * Virtual path: Organizational folders (e.g., `motor.electrical`)
  * Name: The actual data point (e.g., `current`)

See [Topic Convention](/usage/unified-namespace/topic-convention) for complete structure.

## Data Flow Patterns

Data can follow these patterns based on your needs:

### Device Language (\_raw)

Start by exploring your equipment data with original naming:

```
Device → Bridge → _raw → Topic Browser
```

**Example**: `umh.v1.enterprise.chicago.line-1.pump._raw.DB1.DW20`

* Exploration, debugging, quick connectivity
* Site engineers who know the PLC addressing
* Original tags preserved (e.g., `s7_address: "DB1.DW20"`)

### Device Models

Apply business naming directly in bridges:

```
Device → Bridge + Model → _pump_v1 → Applications
         (one step)
```

**Example**: `umh.v1.enterprise.chicago.line-1.pump._pump_v1.inlet_temperature`

* Consistent naming across equipment types
* Operations teams, local dashboards
* Most implementations start here

### Business Models

Transform device data into business KPIs:

```
Multiple _pump_v1 → Stream Processor → _maintenance_v1 → Enterprise Apps
```

**Example**: `umh.v1.enterprise.chicago._maintenance_v1.work_orders.create`

* Aggregated metrics, business records
* Enterprise systems, management dashboards
* Required when scaling across sites

## The Two-Layer Architecture

**Sites and HQ both need their view of the same data.** This isn't a choice between approaches - it's about enabling both layers to work together:

### Layer 1: Device Models (Data Structure Within Equipment)

* Define WHAT data points exist in equipment (temperature, vibration)
* NOT the organizational structure (that's location\_path)
* Sites maintain control of their data definitions
* Original tags preserved in metadata
* Applied directly in bridges (one step)
* Primarily time-series data
* **Result**: Sites trust the system because they built it

### Layer 2: Business Models (Enterprise Metrics)

Created in TWO ways:

1. **Aggregation**: Stream processors combine device models into KPIs
2. **Direct**: Bridges connect to ERP/MES systems for business data

* Creates consistent metrics across sites
* Doesn't disturb site operations
* Multiple departments can create their own views
* Primarily relational data
* **Result**: Everyone gets the metrics they need

The key: Device models describe equipment internals, business models describe enterprise needs.

### Why Both Layers Matter

**Common failure patterns:**

* **Only device models**: Chaos across multiple sites, no standardization
* **Only business models**: Sites lose control, engineers reject the system

**The success formula:** Sites own their data structure (device models), everyone creates their views (business models). This is why stream processors exist - to bridge these layers without forcing change on sites.

## Key Concepts

### Location Path vs Device Model

* **Location Path**: WHERE equipment sits in your organization
  * Example: `enterprise.chicago.packaging.line-1.pump-01`
  * Defined in bridge configuration
  * This is your factory hierarchy
* **Device Model**: WHAT data points exist within that equipment
  * Example: `_pump_v1.temperature`, `_pump_v1.vibration.x-axis`
  * Defines internal data structure of a single device
  * NOT the organizational structure

### Name vs Tag

* **Name**: The data point identifier in UNS topics
* **Tag**: Industry term for a sensor/data point
* We use "name" for broader applicability (not just time-series)

### Virtual Path

Organizational folders within your data model:

* **Example**: `motor.electrical`, `diagnostics.vibration`
* **Purpose**: Groups related data points logically within a device

### Data Contract vs Data Model

* **Data Model**: Defines structure (template)
* **Data Contract**: Enforces structure (runtime validation)
* **Example**: Creating `pump` model auto-creates `_pump_v1` contract

### Time-Series vs Relational

* **Time-Series**: Single value with timestamp
  * Example: `{"timestamp_ms": 1733904005123, "value": 42.5}`
* **Relational**: Multiple fields in one message
  * Example: Work order with 10 fields

See [Payload Formats](/usage/unified-namespace/payload-formats) for details.

### Processing Methods

**In Bridges:**

* `tag_processor`: For time-series data from PLCs/sensors
* `nodered_js`: For relational data from ERP/MES systems

**In Stream Processors:**

* JavaScript expressions: For aggregating and transforming data
* Example: `total: "sensor1 + sensor2 + sensor3"`

## Choosing Your Data Flow

Based on your data source, choose the appropriate path:

| Source                 | Output         | Bridge Processor | Path                          |
| ---------------------- | -------------- | ---------------- | ----------------------------- |
| PLC/Sensor             | Device Model   | `tag_processor`  | Direct to \_pump\_v1          |
| PLC/Sensor             | Raw            | `tag_processor`  | Direct to \_raw (exploration) |
| ERP/MES                | Business Model | `nodered_js`     | Direct to \_workorder\_v1     |
| Multiple Device Models | Business Model | -                | Via Stream Processor          |

### Data Type Alignment

* **Device Models**: 90% time-series data
  * Temperature, pressure, vibration readings
  * Status indicators, counters, running hours
  * Process with: `tag_processor` in bridges
* **Business Models**: 90% relational data
  * Work orders, maintenance schedules
  * Production batches, quality reports
  * Process with: `nodered_js` in bridges OR aggregate via stream processors

## Implementation Patterns

### Pattern 1: Equipment Monitoring

```
PLC tags → Bridge + tag_processor → Device Model (_pump_v1)
Location: enterprise.site.line.pump-01
Model adds: .temperature, .pressure, .status
```

**When**: Connecting industrial equipment with time-series data

### Pattern 2: ERP Integration

```
SAP work orders → Bridge + nodered_js → Business Model (_workorder_v1)
```

**When**: Connecting business systems with relational data

### Pattern 3: Multi-Site Aggregation

```
site1._pump_v1 ─┐
site2._pump_v1 ─├→ Stream Processor → _maintenance_v1
site3._pump_v1 ─┘   (JavaScript expressions)
```

**When**: Creating KPIs from multiple device models

### Pattern 4: Exploration First

```
Unknown PLC → Bridge + tag_processor → _raw → Topic Browser
                                           ↓
                                    Design device model
                                           ↓
                                    Apply in bridge
```

**When**: Understanding new equipment before modeling

## Why Models Are Immutable

**The scenario**: A data scientist builds a dashboard using `_pump_v1` for 500 pumps across 10 sites.

**Without immutability**: Someone modifies `_pump_v1`:

* Dashboard breaks - fields renamed
* Historical queries fail - structure changed
* Other sites stop working
* Weekend ruined

**With immutability**:

1. Create `_pump_v2` with changes
2. Test thoroughly
3. Migrate gradually
4. Deprecate v1 when safe
5. Dashboard keeps working throughout

## Common Questions

### When should I use data models?

Start with `_raw` for exploration. Add models when:

* You need consistent naming across sites
* Multiple applications consume the data
* Validation becomes critical
* You're ready for production

### How do I handle different equipment versions?

Create separate models:

* `_pump_v1` for older pumps
* `_pump_v2` for newer pumps with more sensors
* Stream processors can aggregate both

### What about equipment-specific data?

Use virtual paths to organize:

```
_pump_v1.motor.temperature
_pump_v1.motor.current
_pump_v1.diagnostics.vibration
```

## Next Steps

1. **Define value types**: [Payload Shapes](/usage/data-modeling/payload-shapes)
2. **Create structure**: [Data Models](/usage/data-modeling/data-models)
3. **Enforce validation**: [Data Contracts](/usage/data-modeling/data-contracts)
4. **Transform data**: [Stream Processors](/usage/data-modeling/stream-processors)

## Learn More

* [Getting Started Guide](/getting-started) - See data modeling in action
* [Bridges](/usage/data-flows/bridges) - Apply models during ingestion
* [Topic Convention](/usage/unified-namespace/topic-convention) - Understand topic structure


# Data Models

> This article assumes you've completed the [Getting Started guide](/getting-started) and understand the [data modeling concepts](/usage/data-modeling).

Data models define the hierarchical structure of your industrial data. They create the virtual paths and fields that organize raw data into meaningful information.

## Overview

In the [component chain](/usage/data-modeling#the-component-chain), models provide the structure:

```
Payload Shapes → Data Models → Data Contracts → Data Flows
                      ↑
                 Structure defined here
```

When you create a data model, you're defining:

* Virtual paths - organizational folders (e.g., `vibration`, `motor.electrical`)
* Fields - data endpoints with specific types (e.g., `temperature`, `pressure`)
* Relationships - how components nest and reference each other

## UI Capabilities

The Management Console provides full control over data models:

| Feature              | Available | Notes                                    |
| -------------------- | --------- | ---------------------------------------- |
| View model list      | ✅         | Shows all models with versions           |
| Create models        | ✅         | Visual editor with YAML preview          |
| View model details   | ✅         | Inspect structure and configuration      |
| Create new versions  | ✅         | Models are immutable, edit by versioning |
| Reference sub-models | ✅         | Link to other models via `_refModel`     |
| Delete models        | ✅         | Remove unused model versions             |
| Direct editing       | ❌         | Use "New Version" to modify              |

![Data Models List](/files/enr6hy6BrCH81DtjQJVC)

**What you see in the UI:**

* **Name**: Model identifier (e.g., `cnc`, `pump`, `temperature-sensor`)
* **Instance**: Which UMH Core instance owns the model
* **Description**: Optional description of the model's purpose
* **Latest**: Current version number (v1, v2, etc.)

### Model Actions

Click the three-dot menu (⋮) on any model to access actions:

![Data Model Actions](/files/Tn6l1QgojavJqwSna0rt)

* **Data Model**: View the model's structure and YAML configuration
* **New Version**: Create a new version with modifications (since models are immutable)
* **Delete**: Remove the model (only if not in use by contracts or bridges)

![Data Model Creation](/files/0G6bCArIc3Msj795xoqc)

## Configuration

### Basic Structure

```yaml
datamodels:
  - name: pump                    # Model name
    description: "Pump monitoring" # Optional description
    version:
      v1:                         # Version identifier
        structure:                # Hierarchical definition
          pressure:
            inlet:
              _payloadshape: timeseries-number
            outlet:
              _payloadshape: timeseries-number
```

### How Structure Becomes Topics

Model structure directly maps to UNS topics:

```yaml
structure:
  vibration:        # Creates: .../_pump_v1.vibration
    x-axis:         # Creates: .../_pump_v1.vibration.x-axis
```

**Complete topic path:**

```
umh.v1.enterprise.site._pump_v1.vibration.x-axis
       └─ fixed ─┘     └contract┘└─from model─┘
```

## The Three Building Blocks

### Fields

```yaml
temperature:
  _payloadshape: timeseries-number  # Accepts numeric values
```

**Characteristics:**

* Has `_payloadshape` property
* Creates a topic endpoint that accepts data
* References a [payload shape](/usage/data-modeling/payload-shapes) for validation
* Cannot have child elements

### Folders

```yaml
vibration:           # Folder - no _payloadshape
  x-axis:           # Field inside folder
    _payloadshape: timeseries-number
  y-axis:           # Field inside folder
    _payloadshape: timeseries-number
```

**Characteristics:**

* No `_payloadshape` property
* Groups related fields or other folders
* Creates hierarchy in topic path
* Can nest multiple levels deep

### Sub-Models

Define once, use everywhere:

```yaml
# Define reusable motor model
datamodels:
  - name: motor
    version:
      v1:
        structure:
          rpm:
            _payloadshape: timeseries-number
          temperature:
            _payloadshape: timeseries-number

# Reference in pump model
datamodels:
  - name: pump
    version:
      v1:
        structure:
          pressure:
            inlet:
              _payloadshape: timeseries-number
          motor:           # Include the motor model
            _refModel:
              name: motor
              version: v1
```

Topics created:

* `_pump_v1.pressure.inlet`
* `_pump_v1.motor.rpm`
* `_pump_v1.motor.temperature`

**Benefits:**

* Single source of truth
* Consistent structure across models
* Update once, reflected everywhere

## Version Evolution

Models are immutable once created. To add fields, create a new version:

### Why Immutability?

From the [README](/usage/data-modeling#why-are-models-immutable):

* Models are contracts between teams
* Dashboards depend on stable structure
* Historical data queries must not break

### Evolution Pattern

**Version 1 - Basic:**

```yaml
version:
  v1:
    structure:
      temperature:
        _payloadshape: timeseries-number
```

**Version 2 - Add pressure:**

```yaml
version:
  v2:
    structure:
      temperature:
        _payloadshape: timeseries-number
      pressure:                          # New field
        _payloadshape: timeseries-number
```

**Migration steps:**

1. Create v2 with additions
2. Deploy new bridges using v2
3. Update dashboards to v2
4. Keep v1 running during transition
5. Deprecate v1 when safe

## Relationship to Contracts

Models define structure, but don't enforce it. That's where [data contracts](/usage/data-modeling/data-contracts) come in:

| Component    | Purpose            | Example                       |
| ------------ | ------------------ | ----------------------------- |
| **Model**    | Defines structure  | `pump` model with fields      |
| **Contract** | Enforces structure | `_pump_v1` validates messages |

When you create a model in the UI:

1. Model `pump` version `v1` is created
2. Contract `_pump_v1` is auto-generated
3. Contract becomes available in bridges

Without a contract, a model is just documentation. With a contract, it becomes validation.

## Next Steps

* [Data Contracts](/usage/data-modeling/data-contracts) - Make models mandatory
* [Payload Shapes](/usage/data-modeling/payload-shapes) - Specify data types for fields
* [Stream Processors](/usage/data-modeling/stream-processors) - Transform device models to business models


# Data Contracts

> This article assumes you've completed the [Getting Started guide](/getting-started) and understand the [data modeling concepts](/usage/data-modeling).

Data contracts are the enforcement mechanism that makes data models mandatory. Without a contract, models are just documentation - contracts make validation happen.

## Overview

In the [component chain](/usage/data-modeling#the-component-chain), contracts sit between models and execution:

```
Payload Shapes → Data Models → Data Contracts → Data Flows
                                      ↑
                              Enforcement happens here
```

When you specify a data contract in a bridge, the UNS output plugin validates every message against the associated model.

## UI Capabilities

The Management Console provides read-only access to contracts:

| Feature            | Available | Notes                                           |
| ------------------ | --------- | ----------------------------------------------- |
| View contract list | ✅         | Shows all contracts with their models           |
| Filter/search      | ✅         | Filter by name, instance, or model              |
| View associations  | ✅         | See which model version each contract enforces  |
| View usage         | ✅         | Shows count of stream processors using contract |
| Create contracts   | ❌         | Auto-created when creating models in UI         |
| Edit contracts     | ❌         | Immutable once created                          |
| Delete contracts   | ❌         | Must be done via config.yaml                    |

![Contracts List](/files/xmMNDreZG01CwQsl1io3)

**What you see in the UI:**

* **Name**: Contract identifier (e.g., `_cnc_v1`, `_pump_v2`)
* **Instance**: Which UMH Core instance owns the contract
* **Model**: The model and version being enforced (e.g., `cnc (v1)`)
* **Stream Processors**: Count of processors using this contract

### Auto-Creation via UI

When you create a model in the UI, it automatically generates a matching contract:

1. Create model `pump` version `v1` in UI
2. System auto-creates contract `_pump_v1`
3. Contract immediately available for use in bridges

## Configuration

Access configuration via: Instances → Select instance → `...` → Config File

### Basic Structure

```yaml
datacontracts:
  - name: _pump_v1          # Contract name (used in bridges)
    model:
      name: pump            # References a data model
      version: v1           # Specific version to enforce
```

### Naming Convention

Contracts follow the pattern `_modelname_version`:

* Always start with underscore
* Include model name
* End with version number
* Examples: `_pump_v1`, `_temperature_sensor_v2`, `_workorder_v1`

## Enforcement Mechanism

### Where Validation Happens

```
Bridge → UNS Output Plugin → [Contract Check] → Kafka/Redpanda
                                    ↑
                            Validation happens here
```

The UNS output plugin (`output: uns: {}`) performs validation:

1. **Reads metadata**: Extracts `data_contract` from message
2. **Looks up contract**: Finds the associated model
3. **Validates structure**: Checks topic path matches model
4. **Validates payload**: Ensures data types match shapes
5. **Result**:
   * ✅ Valid → Message published to topic
   * ❌ Invalid → Message rejected, bridge degraded

### Validation Failures

When validation fails:

```
ERROR: schema validation failed for message with topic 'umh.v1.enterprise.site._pump_v1.invalid.path':
Valid virtual_paths are: [pressure, temperature, motor.rpm].
Your virtual_path is: invalid.path
```

Result:

* Message rejected (not published)
* Bridge enters degraded state
* Error logged with details
* Bridge retries with backoff

## Contract Types

### The Special \_raw Contract

```yaml
# No explicit definition needed - always available
msg.meta.data_contract = "_raw";
```

* Accepts any structure
* No validation performed
* Use for exploration and development
* Bridge never goes degraded from data issues

### Model-Based Contracts

```yaml
datacontracts:
  - name: _pump_v1
    model:
      name: pump
      version: v1
```

* Enforces exact model structure
* Validates data types via payload shapes
* Rejects non-conforming messages
* Use for production systems

### Relationship to Data Types

| Data Type       | Common Contracts                    | Validation             |
| --------------- | ----------------------------------- | ---------------------- |
| Raw/Exploration | `_raw`                              | No validation          |
| Device Models   | `_pump_v1`, `_sensor_v1`, `_cnc_v1` | Model-based validation |
| Business Models | `_workorder_v1`, `_maintenance_v1`  | Always strict          |

## Examples

### Simple Device Contract

```yaml
# Model definition
datamodels:
  - name: temperature-sensor
    version:
      v1:
        structure:
          celsius:
            _payloadshape: timeseries-number

# Contract (auto-created or manual)
datacontracts:
  - name: _temperature-sensor_v1
    model:
      name: temperature-sensor
      version: v1

# Usage in bridge
msg.meta.data_contract = "_temperature-sensor_v1";
msg.meta.tag_name = "celsius";
msg.payload = 23.5;
```

## Relationship to Stream Processors

Stream processors don't use contracts directly - they reference models:

```yaml
templates:
  streamProcessors:
    pump_aggregator:
      model:           # Direct model reference
        name: pump
        version: v1
```

However, if a matching contract exists (`_pump_v1`), the stream processor's output will be validated against it automatically.


# Payload Shapes

> This article assumes you've completed the [Getting Started guide](/getting-started) and understand the [data modeling concepts](/usage/data-modeling).

Payload shapes define the structure and validation rules for message payloads. While built-in shapes handle most time-series data, custom shapes enable relational data modeling for complex business records.

## Overview

In the [component chain](/usage/data-modeling#the-component-chain), payload shapes provide the foundation:

```
Payload Shapes → Data Models → Data Contracts → Data Flows
       ↑
  Value types defined here
```

## UI Capabilities

Payload shapes have no UI component:

| Feature         | Available | Notes                                     |
| --------------- | --------- | ----------------------------------------- |
| View shapes     | ❌         | Configuration file only                   |
| Create shapes   | ❌         | Edit config.yaml directly                 |
| Edit shapes     | ❌         | Modify config.yaml                        |
| Built-in shapes | ✅         | Always available, no configuration needed |

**Configuration location**: Select your instance under `Instances` in the menu, press `...`, then `Config File`. Find payload shapes under the `payloadShapes:` section

## Built-in Shapes

UMH provides three built-in shapes that handle 90% of industrial data:

### timeseries-number

For numeric sensor data and measurements.

**Structure:**

```json
{
  "timestamp_ms": 1733904005123,
  "value": 42.5
}
```

**Use cases:** Temperature, pressure, speed, counts, any numeric measurement

### timeseries-string

For text-based status and identifiers.

**Structure:**

```json
{
  "timestamp_ms": 1733904005123,
  "value": "running"
}
```

**Use cases:** Machine states, batch IDs, product codes, operator names

### timeseries-boolean

For on/off process values.

**Structure:**

```json
{
  "timestamp_ms": 1733904005123,
  "value": true
}
```

**Use cases:** Machine running, alarm active, valve open/closed, any true/false state

## When to Use Custom Shapes

Custom shapes are specifically for **relational data** - business records with multiple related fields that must stay together.

**Time-series vs Relational:**

| Data Type                | Shape                  | Example                          |
| ------------------------ | ---------------------- | -------------------------------- |
| Single values over time  | Built-in timeseries-\* | Temperature readings             |
| Complex business records | Custom shape           | Work orders, quality inspections |

Learn more: [Payload Formats](/usage/unified-namespace/payload-formats)

## Configuration

Access configuration via: Instances → Select instance → `...` → Config File

### Defining Custom Shapes

```yaml
payloadShapes:
  work-order:    # Shape name (referenced in models)
    description: "Work order record"
    fields:
      orderId:
        _type: string
      productId:
        _type: string
      quantity:
        _type: number
      status:
        _type: string    # created/in-progress/completed
      operatorId:
        _type: string
      timestamp:
        _type: string    # ISO 8601 format
```

**Key points:**

* Field types are `string`, `number`, `boolean` (see [Data Model Type Definitions](/reference/data-model-type-definitions))
* All fields are required
* Shape names must be unique

### Using Custom Shapes in Models

Reference custom shapes in your data models to create CRUD-like endpoints:

```yaml
dataModels:
  - name: work-order
    version:
      v1:
        structure:
          create:
            _payloadshape: work-order    # Custom shape for new orders
          update:
            _payloadshape: work-order    # Same shape for updates
          delete:
            _payloadshape: work-order    # Or simplified delete shape
```

This creates topics like:

* `enterprise.site._work_order_v1.create` - New work orders
* `enterprise.site._work_order_v1.update` - Update existing orders
* `enterprise.site._work_order_v1.delete` - Delete orders

## Processing Relational Data

### Example: Processing Work Orders

```javascript
// In nodered_js processor for work order creation
msg.meta.data_contract = "_work_order_v1";
msg.meta.tag_name = "create";  // Maps to the 'create' endpoint

// Build work order record
msg.payload = {
  orderId: "WO-" + Date.now(),
  productId: msg.product,
  quantity: msg.qty,
  status: "created",
  operatorId: msg.operator,
  timestamp: new Date().toISOString()
};

return msg;
```

This creates a message at `enterprise.site._work_order_v1.create` with the complete work order data.

## Relationship to Other Components

Payload shapes are referenced by [data models](/usage/data-modeling/data-models), which are then enforced by [data contracts](/usage/data-modeling/data-contracts).

## Next Steps

* **Define structure**: [Data Models](/usage/data-modeling/data-models) - Create hierarchies using shapes
* **Enforce validation**: [Data Contracts](/usage/data-modeling/data-contracts) - Make shapes mandatory
* **Build pipelines**: [Data Flows](/usage/data-flows) - Process data with bridges


# Stream Processors

> This article assumes you've completed the [Getting Started guide](/getting-started) and understand the [data modeling concepts](/usage/data-modeling).

Stream processors are one way to create business models by transforming multiple device model streams into KPIs and metrics.

## Overview

Stream processors aggregate device data into business KPIs:

```
Multiple Device Model Topics → Stream Processor → Business Model Topic
                                     ↑
                           Aggregation happens here
```

Stream processors are for:

* **Aggregating** data from multiple devices
* **Creating** business KPIs from raw metrics
* **Calculating** derived values across equipment
* **Reducing** data volume for cloud transmission

## UI Capabilities

The Management Console provides visual stream processor creation:

| Feature             | Available | Notes                                |
| ------------------- | --------- | ------------------------------------ |
| View processor list | ✅         | Shows all processors with throughput |
| Create processors   | ✅         | Visual or code mode                  |
| Topic selection     | ✅         | Browse and select source topics      |
| Expression mapping  | ✅         | Map topics to model fields           |
| Edit processors     | ✅         | Modify existing processors           |
| Delete processors   | ✅         | Remove unused processors             |

![Stream Processors List](/files/FatN1M4GE40ppwYCDit1)

**What you see in the UI:**

* **Name**: Processor identifier
* **Instance**: Which UMH Core instance runs the processor
* **Throughput**: Messages processed per second

## Creating a Stream Processor (UI)

The UI is the primary way to create stream processors. It generates the YAML configuration automatically.

### Step 1: Basic Configuration

![Add Stream Processor](/files/lUTbEXCmmmHDOiJKBAXm)

1. Click **Add Stream Processor**
2. Enter **Name**: Descriptive identifier
3. Select **Instance**: Where to run the processor
4. Choose **Data Contract**: Output structure (e.g., `_production_v1`)
5. Set **Location levels**: Where in the UNS hierarchy (enterprise, site, etc.)

### Step 2: Topic Selection

![Topic Selection](/files/NDApk2lthPEqjnatjNoo)

Select source topics to aggregate:

* Browse available topics
* Select multiple sources
* Variable names auto-generated (e.g., `_10` for each topic)

### Step 3: Expression Mapping

Map source data to output fields:

* Use variable names from selected topics
* Write expressions (e.g., `_10 + _10_2` for sum)
* Each field in your model needs a mapping

### Step 4: Code Mode (Advanced)

![Code Mode Configuration](/files/nAc5OsipvoC6WbVTUeOs)

Switch to code mode to see or edit the generated YAML:

```yaml
sources:
  # Define your data sources (specific topics)
  source_alias: "{{ .location_path }}._raw.some_field"
mapping:
  # Map source aliases to data model fields
  field_name: "source_alias"
```

**Note**: The UI generates this YAML configuration. You can switch between visual and code modes.

## Configuration

Access configuration via: Instances → Select instance → `...` → Config File

### How Stream Processors Work

Stream processors use templates for reusability and subscribe to specific topics:

```yaml
templates:
  streamProcessors:
    production_aggregator:
      model:
        name: production-metrics
        version: v1
      sources:  # Specific topics to subscribe to
        cnc1: "{{ .location_path }}.cnc-01._cnc_v1.vibration.x-axis"
        cnc2: "{{ .location_path }}.cnc-02._cnc_v1.vibration.x-axis"
        cnc3: "{{ .location_path }}.cnc-03._cnc_v1.vibration.x-axis"
      mapping:  # Transform to model fields
        total-vibration: "cnc1 + cnc2 + cnc3"
        avg-vibration: "(cnc1 + cnc2 + cnc3) / 3"
        machines-running: "(cnc1 > 0 ? 1 : 0) + (cnc2 > 0 ? 1 : 0) + (cnc3 > 0 ? 1 : 0)"
```

**Key concepts:**

* Each processor gets a unique consumer group (for offset tracking)
* Sources define exact topics to subscribe to (not patterns)
* Outputs go to: `umh.v1.{{ .location_path }}._<model>_<version>.<field>`

### Deploying Stream Processors

```yaml
streamprocessors:
  - name: line_a_metrics
    _templateRef: "production_aggregator"
    location:
      0: enterprise
      1: site
      2: line-a
```

### Expression Language

Mappings use JavaScript expressions with dependency-based evaluation:

```yaml
mapping:
  # Direct pass-through
  value: "source1"

  # Math operations
  total: "source1 + source2 + source3"
  average: "(source1 + source2) / 2"

  # Conditionals
  status: "value > 100 ? 'HIGH' : 'NORMAL'"

  # Complex logic
  efficiency: |
    var produced = source1;
    var planned = source2;
    return (produced / planned) * 100;
```

**Evaluation behavior:**

* Static mappings (no dependencies) evaluate on every message
* Dynamic mappings only evaluate when their dependencies arrive

## Relationship to Other Components

Stream processors work with:

* [**Data Models**](/usage/data-modeling/data-models): Define output structure via `model` reference
* [**Data Contracts**](/usage/data-modeling/data-contracts): Enforce output validation
* [**Bridges**](/usage/data-flows/bridges): Provide input data (device model topics)

## Next Steps

* **Create device data**: [Bridges](/usage/data-flows/bridges) - Connect devices first
* **Define structures**: [Data Models](/usage/data-modeling/data-models) - Create output models
* **Learn concepts**: [README](/usage/data-modeling) - Understand data modeling approach


# Historian

A historian is the standard industrial term for the system that records process data over time, so operators can go back and ask what a machine was doing last Tuesday. AVEVA PI and GE Proficy fill that role in a classic plant.

Ours stores data from the Unified Namespace in TimescaleDB rather than a proprietary store, so you can read history back with ordinary SQL and build Grafana dashboards on it. The UNS keeps the live value of every tag; the Historian keeps the past.

{% hint style="info" %}
**Early access.** The Historian is not switched on by default. If you want to use it, get in touch with us on [Discord](https://discord.gg/F9mqkZnm8U) or through your usual UMH contact, and we will show you how to get started.
{% endhint %}

## How it fits together

```
Bridges / Stream Processors
        │
        ▼
  Unified Namespace  ──►  Historian bridge  ──►  TimescaleDB  ──►  Grafana
                          (one per contract)      umh schema
```

You configure the database connection **once per instance**, in the instance's Plugins tab. Every Historian bridge on that instance then reuses it, so no credentials are copied into bridge configs, and changing the connection updates the bridges that reference it.

Three steps get you from a running UNS to a dashboard:

1. Set up the connection (this page). Tell the instance where its database is.
2. [Save data to the Historian](/usage/historian/save-to-historian). Create one bridge per data contract you want to keep.
3. [Query the data](/usage/historian/querying). Copy queries out of the topic browser into Grafana or psql.

## Prerequisites

* **A TimescaleDB database**, reachable from the umh-core instance. If you don't have one, [Recommended UMH Stack](/production/deployment/docker-compose/additional-services/recommended-umh-stack) brings up umh-core, PgBouncer, TimescaleDB, and Grafana together. To add just the database to a running instance, see [TimescaleDB](/production/deployment/docker-compose/additional-services/timescaledb).
* **PostgreSQL 16 or newer**, with the `timescaledb` and `ltree` extensions available. Version 16 is the floor because older `ltree` labels reject hyphens, and location paths such as `line-1` contain them.
* **A login role**, created before the first bridge starts. The bridge logs in as this role and creates the `umh` schema it owns; it cannot create the role itself. A database-level grant is enough, so no privileges on `public` are needed:

  ```sql
  CREATE ROLE umh_owner WITH LOGIN PASSWORD 'change-me';
  GRANT CREATE, CONNECT ON DATABASE umh TO umh_owner;
  ```
* **Data in the UNS**, already flowing. The Historian archives what is in the namespace; it does not read from devices itself.

## Set up the connection

1. Open the instance, go to the **Plugins** tab, and add the **Historian** plugin.
2. Fill in the TimescaleDB connection:

   | Field        | Default     | Notes                                                          |
   | ------------ | ----------- | -------------------------------------------------------------- |
   | **Host**     | required    | Hostname or IP, resolved *from inside the umh-core container*. |
   | **Port**     | `5432`      | Point this at PgBouncer if you use one.                        |
   | **Database** | `umh`       | Must already exist.                                            |
   | **User**     | `umh_owner` | The login role from the prerequisites.                         |
   | **Password** | required    | Stored in `config.yaml`, hidden in the UI, redacted in logs.   |
   | **SSL Mode** | `require`   | `require` or `disable`.                                        |
3. Save. The instance opens a connection and reports the result on the plugin's overview card.

`require` encrypts the connection but does not verify the server certificate. `disable` turns TLS off entirely.

Full certificate verification (`verify-full`) cannot be configured from the Management Console yet; support is coming in a later release. A connection that already sets `sslmode: verify-full` in `config.yaml` keeps working, along with its `sslrootcert`, `sslcert`, and `sslkey` paths.

### Reading the connection status

The overview card is not shown in the plugin list. Go to **Instances** → your instance → **Plugins** → **Historian** and click **Open**. Saving a new connection also leaves you on the plugin list, so that click is needed either way.

The overview card shows the connection settings plus three live fields, refreshed once per second:

| Field         | Meaning                                                                                                                                           |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Reachable** | The endpoint answered. It stays `true` even when the server rejects the credentials. Only a network fault or timeout makes it `false`.            |
| **Auth**      | Whether the server accepted the login role, password, and database name. `unknown` when nothing answered, so authentication could not be checked. |
| **Latency**   | Round-trip time of the check query.                                                                                                               |

The check is a single `SELECT 1` over one pooled connection. It tells you the database is reachable and the credentials work; it says nothing about whether a particular bridge is writing. Per-bridge throughput, health, and errors stay in the **Data Flows** bridge list.

Connections are recycled every five minutes, so a password rotated on the server surfaces as an authentication failure within that window rather than being masked by a long-lived session.

### Editing and removing

Editing the connection updates every bridge that references it, so you don't touch the bridges. Leave the password field blank to keep the stored one; the Management Console never receives it back, so an empty value means "unchanged", not "clear".

Adding a Historian connection is create-only. If one already exists, edit it instead; a repeated add is rejected rather than silently overwriting your settings.

## Where it is stored

The connection lives in the instance's `config.yaml` as a single shared block:

```yaml
historian:
  timescale:
    host: timescaledb.example.com
    port: 5432
    database: umh
    username: umh_owner
    password: change-me
    sslmode: require
```

This block is the one place the credentials are written. Every Historian bridge on the instance reads them from here instead of carrying its own copy.

Bridge templates read it through the reserved `{{ .historian.timescale.* }}` scope: `host`, `port`, `database`, `username`, `password`, `sslmode`, `sslrootcert`, `sslcert`, and `sslkey`. See [Variables](/reference/variables).

A bridge counts as a Historian bridge when its write flow's destination protocol is `historian`. umh-core then targets the bridge's health check at the shared connection rather than at a host and port entered on the bridge, so the check follows the connection whenever you change it.

## Next steps

* [Save data to the Historian](/usage/historian/save-to-historian)
* [Query the Historian](/usage/historian/querying)


# Save data to the Historian

A Historian bridge subscribes to the Unified Namespace and writes one data contract into TimescaleDB. You create one bridge per contract you want to keep.

The bridge is a write flow only: it has no read flow, because its source is the UNS rather than a device.

## Before you start

Set up the instance's [Historian connection](/usage/historian#set-up-the-connection) first. Without it, the bridge has nothing to connect to and deployment fails with an error saying so.

## Create the bridge

1. Go to **Data Flows** → **Add Bridge**, and choose **From Scratch**.
2. In the template list, pick **TimescaleDB (PostgreSQL, auto-connect) — Historian**. The **Vendor** filter narrows the list to the TimescaleDB templates.
3. Select the instance. The connection host and port are prefilled read-only from that instance's Historian connection, so the bridge's health check always targets the configured database.
4. In the write flow's output, set `data_contract_name` to the contract you want to store, written **without the version suffix**: `pump`, not `pump_v1`.
5. Deploy.

There is no read flow to configure and no address table to map, so the protocol and address steps of the general [bridge walkthrough](/usage/data-flows/bridges) don't apply.

{% hint style="warning" %}
**Drop the version, and no dashes.** `data_contract_name` is the bare contract name: no leading underscore, no `_vN` suffix. For the UNS contract `_pump_v1`, write `pump`.

A contract whose name contains a dash cannot be stored by the Historian at all. The name becomes a PostgreSQL table identifier, which allows only lowercase letters, numbers, and underscores.
{% endhint %}

The only value you normally change is `data_contract_name`. Credentials, database, and TLS mode come from the shared connection through `{{ .historian.timescale.* }}`, so they are never entered twice and follow the connection if you later change it.

### Naming the data contract

Every version of a contract shares one set of tables, so `pump` stores `_pump_v1` and `_pump_v2` alike. That is why the version is left off: the tables hold the contract, not one version of it.

The Management Console rejects dashes in new data model names for the table-identifier reason above. Models created before that check keep working everywhere else, but a Historian bridge cannot store them under their own name.

The default is `historian`, the generic UNS time-series contract.

### What the bridge writes

For `data_contract_name: pump`, the bridge creates and fills two hypertables in a dedicated `umh` schema:

| Table                | Contents                                                                                             |
| -------------------- | ---------------------------------------------------------------------------------------------------- |
| `umh.value_pump`     | One row per (tag, timestamp). Numbers and booleans in `value_num`, strings and JSON in `value_text`. |
| `umh.attribute_pump` | The message metadata as a JSON object, rewritten only when the key set changes.                      |

Tag identity lives in the shared dimension tables `umh.topic`, `umh.tag`, and `umh.location`. See [Query the Historian](/usage/historian/querying) for the layout and how to read it back.

The bridge subscribes to the whole UNS by default and drops messages whose contract doesn't match `data_contract_name`, so there is no topic regex to keep in sync. Narrow the source topics only if you want to stop unrelated messages from reaching the output at all.

## Connecting to a different database

The **TimescaleDB (PostgreSQL) — Historian** template is the same output without the shared connection: you enter host and port in the connection step and set the username, password, and SSL mode in the output config yourself. Use it for a database that isn't the instance's Historian, such as a second archive or a customer-managed database with its own role.

Everything below applies to both templates; only where the connection details come from differs.

## Advanced options

The output supports retention, compression, metadata filtering, batching, and timeouts. They are plain YAML fields you add to the output config in the bridge editor. See the [Historian output reference](https://docs.umh.app/benthos-umh/output/historian) for the full field list and defaults.

`compress_after` and `retention` are the exceptions. Both are applied once, when the tables are first created, so editing them later has no effect on an existing database: a config edit must not silently change how production history is compressed or deleted. To change them, update the TimescaleDB policies directly on both hypertables, then set the same value in the bridge config so the drift warning stops.

## Troubleshooting

**The bridge fails immediately after deploying.** The output verifies the connection at startup rather than failing later on the first write. It fails fast on an unreachable host, a PostgreSQL older than 16, a missing `timescaledb` or `ltree` extension, and a role that can connect but cannot `INSERT` into the contract's tables. The error names which one. Check the plugin overview card in the Plugins tab first: if **Auth** is not valid there, fix the connection before looking at the bridge.

**The bridge is running but nothing arrives.** The output logs once when it stores its first message, and once if data is flowing but none of it matches `data_contract_name`. A mismatch between the contract in your topics and the configured name is the usual cause.

**Rows are being dropped.** Two cases can never be written and are dropped rather than retried: a tag whose datatype flips between numeric and text, and two different values at the same millisecond. Both are logged with the tag name and counted on the `historian_rows_poisoned` metric. The [Historian output reference](https://docs.umh.app/benthos-umh/output/historian) has a runbook for resolving them.

Connection loss, deadlocks, a missing grant, and a full disk are all retried rather than dropped. A database restart mid-stream loses nothing: held messages replay, and an identical value at the same timestamp is absorbed.

## Next steps

* [Query the Historian](/usage/historian/querying)


# Query the Historian

Once a [Historian bridge](/usage/historian/save-to-historian) is storing a contract, the data is ordinary TimescaleDB, and any SQL client or Grafana can read it. The fastest route is to copy a query out of the topic browser; the schema further down is for when you write your own.

The Management Console never runs these queries. It generates the SQL and you paste it into Grafana or psql.

## Copy a query from the topic browser

1. Open the **Topic Browser** and select a tag.
2. Choose the **Grafana** or **TimescaleDB** tab.

   The two differ only in the time filter. Grafana emits `$__timeFilter(ts)`, which the dashboard's time picker fills in. TimescaleDB emits a fixed window sized to the resolution, so the query returns roughly 60 points when pasted straight into psql.
3. Adjust the controls:
   * **Aggregate** buckets the data with `time_bucket()` and returns `avg`, `min`, and `max`. Turn it off for the raw rows. Text tags cannot be aggregated, so the toggle is disabled for them and the query always returns raw values.
   * **Resolution** sets the bucket width: `$__interval`, `1 second`, `10 seconds`, `1 minute`, `1 hour`, or `raw`. Grafana starts at `$__interval`, which lets Grafana pick a width from the panel's time range and width; TimescaleDB has no such macro and starts at `1 minute`.
4. Copy it.

The panel is generated from the topic name, not from the database, so it appears for every tag, including ones no Historian bridge has stored yet. The panel says as much. Those queries are valid and return no rows. When the tag's datatype isn't known yet, the query defaults to the numeric column. Switch it to `value_text` if the tag holds strings.

## Schema

Values are stored per contract; identity is shared across contracts.

```mermaid
erDiagram
    "umh.value_pump"     }o--|| "umh.topic" : topic_id
    "umh.attribute_pump" }o--|| "umh.topic" : topic_id
    "umh.topic"          }o--|| "umh.tag" : tag_id
    "umh.topic"          }o--|| "umh.location" : location_id

    "umh.value_pump" {
        bigint      topic_id   PK
        timestamptz ts         PK
        double      value_num
        text        value_text
    }
    "umh.attribute_pump" {
        bigint      topic_id  PK
        timestamptz ts        PK
        jsonb       attribute
    }
    "umh.topic" {
        bigserial topic_id    PK
        bigint    location_id FK
        bigint    tag_id      FK
    }
    "umh.tag" {
        bigserial  tag_id             PK
        text       name
        text       virtual_path
        text       data_contract_name
        value_type value_type
    }
    "umh.location" {
        bigserial location_id PK
        ltree     path
    }
```

`umh.value_pump` and `umh.attribute_pump` are the per-contract tables, named after `data_contract_name`; the three dimension tables are shared. `attribute` is a JSON object, queryable with `attribute->>'key'` and `attribute @> '{...}'`.

`value_type` on `umh.tag` records whether a tag is numeric or text. It is set on first write and cannot change afterwards, which is why a tag that flips datatype is dropped rather than stored.

### Resolving a tag

`umh.get_topic_id(location_path, virtual_path, data_contract, tag_name)` hides that join for single-tag lookups. It is what the generated queries use:

```sql
SELECT ts, value_num
FROM   umh.value_pump
WHERE  topic_id = umh.get_topic_id('enterprise.site.area.line', '', 'pump', 'temperature')
  AND  ts BETWEEN now() - INTERVAL '1 hour' AND now()
ORDER  BY ts;
```

Three things trip up hand-written queries:

* The timestamp column is **`ts`**, a `timestamptz`, not `timestamp` or `time`.
* A tag with no virtual path stores `virtual_path` as the **empty string**, never `NULL`. Passing `NULL` matches nothing and returns an empty result with no error.
* The `data_contract` argument is forgiving: `pump`, `_pump`, and `_pump_v1` all resolve to the same tag.

Location paths are canonicalized into an `ltree`: characters outside `[A-Za-z0-9_-]` become `_`. Hyphens survive, so `line-1` and `line_1` are **different** locations with different `topic_id`s.

### Latest value of every tag

```sql
SELECT DISTINCT ON (v.topic_id)
       l.path::text AS location, g.virtual_path, g.name AS tag, v.ts, v.value_num, v.value_text
FROM   umh.value_pump v
JOIN   umh.topic    t ON t.topic_id    = v.topic_id
JOIN   umh.tag      g ON g.tag_id      = t.tag_id
JOIN   umh.location l ON l.location_id = t.location_id
ORDER  BY v.topic_id, v.ts DESC;
```

This scans each topic's history to find its newest row, which is fine for hundreds of tags. For a dashboard that refreshes often, back the query with a continuous aggregate holding `last(value_num, ts)` per `topic_id` and read that instead.

## Using it from Grafana

Add the database as a PostgreSQL data source, paste the Grafana-flavored query into a panel, and the dashboard's time picker drives `$__timeFilter(ts)`. If you don't have Grafana yet, [Grafana](/production/deployment/docker-compose/additional-services/grafana) covers adding it to a running stack.

Point the data source at PgBouncer rather than TimescaleDB directly if your deployment has one.

## Precision

`value_num` is `DOUBLE PRECISION`, a binary floating-point type. It stores an approximation of the value, which is close enough for sensor readings but wrong for anything that has to come back byte-for-byte: integer counters above 2^53 lose their low digits, and a decimal such as `0.1` is kept as the nearest binary fraction. Route those tags to a text contract, where the value is stored verbatim in `value_text`.

The [Historian output reference](https://docs.umh.app/benthos-umh/output/historian) covers the rest of what the output plugin does: metrics, error classes, throughput tuning, and schema compatibility.


# Management Console

The Management Console is your cloud-based control center for managing UMH Core instances. It provides a modern web interface for deploying, configuring, and monitoring your industrial IoT infrastructure without touching command lines or YAML files.

## Getting Started

1. **Sign up** at [management.umh.app](https://management.umh.app)
2. **Choose authentication**:
   * Email/password
   * Google SSO
   * LinkedIn SSO
3. **Create your first instance** and start collecting data

![Login Screen](/files/l5UD0N85bFCPfmF3HbWh)

## Key Features

### Instance Management

* Deploy new UMH Core instances with one click
* Monitor health, performance, and data flows
* Edit configurations through the UI or YAML
* Access Topic Browser and logs

### Account & Organization

* Company-level management
* User permissions and roles (Super-Admin, custom roles)
* License management
* Early access features toggle for experimental functionality

### Settings & Preferences

* **Advanced Mode**: Unlock power user features
* **Early Access Features**: Try upcoming capabilities like

## Authentication & Security

The Management Console uses Auth0 for secure authentication with support for:

* Traditional email/password
* Single Sign-On (SSO) via Google and LinkedIn
* Enterprise SSO
* Role-based access control (RBAC)

## Detailed Documentation Coming Soon

Full documentation including user management, team collaboration, API access, and enterprise features will be available soon. For now, the UI is designed to be self-explanatory - start by creating your first instance and exploring the features.


# Production

This section provides essential guidance for deploying and maintaining UMH Core in production environments. Topics include system sizing recommendations, key metrics to monitor for optimal performance, update procedures for seamless maintenance, and configuration considerations for corporate firewall environments. Use these resources to ensure reliable, scalable deployments of your industrial data infrastructure.


# Architecture Patterns

Architecture patterns describe how to deploy UMH Core for different operational requirements. Each page explains a topology: what components are involved, how they connect, what the pattern guarantees, and when to use it.

These are not step-by-step guides. They describe architectures so you can choose the right one for your site. For deployment procedures, see [Deployment](/production/deployment). For infrastructure-level recovery (container restarts, storage, node rescheduling), see [High Availability](/production/high-availability).

## Patterns

* [Redundant Data Collection](/production/architecture-patterns/redundant-data-collection) -- two umh-core instances reading from redundant PLCs for zero data loss


# Redundant Data Collection

{% hint style="info" %}
This pattern applies to sites with **redundant PLCs** -- two physical controllers publishing the same data. Redundant PLCs are common in process manufacturing (oil and gas, chemicals, power generation, pharmaceuticals) where the process cannot be safely stopped. If you run standard, non-redundant PLCs -- as most discrete manufacturing sites do -- this page does not apply to you. A single umh-core instance already provides sub-second process recovery via S6, and your container manager handles container-level restarts. See [High Availability](/production/high-availability).
{% endhint %}

## The problem

When a site has redundant PLCs, each controller independently reads the same sensors. If the single umh-core instance collecting data goes down, data is lost until it recovers -- even though a second PLC could have provided the same readings.

## The solution

Run two umh-core instances, each connected to one of the redundant PLCs. Each instance collects into its own [Unified Namespace](/usage/unified-namespace) under a PLC-specific topic:

```
umh.v1.enterprise.line.PLC-A._historian   (umh-core 1, from its own bridge)
umh.v1.enterprise.line.PLC-B._historian   (umh-core 2, from its own bridge)
```

umh-core 2 pushes its data into umh-core 1's UNS. umh-core 1 now has both PLC-specific topics. A [stream processor](/usage/data-flows/stream-processor) then does redundancy resolution: it selects the primary source from the two streams and writes a single consolidated topic:

```
umh.v1.enterprise.line._historian         (consolidated, PLC-agnostic)
```

The higher-level system (a data lake, an MQTT broker, or another factory-wide umh-core instance) consumes this consolidated topic. It does not need to track which PLC provided each reading, but the original source is always traceable through the PLC-specific topics and message metadata.

![Redundant data collection topology: two umh-core instances each collecting from one PLC, umh-core 2 pushes data to umh-core 1 which does redundancy resolution and forwards to the higher-level system](/files/Y1EXbABy9ifrkOLlN853)

All original data from both collectors is retained in the PLC-specific source topics. Redundancy resolution selects the primary source for downstream processing. It does not delete or modify the original records.

If one instance fails, the other continues collecting into its own UNS. When the failed instance recovers, the bridge resumes pushing data and redundancy resolution continues.

## Prerequisites

* **Two redundant PLCs.** Two physical or logical controllers that publish the same data. A single PLC with two IP addresses does not qualify -- you need hardware designed for redundancy. The redundant PLCs must produce equivalent data: the same sensors, the same values, and ideally synchronized timestamps.
* **High-availability deployment.** Both umh-core instances must be deployed so that a single infrastructure failure does not take both down at the same time. See [High Availability](/production/high-availability) for storage and deployment requirements.
* **A redundancy resolution strategy.** The stream processor needs a rule for selecting the primary source. The strategy depends on the protocol and what timestamps the PLCs provide. OPC UA subscriptions provide a device-level SourceTimestamp. If both PLCs are synchronized to a common time source (for example PTP/IEEE 1588), timestamp-based resolution works well. Clock synchronization between the redundant PLCs is a site prerequisite, not something OPC UA or umh-core provides. Polling-based protocols (Modbus, S7) do not provide device-level timestamps and require a different approach, for example last-value-wins within a time window.

## Recovery behavior

Two terms used in the table below:

* **MTTR** (Mean Time To Recovery): how long until data collection resumes after a failure.
* **RPO** (Recovery Point Objective): how much data is lost during the failure. RPO = 0 means no data is lost.

| Failure                        | MTTR               | RPO | What happens                                                                                                                                                                                                          |
| ------------------------------ | ------------------ | --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Process crash inside container | Seconds to minutes | 0   | S6 restarts the process in sub-second, but data collection resumes only after the bridge reconnects to the PLC (seconds to minutes depending on protocol and number of nodes). The other instance was never affected. |
| Container or pod crash         | 30-60 seconds      | 0   | Container manager restarts the container. The other instance continues.                                                                                                                                               |
| Full instance failure          | 2-5 minutes        | 0   | Infrastructure reschedules the instance (depends on failure detection timeout and storage reattachment). The other instance continues. Data already in each instance's UNS is preserved.                              |

RPO is zero in all three cases because the second instance is always collecting independently. Even during the rescheduling window, one instance is still publishing. This guarantee covers single-instance failure only. Simultaneous failure of both instances (for example, a site-wide power loss) is not covered by this pattern. See [High Availability](/production/high-availability) for infrastructure-level mitigations.

**Note on OPC UA reconnection.** After a connection loss, the OPC UA client must re-establish the TCP connection, create a new session, re-browse the node tree, and re-create subscriptions. This typically takes seconds to minutes depending on the server and number of nodes. This is why the redundant pattern matters: the second instance bridges the gap while the first reconnects.

## When this pattern does not apply

* **Your PLC is not redundant.** Connecting two bridges to a single PLC doubles the load on it without improving availability. Redundant collection requires two separate endpoints publishing the same data. A single PLC with two IP addresses does not qualify -- if the controller fails, both addresses go down. (If your goal is to avoid data gaps during planned updates rather than hardware failure, a single PLC with multiple addresses may be sufficient for a zero-downtime update strategy instead.)
* **You only need process-level or container-level recovery.** A standard single umh-core deployment already provides sub-second process recovery and 30-60 second container recovery. No additional topology is needed. See [High Availability](/production/high-availability).
* **You are in discrete manufacturing.** Automotive assembly lines, packaging lines, and machine tools typically use standard PLCs without redundancy. This pattern adds complexity without benefit in those environments.

## Related

* [High Availability](/production/high-availability) -- infrastructure-level recovery (container restarts, storage, node rescheduling)
* [Architecture Patterns](/production/architecture-patterns) -- overview of all deployment patterns


# Deployment

This section covers how to deploy umh-core in production environments.

## Deployment Options

umh-core is distributed as a container image. We provide instructions for using [Docker](/production/deployment/docker) and [Docker Compose](/production/deployment/docker-compose) to deploy it.


# Docker

This section covers deploying umh-core using the Docker CLI.


# Setup

You can run umh-core without the Management Console. But you'll need to edit configuration files directly.

Find the latest version on the [Releases](https://github.com/united-manufacturing-hub/united-manufacturing-hub/releases) page and replace `<VERSION>` with your selected version.

```bash
docker volume create umh-core-data

docker run -d \
  --name umh-core \
  --restart unless-stopped \
  --env AUTH_TOKEN=your-token \
  --env LOCATION_0=your-location \
  --env RELEASE_CHANNEL=stable \
  --env API_URL=https://management.umh.app/api \
  --volume umh-core-data:/data \
  management.umh.app/oci/united-manufacturing-hub/umh-core:<VERSION>
```


# Updating

To update umh-core, stop the container and start a new one with the latest image.

Find the latest version on the [Releases](https://github.com/united-manufacturing-hub/united-manufacturing-hub/releases) page and replace `<VERSION>` with your selected version.

```bash
docker stop umh-core
docker rm umh-core

docker run -d \
  --name umh-core \
  --restart unless-stopped \
  --volume umh-core-data:/data \
  management.umh.app/oci/united-manufacturing-hub/umh-core:<VERSION>
```

**That's it!** Your data is preserved in the `umh-core-data` volume.

> **Note:** On Linux without Docker group membership, prefix commands with `sudo`.

***

> **Using a custom data folder?** If you manually specified a folder path instead of using a Docker volume, see [Container Layout](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/production/deployment/reference/container-layout.md#advanced-custom-data-location) for upgrade instructions.


# Docker Compose

This documentation describes how to deploy umh-core using Docker Compose as an alternative to the standard `docker run` command mentioned in the [Getting Started section](/getting-started).

## Why Docker Compose?

While umh-core can be started with a single `docker run` command, real-world deployments often require additional services such as databases, visualization tools, and reverse proxies. Docker Compose allows you to define and manage multi-container applications in a single, declarative YAML file. Compared to using the Docker CLI alone, this is easier to set up and maintain.

If you are unfamiliar with Docker Compose, refer to the [official Docker Compose documentation](https://docs.docker.com/compose/) for a comprehensive introduction.

## What is Docker Compose

Docker Compose uses YAML files for the declaration of Docker Resources. These files are called `docker-compose.yaml`.

In these files Docker Volumes, Docker Networks and Docker Containers can all be declared and configured to work with eachother. This makes Docker Compose particularly valuable when additional Services like Grafana or TimescaleDB should be deployed alongside umh-core. But it is also valuable if only umh-core is deployed as it keeps the configuration of the container in a file instead of being lost to a one-off command.

For the configuration we recommend to most deployments, see the [Recommended UMH Stack](/production/deployment/docker-compose/additional-services/recommended-umh-stack).


# Setup

This guide walks you through setting up umh-core with Docker Compose, starting from a minimal configuration you can extend with [additional services](/production/deployment/docker-compose/additional-services).

## What you'll need

* [Docker](https://docs.docker.com/get-started/introduction/get-docker-desktop/) installed on your system
* Basic familiarity with the command line

## Minimal Setup

1. Find the latest version on the [Releases](https://github.com/united-manufacturing-hub/united-manufacturing-hub/releases) page and replace `ENTER_VERSION_HERE` with your selected version.
2. Save this in `docker-compose.yaml`.

> **Hint:** If your desired release is `v0.44.31`, the image tag is: `umh-core:v0.44.31`

```yaml
services:
  umh:
    # TODO: set your desired container version here 
    # e.g. `umh-core:v0.44.31`
    image: management.umh.app/oci/united-manufacturing-hub/umh-core:ENTER_VERSION_HERE
    restart: unless-stopped
    volumes:
      - umh-data:/data
    environment:
      # TODO: Enter your instance's Auth Token.
      # You'll find it in your instance's
      # configuration file on management.umh.app.
      - AUTH_TOKEN=your_auth_token
      # TODO: Change the LOCATION_0 parameter
      # to your desired Level 0 Location name
      - LOCATION_0=your_level_0_location
      # Optional: Define more levels
      # - LOCATION_1=your_level_1
      # - LOCATION_2=your_level_2
      # - LOCATION_3=your_level_3
      # - LOCATION_4=your_level_4
      - RELEASE_CHANNEL=stable
      - API_URL=https://management.umh.app/api

volumes:
  umh-data: {}
```

This achieves the same result as the docker cli commands, but the configuration is now documented in a file that you can version control and extend.

## Starting the Stack

Once the configuration is done you can start the stack:

```bash
docker compose up -d
```

We recommend that you familiarise yourself with the docker commands below. They'll come in handy while working with umh-core.

* `docker compose ps`: check the running state of all services
* `docker compose stats`: check resource usage of all services
* `docker compose pull`: re-pull the image tags currently listed in the `docker-compose.yaml`

> **Note:** `docker compose pull` does not move you to a newer UMH release. The image tags here are pinned to a specific version, so the pull only refreshes that same version. To deploy a newer release, change the tag in `docker-compose.yaml` first. See [Updating](/production/deployment/docker-compose/updating).

For more refer to the [official Docker Compose documentation](https://docs.docker.com/compose/).

## Add More Services

While umh-core can operate standalone, most deployments benefit from persistent storage and visualization capabilities.

* [Recommended UMH Stack](/production/deployment/docker-compose/additional-services/recommended-umh-stack): umh-core, Grafana, PgBouncer, and TimescaleDB in one file
* [TimescaleDB](/production/deployment/docker-compose/additional-services/timescaledb): time-series storage, with PgBouncer in front of it
* [Grafana](/production/deployment/docker-compose/additional-services/grafana): create dashboards that run locally from your data
* [nginx](/production/deployment/docker-compose/additional-services/nginx): reverse proxy and SSL termination


# Additional Services

Docker Compose lets you run more than umh-core. Each page in this section explains how to extend `services:` block of your `docker-compose.yaml`, with a complete configuration you can copy.

Most deployments want a database and dashboards, so start with the [Recommended UMH Stack](/production/deployment/docker-compose/additional-services/recommended-umh-stack), which combines umh-core, Grafana, PgBouncer, and TimescaleDB in one file.

To add services one at a time, see [TimescaleDB](/production/deployment/docker-compose/additional-services/timescaledb), [Grafana](/production/deployment/docker-compose/additional-services/grafana), or [nginx](/production/deployment/docker-compose/additional-services/nginx).


# Recommended UMH Stack

This is the Docker Compose configuration we recommend for most deployments: umh-core, Grafana, PgBouncer, and TimescaleDB in one file. If you are new to running umh-core with Docker Compose, read [Setup](/production/deployment/docker-compose/setup) first.

## Why these four services

umh-core collects and standardizes your data, but it does not keep a long history and it does not draw charts. The other three cover that.

* **TimescaleDB** is PostgreSQL with a time-series extension. Sensor readings, production counts, and machine states all arrive with timestamps, which is exactly what it is built to store and query.
* **Grafana** reads from TimescaleDB and turns it into dashboards. On its own it has nothing to query.
* **PgBouncer** sits in front of TimescaleDB as a connection pooler. umh-core bridges can open many simultaneous connections, and PostgreSQL handles a limited number of them (typically 100 by default). PgBouncer pools hundreds of client connections onto a handful of connections.

TimescaleDB is placed in an internal network that only PgBouncer can reach, so nothing else talks to the database directly.

## The configuration

Copy this into `docker-compose.yaml` and fill in the fields marked `TODO`. Find the latest umh-core version on the [Releases](https://github.com/united-manufacturing-hub/united-manufacturing-hub/releases) page and replace `ENTER_VERSION_HERE`.

> **IMPORTANT:** Change the database and Grafana credentials before using this in production!

```yaml
services:
  umh:
    # TODO: set your desired container version here 
    # e.g. `umh-core:v0.44.31`
    image: management.umh.app/oci/united-manufacturing-hub/umh-core:ENTER_VERSION_HERE
    restart: unless-stopped
    volumes:
      - umh-data:/data
    environment:
      # TODO: Enter your instance's Auth Token. 
      # You'll find it in your instance's configuration
      # file on management.umh.app.
      - AUTH_TOKEN=your_auth_token
      # TODO: Change the LOCATION_0 parameter
      # to your desired Level 0 Location name
      - LOCATION_0=your_level_0_location
      # Optional: Define more levels
      # - LOCATION_1=your_level_1
      # - LOCATION_2=your_level_2
      # - LOCATION_3=your_level_3
      # - LOCATION_4=your_level_4
      - RELEASE_CHANNEL=stable
      - API_URL=https://management.umh.app/api

  grafana:
    image: management.umh.app/oci/grafana/grafana:12.3.0
    restart: unless-stopped
    ports:
      - 3000:3000
    environment:
      # TODO: Set your desired username and password here
      # You'll need these credentials to
      # access your local Grafana instance
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
    healthcheck:
      test: ["CMD-SHELL", "curl --fail http://grafana:3000/api/health"]
      interval: 10s
      timeout: 5s
      retries: 3

  timescaledb:
    image: management.umh.app/oci/timescale/timescaledb:2.24.0-pg17
    restart: unless-stopped
    environment:
      - POSTGRES_DB=umh
      # TODO: Set your postgresDB
      # Password and Username here
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes:
      - timescaledb-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h timescaledb"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - timescaledb-network

  pgbouncer:
    image: management.umh.app/oci/edoburu/pgbouncer:v1.24.1-p1
    restart: unless-stopped
    environment:
      - DB_NAME=umh
      # This has to be the same value as
      # timescaledb.environment.POSTGRES_USER
      - DB_USER=postgres
      # This has to be the same value as
      # timescaledb.environment.POSTGRES_PASSWORD
      - DB_PASSWORD=postgres
      - DB_HOST=timescaledb
      - AUTH_TYPE=scram-sha-256
    ports:
      - 5432:5432
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h pgbouncer"]
      interval: 10s
      timeout: 5s
      retries: 5
    depends_on:
      timescaledb:
        condition: service_healthy
    networks:
      - default
      - timescaledb-network

networks:
  default:
  timescaledb-network:
    internal: true

volumes:
  umh-data: {}
  timescaledb-data: {}
  grafana-data: {}
```

Prefer to add one service at a time? Each page carries the same configuration in smaller pieces: [TimescaleDB](/production/deployment/docker-compose/additional-services/timescaledb), [Grafana](/production/deployment/docker-compose/additional-services/grafana).

To start the stack, see [Starting the Stack](/production/deployment/docker-compose/setup#starting-the-stack).

## VM Sizing

Our [**Sizing Guide**](/production/sizing-guide) covers umh-core on its own, and its recommended box is sized for that. Redpanda inside umh-core already budgets roughly 2 GB per core plus headroom, so running a database and Grafana next to it on the minimum box leaves little room. Give the host more memory and disk than the baseline, and watch actual usage before settling on a size.

## What's next

Starting the stack gives you four running containers. Two things still need configuring before a dashboard shows anything:

1. **Get data into TimescaleDB.** umh-core does not write to the database by itself. You configure a flow that reads from the Unified Namespace and writes to PostgreSQL. See [Stand-alone Flow](/usage/data-flows/stand-alone-flow), which covers the `kafka_to_postgresql_historian_bridge`.
2. **Point Grafana at the database.** Add a PostgreSQL data source in Grafana using host `pgbouncer:5432`, database `umh`, and the credentials you set above. See Grafana's [PostgreSQL data source documentation](https://grafana.com/docs/grafana/latest/datasources/postgres/).


# TimescaleDB

In this example you'll learn how to add TimescaleDB and PgBouncer to a umh-core Docker Compose stack. If you are new to running umh-core with Docker Compose, read [Setup](/production/deployment/docker-compose/setup) first.

[TimescaleDB](https://www.timescale.com/) is PostgreSQL with an extension optimized for time-series data. Manufacturing data is inherently time-series: sensor readings, production counts, and machine states all have timestamps.

PgBouncer is a connection pooler for TimescaleDB. umh-core bridges can create many simultaneous connections to the database, and PostgreSQL handles a limited number of them (typically 100 by default), so without pooling a busy system exhausts the limit and connections start failing. Applications connect to PgBouncer on port 5432 and PgBouncer connects to TimescaleDB internally.

## Complete docker-compose.yaml

umh-core with TimescaleDB and PgBouncer. Copy this into `docker-compose.yaml` and fill in the fields marked `TODO`.

> **IMPORTANT**: Change the template's database credentials before using this in production!

```yaml
services:
  umh:
    # TODO: set your desired container version here 
    # e.g. `umh-core:v0.44.31`
    image: management.umh.app/oci/united-manufacturing-hub/umh-core:ENTER_VERSION_HERE
    restart: unless-stopped
    volumes:
      - umh-data:/data
    environment:
      # TODO: Enter your instance's Auth Token.
      # You'll find it in your instance's configuration
      # file on management.umh.app.
      - AUTH_TOKEN=your_auth_token
      # TODO: Change the LOCATION_0 parameter
      # to your desired Level 0 Location name
      - LOCATION_0=your_level_0_location
      # Optional: Define more levels
      # - LOCATION_1=your_level_1
      # - LOCATION_2=your_level_2
      # - LOCATION_3=your_level_3
      # - LOCATION_4=your_level_4
      - RELEASE_CHANNEL=stable
      - API_URL=https://management.umh.app/api

  timescaledb:
    image: management.umh.app/oci/timescale/timescaledb:2.24.0-pg17
    restart: unless-stopped
    environment:
      - POSTGRES_DB=umh
      # TODO: Set your postgresDB Password and Username here
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes:
      - timescaledb-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h timescaledb"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - timescaledb-network

  pgbouncer:
    image: management.umh.app/oci/edoburu/pgbouncer:v1.24.1-p1
    restart: unless-stopped
    environment:
      - DB_NAME=umh
      # This has to be the same value as
      # timescaledb.environment.POSTGRES_USER
      - DB_USER=postgres
      # This has to be the same value as
      # timescaledb.environment.POSTGRES_PASSWORD
      - DB_PASSWORD=postgres
      - DB_HOST=timescaledb
      - AUTH_TYPE=scram-sha-256
    ports:
      - 5432:5432
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h pgbouncer"]
      interval: 10s
      timeout: 5s
      retries: 5
    depends_on:
      timescaledb:
        condition: service_healthy
    networks:
      - default
      - timescaledb-network

networks:
  default:
  timescaledb-network:
    internal: true

volumes:
  umh-data: {}
  timescaledb-data: {}
```

To start the stack, see [Starting the Stack](/production/deployment/docker-compose/setup#starting-the-stack).

## Already running umh-core?

Add the following to the `docker-compose.yaml` you already have. Your existing `umh:` service stays as it is.

**1. Two new services inside `services:`**

```yaml
  timescaledb:
    image: management.umh.app/oci/timescale/timescaledb:2.24.0-pg17
    restart: unless-stopped
    environment:
      - POSTGRES_DB=umh
      # TODO: Set your postgresDB Password and Username here
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes:
      - timescaledb-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h timescaledb"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - timescaledb-network

  pgbouncer:
    image: management.umh.app/oci/edoburu/pgbouncer:v1.24.1-p1
    restart: unless-stopped
    environment:
      - DB_NAME=umh
      # This has to be the same value as
      # timescaledb.environment.POSTGRES_USER
      - DB_USER=postgres
      # This has to be the same value as
      # timescaledb.environment.POSTGRES_PASSWORD
      - DB_PASSWORD=postgres
      - DB_HOST=timescaledb
      - AUTH_TYPE=scram-sha-256
    ports:
      - 5432:5432
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h pgbouncer"]
      interval: 10s
      timeout: 5s
      retries: 5
    depends_on:
      timescaledb:
        condition: service_healthy
    networks:
      - default
      - timescaledb-network
```

**2. A new top-level `networks:` section**

```yaml
networks:
  default:
  timescaledb-network:
    internal: true
```

**3. One more entry under `volumes:`**

```yaml
  timescaledb-data: {}
```

**4. Apply the changes**

```bash
docker compose up -d
```

See [Starting the Stack](/production/deployment/docker-compose/setup#starting-the-stack) for the other compose commands.

## What this declares

* 2 Services called `pgbouncer` and `timescaledb`: PgBouncer acts as a proxy to TimescaleDB. This means the Credentials have to match between these two Services. The `depends_on` entry with `condition: service_healthy` on `pgbouncer` enforces the startup order; the Healthchecks only report whether a Service is healthy. `timescaledb` is isolated in the Network `timescaledb-network`. `pgbouncer` is both in the `default` and the `timescaledb-network`. This makes PgBouncer the only Service which can talk to TimescaleDB directly.
* 2 Networks called `timescaledb-network` and `default`: Without the Network section the Network `default` is always created by default. Services use the Network `default` if no explicit Network configuration is provided. The `timescaledb-network` is configured to be internal which means Services can't reach the internet or any Service outside this Network if they are only connected through this Network.
* 1 Volume called `timescaledb-data`: This is where TimescaleDB stores its data.

Once the stack is running, PostgreSQL is reachable through PgBouncer at `localhost:5432`.

A database on its own has nothing to show you. Most deployments add [Grafana](/production/deployment/docker-compose/additional-services/grafana) on top of it, which is why both are part of the [**Recommended UMH Stack**](/production/deployment/docker-compose/additional-services/recommended-umh-stack).


# Grafana

In this example you'll learn how to add Grafana to a umh-core Docker Compose stack. If you're not sure about umh-core and Docker Compose stacks, start with [Setup](/production/deployment/docker-compose/setup).

[Grafana](https://grafana.com/) is an open-source visualization platform. It allows you to build dashboards showing real-time and historical data based on your umh-core configuration.

Grafana's admin account defaults to `admin` / `admin`. Port 3000 is published to the host, so set both values before you start the stack. The variable names follow Grafana's `GF_<SECTION>_<KEY>` convention, documented in the [Grafana configuration reference](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/).

## Complete docker-compose.yaml

umh-core with Grafana. Copy this into `docker-compose.yaml` and fill in the fields marked `TODO`.

```yaml
services:
  umh:
    # TODO: set your desired container version here 
    # e.g. `umh-core:v0.44.31`
    image: management.umh.app/oci/united-manufacturing-hub/umh-core:ENTER_VERSION_HERE
    restart: unless-stopped
    volumes:
      - umh-data:/data
    environment:
      # TODO: Enter your instance's Auth Token. 
      # You'll find it in your instance's configuration
      # file on management.umh.app.
      - AUTH_TOKEN=your_auth_token
      # TODO: Change the LOCATION_0 parameter
      # to your desired Level 0 Location name
      - LOCATION_0=your_level_0_location
      # Optional: Define more levels
      # - LOCATION_1=your_level_1
      # - LOCATION_2=your_level_2
      # - LOCATION_3=your_level_3
      # - LOCATION_4=your_level_4
      - RELEASE_CHANNEL=stable
      - API_URL=https://management.umh.app/api

  grafana:
    image: management.umh.app/oci/grafana/grafana:12.3.0
    restart: unless-stopped
    ports:
      - 3000:3000
    environment:
      # TODO: Set your desired username and password here
      # You'll need these credentials to
      # access your local Grafana instance
      - GF_SECURITY_ADMIN_USER=admin     
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
    healthcheck:
      test: ["CMD-SHELL", "curl --fail http://grafana:3000/api/health"]
      interval: 10s
      timeout: 5s
      retries: 3

volumes:
  umh-data: {}
  grafana-data: {}
```

To start the stack, see [Starting the Stack](/production/deployment/docker-compose/setup#starting-the-stack).

## Already running umh-core?

Add the following to the `docker-compose.yaml` you already have. Your existing `umh:` service stays as it is.

**1. One new service inside `services:`**

```yaml
  grafana:
    image: management.umh.app/oci/grafana/grafana:12.3.0
    restart: unless-stopped
    ports:
      - 3000:3000
    environment:
      # TODO: Set your desired username and password here
      # You'll need these credentials to
      # access your local Grafana instance
      - GF_SECURITY_ADMIN_USER=admin     
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
    healthcheck:
      test: ["CMD-SHELL", "curl --fail http://grafana:3000/api/health"]
      interval: 10s
      timeout: 5s
      retries: 3
```

**2. One more entry under `volumes:`**

```yaml
  grafana-data: {}
```

**3. Apply the changes**

```bash
docker compose up -d
```

See [Starting the Stack](/production/deployment/docker-compose/setup#starting-the-stack) for the other compose commands.

## Connecting to Grafana

Once the stack is running, Grafana is reachable at `http://localhost:3000`.

To log into Grafana, use the credentials that you defined in `docker-compose.yaml`.

> 💡 Without a persistent database behind it there is nothing to query, so most deployments run Grafana together with TimescaleDB. We call it the [**Recommended UMH Stack**](/production/deployment/docker-compose/additional-services/recommended-umh-stack).


# nginx

Nginx is a high-performance web server and reverse proxy. In a umh-core deployment, nginx can be used as the proxy for incoming HTTP requests, handling SSL and routing requests to the appropriate services.

## Why a proxy?

umh-core pipelines can expose HTTP endpoints using the `http_server` input. These endpoints allow external systems to send data to umh-core. However, these endpoints should be secured. It is possible to manage certificates in each of those inputs but it is inconvenient to maintain as certificates have to be replaced regularly.

## Adding Nginx in the existing Docker Compose Configuration

Add nginx to your `docker-compose.yaml` within the `services` section:

```diff
  services:
+   nginx:
+     image: management.umh.app/oci/library/nginx:1.27.3-alpine
+     restart: unless-stopped
+     ports:
+       - 80:80   # for http access
+       - 443:443 # for https access
```

## Nginx Configuration File

Refer to the [Nginx documentation](https://nginx.org/en/docs/index.html) for how to create your own configuration.

If you want to customize the `nginx.conf` create an `nginx.conf` file in the same directory as your `docker-compose.yaml` and mount it:

```diff
  services:
    umh:
      # ... existing configuration ...
    nginx:
      image: management.umh.app/oci/library/nginx:1.27.3-alpine
      restart: unless-stopped
      ports:
        - 80:80   # for http access
        - 443:443 # for https access
+     volumes:
+       - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
```

Refer to the [Nginx documentation](https://nginx.org/en/docs/index.html) for what you can do in this configuration.

## Adding a simple http/https forward

```conf
# example of an http/https server configuration for forwarding requests to umh
server {
  listen 80;
  listen 443 ssl;
  server_tokens off;

  # Add CORS headers globally for this server
  add_header 'Access-Control-Allow-Origin' '*' always;
  add_header 'Access-Control-Allow-Methods' 'POST,GET,HEAD,OPTIONS' always;
  add_header 'Access-Control-Allow-Headers' 'content-type' always;

  # Handle OPTIONS requests
  if ($request_method = 'OPTIONS') {
      return 204;
  }

  location /example/route/ {
      # docker exposes the service `umh` as a hostname
      # the port is an example for an http_input that has to be created in umh
      proxy_pass http://umh:8285;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
  }
}
```

## SSL

Obtain certificates from a trusted Certificate Authority (CA) or use Let's Encrypt. Place the `*.crt` and `*.key` file in the `certs` directory and create a `nginx.conf` for nginx which uses those certificates.

The file structure should now look similar to this.

```
.
├── certs
│   ├── example.crt
│   └── example.key
├── docker-compose.yaml
└── nginx.conf
```

The `docker-compose.yaml` has to mount `certs` and `nginx.conf` in appropriate places for nginx. This would look similar to this example:

```diff
  services:
    nginx:
      image: management.umh.app/oci/library/nginx:1.27.3-alpine
      restart: unless-stopped
      ports:
        - 80:80   # for http access
        - 443:443 # for https access
+     volumes:
+       - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
+       - ./certs:/etc/nginx/tls:ro
```


# Updating

To update umh-core, edit the image tag in your `docker-compose.yaml` and restart the stack.

1. Find the latest version on the [Releases](https://github.com/united-manufacturing-hub/united-manufacturing-hub/releases) page and replace `ENTER_VERSION_HERE` with your selected version.
2. Update the umh-core image tag in `docker-compose.yaml`:

```yaml
services:
  umh:
    # make sure to include 'v' in the version string like so: v0.44.31
    image: management.umh.app/oci/united-manufacturing-hub/umh-core:ENTER_VERSION_HERE
```

3. Pull the new image and restart:

```bash
docker compose up -d
```

**That's it!** Your data is preserved in the named volumes.

> **Note:** On Linux without Docker group membership, prefix commands with `sudo`.


# Sizing Guide

## Supported Architectures

| Architecture | Status            | Notes                                                                             |
| ------------ | ----------------- | --------------------------------------------------------------------------------- |
| x86\_64      | ✅ Fully supported | Intel/AMD 64-bit processors                                                       |
| ARM64        | ✅ Fully supported | 64-bit ARM processors (AWS Graviton, Apple Silicon, etc.)                         |
| Raspberry Pi | ❌ Not supported   | Redpanda requires 48-bit virtual address space; Raspberry Pi provides only 38-bit |

**Why Raspberry Pi doesn't work:** Redpanda uses the Seastar framework which requires a 48-bit virtual address space. Raspberry Pi's ARM processors only provide 38-bit VA space, causing SIGABRT on startup. This is a fundamental hardware limitation documented in [Redpanda GitHub Issue #1542](https://github.com/redpanda-data/redpanda/issues/1542).

## Recommended Starting Point

**Start with → 2 vCPU · 4 GB RAM · 40 GB SSD**

#### What that box handles

* **≈ 4 bridges instances** (e.g. OPC UA ➜ Redpanda) **plus one bridge instance** that forwards from the local Redpanda to an external MQTT broker
* **≈ 900 tags at 1 message / second each**
* Keeps **seven days** of replayable raw history under the defaults a new instance starts with (`log_retention_ms = 7 days`, `log_cleanup_policy = delete`)
* Runs comfortably below 70% CPU (automatic throttling protection kicks in above this)

#### Disk usage in practice

Redpanda writes **128 MiB segments**; a segment can be deleted only after it is closed.\
With Snappy compression, a typical 200 B JSON payload shrinks to ≈ 50–70 B (3–4× ratio).\
Allowing a 5 GB safety buffer, a 40 GB SSD gives **≈ 35 GB usable history ≙ \~500–700 million messages**.

Size the disk against your message rate, not your tag count: seven days of raw history is what makes replay possible, and time-based retention has no size ceiling of its own. At 700 messages / second — ≈ 423 million records over seven days — `umh.messages` alone needs roughly **40–55 GB**. Measure the real figure on your instance, since payload size drives it.

*Need more?*\
Shorten retention (either during install with `internal.redpanda.redpandaServiceConfig.topic.defaultTopicRetentionMs` or later on the topic level using `rpk`), cap the topic with `defaultTopicRetentionBytes`, or enlarge the disk. A byte cap protects the disk but can cut the replay window below seven days under heavy load.

#### Memory

| Component           | Rule of thumb                                    |
| ------------------- | ------------------------------------------------ |
| Redpanda            | ≈ 2 GB · cores + 1.5 GB head-room (Seastar rule) |
| Agent + supervision | ≈ 150 MB                                         |
| Each extra pipeline | ≈ 100 MB                                         |

#### CPU

**Theoretical Bridge Limits:**

* We recommend **5 bridges per CPU core** (after reserving 1 core for Redpanda)
* Example: 2 CPU cores = (2-1) × 5 = **5 bridges maximum**
* Example: 4 CPU cores = (4-1) × 5 = **15 bridges maximum**

**Dynamic Resource Protection:** Since every bridge has different resource requirements (OPC UA with 10,000 tags uses more CPU than MQTT with 100 tags), we also monitor actual resource usage:

* **CPU Utilization**: Blocks new bridges if CPU usage exceeds 70%
* **CPU Throttling**: Blocks if the container is being throttled. Throttling means the system needs brief CPU bursts (e.g., when processing message batches) but hits the CPU limit, causing delays and degraded performance even if average CPU usage looks acceptable
* **Memory Usage**: Blocks if memory exceeds 80%
* **Disk Usage**: Blocks if disk exceeds 85%

**Redpanda CPU Utilization:** UMH Core runs Redpanda with the `--overprovisioned` flag, which optimizes CPU usage for containerized environments. This disables Seastar's busy-polling reactor model, reducing idle CPU usage from 100% to near-zero when not processing messages. The trade-off is slightly higher latency (microseconds to low milliseconds), which is acceptable for manufacturing data that doesn't require sub-millisecond response times. This is required because UMH Core runs in Docker where CPU pinning doesn't work effectively, and Redpanda shares the container with other processes.

**Automatic Enforcement:** The system will prevent you from deploying new bridges if:

1. You've reached the theoretical limit for your CPU allocation, OR
2. The system detects resource degradation (high CPU, throttling, memory, or disk pressure)

This resource-based blocking is controlled by a feature flag and can be configured in your `config.yaml`:

```yaml
agent:
  enableResourceLimitBlocking: false  # Disable resource-based bridge blocking (default: true)
```

When enabled, this ensures system stability and prevents one bridge from impacting others. If you need more bridges, either:

* Increase CPU allocation (for containerized deployments)
* Upgrade to a larger instance (for VM/bare-metal deployments)
* Optimize existing bridges (reduce polling rates, tag counts, or processing complexity)

#### Resource Limit Error Messages

When the system blocks bridge creation, you'll see clear messages explaining why:

* **Bridge limit**: `Cannot create bridge - limit exceeded (5 bridges maximum with 2.0 CPU cores, 1 core reserved for Redpanda)`
* **CPU throttling**: `CPU throttled (15% of time). Container limited to 2.0 cores, needs more during peaks (host has 8 cores available)`
* **High CPU**: `CPU degraded: CPU utilization critical`
* **High Memory**: `Memory degraded: Memory usage at 85%`
* **High Disk**: `Disk degraded: Disk usage at 90%`

#### Easy vertical scaling

UMH Core is stateless besides the **`/data`** volume. To grow:

1. Stop the container
2. Move or resize the volume / attach it to a bigger VM
3. Start the same image — no reinstall or re-configuration required

#### High availability

For multi-node failover, mount `/data` on storage accessible from multiple nodes. See [High Availability](/production/high-availability).


# High Availability

UMH Core handles high availability without custom failover daemons. Inside the container, [S6](https://skarnet.org/software/s6/) supervises all processes and restarts them on failure (see [Container Layout](/reference/container-layout) for details). Outside the container, your container manager (Docker, Kubernetes, Docker Swarm, Portainer, or any other) handles container restarts and rescheduling.

**Typical recovery times** (actual times depend on configuration and workload):

| Failure                        | Recovery                                                                    |
| ------------------------------ | --------------------------------------------------------------------------- |
| Process crash inside container | Sub-second (S6 restarts the process)                                        |
| Container/pod crash            | 30-60 seconds (depends on restart policy and resource availability)         |
| Node failure                   | 2-5 minutes (depends on failure detection timeout and storage reattachment) |

## How Failover Works

All state lives in the `/data` volume (see [Container Layout](/reference/container-layout) for structure). When a container or node fails, your container manager starts a new container that mounts the same volume. Redpanda recovers from its on-disk data and resumes processing.

## Why Not Custom Failover?

Some industrial software implements custom heartbeat-based failover. This approach predates container orchestration. Today, it duplicates what your container manager already does. It also introduces split-brain risk: the heartbeat network can partition while both nodes remain up, causing both to assume primary ownership.

UMH Core relies on your container manager as the single source of truth for scheduling.

## Storage Requirements

### Single-Node Deployments

If you have one node, storage choice does not affect failover. The node itself is the single point of failure. Redundancy comes from your underlying infrastructure. For example, a VM on vSAN survives host failure, and RAID storage survives disk failure.

### Multi-Node Deployments

For multi-node failover, you need storage accessible from multiple nodes.

The default k3s storage (`local-path`) binds the volume to one node. If that node dies, the container manager cannot reschedule the container elsewhere because the data is stuck on the failed node.

**Compatible storage options** (all support mounting volumes on any node):

* Enterprise SAN or vSAN
* Cloud block storage: EBS, Azure Disk, GCE PD
* Software-defined: Longhorn, Rook-Ceph

**Performance note:** Software-defined storage like Longhorn adds latency due to synchronous replication. For most UMH Core workloads, this is acceptable. For high-throughput scenarios, benchmark your specific workload before committing.

### Longhorn on k3s

If you do not have enterprise storage, Longhorn is a good choice for k3s clusters. See the [Longhorn Quick Installation Guide](https://longhorn.io/docs/latest/deploy/install/) and [K3s-specific configuration](https://longhorn.io/docs/latest/advanced-resources/os-distro-specific/csi-on-k3s/).

## If You Need Zero Data Loss

The standard approach accepts 30-60 second recovery on container failure. If your process cannot tolerate any data loss during a single-instance failure, see [Redundant Data Collection](/production/architecture-patterns/redundant-data-collection) for a topology that provides zero data loss using redundant PLCs and two umh-core instances.

## OPC UA Sessions After Failover

OPC UA sessions must be re-established after any failover. UMH Core reconnects to OPC UA servers automatically, but re-establishment takes a few seconds during which no data is collected from OPC UA sources. The OPC UA protocol requires this.


# Metrics

`http://<device-ip>:8080/metrics` (Prometheus format) exposes:

## Currently Available ✅

* Agent tick & FSM timings (each full reconcile loop < 100 ms by design)
* Per-DFC counters: processed, error, latency, active / idle flag
* Redpanda I/O and disk-utilisation stats


# Migration from Classic

This guide walks through migrating from UMH Classic to UMH Core. The key changes involve data contracts, configuration syntax, and architectural improvements.

## Migration Strategy

The recommended approach is **side-by-side deployment** to minimize risk:

1. **Deploy Core next to Classic** - Point a Bridge at the Classic UNS topics
2. **Cut over producers/consumers** gradually
3. **Shut down Classic** pods once data is verified

## Key Changes from Classic to Core

### Data Contract Migration

#### Core Change: `_historian` → `_raw`

**UMH Classic (Deprecated):**

```yaml
# Bridge configuration
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.data_contract = "_historian";

# Consumer patterns  
topics: ["umh.v1.+.+.+.+._historian.+"]
```

**UMH Core (Current):**

```yaml
# Bridge configuration - Start simple
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.data_contract = "_raw";

# Consumer patterns
input:
  uns:
    topics: ["umh.v1.+.+.+.+._raw.+"]        # For simple sensor data
    topics: ["umh.v1.+.+.+.+._temperature.+"]  # For structured temperature data
```

### Configuration Changes

#### Bridge Configuration

**UMH Classic:**

```yaml
# Old benthos-style configuration
input:
  opcua: 
    # ... configuration
processors:
  - tag_processor:
      defaults: |
        msg.meta.data_contract = "_historian";
output:
  kafka: {}
```

**UMH Core:**

```yaml
# New protocolConverter (Bridge) configuration
protocolConverter:
  - name: device-bridge
    desiredState: active
    protocolConverterServiceConfig:
      location:
        2: "production-line"
        3: "device-name"
      config:
        dataflowcomponent_read:
          benthos:
            input:
              opcua: 
                # ... configuration
            pipeline:
              processors:
                - tag_processor:
                    defaults: |
                      msg.meta.data_contract = "_raw";
            output:
              uns: {}  # Always use UNS output
```

#### Key Differences

| Aspect                   | UMH Classic                 | UMH Core                                                                          |
| ------------------------ | --------------------------- | --------------------------------------------------------------------------------- |
| **Data Contracts**       | `_historian` only           | `_raw` + explicit contracts                                                       |
| **Configuration**        | Direct Benthos config       | Embedded within Bridges/Flows                                                     |
| **Output**               | Direct Kafka                | UNS output (abstracts Kafka)                                                      |
| **Location**             | Manual topic construction   | Automatic hierarchical path construction (supports ISA-95, KKS, or custom naming) |
| **Data Modeling**        | Single payload format       | Structured models + contracts                                                     |
| **Database Integration** | Automatic with `_historian` | Manual with specific contracts                                                    |

## Important: No Default Database Integration

**⚠️ Breaking Change**: UMH Core has **no default data contract** that automatically writes to TimescaleDB.

**UMH Classic:**

```yaml
# _historian automatically integrated with TimescaleDB
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.data_contract = "_historian";  # Auto-saved to database
```

**UMH Core:**

```yaml
# _raw is for simple data only - NO automatic database integration
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.data_contract = "_raw";  # NOT saved to database automatically

# Database integration requires explicit configuration
dataFlow:
  - name: timescale-sink
    dataFlowComponentConfig:
      benthos:
        input:
          uns:
            topics: ["umh.v1.+.+.+.+._raw.+"]
        pipeline:
          processors:
            - mapping: |
                # Manual transformation for database
                root.timestamp = this.timestamp_ms.ts_unix_milli()
                root.value = this.value
                # Extract location from topic
                let parts = metadata("umh_topic").split(".")
                root.enterprise = parts.1
                root.site = parts.2
                root.tag_name = parts.7
        output:
          sql_insert:
            driver: "postgres"
            dsn: "postgres://user:pass@timescale:5432/manufacturing"
            table: "sensor_readings"
```

**Migration Impact:**

* `_raw` data contracts do **NOT** automatically save to TimescaleDB
* You must create explicit database sink flows for any data you want persisted
* This provides more control but requires manual configuration

## Exporting Benthos Configurations

Export your existing Benthos configs from Classic:

```bash
kubectl get configmap <benthos-config-name> -o yaml > classic-benthos-config.yaml
```

Extract the Benthos pipeline section and paste it into Core's `config.yaml → dataFlow:` section.

## What to Migrate

| Component              | Classic Location | Core Location            | Notes                                 |
| ---------------------- | ---------------- | ------------------------ | ------------------------------------- |
| **Benthos pipelines**  | Individual pods  | `config.yaml → dataFlow` | Export configs from ConfigMaps        |
| **Node-RED flows**     | Node-RED pod     | External container       | Run separately, connect via MQTT/HTTP |
| **Grafana dashboards** | Bundled Grafana  | External Grafana         | Point at TimescaleDB                  |
| **TimescaleDB**        | TimescaleDB pod  | External database        | Use Bridge to forward data            |

### 3. Verify Migration

1. **Check topic creation**: Ensure new `_raw` topics are being created
2. **Validate consumers**: Confirm all consumers are receiving data from `_raw` topics
3. **Monitor metrics**: Watch for any data loss or processing errors
4. **Test integrations**: Verify external systems still receive expected data

## Side-by-Side Deployment

Run Core alongside Classic during transition:

```yaml
# Core Bridge reading from Classic UNS
protocolConverter:
  - name: classic-bridge
    desiredState: active
    protocolConverterServiceConfig:
      config:
        dataflowcomponent_read:
          benthos:
            input:
              kafka:
                addresses: ["classic-kafka:9092"]
                topics: ["umh.v1.+.+.+.+._historian.+"]
            pipeline:
              processors:
                - mapping: |
                    # Convert _historian to _raw
                    root = this
                    meta.data_contract = "_raw"
            output:
              uns: {}
```

## Getting Help

* **Migration Issues**: See [Troubleshooting Guide](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/production/troubleshooting.md)
* **Configuration Questions**: Check [Configuration Reference](/reference/configuration-reference)
* **Community Support**: Visit [UMH Community Forum](https://community.umh.app)

## Next Steps

After completing the basic migration:

1. [**Data Modeling**](/usage/data-modeling) - Implement structured data contracts
2. [**Stream Processors**](/usage/data-flows/stream-processor) - Transform raw data to business models
3. [**Production Deployment**](/production) - Scale and secure your UMH Core deployment

### Tag Groups Migration

**UMH Classic (Tag Groups in Payload):**

```yaml
# Classic bridge configuration
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name = "head";  # Groups appear inside payload

# Classic payload with tag groups
{
  "timestamp_ms": 1670001234567,
  "pos": { 
    "x": 12.5,
    "y": 7.3,
    "z": 3.2
  },  
  "temperature": 50.0,
  "collision": false
}

# Classic database storage
# head$pos$x, head$pos$y, head$pos$z, head$temperature, head$collision
```

**UMH Core (Virtual Paths in Topics):**

```yaml
# Core data model with virtual paths
datamodels:
  - name: CNCHead
    version: v1
    structure:
      pos:
        x: { type: timeseries }
        y: { type: timeseries }
        z: { type: timeseries }
      temperature: { type: timeseries }
      collision: { type: timeseries }

# Core stream processor
streamprocessors:
  - name: cnc_head_sp
    contract: _cnc_head:v1
    sources:
      x_raw: "umh.v1.corp.plant.line.cnc._raw.head_pos_x"
      y_raw: "umh.v1.corp.plant.line.cnc._raw.head_pos_y"
      z_raw: "umh.v1.corp.plant.line.cnc._raw.head_pos_z"
      temp_raw: "umh.v1.corp.plant.line.cnc._raw.head_temperature"
      collision_raw: "umh.v1.corp.plant.line.cnc._raw.head_collision"
    mapping:
      pos.x: "x_raw"
      pos.y: "y_raw"  
      pos.z: "z_raw"
      temperature: "temp_raw"
      collision: "collision_raw"

# Result: Individual topics with virtual paths
# umh.v1.corp.plant.line.cnc._cnc_head.pos.x
# umh.v1.corp.plant.line.cnc._cnc_head.pos.y
# umh.v1.corp.plant.line.cnc._cnc_head.pos.z
# umh.v1.corp.plant.line.cnc._cnc_head.temperature
# umh.v1.corp.plant.line.cnc._cnc_head.collision
```

**Migration Benefits:**

* **Granular subscriptions**: Subscribe to just `pos.*` for positioning data
* **Independent scaling**: Different retention for different sensor groups
* **Type safety**: Each topic has explicit schema validation
* **Cross-site consistency**: Same virtual path structure across all CNCs

**Migration Steps:**

1. **Identify tag groups** in Classic payloads (nested JSON objects)
2. **Create data models** with folder structures matching groups
3. **Define stream processors** to map raw inputs to virtual paths
4. **Update consumers** to subscribe to specific virtual path patterns


# Security

This directory contains security documentation for the United Manufacturing Hub platform.

## Component Scope

The UMH platform consists of two main security domains:

### umh-core (Edge Gateway Container)

**Documentation**: `umh-core/deployment-security.md`

Security scope:

* Instance-level authentication (AUTH\_TOKEN)
* Container security (non-root execution, process isolation)
* Edge gateway security architecture
* Protocol converter and data flow security
* Network security for edge deployment
* Supply chain security (vulnerability scanning, dependencies)
* Industrial protocol handling (OPC UA, Modbus, S7, MQTT)

### ManagementConsole (Cloud Platform)

**Documentation**: `management-console/` (separate repository)

Security scope:

* User authentication and multi-factor authentication (MFA)
* Role-based access control (RBAC) for users
* User-level audit trails and action logging
* Cloud security and API protection
* Session management and user permissions
* Organization and team access controls

## Security Responsibility Boundary

**umh-core** handles edge security - authenticating the instance, securing the container, and protecting data flows at the factory edge.

**ManagementConsole** handles user security - authenticating users, controlling access, and securing the cloud platform.

Together they provide defense-in-depth: instance authentication (umh-core) + user authentication (ManagementConsole) + customer infrastructure security.


# ManagementConsole Security

This directory will contain security documentation for the cloud-based Management Console platform.

## Topics

* [**Authentication and Authorization**](/production/security/management-console/authentication-and-authorization) - User and instance authentication, role-based access control, and permission system

## Planned Topics

* **Cloud Infrastructure** - Cloud security architecture and compliance
* **API Security** - API authentication, rate limiting, and input validation

## Related Documentation

For Management Console documentation, see the [ManagementConsole repository](https://github.com/united-manufacturing-hub/ManagementConsole).

## Note

ManagementConsole security documentation is separate from umh-core security documentation:

* **umh-core** focuses on edge deployment security (containers, bridges, network)
* **ManagementConsole** focuses on cloud platform security (users, authentication, API)


# Authentication and Authorization

For authorization and authentication, we use a 2-layer solution:

* **Layer 1**: Identity and company access (session token) - who you are and which company's instances you can access
* **Layer 2**: Permissions (permission grants) - what you can do within that company

This separation provides clear boundaries in our distributed, multi-tenant system. Layer 1 ensures users can only communicate with instances within their company, while Layer 2 provides fine-grained authorization within that company.

***

## How It Works

### Layer 1: Identity and Company Access

**Applicable Standards**: NIST SP 800-63B (Digital Identity Guidelines), IEC 62443-4-2 CR 1.1 (Human User Identification and Authentication), OWASP Authentication Cheat Sheet

Layer 1 handles authentication - proving who you are and which company you belong to. Both users and UMH instances authenticate at this layer.

#### User Authentication

Users authenticate to verify their identity and their access rights to a company. Both authentication methods result in a session token that identifies you and your company:

**Auth0**: Modern authentication that simplifies login and enables integration with enterprise systems such as SAML:

* Each email can be assigned to multiple companies (configured in Auth0)
* Default login uses a one-time password sent via email
* Enterprise customers can customize the login experience:
  * Integration with company SAML or SSO
  * Additional multi-factor authentication methods
* The user is redirected to Auth0 to complete the authentication process

**Legacy**: Email and password authentication (deprecated):

* Each email can only be assigned to one company
* Email addresses are not validated for existence
* Multi-factor authentication is not available
* Password complexity requirements apply (minimum 12 characters, at least one uppercase letter, one digit, and one symbol)

Once logged in, the user remains authenticated until the session token expires.

#### Security Controls

Authentication security is provided by Auth0, which implements:

* **Brute force protection**: Automatic rate limiting and account lockout after failed attempts
* **Credential stuffing detection**: Anomaly detection for automated attacks
* **Session management**: Configurable idle and absolute timeout policies
* **Adaptive MFA**: Risk-based authentication challenges

For enterprise customers with custom Auth0 tenants, additional controls can be configured through the Auth0 dashboard. See [Auth0 Security Documentation](https://auth0.com/docs/secure) for details.

#### Instance Authentication

A UMH instance authenticates using an AUTH\_TOKEN that is generated during the initial setup process by the user who creates the instance.

The AUTH\_TOKEN is a cryptographically secure random token displayed once during instance creation. The user must copy this token and configure it in the UMH instance.

**Security Note**: The user who creates an instance has one-time visibility of the AUTH\_TOKEN. For secure AUTH\_TOKEN storage and rotation procedures at the instance level, see [umh-core deployment security](/production/security/deployment-security#auth_token-in-environment-variable).

The AUTH\_TOKEN serves two purposes:

* **Layer 1 authentication**: A double-hash of the AUTH\_TOKEN is stored in ManagementConsole for authentication. The instance sends the double-hash to prove its identity and receives a session token for subsequent communication.
* **Layer 2 credential encryption**: A single-hash of the AUTH\_TOKEN is used as the credential encryption key. This allows the instance to decrypt its own credentials, while ManagementConsole (which only has the double-hash) cannot.

### Layer 2: Permissions Within Your Company

**Applicable Standards**: NIST SP 800-53 AC-3 (Access Enforcement), NIST SP 800-53 AC-6 (Least Privilege), IEC 62443-4-2 CR 2.1 (Authorization Enforcement), OWASP Authorization Cheat Sheet

Layer 2 uses a hierarchical permission system where each user and instance has a defined role at specific locations within a company.

#### Account Owner

The first user who creates a company becomes the **Account Owner**. This role has special significance:

* The Account Owner cannot be changed or transferred
* Has Admin access to all locations within the company
* Can perform all administrative actions including user invitation and instance creation

> **Design Trade-off: Account Owner Cannot Be Removed**
>
> The Account Owner has permanent Admin access and cannot be transferred or demoted. This ensures a guaranteed recovery path if other admins lose access.
>
> **Best practice:** Treat Account Owner as a break-glass account - use only for initial setup and emergency access. Enable MFA via Auth0, store credentials in a secure password manager, and create separate admin accounts for daily work.
>
> **Enterprise setup:** Decide before registering which account becomes the Owner. Use a dedicated account your organization controls centrally - not the personal account of an employee who might leave, and never a shared login. Invite everyday users from there.

**Why this design**: The Account Owner provides a guaranteed recovery path if other admins lose access or permissions become misconfigured. Since it cannot be removed or demoted, it ensures at least one account always has full control.

#### Locations

Locations represent positions in an organizational tree structure. This flexible path format allows unlimited depth to match your actual organization. You can use ISA-95, KKS, or any organizational naming standard - level 0 (enterprise) is the only required level.

Location paths use the same dot-separated format as [topic paths](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/production/usage/unified-namespace/topic-convention.md):

* `ACME` (enterprise only)
* `ACME.Munich` (enterprise.site)
* `ACME.Munich.Assembly` (enterprise.site.area)
* `ACME.Munich.Assembly.Line1` (enterprise.site.area.line)
* `ACME.Munich.Assembly.Line1.Cell5` (enterprise.site.area.line.workcell)
* can be extended up to more location levels than 5

#### Roles

Three roles control what actions users and instances can perform at their assigned locations:

| Role   | Capabilities                                                                |
| ------ | --------------------------------------------------------------------------- |
| Admin  | Full control including the ability to invite other users at their locations |
| Editor | Can create and modify resources but cannot manage users                     |
| Viewer | Read-only access                                                            |

Users can have different roles at different locations. For example, a user can be an Admin at `ACME.Munich.Assembly.Line1` but only a Viewer at `ACME.Munich.Assembly.Line2`.

**Security context**: These roles control ManagementConsole actions. At the instance level, anyone with write access (Editor or Admin) can deploy bridge configurations that have full access to all instance resources - see [umh-core deployment security](/production/security/deployment-security) for details. The meaningful security boundary is between read-only (Viewer) and write access (Editor/Admin).

#### Permission Inheritance

Permissions are inherited downward: a user with access at `ACME.Munich` automatically has access to everything within Munich, such as `ACME.Munich.Assembly.Line1.Cell5`.

You can also define exceptions to override inherited permissions. For example, a user could have Viewer access at `ACME.Munich.Assembly` but Admin access specifically at `ACME.Munich.Assembly.Line1.Cell5`.

#### User and Instance Management

Only admins and the Account Owner can invite new users and add instances. When inviting users, admins can only grant permissions for locations where they themselves have admin access. This prevents privilege escalation and ensures that permissions flow naturally through the organization. Currently, every admin can create instances in every location. If they would create an instance that is outside of their location permission scope, they would not be able to modify it via the Management Console.

**User Invitation Process**

When an admin invites a new user:

1. **Admin specifies**: Email address, role, and location permissions
2. **System generates**: Invite link + separate invite key (shown only to the admin)
3. **Auth0 sends**: Automatic invitation email to the user
4. **Admin shares**: The invite key through a separate secure channel
5. **User accepts**: Clicks link, authenticates with Auth0, enters invite key

The invite key can only be used once and enables secure key exchange without the backend ever seeing the user's private credentials.

**Why two pieces?** The invite link proves email ownership (via Auth0). The invite key, shared separately, ensures the inviting admin intended this specific person to receive access. This prevents email forwarding attacks.

**Inviting UMH Personnel**

Some support requests go beyond what a quick call can answer - for example when the UMH team sets up bridges or data models for you. For this kind of longer setup work, it is easier if the UMH team member doing the work has access to your company in the Management Console.

Invite UMH team members through the normal invitation flow, using their `@umh.app` email address. These invitations are automatically issued against UMH's Google Workspace connection (i.e. an identity provider, set up in Auth0) instead of your default connection, because UMH staff don't have accounts in your identity provider.

Control stays on both sides. You decide who to invite and can remove them at any time. UMH manages the `@umh.app` identities centrally through Google Workspace, so access is bound to employment: when an employee leaves UMH, disabling their Google Workspace account also cuts their access to your company.

### Access Revocation

**Applicable Standards**: NIST SP 800-53 AC-2 (Account Management), OWASP Session Management Cheat Sheet

Access revocation happens at Layer 1 (session invalidation), not through permission grant expiration:

* **User removal**: When a user is removed from ManagementConsole, their session token is invalidated immediately - they can no longer authenticate
* **Permission updates**: Admins can update user permissions at any time. ManagementConsole validates the user's current permissions from the database on each request, so changes take effect quickly (within the cache window of up to 10 minutes)
* **Instance removal**: Removing an instance from ManagementConsole denies all further communication as authentication in layer 1 fails

> **Design Trade-off: Permission Updates Require Active Session**
>
> When an admin updates a user's permissions, the new permission certificate is automatically applied in the background by the user's frontend worker. However, this only happens while the user is logged in. If the user is offline, they retain their original privileges until their next login.
>
> **Best practice:** To immediately revoke access or demote a user, **remove them entirely** - this invalidates their session without requiring them to be online. You can then re-invite them with the correct permissions.

**What happens when users leave**: When you remove a user, they immediately lose access (session invalidated). Resources they created and users they invited remain - permission grants represent organizational decisions, not personal relationships.

### Session Management

**Applicable Standards**: OWASP Session Management Cheat Sheet, NIST SP 800-63B (Session Binding)

ManagementConsole manages user sessions independently from Auth0 using JWT cookies signed with `JWT_SECRET_KEY`.

**Session Lifecycle**:

* **Creation**: Session token issued after successful Auth0 authentication
* **Storage**: HTTPOnly cookie with Secure and SameSite=Strict attributes
* **Token Validity**: Each JWT token is valid for **14 days**
* **Sliding Window Refresh**: When a token is within **7 days of expiring** and the user is active (e.g., page load), a fresh 14-day token is issued automatically
* **Absolute Session Limit**: Regardless of activity, users must re-authenticate after **30 days** from their original login
* **Termination**: Explicit logout, absolute timeout, or user removal from company

**Session Policies**:

* **Concurrent sessions**: Multiple sessions from different devices are permitted but may lead to unreliable connections. For devices requiring simultaneous access, create separate user accounts
* **Token refresh**: Session extends automatically on API activity when within the last 7 days of the 14-day token validity
* **Cross-device**: Each device maintains its own independent session

> **Current Limitation: No Idle Timeout**
>
> There is no automatic logout after periods of inactivity. A session remains valid as long as:
>
> 1. The token hasn't expired (14 days without activity), AND
> 2. The 30-day absolute limit hasn't been reached
>
> **Best practice:** Log out when leaving your workstation. For environments requiring shorter session lifetimes, consider enterprise SSO with your own timeout policies.

**Forced Logout**: Currently, individual session termination requires user removal and re-invitation. Bulk session revocation for specific users is planned for a future release.

***

## Threat Model

ManagementConsole's authentication system is designed to protect against specific threat actors while accepting certain risks as design trade-offs.

### Trust Boundaries

The authentication system operates across three trust boundaries:

1. **User Browser ↔ ManagementConsole**: TLS-encrypted connections with session tokens stored as HTTPOnly cookies (stateless authentication)
2. **ManagementConsole ↔ Auth0**: OAuth 2.0 flow for user authentication, with Auth0 handling credential storage and MFA
3. **ManagementConsole ↔ umh-core Instances**: AUTH\_TOKEN-based authentication for instance identity, with permission certificates for authorization

### Threat Actors and Protection

| Threat Actor                 | Capabilities                                            | Protection Level                                                                                            |
| ---------------------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| **External Attacker**        | Credential stuffing, phishing, brute force              | **Protected** - Auth0 provides rate limiting, anomaly detection, MFA                                        |
| **Compromised User**         | Access to company data at their permission level        | **Protected** - Session timeout, per-request permission validation, removal invalidates session immediately |
| **Malicious Insider**        | Permission escalation attempts, invite key interception | **Partially Protected** - Backend enforces admin-location rules; two-channel invite delivery                |
| **Account Owner Compromise** | Complete company takeover                               | **Design trade-off** - See Account Owner section above                                                      |

### Attack Scenarios and Mitigations

**Credential Attacks**: Auth0 protects against credential stuffing and brute force through automatic rate limiting and anomaly detection. For enterprise customers, MFA can be enforced for all users.

**Session Hijacking**: Session tokens are stored in HTTPOnly cookies (not accessible via JavaScript), preventing XSS-based token theft. All connections use TLS encryption.

**Cross-Company Access**: Each company is isolated through Auth0 organizations. Users authenticate to specific companies and cannot access data from other organizations.

**Privilege Escalation**: Admins can only grant permissions for locations where they have admin access. The backend validates all permission operations, preventing UI-based manipulation.

**Phishing Attacks**: The two-channel invite system (email link + separate invite key) prevents email-only attacks. Users must have both pieces to join a company.

### Out of Scope

ManagementConsole authentication does NOT protect against:

* **Compromised umh-core instances**: Instance-level security is covered in [umh-core deployment security](/production/security/deployment-security#threat-model-simplified)
* **Physical access to user devices**: Users are responsible for device security
* **Auth0 infrastructure compromise**: Auth0 is responsible for their platform security (see Shared Responsibility Model)

***

## Shared Responsibility Model

Security is a shared responsibility between UMH and our customers. This section clarifies who is responsible for what. For instance-level security responsibilities, see [umh-core deployment security - Shared Responsibility Model](/production/security/deployment-security#shared-responsibility-model).

### We (UMH) Are Responsible For

| Area                          | Our Responsibility                                                                       |
| ----------------------------- | ---------------------------------------------------------------------------------------- |
| Authentication Infrastructure | Auth0 integration, session token generation, secure credential hashing                   |
| Authorization Framework       | RBAC system, permission inheritance, location-based access control                       |
| Secure Defaults               | Password complexity requirements, invite key separation, double-hash storage             |
| Platform Security             | ManagementConsole application security, API security, session management                 |
| Audit Logging                 | Recording authentication events and permission changes (see note below)                  |
| UMH Personnel Access          | Identity, authentication, and offboarding of `@umh.app` accounts invited to your company |

> **Known Limitation: Audit Logging**
>
> Comprehensive audit logging with user-accessible logs, configurable retention, and SIEM integration is planned but not yet fully implemented. Currently, authentication events are logged internally but not exposed through a user interface. Enterprise customers requiring detailed audit trails should contact UMH support to discuss available options.

### You (Customer) Are Responsible For

| Area                    | Your Responsibility                                                            |
| ----------------------- | ------------------------------------------------------------------------------ |
| Account Owner Security  | Protecting the Account Owner credentials (recovery account)                    |
| AUTH\_TOKEN Management  | Secure storage and transmission of instance AUTH\_TOKENs                       |
| Invite Key Distribution | Sharing invite keys through secure out-of-band channels                        |
| User Lifecycle          | Promptly removing users who leave your organization                            |
| Access Reviews          | Periodic review of user permissions and access levels                          |
| Enterprise SSO          | Configuration and security of your SAML/SSO identity provider                  |
| UMH Personnel           | Deciding which UMH personnel to invite and removing them when no longer needed |

### Shared Responsibilities

| Area              | Details                                                                             |
| ----------------- | ----------------------------------------------------------------------------------- |
| Permission Design | UMH provides the RBAC framework; you define appropriate roles per location          |
| Incident Response | UMH monitors platform; you monitor for compromised credentials                      |
| Compliance        | UMH provides security controls; you ensure usage meets your compliance requirements |

***

## Compliance Alignment

This section maps ManagementConsole security controls to industry standards.

### Authentication Standards

| Standard              | Requirement                                            | Implementation                             |
| --------------------- | ------------------------------------------------------ | ------------------------------------------ |
| OWASP Authentication  | MFA, brute force protection, secure session management | Auth0 provides all authentication controls |
| NIST SP 800-63B AAL2  | Multi-factor authentication, session timeout ≤24hr     | Auth0 MFA + configurable session lifetime  |
| IEC 62443 SR 1.1      | Human user identification and authentication           | Auth0 unique user IDs + authentication     |
| IEC 62443 SR 1.1 RE 2 | MFA for untrusted networks (SL2+)                      | Auth0 MFA enforced for all external access |

### Authorization Standards

| Standard            | Requirement                             | Implementation                                     |
| ------------------- | --------------------------------------- | -------------------------------------------------- |
| OWASP Authorization | Least privilege, deny by default, RBAC  | Platform implements role-based access control      |
| NIST SP 800-53 AC-3 | Access enforcement at all access points | ManagementConsole enforces via database lookups    |
| NIST SP 800-53 AC-6 | Least privilege                         | Viewer/Editor/Admin roles with minimal permissions |
| IEC 62443 SR 2.1    | Authorization enforcement for all users | Platform validates permissions on each request     |

### Session Management

| Standard        | Requirement                            | Implementation                                                               |
| --------------- | -------------------------------------- | ---------------------------------------------------------------------------- |
| OWASP Session   | Token rotation, secure cookies, logout | Backend issues HTTPOnly JWT cookies (14-day sliding window, 30-day absolute) |
| NIST SP 800-63B | Idle timeout ≤1hr (AAL2)               | Not yet implemented (30-day absolute timeout only)                           |

**Note**: Auth0 handles authentication only. ManagementConsole backend manages sessions independently using its own JWT cookies signed with `JWT_SECRET_KEY`.

### Accepted Limitations

| Standard             | Requirement                     | Current Status                                 |
| -------------------- | ------------------------------- | ---------------------------------------------- |
| NIST SP 800-63B AAL2 | Idle timeout ≤1hr               | Not implemented (30-day absolute timeout only) |
| NIST AC-2(3)         | Disable dormant accounts        | Not implemented (manual process)               |
| IEC 62443 SL3        | MFA for all interfaces          | MFA for external only (SL2 compliant)          |
| OWASP                | Immediate permission revocation | Permission updates require user acceptance     |

### Target Security Level

ManagementConsole targets **IEC 62443 Security Level 2 (SL2)**, appropriate for:

* Protection against intentional violation using simple means
* Cybercrime-level threat actors with generic skills
* Standard manufacturing and industrial operations

For critical infrastructure requiring SL3+, contact UMH for enterprise security options.

***

### References

* [OWASP Authentication Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html)
* [OWASP Authorization Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html)
* [NIST SP 800-63B Digital Identity Guidelines](https://pages.nist.gov/800-63-4/sp800-63b.html)
* [NIST SP 800-53 Access Control](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)
* [ISA/IEC 62443 Series of Standards](https://www.isa.org/standards-and-publications/isa-standards/isa-iec-62443-series-of-standards)

#### Deployment Considerations

The permission infrastructure exists and controls what users see in ManagementConsole, but UMH Core does not yet validate individual user permissions when executing commands. Similarly, users accept messages from all instances within their company without per-instance verification. This means that within a company, all authenticated users can execute all actions on UMH Core instances, regardless of their assigned role or location permissions.

The permission system provides fine-grained authorization that is ready for enforcement once validation is implemented.

#### Auth0 Organization Linking

Each company can be linked to an Auth0 organization, enabling:

* Single sign-on through your corporate identity provider
* Centralized user management in Auth0
* Multi-company access with one email (each company links to its own Auth0 org)

The link is established during company setup and validated during every user onboarding - users can only join companies that match their Auth0 organization.

**Connections**:

An Auth0 organization accepts logins through one or more *connections*. A connection is an identity source, such as your corporate SAML/OIDC provider, a social login like Google, or an email one-time password.

If you have an enterprise plan, you pick a **default connection** for your team. All members sign in through it, and invitations go through it too. Point the default connection at your own identity provider. Your usual SSO, MFA, and offboarding rules then control who gets in.

UMH staff are the one exception. You don't set up a user for them in your identity provider. Their invitations use UMH's internal SSO instead (see [Inviting UMH Personnel](#inviting-umh-personnel)).


# umh-core Security

## Shared Responsibility Model

### We are responsible for:

* **Software supply chain** (container images, SBOM, vulnerability scanning via Aikido/FOSSA)
* **Secure defaults** (non-root execution, TLS enabled by default, no default passwords)
* **Clear documentation** of protocol limitations and security considerations
* **Regular security updates** via our Docker registry and documented release process

### You are responsible for:

* **Infrastructure and runtime** (Docker/Kubernetes configuration, host OS security, network architecture)
* **Secrets lifecycle** (AUTH\_TOKEN storage, rotation, access controls)
* **Volume permissions** (ensuring `/data` is writable by UID 1000)
* **Monitoring and incident response** (log aggregation, security monitoring, forensics)
* **Deployment security** (capabilities, AppArmor/SELinux, resource limits, network policies)
* **Network segmentation and zone placement** (deploy at Purdue Level 3 per IEC 62443-3-3 zone architecture, firewall rules for OT/IT boundaries)
* **PLC polling rate configuration** (configure polling intervals per device specifications to prevent overload)
* **Physical security** (secure deployment locations and restrict physical access per IEC 62443-3-3 SR 5.1)
* **OT safety systems** (umh-core must not be integrated into safety-instrumented systems; see IEC 61508/61511)
* **Backup and disaster recovery** (configuration backups, persistent volume snapshots, tested restore procedures per business continuity requirements)
* **High availability**: use storage accessible from multiple nodes. See [High Availability](/production/high-availability).
* **Security event monitoring** (SIEM integration if required, intrusion detection systems)
* **Corporate CA certificate management** (adding certificates for TLS inspection scenarios)
* **Reading this documentation** - we provide secure software, you must deploy it securely

**This aligns with cloud vendor models** - we secure the software, you secure the deployment environment.

**For detailed OWASP/CIS compliance guidance**, see:

* [OWASP Docker Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html)
* [CIS Docker Benchmark](https://www.cisecurity.org/benchmark/docker)
* [Kubernetes Security Best Practices](https://kubernetes.io/docs/concepts/security/)

***

## Security Capabilities

This section documents umh-core's security features across container security, access control, network architecture, cryptography, supply chain integrity, and industrial protocol handling. Each capability is mapped to applicable industry standards (NIST, IEC 62443, OWASP) with implementation details and known limitations.

## Container Security

**Applicable Standards**: OWASP Docker Security Cheat Sheet requirement 2 (non-root user), NIST SP 800-190 (least privilege principle for containers), IEC 62443-4-2 CR 2.1 (authorization enforcement and least privilege at Security Level 2)

**Implementation**: All umh-core processes run as non-root user umhuser with UID 1000, preventing privilege escalation attacks. Each component operates in a separate process namespace isolated from the host, with limited filesystem access restricted to explicitly mounted paths only. The minimal Alpine Linux base image reduces the attack surface by excluding unnecessary system utilities and libraries.

**Deployment Considerations**: All bridges and [data flows](/usage/data-flows) run as the same Linux user within the container, with no per-component user isolation. This design prioritizes non-root security over internal process isolation, as user switching requires root capabilities unavailable in non-root containers.

**Note**: Mounted volumes must be writable by UID 1000. See [Filesystem Access](#filesystem-access).

⚠️ **Safety System Warning**: umh-core is NOT SIL-rated and must not be used in safety-instrumented systems. Refer to IEC 61508/61511 for safety requirements.

***

## Logging and Audit Trail

**Applicable Standards**: NIST SP 800-53 AU-2 and AU-3 (comprehensive audit logging with timestamps, event types, and outcomes), NIST SP 800-92 (log generation, storage, and protection), IEC 62443-4-2 CR 2.8 (auditable events for security-relevant actions)

**Implementation**: All services write structured logs to /data/logs/ with S6 supervision, using TAI64N timestamps for precise event ordering. FSM state transitions are tracked for all components, with rolling log rotation to manage storage. Logs capture configuration changes, component lifecycle events, and connection status for all industrial protocols and data flows.

**Deployment Considerations**: Current logging tracks system events and component states but does not capture individual user actions performed through the ManagementConsole interface. User-level audit trails for configuration changes are planned for future releases.

***

## Access Control and Authentication

**Applicable Standards**: NIST SP 800-53 AC-6 (least privilege access control), IEC 62443-4-2 CR 1.2 (software process and device identification and authentication), NIST SP 800-171 IA-2 (identification and authentication of organizational users)

**Implementation**: Each umh-core instance authenticates to the ManagementConsole using a unique AUTH\_TOKEN shared secret combined with a per-instance InstanceUUID. The AUTH\_TOKEN is user-configured with no default values, and ManagementConsole provides the user authentication layer for operator access. TLS encryption protects authentication tokens during transmission to management.umh.app.

**Deployment Considerations**: The current AUTH\_TOKEN is instance-level and shared across all components within the container. Future releases will implement per-message authentication to enable fine-grained access control and authorization.

***

## Network Security

**Applicable Standards**: NIST SP 800-82 (defense in depth for OT/ICS environments and network segmentation), IEC 62443-3-3 SR 5.1 (zone and conduit architecture for edge gateway deployment), IEC 62443-4-2 CR 5.1 (network segmentation support)

**Implementation**: umh-core follows an edge-only architecture designed for deployment at the OT/IT boundary (Purdue Level 3 (DMZ between OT and IT networks)), with outbound HTTPS connections only to management.umh.app for configuration synchronization. Typical deployment: umh-core in DMZ, outbound connections to cloud, inbound connections from OT networks only. No direct internet exposure. By default, no services listen for inbound internet connections. However, customers can configure data flows with HTTP server inputs that expose ports within the container. Network isolation is enforced through separate network namespaces unless explicitly configured otherwise.

Deploy umh-core in a DMZ with firewalls on both OT and IT boundaries. See IEC 62443-3-3 for zone architecture.

**Customer-Configured HTTP Servers**: Customer-configured HTTP servers require coordination between application teams (who configure bridges via ManagementConsole) and security/infrastructure teams (who must explicitly expose container ports). This two-team requirement ensures managed attack surface. umh-core does not provide built-in authentication for HTTP inputs. Deploy behind your standard reverse proxy (nginx, HAProxy) for authentication and TLS.

**Deployment Considerations**: The edge-only design requires outbound internet access to function and is not suitable for air-gapped or fully disconnected environments. Network segmentation and firewall rules must be configured properly by the customer to achieve defense in depth.

***

## Cryptography and TLS

**Applicable Standards**: NIST SP 800-52 (TLS 1.2+ configuration with modern cipher suites), NIST SP 800-53 SC-8 (transmission confidentiality and integrity), IEC 62443-4-2 CR 4.3 (use of cryptography conforming to applicable standards and regulations)

**Implementation**: All connections to management.umh.app use TLS 1.2 or higher with modern cipher suites including AES-GCM and ChaCha20-Poly1305. Certificate validation is enabled by default using Go's standard library crypto packages, and all cryptographic operations follow current industry standards for secure communications. This is the secure default configuration.

**Deployment Considerations**: An ALLOW\_INSECURE\_TLS configuration option is available for corporate environments with TLS inspection, which disables certificate validation when enabled. This option should only be used behind trusted corporate firewalls where inspection is performed, as it makes the system vulnerable to man-in-the-middle attacks. When ALLOW\_INSECURE\_TLS is enabled, the minimum TLS version is reduced to TLS 1.0 to maximize compatibility with corporate TLS inspection proxies (note: TLS 1.0 is only available with this flag and does not meet the SSLabs Grade B standard). This setting should only be used behind trusted corporate firewalls.

***

## Supply Chain Security

**Applicable Standards**: OWASP Docker Security Cheat Sheet requirement 0 (regular image updates and vulnerability scanning), NIST SP 800-161 (SBOM generation, vulnerability management, and supply chain risk controls), IEC 62443-4-1 SR-5 (product defect management and security vulnerability tracking)

**Implementation**: All container images undergo automated vulnerability scanning with Aikido. SBOM and OSS License Compliance is tracked through FOSSA. The compliance dashboard at trust.umh.app tracks standards alignment (such as ISO27001 or NIST) through Vanta.

**Deployment Considerations**: Supply chain security depends on timely updates from upstream dependencies and proper image verification during deployment.

***

## Industrial Protocol Security

**Applicable Standards**: OWASP OT Top 10 number 9 (legacy protocol security limitations and risks), IEC 62443-3-3 SR 5.1 (network segmentation for operational technology protocols), IEC 62443-4-2 CR 3.1 (communication integrity requirements)

**Implementation**: umh-core supports industrial protocols including OPC UA (with certificate-based security), MQTT (with TLS support), Modbus TCP, and S7 communication. Connections are managed through protocol-specific bridges with configurable security parameters where the underlying protocol provides security features.

**Deployment Considerations**: Modbus TCP and S7 protocols lack native encryption capabilities due to inherent protocol design limitations. Compensating controls include network segmentation to isolate OT zones from IT and internet networks, physical security requirements for deployment locations, and firewall rules to restrict protocol access by IP address. These compensating controls must be implemented by the customer as part of their overall IEC 62443-3-3 zone architecture.

***

## PLC Integration Limits

**Warning**: Aggressive polling can overwhelm PLCs and cause CPU overload or crashes. Each PLC model has different limits for concurrent connections, polling rates, and tag counts. Consult your device manual for specifications and test polling rates in non-production before deployment. S7 protocol is particularly sensitive to connection overload.

***

## Threat Model (Simplified)

umh-core primarily protects against unintentional compromise of external industrial systems due to vulnerabilities in our software. The non-root execution model prevents privilege escalation, the minimal network attack surface reduces exposure, and TLS is enabled by default for all external communications. Supply chain risks are mitigated through vulnerability scanning and dependency tracking. The pull-based deployment model (see Deployment Model below) prevents misconfiguration that could expose industrial protocols to the internet.

umh-core does not protect against malicious operators or accidental misconfiguration. An operator with access to the ManagementConsole UI or direct filesystem access to config.yaml can deploy bridge configurations that connect to external industrial systems, exfiltrate the AUTH\_TOKEN via outbound network requests, or read sensitive data from mounted volumes. Accidental misconfiguration poses similar risks - operators copying configuration snippets from the internet (blog posts, forums, examples) may inadvertently expose ports, enable insecure settings, or introduce vulnerable patterns without understanding the security implications. Similarly, the system cannot protect against compromise of the underlying container runtime, host operating system, or Kubernetes control plane.

This model aligns with industry-standard edge gateway security - we secure our software, you secure your infrastructure.

***

## Deployment Model: Edge-Only Architecture

umh-core is designed for edge-only deployment with a specific network architecture. The system requires outbound HTTPS connections to management.umh.app for configuration synchronization and supports outbound connections to data sources including MQTT brokers, OPC UA servers, Modbus devices, and APIs. No services are designed for inbound internet connections.

The typical deployment location is on the factory floor, behind corporate firewall, on-premises at customer sites. This architecture reduces the attack surface by eliminating services that listen for inbound internet connections. The ManagementConsole queues configuration changes that umh-core instances pull on their own schedule for execution. This aligns with network segmentation best practices where umh-core sits between OT networks and IT infrastructure at the boundary layer.

The system is not designed for air-gapped environments. umh-core requires outbound internet access to management.umh.app to function and cannot operate in fully disconnected deployments.

***

## What umh-core Accesses and Why

### Filesystem Access

**Required**: `/data` directory (persistent storage)

* Configuration files (config.yaml with AUTH\_TOKEN)
* Logs (rolling logs for all services)
* Redpanda data (message broker storage)

**Optional**: Customer-defined mounts for file-based inputs

* CSV/JSON/XML data files from network shares
* Log files from other systems
* Production reports from local disks

**Why**: Bridges need to read data files and persist configuration across container restarts.

**Access pattern**: Access depends on mount configuration (e.g., `:ro` for read-only, `:rw` for read-write).

#### Volume Permission Requirements

The `/data` directory must be writable by UID 1000 (umhuser). Set ownership before starting:

```bash
sudo chown -R 1000:1000 /path/to/data
```

***

### Network Access

**Required outbound**: HTTPS to `management.umh.app` for configuration sync and status reporting.

See [Network Configuration](/production/security/deployment-security/network-configuration) for details on proxy settings and TLS inspection.

**Data source connections**: Customer-defined connections to industrial devices and data sources.

**Inbound**: No services designed for internet exposure (edge-only deployment).

***

### Process Model

All components run as single non-root user with UID 1000 named umhuser. This includes the main umh-core agent, all bridges implemented as benthos-umh instances, all data flows, the Redpanda broker, and internal services.

See [Container Security](#container-security) above for details on non-root execution model, process isolation, and filesystem access restrictions.

***

## Deployment Considerations

### AUTH\_TOKEN in Environment Variable

**Category**: Deployment Consideration (cannot fix in single-container architecture)

**Issue**: AUTH\_TOKEN shared secret flow:

* **Input**: Environment variable (`AUTH_TOKEN=xxx`) during first container start
* **Persistent storage**: Written to `/data/config.yaml` automatically
* **Subsequent starts**: Read from config.yaml (environment variable no longer required)

**Why this design**: Ensures configuration persists across container restarts without requiring environment variables every time. Once set via environment variable or ManagementConsole, AUTH\_TOKEN is stored in `/data/config.yaml` on the persistent volume.

**Security implication**: Both storage locations are readable by all processes running as umhuser with UID 1000. This includes all bridges handling protocol converters, data flows, and stream processors, any process started within the container, and any code executed via bridge configurations.

**Risk**: Malicious bridge configuration could exfiltrate AUTH\_TOKEN via outbound network requests.

**What you should do**:

1. **Understand the security model**: This follows standard container secret management patterns used by Docker and Kubernetes. All processes within the container share the secret, which is industry-standard for single-container architectures. For enhanced secret isolation, enterprise deployments can integrate with HashiCorp Vault or similar secret management systems.
2. **Monitor** network connections for unexpected outbound traffic (exfiltration attempts)
3. **Rotate if compromised**:
   * Create new instance in ManagementConsole
   * Copy new AUTH\_TOKEN to deployment configuration
   * Update container environment variable or config.yaml
   * Remove old instance from ManagementConsole

***

### No User/Process Isolation Between Bridges

**Category**: Accepted Risk (design trade-off)

**What this means**: All bridges (protocol converters, data flows, stream processors) run as the same Linux user (UID 1000, umhuser). There is no user-level or process-level isolation between different bridges within the container.

### No Per-User Access Control Within Instance

**Category**: Deployment Consideration (planned for future releases)

**Issue**: AUTH\_TOKEN authorizes the umh-core instance to communicate with ManagementConsole. Once authorized, any user within that organization can perform all actions on this instance through the ManagementConsole UI. Per-user and per-action access restrictions at the instance level are not currently implemented but are planned for future releases.

### Logs Accessible to All Processes

**Category**: Deployment Consideration (non-root container design)

**Issue**: All processes running as umhuser (UID 1000) can read and potentially modify log files in /data/logs/. This is a consequence of the non-root container design. Customers requiring tamper-proof logs should implement log forwarding to external SIEM systems with append-only storage.

**What this does not mean**: Bridges do not interfere with each other's data processing, as Redpanda isolates message flows by topic. Bridges are not resource-limited together; each bridge can have separate CPU and memory limits configured via s6-softlimit. Data is not shared between bridges, as each benthos instance maintains separate configuration and state.

**Technical constraint**: Per-bridge user isolation is not possible in non-root containers. Process-level user switching requires CAP\_SETUID and CAP\_SETGID capabilities, which are only available to root processes.

**Design decision**: Non-root container security prioritized over per-bridge user isolation.

**Security implications**:

Because all bridges run as the same user, they share access within the container. All bridges can read the /data/config.yaml file containing AUTH\_TOKEN, access the same mounted directories, view each other's environment variables, and read each other's configuration files.

The container boundary remains enforced despite shared user access. Bridges cannot access the host filesystem except through explicitly mounted paths, cannot see host processes, and network isolation applies unless explicitly disabled with host networking mode.

Non-root execution provides security benefits that justify this trade-off. The container cannot escalate to root privileges even if a bridge is compromised. This follows the standard Docker security model with defense in depth. The design is compatible with restricted Kubernetes environments that do not permit privileged containers or special permissions.

***

### TLS Certificate Validation Can Be Disabled

**Category**: Accepted Risk (corporate firewall compatibility)

**Issue**: `ALLOW_INSECURE_TLS=true` option disables certificate validation for:

* **Connection to management.umh.app** (configuration sync and status reporting)
* **Bridge connections** to data sources (HTTPS APIs, MQTTS brokers, etc.)

**Why this option exists**: Corporate firewalls often perform TLS inspection (MITM), and adding corporate CA certificates is complex.

**Risk**: MITM attacks possible if misused:

* **Management connection**: Attacker could intercept AUTH\_TOKEN during transmission
* **Bridge connections**: Attacker could intercept or modify industrial data in transit

**Usage guidance**: Only use behind trusted corporate firewall where TLS inspection is performed. See [Network Configuration](/production/security/deployment-security/network-configuration) for details on adding corporate CA certificates (preferred) vs using `ALLOW_INSECURE_TLS=true`.

***

## Security Best Practices

### What to Mount

**Required**: `/data` for persistent storage (configs, logs, certificates)

**Common additional mounts**:

* Data files: Network shares with CSV/JSON/XML production data
* Log files: From PLCs or other systems
* Use read-only mounts when bridges only need to read

### What NOT to Mount

* Entire host filesystem (`/`)
* Docker socket (`/var/run/docker.sock`)
* Host `/etc` or `/var` directories

***

## Enterprise Security Features

The following security capabilities and documentation require an enterprise license:

* Security testing reports and penetration test results
* Software Bill of Materials (SBOM) access and vulnerability disclosure timelines
* Incident response playbooks and operational runbooks
* End-of-life policy, extended support, and long-term security update commitments
* Service Level Agreements (SLAs) for security patch response times
* Compliance attestations and audit support documentation

Enterprise license provides compliance documentation and SLA-backed support, not additional security features.

For enterprise licensing information, contact the United Manufacturing Hub sales team.


# Network Configuration

Network requirements and configuration for umh-core edge deployments.

## Outbound Connections

umh-core requires outbound connectivity to the Management Console:

| Destination          | Protocol | Port | Purpose                                                |
| -------------------- | -------- | ---- | ------------------------------------------------------ |
| `management.umh.app` | HTTPS    | 443  | Configuration sync, status reporting, action retrieval |

No inbound connections are required from the internet.

## Corporate Firewall Configuration

### TLS Inspection (MITM)

If your corporate firewall performs TLS inspection, you may see certificate errors. Solutions:

**Option 1: Add Corporate CA Certificate (Recommended)**

Add your corporate CA certificate to the container's trusted certificates.

**Option 2: Disable Certificate Validation (Last Resort)**

```yaml
# config.yaml
agent:
  communicator:
    allowInsecureTLS: true  # WARNING: Only use if corporate firewall blocks secure connections
```

Or via environment variable:

```bash
docker run -e ALLOW_INSECURE_TLS=true management.umh.app/oci/united-manufacturing-hub/umh-core:<VERSION>
```

**Security Warning:** The `allowInsecureTLS` option disables certificate validation. Only use this if:

* You're behind a corporate firewall that you trust
* You cannot add your corporate CA certificate
* You understand the security implications

## Proxy Configuration

If your network requires a proxy:

```bash
docker run \
  -e HTTP_PROXY=http://proxy.company.com:8080 \
  -e HTTPS_PROXY=https://proxy.company.com:8080 \
  -e NO_PROXY=localhost,127.0.0.1,.local \
  management.umh.app/oci/united-manufacturing-hub/umh-core:<VERSION>
```

Supported environment variables: `HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY` (and their lowercase variants).

### Authenticated Proxies

Include credentials in the proxy URL:

```bash
-e HTTP_PROXY=http://username:password@proxy.company.com:8080
```

Supported proxy types: HTTP and HTTPS.

## Common Configuration

In most corporate environments, proxy usage and TLS inspection go together. If you need to configure a proxy, you'll likely also need to add your corporate CA certificate to handle TLS inspection.


# Reference

This section provides comprehensive reference documentation for the United Manufacturing Hub (UMH) Core components. Here you'll find detailed technical specifications, configuration options, and structural documentation to help you implement, configure, and troubleshoot your UMH deployment.

The reference materials include:

* **Configuration Reference**: Complete documentation of the central configuration file (`/data/config.yaml`), including all available fields, data types, and examples
* **Container Layout**: Detailed overview of the container's file system structure, data organization, and persistent volume layout
* **Template Variables**: Comprehensive glossary of all available template variables for protocol converter configurations, including connection, location, and system variables
* **State Machines**: Reference documentation for state machine implementations and workflows
* **HTTP API**: GraphQL API reference for Topic Browser queries and programmatic access

Navigate through these pages to find specific configuration parameters, understand system architecture, or look up technical details for your UMH implementation.


# Configuration Reference

This is the reference for the central config `/data/config.yaml`

> **File location:** The container mounts `/data` as a writable volume.\
> **Hot-reload:** The Agent polls the file every tick; valid changes are applied automatically.\
> **UI Terminology:** In the Management Console UI, `protocolConverter:` is shown as "Bridges" and `dataFlow:` is shown as "Stand-alone Flows"

## YAML vs UI: Choose Your Workflow

UMH Core supports both **direct YAML editing** and **Management Console UI** for configuration. Each approach has distinct advantages:

### YAML Configuration Benefits

* **Version Control**: Every change is tracked with Git — see who changed what, when, and why
* **GitOps Integration**: Automate deployments, rollbacks, and multi-environment promotion
* **AI/LLM Integration**: Tools like Cursor and ChatGPT can generate, modify, and optimize configurations
* **Templating Power**: Create and connect hundreds of machines with a few keystrokes using YAML anchors
* **IDE Support**: Syntax highlighting, validation, autocomplete, and refactoring tools

### UI Configuration Benefits

* **Zero Code**: Any OT person or business user can click and configure without YAML knowledge
* **Input Validation**: Real-time validation prevents configuration errors before deployment
* **Visual Guidance**: Wizards and forms guide users through complex protocol configurations
* **Live Preview**: See the impact of changes before applying them

### The Best of Both Worlds

**UMH Core provides two-way synchronization:** changes made in the UI are reflected in the YAML file, and YAML edits are shown in the UI. This means:

* **OT teams** can use the UI for day-to-day operations
* **IT teams** can use YAML for infrastructure-as-code workflows
* **Both approaches** maintain full version control and auditability
* **Scaling operations** benefit from templating while **individual adjustments** benefit from UI convenience

> **Pro Tip**: Start with the UI to understand the configuration structure, then switch to YAML for advanced templating and automation.

## Terminology Reference

| Current Term (UMH Core) | Legacy Term (UMH Classic) | YAML Key             | Description                                             |
| ----------------------- | ------------------------- | -------------------- | ------------------------------------------------------- |
| Bridge                  | Protocol Converter        | `protocolConverter:` | Connects external devices to UNS with health monitoring |
| Stand-alone Flow        | Data Flow Component (DFC) | `dataFlow:`          | Point-to-point data processing pipelines                |
| Stream Processor        | Stream Processor          | `dataFlow:`          | Processes data within UNS (upcoming feature)            |

> **Note:** YAML configuration keys retain legacy names for backward compatibility. The Management Console UI uses the current terminology.

### Agent - Runtime & Device identity

| Field                           | Type               | Default             | Purpose                                                                                                                                                                                                                                                                                                                                              |
| ------------------------------- | ------------------ | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `metricsPort`                   | `int`              | **9102**            | Exposes Prometheus metrics for the container.                                                                                                                                                                                                                                                                                                        |
| `location`                      | map `int → string` | –                   | Hierarchical location path (level0-4+) that identifies this instance. **Level 0 (enterprise) is mandatory**. Can follow ISA-95, KKS, or any organizational naming standard.                                                                                                                                                                          |
| `communicator.apiUrl`           | `string`           | – (console-managed) | HTTPS endpoint of the Management Console.                                                                                                                                                                                                                                                                                                            |
| `communicator.authToken`        | `string`           | –                   | API Key issued by the console. Can be set via `AUTH_TOKEN` env-var.                                                                                                                                                                                                                                                                                  |
| `communicator.allowInsecureTLS` | `bool`             | `false`             | Skip TLS verification [https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/production/corporate-firewalls.md](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/production/corporate-firewalls.md "mention"). Can be set via `ALLOW_INSECURE_TLS` env-var. |

**Location levels**

| Index | Generic Level | ISA-95 Example     | KKS Example     | Other Examples  |
| ----- | ------------- | ------------------ | --------------- | --------------- |
| `0`   | Enterprise    | `enterprise`       | `powerplant`    | `acme-inc`      |
| `1`   | Site/Region   | `site`/`plant`     | `unit-group`    | `cologne-plant` |
| `2`   | Area/Zone     | `area`/`line`      | `unit`          | `cnc-line`      |
| `3+`  | Work Cell+    | `work-cell`, `plc` | `component-grp` | `plc123`        |

```yaml
agent:
  metricsPort: 9102
  communicator:
    apiUrl: "https://api.management.umh.app"
    authToken: "${AUTH_TOKEN}"
  location:
    0: acme-inc
    1: plant1
    2: press-shop
    
```

### DataFlow - Stand-alone Flows

A list of all [stand-alone flows](/usage/data-flows/stand-alone-flow) (UI: "Stand-alone Flows", YAML: `dataFlow:`). Each entry spins up one Benthos-UMH instance.

| Field                                 | Type                  | Required | Description                                                                                                                                                                                                                                                                                                                            |
| ------------------------------------- | --------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                                | `string`              | ✔        | Unique within the file.                                                                                                                                                                                                                                                                                                                |
| `desiredState`                        | `active` \| `stopped` | ✔        | Agent will converge to this state.                                                                                                                                                                                                                                                                                                     |
| `dataFlowComponentConfig.debug_level` | `bool`                | ✗        | Enable verbose logging. When `true`, enables DEBUG log level and detailed operational logging (e.g., input/output operations, processing steps, data transformations). When `false` (default), uses INFO log level. **Use only for troubleshooting** - debug logging generates significantly more log data and may impact performance. |
| `dataFlowComponentConfig.benthos`     | object                | ✔        | Inline Benthos config (inputs, pipeline, outputs).                                                                                                                                                                                                                                                                                     |

```yaml
dataFlow:
- name: opcua-to-uns
  desiredState: active
  dataFlowComponentConfig:
    debug_level: false  # Enable detailed logging for troubleshooting
    benthos:
      input:
        opcua:
          endpoint:  "opc.tcp://192.168.0.50:4840"
          nodeIDs:   ["ns=2;s=FolderNode"]
      pipeline:
        processors:
          - tag_processor:
              defaults: |
                msg.meta.location_path = "acme.plant1.press-shop.plc1"
                msg.meta.data_contract = "_raw"
                msg.meta.tag_name      = "value"
                return msg;
      output:
        uns: {}           # write into embedded Redpanda
```

**Quick field map**

| Path                  | Notes                                                                          |
| --------------------- | ------------------------------------------------------------------------------ |
| `input.*`             | Any supported Benthos-UMH input (OPC UA, S7comm, Modbus, MQTT, Kafka, …).      |
| `pipeline.processors` | Standard Benthos processors **plus** `tag_processor` & `nodered_js`.           |
| `output.*`            | Usually `uns: {}` for Unified Namespace. MQTT, HTTP, SQL, etc. also available. |

*For complete input/output syntax see* [*Benthos-UMH Documentation*](https://docs.umh.app/benthos-umh)

### Bridge – Bridge from Device to UNS

> 🚧 **Roadmap Item**: Bridges are under active development. Current functionality includes connection monitoring and basic read/write flows.

A [**bridge**](/usage/data-flows/bridges) (UI: "Bridges", YAML: `protocolConverter:`) ingests data from a field device (e.g. OPC UA server, Siemens S7, Modbus controller) and pushes it into the Unified Namespace (UNS), and vice versa.\\

It combines a **connection probe** and two **stand-alone data flows (one for reading and one for writing)** under a single name and lifecycle.

```yaml
protocolConverter:
- name: press-opcua
  desiredState: active
  protocolConverterServiceConfig:
    debug_level: false  # Enable detailed logging for troubleshooting
    location:
      2: press1
    config:
      connection:
        nmap:
          target: "{{ .IP }}"
          port: "{{ .PORT }}"
      dataflowcomponent_read:
        benthos:
          input:
            opcua:
              endpoint: "opc.tcp://{{ .IP }}:{{ .PORT }}"
              nodeIDs: ["ns=2;s=MachineFolder"]
          pipeline:
            processors:
              - tag_processor:
                  defaults: |
                    msg.meta.location_path = "{{ .location_path }}"
                    msg.meta.data_contract = "_raw"
                    msg.meta.tag_name      = msg.meta.opcua_tag_name
                    return msg;
          output:
            uns: {}
    variables:
      # Optional templating support
      IP: "192.168.0.50"
      PORT: "4840"
```

#### Key fields

| Field                                        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                                       | Unique ID for this converter instance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `desiredState`                               | `active` to run immediately, or `stopped` to keep it defined but off.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `protocolConverterServiceConfig.location`    | Appended to the global `agent.location`. Optional, but useful to identify per-machine data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `protocolConverterServiceConfig.debug_level` | Enable verbose logging for troubleshooting. When `true`, enables DEBUG log level and protocol-specific debugging (e.g., OPC UA packet inspection via `OPC_DEBUG=1` environment variable, detailed connection diagnostics). When `false` (default), uses INFO log level. **Use only for troubleshooting** - debug logging generates significantly more log data and may impact performance. Disable after resolving issues. **Note:** Same structure as stand-alone flows - `debug_level` is a direct child of the service config, NOT inside `config:` wrapper. |
| `template.connection.nmap`                   | TCP liveness check to decide if the device is reachable.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `template.dataflowcomponent_read.benthos`    | Benthos pipeline to pull and forward data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `template.dataflowcomponent_write.benthos`   | Benthos pipeline to push and forward data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `variables`                                  | Optional Go-template variables, referenced via `{{ .VARNAME }}` in the template. See [Template Variables Reference](/reference/variables#variables) for details.                                                                                                                                                                                                                                                                                                                                                                                                |

### Benthos-UMH Processors

UMH Core includes specialized processors for industrial data:

#### tag\_processor 🚧

> **🚧 Roadmap Item**: The current `tag_processor` implementation follows the benthos-umh pattern with tag names in payloads. With the next UMH Core release, `tag_processor` will be updated to align with the new data model where:
>
> * Tag names are only in topics (not in payloads)
> * Metadata is not included in message payloads
> * Output follows standard [timeseries payload format](/usage/unified-namespace/payload-formats)

For current implementation details, see [Benthos-UMH Tag Processor Documentation](https://docs.umh.app/benthos-umh/processing/tag-processor).

```yaml
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.location_path = "{{ .location_path }}";
          msg.meta.data_contract = "_raw";
          msg.meta.tag_name = "temperature";
          return msg;
```

**Data Contract Guidelines:**

* Use `_raw` for simple sensor data and initial device integration
* Use explicit contracts (e.g., `_temperature`, `_pump`) with [data models](/usage/data-modeling) for structured enterprise data
* **Migration from UMH Classic:** See [Migration from UMH Classic to UMH Core](/production/migration-from-classic) for `_historian` contract migration instructions

For detailed documentation, see [Benthos-UMH Tag Processor](https://docs.umh.app/benthos-umh/processing/tag-processor).

#### nodered\_js 🚧

Process data using Node-RED-style JavaScript:

```yaml
processors:
  - nodered_js:
      code: |
        msg.payload = msg.payload * 1.8 + 32;  // Convert C to F
        msg.topic = "temperature_fahrenheit";
        return msg;
```

For complete documentation, see [Benthos-UMH Node-RED JavaScript Processor](https://docs.umh.app/benthos-umh/processing/node-red-javascript-processor).

### Industrial Input Protocols

UMH Core supports 50+ industrial protocols via Benthos-UMH. For complete, up-to-date configuration examples, see [Benthos-UMH Input Documentation](https://docs.umh.app/benthos-umh/input/).

**Popular protocols include:**

* [OPC UA](https://docs.umh.app/benthos-umh/input/opc-ua-input) - Industry standard automation
* [Modbus](https://docs.umh.app/benthos-umh/input/modbus) - TCP/RTU serial communication
* [Siemens S7](https://docs.umh.app/benthos-umh/input/siemens-s7) - Direct PLC access
* [Ethernet/IP](https://docs.umh.app/benthos-umh/input/ethernet-ip) - Allen-Bradley devices

**UMH Core Integration Pattern:**

```yaml
protocolConverter:
  - name: device-bridge
    protocolConverterServiceConfig:
      config:
        dataflowcomponent_read:
          benthos:
            input:
              # ANY benthos-umh protocol input
            pipeline:
              processors:
                - tag_processor: { /* UNS metadata setup */ }
            output:
              uns: {}
```

### GraphQL API - Topic Browser 🚧

GraphQL API for querying Unified Namespace topics.

| Field         | Type       | Default | Description                               |
| ------------- | ---------- | ------- | ----------------------------------------- |
| `enabled`     | `bool`     | `false` | Enable GraphQL API (Early Access; opt in) |
| `port`        | `int`      | `8090`  | HTTP port for GraphQL endpoint            |
| `debug`       | `bool`     | `false` | Enable GraphiQL playground at `/`         |
| `corsOrigins` | `[]string` | `[]`    | CORS origins (empty = allow all)          |

```yaml
agent:
  graphql:
    enabled: false   # Default: false; set true to opt in (Early Access)
    port: 8090
    debug: false
    corsOrigins: 
      - "http://localhost:3000"
      - "https://your-app.com"
```

**Endpoints:**

* **GraphQL API:** `POST /graphql`
* **GraphiQL Playground:** `GET /` (debug mode only)
* **CORS Preflight:** `OPTIONS /graphql`

For complete API reference, see [Topic Browser GraphQL API](/reference/http-api/topic-browser-graphql).

### Internal - Built-In Services (expert)

UMH Core injects this section automatically.

```yaml
internal:
  redpanda:
    desiredState: active
    redpandaServiceConfig:
      topic:
        defaultTopicRetentionMs: 604800000   # 7 days
        defaultTopicCleanupPolicy: delete    # keep 7 days of raw records
      resources:
        maxCores: 1
        memoryPerCoreInBytes: 2147483648
  topicbrowser:
    desiredState: active  # Topic Browser service (auto-enabled)
```

> **Do not edit** unless instructed by UMH support; invalid settings can brick the stack.

#### Topic retention on new instances

On its very first boot — when there is no `config.yaml` in `/data` yet — an instance writes `defaultTopicCleanupPolicy: delete` and `defaultTopicRetentionMs: 604800000` into the file it creates. Together these keep seven days of raw records in `umh.messages`, so a new consumer group can replay the full history; the Historian and any other late-joining consumer depend on this.

Two consequences to be aware of:

* **Disk usage scales with throughput, not tag count.** Seven days of raw records is far more than a compacted snapshot. See the [Sizing Guide](/production/sizing-guide) before running a high-throughput instance, and cap the topic with `defaultTopicRetentionBytes` if disk is tighter than the retention window.
* **The Topic Browser tree is rebuilt from the retained window.** After a restart, a tag that has not published for more than seven days is missing from the tree until it publishes again.

Instances that already have a `config.yaml` keep the policy they run on today, which is `compact` unless it was set explicitly — upgrading never changes their retention. To move an existing instance to seven-day replay, set `defaultTopicCleanupPolicy: delete` yourself. Records that compaction already collapsed cannot be recovered.

### Validation & tips

* **YAML anchors** are supported – useful for re-using input or output snippets.
* The Agent logs schema errors and refuses to activate a malformed DFC.

## Related Documentation

* [**Bridges**](/usage/data-flows/bridges) - Device connectivity patterns
* [**Stand-alone Flows**](/usage/data-flows/stand-alone-flow) - Custom data processing
* [**Data Modeling**](/usage/data-modeling) - Structure your industrial data
* [**State Machines**](/reference/state-machines) - Component lifecycle management

## External References

* [**Benthos-UMH Documentation**](https://docs.umh.app/benthos-umh) - Complete protocol and processor reference
* [**Management Console**](https://management.umh.app) - Web-based configuration interface


# Data Model Type Definitions

> This is a reference document. Use it to look up the types that a data model field accepts and the errors the validator may raise.

A field's value type is declared at one of two levels. A data model field references a type with `_payloadshape` (a named payload shape) or `_refModel` (another data model). Inside a payload shape, each field declares a scalar directly with `_type`.

## Field Types (`_type`)

A [payload shape](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/reference/payload-shapes.md) field's `_type` is one of:

| `_type`   | Meaning                             | Example value |
| --------- | ----------------------------------- | ------------- |
| `string`  | Text                                | `"running"`   |
| `number`  | Any numeric value, decimal or whole | `42.5`, `42`  |
| `boolean` | True / false                        | `true`        |

Any other value is rejected: `unsupported UMH type: <type>`.

## Timeseries payload shapes are inherently typed

When defining the `_payloadshape` to be timeseries data, the name of the payload shape already states the type. As such, no additional type definition is needed.

| Payload shape        | `value` type | Use for                         |
| -------------------- | ------------ | ------------------------------- |
| `timeseries-number`  | `number`     | Numeric readings (temperature…) |
| `timeseries-string`  | `string`     | Text status values              |
| `timeseries-boolean` | `boolean`    | On/off, true/false state        |

Each built-in timeseries shape also carries `timestamp_ms` (a `number`) next to `value`. Both fields come from the built-in shape definition, so you never declare `timestamp_ms` yourself. See [Built-in Shapes](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/reference/payload-shapes.md#built-in-shapes) for the full payload structure.

A `_payloadshape` reference must name one of the three built-ins above or a shape defined under `payloadShapes:`. Any other name fails validation with `referenced payload shape '<name>' does not exist`.

```yaml
temperature:
  _payloadshape: timeseries-number
status:
  _payloadshape: timeseries-string
```

## Examples

Data model with several timeseries fields:

```yaml
dataModels:
  - name: temperature
    version:
      v1:
        structure:
          temperature:
            _payloadshape: timeseries-number
          unit:
            _payloadshape: timeseries-string
```

Nested folders, and a field that references another data model (`_refModel`):

```yaml
dataModels:
  - name: complex-model
    version:
      v1:
        structure:
          sensor: # folder here
            temp_reading:
              _payloadshape: timeseries-number
            temp_unit:
              _refModel:
                name: temperature
                version: v1
          metadata:
            _refModel:
              name: device-info
              version: v1
```

Multiple versions of one model:

```yaml
dataModels:
  - name: sensor-data
    version:
      v1:
        structure:
          value:
            _payloadshape: timeseries-number
      v2:
        structure:
          value:
            _payloadshape: timeseries-number
          status:
            _payloadshape: timeseries-string
```

Define a custom [payload shape](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/reference/payload-shapes.md) (top-level `payloadShapes:`), then reference it:

```yaml
payloadShapes:
  work-order:
    description: Work order record
    fields:
      orderId:
        _type: string
      quantity:
        _type: number
      price:
        _type: number
      active:
        _type: boolean

dataModels:
  - name: orders
    version:
      v1:
        structure:
          order:
            _payloadshape: work-order
```

Rules a data model field must follow:

* A leaf field references its type with `_payloadshape` or `_refModel`, never both.
* A folder (a field with subfields) has neither `_payloadshape` nor `_refModel`.

## Validation Errors

| Error message                                                      | Cause                                                              | Fix                                                             |
| ------------------------------------------------------------------ | ------------------------------------------------------------------ | --------------------------------------------------------------- |
| `referenced payload shape '<name>' does not exist`                 | `_payloadshape` names a shape that is not built-in and not defined | Use a built-in, or define the shape first                       |
| `unsupported UMH type: <type>`                                     | A field `_type` is not a supported type                            | Use `string`, `number`, or `boolean`                            |
| `field cannot have both _payloadshape and _refModel`               | A leaf field sets both keys                                        | Keep one                                                        |
| `leaf nodes must contain _payloadshape, _relational, or _refModel` | A leaf field has no value-type key                                 | Add `_payloadshape` or `_refModel`, or give the field subfields |
| `non-leaf nodes (folders) cannot have _payloadshape`               | A field with subfields also sets `_payloadshape`                   | Remove `_payloadshape` from the folder                          |

## Related

* [Data Modeling](/reference) - Concepts and the component chain
* [Payload Shapes](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/reference/payload-shapes.md) - Built-in and custom shapes
* [Data Models](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/reference/data-models.md) - Structure, `_refModel`, versions
* [Data Contracts](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/reference/data-contracts.md) - Enforcement at ingress
* [Payload Formats](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/unified-namespace/payload-formats.md) - UNS payload structure

> ### Note: the `integer` type
>
> The YAML validator also accepts `_type: integer` inside a custom payload shape defined in an instance's config file. However, there is no built-in `timeseries-integer` shape and the data-model editor does not offer it, so most models never need it. Use `number` for numeric values. `integer` is only relevant when a custom payload shape must reject fractional values (for example a discrete count or an ID).


# Container Layout

```
/data
 ├─ config.yaml           # See also configuration reference
 ├─ config-backups/       # Timestamped config.yaml backups (last 100 retained)
 ├─ logs/                 # Rolling logs for agent, every data flow, Redpanda …
 ├─ redpanda/             # Redpanda data & WALs (backup-worthy)
 ├─ services/             # S6 service directories (only when S6_PERSIST_DIRECTORY=true)
 └─ hwid                  # Device fingerprint sent to the console

/tmp/umh-core-services/   # S6 service directories (default, cleared on restart)

/run/service/             # S6 scan directory (contains symlinks to service directories)
```

Mount **one persistent volume** (e.g. `umh-core-data`) to `/data` and you're done.

#### Advanced: Custom Data Location

If you need control over the exact data location (e.g., for compliance or backup requirements), you can use a custom folder instead of a Docker volume:

```bash
mkdir -p /path/to/umh-core-data
# Ensure container user has write access
chown -R 1000:1000 /path/to/umh-core-data
docker run -d --name umh-core -v /path/to/umh-core-data:/data:z ...
```

On SELinux systems (RHEL, Rocky), the `:z` flag allows Docker to relabel the directory. It's harmlessly ignored on other systems.

#### Upgrading Custom Folders to v0.44+

Version 0.44+ runs as a non-root user. If upgrading from an older version with a custom data folder, fix permissions first:

```bash
docker stop umh-core
docker rm umh-core

# Fix permissions for non-root container
sudo chown -R 1000:1000 /path/to/umh-core-data

docker run -d \
  --name umh-core \
  --restart unless-stopped \
  -v /path/to/umh-core-data:/data:z \
  management.umh.app/oci/united-manufacturing-hub/umh-core:<VERSION>
```

#### Migrating to Docker Volumes (Optional)

If you want to switch from a custom folder to a Docker volume:

```bash
docker stop umh-core
docker rm umh-core

# Create volume and copy data
docker volume create umh-core-data
docker run --rm \
  -v /path/to/umh-core-data:/source:ro,z \
  -v umh-core-data:/target \
  alpine sh -c "cp -av /source/. /target/"

# Fix permissions and start
docker run --rm -v umh-core-data:/data alpine chown -R 1000:1000 /data

docker run -d \
  --name umh-core \
  --restart unless-stopped \
  -v umh-core-data:/data \
  management.umh.app/oci/united-manufacturing-hub/umh-core:<VERSION>
```

Keep your old folder as backup for 24-48 hours before deleting.

### /config.yaml

See also [Configuration Reference](/reference/configuration-reference)

### /logs

| File/dir                                 | What it is for                                                                                 | When it appears                                                                                                     |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **`current`**                            | The file S6 is *actively* appending log lines to. Keep an eye on this one with `tail -f`.      | Always – it is recreated immediately after every rotation. ([Skarnet](https://skarnet.org/software/s6/s6-log.html)) |
| **`previous`**                           | A temporary name used *during* a rotation; disappears once rotation completes.                 | Only while a rotation is in flight. ([Skarnet](https://skarnet.org/software/s6/s6-log.html))                        |
| **`@<timestamp>.s`**                     | An archived log that was rotated *cleanly*. The timestamp is the moment the rotation occurred. | After every successful rotation. ([Skarnet](https://skarnet.org/software/s6/s6-log.html))                           |
| **`@<timestamp>.u`**                     | A “unfinished” archive – it was the `current` file when the container was killed.              | Only after an un-clean shutdown. ([Skarnet](https://skarnet.org/software/s6/s6-log.html))                           |
| `lock`, `state`, `processed`, `newstate` | Book-keeping files S6-log uses while rotating or while a post-processor runs.                  | Internal – you normally ignore them. ([Skarnet](https://skarnet.org/software/s6/s6-log.html))                       |

#### The life-cycle in practice

1. **Normal running** – all services write to their own `current` file.
2. **Size hits 1 MB** – S6 atomically renames `current` to a name such as `@20250530T131218Z.s`, then immediately creates a fresh empty `current`. ([Skarnet](https://skarnet.org/software/s6/s6-log.html))
3. **Prune** – if the directory now has > 20 archives, the oldest ones are deleted so the newest 20 remain. ([Skarnet](https://skarnet.org/software/s6/s6-log.html))
4. **You read logs** – use:

   ```bash
   # live stream
   tail -f /data/logs/<service>/current

   # inspect an old file (the '@…s' ones are plain text)
   less /data/logs/<service>/@20250530T131218Z.
   ```

### /redpanda

The Redpanda data directory.

### HWID

A unique identifier for that UMH Core installation. Useful for troubleshooting.

### S6 Service Directories

UMH Core uses S6 overlay for service supervision. Service directories contain the runtime state and configuration for each managed service (Benthos, Redpanda, monitors, etc.).

By default, these directories are created in `/tmp/umh-core-services/` which is **cleared on container restart**, ensuring a clean state. This prevents issues from stale supervisor state files.

For debugging purposes, you can set `S6_PERSIST_DIRECTORY=true` to use `/data/services/` instead, which persists across container restarts. This allows inspection of S6 supervisor state files when troubleshooting service startup issues.

The `/run/service/` directory contains symlinks pointing to the actual service directories, which S6's scanner monitors for changes.

See the [Environment Variables](/reference/environment-variables) reference for more details on `S6_PERSIST_DIRECTORY`.


# State Machines

**State machines are the core orchestration mechanism in UMH Core.** Every component is managed by a finite state machine (FSM) with clearly defined states and transitions. This provides predictable, observable behavior and enables reliable error handling and recovery.

## How It Works

UMH Core uses hierarchical state machines where components build upon each other:

* **Bridge** = Connection + Read Flow + Write Flow
* **Flow** = Benthos instance with lifecycle management
* **Benthos Flow** = Individual Benthos process with detailed startup phases
* **Connection** = Network probe service (typically nmap-based)

Each component inherits lifecycle states (`to_be_created`, `creating`, `removing`, `removed`) and adds operational states specific to its function. The Agent continuously reconciles desired vs actual state, triggering appropriate transitions based on observed conditions.

## 1 — Redpanda Service

| State           | What it means                                                                         | How it is entered                                     | How it leaves                                                                                                                                           |
| --------------- | ------------------------------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **stopped**     | `redpanda` process not running.                                                       | *stop\_done* from **stopping**, or initial create.    | *start* → **starting**                                                                                                                                  |
| *starting*      | S6 launching broker; health checks pending.                                           | *start* event from **stopped**.                       | <p><em>start\_done</em> → <strong>idle</strong><br><em>start\_failed</em> → <strong>stopped</strong></p>                                                |
| **idle**        | Broker healthy, **no data for 30 s** (default idle window).                           | *start\_done* or *no\_data\_timeout* from **active**. | <p><em>data\_received</em> → <strong>active</strong><br><em>degraded</em> → <strong>degraded</strong><br><em>stop</em> → <strong>stopping</strong></p>  |
| **active**      | Broker healthy & `BytesIn/OutPerSec` > 0.                                             | *data\_received* from **idle**.                       | <p><em>no\_data\_timeout</em> → <strong>idle</strong><br><em>degraded</em> → <strong>degraded</strong><br><em>stop</em> → <strong>stopping</strong></p> |
| ⚠️ **degraded** | Broker running but ≥1 health‑check failing (`disk-space-low`, `cpu-saturated`, etc.). | *degraded* from **idle/active**.                      | <p><em>recovered</em> → <strong>idle</strong><br><em>stop</em> → <strong>stopping</strong></p>                                                          |
| *stopping*      | Graceful shutdown (draining clients).                                                 | *stop* from any running state.                        | *stop\_done* → **stopped**                                                                                                                              |

***

## 2 — Container Monitor

| State                | Meaning                                    | Enter trigger                                                    | Exit trigger                                       |
| -------------------- | ------------------------------------------ | ---------------------------------------------------------------- | -------------------------------------------------- |
| **active**           | CPU < 85 %, RAM < 90 %, Disk < 90 %.       | *metrics\_all\_ok* after monitor start **or** from **degraded**. | *metrics\_not\_ok* → **degraded**                  |
| ⚠️ **degraded**      | One of the above limits breached for 15 s. | *metrics\_not\_ok*                                               | *metrics\_all\_ok* → **active**                    |
| monitoring\_stopped  | Watchdog disabled.                         | *stop\_monitoring\_done*                                         | *start\_monitoring* → monitoring\_starting         |
| monitoring\_starting | Monitor service booting.                   | *start\_monitoring*                                              | *start\_monitoring\_done* → **degraded** (initial) |
| monitoring\_stopping | Monitor shutting down.                     | *stop\_monitoring*                                               | *stop\_monitoring\_done* → monitoring\_stopped     |

***

## 3 — Agent Monitor

| State                | Meaning                                      | Enter                    | Exit                                               |
| -------------------- | -------------------------------------------- | ------------------------ | -------------------------------------------------- |
| **active**           | Agent connected & internal tasks OK.         | *metrics\_all\_ok*       | *metrics\_not\_ok* → **degraded**                  |
| ⚠️ **degraded**      | Cloud unreachable / auth error / task panic. | *metrics\_not\_ok*       | *metrics\_all\_ok* → **active**                    |
| monitoring\_stopped  | Agent health monitor off.                    | *stop\_monitoring\_done* | *start\_monitoring* → monitoring\_starting         |
| monitoring\_starting | Starting health checks.                      | *start\_monitoring*      | *start\_monitoring\_done* → **degraded** (initial) |
| monitoring\_stopping | Halting checks.                              | *stop\_monitoring*       | *stop\_monitoring\_done* → monitoring\_stopped     |

***

## 4 — Bridge

### Aggregate Bridge FSM

| State                              | Meaning                                       | Status Reason Examples                           | Enter                                              | Exit                                                                                                                                                               |
| ---------------------------------- | --------------------------------------------- | ------------------------------------------------ | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **stopped**                        | All sub‑services stopped.                     | `"stopped"`                                      | *stop\_done* or after create.                      | *start* → **starting\_connection**                                                                                                                                 |
| *starting\_connection*             | Waiting for connection to establish.          | `"starting: waiting for connection"`             | *start*                                            | *start\_connection\_up* → **starting\_redpanda**                                                                                                                   |
| *starting\_redpanda*               | Connection up, waiting for message broker.    | `"starting: redpanda not healthy"`               | *start\_connection\_up*                            | *start\_redpanda\_up* → **starting\_dfc**                                                                                                                          |
| *starting\_dfc*                    | Connection + Redpanda up, waiting for flow.   | `"starting: flow not running"`                   | *start\_redpanda\_up*                              | <p><em>start\_dfc\_up</em> → <strong>idle</strong><br><em>start\_failed\_dfc\_missing</em> → <strong>starting\_failed\_dfc\_missing</strong></p>                   |
| **starting\_failed\_dfc**          | Flow component failed to start.               | `"starting failed: flow in error state"`         | *start\_failed\_dfc*                               | Manual retry or removal                                                                                                                                            |
| **starting\_failed\_dfc\_missing** | No flow configured.                           | `"starting failed: no flows configured"`         | *start\_failed\_dfc\_missing*                      | *start\_retry* (when flow added) or removal                                                                                                                        |
| **idle**                           | All healthy, **no data for 30 s**.            | `"idling: no messages processed in 60s"`         | *start\_dfc\_up*, *no\_data\_timeout*, *recovered* | <p><em>data\_received</em> → <strong>active</strong><br><em>degraded</em> events → <strong>degraded\_</strong>\*<br><em>stop</em> → <strong>stopping</strong></p>  |
| **active**                         | Processing data through flows.                | `""` (empty when fully healthy)                  | *data\_received*                                   | <p><em>no\_data\_timeout</em> → <strong>idle</strong><br><em>degraded</em> events → <strong>degraded\_</strong>\*<br><em>stop</em> → <strong>stopping</strong></p> |
| ⚠️ **degraded\_connection**        | Connection lost/flaky after successful start. | `"connection degraded: probe timeout after 30s"` | *connection\_unhealthy*                            | <p><em>recovered</em> → <strong>idle</strong><br><em>stop</em> → <strong>stopping</strong></p>                                                                     |
| ⚠️ **degraded\_redpanda**          | Message broker issues after successful start. | `"redpanda degraded: not responding"`            | *redpanda\_degraded*                               | <p><em>recovered</em> → <strong>idle</strong><br><em>stop</em> → <strong>stopping</strong></p>                                                                     |
| ⚠️ **degraded\_dfc**               | Flow component issues after successful start. | `"flow degraded: benthos service not running"`   | *dfc\_degraded*                                    | <p><em>recovered</em> → <strong>idle</strong><br><em>stop</em> → <strong>stopping</strong></p>                                                                     |
| ⚠️ **degraded\_other**             | Inconsistent component states detected.       | `"other degraded: inconsistent states"`          | *degraded\_other*                                  | <p><em>recovered</em> → <strong>idle</strong><br><em>stop</em> → <strong>stopping</strong></p>                                                                     |
| *stopping*                         | Stopping all components.                      | `"stopping"`                                     | *stop*                                             | *stop\_done* → **stopped**                                                                                                                                         |

### 4.1 Connection Service FSM

| State           | Meaning                         |
| --------------- | ------------------------------- |
| *starting*      | Probe service launching.        |
| **up**          | Target reachable.               |
| **down**        | Target unreachable.             |
| ⚠️ **degraded** | Flaky / intermittent responses. |
| *stopping*      | Probe shutting down.            |
| **stopped**     | Probe disabled.                 |

### 4.2 Benthos Flow (Source /Sink)

| State                                                  | Meaning                                                |
| ------------------------------------------------------ | ------------------------------------------------------ |
| **stopped**                                            | Service file present, process not running.             |
| *starting*                                             | S6 launched process.                                   |
| *starting\_config\_loading*                            | Benthos parsing YAML pipeline.                         |
| *starting\_waiting\_for\_healthchecks*                 | Pipeline loaded; waiting for plugin health.            |
| *starting\_waiting\_for\_service\_to\_remain\_running* | Stability grace period.                                |
| **idle**                                               | Flow running, no msgs for idle window.                 |
| **active**                                             | Processing messages.                                   |
| ⚠️ **degraded**                                        | Flow running but error state (e.g., endpoint retries). |
| *stopping*                                             | Graceful SIGTERM underway.                             |

> **Idle/Active timeout:** default 30 s (`BRIDGE_IDLE_WINDOW`).

***

## 5 — Topic Browser Service

The Topic Browser service manages real-time topic discovery and caching.

| State                     | Description             | Enter Trigger                 | Exit Trigger                                 |
| ------------------------- | ----------------------- | ----------------------------- | -------------------------------------------- |
| **stopped**               | Service not running     | Initial state or *stop\_done* | *start* → **starting**                       |
| *starting*                | Service initialization  | *start*                       | *benthos\_started* → **starting\_benthos**   |
| *starting\_benthos*       | Benthos starting        | *benthos\_started*            | *redpanda\_started* → **starting\_redpanda** |
| *starting\_redpanda*      | Redpanda connection     | *redpanda\_started*           | *start\_done* → **idle**                     |
| **idle**                  | Healthy, no active data | *start\_done* or *recovered*  | *data\_received* → **active**                |
| **active**                | Processing topic data   | *data\_received*              | *no\_data\_timeout* → **idle**               |
| ⚠️ **degraded\_benthos**  | Benthos degraded        | *benthos\_degraded*           | *recovered* → **idle**                       |
| ⚠️ **degraded\_redpanda** | Redpanda degraded       | *redpanda\_degraded*          | *recovered* → **idle**                       |
| *stopping*                | Graceful shutdown       | *stop*                        | *stop\_done* → **stopped**                   |

**Default:** Active (runs automatically)\
**Transitions:** idle ↔ active based on topic activity\
**Recovery:** Automatic from degraded states when underlying services recover

***

### Quick Defaults

| Parameter              | Default | Source Const / Env     |
| ---------------------- | ------- | ---------------------- |
| Idle window (Redpanda) | 30 s    | `REDPANDA_IDLE_WINDOW` |
| Idle window (Bridge)   | 30 s    | `BRIDGE_IDLE_WINDOW`   |
| Container CPU limit    | 85 %    | `CONTAINER_CPU_LIMIT`  |
| Container RAM limit    | 90 %    | `CONTAINER_RAM_LIMIT`  |
| Container Disk limit   | 90 %    | `CONTAINER_DISK_LIMIT` |


# Environment Variables

This is the reference for all environment variables used by umh-core. These variables override or supplement configuration from `config.yaml`.

| Name                   | Values                                                                     | Description                                                                                                                                                                                                                              |
| ---------------------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| LOGGING\_LEVEL         | ERROR, WARN, INFO, DEBUG                                                   | Controls log verbosity. DEBUG shows detailed internal operations                                                                                                                                                                         |
| AUTH\_TOKEN            | (Base64 encoded token)                                                     | Management Console authentication token. Overrides auth token from config.yaml                                                                                                                                                           |
| API\_URL               | <https://management.umh.app/api>, <https://staging.management.umh.app/api> | Management Console API endpoint. Use staging for testing environments                                                                                                                                                                    |
| RELEASE\_CHANNEL       | enterprise, stable, nightly                                                | Auto-update channel. Enterprise = most stable, nightly = latest features                                                                                                                                                                 |
| ALLOW\_INSECURE\_TLS   | true, false                                                                | Skip TLS certificate verification. Use for corporate firewalls with MITM proxies                                                                                                                                                         |
| S6\_PERSIST\_DIRECTORY | true/1/TRUE/True, (unset/false/other)                                      | Controls S6 service directory persistence. Truthy values (true, 1, TRUE, etc.) = /data/services (persist across container restarts for debugging). Default/false = /tmp/umh-core-services (cleared on container restart for fresh state) |
| LOCATION\_<0-6>        | (string values)                                                            | Sets Agent location path levels 0-6. Example: LOCATION\_0=factory, LOCATION\_1=line1                                                                                                                                                     |


# Variables

## Overview

UMH Core uses a three-tier variable system for protocol converter templates that enables flexible configuration and runtime customization:

* **User Variables**: Defined in the `variables:` section of your YAML configuration, flattened to top-level access
* **Internal Variables**: Runtime-injected system metadata and computed values
* **Global Variables**: Fleet-wide settings (**NOT YET IMPLEMENTED**)

## Variable Precedence

Variables follow a clear precedence hierarchy:

1. **User Variables** (highest priority) - Override all others when flattened by VariableBundle
2. **Internal Variables** - System-generated values
3. **Global Variables** - Fleet-wide defaults (**NOT YET IMPLEMENTED**)

## Available Variables

### Connection Variables

These variables provide device connection information, typically auto-injected from Connection configurations or defined by users.

| Variable      | Description                   | Example Value   | Source          | Status  |
| ------------- | ----------------------------- | --------------- | --------------- | ------- |
| `{{ .IP }}`   | Device IP address or hostname | `192.168.1.100` | User/Connection | Current |
| `{{ .PORT }}` | Device communication port     | `4840`          | User/Connection | Current |

**Usage Example:**

```yaml
protocolConverter:
  config: |
    input:
      opcua:
        endpoint: "opc.tcp://{{ .IP }}:{{ .PORT }}/OPCUA/SimulationServer"
```

### Location Variables

These variables provide hierarchical location information for data organization and topic routing.

| Variable               | Description                       | Example Value                                | Source   | Status  |
| ---------------------- | --------------------------------- | -------------------------------------------- | -------- | ------- |
| `{{ .location_path }}` | Dot-separated hierarchical path   | `factory.line1.machine1`                     | Internal | Current |
| `{{ .location }}`      | Location as numbered map          | `{"0":"factory","1":"line1","2":"machine1"}` | Internal | Current |
| `{{ .location.0 }}`    | Enterprise level (first location) | `factory`                                    | Internal | Current |
| `{{ .location.1 }}`    | Site level (second location)      | `line1`                                      | Internal | Current |
| `{{ .location.2 }}`    | Area level (third location)       | `machine1`                                   | Internal | Current |

**Usage Example:**

```yaml
protocolConverter:
  config: |
    output:
      mqtt:
        topic: "umh/v1/{{ .location_path }}/data"
        # Or access individual levels:
        # topic: "umh/v1/{{ .location.0 }}/{{ .location.1 }}/data"
```

### Internal System Variables

These variables provide runtime metadata and system integration information.

| Variable                     | Description                                | Example Value                     | Source   | Status                     |
| ---------------------------- | ------------------------------------------ | --------------------------------- | -------- | -------------------------- |
| `{{ .internal.bridged_by }}` | Bridge identifier that created this config | `protocol-converter-node1-pc123`  | Internal | Current                    |
| `{{ .internal.id }}`         | Protocol converter instance ID             | `pc-123`                          | Internal | Current                    |
| `{{ .internal.umh_topic }}`  | UMH topic for write flows                  | `umh/v1/factory/line1/_historian` | Internal | Current (write flows only) |

**Usage Example:**

```yaml
protocolConverter:
  config: |
    output:
      mqtt:
        topic: "{{ .internal.umh_topic }}"
        metadata:
          bridged_by: "{{ .internal.bridged_by }}"
```

### Global Variables (Not Yet Implemented)

These variables will provide fleet-wide configuration when implemented.

| Variable                   | Description                | Example Value    | Source | Status              |
| -------------------------- | -------------------------- | ---------------- | ------ | ------------------- |
| `{{ .global.cluster_id }}` | Multi-cluster identifier   | `cluster-west-1` | Global | **NOT IMPLEMENTED** |
| `{{ .global.* }}`          | Other fleet-wide variables | varies           | Global | **NOT IMPLEMENTED** |

## Deprecated Variables

The following variables should be replaced in your configurations:

| Deprecated Variable  | Status       | Replacement   | Action Required               |
| -------------------- | ------------ | ------------- | ----------------------------- |
| `{{ .HOST }}`        | Deprecated   | `{{ .IP }}`   | Replace in all configurations |
| `{{ .DEVICE_IP }}`   | Inconsistent | `{{ .IP }}`   | Standardize usage             |
| `{{ .DEVICE_PORT }}` | Inconsistent | `{{ .PORT }}` | Standardize usage             |

## User-Defined Variables

You can define custom variables in the `variables:` section of your configuration. These are flattened to top-level access and override any internal variables with the same name.

**Common patterns:**

* `{{ .SCAN_RATE }}` - Polling intervals
* `{{ .TAG_PREFIX }}` - Tag naming prefixes
* `{{ .USERNAME }}` - Authentication credentials
* `{{ .PASSWORD }}` - Authentication credentials

**Example:**

```yaml
protocolConverter:
  variables:
    SCAN_RATE: "1000ms"
    TAG_PREFIX: "PLC1_"
  config: |
    input:
      opcua:
        endpoint: "opc.tcp://{{ .IP }}:{{ .PORT }}/OPCUA/SimulationServer"
        poll_interval: "{{ .SCAN_RATE }}"
        tag_prefix: "{{ .TAG_PREFIX }}"
```

## Variable Sources

Understanding where variables come from helps with troubleshooting:

* **Connection Variables** (`{{ .IP }}`, `{{ .PORT }}`): Auto-injected from Connection configurations attached to Bridges
* **Location Variables** (`{{ .location_path }}`, `{{ .location }}.*`): Computed from agent location + bridge location by BuildRuntimeConfig
* **Internal Variables** (`{{ .internal.* }}`): Runtime-injected system metadata
* **User Variables**: Explicitly defined in `variables:` section
* **Global Variables**: Fleet-wide settings (**NOT YET IMPLEMENTED**)

## Troubleshooting

### Variable Not Found

If a variable is not resolving:

1. Check spelling and case sensitivity
2. Verify the variable is defined in the `variables:` section (for user variables)
3. Ensure Connection is properly attached (for `.IP`/`.PORT`)
4. Check that location path is configured (for location variables)

### Deprecated Variable Warnings

Replace deprecated variables with their current equivalents:

* `{{ .HOST }}` → `{{ .IP }}`
* `{{ .DEVICE_IP }}` → `{{ .IP }}`
* `{{ .DEVICE_PORT }}` → `{{ .PORT }}`

For questions about variables not listed here, check the [configuration reference](/reference/configuration-reference) or consult the UMH Core documentation.


# HTTP API Reference

{% hint style="info" %}
**Early Access.** The GraphQL API lets developers query the Unified Namespace programmatically. It's experimental and off by default. To turn it on, set `agent.graphql.enabled: true` in the `agent:` block of your `umh-core` instance configuration. See [Configuration](/reference/http-api/topic-browser-graphql#configuration) for all options. Tell us what you think.
{% endhint %}

## GraphQL API

* **Default:** Disabled; enable with `agent.graphql.enabled: true` (port 8090)
* **Endpoint:** `POST /graphql`
* **GraphiQL:** Available at `/` when debug enabled
* **Purpose:** Query Unified Namespace topics

## Topic Browser GraphQL

* [Complete GraphQL API Reference](/reference/http-api/topic-browser-graphql)
* [Schema Documentation](/reference/http-api/topic-browser-graphql#schema)
* [Query Examples](/reference/http-api/topic-browser-graphql#examples)

## Monitoring

* **Metrics:** See [Production Metrics](/production/metrics)
* **Health:** Via FSM state monitoring


# Topic Browser GraphQL API

{% hint style="info" %}
**Early Access.** The GraphQL API lets developers query the Unified Namespace programmatically. It's experimental and off by default. To turn it on, set `agent.graphql.enabled: true` in the `agent:` block of your `umh-core` instance configuration. See [Configuration](#configuration) for all options. Tell us what you think.
{% endhint %}

## Overview

Query Unified Namespace topics and metadata in real-time.

* **Endpoint:** `http://localhost:8090/graphql`
* **Default:** Disabled; opt in with `agent.graphql.enabled: true`
* **Authentication:** None (open by default)

## Schema

### Queries

```graphql
type Query {
  topics(filter: TopicFilter, limit: Int): [Topic!]!
  topic(topic: String!): Topic
}
```

### Types

```graphql
type Topic {
  topic: String!
  metadata: [MetadataEntry!]!
  lastEvent: Event    # Latest event only
}

union Event = TimeSeriesEvent | RelationalEvent

type TimeSeriesEvent {
  producedAt: String!
  scalarType: String!
  numericValue: Float
  stringValue: String
  booleanValue: Boolean
}

type RelationalEvent {
  producedAt: String!
  json: String!
}

input TopicFilter {
  text: String
  meta: [MetaExpr!]
}

input MetaExpr {
  key: String!
  eq: String!
}
```

## Example Queries

### All topics:

```graphql
{ topics { topic metadata { key value } } }
```

### Filter by text:

```graphql
{ topics(filter: { text: "temperature" }) { topic } }
```

### Filter by metadata:

```graphql
{
  topics(filter: { 
    meta: [{ key: "data_contract", eq: "_pump" }] 
  }) {
    topic
    lastEvent {
      ... on TimeSeriesEvent {
        producedAt
        numericValue
      }
    }
  }
}
```

### Single topic:

```graphql
{
  topic(topic: "umh.v1.acme.plant1.line4.sensor1._raw.temperature") {
    metadata { key value }
    lastEvent {
      ... on TimeSeriesEvent {
        producedAt
        numericValue
        scalarType
      }
    }
  }
}
```

## Using curl

### Basic query:

```bash
curl -X POST http://localhost:8090/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ topics(limit: 3) { topic } }"}'
```

### Filter query:

```bash
curl -X POST http://localhost:8090/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ topics(filter: { text: \"pump\" }) { topic lastEvent { ... on TimeSeriesEvent { numericValue } } } }"}'
```

## Limitations

* **History:** Latest event only (no historical data)
* **Subscriptions:** Not supported (queries only)
* **Rate Limiting:** None (use responsibly)
* **Scope:** Single UMH instance only

## Multi-Instance Behavior

* Each UMH Core instance has its own Topic Browser
* GraphQL API returns topics from local instance only
* Management Console aggregates all instances into unified view

## Configuration

```yaml
agent:
  graphql:
    enabled: false   # Default: false; set true to opt in
    port: 8090      # Default: 8090
    debug: false    # Set true for GraphiQL UI
    corsOrigins: [] # Default: allows all origins
```

## Security

* **Authentication:** Currently open (no tokens required)
* **CORS:** Configurable via `corsOrigins` setting
* **Network Security:** Recommended for production deployments
* **Port Access:** Ensure port 8090 is appropriately secured


# UMH Core vs UMH Classic

## Why a single Docker container?

* **Enterprise deployment reality**
  * *In most enterprises*: **tight operating system images, firewalls, and network hoops** turn Kubernetes setup into friction. Custom load-balancers, network policies, storage classes, and custom upgrade paths create endless variables.
  * *UMH Core*: **almost anything that runs Docker works** — from MacBook terminals to edge boxes to enterprise K8s clusters.
* **Deployment complexity**
  * *UMH Classic*: many pods, sidecars, service meshes → a forest of YAML.
  * *UMH Core*: **one image, one command**, sub-second startup.
* **Observability & recovery**
  * *Classic*: error clues spread across pod logs, `kubectl` events and load-balancers. Kubernetes states (`Pending`, `Running`, …) are usually not what the user would expect—`Running` does **not** guarantee the pod is healthy.
  * *Core*: **each component has well-defined states** and is recovered if the state is not healthy (e.g., incl. watching out for error or warning logs, metrics checks, etc.); S6 restarts failed processes instantly.
* **Real-time responsiveness**
  * *Classic*: waiting thirty seconds for a new pipeline to launch due to Kubernetes overhead
  * *Core*: **ticks every 100 milliseconds**, so configuration changes feel instant
* **Version management**
  * *Classic*: drift between Node-RED plug-ins, broker, bridges.
  * *Core*: *bump one line in the Dockerfile → ship vX.Y* — everything already integration-tested.
* **Edge resource footprint**
  * *Classic*: edge boxes fight the kube-control-plane for CPU/RAM.
  * *Core*: no kubelet, no sidecars, no HiveMQ ⇒ **lighter RAM/CPU usage**.

We collapse the stack into a single supervised image, giving enterprises less integration pain and users deterministic behaviour. Inside that image the **Agent, Bridges, Redpanda, and Benthos** still run as separate S6-managed processes, so you keep modularity without the network overhead.

## Isn't that just going back to a monolith? *(Micro-services vs Monolith)*

A "modular monolith" is the sweet spot for most factories:

* **You don't split a five-piece toolkit into hundreds of micro-services** – yet one big binary (like some protocol converters out there) is risky. If one protocol converter fails, it takes all the rest with it.
* **UMH Core** packs the essentials (Agent, Benthos pipelines, Redpanda buffer) in one container but launches **every single part as its own S6-supervised process**. Crash one, the others stay green.
* **Scale when it hurts:** Need more juice later for select pipelines? Break that process out into its own container.
* **Micro-services tax** – separate repos, CI/CD pipelines, service discovery, network latency – only pays off when you reach FAANG-scale org size.

***

## "So you ditched Kubernetes? I thought you were cloud-native!"

We didn't ditch it—**we de-coupled from it**.

* **The enterprise Kubernetes reality:** Organizations run their K8s clusters with custom load balancers, network policies, custom storage classes, and their own upgrade paths. Building one product that survives every combination and ships an SLA is a nightmare.
* **Our solution:** **UMH Core is&#x20;*****more*****&#x20;k8s-friendly** because it's self-contained. Drop the same image into k3s, OpenShift, EKS or bare Docker; no hidden assumptions about CNIs, StorageClasses, or LBs.
* With one container, those variables disappear. **Kubernetes becomes an optional scheduler, not a hard dependency.**

***

## Do we lose Kubernetes advantages?

No – the two big ones are still there:

**Scalability**: Deploy one UMH Core instance per production line, each forwarding to a central site instance. This ISA-95 style hierarchy naturally distributes workload across your infrastructure.

**Failover**: Your container manager restarts crashed containers (30-60 seconds) and reschedules after node failure (2-5 minutes). See [High Availability](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/production/high-availability.md).

## Why is Kafka (Redpanda) embedded instead of its own container?

The UNS needs a **policy wall**. Every byte passes schema validation before it hits the log. Exposing raw Redpanda ports would let clients inject malformed data, bypassing Bridges. Embedding gives us:

* On-the-fly topic/ACL/retention re-config from the Agent.
* Enforcement of Matthew Parris' **portal / bridge / firewall** model.

***

## Where is MQTT?

1. **Bring-your-own broker** (HiveMQ CE, Mosquitto) in a side-car, then connect via a Bridge.
2. **UMH MQTT plug-in** (road-map) – a slim container that wraps Redpanda with an MQTT façade plus schema enforcement.

## Where is Benthos running?

Each Bridge launches its own Benthos pipeline **inside** the image; S6 supervises it. If that flow crashes, S6 restarts just that pipeline, leaving the rest untouched.

## Where is Node-RED?

Outside the image, on purpose. Node-RED plug-ins break often; letting plant IT run their preferred image means **we don't accidentally nuke custom flows** during an update.

***

## Why not ship "just a Helm chart"?

Kubernetes distros vary wildly (OpenShift SCCs, Rancher CNIs, air-gapped k3s, …). One universal chart devolves into a spaghetti of `if/else` we can't certify. **A single image sidesteps 90 % of those edge-case failures.**

***

## Why not let users own the Helm chart (self-managed)?

Our value prop is **curated integration + SLA**. If every customer patches their own chart, swaps Grafana versions, tweaks sidecars, we lose traceability and can't guarantee hot-fixes.

***

## How do I migrate from UMH Classic?

1. **Side-by-side:** deploy `umh-core` next to Classic, point a new Bridge at the old UNS topics.
2. **Cut-over:** switch producers to the new UNS, retire Classic components incrementally (Node-RED flows, Grafana, Timescale).
3. **Clean-up:** decommission the Helm stack once data is stable in UMH Core.

## What happened to Grafana, Timescale (Historian), and other "extras"?

**Short answer:** anything that needs heavy customization or third-party plug-ins now lives **outside** the core image, so you keep full version control.

* **Grafana + TimescaleDB (Historian)**\
  Deployed as a separate Docker or Helm package—or via the upcoming *UMH Historian* plug-in. Many plants already run their own Grafana stack or rely on community plug-ins; we don't want to auto-upgrade and break dashboards.
* **Node-RED**\
  Lives in its own container for the same reason: community plug-ins change fast, and site teams often pin specific versions.
* **MQTT broker**\
  Bring your own HiveMQ CE, Mosquitto, etc., or use the soon-to-ship *UMH MQTT* plug-in, which wraps Redpanda with an MQTT façade plus schema enforcement.

**Rule of thumb:** if a service demands lots of plug-ins or customization, we keep it out of the core image and provide a starter Compose/Helm file instead.

## How does UMH Core track health if it's not using Kubernetes pod states?

* **Remember Kubernetes?** A green "Running" pod in Kubernetes only means the container has started — not that your application is healthy.
* **In UMH Core, green actually means healthy.** Every component (Bridge, Benthos pipeline, Redpanda, etc.) is wrapped in a **finite-state machine** managed by the Agent: `starting → active → idle → degraded → stopped`.
* The Agent evaluates **metrics + logs + live config** every 100 ms. A Bridge stuck in a retry loop goes to `degraded`, a pipeline with zero throughput stays `idle`, etc. It fires warnings if anything drifts, and restarts a pipeline the moment a new config lands.
* States and counters are exported via `/metrics`, so you can alert on "connection *degraded* for > 30 s", not just pod restarts.

This offers **finer-grained insight** than Kubernetes' generic `Running/CrashLoopBackOff`.

## How do I connect my MQTT devices now?

Run your own broker (Mosquitto, HiveMQ CE, EMQX, etc.), then create a Bridge in UMH Core that subscribes to the topics you care about and inserts the messages into the Unified Namespace.\
This keeps MQTT exactly where most plants like it—outside the UNS as a "portal" layer—while UMH Core enforces schema and ACLs before data lands in the clean store, and before you forward it to other nodes.

## I don't want Kafka / Redpanda — I want MQTT to be **my** UNS

No problem.\
UMH Core's internal Redpanda is just a **hidden workhorse** for buffering and exactly-once processing; you don't interact with it directly.

1. **Keep (or spin up) your own MQTT broker** — Mosquitto, HiveMQ CE, EMQX, or the upcoming UMH MQTT plug-in.
2. **Create a Bridge** in UMH Core that subscribes to—or publishes from—the topics you choose.
3. From an application point of view, your Unified Namespace *is* MQTT.\
   Redpanda just ensures messages are durable, ordered, and schema-checked before they reach consumers.

You never have to operate or even "see" Redpanda; UMH Core embeds and manages it automatically.

## How do I connect two UMH Core instances?

**Short answer:** create a **Bridge** on each side that speaks the upcoming *umh-core-API* output/input plug-in.

> 🚧 **Roadmap**: The *umh-core-API* plugin is under development to enable direct Core-to-Core communication.

* It's the most firewall-friendly protocol in enterprise networks.
* SSL/TLS is easy to terminate and audit.
* A mature tool-chain (proxies, WAFs, tracing) already exists.
* With idempotent writes we achieve MQTT-style **QoS 1** (at-least-once, de-duplication on the receiving side).

*(Until this feature ships, you can already link cores via Kafka Bridges.)*

## How can I **view** or **query** my Unified Namespace?

* **Topic Browser** (road-map) – a point-and-click tree inside the Management Console to inspect live values.
* **REST API** – two endpoints:
  1. `POST /api/v1/ingest` – push messages into UMH Core (used by the core-to-core Bridge).
  2. `GET /api/v1/uns/tree` – fetch the namespace tree, including latest values and metadata.

These endpoints are secured by mTLS or JWT and respect the same ACLs the Bridges enforce.

***

## What is open-source now?

* **Was:** only the Helm chart was OSS; the Agent and most internals were closed-source.
* **Now:** with UMH Core **everything except the Management Console is Apache-2**.
  * Agent, Bridges, S6 service definitions, Redpanda integration, FSM health logic – all public.
  * The Management Console UI remains free-to-use (Community Edition)


# Introduction

benthos-umh is a specialized extension of Benthos (now known as Redpanda Connect) developed by the [United Manufacturing Hub (UMH)](https://www.umh.app). Tailored for the manufacturing industry, benthos-umh integrates additional manufacturing protocols such as OPC UA, Siemens S7, and Modbus.

## 📚 Documentation

**If you're viewing this on GitHub, please visit our full documentation at** [**docs.umh.app/benthos-umh**](https://docs.umh.app/benthos-umh)**.**


# Input

This section covers Benthos input plugins for collecting data from various industrial systems and protocols. Input plugins allow you to ingest data from PLCs, sensors, and other industrial devices into your Benthos data pipelines.

## Available Input Plugins

* [**Sparkplug B Input**](/benthos-umh/input/sparkplug-b-input) - Ingests data from MQTT brokers using the Sparkplug B specification. Acts as a Host (Secondary or Primary) to consume Sparkplug B messages from Edge Nodes and converts them to UMH-Core format with automatic hierarchy mapping.
* [**OPC UA Input**](/benthos-umh/input/opc-ua-input) - Connects to OPC UA servers to browse and subscribe to nodes. Supports various data types, authentication methods, and provides comprehensive metadata for each message.
* [**Modbus**](/benthos-umh/input/modbus) - Communicates with Modbus devices supporting coils, discrete inputs, holding registers, and input registers. Features register optimization, device-specific workarounds, and extensive configuration options.
* [**ifm IO-Link Master / SensorConnect**](/benthos-umh/input/ifm-io-link-master-sensorconnect) - Integrates with ifm electronic's IO-Link Master devices (AL1350, AL1352) and EIO404 Bluetooth mesh stations. Processes sensor data using IODD files for human-readable output.
* [**Beckhoff ADS (Community)**](/benthos-umh/input/beckhoff-ads-community) - Community-supported plugin for Beckhoff's ADS protocol. Supports batch reading, notifications, and symbol-based communication with Beckhoff PLCs.
* [**Siemens S7**](/benthos-umh/input/siemens-s7) - Connects to Siemens S7 PLCs via the native S7 protocol, enabling cyclic or on-demand reads of data blocks, inputs/outputs, and markers with configurable polling interval.
* [**Ethernet/IP**](/benthos-umh/input/ethernet-ip) - Interfaces with Rockwell and other Ethernet/IP compatible controllers, mainly supporting the CIP protocol.
* [**UNS Input**](/benthos-umh/input/uns-input) - Connects to UMH Core's Unified Namespace to consume and filter messages using regex patterns against UMH topic keys.
* [**More Input Plugins**](https://docs.redpanda.com/redpanda-connect/components/inputs/about/) - Additional built-in input plugins available in Benthos/Redpanda Connect for various data sources.

## Choosing the Right Input Plugin

* Use **Sparkplug B Input** when consuming data from existing Sparkplug B Edge Nodes or integrating with Sparkplug B-enabled devices
* Use **OPC UA Input** for modern industrial systems that support the OPC UA standard
* Use **Modbus** for legacy industrial devices and PLCs that communicate via Modbus protocol
* Use **ifm IO-Link Master** when working with ifm electronic's IO-Link infrastructure and sensors
* Use **Beckhoff ADS** specifically for Beckhoff PLC systems and TwinCAT environments
* Use **Siemens S7** for industrial Siemens devices that communicate via Siemens protocol
* Use **Ethernet/IP** for CompactLogix, ControlLogix, Micro800er series, that communicate via CIP protocol
* Use **UNS Input** when working within UMH Core to consume and filter messages from the Unified Namespace
* Explore **additional input plugins** for other protocols like HTTP, MQTT, databases, file systems, and more


# Sparkplug B (Input)

## Overview

The **Sparkplug B Input plugin** allows the United Manufacturing Hub (UMH) to ingest data from MQTT brokers using the Sparkplug B specification. It subscribes to Sparkplug B MQTT topics (e.g., device birth/data/death messages) and converts the incoming Protobuf payloads into UMH-compatible messages. It maintains the stateful context required by Sparkplug B – tracking device birth certificates, metric alias mapping, and sequence numbers – so that incoming data is interpreted correctly.

This input plugin is designed to seamlessly integrate Sparkplug-enabled edge devices into the UMH **Unified Namespace**. It automatically decodes Sparkplug messages and enriches them with metadata (such as metric names, types, and timestamps) to fit the UMH-Core data model.

## Sparkplug B in UMH Architecture

### UMH's Modified Parris Method

UMH implements a **Modified Parris Method** that distributes hierarchy across both `device_id` and `metric_name` fields instead of cramming everything into `GroupID`. This approach provides significant advantages for multi-site deployments (e.g., a single UMH instance can ingest data from multiple sites):

**Key Innovation**:

* **Location Hierarchy** → `device_id`: `"enterprise.site.area.line"` → `"enterprise:site:area:line"`
* **Virtual Path Hierarchy** → `metric_name`: Supports multiple separators:
  * Colons: `"motor:diagnostics:temperature"` → virtual\_path=`"motor.diagnostics"` + tag\_name=`"temperature"`
  * Slashes: `"motor/diagnostics/temperature"` → virtual\_path=`"motor.diagnostics"` + tag\_name=`"temperature"`
  * Dots: `"motor.diagnostics.temperature"` → virtual\_path=`"motor.diagnostics"` + tag\_name=`"temperature"`

### Architecture Roles

**Sparkplug B Input Plugin** (this plugin):

* **Role**: Host (Secondary Host by default, Primary Host optional)
* **Function**: Consumes Sparkplug B messages from external systems
* **Output**: Converts to UMH-Core format for UNS integration

**Sparkplug B Output Plugin** ([see output documentation](/benthos-umh/output/sparkplug-b-output)):

* **Role**: Edge Node
* **Function**: Publishes UMH-Core data as Sparkplug B messages
* **Input**: Receives UMH-Core format from UNS

### Integration with UMH Unified Namespace

The Sparkplug B plugins integrate seamlessly with the UMH UNS architecture:

**Data Ingestion Flow**:

External Sparkplug B Systems → Sparkplug B Input Plugin (Host) → [tag\_processor](/benthos-umh/processing/tag-processor) → [UNS Output](/benthos-umh/output/uns-output) → UNS

**Data Publication Flow**: UNS → [UNS Input](/benthos-umh/input/uns-input) → UMH-Core Format → [Sparkplug B Output Plugin](/benthos-umh/output/sparkplug-b-output) (Edge Node) → External Systems

### Why Modified Parris Method Matters

Unlike the original Parris Method which creates separate state management per GroupID, UMH's approach enables unified state management across all organizational levels by preserving hierarchy in `device_id` and `metric_name` fields, allowing scalable multi-enterprise/multi-site data ingestion without state explosion.

## Quick Start

Most users should use this simple configuration to read Sparkplug B data:

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "DeviceLevelTest"
    # role: "secondary_passive" is default - safest for brownfield deployments

pipeline:
  processors:
    - tag_processor:
        defaults: |
          // ============================================================
          // AUTOMATIC CONVERSION (works for 95% of cases)
          // ============================================================
          // SparkplugB fields are auto-converted to UMH format:
          // • Separators: colons/slashes → dots (priority: colon > slash > dot)
          // • Device identifiers → location_path: "Plant:Area" → "Plant.Area"
          // • Metric name → tag_name: "sensors/temp/value" → "value"
          // • Metric path → virtual_path: "sensors/temp/value" → "sensors.temp"

          msg.meta.location_path = msg.meta.umh_location_path;  // Auto-converted location
          msg.meta.data_contract = "_historian";                 // Time-series storage
          msg.meta.tag_name = msg.meta.umh_tag_name;            // Extracted tag name
          msg.payload = msg.payload.value;                      // SparkplugB metric value
          msg.meta.timestamp_ms = msg.meta.spb_timestamp;       // Native SparkplugB timestamp

          // Only set virtual_path if present (Benthos cannot store empty strings)
          if (msg.meta.umh_virtual_path) {
            msg.meta.virtual_path = msg.meta.umh_virtual_path;
          }

          return msg;

output:
  uns: {}
```

This configuration reads Sparkplug B messages from the configured group and converts them to UMH-Core format. The default `secondary_passive` role is read-only and won't interfere with existing Sparkplug infrastructure, making it safe to run multiple instances for load balancing and redundancy.

> `identity.group_id` is also the subscription filter. By default the plugin subscribes to `spBv1.0/<group_id>/#`, so the example above ingests only the `DeviceLevelTest` group. To listen to multiple groups, use `subscription.groups: ["GroupA", "GroupB"]`. To subscribe to every group on the broker, use the MQTT wildcard: `subscription.groups: ["+"]`. The subscribed topics are printed at startup (for example, `Operating as secondary_passive - subscribing to: [spBv1.0/DeviceLevelTest/#]`).

**To publish data as Sparkplug B**: After processing in the UNS, use the [Sparkplug B Output plugin](/benthos-umh/output/sparkplug-b-output) to convert UMH-Core data back to Sparkplug B format for external systems.

### Sparkplug B to UMH-Core Mapping

Here's how a Sparkplug B message maps to UMH-Core using the Modified Parris Method:

**Input Sparkplug B Message:**

* **Topic**: `spBv1.0/FactoryA/DDATA/EdgeNode1/enterprise:factory:line1:station1`
* **Metric Name**: `sensors:ambient:temperature` (or `sensors/ambient/temperature` or `sensors.ambient.temperature`)
* **Payload**: Protobuf with metric alias, value 23.5, timestamp

**↓ Results in Structured JSON Message:**

**Payload:**

```json
{
  "name": "sensors:ambient:temperature",
  "alias": 42,
  "value": 23.5
}
```

**Metadata:**

```json
{
  "spb_group_id": "FactoryA",
  "spb_edge_node_id": "EdgeNode1",
  "spb_device_id": "enterprise:factory:line1:station1",
  "spb_metric_name": "sensors:ambient:temperature",
  "spb_message_type": "DDATA",
  "umh_location_path": "enterprise.factory.line1.station1",
  "umh_virtual_path": "sensors.ambient",
  "umh_tag_name": "temperature"
}
```

**Key Transformations:**

1. **Device ID to Location Path**: `enterprise:factory:line1:station1` → `location_path: "enterprise.factory.line1.station1"` (colons → dots)
2. **Metric Name Parsing**: Splits on last separator (priority: colon > slash > dot)
   * `sensors:ambient:temperature` → `virtual_path: "sensors.ambient"` + `tag_name: "temperature"`
   * `sensors/ambient/temperature` → `virtual_path: "sensors.ambient"` + `tag_name: "temperature"`
   * `sensors.ambient.temperature` → `virtual_path: "sensors.ambient"` + `tag_name: "temperature"`
3. **Sparkplug Protobuf**: Metric value and alias → Structured JSON format `{"name": "...", "alias": X, "value": Y}`
4. **Topic Components**: Group/EdgeNode from MQTT topic used for `spb_group_id` and `spb_edge_node_id` metadata

**Reverse Transformation**: The [Sparkplug B Output plugin](/benthos-umh/output/sparkplug-b-output) performs the inverse transformation to convert UMH-Core messages back to Sparkplug B format.

## Configuration Reference

### MQTT Section

| Field                       | Type       | Default                     | Description              |
| --------------------------- | ---------- | --------------------------- | ------------------------ |
| `mqtt.urls`                 | `[]string` | **required**                | List of MQTT broker URLs |
| `mqtt.client_id`            | `string`   | `"benthos-sparkplug-input"` | MQTT client identifier   |
| `mqtt.credentials.username` | `string`   | `""`                        | MQTT username            |
| `mqtt.credentials.password` | `string`   | `""`                        | MQTT password            |
| `mqtt.qos`                  | `int`      | `1`                         | MQTT QoS level           |
| `mqtt.keep_alive`           | `duration` | `"60s"`                     | MQTT keep alive interval |
| `mqtt.connect_timeout`      | `duration` | `"30s"`                     | Connection timeout       |
| `mqtt.clean_session`        | `bool`     | `true`                      | MQTT clean session flag  |

### Identity Section

| Field                   | Type     | Default      | Description                                                                                                                                                     |
| ----------------------- | -------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `identity.group_id`     | `string` | **required** | Sparkplug B Group ID. Also acts as the default subscription filter (`spBv1.0/<group_id>/#`). Override via `subscription.groups`.                                |
| `identity.edge_node_id` | `string` | `""`         | Required for the `primary` role, where it is used as the Sparkplug v3.0 `host_id` in the STATE topic (`spBv1.0/STATE/<host_id>`). Optional for secondary roles. |

### Role Section

| Field  | Type     | Default               | Description                                     |
| ------ | -------- | --------------------- | ----------------------------------------------- |
| `role` | `string` | `"secondary_passive"` | Operating role for the Sparkplug B input plugin |

**Available Roles:**

* `"secondary_passive"` (default): Read-only consumer, no rebirth commands sent
* `"secondary_active"`: Consumer that can request rebirths when needed
* `"primary"`: Full Primary Host with STATE publishing and session management

### Rebirth Configuration

| Field                      | Type       | Default | Description                                                                                                                                                                                                                                                                       |
| -------------------------- | ---------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `request_birth_on_connect` | `bool`     | `true`  | Send REBIRTH when DATA arrives from a node with no prior BIRTH on this bridge. Typical after a bridge restart. Ignored under `secondary_passive`. Controls only the discovery path; sequence-gap and unresolved-aliases recovery always run for `secondary_active` and `primary`. |
| `birth_request_throttle`   | `duration` | `"1s"`  | Minimum time between REBIRTH commands to the same node, shared across every rebirth reason. Collapses simultaneous discovery, sequence-gap, and unresolved-aliases signals into one broker command per window. Set to `0` to disable throttling.                                  |

### Hierarchy Section

| Field                           | Type   | Default | Description                                                                                                                                                               |
| ------------------------------- | ------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `include_edge_node_in_location` | `bool` | `false` | Nest device-level data under its Sparkplug edge node. When `true`, device data maps to `location_path = <edge_node>.<device>`; node-level data (no device) is unaffected. |

**When to enable.** The default decode treats `device_id` as the full location path (UMH's Modified Parris Method, where `enterprise:site:area` → `enterprise.site.area` and the edge node is only a session identity). That is correct for UMH-published streams, but a **native / brownfield** Sparkplug producer uses `Group / EdgeNode / Device` as genuine nested levels — the `device_id` is just a device name. With the default, such device data lands at the top of the hierarchy and identically-named devices on different edge nodes collide. Set `include_edge_node_in_location: true` to put each device under the edge node that owns it:

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "FactoryA"
    include_edge_node_in_location: true
```

| Topic                                 | `include_edge_node_in_location: false` (default) | `true`                |
| ------------------------------------- | ------------------------------------------------ | --------------------- |
| `spBv1.0/g/DDATA/Line1/IO Controller` | `IO_Controller`                                  | `Line1.IO_Controller` |
| `spBv1.0/g/DDATA/Line2/IO Controller` | `IO_Controller` (collides with Line1)            | `Line2.IO_Controller` |
| `spBv1.0/g/NDATA/Line1` (node-level)  | `Line1`                                          | `Line1` (unchanged)   |

Leave it `false` for Parris-encoded publishers, whose `device_id` already carries the full location path — enabling it there would prepend the edge node and corrupt the path.

> **No-rebuild workaround:** on the default template you can achieve the same nesting in the bridge's `tag_processor` by setting `location_path` from `spb_edge_node_id_sanitized` and `virtual_path` from `spb_device_id_sanitized`.

### Extension Decoding

Sparkplug B lets publishers carry custom data in proto2 extensions of two messages: `Payload.MetaData` (per-metric metadata) and `Payload.MetricValueExtension` (the metric value). The standard decode keeps those bytes but cannot name them, because they are not in the built-in schema. Supply the extension definitions and the plugin decodes them per metric.

| Field                          | Type     | Default | Description                                                                                     |
| ------------------------------ | -------- | ------- | ----------------------------------------------------------------------------------------------- |
| `decode_extensions.extensions` | `string` | `""`    | Inline proto2 schema declaring the extensions to decode. Empty (the default) disables decoding. |

Write only the `package` and `extend` blocks, plus any custom message types or well-known-type imports (for example `google/protobuf/timestamp.proto`). The plugin compiles the snippet as proto2 and adds the Sparkplug import, so do **not** write a `syntax` line or import the Sparkplug schema yourself. The extendees must use these fully-qualified names:

* `org.eclipse.tahu.protobuf.Payload.MetaData`
* `org.eclipse.tahu.protobuf.Payload.MetricValueExtension`

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "DeviceLevelTest"
    decode_extensions:
      extensions: |
        package example;
        extend org.eclipse.tahu.protobuf.Payload.MetaData {
          optional int64 extra_value = 9;
        }
```

For each metric that carries an extension, the plugin attaches:

* `spb_ext_<field>` — one metadata key per scalar extension (for example, `spb_ext_extra_value`), usable directly in a `tag_processor` with no protobuf handling.
* `spb_metric_decoded` — the full decoded metric as a JSON string, where extensions appear as `[package.field]` keys. Use it for message-typed extensions: `JSON.parse(msg.meta.spb_metric_decoded)`.

Metrics without an extension are emitted unchanged. The snippet compiles once at startup; an unparseable snippet, a snippet with no extensions, or two scalar extensions whose names map to the same `spb_ext_*` key fails the bridge at startup with the offending line. Extensions are a proto2-only feature, so the snippet must be proto2; the device's own implementation language is irrelevant, since the wire format is identical.

***

## Technical Details

### Operating Roles Explained

The Sparkplug B input plugin offers three operating roles that automatically configure all necessary settings. These roles provide clear choices for different deployment scenarios while eliminating the complexity of understanding Sparkplug B roles.

#### Role Overview

| Role                | Name                     | Description                             | Safe for Brownfield |
| ------------------- | ------------------------ | --------------------------------------- | ------------------- |
| `secondary_passive` | Secondary Host (Muted)   | Read-only consumer, no rebirth commands | ✅ Yes (Default)     |
| `secondary_active`  | Secondary Host (Unmuted) | Consumer that can request rebirths      | ✅ Yes               |
| `primary`           | Primary Host             | Full host with STATE publishing         | ⚠️ May conflict     |

#### Role: `secondary_passive` (Default)

**What it does:**

* Operates as a read-only Secondary Host
* **Does NOT send rebirth commands** (fully passive)
* Does NOT publish STATE messages
* Safe to run multiple instances for scalability
* Consumes Sparkplug B messages from the configured group without interfering (set `subscription.groups: ["+"]` to consume every group)

**When to use:**

* ✅ **Default choice** - Safest option for any deployment
* ✅ **Brownfield deployments** - Existing infrastructure remains undisturbed
* ✅ **Multi-consumer environments** - Prevents rebirth storms
* ✅ **Uncertain scenarios** - When you're not sure about the infrastructure

**Configuration:**

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "FactoryA"
    # role: "secondary_passive" is the default
```

**Technical behavior:** Pure consumer role with no command publishing capabilities.

#### Role: `secondary_active`

**What it does:**

* Operates as an active Secondary Host
* **Can send rebirth commands** when needed
* Does NOT publish STATE messages
* Can run multiple instances (though rebirth storms possible)
* Actively manages alias resolution through rebirth requests

**When to use:**

* ✅ **Single consumer deployments** - You're the only Sparkplug B consumer
* ✅ **Controlled environments** - You understand the rebirth implications
* ✅ **Active data management needed** - You need to request fresh BIRTH certificates

**Configuration:**

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "FactoryA"
    role: "secondary_active"
```

**Technical behavior:** Secondary Host with NCMD/DCMD publishing for rebirth requests.

#### Role: `primary`

**What it does:**

* Operates as the Primary Host per Sparkplug B specification
* Publishes STATE messages for Edge Node coordination
* Monitors sequence numbers and manages sessions
* Single instance only (multiple Primary Hosts conflict)
* Full control over Edge Node behavior

**When to use:**

* ✅ **Greenfield deployments** - UMH is your only Sparkplug B application
* ✅ **Full control needed** - You want Edge Nodes to buffer data when offline
* ✅ **Spec compliance required** - Strict Sparkplug B v3.0 compliance

**Configuration:**

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "FactoryA"
      edge_node_id: "UMH_Primary"  # Used as the Sparkplug v3.0 host_id
    role: "primary"
```

> For the `primary` role, `identity.edge_node_id` is the Sparkplug v3.0 `host_id`. With the config above, the plugin publishes STATE messages on `spBv1.0/STATE/UMH_Primary`. The startup logs make this explicit: `Primary Host: using identity.edge_node_id='UMH_Primary' as Sparkplug v3.0 host_id for STATE topic spBv1.0/STATE/UMH_Primary`. The field is named `edge_node_id` for consistency with the secondary roles, but in `primary` mode it identifies the host.

**Technical behavior:** Full Primary Host with STATE publishing and session management.

#### Understanding Rebirth Storms

**What is a rebirth storm?** A cascading effect that occurs when multiple Secondary Hosts simultaneously request rebirths from Edge Nodes, potentially overwhelming the MQTT infrastructure with redundant BIRTH messages.

**Scenario:**

1. Edge Node publishes DDATA with aliases
2. Multiple consumers don't have the alias mappings
3. All consumers simultaneously request rebirth
4. Edge Node publishes NBIRTH/DBIRTH for each request
5. Network and broker become saturated

**Prevention:**

* Use `secondary_passive` role (default) in multi-consumer environments
* Only use `secondary_active` when you're the sole consumer
* Coordinate rebirth requests if multiple active consumers are necessary

> **How throttling prevents rebirth storms.** This plugin rate-limits REBIRTH commands per node via `birth_request_throttle` (default `1s`). Without it, the alias-recovery path would loop until the next BIRTH arrives: every DATA in the round-trip window references the same uncached aliases and triggers another rebirth. Suppressed rebirths log at `info` when the throttle has been active longer than 100ms; same-dispatch co-fires (multiple reasons triggered by one DATA) log at `debug`. Throttling is an implementation choice, not Sparkplug B v3.0 behavior; the spec uses MAY for the Host's rebirth obligation and does not specify pacing. Set `birth_request_throttle: 0` to disable.

#### Choosing the Right Role

**Start with `secondary_passive` if:**

* You have any existing Sparkplug B infrastructure
* Multiple applications consume the same data
* You're unsure about the deployment environment
* You want the safest, most compatible option

**Consider `secondary_active` if:**

* You're the only Sparkplug B consumer
* You need to actively request BIRTH certificates
* You understand and can manage rebirth timing
* Network bandwidth isn't a concern

**Use `primary` only if:**

* You're building a greenfield Sparkplug B system
* No other Primary Hosts exist in your infrastructure
* You need Edge Nodes to respect your online/offline state
* You require full Sparkplug B specification compliance

### Advanced Configuration Options

For users who need to override the default subscription behavior, `subscription.groups` lets you listen to multiple groups or to every group on the broker:

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "FactoryA"
      edge_node_id: "CustomHost"  # Optional for secondary role
    role: "secondary_passive"

    # Override the default subscription filter
    subscription:
      groups: ["FactoryA", "FactoryB"]   # Listen to several groups
      # groups: ["+"]                      # Or: subscribe to every group (MQTT wildcard)

    # Future options (planned):
    # include_data_contract_in_device_id: true  # Add data contract to device ID
```

When `subscription.groups` is omitted or empty, the plugin filters to `identity.group_id` (`spBv1.0/<group_id>/#`). Set this field only when you need to listen to groups other than your identity, or to opt back into the all-groups behavior with `["+"]`.

### STATE Topic Behavior (primary role only)

When using `primary` role, the plugin publishes STATE messages according to Sparkplug B v3.0:

```
spBv1.0/STATE/<host_id>
```

* The `edge_node_id` is used as the `host_id`
* STATE topics do NOT include `group_id` (per specification)
* This allows Edge Nodes across all groups to detect the Primary Host

### Deployment Considerations

#### Multiple Instance Support

**`secondary_passive` role (default)**: ✅ **Safe for multiple instances** - No STATE conflicts, no rebirth storms, load balancing friendly

**`secondary_active` role**: ✅ **Can run multiple instances** - But be aware of potential rebirth storms

**`primary` role**: ⚠️ **Single instance only** - Publishes STATE messages for host arbitration

### Metadata Enrichment

The plugin attaches comprehensive Sparkplug-specific metadata fields to each output message. These are organized into **primary fields** (commonly used) and **secondary fields** (for advanced use cases):

#### Primary Metadata Fields

These are the main metadata fields that most users will need for processing Sparkplug messages:

* `spb_message_type`: The Sparkplug message type (e.g., "NBIRTH", "NDATA", "NDEATH", "DBIRTH", "DDATA", "DDEATH")
* `spb_group_id`: The Sparkplug Group ID of the source message
* `spb_edge_node_id`: The Edge Node ID (equipment or gateway name)
* `spb_device_id`: The Device ID (for metrics from devices under an edge node, empty for node-level messages)
* `spb_device_key`: Combined device identifier in format "group\_id/edge\_node\_id" or "group\_id/edge\_node\_id/device\_id"
* `spb_topic`: The original MQTT topic the message was received from
* `spb_metric_name`: The Sparkplug metric name (either from name field or alias\_X format)

#### Secondary Metadata Fields (Advanced)

These fields provide additional Sparkplug context and are primarily for debugging or advanced processing:

* `spb_group`: Same as `spb_group_id` (alternative field name)
* `spb_edge_node`: Same as `spb_edge_node_id` (alternative field name)
* `spb_device`: Same as `spb_device_id` (alternative field name)
* `spb_sequence`: The sequence number of the Sparkplug message
* `spb_bdseq`: The birth-death sequence number of the session
* `spb_timestamp`: The timestamp (in epoch ms) of the metric. The per-metric timestamp is used when present, falling back to the payload-level timestamp otherwise. This preserves each sample's own capture time — important for buffered or historical metrics that share one payload but were sampled at different moments.
* `spb_datatype`: The Sparkplug data type of the metric (e.g. "Int32", "Double", "Boolean")
* `spb_alias`: The alias number of the metric (for debugging alias resolution)
* `spb_is_historical`: Set to "true" if the metric was flagged as historical

#### Special Message Types

For STATE messages, the plugin sets:

* `event_type`: "state\_change"
* `node_state`: The state value ("ONLINE" or "OFFLINE")

For NDEATH/DDEATH messages, the plugin sets:

* `event_type`: "device\_offline"

#### UMH Conversion Metadata (Optional)

When UMH conversion is successful, additional metadata is added:

* `umh_conversion_status`: "success", "failed", "skipped\_insufficient\_data", or "failed\_no\_value"
* `umh_location_path`: Converted UMH location path (dots format, sanitized)
* `umh_tag_name`: UMH tag name extracted from metric name (sanitized)
* `umh_data_contract`: UMH data contract (e.g., "\_raw", "\_historian")
* `umh_virtual_path`: UMH virtual path if present in metric name (sanitized)
* `umh_conversion_error`: Error message if conversion failed

**Automatic Sanitization**: The plugin handles Sparkplug metric names to ensure UMH compatibility through a clear architectural boundary at the format conversion layer:

1. **Input Processing**: Sparkplug messages are received with their original metric names intact (no preprocessing)
2. **Format Conversion**: The format converter parses and sanitizes during the conversion to UMH format:
   * Splits metric names to extract virtual paths (priority: colon > slash > dot)
   * Example: `vpath:segment:metric` → virtual\_path=`vpath.segment`, tag\_name=`metric`
   * Trims leading/trailing separators before parsing
3. **Sanitization**: Applied only at the conversion boundary to preserve data integrity

Sanitization rules (applied during conversion):

* Hierarchy separators (`/`, `:`) are converted to dots (`.`) to preserve structure
* Invalid characters are replaced with underscores (`_`)
* Valid characters are: `a-z`, `A-Z`, `0-9`, `.`, `_`, `-`
* Multiple consecutive dots are collapsed into a single dot
* Leading and trailing dots are removed

Examples:

* `Refrigeration/Tower1/Pumps/chemHOA` → `Refrigeration.Tower1.Pumps.chemHOA`
* `Device@Name#123` → `Device_Name_123`
* `Area/Zone@1/Device#2` → `Area.Zone_1.Device_2`
* `Path//with///slashes` → `Path.with.slashes` (double dots prevented)
* `/hello123/test/` → virtual\_path=`hello123`, tag\_name=`test` (slashes trimmed, then split)
* `vpath:segment:metric` → virtual\_path=`vpath.segment`, tag\_name=`metric` (colons to dots)

When sanitization occurs, the plugin preserves the original values in metadata:

* `spb_original_metric_name`: The original metric name before sanitization (if different from spb\_metric\_name)
* `spb_original_device_id`: The original device ID before sanitization (if different from spb\_device\_id)

The sanitized values are available in the UMH metadata fields:

* `umh_location_path`: The sanitized location path (from device ID)
* `umh_virtual_path`: The sanitized virtual path (from metric name)
* `umh_tag_name`: The sanitized tag name (from metric name)

This ensures that messages with non-compliant Sparkplug names are automatically converted to valid UMH topics while preserving the original values for reference.

**Usage Recommendation**: Use the **primary metadata fields** for most processing logic. The alternative `spb_` prefixed fields are provided for consistency and advanced debugging scenarios.

### Message Processing

The Sparkplug B input plugin **always splits metrics** into individual messages to ensure UMH-Core format compliance. Each Sparkplug metric becomes a separate Benthos message for downstream processing.

*Note: This behavior is required for UMH-Core format and cannot be disabled.*

## Stateless Architecture Considerations

### Understanding bdSeq (Birth-Death Sequence) in Sparkplug B

The Sparkplug B input plugin processes **bdSeq** values from incoming Edge Node messages. Understanding bdSeq behavior is important for monitoring Edge Node session lifecycle:

**Specification-Compliant Edge Nodes:**

* bdSeq should increment by +1 for each new MQTT session
* Example: Session 1: bdSeq=0 → Session 2: bdSeq=1 → Session 3: bdSeq=2

**Stateless Edge Nodes (like Benthos Sparkplug B Output):**

* bdSeq may reset to 0 on Edge Node component restart
* This is common in container-based or stateless Edge Node implementations
* Still compliant within individual component lifecycles

### Impact on Input Processing

**What to Expect:**

* Edge Nodes may send bdSeq=0 after restarts (not necessarily the first session)
* bdSeq jumps or resets indicate Edge Node restarts or different implementations
* This is normal behavior for stateless architectures

**Recommendation:** The input plugin handles both persistent and stateless Edge Node bdSeq patterns correctly. No special configuration is needed - the plugin automatically adapts to different Edge Node implementations and their bdSeq behaviors.


# OPC UA (Input)

The plugin is designed to browse and subscribe to all child nodes within a folder for each configured NodeID, provided that the NodeID represents a folder. It features a recursion depth of up to 10 levels, enabling thorough exploration of nested folder structures. The browsing specifically targets nodes organized under the OPC UA 'Organizes' relationship type, intentionally excluding nodes under 'HasProperty' and 'HasComponent' relationships. Additionally, the plugin does not browse Objects represented by red, blue, or green cube icons in UAExpert.

Subscriptions are selectively managed, with tags having a DataType of null being excluded from subscription. Also, by default, the plugin does not subscribe to the properties of a tag, such as minimum and maximum values.

**Datatypes**

The plugin has been rigorously tested with an array of datatypes, both as single values and as arrays. The following datatypes have been verified for compatibility:

* `Boolean`
* `Byte`
* `DateTime`
* `Double`
* `Enumeration`
* `ExpandedNodeId`
* `Float`
* `Guid`
* `Int16`
* `Int32`
* `Int64`
* `Integer`
* `LocalizedText`
* `NodeId`
* `Number`
* `QualifiedName`
* `SByte`
* `StatusCode`
* `String`
* `UInt16`
* `UInt32`
* `UInt64`
* `UInteger`
* `ByteArray`
* `ByteString`
* `Duration`
* `LocaleId`
* `UtcTime`
* `Variant`
* `XmlElement`

There are specific datatypes which are currently not supported by the plugin and attempting to use them will result in errors. These include:

* Two-dimensional arrays
* UA Extension Objects
* Variant arrays (Arrays with multiple different datatypes)

**Authentication and Security**

In benthos-umh, we design security and authentication to be as robust as possible while maintaining flexibility. The software automates the process of selecting the highest level of security offered by an OPC-UA server for the selected Authentication Method, but the user can specify their own Security Policy / Security Mode if they want (see below at Configuration options)

**Supported Authentication Methods**

* **Anonymous**: No extra information is needed. The connection uses the highest security level available for anonymous connections.
* **Username and Password**: Specify the username and password in the configuration. The client opts for the highest security level that supports these credentials.
* **Certificate (Future Release)**: Certificate-based authentication is planned for future releases.

**Metadata outputs**

The plugin provides metadata for each message, that can be used to create a topic for the output, as shown in the example above. The metadata can also be used to create a unique identifier for each message, which is useful for deduplication.

| Metadata                 | Description                                                                                                                                          |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `opcua_tag_name`         | The sanitized ID of the Node that sent the message. This is always unique between nodes                                                              |
| `opcua_tag_path`         | A dot-separated path to the tag, created by joining the BrowseNames.                                                                                 |
| `opcua_tag_group`        | Other name for `opcua_tag_path`                                                                                                                      |
| `opcua_tag_type`         | The data type of the node optimized for benthos, which can be either a number, string or bool. For the original one, check out `opcua_attr_datatype` |
| `opcua_source_timestamp` | The SourceTimestamp of the OPC UA node                                                                                                               |
| `opcua_server_timestamp` | The ServerTimestamp of the OPC UA node                                                                                                               |
| `opcua_attr_nodeid`      | The NodeID attribute of the Node as a string                                                                                                         |
| `opcua_attr_nodeclass`   | The NodeClass attribute of the Node as a string                                                                                                      |
| `opcua_attr_browsename`  | The BrowseName attribute of the Node as a string                                                                                                     |
| `opcua_attr_description` | The Description attribute of the Node as a string                                                                                                    |
| `opcua_attr_accesslevel` | The AccessLevel attribute of the Node as a string                                                                                                    |
| `opcua_attr_datatype`    | The DataType attribute of the Node as a string                                                                                                       |
| `opcua_attr_statuscode`  | The OPC UA quality/status code for the data value (e.g., "Good", "BadNodeIdUnknown"). Indicates the reliability of the value.                        |

**Note:** String values are delivered exactly as the server sends them. However, `tag_processor` auto-detection converts numeric-looking strings (e.g., serial codes) back to lossy numbers. To keep them as strings, set `msg.meta.datatype = "string"` for those tags in the processing section.

Taking as example the following OPC-UA structure:

```
Root
└── ns=2;s=FolderNode
    ├── ns=2;s=Tag1
    ├── ns=2;s=Tag2
    └── ns=2;s=SubFolder
        ├── ns=2;s=Tag3
        └── ns=2;s=Tag4
```

Subscribing to `ns=2;s=FolderNode` would result in the following metadata:

| `opcua_tag_name` | `opcua_tag_group`      |
| ---------------- | ---------------------- |
| `Tag1`           | `FolderNode`           |
| `Tag2`           | `FolderNode`           |
| `Tag3`           | `FolderNode.SubFolder` |
| `Tag4`           | `FolderNode.SubFolder` |

**Configuration Options**

The following options can be specified in the `benthos.yaml` configuration file:

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
    username: 'your-username'  # optional (default: unset)
    password: 'your-password'  # optional (default: unset)
    insecure: false | true # DEPRECATED, see below
    securityMode: None | SignAndEncrypt # optional (default: unset)
    securityPolicy: None | Basic128Rsa15 | Basic256 | Basic256Sha256  # optional (default: unset)
    serverCertificateFingerprint: 'sha3-fingerprint-of-cert' # optional (default: unset)
    clientCertificate: 'your-fixed-base64-encoded-certificate' # optional (default: unset)
    userCertificate: 'base64-encoded-user-PEM-certificate' # optional (default: unset)
    userPrivateKey: 'base64-encoded-user-PEM-private-key' # optional (default: unset)
    subscribeEnabled: false | true # optional (default: false)
    useHeartbeat: false | true # optional (default: false)
    pollRate: 1000 # optional (default: 1000) The rate in milliseconds at which to poll the OPC UA server when not using subscriptions
    autoReconnect: false | true # optional (default: false)
    reconnectIntervalInSeconds: 5 # optional (default: 5) The rate in seconds at which to reconnect to the OPC UA server when the connection is lost
    # Advanced options - only modify if you understand your OPC UA server's behavior
    queueSize: 10 # optional (default: 10) Number of subscription notifications to buffer
    samplingInterval: 0.0 # optional (default: 0.0) Sampling interval in milliseconds for subscriptions
```

**Endpoint**

You can specify the endpoint in the configuration file. Node endpoints are automatically discovered and selected based on the authentication method.

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
```

**Node IDs**

You can specify the node IDs in the configuration file (currently only namespaced node IDs are supported):

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
```

**Username and Password**

If you want to use username and password authentication, you can specify them in the configuration file:

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
    username: 'your-username'
    password: 'your-password'
```

**User Certificate and Private Key**

* **Keys**: `userCertificate`, `userPrivateKey`
* **Description**: Credentials for User certificate-based authentication.
* `userCertificate`: Base64-encoded certificate in either PEM (.pem) or DER (.der) format.
* `userPrivateKey`: Base64-encoded private key in PEM (.pem) format only.
* Certificate-based authentication provides stronger security than username/password for high-security environments.
* Proper protection of the private key and certificate validation on both client and server are essential.
* **Configuration Example**:

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
    securityMode: SignAndEncrypt
    securityPolicy: Basic256Sha256
    userCertificate: 'base64-encoded certificate (.pem or .der)'
    userPrivateKey: 'base64-encoded private key (.pem only)'
```

**Security Options**

> To ensure a fully secure connection, you must explicitly configure all of the following security options. However, if these settings seem overwhelming, you can leave them unspecified. In that case, **benthos-umh** will automatically scan for and connect to available endpoints until it succeeds—and then it will log the recommended security settings for your future configuration.

OPC UA supports various security modes and security policies. These options define how messages are signed or encrypted and which cryptographic algorithms are used. In the configuration, you can specify the following:

* **Security Mode**: Defines the level of security applied to messages.
  * **Key**: `securityMode`
  * **Values**:
    * **None**: No security is applied; messages are neither signed nor encrypted.
    * **Sign**: Messages are signed for integrity and authenticity but not encrypted.
    * **SignAndEncrypt**: The highest level of security where messages are both signed and encrypted.
* **Security Policy**: Specifies the cryptographic algorithms used for signing/encrypting messages.
  * **Key**: `securityPolicy`
  * **Values**:
    * **None**: No security applied.
    * **Basic128Rsa15** (**deprecated**): Insecure due to SHA-1. Often disabled on servers by default.
    * **Basic256** (**deprecated**): Insecure due to SHA-1. Often disabled on servers by default.
    * **Basic256Sha256**: Recommended. Uses SHA-256 and provides stronger security.
* **Server Certificate Fingerprint**:
  * **Key**: `serverCertificateFingerprint`
  * **Description**: A SHA3-512 hash of the server’s certificate, used to verify you are connecting to the correct server.
  * If you specify this field, the client will verify that the server’s certificate matches the given fingerprint. If there’s a mismatch, the connection is rejected.
  * If omitted while **still using encryption** (`Sign` or `SignAndEncrypt`), the client will attempt to connect and then **log** the server’s actual fingerprint. You can copy that fingerprint into your config to be certain you’re connecting to the intended server.
  * In future releases, omitting the fingerprint may become a warning or block deployment in certain environments.
* **Client Certificate**:
  * **Key**: `clientCertificate`
  * **Description**: A Base64‐encoded PEM bundle (certificate + private key).
  * When using encryption (`Sign` or `SignAndEncrypt`), the client must present a certificate to the server. If you **do not** provide one, the system **auto‐generates** a random certificate at startup.
  * The auto‐generated certificate is logged in Base64 so you can copy/paste it into your configuration. This allows the server to trust the same client certificate across restarts instead of generating a new one each time.
  * Whenever a certificate is created, and the OPC UA server's settings do not allow automatic acceptance of client certificates, you will need to manually trust the client certificate in the server's settings. The client's name will be displayed, enabling you to uniquely identify it in the certificate list.

If you want to connect with security options, you will at least have to provide the following sample:

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
    securityMode: SignAndEncrypt
    securityPolicy: Basic256Sha256
    serverCertificateFingerprint: 'sha3-fingerprint-of-cert'
    clientCertificate: 'your-fixed-base64-encoded-certificate' # optional but recommended
```

**Insecure Mode**

This is now deprecated. By default, benthos-umh will now connect via SignAndEncrypt and Basic256Sha256 and if this fails it will fall back to insecure mode.

**Pull and Subscribe Methods**

Benthos-umh supports two modes of operation: pull and subscribe. In pull mode, it pulls all nodes every second, regardless of changes. In subscribe mode, it only sends data when there's a change in value, reducing unnecessary data transfer.

| Method    | Advantages                                                                                                                                                                                                                                         | Disadvantages                                                                                                                         |
| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| Pull      | <p>- Provides real-time data visibility, e.g., in MQTT Explorer.<br>- Clearly differentiates between 'no data received' and 'value did not change' scenarios, which can be crucial for documentation and proving the OPC-UA client's activity.</p> | - Results in higher data throughput as it pulls all nodes at the configured poll rate (default: every second), regardless of changes. |
| Subscribe | - Data is sent only when there's a change in value, reducing unnecessary data transfer.                                                                                                                                                            | - Less visibility into real-time data status, and it's harder to differentiate between no data and unchanged values.                  |

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
    subscribeEnabled: true
```

**UseHeartbeat**

If you are unsure if the OPC UA server is actually sending new data, you can enable `useHeartbeat` by setting it to true. It will automatically subscribe to the OPC UA server time, and will re-connect automatically if it does not receive an update within 10 seconds.

```yaml
input:
  opcua:
    useHeartbeat: true
```

**Browse Hierarchical References (Option until version 0.5.2)**

**NOTE**: This property is removed in version 0.6.0 and made as a standard way to browse OPCUA nodes. From version 0.6.0 onwards, opcua\_plugin will browse all nodes with Hierarchical References.

The plugin offers an option to browse OPCUA nodes by following Hierarchical References. By default, this feature is disabled (`false`), which means the plugin will only browse a limited subset of reference types, including:

* `HasComponent`
* `Organizes`
* `FolderType`
* `HasNotifier`

When set to `true`, the plugin will explore a broader range of node references. For a deeper understanding of the different reference types, refer to the [Standard References Type documentation](https://qiyuqi.gitbooks.io/opc-ua/content/Part3/Chapter7.html).

**Recommendation**: Enable this option (`browseHierarchicalReferences: true`) for more comprehensive node discovery.

```yaml
input:
  opcua:
    browseHierarchicalReferences: true
```

**Auto Reconnect**

If the connection is lost, the plugin will automatically reconnect to the OPC UA server. This is useful if the OPC UA server is unstable or if the network is unstable.

```yaml
input:
  opcua:
    autoReconnect: true
```

**Reconnect Interval**

The interval in seconds at which to reconnect to the OPC UA server when the connection is lost. This is only used if `autoReconnect` is set to true.

```yaml
input:
  opcua:
    reconnectIntervalInSeconds: 5
```

**Advanced Configuration Options**

> **WARNING**: The following options are for advanced users only. Modifying these settings without understanding your OPC UA server's behavior and limitations can lead to performance issues, memory problems, or connection failures. Setting those configuration options, does **not** mean that the server will respect that settings. **Leave these at their default values unless you have specific performance requirements and understand the implications.**

**Queue Size**

The `queueSize` parameter controls how many subscription notifications are buffered internally before being processed.

* **Key**: `queueSize`
* **Default**: `10`
* **Description**: This parameter determines the internal buffer size for handling subscription notifications from the OPC UA server. A larger queue can handle burst notifications but uses more memory.

**Risks of incorrect configuration**:

* **Too small**: May cause notification loss during high-frequency periods
* **Too large**: Excessive memory usage that could impact system performance

```yaml
input:
  opcua:
    queueSize: 20  # Only increase if you experience notification loss
```

**Sampling Interval**

The `samplingInterval` parameter controls how frequently the OPC UA server samples the underlying data source for subscription notifications.

* **Key**: `samplingInterval`
* **Default**: `0.0` (fastest possible sampling)
* **Unit**: Milliseconds
* **Description**: Defines the rate at which the server samples the data source. A value of `0.0` means "as fast as possible" according to the server's capabilities. Higher values reduce sampling frequency.

**Server behavior dependency**:

* Some servers ignore this parameter and use their own internal sampling rates
* The actual sampling rate depends on the server's implementation and capabilities

```yaml
input:
  opcua:
    samplingInterval: 1000.0  # Sample every 1 second instead of as fast as possible
```

## Server Profiles and Performance Tuning

Server profiles automatically optimize OPC UA connection parameters based on detected server type. The system queries the server's manufacturer and product information to select the best profile.

### What Are Server Profiles?

Profiles auto-optimize OPC UA connection parameters based on detected server type. The system queries the server's manufacturer and product information (ServerInfo nodes) to select appropriate tuning.

**Two performance parameters:**

1. **Workers (Browse Phase)**: Concurrent goroutines discovering the OPC UA node tree
   * Example: 10,000 nodes with 5 workers ≈ 2,000 Browse calls per worker vs 10,000 sequential calls
2. **Batch Size (Subscribe Phase)**: Nodes per CreateMonitoredItems call
   * Larger batches = faster subscription setup
   * Too large = server rejection or performance degradation

### Available Profiles

Profiles are automatically detected based on server manufacturer/product information:

* **Auto**: Unknown servers (conservative defaults)
* **High-Performance**: Manual override for known high-capacity infrastructure
* **Ignition**: Inductive Automation Ignition Gateway
* **Kepware**: PTC Kepware KEPServerEX
* **S7-1200**: Siemens S7-1200 PLCs (hardware limit: 1000 monitored items)
* **S7-1500**: Siemens S7-1500 PLCs (hardware limit: 10000 monitored items)
* **Prosys**: Prosys Simulation Server

**Profile values** are defined in [`opcua_plugin/server_profiles.go`](https://github.com/united-manufacturing-hub/benthos-umh/blob/main/opcua_plugin/server_profiles.go) with vendor documentation citations.

**Key insight**: Profile values are production-safe limits from vendor docs and real-world testing, NOT server-reported theoretical maximums. Example: S7-1200 reports `MaxMonitoredItemsPerCall=1000` but profile uses 100 (values >200 cause 50× performance degradation per [Siemens docs](https://cache.industry.siemens.com/dl/files/846/109755846/att_1163306/v4/109755846_TIA_Portal_OPC_UA_system_limits.pdf)).

### Performance Impact

Examples of profile-specific optimizations:

* **S7-1200**: Batch size limited to 100 (server reports 1000, but >200 causes 50× degradation - [Siemens docs](https://cache.industry.siemens.com/dl/files/846/109755846/att_1163306/v4/109755846_TIA_Portal_OPC_UA_system_limits.pdf))
* **Ignition/Kepware**: Batch size of 1000 enables 10× faster subscription setup compared to conservative 100. But limited browse worker amount as Eclipse Milo, the OPC UA library that Ignition uses, has a limit on concurrent operations per OPC UA session fo by default 64.
* **Prosys**: Batch size of 800 prevents simulation server unresponsiveness with large node counts

### Dynamic Worker Scaling

During the Browse phase, the system automatically adjusts worker concurrency based on measured server response time. This optimization balances performance (faster browsing with more workers) against server load (preventing overload).

**How it works:**

1. **Measurement**: The system samples response times from 5 consecutive Browse operations
2. **Target latency**: 250ms per Browse request (default)
3. **Scaling logic**:
   * If average response > 250ms → reduce workers by 1 (down to profile's MinWorkers)
   * If average response < 250ms → increase workers by 1 (up to profile's MaxWorkers)
   * If average response ≈ 250ms → no adjustment

**Bounds enforcement**: Worker count always respects the ServerProfile's MinWorkers and MaxWorkers limits. These limits are **global across ALL NodeIDs** being browsed - if you configure 3 NodeIDs with MaxWorkers=20, all 3 NodeIDs share a single pool of 20 workers (not 20 workers per NodeID = 60 total). The system cannot scale beyond profile-defined hardware constraints.

**Why this matters:**

* **Performance**: Automatically finds optimal worker count for each server's capability
* **Safety**: Prevents server overload by reducing workers when response times increase
* **Gradual adaptation**: Fine-grained control with ±1 worker adjustments provides smoother scaling
* **Adaptability**: Adjusts to changing server conditions during long Browse operations
* **Global resource management**: MaxWorkers controls total system concurrency regardless of NodeID count, preventing server overload when browsing multiple large folders simultaneously

Example: An S7-1500 profile with MaxWorkers=50 and **2 NodeIDs** might start with 10 workers **shared across both browse operations**. If Browse responses average 100ms (< 250ms target), the shared worker pool gradually increases to 50 maximum **for all browse operations combined**: 10 → 11 → 12 → 13. If responses slow to 400ms, workers reduce **across all active browse operations**: 13 → 12 → 11 to maintain stability.

### Manual Profile Override

Override auto-detection by setting profile explicitly:

```yaml
input:
  opcua:
    endpoint: "opc.tcp://10.0.0.1:4840"
    profile: "high-performance"  # Override auto-detection
```

Use case: Force high-performance profile when server manufacturer string doesn't match known vendors but infrastructure is validated for aggressive batching.

## Metrics

### opcua\_subscription\_failures\_total

Counter tracking OPC UA subscription failures by reason and node ID.

**Labels:**

* `reason`: Failure classification (`filter_not_allowed`, `filter_unsupported`, `node_id_unknown`, `node_id_invalid`, `other`)
* `node_id`: OPC UA NodeID that failed to subscribe

**Common Failure Reasons:**

| Reason               | Description                     | Resolution                                                     |
| -------------------- | ------------------------------- | -------------------------------------------------------------- |
| `filter_not_allowed` | Server rejected deadband filter | Only numeric types (Int, UInt, Float, Double) support deadband |
| `filter_unsupported` | Server doesn't support filters  | Disable deadband: `deadbandType: none`                         |
| `node_id_unknown`    | NodeID doesn't exist            | Verify NodeID configuration                                    |
| `node_id_invalid`    | NodeID syntax error             | Fix NodeID string format                                       |
| `other`              | Other subscription errors       | Check server logs                                              |


# Modbus

The Modbus plugin facilitates communication with various types of Modbus devices. It supports reading from four types of registers: coils, discrete inputs, holding registers, and input registers. Each data item configuration requires specifying the register type, address, and the data type to be read. The plugin supports multiple data types including integers, unsigned integers, floats, and strings across different sizes and formats.

Data reads can be configured to occur at a set interval, allowing for consistent data polling. Advanced features like register optimization and workarounds for device-specific quirks are also supported to enhance communication efficiency and compatibility.

**Metadata Outputs**

For each read operation, the plugin outputs detailed metadata that includes various aspects of the read operation, which can be utilized to effectively tag, organize, and utilize the data within a system. This metadata encompasses identifiers, data types, and register specifics to ensure precise tracking and utilization of the Modbus data.

Below is the extended metadata output schema provided by the plugin:

| Metadata                     | Description                                                                     |
| ---------------------------- | ------------------------------------------------------------------------------- |
| `modbus_tag_name`            | Sanitized tag name, with special characters removed for compatibility.          |
| `modbus_tag_name_original`   | Original tag name, as defined in the device configuration.                      |
| `modbus_tag_datatype`        | Original Modbus data type of the tag.                                           |
| `modbus_tag_datatype_json`   | Data type of the tag suitable for JSON representation: number, bool, or string. |
| `modbus_tag_address`         | String representation of the tag's Modbus address.                              |
| `modbus_tag_length`          | The length of the tag in registers, relevant for string or array data types.    |
| `modbus_tag_register`        | The specific Modbus register type where the tag is located.                     |
| `modbus_tag_slaveid`         | The slave ID where the tag is coming from                                       |
| `modbus_tag_unified_address` | Unified dotted address string (e.g. `temperature.holding.100.INT16`)            |

This enhanced metadata schema provides comprehensive data for each read operation, ensuring that users have all necessary details for effective data management and application integration.

**Configuration Options**

Below are the comprehensive configuration options available in the configuration file for the Modbus plugin. This includes settings for device connectivity, data reading intervals, optimization strategies, and detailed data item configurations.

```yaml
input:
  modbus:
    controller: 'tcp://localhost:502'
    transmissionMode: 'TCP'
    slaveIDs:
      - 1
    timeout: '1s'
    busyRetries: 3
    busyRetriesWait: '200ms'
    timeBetweenReads: '1s'
    optimization: 'none'
    byteOrder: 'ABCD'
    addresses:
      - name: "firstFlagOfDiscreteInput"
        register: "discrete"
        address: 1
        type: "BIT"
        output: "BOOL"
      - name: "zeroElementOfInputRegister"
        register: "input"
        address: 0
        type: "UINT16"
```

**Per-Slave Address Example**

When different slaves expose different registers, you can assign addresses to specific slaves:

```yaml
input:
  modbus:
    controller: 'tcp://192.168.1.100:502'
    slaveIDs: [1, 2, 3]
    addresses:
      - name: "temperature"
        register: "holding"
        address: 100
        type: "INT16"
        slaveID: 1          # Only slave 1
      - name: "pressure"
        register: "holding"
        address: 200
        type: "INT16"
        slaveID: 2          # Only slave 2
      - name: "status"
        register: "coil"
        address: 10
        type: "BIT"
        # No slaveID → read from ALL slaves (1, 2, 3)
```

**Controller**

Specifies the network address of the Modbus controller:

```yaml
input:
  modbus:
    controller: 'tcp://localhost:502'
```

**Transmission Mode**

Defines the Modbus transmission mode. Can be "TCP" (default), "RTUOverTCP", "ASCIIOverTCP":

```yaml
input:
  modbus:
    transmissionMode: 'TCP'
```

**Slave IDs**

Configure the modbus slave IDs :

```yaml
input:
  modbus:
    slaveIDs:
      - 1
      - 2
```

For backward compatibility, there is also `slaveID: 1`, which allows setting only a single Modbus slave.

**Retry Settings & Timeout**

Configurations to handle retries in case of communication failures:

```yaml
input:
  modbus:
    busyRetries: 3
    busyRetriesWait: '200ms'
    timeout: '1s'
```

**Time Between Reads**

Defines how frequently the Modbus device should be polled:

```yaml
input:
  modbus:
    timeBetweenReads: '1s'
```

**Optimization**

The Modbus plugin offers several strategies to optimize data read requests, enhancing efficiency and reducing network load when interacting with Modbus devices. These strategies are designed to adjust the organization and batching of requests based on device capabilities and network conditions.

The available optimization strategies are:

* **none**: This is the default setting where no optimization is applied. The plugin groups read requests according to the defined metrics without further optimization. Suitable for systems with simple setups or minimal performance requirements.
* **max\_insert**: Enhances efficiency by collating read requests across all defined metrics and filling in gaps (non-consecutive registers) to minimize the total number of requests. This strategy is ideal for complex systems with numerous data points, as it significantly reduces network traffic and processing time.
* **shrink**: Reduces the size of each request by stripping leading and trailing fields marked with an omit flag. This can decrease the overall data payload and improve processing times, especially when many fields are optional or conditional.
* **rearrange**: Allows rearranging fields between requests to reduce the number of registers accessed while maintaining the minimal number of requests. This strategy optimizes the order of fields to minimize the spread across registers.
* **aggressive**: Similar to "rearrange" but allows mixing of groups. This approach may reduce the number of requests at the cost of accessing more registers, potentially touching more data than necessary to consolidate requests.

Each strategy can be tailored with parameters such as `OptimizationMaxRegisterFill` to control how aggressively the system attempts to optimize data reads. For example, the `max_insert` option can be configured to limit the number of additional registers filled to reduce gaps:

```yaml
input:
  modbus:
    optimization: 'max_insert'
    optimizationMaxRegisterFill: 10
```

Additional Configuration for Optimization Strategies:

* **OptimizationMaxRegisterFill**: Specifies the maximum number of registers the optimizer is allowed to insert between non-consecutive registers in the `max_insert` strategy.

**Byte Order**

The `byteOrder` configuration specifies how bytes within the registers are ordered, which is essential for correctly interpreting the data read from a Modbus device. Different devices or systems may represent multi-byte data types (like integers and floating points) in various byte orders. The options are:

* **ABCD**: Big Endian (Motorola format) where the most significant byte is stored first.
* **DCBA**: Little Endian (Intel format) where the least significant byte is stored first.
* **BADC**: Big Endian with byte swap where bytes are stored in a big-endian order but each pair of bytes is swapped.
* **CDAB**: Little Endian with byte swap where bytes are stored in little-endian order with each pair of bytes swapped.

```yaml
input:
  modbus:
    byteOrder: 'ABCD'
```

**Modbus Workaround**

The Modbus plugin incorporates specific workarounds to address compatibility and performance issues that may arise with various Modbus devices. These workarounds ensure the plugin can operate efficiently even with devices that have unique quirks or non-standard Modbus implementations.

```yaml

input:
  modbus:
    workarounds:
      pauseAfterConnect: '500ms'
      oneRequestPerField: true
      readCoilsStartingAtZero: true
      timeBetweenRequests: '100ms'
      stringRegisterLocation: 'upper'
```

1. **Pause After Connect**

* **Description**: Introduces a delay before sending the first request after establishing a connection.
* **Purpose**: This is particularly useful for slow devices that need time to stabilize a connection before they can process requests.
* **Default**: `0s`
* **Configuration Example**:

  ```yaml
  pauseAfterConnect: '500ms'
  ```

2. **One Request Per Field**

* **Description**: Configures the system to send each field request separately.
* **Purpose**: Some devices may have limitations that prevent them from handling multiple field requests in a single Modbus transaction. Isolating requests ensures compatibility.
* **Default**: `false`
* **Configuration Example**:

  ```yaml
  oneRequestPerField: true
  ```

3. **Read Coils Starting at Zero**

* **Description**: Adjusts the starting address for reading coils to begin at address 0 instead of 1.
* **Purpose**: Certain devices may map their coil addresses starting from 0, which is non-standard but not uncommon.
* **Default**: `false`
* **Configuration Example**:

  ```yaml
  readCoilsStartingAtZero: true
  ```

4. **Time Between Requests**

* **Description**: Sets the minimum interval between consecutive requests to the same device.
* **Purpose**: Prevents the overloading of Modbus devices by spacing out the requests, which is critical in systems where devices are sensitive to high traffic.
* **Default**: `0s`
* **Configuration Example**:

  ```yaml
  timeBetweenRequests: '100ms'
  ```

5. **String Register Location**

* **Description**: Specifies which part of the register to use for string data after byte-order conversion.
* **Options**:
  * `lower`: Uses only the lower byte of each register.
  * `upper`: Uses only the upper byte of each register.
  * If left empty, both bytes of the register are used.
* **Purpose**: Some devices may place string data only in specific byte locations within a register, necessitating this adjustment for correct string interpretation.
* **Default**: Both bytes used.
* **Configuration Example**:

  ```yaml
  stringRegisterLocation: 'upper'
  ```

**Unified Address Format (`unifiedAddresses`)**

The preferred way to configure Modbus addresses is using the unified `unifiedAddresses` string list. Each address is a single string with the format:

```
name.register.address.type[:key=value]*
```

**Required positional segments** (dot-separated):

1. `name` — tag name (no dots allowed)
2. `register` — one of: `coil`, `discrete`, `holding`, `input`
3. `address` — numeric 0–65535
4. `type` — one of: `BIT`, `INT8L`, `INT8H`, `UINT8L`, `UINT8H`, `INT16`, `UINT16`, `INT32`, `UINT32`, `INT64`, `UINT64`, `FLOAT16`, `FLOAT32`, `FLOAT64`, `STRING`

**Optional key-value pairs** (colon-separated):

* `slaveID=<0-255>` — restrict to specific slave ID (default: 0 = all)
* `length=<n>` — register count, only valid for STRING type
* `bit=<0-15>` — bit number, only valid for BIT type
* `scale=<float>` — scaling factor
* `output=<INT64|UINT64|FLOAT64|STRING|BOOL|UINT16>` — output type

```yaml
input:
  modbus:
    controller: 'tcp://localhost:502'
    slaveIDs: [1]
    unifiedAddresses:
      - "temperature.holding.100.INT16"
      - "motor_status.discrete.1.BIT:bit=3"
      - "serial_number.holding.200.STRING:length=10"
      - "pressure.holding.300.FLOAT32:scale=0.1:output=FLOAT64:slaveID=2"
```

**Migration from `addresses` to `unifiedAddresses`**

| Old format                                                                              | New format                            |
| --------------------------------------------------------------------------------------- | ------------------------------------- |
| `name: "temp"`, `register: "holding"`, `address: 100`, `type: "INT16"`                  | `temp.holding.100.INT16`              |
| `name: "flag"`, `register: "discrete"`, `address: 1`, `type: "BIT"`, `bit: 3`           | `flag.discrete.1.BIT:bit=3`           |
| `name: "serial"`, `register: "holding"`, `address: 200`, `type: "STRING"`, `length: 10` | `serial.holding.200.STRING:length=10` |

> **Note**: The `unifiedAddresses` and `addresses` fields are mutually exclusive. Using both will result in an error.

**Addresses (Deprecated)**

> **Deprecated**: Use the `unifiedAddresses` string list format above instead.

The Modbus plugin provides a highly configurable way to specify which data points (addresses) to read from Modbus devices. Each address configuration allows precise definition of what data to read, how it's interpreted, and how it should be scaled or formatted before use.

```yaml
input:
  modbus:
    addresses:
      - name: "firstFlagOfDiscreteInput"
        register: "discrete"
        address: 1
        type: "BIT"
        output: "BOOL"
      - name: "zeroElementOfInputRegister"
        register: "input"
        address: 0
        type: "UINT16"
```

1. **Name**

* **Description**: Identifier for the data point being configured.
* **Configuration Example**:

  ```yaml
  name: "TemperatureSensor"
  ```

2. **Register**

* **Description**: Specifies the type of Modbus register to query. Options include "coil", "discrete", "holding", or "input".
* **Default**: "holding"
* **Configuration Example**:

  ```yaml
  register: "holding"
  ```

3. **Address**

* **Description**: The Modbus register address from which data should be read.
* **Configuration Example**:

  ```yaml
  address: 3
  ```

4. **Type**

* **Description**: Specifies the data type of the field, which determines how the data read from the register is interpreted. This setting is crucial as it affects how the raw data from Modbus registers is processed and used. The available data types cater to various data resolutions and formats, ranging from single-bit signals to full 64-bit precision, including special formats for strings and floating-point numbers.
* **Options**:
  * `BIT`: Single bit of a register.
  * `INT8L`: 8-bit integer (low byte).
  * `INT8H`: 8-bit integer (high byte).
  * `UINT8L`: 8-bit unsigned integer (low byte).
  * `UINT8H`: 8-bit unsigned integer (high byte).
  * `INT16`: 16-bit integer.
  * `UINT16`: 16-bit unsigned integer.
  * `INT32`: 32-bit integer.
  * `UINT32`: 32-bit unsigned integer.
  * `INT64`: 64-bit integer.
  * `UINT64`: 64-bit unsigned integer.
  * `FLOAT16`: 16-bit floating point (IEEE 754).
  * `FLOAT32`: 32-bit floating point (IEEE 754).
  * `FLOAT64`: 64-bit floating point (IEEE 754).
  * `STRING`: A sequence of bytes converted to a string.

5. **Length**

* **Description**: Number of registers to read, primarily used when the data type is "STRING".
* **Default**: 0
* **Configuration Example**:

  ```yaml
  length: 2
  ```

6. **Bit**

* **Description**: Relevant only for BIT data type, specifying which bit of the register to read.
* **Default**: 0
* **Configuration Example**:

  ```yaml
  bit: 7
  ```

7. **Scale**

* **Description**: A multiplier applied to the numeric data read from the register, used to scale values to the desired range or unit.
* **Default**: 0.0
* **Configuration Example**:

  ```yaml
  scale: 0.1
  ```

8. **Output**

* **Description**: Specifies the data type of the output field. Options include "INT64", "UINT64", "FLOAT64", or "native" (which retains the original data type without conversion).
* **Default**: Defaults to FLOAT64 if "scale" is provided and to the input "type" class otherwise (i.e. INT\* -> INT64, etc).
* **Configuration Example**:

  ```yaml
  output: "FLOAT64"
  ```

9. **Slave ID (per-address)**

* **Description**: Optionally restrict this address to a specific slave ID. When set, only the specified slave will read this address. When omitted or set to 0, all configured slaves (from the top-level `slaveIDs`) will read this address.
* **Default**: 0 (all slaves)
* **Validation**: If non-zero, the value must appear in the top-level `slaveIDs` list.
* **Configuration Example**:

  ```yaml
  slaveID: 2
  ```


# ifm IO-Link Master / "sensorconnect"

The SensorConnect plugin facilitates communication with ifm electronic’s IO-Link Masters devices, such as the AL1350 or AL1352 IO-Link Masters.\\

It also supports EIO404 Bluetooth mesh base stations with EIO344 Bluetooth mesh IO-Link adapters.\
It enables the integration of sensor data into Benthos pipelines by connecting to the device over HTTP and processing data from connected sensors, including digital inputs and IO-Link devices.\
The plugin handles parsing and interpreting IO-Link data using IODD files, converting raw sensor outputs into human-readable data.

It was previously known as [sensorconnect](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/golang/cmd/sensorconnect).

**Configuration**

Below is an example configuration demonstrating all available options for the sensorconnect plugin. This includes settings for device connectivity, IODD API URLs, and detailed device-specific configurations.

```yaml
input:
  sensorconnect:
    device_address: '192.168.0.1' # IP address of the IO-Link Master
    iodd_api: 'https://management.umh.app/iodd' # URL of the IODD API
    devices:
      - device_id: 1234
        vendor_id: 5678
        iodd_url: "https://example.com/iodd/device1234.xml"
      - device_id: 2345
        vendor_id: 6789
        iodd_url: "https://example.com/iodd/device2345.xml"
```

**Configuration Options**

**Device Address**

Specifies the IP address of the ifm IO-Link Master device

```yaml
input:
  sensorconnect:
    device_address: '192.168.0.1'
```

**IODD API**

Defines the URL of the IODD API, which is used to fetch IODD files for connected devices. Defaults to `https://management.umh.app/iodd` and should not be changed except for development purposes.

```yaml
input:
  sensorconnect:
    iodd_api: 'https://management.umh.app/iodd'
```

**Devices**

Provides a list of devices to provide for a given device\_id and vendor\_id, a fallback iodd\_url (in case the IODD file is not available via the IODD API).

```yaml
input:
  sensorconnect:
    devices:
      - device_id: 509 # Device ID of the IO-Link device
        vendor_id: 2035 # Vendor ID of the IO-Link device
        iodd_url: "https://yourserver.com/iodd/KEYENCE-FD-EPA1-20230410-IODD1.1.xml" # URL of the IODD file for the device. You might need to download this from the vendors website and self-host it.
```

**Output**

The payload of each message is a JSON object containing the sensor data, structured according to the data provided by the connected device. The exact structure of the payload depends on the specific sensors connected to the SensorConnect device and the data they provide.

Example for a VVB001 vibration sensor:

```json
{
  "Crest": 41,
  "Device status": 0,
  "OUT1": true,
  "OUT2": true,
  "Temperature": 394,
  "a-Peak": 2,
  "a-Rms": 0,
  "v-Rms": 0
}

```

**Metadata Outputs**

For each read operation, the plugin outputs detailed metadata that includes various aspects of the read operation, which can be utilized to effectively tag, organize, and utilize the data within a system.

Below is the extended metadata output schema provided by the plugin:

| Metadata                                 | Description                                                       |
| ---------------------------------------- | ----------------------------------------------------------------- |
| `sensorconnect_port_mode`                | The mode of the port, e.g., digital-input or io-link.             |
| `sensorconnect_port_number`              | The number of the port on the ifm IO-Link Master device.          |
| `sensorconnect_port_iolink_vendor_id`    | The IO-Link vendor ID of the connected device (if applicable).    |
| `sensorconnect_port_iolink_device_id`    | The IO-Link device ID of the connected device (if applicable).    |
| `sensorconnect_port_iolink_product_name` | The product name of the connected IO-Link device (if applicable). |
| `sensorconnect_port_iolink_serial`       | The serial number of the connected IO-Link device.                |
| `sensorconnect_device_product_code`      | The product code of the connected IO-Link device.                 |
| `sensorconnect_device_serial_number`     | The serial number of the connected IO-Link device                 |


# Beckhoff ADS (community)

Input for Beckhoff's ADS protocol. Supports batch reading and notifications. Beckhoff recommends limiting notifications to approximately 500 to avoid overloading the controller.\
This input only supports symbols and not direct addresses.

This plugin is community supported only. If you encounter any issues, check out the [original repository](https://github.com/RuneRoven/benthosADS) for more information, or ask around in our Discord.

## Minimal Example — TwinCAT 3, automatic route registration (Docker)

```yaml
input:
  ads:
    targetIP: '192.168.1.100'
    targetAMS: '192.168.1.100.1.1'
    runtimePort: 851
    hostAMS: 'auto'
    routeUsername: 'Administrator'
    routePassword: '1'
    routeHostAddress: '192.168.1.50'   # Docker host IP on PLC network
    symbols:
      - "MAIN.MyVariable"
      - "GVL_ProcessData.nCounter"
pipeline:
  processors:
    - tag_processor:
        defaults: |-
          msg.meta.location_path = "beckhoff.twincat.plc";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name      = msg.meta.symbol_name;
          return msg;
output:
  uns: {}
```

## Minimal Example — TwinCAT 3, static route (no auto-registration)

```yaml
input:
  ads:
    targetIP: '192.168.1.100'
    targetAMS: '192.168.1.100.1.1'
    runtimePort: 851
    hostAMS: '192.168.1.50.1.1'        # Must match static route on PLC
    symbols:
      - "MAIN.MyVariable"
      - "GVL_ProcessData.nCounter"
pipeline:
  processors:
    - tag_processor:
        defaults: |-
          msg.meta.location_path = "beckhoff.twincat.plc";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name      = msg.meta.symbol_name;
          return msg;
output:
  uns: {}
```

## Minimal Example — TwinCAT 2, automatic route registration (Docker)

```yaml
input:
  ads:
    targetIP: '192.168.1.100'
    targetAMS: '192.168.1.100.1.1'
    runtimePort: 801
    hostAMS: 'auto'
    routeUsername: 'Administrator'
    routePassword: '1'
    routeHostAddress: '192.168.1.50'   # Docker host IP on PLC network
    symbols:
      - ".myVariable"                  # TC2 global variable (dot prefix)
      - ".nCounter"
pipeline:
  processors:
    - tag_processor:
        defaults: |-
          msg.meta.location_path = "beckhoff.twincat.plc";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name      = msg.meta.symbol_name;
          return msg;
output:
  uns: {}
```

## Minimal Example — TwinCAT 2, static route (no auto-registration)

```yaml
input:
  ads:
    targetIP: '192.168.1.100'
    targetAMS: '192.168.1.100.1.1'
    runtimePort: 801
    hostAMS: '192.168.1.50.1.1'        # Must match static route on PLC
    symbols:
      - ".myVariable"                  # TC2 global variable (dot prefix)
      - ".nCounter"
pipeline:
  processors:
    - tag_processor:
        defaults: |-
          msg.meta.location_path = "beckhoff.twincat.plc";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name      = msg.meta.symbol_name;
          return msg;
output:
  uns: {}
```

## Full Config Reference

```yaml
input:
  ads:
    targetIP: 192.168.3.70            # Required
    targetAMS: 5.3.12.111.1.1         # Required
    targetPort: 48898                 # Optional, default: 48898
    runtimePort: 851                  # Optional, default: 801. Use 851 for TC3, 801 for TC2
    hostAMS: auto                     # Optional, default: auto
    hostPort: 10500                   # Optional, default: 10500
    readType: notification            # Optional, default: notification
    maxDelay: 100                     # Optional, default: 100
    cycleTime: 1000                   # Optional, default: 1000
    intervalTime: 1000                # Optional, default: 1000
    transmissionMode: serverOnChange  # Optional, default: serverOnChange
    loadSymbols: false                # Optional, default: false. Required for whole struct/array symbols
    logLevel: disabled                # Optional, default: disabled
    routeUsername: ""                 # Optional. If set, triggers automatic UDP route registration
    routePassword: ""                 # Optional
    routeHostAddress: ""              # Optional. Usually required in Docker bridge networking
    symbols:
      - MAIN.MYTRIGGER:0:10           # 0ms max delay, 10ms cycle time
      - MAIN.myInt                    # uses plugin-level defaults
      - ".superDuperInt"              # TC2 global variable (must start with `.`)
```

## Connection to ADS

When connecting to an ADS device you connect to a router which then routes the traffic to the correct device using the AMS net ID. There are basically 3 ways for setting up the connection:

1. **TwinCAT Connection Manager**: Use the TwinCAT connection manager locally on the host, scan for the device and add a connection using the correct credentials for the PLC.
2. **Static route on PLC**: Log in to the PLC using the TwinCAT system manager and add a static route from the PLC to the client. This is the preferred way when using benthos on a Kubernetes cluster since you have no good way of installing the connection manager.
3. **Automatic route registration (UDP)**: Use the `routeUsername` and `routePassword` config fields to have the plugin automatically register a route on the PLC before connecting. See the [Route Registration](#route-registration) section below.

### Docker and Kubernetes

ADS works from inside Docker containers with default bridge networking — **no `host_network`, no port forwarding, and no open ports are needed**. All ADS traffic (requests, responses, and notifications) flows over a single outbound TCP connection to port 48898. The PLC never initiates connections back to the client; it sends all responses and notifications on the same TCP socket the client opened.

The only requirement is that the `hostAMS` value matches a route registered on the PLC. When running in Docker with bridge networking:

* **`routeHostAddress` must be set** to the Docker host's IP on the PLC network (e.g. `192.168.1.50`). This tells the PLC which IP address to associate with the route. If left empty, it auto-detects the container's bridge IP which is not routable from the PLC.
* **`hostAMS` can be set explicitly** to `routeHostAddress` + `.1.1` (e.g. `192.168.1.50.1.1`), or left as `auto` — when route registration is configured with `routeHostAddress`, `auto` will correctly derive the AMS NetID from `routeHostAddress` instead of the container's bridge IP.
* **A route must exist on the PLC** for the `hostAMS` NetID. This can be added manually in TwinCAT System Manager, or automatically via the `routeUsername`/`routePassword` config fields.
* **`hostPort` is optional** (default 10500). It is a logical AMS port used in protocol headers, not a network port. Any value works.

**Option A: Automatic route registration (recommended)**

The plugin registers a route on the PLC automatically via UDP before connecting. No manual PLC configuration needed:

```yaml
input:
  ads:
    targetIP: '192.168.1.100'
    targetAMS: '192.168.1.100.1.1'
    runtimePort: 851
    hostAMS: 'auto'                      # Derives AMS NetID from routeHostAddress
    routeUsername: 'Administrator'        # Triggers automatic route registration
    routePassword: '1'
    routeHostAddress: '192.168.1.50'     # Docker HOST IP (required in bridge networking)
    readType: 'notification'
    symbols:
      - "MAIN.MyVariable"
```

You can also set `hostAMS` explicitly if you prefer:

```yaml
    hostAMS: '192.168.1.50.1.1'          # Explicit: Docker HOST IP + .1.1
    routeHostAddress: '192.168.1.50'     # Must match
```

**Option B: Static route on PLC**

If you prefer not to use automatic registration, add a static route on the PLC via TwinCAT System Manager pointing to the Docker host's IP. Then configure `hostAMS` to match — no `routeUsername`/`routePassword` needed:

```yaml
input:
  ads:
    targetIP: '192.168.1.100'
    targetAMS: '192.168.1.100.1.1'
    runtimePort: 851
    hostAMS: '192.168.1.50.1.1'         # Must match the route on the PLC
    readType: 'notification'
    symbols:
      - "MAIN.MyVariable"
```

**Option C: host\_network or macvlan**

When the container has a routable IP on the PLC network, `hostAMS: auto` works without `routeHostAddress`:

```yaml
input:
  ads:
    targetIP: '192.168.1.100'
    targetAMS: '192.168.1.100.1.1'
    runtimePort: 851
    hostAMS: 'auto'                     # Auto-derive from container's real IP
    routeUsername: 'Administrator'       # Optional: auto-register route
    routePassword: '1'
    readType: 'notification'
    symbols:
      - "MAIN.MyVariable"
```

#### Configuration Parameters

| Parameter            | Required | Default          | Description                                                                                                                                                                                                                                                                    |
| -------------------- | -------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **targetIP**         | Yes      | —                | IP address of the Beckhoff PLC                                                                                                                                                                                                                                                 |
| **targetAMS**        | Yes      | —                | AMS net ID of the target                                                                                                                                                                                                                                                       |
| **symbols**          | Yes      | —                | List of symbols to read from (see [Symbols Format](#symbols-format) below)                                                                                                                                                                                                     |
| **targetPort**       | No       | `48898`          | Port of the target internal gateway                                                                                                                                                                                                                                            |
| **runtimePort**      | No       | `801`            | Runtime port of PLC system, 800–899. TwinCAT 2 uses 800–850 (usually 801), TwinCAT 3 uses 851–899 (usually 851)                                                                                                                                                                |
| **hostAMS**          | No       | `auto`           | Host AMS net ID. Usually the IP address + `.1.1`. Must match a route on the PLC. `auto` derives it from `routeHostAddress` if set, otherwise from the outbound connection's local IP                                                                                           |
| **hostPort**         | No       | `10500`          | AMS source port used in protocol headers. This is a logical port, not a network port. Any arbitrary value works                                                                                                                                                                |
| **readType**         | No       | `notification`   | Read type for the symbols. `interval` polls at `intervalTime`; `notification` uses PLC push updates (see [Interval vs Notification](#interval-vs-notification))                                                                                                                |
| **maxDelay**         | No       | `100`            | Default max delay for sending notifications in ms. Maximum time after value change before PLC must send the notification                                                                                                                                                       |
| **cycleTime**        | No       | `1000`           | Default cycle time for notification handler in ms. How often the PLC scans for changes. Use a low value for triggers that are only true/false for 1 PLC cycle                                                                                                                  |
| **intervalTime**     | No       | `1000`           | Interval time between reads in ms (only used when `readType` is `interval`)                                                                                                                                                                                                    |
| **requestTimeout**   | No       | `5000`           | Timeout for individual ADS requests in ms. Increase for slow PLCs or large symbol tables                                                                                                                                                                                       |
| **transmissionMode** | No       | `serverOnChange` | Notification transmission mode. Only applies when `readType` is `notification`. Options: `serverOnChange`, `serverCycle`, `serverOnChange2`, `serverCycle2` (see [Transmission Modes](#transmission-modes))                                                                    |
| **loadSymbols**      | No       | `false`          | Download the full symbol and datatype table from the PLC on connect. Required for subscribing to whole struct or array symbols. May cause brief real-time jitter on the PLC during initial connection. See [Struct and Array Symbols](#struct-and-array-symbols)               |
| **logLevel**         | No       | `disabled`       | Log level for ADS connection (`disabled`, `error`, `warn`, `info`, `debug`, `trace`). At `debug`/`trace`, ADS error codes show human-readable descriptions                                                                                                                     |
| **routeUsername**    | No       | `""`             | Username for automatic UDP route registration on the PLC. If set, a route is registered before connecting (see [Route Registration](#route-registration))                                                                                                                      |
| **routePassword**    | No       | `""`             | Password for automatic UDP route registration on the PLC                                                                                                                                                                                                                       |
| **routeHostAddress** | No       | `""`             | IP address the PLC associates with the route. Required in Docker bridge networking (set to Docker host's IP). When `hostAMS` is `auto`, the AMS NetID is also derived from this. Auto-detected from outbound connection if empty (only correct with `host_network` or macvlan) |

**Symbols Format**

Symbols are specified as `name[:opt1[:opt2...]]`. Options are positional integers or `key=value` pairs.

| Format                   | maxDelay | cycleTime |
| ------------------------ | -------- | --------- |
| `MAIN.var`               | default  | default   |
| `MAIN.var:50:100`        | 50       | 100       |
| `MAIN.var:50`            | 50       | default   |
| `MAIN.var::100`          | default  | 100       |
| `MAIN.var:cycleTime=100` | default  | 100       |
| `MAIN.var:maxDelay=50`   | 50       | default   |

* `MAIN.MYBOOL` — variable in the main program, uses default maxDelay and cycleTime
* `MAIN.MYTRIGGER:0:10` — variable with 0ms max delay and 10ms cycle time
* `MAIN.MYTRIGGER::10` — default max delay, 10ms cycle time
* `.superDuperInt` — global variable (must start with `.`)

**TwinCAT 3** uses GVL-prefixed symbols: `GVL_ProcessData.nCounter`, `MAIN.MyVariable`

**TwinCAT 2** uses a flat namespace with dot prefix: `.nCounter`, `.myVariable`

**Struct and Array Symbols**

Two approaches for reading structured PLC data:

**Option A — Dot-notation (recommended, no extra config)**

Subscribe to individual primitive members using their full dot-path:

```yaml
symbols:
  - "MAIN.MachineStatus.Motor1.fSpeed"
  - "MAIN.MachineStatus.Motor1.bRunning"
  - "MAIN.MachineStatus.Pressure.fValue"
```

Each symbol fires independently on change and returns a primitive value. No `loadSymbols` needed.

**Option B — Whole struct subscription (requires `loadSymbols: true`)**

Subscribe to the struct symbol directly — returns a nested JSON object:

```yaml
input:
  ads:
    targetIP: "192.168.1.10"
    targetAMS: "5.1.2.3.1.1"
    loadSymbols: true
    symbols:
      - "MAIN.MachineStatus"
```

Output:

```json
{
  "Motor1": { "fSpeed": 1450.5, "bRunning": true },
  "Pressure": { "fValue": 3.2 }
}
```

|                        | Option A (dot-notation) | Option B (whole struct) |
| ---------------------- | ----------------------- | ----------------------- |
| `loadSymbols` required | No                      | Yes                     |
| Output per change      | One primitive per field | Whole struct as JSON    |
| PLC jitter risk        | None                    | Brief on connect        |

#### Transmission Modes

> **Note:** `transmissionMode` only applies when `readType` is `notification`. When using `readType: interval`, the plugin sends plain ADS Read commands to the PLC at each interval — no notification mechanism is involved, and `transmissionMode` is ignored.

The `transmissionMode` field controls how the PLC's internal notification handler sends updates back to the client. The available modes are:

| Mode              | Value | Description                                                                                                                                                                                                 |
| ----------------- | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `serverOnChange`  | 4     | **(Default)** The PLC scans for changes at the configured `cycleTime` interval and sends a notification only when the value has changed. This is the most efficient mode for most use cases.                |
| `serverCycle`     | 3     | The PLC sends the current value at every `cycleTime` interval, regardless of whether the value has changed. Useful when you need a constant data stream or heartbeat.                                       |
| `serverOnChange2` | 6     | Enhanced version of `serverOnChange` available on newer TwinCAT 3 firmware. Supports more efficient internal handling on the PLC side. **Automatically falls back** to `serverOnChange` on older PLCs.      |
| `serverCycle2`    | 5     | Enhanced version of `serverCycle` available on newer TwinCAT 3 firmware. Same behavior as `serverCycle` but with improved internal efficiency. **Automatically falls back** to `serverCycle` on older PLCs. |

**Choosing a mode:**

* Use `serverOnChange` (default) for event-driven data where you only care about changes
* Use `serverCycle` when you need periodic snapshots regardless of changes
* The `2` variants (`serverOnChange2`, `serverCycle2`) can be used safely on any PLC — the plugin automatically detects older PLCs and falls back to the v1 equivalent

```yaml
input:
  ads:
    transmissionMode: 'serverOnChange'   # default, sends only on value change
    # transmissionMode: 'serverCycle'    # sends at every cycle regardless of change
    # transmissionMode: 'serverOnChange2' # enhanced, auto-falls back on older PLCs
    # transmissionMode: 'serverCycle2'    # enhanced cyclic, auto-falls back on older PLCs
```

#### Interval vs Notification

The `interval` and `notification` read types can produce similar-looking results (periodic data), but they work differently under the hood:

* **`interval`**: The client polls the PLC — sends an ADS Read request for each symbol at every `intervalTime` interval. Simple, no PLC notification overhead, and not subject to the \~500-notification limit.
* **`notification` + `serverOnChange`**: The PLC pushes data only when a value changes. Most efficient for event-driven data. Subject to the \~500-notification limit per connection.
* **`notification` + `serverCycle`**: The PLC pushes data at every `cycleTime` interval regardless of changes. Similar result to `interval` but PLC-driven — more precise timing with no request/response overhead per cycle. Subject to the \~500-notification limit.

| Aspect                 | `interval`                       | `notification` + `serverOnChange`  | `notification` + `serverCycle` |
| ---------------------- | -------------------------------- | ---------------------------------- | ------------------------------ |
| Who drives             | Client polls                     | PLC pushes on change               | PLC pushes on timer            |
| Network per cycle      | Request + response               | Push only                          | Push only                      |
| Sends unchanged values | Yes                              | No                                 | Yes                            |
| Timing precision       | Subject to network latency       | PLC real-time task                 | PLC real-time task             |
| PLC notification limit | No limit                         | \~500 max                          | \~500 max                      |
| Best for               | Large symbol lists, simple setup | Event-driven data (most use cases) | Precise periodic sampling      |

#### Route Registration

The plugin can automatically register a route on the PLC using the Beckhoff UDP route protocol (port 48899). This removes the need to manually add routes in the TwinCAT System Manager.

**How it works:**

1. Before establishing the TCP connection, the plugin sends a UDP route registration packet to port 48899 on the PLC
2. The packet tells the PLC: "Associate AMS NetID X with IP address Y"
3. The PLC adds this as a runtime route (may not be visible in TwinCAT System Manager)
4. The normal ADS TCP connection is then established over port 48898

Setting `routeUsername` activates automatic route registration. If route registration fails (e.g. UDP response lost due to NAT), the plugin logs a warning and still attempts the TCP connection — the route may have been created despite the missing response.

**Parameters:**

* `routeUsername` / `routePassword`: PLC administrator credentials. Same as used in TwinCAT System Manager to add routes
* `routeHostAddress`: The IP address the PLC associates with this client. In Docker with bridge networking, this must be set to the Docker host's IP on the PLC network. When `hostAMS` is `auto`, the AMS NetID is derived from this address. Auto-detected if empty (only correct with `host_network` or macvlan)

**Network requirements:**

* UDP port 48899 must be reachable on the PLC from the client (for route registration)
* TCP port 48898 must be reachable on the PLC from the client (outbound — works through any NAT)

#### Reconnection

The plugin automatically reconnects when the TCP connection is lost (e.g. network cable unplugged, PLC restart). Aggressive TCP keepalive probes detect dead connections within \~13 seconds. On reconnect, the plugin:

1. Re-establishes the TCP connection (retries indefinitely with 5s interval)
2. Reloads the symbol table from the PLC
3. Re-subscribes all notification handles

No manual intervention is needed.

#### Output

This outputs for each address a single message with the payload being the value that was read. To distinguish messages, you can use meta("symbol\_name") in a following benthos bloblang processor.

## Testing

Tested and verified:

### CX7000, TwinCAT 3

* Notifications from Docker container with bridge networking (no host\_network)
* Automatic UDP route registration from Docker bridge networking
* Static route with explicit hostAMS (no route registration)
* Reconnection after network loss with automatic notification re-subscribe
* Sum/batch commands for read, add notification, and delete notification

### CX1020, TwinCAT 2

* Read batches, Add notifications, different cycle times and max delay
* Different datatypes, INT, INT16, UINT, DINT, BOOL, STRUCT, and more
* Automatic fallback from sum commands to individual calls
* Automatic fallback from v2 transmission modes to v1
* Reconnection after network loss with automatic notification re-subscribe


# Siemens S7

This input is tailored for the S7 communication protocol, facilitating a direct connection with S7-300, S7-400, S7-1200, and S7-1500 series PLCs.

For more modern PLCs like the S7-1200 and S7-1500 the following two changes need to be done to use them:

1. "Optimized block access" must be disabled for the DBs we want to access
2. In the "Protection" section of the CPU Properties, enable the "Permit access with PUT/GET" checkbox

**Configuration**

```yaml
input:
  s7comm:
    tcpDevice: '192.168.0.1' # IP address of the S7 PLC (optionally with port, e.g., '192.168.0.1:102')
    rack: 0                  # Rack number of the PLC. Defaults to 0
    slot: 1                  # Slot number of the PLC. Defaults to 1
    batchMaxSize: 480        # DEPRECATED: This field never worked correctly because batches were calculated before connecting to the PLC, so the actual negotiated PDU size was unknown. PDU size is now automatically negotiated during connection.
    timeout: 10              # Timeout in seconds for connections and requests. Default to 10
    disableCPUInfo: false    # Set this to true to not fetch CPU information from the PLC
    addresses:               # List of addresses to read from
      - "DB1.DW20"     # Accesses a double word at location 20 in data block 1
      - "DB1.S30.10"   # Accesses a 10-byte string at location 30 in data block 1
```

**Configuration Parameters**

* **tcpDevice**: IP address or hostname of the Siemens S7 PLC, optionally with port (e.g., `192.168.0.1:102`). If no port is specified, the default S7 port 102 is used.
* **rack**: Identifies the physical location of the CPU within the PLC rack.
* **slot**: Identifies the specific CPU slot within the rack.
* **timeout**: Timeout duration in seconds for connection attempts and read requests.
* **disableCPUInfo**: Set this to true to not fetch CPU information from the PLC. Should be used when you get the error 'Failed to get CPU information'
* **addresses**: List of PLC memory addresses to read. See [Address Format](#address-format) below.

## Address Format

Each address tells benthos-umh **where** to read in the PLC memory and **what data type** to expect.

**For Data Blocks** (block number required):

```
DB<number>.<type><offset>[.<extra>]
```

Example: `DB1.DW20` — double word at offset 20 in data block 1.

**For all other areas** (no block number):

```
<area>.<type><offset>[.<extra>]
```

Example: `PE.W0` — input word at offset 0.

Breaking this down:

| Part     | What it means                 | Example                                   |
| -------- | ----------------------------- | ----------------------------------------- |
| `area`   | Memory area in the PLC        | `DB`, `MK`, `PE`, `PA`, `C`, `T`          |
| `number` | Which data block (DB only)    | `1` in `DB1` (Data Block 1)               |
| `type`   | Data type to read             | `DW`, `X`, `S`, `R`, etc.                 |
| `offset` | Byte position within the area | `20` in `DB1.DW20` (starts at byte 20)    |
| `extra`  | Required for some types only  | Bit number for `X`, string length for `S` |

Only **DB** (Data Blocks) uses a block number.

### Memory Areas

| Area | Name           | Description                            | Example    |
| ---- | -------------- | -------------------------------------- | ---------- |
| `DB` | Data Block     | Main data storage — most commonly used | `DB1.DW20` |
| `PE` | Process Input  | Physical inputs (sensors, switches)    | `PE.B0`    |
| `PA` | Process Output | Physical outputs (actuators, relays)   | `PA.W0`    |
| `MK` | Merker (Flags) | Internal boolean/word flags            | `MK.W0`    |
| `C`  | Counter        | Hardware counters                      | `C.W0`     |
| `T`  | Timer          | Hardware timers                        | `T.W0`     |

### Data Types

| Type | Name           | Size     | Output Type                 | Extra Required?        | Example Address |
| ---- | -------------- | -------- | --------------------------- | ---------------------- | --------------- |
| `X`  | Bit            | 1 bit    | `bool`                      | Yes — bit number (0–7) | `DB1.X5.2`      |
| `B`  | Byte           | 1 byte   | `uint8`                     | No                     | `DB1.B10`       |
| `C`  | Char           | 1 byte   | `string` (single character) | No                     | `DB1.C10`       |
| `W`  | Word           | 2 bytes  | `uint16`                    | No                     | `DB1.W20`       |
| `I`  | Integer        | 2 bytes  | `int16` (signed)            | No                     | `DB1.I20`       |
| `DW` | Double Word    | 4 bytes  | `uint32`                    | No                     | `DB1.DW100`     |
| `DI` | Double Integer | 4 bytes  | `int32` (signed)            | No                     | `DB1.DI100`     |
| `R`  | Real           | 4 bytes  | `float32`                   | No                     | `DB1.R200`      |
| `DT` | Date/Time      | 8 bytes  | `int64` (Unix nanoseconds)  | No                     | `DB1.DT0`       |
| `S`  | String         | variable | `string`                    | Yes — max length (≥ 1) | `DB1.S30.10`    |

### The Extra Parameter

Two data types require the extra parameter (the part after the second `.`):

**Bit (`X`) — specify which bit (0–7) within the byte:**

```
DB1.X5.2
       │ └─ bit 2 (third bit, counting from 0)
       └─── byte offset 5
```

Bit numbering: `0` is the least significant bit, `7` is the most significant.

**String (`S`) — specify the maximum string length:**

```
DB1.S30.10
        │  └─ max 10 characters
        └──── byte offset 30
```

The PLC stores strings with a 2-byte header (max length + actual length), so `DB1.S30.10` reads 12 bytes starting at offset 30.

All other types must **not** have an extra parameter.

### Examples

```yaml
addresses:
  # Data Block reads
  - "DB1.X0.0"      # Bit 0 of byte 0 — a boolean flag
  - "DB1.DW8"       # Unsigned 32-bit double word at offset 8
  - "DB1.R16"       # 32-bit float at offset 16
  - "DB1.S28.50"    # String of up to 50 chars starting at offset 28

  # Process Inputs (PE) — reading from sensors, switches, etc.
  - "PE.X0.0"       # Input bit 0 — e.g., a digital sensor
  - "PE.X0.7"       # Input bit 7
  - "PE.B2"         # Input byte at offset 2
  - "PE.W4"         # Input word at offset 4 — e.g., an analog sensor value

  # Process Outputs (PA) — reading back output states
  - "PA.X0.0"       # Output bit 0 — e.g., a relay state
  - "PA.W0"         # Output word at offset 0

  # Merker / Flags (MK)
  - "MK.X0.0"       # Merker bit — internal boolean flag
  - "MK.W10"        # Merker word at offset 10
```

### Mapping from TIA Portal

When reading addresses from a TIA Portal project, map them like this:

| TIA Portal    | benthos-umh | Notes                                                   |
| ------------- | ----------- | ------------------------------------------------------- |
| `DB1.DBX 5.2` | `DB1.X5.2`  | Data bit — drop "DB" prefix from type                   |
| `DB1.DBB 10`  | `DB1.B10`   | Data byte                                               |
| `DB1.DBW 20`  | `DB1.W20`   | Data word                                               |
| `DB1.DBD 100` | `DB1.DW100` | Data double word (unsigned)                             |
| `DB1.DBD 100` | `DB1.DI100` | Data double word (signed) — same offset, different type |
| `DB1.DBD 200` | `DB1.R200`  | Data real (float)                                       |
| `M 0.0`       | `MK.X0.0`   | Merker bit                                              |
| `MW 10`       | `MK.W10`    | Merker word                                             |
| `I 0.0`       | `PE.X0.0`   | Input bit                                               |
| `IW 0`        | `PE.W0`     | Input word                                              |
| `Q 0.0`       | `PA.X0.0`   | Output bit                                              |
| `QW 0`        | `PA.W0`     | Output word                                             |

**Output**

Similar to the OPC UA input, this outputs for each address a single message with the payload being the value that was read. To distinguish messages, you can use meta("s7\_address") in a following benthos bloblang processor.

**Batching Behavior**

Addresses are automatically split into batches based on S7 protocol constraints:

* **Max 20 items per request**: The S7 protocol limits AGReadMulti to 20 addresses per request
* **PDU size**: The request and response must fit within the negotiated PDU size (typically 240-480 bytes depending on the PLC model)

When more than 20 addresses are configured or the combined data exceeds PDU limits, multiple sequential requests are made. This has implications:

* **Timing**: Addresses in different batches are read at slightly different times, resulting in different timestamps
* **Performance**: More batches means more round-trips to the PLC, increasing total read time


# Ethernet/IP

The plugin is designed to read data from configured tags within Rockwell Automation PLCs that support the CIP protocol over Ethernet/IP, such as **ControlLogix**, **CompactLogix**, and **Micro820**. It uses the [`gologix`](https://github.com/danomagnum/gologix) driver — a native Go implementation modeled after pylogix — to establish and maintain communication with the controller.

Currently, the plugin behaves similarly to the `s7comm` plugin: you must explicitly configure the **tag names** and **data types** you want to read. Automatic browsing or discovery of tags is **not yet implemented**. As such, you should already know the tag names and structures from the PLC program (e.g., via Studio 5000 or Connected Components Workbench).

> ⚠️ Support is limited to modern controllers that use **CIP over Ethernet/IP**. Legacy PLCs such as **PLC-5**, **SLC-500**, or **MicroLogix** models using **PCCC** are *not supported*.

In future versions, support for **browsing** and **listing available tags** directly from the controller may be added to improve usability and reduce manual configuration. **Datatypes**

The plugin is being tested with multiple datatypes, therefore the following datatypes are verified for compatibility:

* `Boolean`
* `Byte`
* `Int8`
* `Int16`
* `Int32`
* `Int64`
* `UInt16`
* `UInt32`
* `UInt64`
* `Float32`
* `Float64`
* `String`
* `Array of Byte`
* `Array of Int8`
* `Array of Int16`
* `Array of Int32`
* `Array of Int64`
* `Array of Uint16`
* `Array of Uint32`
* `Array of Uint64`
* `Array of Float32`
* `Array of Float64`
* `Array of String`

**Metadata outputs**

The plugin provides metadata for each message, that can be used to create a topic for the output, as shown in the example above. The metadata can also be used to create a unique identifier for each message, which is useful for deduplication.

| Metadata       | Description                                                                                                          |
| -------------- | -------------------------------------------------------------------------------------------------------------------- |
| `eip_tag_name` | The Name of the Tag that sent the message. If an alias was specified it will replace the original naming of the tag. |
| `eip_tag_path` | A dot-separated path to the tag, which is usually used for TagSets and Tags.                                         |
| `eip_tag_type` | The data type of the node optimized for benthos, which can be either a number, string or bool.                       |

**Configuration Options**

The following options can be specified in the `benthos.yaml` configuration file:

```yaml
input:
  ethernetip:
    endpoint: '127.0.0.1:44818'
    pollRate: 1000 # optional (default: 1000) The rate in milliseconds at which to poll the EthernetIP plc
    listAllTags: false | true # (currently not supported)
    useMultiRead: false | true # (currently not supported)
    attributes:
      - path: "1-1-1" # specify the path like `Class-Instance-Attribute`
        type: "int16"
        alias: "vendorID" # optional (default: unset)
    tags:
      - name: "tagName"
        type: "bool"
        alias: "tagAlias" # optional (default: unset)
        length: 1 # optional (default: 1) you only need to set this for arrays
```

**Endpoint**

You can specify the endpoint in the configuration file. Node endpoints are automatically discovered and selected based on the authentication method.

```yaml
input:
  ethernetip:
    endpoint: '127.0.0.1:44818'
    tags:
      - name: "tagName"
        type: "bool"
```

**Attributes**

You can specify the Attributes in the configuration file:

```yaml
input:
  ethernetip:
    endpoint: '127.0.0.1:44818'
    attributes:
      - path: "1-1-1" # specify the path like `Class-Instance-Attribute`
        type: "int16"
        alias: "vendorID" # optional (default: unset)
```

* **Key**: `path`
* **Description**: The `path` always consists of this pattern: "Class-Instance-Attribute". Path to the specific attribute you want to read. Attributes are usually used for some device specific information, therefore you have to check your device's manual to check whether a specific attribute exists.
* **Example**: `1-1-1`
* **Key**: `type`
* **Description**: The `type` specifies the attributes type, which has to be set correctly otherwise you will receive an error. You can also get this from your device's manual.
* **Example**: `int16`
* **Key**: `alias`
* **Description**: The `alias` is an option to store your data into a specific name, so you can later easilier access this attribute.
* **Example**: `testAlias`

**Tags**

You can specify the Tags in the configuration file:

```yaml
input:
  ethernetip:
    endpoint: '127.0.0.1:44818'
    tags:
      - name: "testInt16" # specify the path like `Class-Instance-Attribute`
        type: "int16"
        alias: "counter" # optional (default: unset)
```

* **Key**: `name`
* **Description**: The `name` is basically the name of the tag you want to read data from. This is specified in your device's software and you have to know the name of that.
* **Example**: `testInt16`
* **Key**: `type`
* **Description**: The `type` specifies the tags type, which has to be set correctly otherwise you will receive an error. You can also get this from your device's software.
* **Example**: `int16`
* **Key**: `alias`
* **Description**: The `alias` is an option to store your data into a specific name, so you can later easilier access this tags data.
* **Example**: `testAlias`
* **Key**: `length` (only if you read from a `type` e.g. `arrayofint16`)
* **Description**: The `length` is an option to set the length of the data you want to read. This is only needed when reading arrays. Otherwise you will receive an error here.
* **Example**: 3

**Important:** The functionality of reading tag-sets in specific is not yet implemented and will need further investigation.

**List all tags (not yet implemented)**

This is not yet implemented and currently not set to a specific timeline.

**MultiRead (not yet implemented)**

This is not yet implemented and currently not set to a specific timeline, but will improve reading time out of your plc if it supports this service.


# UNS Input

> **Works exclusively with UMH Core** Inside UMH Core you can leave the block empty — `uns: {}`. If you run the plugin elsewhere, at minimum set `broker_address`.

#### 1 Quick-start (99 % of users)

**Single topic pattern:**

```yaml
input:
  uns:                     # nothing else needed on UMH Core
    umh_topic: "umh\\.v1\\..+" # optional regex – here "take everything"
pipeline:
  processors:
    - tag_processor:       # enrich, rename, move to UMH data model
        defaults: |
          msg.meta.location_path = "enterprise.demo.plant1.line1.plc1";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name      = "value";
          return msg;
output:
  uns: {}                  # hand the curated data back to UMH Core
```

**Multiple topic patterns (preferred for multiple filters):**

```yaml
input:
  uns:
    umh_topics:            # list of regex patterns
      - "umh\\.v1\\.acme\\.berlin\\..+"
      - "umh\\.v1\\.acme\\.munich\\..+"
      - "umh\\.v1\\.acme\\.paris\\.production\\..+"
pipeline:
  processors:
    - tag_processor:       # same pipeline as above
        defaults: |
          msg.meta.location_path = "enterprise.demo.plant1.line1.plc1";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name      = "value";
          return msg;
output:
  uns: {}
```

| What               | Default inside UMH Core                                                                                                          |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------- |
| **Broker address** | `localhost:9092` (embedded Redpanda)                                                                                             |
| **Kafka topic**    | **`umh.messages`** (hard-wired)                                                                                                  |
| **Consumer group** | `uns_plugin`                                                                                                                     |
| **Filter regex**   | `.*` → receive all keys; override with `umh_topic:` (single pattern) or `umh_topics:` (multiple patterns) to limit what you pull |

#### 2 Optional overrides

**Single topic pattern:**

```yaml
input:
  uns:
    umh_topic:           "umh\\.v1\\.acme\\.berlin\\..*"   # only Berlin site
    broker_address:  "edge-redpanda:9092"
    consumer_group:  "analytics_reader"
```

**Multiple topic patterns:**

```yaml
input:
  uns:
    umh_topics:          # list of regex patterns
      - "umh\\.v1\\.acme\\.berlin\\..*"
      - "umh\\.v1\\.acme\\.munich\\..*"
    broker_address:  "edge-redpanda:9092"
    consumer_group:  "analytics_reader"
```

| Field            | Purpose & Default                                                                                                              |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `umh_topic`      | **Single regex pattern** against the Kafka key (UMH topic). Default `.*`. Cannot be used with `umh_topics`.                    |
| `umh_topics`     | **List of regex patterns** against the Kafka key (UMH topic). Preferred for multiple filters. Cannot be used with `umh_topic`. |
| `broker_address` | Comma-separated bootstrap list. Default `localhost:9092`.                                                                      |
| `consumer_group` | Consumer-group ID (offset tracking). Default `uns_plugin`.                                                                     |
| `topic`          | **\[DEPRECATED]** Use `umh_topic` instead. Still supported for backward compatibility.                                         |

#### 3 What the plugin does behind the scenes

1. **Poll** `umh.messages` continuously.
2. **Filter** records whose umh\_topic matches the topic regex (fast path if you only need a subset).
3. **Convert** each record into a Benthos message:

* Payload → Value (raw bytes).
* Metadata →
  * `kafka_timestamp_ms` - Kafka record timestamp in milliseconds (when the record was written to Kafka)
  * `kafka_msg_key` - Original Kafka message key (same as umh\_topic)
  * `kafka_topic` - Physical Kafka topic name (e.g., "umh.messages")
  * `umh_topic` - **UMH topic structure** from Kafka message key (e.g., "umh.v1.enterprise.plant1.\_historian.temperature")
  * Every Kafka header becomes `meta(...)` for processors downstream

4. **Batch-safe ACK** – commits offsets only when the whole Benthos batch succeeds.

**Performance notes**

* A very complex regular expression can cost CPU (back-tracking). Stick to anchored, specific patterns such as `umh\\.v1\\.acme\\.(berlin|munich)\\..*` rather than `.*temperature.*` if throughput matters.

#### 4 Typical end-to-end flow

```mermaid
graph LR
  A[Redpanda umh.messages] -->|uns input| B[Tag Processor]
  B -->|curated<br>topic & payload| C[uns output]
  C --> D[Redpanda umh.messages]
```

Use this pattern when you need to tap into the Unified Namespace, enrich / transform the stream, and publish the result back (or to another cluster).

#### 5 FAQs / Troubleshooting

* **"No messages appear"** – your regex under `umh_topic:` or `umh_topics:` filtered everything out. Start with `.*` (or `[".*"]` for umh\_topics), verify flow, then tighten the pattern.
* **"consumer group lag grows forever"** – your pipeline never ACKs. Check downstream processors / outputs for errors; the input only commits on ACK.


# Libraries

This section contains documentation for core libraries and utilities used across UMH components.

These libraries provide shared functionality that is used by multiple plugins and components throughout the system.

## Available Libraries

* [UMH Topic Parser](/benthos-umh/libraries/umh-topic-parser) - High-performance parsing, validation, and construction utilities for UMH topics


# UMH Topic Parser

The UMH Topic Parser is a high-performance Go package that provides parsing, validation, and construction utilities for UMH (Unified Manufacturing Hub) topics. It ensures that all topics conform to the UMH specification and can be safely used as Kafka message keys.

## Overview

UMH topics follow a strict hierarchical structure that represents the physical and logical organization of manufacturing systems. This package centralizes topic handling across all UMH components, providing consistent validation and preventing bad data from entering the UNS.

### Topic Structure

UMH topics follow this format:

```
umh.v1.<location_path>.<data_contract>[.<virtual_path>].<name>
```

**Components:**

* **location\_path**: 1-N hierarchical levels representing physical organization (enterprise.site.area.line...)
* **data\_contract**: Service/schema identifier starting with underscore (\_historian, \_analytics, etc.)
* **virtual\_path**: Optional logical grouping (axis.x.position, diagnostics.\*, etc.)
* **name**: Final identifier for the specific data point

### Validation Rules

The package enforces these fixed rules:

* **Location path levels**: MUST NOT start with underscore, at least 1 level required
* **Data contract**: MUST start with underscore, cannot be just "\_"
* **Virtual path**: CAN start with underscore, optional
* **Name**: CAN start with underscore, required
* **Kafka compatibility**: Only `[a-zA-Z0-9._-]` characters allowed
* **No consecutive dots, leading dots, or trailing dots**

## Examples

### Valid Topics

```
umh.v1.enterprise._historian.temperature
umh.v1.acme.berlin._historian.pressure
umh.v1.factory.line1.station2._raw.motor.diagnostics.vibration
umh.v1.plant._analytics.efficiency._kpi.oee
```

### Invalid Topics

```
umh.v1._enterprise._historian.temp     // location cannot start with _
umh.v1.factory.historian.temp          // data contract must start with _
umh.v1.factory._._historian.temp       // data contract cannot be just _
umh.v1.factory.._historian.temp        // empty location level
umh.v1.factory._historian.temp@ture    // invalid characters
```

## Usage

### Basic Parsing and Validation

```go
import "github.com/united-manufacturing-hub/benthos-umh/pkg/umh/topic"

// Parse and validate a topic string
uns, err := topic.NewUnsTopic("umh.v1.enterprise.site._historian.temperature")
if err != nil {
    log.Fatal(err)
}

// Use the topic as a Kafka key
kafkaKey := uns.AsKafkaKey()

// Access parsed components
info := uns.Info()
fmt.Println("Level0:", info.Level0)                          // "enterprise"
fmt.Println("Location Sublevels:", info.LocationSublevels)   // ["site"]
fmt.Println("Data Contract:", info.DataContract)             // "_historian"
fmt.Println("Name:", info.Name)                              // "temperature"
```

### Programmatic Construction

```go
// Build a simple topic
topic, err := topic.NewBuilder().
    SetLevel0("enterprise").
    SetDataContract("_historian").
    SetName("temperature").
    Build()

// Build a complex topic with all components
complexTopic, err := topic.NewBuilder().
    SetLocationLevels("enterprise", "site", "area", "line").
    SetDataContract("_historian").
    SetVirtualPath("motor.axis.x").
    SetName("position").
    Build()
```

### High-Performance Scenarios

For high-throughput scenarios, reuse builder instances:

```go
builder := topic.NewBuilder()

for _, sensor := range sensors {
    topic, err := builder.Reset().
        SetLocationPath(sensor.Location).
        SetDataContract("_historian").
        SetName(sensor.Name).
        Build()
    
    if err != nil {
        log.Printf("Invalid topic for sensor %s: %v", sensor.Name, err)
        continue
    }
    
    // Process topic...
}
```

### Location Path Handling

```go
// Different ways to set location paths
builder := topic.NewBuilder()

// Method 1: Dot-separated string
builder.SetLocationPath("enterprise.site.area")

// Method 2: Individual levels
builder.SetLocationLevels("enterprise", "site", "area")

// Method 3: Incremental building
builder.SetLevel0("enterprise").
    AddLocationLevel("site").
    AddLocationLevel("area")

// Method 4: Get current path from builder
currentPath := builder.GetLocationPath() // Returns: "enterprise.site.area"
```

## Performance Characteristics

The package is optimized for high-throughput scenarios:

* **Simple topic parsing**: \~656ns per operation
* **Complex topic parsing**: \~1322ns per operation
* **Simple topic construction**: \~751ns per operation
* **Complex topic construction**: \~1436ns per operation
* **Memory allocations**: 3-5 allocations per topic creation
* **Read operations**: <1ns (String(), Info(), AsKafkaKey() are essentially free)

### Thread Safety

* **UnsTopic instances**: Immutable and safe for concurrent use
* **Builder instances**: NOT thread-safe (use separate builders per goroutine)
* **Package-level functions**: Thread-safe

## Integration Patterns

### Message Processing

```go
func processMessage(topicStr string, payload []byte) error {
    topic, err := topic.NewUnsTopic(topicStr)
    if err != nil {
        return fmt.Errorf("invalid topic: %w", err)
    }

    // Use topic for routing, validation, etc.
    return routeMessage(topic, payload)
}
```

### Location-Based Routing

```go
func routeByLocation(topic *topic.UnsTopic) string {
    info := topic.Info()
    totalLevels := 1 + len(info.LocationSublevels) // level0 + sublevels
    switch totalLevels {
    case 1:
        return "enterprise-router"
    case 2:
        return "site-router"
    default:
        return "local-router"
    }
}
```

### Bulk Topic Generation

```go
func createTopics(sensors []Sensor) ([]*topic.UnsTopic, error) {
    builder := topic.NewBuilder()
    topics := make([]*topic.UnsTopic, 0, len(sensors))

    for _, sensor := range sensors {
        topic, err := builder.Reset().
            SetLocationPath(sensor.LocationPath).
            SetDataContract("_historian").
            SetName(sensor.Name).
            Build()
        if err != nil {
            return nil, err
        }
        topics = append(topics, topic)
    }

    return topics, nil
}
```

## Error Handling

The package provides detailed error messages for debugging:

```go
topic, err := topic.NewUnsTopic("umh.v1._enterprise._historian.temp")
if err != nil {
    fmt.Println(err) // "level0 cannot start with underscore"
}
```

Error messages include:

* Specific validation rule that failed
* Position information for parsing errors
* Suggestions for common mistakes

## Migration Guide

### From Manual String Parsing

**Before:**

```go
parts := strings.Split(topicStr, ".")
if len(parts) < 5 || !strings.HasPrefix(parts[0], "umh") {
    return errors.New("invalid topic")
}
// ... manual validation logic
level0 := parts[2]
dataContract := parts[3] // This might be wrong!
```

**After:**

```go
topic, err := topic.NewUnsTopic(topicStr)
if err != nil {
    return err
}
info := topic.Info()
level0 := info.Level0
dataContract := info.DataContract // Correctly parsed
```

### From Existing Validation Code

The topic parser replaces manual validation logic throughout the codebase:

1. **topic\_browser\_plugin**: Replace `topicToUNSInfo()` and manual parsing
2. **uns\_output**: Replace manual topic validation
3. **tag\_processor\_plugin**: Replace `constructUMHTopic()` with Builder
4. **classic\_to\_core\_plugin**: Replace manual topic parsing

## API Reference

### Core Types

```go
// UnsTopic represents a validated UMH topic
type UnsTopic struct { /* ... */ }

// TopicInfo contains parsed topic components (protobuf-generated)
// Located in pkg/umh/topic/proto package
type TopicInfo struct {
    Level0            string   // Enterprise level
    LocationSublevels []string // Additional location levels
    DataContract      string   // Data contract (_historian, etc.)
    VirtualPath       *string  // Optional virtual path (pointer, can be nil)
    Name              string   // Final name segment
    Metadata          map[string]string // Kafka header metadata
}

// Builder provides fluent topic construction
type Builder struct { /* ... */ }
```

### Key Methods

```go
// Parsing and validation
func NewUnsTopic(topic string) (*UnsTopic, error)

// Topic access
func (u *UnsTopic) String() string
func (u *UnsTopic) AsKafkaKey() string  
func (u *UnsTopic) Info() *proto.TopicInfo

// Field access (direct field access, not methods)
info.Level0                // string
info.LocationSublevels     // []string
info.DataContract          // string
info.VirtualPath           // *string (can be nil)
info.Name                  // string
info.Metadata              // map[string]string

// Builder methods
func NewBuilder() *Builder
func (b *Builder) SetLevel0(level0 string) *Builder
func (b *Builder) SetLocationPath(path string) *Builder
func (b *Builder) SetLocationLevels(level0 string, additional ...string) *Builder
func (b *Builder) AddLocationLevel(level string) *Builder
func (b *Builder) SetDataContract(contract string) *Builder
func (b *Builder) SetVirtualPath(path string) *Builder
func (b *Builder) SetName(name string) *Builder
func (b *Builder) Build() (*UnsTopic, error)
func (b *Builder) BuildString() (string, error)
func (b *Builder) GetLocationPath() string
func (b *Builder) Reset() *Builder
```

## Testing

The package includes comprehensive test coverage:

* **100+ test cases** covering valid and invalid topics
* **Performance benchmarks** for all operations
* **Concurrency tests** for thread safety
* **Edge case validation** for boundary conditions

Run tests:

```bash
cd pkg/umh/topic
go test -v
go test -bench=. -benchmem
```


# Processing

This section covers Benthos processors for data transformation and message handling. Processors allow you to modify, filter, and enrich messages as they flow through your data pipelines.

## Available Processors

* [**Tag Processor**](/benthos-umh/processing/tag-processor) - Optimized for processing tags and time series data, converting them to the UMH data model within the `_historian` data contract. Provides automatic message formatting, metadata generation, and structured processing stages.
* [**Classic to Core Processor**](/benthos-umh/processing/classic-to-core-processor) - Converts UMH Historian Data Contract format messages into Core format, following the "one tag, one message, one topic" principle.
* [**Downsampler**](/benthos-umh/processing/downsampler) - Reduces time-series data volume by filtering out insignificant changes using configurable algorithms. Integrates with UMH data pipelines to compress historian data while preserving significant trends.
* [**Topic Browser**](/benthos-umh/processing/topic-browser) - **\[Internal]** Transforms UMH messages into structured data for the Topic Browser interface. Extracts hierarchical topic information and event data to enable real-time topic exploration, metadata search, and debugging in the Management Console.
* [**Stream Processor**](/benthos-umh/processing/stream-processor) - Collects timeseries data from multiple UNS sources and generates transformed messages using JavaScript expressions. Implements dependency-based evaluation where static mappings are evaluated on every message while dynamic mappings only evaluate when their dependencies are received.
* [**Node-RED JavaScript Processor**](/benthos-umh/processing/node-red-javascript-processor) - Provides full control over payload and metadata through custom JavaScript code. Use this processor when you need complex transformations, conditional logic, or custom processing beyond standard tag handling.
* [**More Processors**](https://docs.redpanda.com/redpanda-connect/components/processors/about/) - Additional built-in processors available in Benthos/Redpanda Connect for various data processing needs.

## Which Processor to Choose?

* Use **Tag Processor** when working with structured time series data that needs to conform to the UMH data model
* Use **Classic to Core Processor** when migrating from legacy UMH Historian Data Contract format to modern Core format
* Use **Downsampler** after tag\_processor to reduce data volume while preserving important changes in time-series data
* Use **Topic Browser** (internal) when implementing systems that need to consume and display UMH topic hierarchies and event streams
* Use **Stream Processor** when you need to combine data from multiple UNS sources with state management, perform complex JavaScript transformations, and generate new UMH topics based on a model/version data contract
* Use **Node-RED JavaScript Processor** when you need maximum flexibility and custom processing logic
* Explore the **additional processors** for specific use cases like JSON manipulation, HTTP requests, caching, and more


# Tag Processor

The Tag Processor is designed to prepare incoming data for the UMH data model. It processes messages through three configurable stages: defaults, conditional transformations, and advanced processing, all using a Node-RED style JavaScript environment.

Use the `tag_processor` compared to the `nodered_js` when you are processing tags or time series data and converting them to the UMH data model within the `_historian` data contract. This processor is optimized for handling structured time series data, automatically formats messages, and generates appropriate metadata.

The JavaScript stages use the same environment as the `nodered_js` processor. For the full list of available globals (`msg`, `console`, `cache`), see the [JavaScript API Reference](/benthos-umh/processing/javascript-api).

**Message Formatting Behavior**

The processor automatically formats different input types into a consistent structure with a "value" field:

1. **Simple Values (numbers, strings, booleans)**\
   Input:

```json
42
```

Output:

```json
{
  "value": 42
}
```

Input:

```json
"test string"
```

Output:

```json
{
  "value": "test string"
}
```

Input:

```json
true
```

Output:

```json
{
  "value": true
}
```

2. **Arrays** (converted to JSON string representation)\
   Input:

```json
["a", "b", "c"]
```

Output:

```json
{
  "value": "[\"a\",\"b\",\"c\"]"
}
```

3. **Objects** (preserved as JSON objects)\
   Input:

```json
{
  "key1": "value1",
  "key2": 42
}
```

Output:

```json
{
  "value": "{\"key1\": \"value1\",\"key2\": 42}"
}
```

4. **Numbers** (preserved as numbers)\
   Input:

```json
23.5
```

Output:

```json
{
  "value": 23.5
}
```

Input:

```json
42
```

Output:

```json
{
  "value": 42
}
```

This consistent formatting ensures that:

* All messages have a "value" field
* Simple types (numbers, strings, booleans) are preserved as-is
* Complex types (arrays, objects) are converted to JSON string representations
* Numbers are always preserved as numeric types (integers or floats)
* Numeric-looking strings (e.g., `"2340925"`) stay strings and keep their exact digits; set `msg.meta.datatype = "number"` to convert them explicitly

> **Breaking change in v4.0.0**: Arrays now serialize to JSON format `["a","b","c"]` instead of space-separated format `[a b c]`. This preserves type information and enables array parsing in downstream processors.

**Configuration**

```yaml
pipeline:
  processors:
    - tag_processor:
        defaults: |

          // Set default location hierarchy and datacontract
          msg.meta.location_path = "enterprise.plant1.machiningArea.cnc-line.cnc5.plc123";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name = "value";
          msg.payload = msg.payload; //does not modify the payload
          return msg;
        conditions:
          - if: msg.meta.opcua_node_id === "ns=1;i=2245"
            then: |
              // Set path hierarchy and tag name for specific OPC UA node
              msg.meta.virtual_path = "axis.x.position";
              msg.meta.tag_name = "actual";
              return msg;
        advancedProcessing: |
          // Optional advanced message processing
          // Example: double numeric values
          msg.payload = parseFloat(msg.payload) * 2;
          return msg;
```

**Processing Stages**

1. **Defaults**
   * Sets initial metadata values
   * Runs first on every message
   * Must return a message object
2. **Conditions**
   * List of conditional transformations
   * Each condition has an `if` expression and a `then` code block
   * Runs after defaults
   * Must return a message object
3. **Advanced Processing**
   * Optional final processing stage
   * Can modify both metadata and payload
   * Must return a message object

**How Metadata Works**

The tag processor uses a two-step process for handling metadata:

1. **JavaScript Stage - Working with `msg.meta`**

   In your JavaScript code (defaults, conditions, advancedProcessing), you work with the `msg.meta` object:

   ```javascript
   // In JavaScript processing stages
   msg.meta.location_path = 'enterprise.site.area'
   msg.meta.data_contract = '_historian'
   msg.meta.tag_name = 'temperature'
   msg.meta.virtual_path = 'axis.x.position'
   ```

   At this stage, these are just JavaScript object properties - they're not yet actual Benthos metadata.
2. **Conversion to Benthos Metadata**

   After the JavaScript processing is complete, the tag processor extracts all properties from the `msg.meta` JavaScript object and converts them to actual Benthos message metadata.
3. **Final Message Structure**

   **Important**: Properties set on `msg.meta` become message metadata, **not** part of the payload.

   **Metadata** (accessible via message metadata):

   * `location_path`: "enterprise.site.area"
   * `data_contract`: "\_historian"
   * `tag_name`: "temperature"
   * `virtual_path`: "axis.x.position"
   * `umh_topic`: "umh.v1.enterprise.site.area.\_historian.axis.x.position.temperature" (auto-generated)
   * `topic`: "umh.v1.enterprise.site.area.\_historian.axis.x.position.temperature" (auto-generated, deprecated)

   **Payload** (the actual message content):

   ```json
   {
     "value": 23.5,
     "timestamp_ms": 1647890123456
   }
   ```

**Integration with UNS Output**

The tag processor is typically used together with the `uns` output plugin, which is the standard way to publish data into the United Manufacturing Hub. Here's how metadata flows from tag processor to UNS:

1. **Metadata to UNS Headers**: All metadata fields set by the tag processor become UNS headers, preserving the complete context and routing information.
2. **UMH Topic as Routing Key**: The auto-generated `umh_topic` metadata becomes the UNS routing key, enabling efficient message routing and topic-based subscriptions.
3. **Payload Preservation**: The structured payload (with `value` and `timestamp_ms`) is published as-is to the UNS.

**Complete Flow Example:**

```yaml
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.location_path = "enterprise.site.area";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name = "temperature";
          msg.meta.custom_field = "sensor_data";
          return msg;
  output:
    uns: {}
```

**Input Message**: `23.5`

**UNS Record Result**:

* **UNS Topic**: `umh.messages` (fixed UNS topic)
* **UNS Routing Key**: `umh.v1.enterprise.site.area._historian.temperature` (from `umh_topic` metadata)
* **UNS Value**: `{"value": 23.5, "timestamp_ms": 1647890123456}` (structured payload)
* **UNS Headers**:

  ```
  location_path: "enterprise.site.area"
  data_contract: "_historian"
  tag_name: "temperature"
  umh_topic: "umh.v1.enterprise.site.area._historian.temperature"
  topic: "umh.v1.enterprise.site.area._historian.temperature"
  custom_field: "sensor_data"
  bridged_by: "umh-core"
  ```

**Metadata Fields**

The processor uses the following metadata fields:

**Required Fields:**

* `location_path`: Hierarchical location path in dot notation (e.g., "enterprise.site.area.line.workcell.plc123")
* `data_contract`: Data schema identifier (e.g., "\_historian", "\_analytics")
* `tag_name`: Name of the tag/variable (e.g., "temperature", "pressure")

**Optional Fields:**

* `virtual_path`: Logical, non-physical grouping path in dot notation (e.g., "axis.x.position")
* `datatype`: Forces the output value type (`"string"`, `"number"`, or `"bool"`) instead of auto-detection. Set to `"number"` to convert a numeric string (e.g., `"42"`) into a number; without it, a string value always stays a string

**Generated Fields:**

* `umh_topic`: Automatically generated from the above fields in the format:

  ```
  umh.v1.<location_path>.<data_contract>.<virtual_path>.<tag_name>
  ```

  Empty or undefined fields are skipped, and dots are normalized.

**Message Structure**

Messages in the Tag Processor follow the Node-RED style format:

```javascript
{
  payload: {
    // The message content - can be a simple value or complex object
    "value": 23.5,
    "timestamp_ms": 1733903611000
  },
  meta: {
    // Required fields
    location_path: "enterprise.site.area.line.workcell.plc123",  // Hierarchical location path
    data_contract: "_historian",                                 // Data schema identifier
    tag_name: "temperature",                                     // Name of the tag/variable

    // Optional fields
    virtual_path: "axis.x.position",                            // Logical grouping path

    // Generated field (by processor)
    umh_topic: "umh.v1.enterprise.site.area.line.workcell.plc123._historian.axis.x.position.temperature",

    // Input-specific fields (e.g., from OPC UA)
    opcua_node_id: "ns=1;i=2245",
    opcua_tag_name: "temperature_sensor_1",
    opcua_tag_group: "sensors.temperature",
    opcua_tag_path: "sensors.temperature",
    opcua_tag_type: "number",
    opcua_source_timestamp: "2024-03-12T10:00:00Z",
    opcua_server_timestamp: "2024-03-12T10:00:00.001Z",
    opcua_attr_nodeid: "ns=1;i=2245",
    opcua_attr_nodeclass: "Variable",
    opcua_attr_browsename: "Temperature",
    opcua_attr_description: "Temperature Sensor 1",
    opcua_attr_accesslevel: "CurrentRead",
    opcua_attr_datatype: "Double"
  }
}
```

**Examples**

1. **Basic Defaults Processing**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise.plant1.machiningArea.cnc-line.cnc5.plc123";
    msg.meta.data_contract = "_historian";
    msg.meta.tag_name = "actual";
    return msg;
```

Input:

```json
23.5
```

Output:

```json
{
  "value": 23.5,
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise.plant1.machiningArea.cnc-line.cnc5.plc123._historian.actual`

2. **OPC UA Node ID Based Processing**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise.plant1.machiningArea.cnc-line.cnc5.plc123";
    msg.meta.data_contract = "_historian";
    return msg;
  conditions:
    - if: msg.meta.opcua_attr_nodeid === "ns=1;i=2245"
      then: |
        msg.meta.virtual_path = "axis.x.position";
        msg.meta.tag_name = "actual";
        return msg;
```

Input with metadata `opcua_attr_nodeid: "ns=1;i=2245"`:

```json
23.5
```

Output:

```json
{
  "value": 23.5,
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise.plant1.machiningArea.cnc-line.cnc5.plc123._historian.axis.x.position.actual`

3. **Moving Folder Structures in Virtual Path**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise.plant1";
    msg.meta.data_contract = "_historian";
    msg.meta.virtual_path = msg.meta.opcua_tag_path;
    msg.meta.tag_name = msg.meta.opcua_tag_name;
    return msg;
  conditions:
    # Move the entire DataAccess_AnalogType folder and its children into axis.x
    - if: msg.meta.opcua_tag_path && msg.meta.opcua_tag_path.includes("DataAccess_AnalogType")
      then: |
        msg.meta.location_path += ".area1.machining_line.cnc5.plc123";
        msg.meta.virtual_path = "axis.x." + msg.meta.opcua_tag_path;
        return msg;
```

Input messages with OPC UA tags:

```javascript
// Original tag paths from OPC UA:
// DataAccess_AnalogType
// DataAccess_AnalogType.EURange
// DataAccess_AnalogType.Min
// DataAccess_AnalogType.Max
```

Output UMH topics will be:

```
umh.v1.enterprise.plant1.area1.machining_line.cnc5.plc123._historian.axis.x.DataAccess_AnalogType
umh.v1.enterprise.plant1.area1.machining_line.cnc5.plc123._historian.axis.x.DataAccess_AnalogType.EURange
umh.v1.enterprise.plant1.area1.machining_line.cnc5.plc123._historian.axis.x.DataAccess_AnalogType.Min
umh.v1.enterprise.plant1.area1.machining_line.cnc5.plc123._historian.axis.x.DataAccess_AnalogType.Max
```

This example shows how to:

* Match an entire folder structure using `includes("DataAccess_AnalogType")`
* Move all matching nodes into a new virtual path prefix (`axis.x`)
* Preserve the original folder hierarchy under the new location
* Apply consistent location path for the entire folder structure

4. **Advanced Processing with getLastPayload**

getLastPayload is a function that returns the last payload of a message that was avaialble in Kafka. Remember that you will get the full payload, and might still need to extract the value you need.

**This is not yet implemented, but will be available in the future.**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise.site.area.line.workcell";
    msg.meta.data_contract = "_analytics";
    msg.meta.virtual_path = "work_order";
    return msg;
  advancedProcessing: |
    msg.payload = {
      "work_order_id": msg.payload.work_order_id,
      "work_order_start_time": umh.getLastPayload("enterprise.site.area.line.workcell._historian.workorder.work_order_start_time").work_order_start_time,
      "work_order_end_time": umh.getLastPayload("enterprise.site.area.line.workcell._historian.workorder.work_order_end_time").work_order_end_time
    };
    return msg;
```

Input:

```json
{
  "work_order_id": "WO123"
}
```

Output:

```json
{
  "value": "{\"work_order_id\":\"WO123\",\"work_order_start_time\":\"2024-03-12T10:00:00Z\",\"work_order_end_time\":\"2024-03-12T18:00:00Z\"}",
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise.site.area.line.workcell._analytics.work_order`

4. **Dropping Messages Based on Value**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise";
    msg.meta.data_contract = "_historian";
    msg.meta.tag_name = "temperature";
    return msg;
  advancedProcessing: |
    if (msg.payload < 0) {
      // Drop negative values
      return null;
    }
    return msg;
```

Input:

```json
-10
```

Output: Message is dropped (no output)

Input:

```json
10
```

Output:

```json
{
  "value": 10,
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise._historian.temperature`

5. **Duplicating Messages for Different Data Contracts**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise";
    msg.meta.data_contract = "_historian";
    msg.meta.tag_name = "temperature";
    return msg;
  conditions:
    - if: true
      then: |
        msg.meta.location_path += ".production";
        return msg;
  advancedProcessing: |
    // Create two versions of the message:
    // 1. Original value for historian
    // 2. Doubled value for custom
    let doubledValue = msg.payload * 2;

    msg1 = {
      payload: msg.payload,
      meta: { ...msg.meta, data_contract: "_historian" }
    };

    msg2 = {
      payload: doubledValue,
      meta: { ...msg.meta, data_contract: "_custom", tag_name: msg.meta.tag_name + "_doubled" }
    };

    return [msg1, msg2];
```

Input:

```json
23.5
```

Output 1 (Historian):

```json
{
  "value": 23.5,
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise.production._historian.temperature`

Output 2 (custom):

```json
{
  "value": 47,
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise.production._custom.temperature_doubled`

6. **Processing Full MQTT Message Payload**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise.area._workorder";
    msg.meta.data_contract = "_workorder";
    msg.meta.virtual_path = "new";
    msg.meta.tag_name = "maintenance";
    return msg;
```

Input:

```json
{
  "maintenanceSchedule": {
    "eventType": "ScheduledMaintenance",
    "eventId": "SM-20240717-025",
    "timestamp": "2024-07-17T13:00:00Z",
    "equipmentId": "InjectionMoldingMachine5",
    "equipmentName": "Engel Victory 120",
    "scheduledDate": "2024-07-22",
    "maintenanceType": "Preventive",
    "description": "Inspection and cleaning of injection unit and mold.",
    "maintenanceDuration": "6 hours",
    "assignedTo": {
      "employeeId": "EMP-5005",
      "name": "Hans Becker"
    },
    "status": "Scheduled",
    "partsRequired": [
      {
        "partId": "NOZZLE-015",
        "description": "Injection Nozzle",
        "quantity": 1
      }
    ],
    "notes": "Replace worn nozzle to prevent defects."
  }
}
```

Output:

```json
{
  "value": "{\"maintenanceSchedule\":{\"eventType\":\"ScheduledMaintenance\",\"eventId\":\"SM-20240717-025\",\"timestamp\":\"2024-07-17T13:00:00Z\",\"equipmentId\":\"InjectionMoldingMachine5\",\"equipmentName\":\"Engel Victory 120\",\"scheduledDate\":\"2024-07-22\",\"maintenanceType\":\"Preventive\",\"description\":\"Inspection and cleaning of injection unit and mold.\",\"maintenanceDuration\":\"6 hours\",\"assignedTo\":{\"employeeId\":\"EMP-5005\",\"name\":\"Hans Becker\"},\"status\":\"Scheduled\",\"partsRequired\":[{\"partId\":\"NOZZLE-015\",\"description\":\"Injection Nozzle\",\"quantity\":1}],\"notes\":\"Replace worn nozzle to prevent defects.\"}}",
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise.area._workorder.maintenance`

7. **Setting Custom Timestamps**

By default, the tag processor uses the current time. You can set custom timestamps using the `timestamp_ms` metadata field (Unix milliseconds or RFC3339Nano as string):

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise.site.area";
    msg.meta.data_contract = "_historian";
    msg.meta.tag_name = "temperature";

    // Use OPC UA timestamp either as Unix milliseconds
    if (msg.meta.opcua_source_timestamp) {
      msg.meta.timestamp_ms = new Date(msg.meta.opcua_source_timestamp).getTime().toString();
    }
    // or use it directly as RFC3339Nano
    if (msg.meta.opcua_source_timestamp) {
      msg.meta.timestamp_ms = msg.meta.opcua_source_timestamp;
    }
    // Or use Sparkplug B timestamp directly
    if (msg.meta.spb_timestamp) {
      msg.meta.timestamp_ms = msg.meta.spb_timestamp;
    }

    return msg;
```

**Note:** In the `tag_processor`, the resulting payload will always include `timestamp_ms` and one additional key corresponding to the `tag_name`. If you need to fully control the resulting payload structure, consider using the `nodered_js` processor instead. You can set the topic and payload manually, as shown below:

```yaml
pipeline:
  processors:
    - nodered_js:
        code: |
          // set kafka topic manually
          msg.meta.umh_topic = "umh.v1.enterprise.site.area._workorder.new"

          // only take two fields from the payload
          msg.payload = {
            "maintenanceSchedule": {
              "eventType": msg.payload.maintenanceSchedule.eventType,
              "description": msg.payload.maintenanceSchedule.description
            }
          }
          return msg;
```


# Classic to Core Processor

The `classic_to_core` processor converts UMH Historian schema format messages into Core format, following the "one tag, one message, one topic" principle. This processor is essential for migrating from Classic historian schemas to Core data architecture.

## Overview

The Classic to Core Processor transforms single messages containing multiple values and timestamps into separate Core format messages, each containing a single value. It handles nested tag groups by flattening them into intuitive dot-notation paths and reconstructs topics according to Core conventions.

## When to Use

Use the `classic_to_core` processor when you need to:

* **Process Classic Data in Core**: Convert Classic UMH Historian schema messages to Core format for processing in Core-based systems
* **Use Core Processors**: Enable the use of processors that rely on the Core data model (like downsampler)

## Quick Start

```yaml
pipeline:
  processors:
    - classic_to_core: {}
```

### Configuration Options

| Parameter              | Type   | Default | Description                                         |
| ---------------------- | ------ | ------- | --------------------------------------------------- |
| `target_data_contract` | string | `""`    | Target data contract. If empty, uses input's schema |

## Message Transformation

### Basic Conversion

The processor transforms single messages with multiple values into separate Core format messages:

**Input (Historian Schema):**

* Topic: `umh.v1.acme._historian.weather`
* Payload:

```json
{
  "timestamp_ms": 1717083000000,
  "temperature": 23.4,
  "humidity": 42.1
}
```

* target\_data\_contract: "\_raw"

**Output (Core Format):**

Message 1:

* Topic: `umh.v1.acme._raw.weather.temperature`
* Payload: `{"value": 23.4, "timestamp_ms": 1717083000000}`
* Metadata: `umh_topic: umh.v1.acme._raw.weather.temperature`

Message 2:

* Topic: `umh.v1.acme._raw.weather.humidity`
* Payload: `{"value": 42.1, "timestamp_ms": 1717083000000}`
* Metadata: `umh_topic: umh.v1.acme._raw.weather.humidity`

### Tag Groups (Nested Objects)

The processor flattens nested tag groups using dot notation:

**Input (Historian Schema with Tag Groups):**

* Topic: `umh.v1.acme._historian.cnc-mill`
* Payload:

```json
{
  "timestamp_ms": 1670001234567,
  "pos": {
    "x": 12.5,
    "y": 7.3,
    "z": 3.2
  },
  "temperature": 50.0,
  "collision": false
}
```

* target\_data\_contract: "\_raw"

**Output (Core Format with Flattened Tags):**

The processor creates 5 separate messages:

1. Topic: `umh.v1.acme._raw.cnc-mill.pos.x`
   * Payload: `{"value": 12.5, "timestamp_ms": 1670001234567}`
2. Topic: `umh.v1.acme._raw.cnc-mill.pos.y`
   * Payload: `{"value": 7.3, "timestamp_ms": 1670001234567}`
3. Topic: `umh.v1.acme._raw.cnc-mill.pos.z`
   * Payload: `{"value": 3.2, "timestamp_ms": 1670001234567}`
4. Topic: `umh.v1.acme._raw.cnc-mill.temperature`
   * Payload: `{"value": 50.0, "timestamp_ms": 1670001234567}`
5. Topic: `umh.v1.acme._raw.cnc-mill.collision`
   * Payload: `{"value": false, "timestamp_ms": 1670001234567}`

### Array Handling

Arrays are converted to string representation to ensure UMH-Core compliance (scalar values only):

**Input (Historian Schema with Arrays):**

* Topic: `umh.v1.factory._historian.sensors`
* Payload:

```json
{
  "timestamp_ms": 1670001234567,
  "measurements": [23.4, 24.1, 22.8],
  "sensor_ids": ["temp1", "temp2", "temp3"],
  "status_flags": [true, false, true],
  "mixed_data": ["sensor", 42, true, null]
}
```

* target\_data\_contract: "\_raw"

**Output (Core Format with Array Conversion):**

The processor creates 4 separate messages with arrays converted to strings:

1. Topic: `umh.v1.factory._raw.sensors.measurements`
   * Payload: `{"value": "[23.4 24.1 22.8]", "timestamp_ms": 1670001234567}`
2. Topic: `umh.v1.factory._raw.sensors.sensor_ids`
   * Payload: `{"value": "[temp1 temp2 temp3]", "timestamp_ms": 1670001234567}`
3. Topic: `umh.v1.factory._raw.sensors.status_flags`
   * Payload: `{"value": "[true false true]", "timestamp_ms": 1670001234567}`
4. Topic: `umh.v1.factory._raw.sensors.mixed_data`
   * Payload: `{"value": "[sensor 42 true <nil>]", "timestamp_ms": 1670001234567}`

**Array Conversion Rules:**

* Arrays are converted to space-separated string format: `[item1 item2 item3]`
* Empty arrays become `[]`
* Single-item arrays become `[item]`
* All array elements are converted using Go's `fmt.Sprintf("%v", item)` format
* Null values in arrays appear as `<nil>`
* Arrays within nested objects are also converted to strings

## Advanced Configuration

### Using Input Schema

When `target_data_contract` is not specified, the processor uses the input's schema:

```yaml
classic_to_core:
  # target_data_contract not specified - uses input's schema
```

**Input:**

* Topic: `umh.v1.acme._historian.weather`

**Output:**

* Topic: `umh.v1.acme._historian.weather.pressure`
* Maintains the original `_historian` schema

## Topic Transformation

The processor parses Classic topics and reconstructs them for Core format:

| Component     | Classic Example          | Core Example             | Description                    |
| ------------- | ------------------------ | ------------------------ | ------------------------------ |
| Prefix        | `umh.v1`                 | `umh.v1`                 | Unchanged                      |
| Location Path | `enterprise.plant1.area` | `enterprise.plant1.area` | Unchanged                      |
| Schema        | `_historian`             | `_raw` (configurable)    | Updated based on configuration |
| Context       | `weather`                | `weather`                | Becomes virtual\_path          |
| Field Name    | N/A                      | `temperature`            | Added for each field           |

**Example Transformation:**

* **Input Topic**: `umh.v1.enterprise.plant1.area._historian.weather`
* **Output Topic**: `umh.v1.enterprise.plant1.area._raw.weather.temperature`

## Metadata Handling

The processor sets the following metadata fields:

**Generated Metadata Fields:**

* `topic`: The new Core topic
* `umh_topic`: Same as topic (enables direct use with `uns_output`)
* `location_path`: Extracted from original topic
* `data_contract`: The target data contract (or input's schema if not specified)
* `tag_name`: The field name
* `virtual_path`: Original context (if present)

**Metadata Preservation:** Original metadata is always preserved alongside new fields.

## Performance & Reliability

### Built-in Safeguards

The processor includes several safeguards for production use:

* **Recursion Limit**: Maximum recursion depth of 10 levels for flattening nested tag groups
* **Message Size Limit**: Maximum of 1000 tags per message to prevent memory exhaustion
* **Topic Validation**: Strict UMH v1 topic format validation with proper error handling
* **Comprehensive Metrics**: Tracks processing counts, errors, and limit violations for monitoring

### Performance Considerations

* **Message Expansion**: Each input message creates N output messages (N = number of data fields)
* **Memory Usage**: Metadata is copied for each output message
* **Processing Overhead**: Minimal - efficient string parsing with optimized allocations

## Error Handling

The processor includes comprehensive error handling and logging for non-standard messages. Each error condition is logged with detailed information to help diagnose issues:

| Error Condition          | Log Message Example                                          | Behavior                      | Metrics               |
| ------------------------ | ------------------------------------------------------------ | ----------------------------- | --------------------- |
| Invalid JSON             | "failed to parse as structured data: ..."                    | Message skipped, error logged | `messages_errored`    |
| Missing timestamp\_ms    | "timestamp field 'timestamp\_ms' not found in payload"       | Message skipped, error logged | `messages_errored`    |
| Invalid timestamp format | "failed to parse timestamp: ..."                             | Message skipped, error logged | `messages_errored`    |
| Missing topic metadata   | "no topic found in message metadata"                         | Message skipped, error logged | `messages_errored`    |
| Invalid topic format     | "invalid topic structure, expected at least 4 parts: ..."    | Message skipped, error logged | `messages_errored`    |
| Invalid UMH prefix       | "invalid UMH topic prefix, expected 'umh.v1': ..."           | Message skipped, error logged | `messages_errored`    |
| Missing data contract    | "no data contract found in topic: ..."                       | Message skipped, error logged | `messages_errored`    |
| Missing location path    | "missing location path in topic: ..."                        | Message skipped, error logged | `messages_errored`    |
| Recursion depth exceeded | "Maximum recursion depth of 10 reached, stopping flattening" | Flattening stopped at limit   | `recursion_limit_hit` |
| Too many tags            | "Message exceeds maximum tag limit of 1000, truncating"      | Processing stopped at limit   | `tag_limit_exceeded`  |

All error messages include the original message content and specific details about what went wrong, making it easy to identify and fix issues with non-standard messages.

## Metrics

The processor exposes comprehensive metrics for monitoring:

| Metric                | Type    | Description                             |
| --------------------- | ------- | --------------------------------------- |
| `messages_processed`  | Counter | Total input messages processed          |
| `messages_errored`    | Counter | Messages that failed processing         |
| `messages_expanded`   | Counter | Total output messages created           |
| `messages_dropped`    | Counter | Messages dropped due to configuration   |
| `recursion_limit_hit` | Counter | Times recursion depth limit was reached |
| `tag_limit_exceeded`  | Counter | Times tag limit was exceeded            |

These metrics can be used to monitor the health of the processor and identify patterns of non-standard messages that need attention.

## Complete Integration Example

Here's a complete Benthos configuration for migrating from Classic to Core format:

```yaml
input:
  kafka:
    addresses: ['kafka:9092']
    topics: ['umh.v1.+.+.+._historian.+']
    consumer_group: historian-to-core-migration

pipeline:
  processors:
    - classic_to_core: {}

output:
  # Use UNS output for seamless integration
  uns_output: {}
```

**This configuration:**

1. Consumes all Classic `_historian` topics
2. Converts them to Core format using the `_raw` data contract
3. Publishes to individual Core topics via `uns_output`
4. Uses `umh_topic` metadata for automatic topic routing
5. Preserves all metadata for downstream processing

## Troubleshooting

### Common Issues

**No output messages**

* Check that input has valid JSON with timestamp\_ms field
* Verify topic metadata is present
* Ensure input topics follow Classic format: `umh.v1.<location>._historian.<context>`

**Missing fields**

* Check logs for parsing errors
* Ensure fields are not nested beyond maximum recursion depth (10 levels)

**Wrong topics**

* Validate input topic format
* Check `target_data_contract` configuration
* Verify UMH v1 topic structure

**Timestamp errors**

* Check that timestamp\_ms field contains numeric values (int, float, or string numbers)
* Ensure timestamp values are valid Unix timestamps


# Downsampler

### Downsampler in a Nutshell

The UMH **Downsampler** sits between the field bus and your persistence layer and performs one task with provable bounds: **it discards samples that do not contribute new information**.

#### Supported compression modes

* **Dead-band** – forwards every *value change*, suppresses byte-for-byte repeats, and—if `max_time` is set—adds an audit heartbeat.
* **Swinging-Door Trending (SDT)** – maintains a slope-aware envelope; a sample is forwarded only when it would exceed a fixed vertical error bound.

Wildcard overrides let you apply a conservative baseline (`deadband {threshold: 0}`) and refine it only where tighter compression is beneficial.

#### Recommended deployment points

1. **First hop into the Unified Namespace** *Configuration:* `deadband {threshold: 0, max_time: 30m}` *Effect:* Removes pure duplicates while guaranteeing one confirmatory message every 30 min. No latency is introduced.
2. **Noise suppression anywhere in the stream** *Configuration:* `deadband {threshold: 2 × σ_noise, max_time: 1h}` *Effect:* Removes fluctuations beneath the sensor's specified noise floor yet retains all legitimate steps. No latency is introduced.
3. **Final stage before the historian** *Configuration:* `swinging_door {threshold: 2 × σ_noise, min_time: physics_limit, max_time: 1h}` *Effect:* Achieves 95–99 % volume reduction while preserving slope integrity and audit heartbeat. Introduces latency of up to `max_time` as the algorithm needs to buffer the last point before it can be emitted.

Discrete counters and alarms should remain on `deadband {threshold: 0}`; SDT's emit-previous logic could delay a critical state change.

> **Compliance note** – Both algorithms enforce an absolute error bound and maintain a heartbeat, but they still remove data. If your GxP / 21 CFR Part 11 process mandates a full raw stream, archive the unfiltered UMH-core topic in parallel.

***

### 1 Quick-start Configuration

```yaml
processors:
  - downsampler:
      default:                     # safe baseline for every topic
        late_policy: passthrough   # global policy for late arrivals
        deadband:
          threshold: 0             # keep every change
          max_time: "30m"          # 30-minute heartbeat
      overrides:                   # finer control
        - pattern: "*.temperature"
          deadband:
            threshold: 0.1         # 2 × 0.05 °C noise
        - pattern: "*.furnace*"
          late_policy: drop        # can override per pattern
          swinging_door:
            threshold: 0.1
            min_time: "5s"
            max_time: "1h"
```

*Nothing else is required.* Messages that aren't strict UMH-core time-series (`value` + `timestamp_ms`) pass straight through.

**Pattern Matching:** The `pattern` field uses **shell-style glob patterns** (not regex). Supported wildcards:

* `*` matches any sequence of characters (e.g., `"*.temperature"` matches topics ending in "temperature")
* `?` matches any single character
* `[abc]` matches any character in brackets
* Examples: `"*.temperature"`, `"*sensor*"`, `"temp_?"`, `"sensor[12]"`

**Important:** When an override pattern specifies both `deadband` and `swinging_door` algorithms, the more advanced **swinging door** algorithm takes precedence and deadband parameters are ignored.

***

### Algorithm & Parameter Deep Dive

The Downsampler exposes just four parameters; once you understand their purpose you can configure the system with confidence.

| Parameter     | Purpose                                                                                                                   | Typical setting                                                                                                                                | Issue it prevents                                                               |
| ------------- | ------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| `threshold`   | Maximum absolute deviation allowed before a new point must be stored (same units as the signal). Optional, defaults to 0. | **2 × sensor-noise σ** · If σ is unknown, inspect a steady period and take ± peak spread. **Default 0** = keep any change, drop exact repeats. | Removes pure measurement noise without masking genuine step changes.            |
| `min_time`    | Smallest physically realistic interval between meaningful changes. Available only on `swinging_door`                      | Fastest credible process period (e.g. 1 s for furnaces, 50 ms for servo torque). `0` = off                                                     | Suppresses transients caused by bursty drivers or unstable links.               |
| `max_time`    | Heart-beat that forces an output even during flat periods; also flushes any SDT buffer.                                   | 15 min – 1 h (aligns with 21 CFR §11 "system liveness").                                                                                       | Ensures line-flat sensors remain visible and internal buffers stay bounded.     |
| `late_policy` | Action for out-of-order samples.                                                                                          | `passthrough` (default) or `drop`.                                                                                                             | Lets you balance historical accuracy against traffic volume on skewed networks. |

#### Dead-band

* **Rule** Emit when `|v − last| ≥ threshold` or when `max_time` expires.
* **Best suited to** Duplicate removal, discrete states, counters, Boolean flags.
* **Tuning notes**
  * `threshold` is optional and defaults to 0 if not specified.
  * `threshold = 2 × σ_noise` removes sensor noise yet preserves real transitions.
  * `threshold = 0` (default) provides safe, universal de-duplication.
  * Keep `max_time` within 15 – 60 min to satisfy audit requirements.

#### Swinging-Door Trending (SDT)

* **Rule** Maintain rotating upper and lower "doors" that bound the current slope; emit the **previous** point when a new sample would violate the envelope.
* **Additional parameters**
  * `threshold` – vertical tolerance (same intuition as for dead-band).
  * `min_time` – enforces physical plausibility for fast-changing signals.
  * `max_time` – still provides the audit heart-beat and buffer flush.
* **Why choose SDT** Captures long ramps with just two points, eliminating the stair-step pattern created by naïve dead-band on slow trends, while mathematically guaranteeing the reconstruction error remains ≤ `threshold`.

### Edge-case Handling & Internal Behaviour

The Downsampler runs each series in its own finite-state machine and guards two invariants:

1. **No silent loss** A point is acknowledged only after it—or the predecessor that still sits in the buffer—has been forwarded downstream.
2. **Bounded memory** At most **one** candidate is stored per series; an idle-flush watchdog empties it after `max_time`.

***

#### How incoming records are routed

| Payload type            | Processing path                                    | Reasoning                                                                                 |
| ----------------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| *Numeric* (int / float) | Selected algorithm (`deadband` or `swinging_door`) | These are the signals where compression counts.                                           |
| *Boolean, string*       | Change detector (`value ≠ last_value`)             | For states and text, only transitions matter; thresholds are irrelevant.                  |
| Other JSON structures   | **Bypass** (fail-open)                             | Complex objects could hide multiple semantics—forward untouched rather than risk pruning. |

***

#### Late-arrival strategy

*The timestamp of each processed sample is compared with the most recent one already seen for that series.*

* `passthrough` (default) Forward the late sample raw, flagging it with `meta:late_oos=true` (oos = "out of sequence").
* `drop` Discard and increment a metric counter—useful when stale data has no value but bandwidth is critical.

Both modes preserve at-least-once delivery for in-order traffic; only you decide what to do with stragglers.

***

#### Buffered-emit logic (needed by SDT)

1. A candidate point is retained while the doors stay open.
2. When a violation occurs, the **previous** candidate is released and the new sample becomes the fresh candidate.
3. If traffic stops before the doors close, the watchdog flushes the last candidate after `max_time` to guarantee visibility.

The same mechanism ensures a graceful shutdown: during a Benthos drain, every buffered point is emitted before the plug-in confirms closure.

#### Dynamic, per-message compression hints

Instead of hard-coding a rule map inside the *downsampler* you can let each message declare its own compression wishes in **metadata**. The downsampler simply reads the hints, validates them, and uses them in place of the global/default settings. This is useful if you want to configure everything for a single tag in a single place - the tag\_processor.

***

**Metadata keys**

| Meta key         | Type / allowed values                    | Maps to parameter             |
| ---------------- | ---------------------------------------- | ----------------------------- |
| `ds_algorithm`   | `"deadband"` \| `"swinging_door"`        | algorithm switch              |
| `ds_threshold`   | float                                    | `threshold`                   |
| `ds_min_time`    | Go/ISO-style duration string (`"750ms"`) | `min_time` (SDT)              |
| `ds_max_time`    | duration string (`"15m"`, `"1h"`, …)     | `max_time`                    |
| `ds_late_policy` | `"passthrough"` \| `"drop"`              | late-arrival rule             |
| `ds_ignore`      | any non-empty value                      | bypass downsampler completely |

**Precedence**

```
per-message metadata  →  pattern override  →  global default
```

If any field is missing the cascade falls back to the next level, so you can override only what matters.

**Complete bypass with `ds_ignore`**

In UMH deployments, the downsampler is typically enabled by default with conservative settings to provide data compression benefits across all time-series data. However, certain message types require complete bypass of any downsampling logic to ensure data integrity.

The `ds_ignore` metadata key provides this capability and is designed to be used primarily in the **tag\_processor** to selectively bypass downsampling based on message characteristics:

* **Any non-empty value** for `ds_ignore` will cause the message to skip all downsampling logic
* The message passes through unchanged, preserving all original data and metadata
* A `downsampled_by: "ignored"` metadata field is added for traceability
* No series state is created or modified for ignored messages
* Ignored messages are counted in the `messages_ignored` metric

**Common UMH use cases**

**Critical alarms and alerts:**

```yaml
pipeline:
  processors:
    - tag_processor:
        conditions:
          - if: msg.meta.tag_name === "emergency_stop" || msg.meta.tag_name.endsWith("_alarm")
            then: |
              msg.meta.ds_ignore = "true"
              return msg;
    - downsampler: 
        default:
          deadband:
            threshold: 1.0
            max_time: "30m"
```

**High-precision measurements:**

```yaml
pipeline:
  processors:
    - tag_processor:
        conditions:
          - if: msg.meta.virtual_path && msg.meta.virtual_path.includes("calibration")
            then: |
              msg.meta.ds_ignore = "bypass_precision_data"
              return msg;
    - downsampler: 
        default:
          deadband:
            threshold: 0.1
```

**State change events:**

```yaml
pipeline:
  processors:
    - tag_processor:
        conditions:
          - if: msg.meta.data_contract === "_state" || msg.meta.tag_name.startsWith("mode_")
            then: |
              msg.meta.ds_ignore = "preserve_state_changes"
              return msg;
    - downsampler: 
        default:
          deadband:
            threshold: 2.0
```

This approach allows UMH to benefit from automatic downsampling for most data while preserving critical information that requires every data point to be stored without any filtering or compression.

***

**Tag-processor snippet – picking the right algorithm on the fly**

```yaml
pipeline:
  processors:
    - tag_processor:
        defaults: |
          # generic baseline for every tag
          msg.meta.location_path = "enterprise.plant1"
          msg.meta.data_contract = "_historian"

          # safe, zero-risk compression unless told otherwise
          msg.meta.ds_algorithm       = "deadband"
          msg.meta.ds_threshold  = 0
          msg.meta.ds_max_time   = "30m"
          return msg;

        conditions:
          - if: msg.meta.virtual_path.startsWith("furnace.")
            then: |
              msg.meta.ds_algorithm      = "swinging_door"
              msg.meta.ds_threshold = 0.5
              msg.meta.ds_min_time  = "5s"
              msg.meta.ds_max_time  = "1h"
              return msg;
    - downsampler: {}
```

***

#### Numerical edge cases covered in tests

| Scenario                              | Guard rail in code                                                     |
| ------------------------------------- | ---------------------------------------------------------------------- |
| `threshold < 0`                       | Configuration rejected at startup.                                     |
| `threshold = 0`                       | Legal—drops exact repeats, keeps any change.                           |
| `max_time = 0`                        | Treated as *unset* (no heart-beat, but still at-least-once buffering). |
| `min_time > max_time`                 | Validation error to prevent deadlocks.                                 |
| IEEE-754 extremes (`±Inf`, `NaN`)     | Sample is bypassed with a warning; counting metrics record the event.  |
| Clock skew (non-monotonic timestamps) | Logged and routed through late-arrival policy; compression continues.  |

With these guards the Downsampler behaves deterministically across PLC glitches, network jitter and even deliberate fuzz-test assaults—yet still errs on the side of passing the data through rather than dropping it.

#### How Swinging-Door Trending keeps the essentials—step by step

1. **Segment start (point A, green).** From the very first sample in a segment the algorithm opens two straight "doors": an **upper** line and a **lower** line. *Their slopes mark the fastest rise and the slowest rise allowed while the signal stays inside the ± compDev error band.*&#x20;
2. **Each new sample narrows the door.** Every arriving point recalculates the steepest and flattest slopes that would still stay within ± *compDev*. Geometrically the cone can only shrink, so the blue lines in the pictures pivot closer together frame by frame.

   <img src="/files/hugxe0AMaGKPgsIeOW4h" alt="Door narrows further" width="600">

   &#x20;

   <img src="/files/YHw3QqsLySXt6UgLRMVs" alt="Almost closed" width="600">
3. **Two working points—P and S.**
   * **P** (black) is simply the *latest* raw sample.
   * **S** (red, "snapshot") is the *candidate* we might archive. Unlike a dead-band filter we do **not** decide about P immediately; first we test whether S can still represent the trend after P tightens the cone.
4. **Door-slam event = keep S, start over.** When the upper-slope line crosses the lower-slope line the cone has closed (see img 4 / final GIF frame).

   * S is now guaranteed to be the last point that still satisfies the error bound → we **emit** S.
   * P becomes the new anchor A of the next segment, and the door re-opens to its full width.

   <img src="/files/I8rDdHD5WiRw65sfYYQ1" alt="Door closed, snapshot flushed" width="600">
5. **Heartbeat and physics limits.** *`max_time`* forces an emit even if the cone never closes (flat signals, watchdog audit). *`min_time`* ignores arrivals that come *so* fast the physical process could not have changed—useful for ovens versus millisecond-level servo data.
6. **The GIF puts it all together.** Blue cone lines swing, the red snapshot jumps forward, only a handful of black-× points survive, yet the reconstructed line never drifts more than ± *compDev* from the raw signal.

   <img src="/files/pWOTQ40jwVpSk3ZNeOtz" alt="GIF" width="600">

Image Source: [emrumo/swingingdoor](https://github.com/emrumo/swingingdoor)

***

**Practical take-away**

* OT engineers still see smooth ramps without the "stair-step" artefacts a naïve dead-band would create.
* IT teams hold up to **99 % fewer rows** and move far less data, with mathematically bounded error and an audit heartbeat that satisfies 21 CFR §11 / GxP record-liveness rules.


# Topic Browser

{% hint style="info" %}
**Internal UMH Component**: This processor is designed for internal UMH system use and enables the Topic Browser feature in the Management Console. It transforms raw UMH messages into structured data for real-time topic exploration and debugging.
{% endhint %}

The Topic Browser Processor extracts hierarchical topic information and event data from UMH messages, providing the data foundation for the Topic Browser interface. It enables users to explore the Unified Namespace in real-time, search topics by metadata, and inspect both current values and historical event streams.

## Business Logic Overview

### What the Topic Browser Shows Users

The Topic Browser transforms the raw stream of UMH messages into an organized, searchable interface:

1. **Hierarchical Topic Tree**: Displays the ISA-95 hierarchy (enterprise → site → area → line → etc.) with real-time population as new topics appear
2. **Live Topic Discovery**: Shows new topics the moment they start publishing data, with visual indicators for recently active topics
3. **Metadata Search & Filtering**: Enables searching by headers like `unit=°C`, `manufacturer=Siemens`, or `plc_address` without scanning live message traffic
4. **Real-Time Value Display**: Shows the latest value for each topic with timestamp and data quality indicators
5. **Event History**: Provides a scrollable timeline of recent values for debugging and trend analysis
6. **Topic Debugging**: Displays raw message headers, routing history, and timing information for troubleshooting

### User Journey Through Topic Browser

| Phase               | User Action               | UI Response                          | Underlying Data                                    |
| ------------------- | ------------------------- | ------------------------------------ | -------------------------------------------------- |
| **Discovery**       | Opens Topic Browser       | Tree auto-populates with live topics | `uns_map` contains all active topics with metadata |
| **Search**          | Types "temperature"       | Tree filters, shows matching topics  | Search runs against cached `TopicInfo.metadata`    |
| **Inspection**      | Clicks a topic            | Shows latest value + sparkline       | Latest `EventTableEntry` + recent event history    |
| **Debugging**       | Expands "Headers" section | Shows all metadata for topic         | `TopicInfo.metadata` merged from recent messages   |
| **Troubleshooting** | Views "Message History"   | Timeline of recent values            | Ring buffer of `EventTableEntry` objects           |

## Message Processing Contract

### Processing Architecture

The processor implements a **ring buffer + delayed ACK** architecture:

#### Ring Buffer Strategy

* **Per-topic buffers**: Each topic maintains a ring buffer of latest events
* **Buffer size**: Configurable (default: 10 events per topic per interval)
* **Overflow handling**: Automatic overwrite of oldest events when buffer full
* **Rate limiting**: Prevents memory exhaustion during startup topic replay scenarios
* **Emission interval**: Configurable timer-based emission (default: 1 second)

#### Delayed ACK Pattern

* **Buffering**: Messages are buffered until emission interval elapses
* **In-place ACK**: Buffered messages are ACKed in-place when emission succeeds (not forwarded downstream)
* **Single emission**: Only the protobuf bundle is forwarded downstream, original messages are consumed
* **Failure handling**: Emission failure prevents ACK (messages will be retried)
* **Memory safety**: Buffer size limits protect against unbounded growth

### Input Requirements

The processor expects UMH messages with:

* **umh\_topic metadata**: Topic hierarchy string (e.g., `umh.v1.enterprise.plant1._historian.temperature`)
* **Valid JSON payload**: Either UMH-Core time-series format or relational data
* **Kafka headers**: Optional metadata for topic enrichment (unit, manufacturer, etc.)

### Output Emission Rules

The processor follows a strict emission contract that optimizes network traffic:

#### Critical: Only Protobuf Bundle is Forwarded

* **Single output**: Only the compressed protobuf bundle (STARTSTARTSTART format) is sent downstream
* **Original messages**: Input UMH messages are ACKed but **NOT forwarded** downstream
* **No duplication**: You will never see both the protobuf bundle AND the original messages in output
* **Clean pipeline**: Downstream consumers only receive the structured protobuf data

#### UNS Map Emission

* **When emitted**: Always emitted with complete topic tree in every emission interval
* **What's included**: The ENTIRE current topic tree (all topics and their cumulative metadata)
* **Why entire tree**: Stateless consumption - downstream gets complete state each time
* **No change detection**: Always emits full tree for complete state consistency

#### Events Emission

* **What's included**: ALL successfully processed messages from the current batch
* **No synthetic data**: Processor never fabricates additional events
* **Failed messages**: Logged and counted but don't appear in output

#### Possible Output Scenarios

1. **Both uns\_map + events**: Complete topic tree plus ring-buffered events (most common)
2. **Events only**: Ring-buffered events with previously known topic tree
3. **Topics only**: Complete topic tree without events (edge case)
4. **No output**: No messages processed since last emission interval

#### Message-Driven Behavior (Important)

* **Emission trigger**: Emissions ONLY occur when messages are actively being processed
* **No heartbeats**: No timer-based heartbeats if no messages arrive
* **Low-traffic impact**: Low-traffic UNS scenarios may experience extended delays between emissions
* **Downstream considerations**: Consumers should expect gaps in emission timing during quiet periods
* **Design rationale**: Processor is message-driven, not time-driven for resource efficiency

### Data Structures

#### TopicInfo (Topic Metadata)

```protobuf
message TopicInfo {
  string level0 = 1;                    // Enterprise/root level (mandatory)
  repeated string location_sublevels = 2; // [site, area, line, workcell, ...] 
  string data_contract = 3;             // "_historian", "_analytics", etc.
  optional string virtual_path = 4;     // Non-physical grouping (axis.x.position)
  string name = 5;                      // Final segment (temperature, pressure)
  map<string,string> metadata = 6;     // Aggregated headers (unit, manufacturer, etc.)
}
```

**Business Logic**:

* Represents the "where" and "what" of each signal in the system
* Metadata accumulated from all messages for complete topic state
* Always includes cumulative metadata (all keys ever seen for the topic)
* Updated with latest values using last-write-wins strategy

#### EventTableEntry (Message Data)

```protobuf
message EventTableEntry {
  string uns_tree_id = 1;              // xxHash of TopicInfo for efficient joins
  oneof payload {
    TimeSeriesPayload ts = 10;         // Scalar value + timestamp
    RelationalPayload rel = 11;        // Full JSON document
  }
  EventKafka raw_kafka_msg = 5;        // Original headers + payload for debugging
  repeated string bridged_by = 6;      // Benthos routing history
  uint64 produced_at_ms = 7;           // Kafka write timestamp
}
```

**Business Logic**:

* Represents individual message events with timestamps
* `uns_tree_id` links back to topic hierarchy without repeating strings
* Raw message preserved for debugging network issues or data quality problems

### Topic Hierarchy Processing

The processor parses UMH topic strings into structured hierarchies:

```
umh.v1.enterprise.site.area.line._historian.axis.x.position
        ↓
Level0: "enterprise"
LocationSublevels: ["site", "area", "line"] 
DataContract: "_historian"
VirtualPath: "axis.x"
Name: "position"
```

**Business Rules**:

* level0 is mandatory
* Location sublevels are dynamic (can be 0 to N levels)
* Data contract must start with underscore and cannot be the final segment
* Virtual path is optional logical grouping
* Name is always the final segment

## Performance Optimizations

### Data Serialization Strategy

The processor uses **direct protobuf serialization** for all output:

#### Serialization Approach

* **Strategy**: Direct protobuf serialization without compression
* **No threshold**: All payloads are processed using the same format
* **Implementation**: Direct binary serialization for efficiency
* **Detection**: Downstream consumers parse protobuf directly

#### Performance Benefits

* **Reduced CPU overhead**: Eliminates compression/decompression cycles
* **Simplified processing**: Direct protobuf parsing without decompression
* **Lower memory usage**: No compression buffer allocations
* **Faster processing**: Optimized for small to medium payloads common in UMH

### LRU Cache Optimization

Topic metadata caching prevents unnecessary re-transmission:

#### Cache Strategy

* **Key**: UNS Tree ID (xxHash of topic hierarchy)
* **Value**: Merged headers from recent messages
* **Size**: Configurable (default 50,000 entries)
* **Eviction**: Least Recently Used

#### Metadata Accumulation Logic

```
New message → Extract headers → Hash topic hierarchy → Cache lookup
                                                           ↓
Cache miss: Store headers in cache + update full topic map
Cache hit: Merge headers (last-write-wins) → Update cache + full topic map
                                                           ↓
                            Full topic map always emitted with complete state
```

**Result**: Cumulative metadata ensures complete topic state is always available downstream

### Protobuf Schema Design

Efficient binary serialization with forward/backward compatibility:

#### Space Optimizations

* Hash-based topic references instead of repeated strings
* Oneof payload fields to avoid null checks
* Varint encoding for timestamps and counters
* Optional fields for sparse data

#### Network Format

```
STARTSTARTSTART
<hex-encoded-protobuf-data>
ENDDATAENDDATAENDDATA  
<unix-timestamp-ms>
ENDENDENDEND
```

**Purpose**: Enables efficient parsing and latency measurement in downstream systems

## Configuration

### Basic Configuration

```yaml
processors:
  - topic_browser:
      lru_size: 50000                              # Cache size (default: 50,000 entries)
      emit_interval: "1s"                          # Base emit interval - CPU-aware controller adapts this (default: 1s)
      max_events_per_topic_per_interval: 1        # Ring buffer size per topic - burst protection (default: 1)
      max_buffer_size: 10000                     # Safety limit for total buffered messages (default: 10,000)
```

## CPU-Aware Adaptive Behavior

The topic browser processor automatically adapts its emit intervals based on CPU load and payload patterns **without requiring any configuration**. This intelligent resource management ensures optimal performance while preventing CPU saturation.

### Algorithm Overview

* **CPU Monitoring**: Samples CPU usage every 200ms using `syscall.Getrusage`
* **EMA Smoothing**: Applies exponential moving average (α=0.2) to prevent oscillation
* **Payload Awareness**: Adjusts intervals based on message volume patterns
* **Adaptive Range**: Emit intervals dynamically adjust between 1s-15s
* **Gradual Changes**: Maximum 2s adjustment per cycle for stability

### Behavior by Load Pattern

| Scenario           | CPU Load | Payload Size    | Adaptive Interval   | Result                            |
| ------------------ | -------- | --------------- | ------------------- | --------------------------------- |
| High traffic burst | >90%     | Large (>50KB)   | 1s (minimum)        | Fast emission, better compression |
| Normal operation   | 70-90%   | Medium (1-10KB) | 4-8s                | Balanced performance              |
| Low traffic        | <70%     | Small (<1KB)    | 8-15s               | CPU conservation                  |
| CPU saturation     | >90%     | Any             | Increases gradually | Prevents overload                 |

### Exposed Metrics

The processor exposes additional Prometheus metrics for operational visibility:

* `cpu_load_percent`: Current CPU usage percentage (0-100)
* `active_emit_interval_seconds`: Current adaptive emit interval in milliseconds
* `active_topics_count`: Number of active topics being tracked

### Configuration Parameters

| Parameter                           | Default | Purpose                                        | Tuning Guidance                                  |
| ----------------------------------- | ------- | ---------------------------------------------- | ------------------------------------------------ |
| `lru_size`                          | 50,000  | LRU cache size for cumulative metadata storage | Adjust based on topic cardinality                |
| `emit_interval`                     | 1s      | Base emit interval for CPU-aware adaptation    | Used as starting point for adaptive algorithm    |
| `max_events_per_topic_per_interval` | 1       | Ring buffer size per topic (burst protection)  | Increased from 10→1 for maximum burst protection |
| `max_buffer_size`                   | 100,000 | Safety limit for total buffered messages       | Set based on available memory                    |

### Sizing Guidelines

| Environment      | Topics         | LRU Size | Ring Buffer | Memory Usage |
| ---------------- | -------------- | -------- | ----------- | ------------ |
| Small plant      | <1,000         | 5,000    | 10          | \~5MB        |
| Medium plant     | 1,000-10,000   | 25,000   | 10          | \~25MB       |
| Large enterprise | 10,000-100,000 | 100,000  | 15          | \~100MB      |
| Very large       | >100,000       | 250,000  | 20          | \~250MB      |

## Integration Points

### Upstream Data Sources

* **uns-input plugin**: Provides umh\_topic metadata and Kafka timestamps
* **tag-processor**: Generates structured UMH messages with metadata headers
* **Bridge plugins**: Add routing and device metadata to headers

### Downstream Consumers

* **stdout output**: Delivers formatted messages to umh-core
* **FSM (Finite State Machine)**: Maintains canonical topic tree and event history
* **Management Console**: Renders Topic Browser interface from FSM data

### Data Flow

```
[OPC UA/Modbus] → [Bridge] → [tag-processor] → [topic-browser] → [stdout] → [umh-core FSM] → [UI]
                     ↓            ↓               ↓                           
                  Headers    UMH Format    Structured Data              
```

## Edge Cases & Error Handling

### Message Processing Failures

* **Invalid topic format**: Message skipped, error logged, failure metric incremented
* **Missing umh\_topic**: Message skipped with detailed logging
* **Malformed JSON**: Message skipped, error logged, processing continues
* **Duplicate timestamps**: Last-write-wins for headers, all events preserved

### Cache Management

* **Cache eviction**: Evicted topics re-emitted on next access (acceptable trade-off)
* **Memory pressure**: LRU automatically manages memory within configured bounds
* **Thread safety**: Mutex protection prevents race conditions in concurrent processing

### Network & Serialization

* **Protobuf failures**: Return error immediately (no partial emission)
* **Serialization failures**: Return error immediately (no partial data emission)
* **Large payloads**: Direct protobuf serialization handles up to multi-megabyte bundles efficiently

### Ring Buffer Edge Cases

* **Buffer overflow**: Oldest events automatically discarded when ring buffer full
* **Burst traffic**: Startup topic replay scenarios handled gracefully via ring buffer limits
* **Memory safety**: Ring buffer size limits prevent unbounded memory growth
* **Event ordering**: Ring buffer maintains chronological order within each topic

## Troubleshooting

### Common Issues

**No topics appearing in browser**:

* Check umh\_topic metadata is present in messages
* Verify topic format follows UMH conventions (umh.v1....)
* Ensure messages reach the processor (check input metrics)

**Seeing duplicate messages in output**:

* This should NOT happen - if you see both protobuf bundles AND original messages, there's a configuration issue
* The processor should only emit protobuf bundles (STARTSTARTSTART format)
* Original UMH messages are consumed and ACKed, not forwarded
* Check for multiple processors or incorrect pipeline configuration

**Performance degradation**:

* Monitor LRU cache hit rate (should be >90%)
* Check for extremely high topic cardinality
* Verify ring buffer isn't overflowing excessively (check events\_overwritten metric)
* Ensure emission intervals aren't too frequent for your traffic volume

**Memory usage growth**:

* Reduce lru\_size if memory constrained
* Lower max\_events\_per\_topic\_per\_interval to reduce ring buffer memory
* Reduce max\_buffer\_size to limit total buffered messages
* Check for topic metadata churn causing cache misses
* Monitor for header proliferation per topic

**Emission delays**:

* Verify messages are actively flowing (processor is message-driven)
* Check emit\_interval setting vs required latency
* Monitor for processing bottlenecks causing buffer delays

### Metrics to Monitor

* `messages_processed`: Successfully processed messages (should increase steadily)
* `messages_failed`: Failed processing attempts (should remain low)
* `events_overwritten`: Ring buffer overflow events (monitor for excessive values)
* `total_events_emitted`: Total events sent downstream (for throughput monitoring)
* `ring_buffer_utilization`: Ring buffer usage patterns (currently tracked but not actively used)
* `flush_duration`: Time taken for buffer flush operations (performance monitoring)
* `emission_size`: Size of emitted protobuf bundles (network usage monitoring)

**Additional Metrics Available**:

* Ring buffer utilization: Monitor per-topic buffer usage patterns
* Downstream latency: Monitor time from message receipt to UI display


# Stream Processor

The Stream Processor collects timeseries data from multiple UNS sources, maintains state for variable mappings, and generates transformed messages using JavaScript expressions.

## When to Use This Processor

Use Stream Processor when you need to combine data from multiple UNS sources. Example: Calculate pump efficiency from separate flow and temperature sensors.

**Required input format**: UMH timeseries (`value` + `timestamp_ms`)

## Configuration

```yaml
processors:
  - stream_processor:
      mode: "timeseries"              # Only "timeseries" supported
      output_topic: "umh.v1.enterprise.site.area"  # Base output topic
      model:
        name: "pump"                  # Model name for data contract
        version: "v1"                 # Model version
      sources:                        # Map variables to UNS topics
        press: "umh.v1.enterprise.site.area._raw.pressure"
        temp: "umh.v1.enterprise.site.area._raw.temperature"
        run: "umh.v1.enterprise.site.area._raw.running"
      mapping:                        # JavaScript expressions
        # Static (no dependencies)
        serialNumber: '"SN-P42-008"'
        # Dynamic (with dependencies)
        pressure: "press + 4.00001"              # Depends on 'press'
        temperatureC: "(temp - 32) * 5/9"        # Depends on 'temp'
        motor:
          rpm: "press / 4"                       # Nested, depends on 'press'
        status: 'run ? "active" : "inactive"'    # Depends on 'run'
        efficiency: "(press / temp) * 100"       # Depends on both
```

| Field           | Type   | Required | Description                   |
| --------------- | ------ | -------- | ----------------------------- |
| `mode`          | string | Yes      | Only `"timeseries"` supported |
| `output_topic`  | string | Yes      | Base UMH topic for output     |
| `model.name`    | string | Yes      | Model name for data contract  |
| `model.version` | string | Yes      | Model version                 |
| `sources`       | map    | Yes      | Variable name → UNS topic     |
| `mapping`       | object | No       | JavaScript expressions        |

## How It Works

### Dependency Tracking

* **Static mappings**: Expressions without variables evaluate on every message (e.g., `"SN-12345"`)
* **Dynamic mappings**: Expressions with variables evaluate when those variables update (e.g., `press * 2`)
* **Multiple dependencies**: Wait for all variables before evaluating (e.g., `press / temp`)

### JavaScript Expression Engine

The Stream Processor uses the same underlying [goja](https://github.com/dop251/goja) JavaScript engine as the [Node-RED JavaScript Processor](/benthos-umh/processing/node-red-javascript-processor), but with different configurations optimized for different use cases:

**Stream Processor JavaScript:**

* Evaluates single expressions (not full functions)
* Pre-compiles at startup
* Auto-detects variable dependencies
* Blocks dangerous operations (eval, require, process)

**Comparison with Node-RED JavaScript Processor:**

* Node-RED JS runs full JavaScript functions with `return` statements
* Stream Processor evaluates expressions directly (no function wrapper needed)
* Both use security sandboxing to prevent dangerous operations
* Node-RED JS allows console.log() for debugging, Stream Processor blocks it
* Stream Processor adds expression pre-compilation and dependency analysis

**Security Features (shared with Node-RED JS):**

* Maximum call stack size limits to prevent stack overflow
* Blocked dangerous globals (eval, Function, require, process, setTimeout)
* Safe built-in objects available (Math, JSON, Date, String, Number)
* Execution within a sandboxed environment

### State Management

* Stores latest value for each configured source
* Values persist until processor restarts
* Enables calculations across async data sources

## Message Processing

### Input Requirements

1. **Format**: `{"value": 25.5, "timestamp_ms": 1647890123456}`
2. **Metadata**: Must have `umh_topic` field
3. **Topic Match**: Must match a configured source

### Processing Steps

1. Match `umh_topic` to configured sources
2. Store value in state (e.g., `press = 25.5`)
3. Evaluate static mappings
4. Evaluate dynamic mappings with satisfied dependencies
5. Generate output messages

### Output Topics

Format: `<output_topic>.<data_contract>[.<virtual_path>].<name>`

* `output_topic`: From configuration
* `data_contract`: `_<model_name>_<model_version>`
* `virtual_path`: For nested mappings
* `name`: Field name

Examples:

* `umh.v1.enterprise.site.area._pump_v1.pressure`
* `umh.v1.enterprise.site.area._pump_v1.motor.rpm` (nested)

## JavaScript Expressions

### Allowed Operations

```javascript
// Math
Math.PI * 2
Math.sqrt(press * press + temp * temp)

// Arithmetic
press * 2 + 1
(temp - 32) * 5/9

// Conditionals
press > 100 ? 'high' : 'normal'
run ? 'active' : 'inactive'

// Strings
"Pump-" + press
`Temperature: ${temp}°C`

// JSON
JSON.stringify({pressure: press, temp: temp})

// Date
Date.now()
```

### Blocked Operations

* `eval()` - No dynamic code
* `Function()` - No function constructors
* `require()` - No module imports
* `process` - No process access
* `setTimeout/setInterval` - No async
* `console` - No console access

## Examples

### Sensor Calibration

```yaml
stream_processor:
  mode: "timeseries"
  output_topic: "umh.v1.plant1.line5.cell3"
  model:
    name: "sensor_calibrated"
    version: "v1"
  sources:
    raw_temp: "umh.v1.plant1.line5.cell3._raw.temperature"
    raw_press: "umh.v1.plant1.line5.cell3._raw.pressure"
  mapping:
    temperature: "raw_temp + 0.5"  # Offset correction
    pressure: "raw_press * 1.02"   # Scale correction
```

### Machine Status

```yaml
stream_processor:
  mode: "timeseries"
  output_topic: "umh.v1.factory.assembly"
  model:
    name: "machine_status"
    version: "v2"
  sources:
    speed: "umh.v1.factory.assembly._raw.conveyor_speed"
    temp: "umh.v1.factory.assembly._raw.motor_temp"
    running: "umh.v1.factory.assembly._raw.is_running"
  mapping:
    status: |
      running ? (temp > 80 ? "warning" : "running") : "stopped"
    alert: "temp > 90"
    efficiency: "running ? (speed / 100) * 100 : 0"
```

### Boiler Efficiency

```yaml
stream_processor:
  mode: "timeseries"
  output_topic: "umh.v1.site.boiler"
  model:
    name: "efficiency_calc"
    version: "v1"
  sources:
    flow_in: "umh.v1.site.boiler._raw.input_flow"
    flow_out: "umh.v1.site.boiler._raw.output_flow"
    temp_in: "umh.v1.site.boiler._raw.input_temp"
    temp_out: "umh.v1.site.boiler._raw.output_temp"
  mapping:
    flow_diff: "flow_out - flow_in"
    temp_diff: "temp_out - temp_in"
    thermal_efficiency: |
      (flow_out * temp_out - flow_in * temp_in) / (flow_in * temp_in) * 100
    heat_transfer: "(flow_out - flow_in) * (temp_out - temp_in) * 500"
```

### Nested Metrics

```yaml
stream_processor:
  mode: "timeseries"
  output_topic: "umh.v1.plant.packaging"
  model:
    name: "machine_metrics"
    version: "v3"
  sources:
    speed: "umh.v1.plant.packaging._raw.belt_speed"
    count: "umh.v1.plant.packaging._raw.package_count"
    weight: "umh.v1.plant.packaging._raw.total_weight"
  mapping:
    performance:
      speed: "speed"
      throughput: "count / 60"  # packages/min
    quality:
      avg_weight: "weight / count"
      deviation: "Math.abs((weight / count) - 1.5)"  # from target
```

Output topics:

* `umh.v1.plant.packaging._machine_metrics_v3.performance.speed`
* `umh.v1.plant.packaging._machine_metrics_v3.performance.throughput`
* `umh.v1.plant.packaging._machine_metrics_v3.quality.avg_weight`
* `umh.v1.plant.packaging._machine_metrics_v3.quality.deviation`

## Processing Behavior

### Static vs Dynamic Evaluation

**Static** (every message): `"SN-12345"`, `Date.now()`, `Math.PI`

**Dynamic** (when dependencies update): `press * 2`, `press / temp`, `run ? "active" : "stopped"`

### Metadata Preservation

* `umh_topic`: Replaced with output topic
* All other metadata: Passed through

### Error Handling

* **Invalid JSON**: "Invalid timeseries format from umh.v1.plant.sensor - expected {value, timestamp\_ms}. Skipping message."
* **Missing Dependencies**: "Cannot evaluate 'efficiency' - waiting for 'temp' (last value: 65.2 from 30s ago)"
* **JavaScript Error**: "Failed to evaluate 'motor.rpm': Cannot divide by zero (press=0). Using last valid value: 1250."
* **Missing umh\_topic**: "Message missing umh\_topic metadata. Check input configuration."

## Troubleshooting

### No Output Messages

**Check umh\_topic exists:**

```bash
# Verify metadata in your pipeline
echo '{"value": 42}' | benthos -c your_config.yaml
# Should show: Message missing umh_topic metadata
```

**Verify topic matches source:**

```yaml
sources:
  press: "umh.v1.plant._raw.pressure"  # Must match exactly
```

### Missing Outputs

**Check dependencies:**

* `efficiency: "press / temp"` needs both variables
* Look for "waiting for 'temp'" in logs

**Test expressions:**

```javascript
// Test in browser console first
var press = 100, temp = 50;
console.log(press / temp);  // Should be 2
```

### Performance Issues

* Simplify expressions: `press * 2` instead of complex math
* Check message rate: >1000 msg/s may need optimization
* Monitor memory if storing many variables

### Common Errors

| Error                         | Cause                   | Fix                         |
| ----------------------------- | ----------------------- | --------------------------- |
| "undefined is not a function" | Using blocked function  | Use allowed operations only |
| "variable not defined"        | Typo in variable name   | Check sources configuration |
| "unexpected token"            | JavaScript syntax error | Test in browser console     |
| "Cannot read null"            | Missing null check      | Add: `temp ? temp * 2 : 0`  |


# Node-RED JavaScript Processor

The Node-RED JavaScript processor allows you to write JavaScript code to process messages in a style similar to Node-RED function nodes. This makes it easy to port existing Node-RED functions to Benthos or write new processing logic using familiar JavaScript syntax.

Use the `nodered_js` processor instead of the `tag_processor` when you need full control over the payload and require custom processing logic that goes beyond standard tag or time series data handling. This processor allows you to write custom JavaScript code to manipulate both the payload and metadata, providing the flexibility to implement complex transformations, conditional logic, or integrate with other systems.

For the full list of available JavaScript globals (`msg`, `console`, `cache`, `protobuf`), see the [JavaScript API Reference](/benthos-umh/processing/javascript-api).

**Configuration**

```yaml
pipeline:
  processors:
    - nodered_js:
        code: |
          // Your JavaScript code here
          return msg;
```

**Message Format**

Messages in Benthos and in the JavaScript processor are handled differently:

**In Benthos/Bloblang:**

```yaml
# Message content is the message itself
root = this   # accesses the message content

# Metadata is accessed via meta() function
meta("some_key")   # gets metadata value
meta some_key = "value"   # sets metadata
```

**In JavaScript (Node-RED style):**

```javascript
// Message content is in msg.payload
msg.payload   // accesses the message content

// Metadata is in msg.meta
msg.meta.some_key   // accesses metadata
```

The processor automatically converts between these formats.

**Examples**

1. **Pass Through Message**\
   Input message:

```json
{
  "temperature": 25.5,
  "humidity": 60
}
```

Metadata:

```yaml
sensor_id: "temp_1"
location: "room_a"
```

JavaScript code:

```yaml
pipeline:
  processors:
    - nodered_js:
        code: |
          // Message arrives as:
          // msg.payload = {"temperature": 25.5, "humidity": 60}
          // msg.meta = {"sensor_id": "temp_1", "location": "room_a"}

          // Simply pass through
          return msg;
```

Output: Identical to input

2. **Modify Message Payload**\
   Input message:

```json
["apple", "banana", "orange"]
```

JavaScript code:

```yaml
pipeline:
  processors:
    - nodered_js:
        code: |
          // msg.payload = ["apple", "banana", "orange"]
          msg.payload = msg.payload.length;
          return msg;
```

Output message:

```json
3
```

3. **Create New Message**\
   Input message:

```json
{
  "raw_value": 1234
}
```

JavaScript code:

```yaml
pipeline:
  processors:
    - nodered_js:
        code: |
          // Create new message with transformed data
          var newMsg = {
            payload: {
              processed_value: msg.payload.raw_value * 2,
              timestamp: Date.now()
            }
          };
          return newMsg;
```

Output message:

```json
{
  "processed_value": 2468,
  "timestamp": 1710254879123
}
```

4. **Drop Messages (Filter)**\
   Input messages:

```json
{"status": "ok"}
{"status": "error"}
{"status": "ok"}
```

JavaScript code:

```yaml
pipeline:
  processors:
    - nodered_js:
        code: |
          // Only pass through messages with status "ok"
          if (msg.payload.status === "error") {
            return null;  // Message will be dropped
          }
          return msg;
```

Output: Only messages with status "ok" pass through

5. **Working with Metadata**\
   Input message:

```json
{"value": 42}
```

Metadata:

```yaml
source: "sensor_1"
```

JavaScript code:

```yaml
pipeline:
  processors:
    - nodered_js:
        code: |
          // Add processing information to metadata
          msg.meta.processed = "true";
          msg.meta.count = "1";

          // Modify existing metadata
          if (msg.meta.source) {
            msg.meta.source = "modified-" + msg.meta.source;
          }

          return msg;
```

Output message: Same as input

Output metadata:

```yaml
source: "modified-sensor_1"
processed: "true"
count: "1"
```

Equivalent Bloblang:

```coffee
meta processed = "true"
meta count = "1"
meta source = "modified-" + meta("source")
```

6. **String Manipulation**\
   Input message:

```json
"hello world"
```

JavaScript code:

```yaml
pipeline:
  processors:
    - nodered_js:
        code: |
          // Convert to uppercase
          msg.payload = msg.payload.toUpperCase();
          return msg;
```

Output message:

```json
"HELLO WORLD"
```

7. **Numeric Operations**\
   Input message:

```json
42
```

JavaScript code:

```yaml
pipeline:
  processors:
    - nodered_js:
        code: |
          // Double a number
          msg.payload = msg.payload * 2;
          return msg;
```

Output message:

```json
84
```

8. **Logging**\
   Input message:

```json
{
  "sensor": "temp_1",
  "value": 25.5
}
```

Metadata:

```yaml
timestamp: "2024-03-12T12:00:00Z"
```

JavaScript code:

```yaml
pipeline:
  processors:
    - nodered_js:
        code: |
          // Log various aspects of the message
          console.log("Processing temperature reading:" + msg.payload.value);
          console.log("From sensor:" + msg.payload.sensor);
          console.log("At time:" + msg.meta.timestamp);

          if (msg.payload.value > 30) {
            console.warn("High temperature detected!");
          }

          return msg;
```

Output: Same as input, with log messages in Benthos logs

9. **Returning an Array (Fan-out)**\
   A function may return an array of message objects to publish one output message per element. Each element must be a message object (`{payload, meta}`); `null`/`undefined` elements are skipped, and a non-object element drops the input (the rest of the batch continues).

   Input message:

```json
{
  "records": [
    {"id": 1, "temp": 22},
    {"id": 2, "temp": 23}
  ]
}
```

JavaScript code:

```yaml
pipeline:
  processors:
    - nodered_js:
        code: |
          // Publish one message per record, propagating the input's metadata
          return msg.payload.records.map(r => ({payload: r, meta: msg.meta}));
```

Output: Two messages, `{id: 1, temp: 22}` and `{id: 2, temp: 23}`, each carrying the input's metadata.

This is the typical pattern for read bridges that fetch a JSON array from an API (for example, an ERP system) and need one UNS message per record. Returning `null` or `undefined` drops the input (no outputs); returning an empty array `[]` or an all-`null` array `[null, null]` also drops the input and counts as a single drop in the `messages_dropped` metric. The `messages_processed` counter counts output messages, not inputs; under array fan-out one input can yield several processed messages, so use `messages_dropped` directly for drop-rate alerts rather than a `dropped/(dropped+processed)` ratio.

If the function throws an exception, or returns a value that is not a message object or `null`, that message is dropped: it is absent from the output, counted in `messages_dropped` (reason `js_throw`, `bad_return`, or `bad_array_element`), and a warning is logged. The rest of the batch continues normally. Returning `null` or `undefined` is a normal drop, not an error.

**Performance Comparison**

When choosing between Node-RED JavaScript and Bloblang for message processing, consider the performance implications. Here's a benchmark comparison of both processors performing a simple operation (doubling a number) on 1000 messages:

**JavaScript Processing:**

* Median: 15.4ms
* Mean: 20.9ms
* Standard Deviation: 9.4ms
* Range: 13.8ms - 39ms

**Bloblang Processing:**

* Median: 3.7ms
* Mean: 4ms
* Standard Deviation: 800µs
* Range: 3.3ms - 5.6ms

**Key Observations:**

1. Bloblang is approximately 4-5x faster for simple operations
2. Bloblang shows more consistent performance (smaller standard deviation)
3. However, considering typical protocol converter workloads (around 1000 messages/second), the performance difference is negligible for most use cases. The JavaScript processor's ease of use and familiarity often outweigh the performance benefits of Bloblang, especially for smaller user-generated flows.

Note that these benchmarks represent a simple operation. The performance difference may vary with more complex transformations or when using advanced JavaScript features.


# JavaScript API Reference

This page documents the global objects available in the JavaScript environment shared by the `nodered_js` and `tag_processor` processors. The engine is goja (ES5.1 with some ES6 features) — no Node.js APIs are available.

## msg

The message object. Contains the payload and metadata of the current message.

```javascript
msg.payload    // The message content (any JSON type)
msg.meta       // Metadata key-value pairs (strings)
```

**Return behavior:**

* `return msg;` — pass the message through (modified or not)
* `return null;` or `return undefined;` — drop the message
* `return { payload: ..., meta: ... };` — create a new message

**Example:**

```javascript
msg.payload = msg.payload * 2;
msg.meta.processed = "true";
return msg;
```

## console

Logging functions that write to the Benthos logger.

```javascript
console.debug(...)  // DEBUG level
console.log(...)    // INFO level
console.info(...)   // INFO level
console.warn(...)   // WARN level
console.error(...)  // ERROR level
```

Accepts multiple arguments: `console.log("value is", msg.payload.value)`

## cache

Key-value store for maintaining state across messages. Persists across all messages for the lifetime of the Benthos process. In-memory only, lost on restart. Supports any JSON-compatible type: strings, numbers, booleans, objects, arrays.

The cache is automatic and requires no configuration.

```javascript
cache.set(key, value)    // Store a value under key (string)
cache.get(key)           // Retrieve a value, logs error if key not found
cache.exists(key)        // Returns true if key exists, false otherwise
cache.delete(key)        // Remove a key
```

Always use `cache.exists(key)` before `cache.get(key)` to avoid error logs on missing keys.

```javascript
if (cache.exists("counter")) {
  var count = cache.get("counter");
} else {
  var count = 0;
}
```

### Counter

```javascript
var count = 0;
if (cache.exists("count")) { count = cache.get("count"); }
count++;
cache.set("count", count);
msg.payload = count;
return msg;
```

### Previous value comparison

```javascript
var prev = null;
if (cache.exists("last_value")) {
  prev = cache.get("last_value");
}
var delta = 0;
if (prev !== null) {
  delta = msg.payload.value - prev;
}
cache.set("last_value", msg.payload.value);
msg.payload.delta = delta;
return msg;
```

### History (last N values)

```javascript
var history = [];
if (cache.exists("history")) {
  history = cache.get("history");
}
history.push(msg.payload.value);
if (history.length > 10) history.shift();
cache.set("history", history);
return msg;
```

### Alarm state tracking

```javascript
var alarmed = false;
if (cache.exists("alarm_active")) {
  alarmed = cache.get("alarm_active");
}
if (msg.payload.value > 100 && !alarmed) {
  cache.set("alarm_active", true);
  msg.meta.alarm = "triggered";
  return msg;
}
if (msg.payload.value <= 100 && alarmed) {
  cache.set("alarm_active", false);
  msg.meta.alarm = "cleared";
  return msg;
}
return msg;
```

### Cycle time between events

```javascript
var lastMs = null;
if (cache.exists("last_event_ms")) {
  lastMs = cache.get("last_event_ms");
}
if (lastMs !== null) {
  msg.payload.cycle_time_ms = Date.now() - lastMs;
}
cache.set("last_event_ms", Date.now());
return msg;
```

### Limitations

* **In-memory only** — state is lost when the Benthos process restarts. A persistent backend is planned.
* **No size limits** — the cache grows unboundedly if keys are never deleted. Use `cache.delete` to clean up unused keys. A memory safeguard (threshold, eviction) is planned.
* **Cache scope in `tag_processor`** — the cache is shared across all stages (`defaults`, `conditions`, `advancedProcessing`). A value set in `defaults` is visible in `advancedProcessing` within the same message.

## protobuf

Decode and encode protobuf messages inline, against a schema passed as a base64-encoded `FileDescriptorSet` (no files on disk). Useful for reading data the standard inputs don't decode — for example the raw Sparkplug B metric bytes attached by the `sparkplug_b` input's `passthrough_raw_metric` flag, including proto2 extension fields.

```javascript
protobuf.decode(dataB64, descriptorSetB64, msgName)  // base64 proto bytes -> object
protobuf.encode(obj, descriptorSetB64, msgName)      // object -> base64 proto bytes
```

* `descriptorSetB64` — base64 of a self-contained `FileDescriptorSet`. Compile it once with `protoc --include_imports --descriptor_set_out=schema.pb your.proto`, then base64-encode `schema.pb` and paste the string into your script.
* `msgName` — fully-qualified message name, e.g. `com.example.Payload.Metric` (no leading dot).
* The decoded object follows protojson conventions: `int64`/`uint64` and `bytes` come back as strings, enums as their names, and **proto2 extensions appear as `[package.extension]` keys**. For `encode`, pass 64-bit integers as strings.
* Both functions throw on error (invalid base64, unknown message, malformed descriptor set); wrap calls in `try/catch` to handle failures in script.

```javascript
// Decode the raw Sparkplug metric attached by passthrough_raw_metric, reading an extension field.
var DESC = "CtIB..."; // base64 FileDescriptorSet, compiled once
var metric = protobuf.decode(msg.meta.spb_metric_raw, DESC, "com.example.Payload.Metric");
msg.payload = { value: metric.value, extra: metric["[com.example.my_extension]"] };
return msg;
```

Available in both `nodered_js` and `tag_processor` — they share the same JavaScript environment.


# Output

This section covers Benthos output plugins for writing data to various industrial systems and external destinations. Output plugins allow you to send processed data from your Benthos pipelines to PLCs, databases, message brokers, and other systems.

## Available Output Plugins

* [**OPC UA Output**](/benthos-umh/output/opc-ua-output) - Writes data to OPC UA servers with optional read-back confirmation (handshake). Supports multiple data types and provides safe setpoint operations for industrial control systems.
* [**UNS Output**](/benthos-umh/output/uns-output) - Publishes batched messages to the Unified Namespace in `umh-core`, auto-deriving `umh-topic`, sanitising keys, and writing them to the internal Kafka broker.
* [**Sparkplug B Output**](/benthos-umh/output/sparkplug-b-output) - Publishes batched messages to an MQTT Broker with the protocol Sparkplug B.
* [**Snowflake PUT Output**](https://github.com/united-manufacturing-hub/benthos-umh/tree/main/docs/output/snowflake-put.md) - Writes batched messages to a Snowflake stage and optionally triggers Snowpipe ingestion. Ported from warpstreamlabs/bento.
* [**TimescaleDB Historian Output**](https://github.com/united-manufacturing-hub/benthos-umh/tree/main/docs/output/historian.md) - Saves one UNS data contract into TimescaleDB using the UMH Historian schema, bootstrapping the schema and de-duplicating metadata, with a startup check that fails the bridge on a misconfigured database.
* [**More Output Plugins**](https://docs.redpanda.com/redpanda-connect/components/outputs/about/) - Additional built-in output plugins available in Benthos/Redpanda Connect for various destinations.

## Choosing the Right Output Plugin

* Use **OPC UA Output** when writing setpoints, commands, or data back to OPC UA-enabled industrial systems with confirmation requirements
* Use **UNS Output** to publish data into the Unified Namespace via `umh-core`, where topics, batching, key sanitisation are handled automatically.
* Use **Sparkplug B Output** to publish data to an MQTT Broker, into a custom data pipeline. Sparkplug B is typically used for edge device use cases where state awareness is required.
* Use **TimescaleDB Historian Output** to persist a UNS data contract as queryable time-series in TimescaleDB for dashboards and historical analysis.
* Explore **additional output plugins** for databases (PostgreSQL, MySQL, InfluxDB), message brokers (MQTT, Kafka), cloud services (AWS, Azure, GCP), file systems, HTTP APIs, and more


# Sparkplug B (Output)

## Overview

The **Sparkplug B Output plugin** allows the United Manufacturing Hub (UMH) to publish industrial IoT data to MQTT brokers using the Sparkplug B specification. It acts as an **Edge Node** in the Sparkplug B ecosystem, converting UMH-Core messages into standardized MQTT-based Sparkplug B protocol with protobuf encoding and alias management.

Sparkplug B is an open standard for MQTT-based industrial IoT communication that minimizes bandwidth usage through metric aliases and efficient protobuf encoding.

### Why Edge Node Only?

This output plugin **always operates as an Edge Node** because:

1. **Role Clarity**: In Sparkplug B architecture, data sources (PLCs, sensors, gateways) are Edge Nodes, while data consumers (SCADA, historians) are Hosts
2. **UMH Philosophy**: UMH acts as a data source when publishing to external systems, naturally fitting the Edge Node role
3. **No Conflicts**: Edge Nodes don't publish STATE messages, avoiding conflicts with existing Primary Hosts in your infrastructure
4. **Responds to Hosts**: Edge Nodes listen for rebirth commands from Host applications, enabling proper Sparkplug B session management

The complementary [Sparkplug B Input plugin](/benthos-umh/input/sparkplug-b-input) handles the Host role for consuming Sparkplug B data.

**UMH-Core Format Requirement**: This output plugin only accepts data in the UMH-Core format (`{"value": X, "timestamp_ms": Y}`). When using the `uns` input plugin, data is already in the correct format. For other input sources, use the `tag_processor` to convert data to UMH-Core format before this output plugin.

**For Sparkplug B Architecture Overview**: See the [Sparkplug B Input plugin documentation](/benthos-umh/input/sparkplug-b-input) for a comprehensive explanation of:

* UMH's Modified Parris Method and how it differs from industry standards
* Integration with the UMH Unified Namespace architecture
* Host vs Edge Node roles and their relationship

This output plugin implements the **Edge Node** role that complements the **Host** role of the input plugin.

## Quick Start

```yaml
input:
  uns: {}

output:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "FactoryA"
      edge_node_id: "EdgeNode1"
      # device_id is optional - if not specified, generated from location_path metadata
```

This configuration reads UMH-Core data and publishes it as Sparkplug B messages. The output plugin always acts as an Edge Node in the Sparkplug B ecosystem.

### UMH-Core to Sparkplug B Mapping

Here's how a UMH-Core message maps to Sparkplug B:

**Configuration:**

```yaml
identity:
  group_id: "FactoryA"
  edge_node_id: "EdgeNode1"
```

**Input UMH-Core Message:**

**Payload:**

```json
{
  "value": 23.5,
  "timestamp_ms": 1672531200000
}
```

**Metadata:**

```json
{
  "location_path": "enterprise.factory.line1.station1",
  "virtual_path": "sensors.ambient", 
  "tag_name": "temperature",
  "data_contract": "_sparkplug"
}
```

**↓ Results in Sparkplug B Message:**

* **Topic**: `spBv1.0/FactoryA/DDATA/EdgeNode1/enterprise:factory:line1:station1`
* **Metric Name**: `sensors:ambient:temperature` (virtual\_path + tag\_name joined with colons)
* **Payload**: Protobuf with metric alias, value 23.5, timestamp

**Key Transformations:**

1. **Location Path**: `enterprise.factory.line1.station1` → Device ID `enterprise:factory:line1:station1` (dots → colons)
2. **Virtual Path + Tag Name**: `sensors.ambient` + `temperature` → Metric Name `sensors:ambient:temperature` (joined with colons)
   * Note: The input plugin can parse metrics with colons, slashes, or dots as separators
3. **UMH-Core Format**: `{"value": 23.5, "timestamp_ms": 1672531200000}` → Sparkplug protobuf metric
4. **Topic Structure**: Uses configured `group_id` and `edge_node_id` from output plugin configuration

## Configuration Reference

### MQTT Section

| Field                       | Type       | Default                      | Description              |
| --------------------------- | ---------- | ---------------------------- | ------------------------ |
| `mqtt.urls`                 | `[]string` | **required**                 | List of MQTT broker URLs |
| `mqtt.client_id`            | `string`   | `"benthos-sparkplug-output"` | MQTT client identifier   |
| `mqtt.credentials.username` | `string`   | `""`                         | MQTT username            |
| `mqtt.credentials.password` | `string`   | `""`                         | MQTT password            |
| `mqtt.qos`                  | `int`      | `1`                          | MQTT QoS level           |
| `mqtt.keep_alive`           | `duration` | `"60s"`                      | MQTT keep alive interval |
| `mqtt.connect_timeout`      | `duration` | `"30s"`                      | Connection timeout       |
| `mqtt.clean_session`        | `bool`     | `true`                       | MQTT clean session flag  |

### Identity Section

| Field                   | Type     | Default      | Description                                                                                      |
| ----------------------- | -------- | ------------ | ------------------------------------------------------------------------------------------------ |
| `identity.group_id`     | `string` | **required** | Sparkplug B Group ID                                                                             |
| `identity.edge_node_id` | `string` | **required** | Static Edge Node ID for Sparkplug B compliance (must be consistent throughout session)           |
| `identity.device_id`    | `string` | `""`         | Device ID (empty for node-level messages, auto-generated from message metadata if not specified) |

### Advanced Configuration (Optional)

For advanced users who want to define static metric aliases:

| Field     | Type       | Default  | Description                                                       |
| --------- | ---------- | -------- | ----------------------------------------------------------------- |
| `metrics` | `[]object` | optional | List of static metric definitions (for advanced alias management) |

### Metric Definition

Each metric in the `metrics` array supports:

| Field              | Type     | Required | Description                           |
| ------------------ | -------- | -------- | ------------------------------------- |
| **name**           | `string` | **yes**  | Human-readable metric name            |
| **alias**          | `uint64` | **yes**  | Unique numeric alias (1-65535)        |
| **type**           | `string` | **yes**  | Sparkplug B data type                 |
| **value\_from**    | `string` | **yes**  | JSON field name containing the value  |
| **units**          | `string` | no       | Engineering units (e.g., "°C", "bar") |
| **is\_historical** | `bool`   | no       | Whether this is historical data       |
| **metadata**       | `object` | no       | Additional key-value metadata         |

### Behaviour Section

| Field                             | Type   | Default | Description                                                               |
| --------------------------------- | ------ | ------- | ------------------------------------------------------------------------- |
| `behaviour.auto_extract_tag_name` | `bool` | `true`  | Whether to automatically extract tag\_name from message metadata          |
| `behaviour.retain_last_values`    | `bool` | `true`  | Whether to retain last known values for BIRTH messages after reconnection |

## Edge Node Behavior

### Automatic Session Management

As a Sparkplug B Edge Node, this plugin handles the complete session lifecycle:

1. **Connection**: Publishes NBIRTH with all configured metrics and bdSeq
2. **Device Discovery**: Publishes DBIRTH when new devices (location\_paths) appear
3. **Data Flow**: Publishes DDATA with efficient alias-based encoding
4. **Disconnection**: NDEATH published automatically via MQTT Last Will Testament

### Rebirth Command Handling

The Edge Node listens for rebirth requests from Host applications on the NCMD topic:

```
spBv1.0/<group_id>/NCMD/<edge_node_id>
```

**When a rebirth is requested:**

1. bdSeq increments by +1
2. Republishes NBIRTH with all node-level metrics
3. Republishes DBIRTH for all known devices
4. Resumes normal DDATA publishing

**Why This Matters:**

* Hosts can request fresh BIRTH certificates after restart
* Ensures alias mappings stay synchronized
* Maintains Sparkplug B session integrity

## Data Format Requirements

This output plugin requires input data to be in **UMH-Core format**:

```json
{
  "value": 25.4,
  "timestamp_ms": 1672531200000
}
```

**Compatible Input Sources:**

* ✅ `uns` input plugin (already in UMH-Core format)
* ✅ Any input + `tag_processor` (converts to UMH-Core format)

**Required Message Metadata:**

* `location_path`: Hierarchical location (e.g., "enterprise.factory.line1.station1")
* `tag_name`: Metric name (e.g., "temperature", "pressure")
* `data_contract`: Data contract identifier (e.g., "\_sparkplug")
* `virtual_path`: Optional sub-path within device (e.g., "sensors.ambient")

## Stateless Architecture Limitations

### bdSeq (Birth-Death Sequence) Behavior

The Sparkplug B output plugin implements **bdSeq** (Birth-Death Sequence) according to the Sparkplug B v3.0 specification:

**Within Component Lifetime** (✅ Specification Compliant):

* bdSeq starts at 0 for the first MQTT session
* bdSeq increments by +1 for each subsequent MQTT reconnection session
* Example: Session 1: bdSeq=0 → Session 2: bdSeq=1 → Session 3: bdSeq=2

**Across Component Restarts** (⚠️ Stateless Limitation):

* bdSeq resets to 0 when the Benthos component is restarted
* This is a fundamental limitation of Benthos's stateless architecture
* No persistence mechanism is available (no database/disk storage)

### What This Means for Users

**Expected Behavior:**

```
Component Start 1: bdSeq=0 → reconnect → bdSeq=1 → reconnect → bdSeq=2
Component Restart: bdSeq=0 (resets)
Component Start 2: bdSeq=0 → reconnect → bdSeq=1 → reconnect → bdSeq=2
```

**Impact:**

* **Acceptable** for most Sparkplug deployments where Edge Nodes naturally reset bdSeq on restart
* **Compatible** with brownfield deployments and development environments
* **Limitation** for deployments requiring persistent bdSeq across component restarts

### Recommendation

This stateless behavior is acceptable for the majority of Sparkplug B use cases. Many industrial Edge Node implementations also reset bdSeq on restart. If your specific use case requires persistent bdSeq across component restarts, consider using a dedicated Sparkplug B implementation with persistent storage capabilities.


# UNS (Output)

> **Works exclusively with UMH Core**

## 1 Quick-start (99 % of users)

```yaml
pipeline:
  processors:
    - tag_processor:            # or Bloblang / Node-RED JS
        defaults: |
          // Minimal example
          msg.meta.location_path = "enterprise.plant1.machiningArea.cnc-line.cnc5.plc123";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name      = "value";
          // tag_processor now auto-creates msg.meta.umh_topic
          return msg;

output:
  uns: {}                       # nothing else needed on UMH Core. Will automatically use msg.meta.umh_topic from the tag_processor and write to the internal redpanda.
```

*Open the **Topic Browser** (Management Console → Unified Namespace) to watch the live values.*

## 2 - Optional overrides

```yaml
output:
  uns:
    umh_topic:      "${! meta(\"umh_topic\") }"   # Must follow `umh.v1.<…>` naming. If not specified, will take the `umh_topic` from the metadata fields (e.g., from msg.meta.umh_topic)
    bridged_by:     "umh-core"                    # Traceability header. Default `umh-core`; overridden automatically by protocol-converters inside UMH Core.
```

## 3 - What the plugin does behind the scenes

1. **Batching** 100 messages *or* 100 ms – whichever comes first.
2. **Sanitising** Illegal chars in the key become "\_".
3. **Kafka/Redpanda** Each message will be stored in the Kafka topic `umh.messages` with the Kafka key of the umh\_topic
4. **Headers** All Benthos metadata (except `kafka_*`) plus `bridged_by` are forwarded as Kafka headers.
5. **Topic check** If `umh.messages` is missing the plugin creates it (1 partition, `compact,delete`).

## Troubleshooting / FAQs

* **"topic is not set or is empty"** – your pipeline never wrote `msg.meta.umh_topic`. Add a `tag_processor` (auto) or a Bloblang line: `meta umh_topic = "umh.v1.demo.plant1.line1._historian.temperature"`
* **I am not using umh-core, but I still want to use the uns output plugin. How can I do that?** - there is a configuration variable called `broker_address` which you can point to any redpanda broker.


# OPC UA (Output)

The **OPC UA output** plugin writes data into an OPC UA server (e.g., a PLC). This plugin supports optional read-back (handshake) to confirm the write and dynamic configuration through interpolated fields.

> **Data Transformations**\
> It is recommended to perform JSON-to-field transformations *before* this plugin (e.g., via Node-RED JavaScript or [Bloblang](https://www.benthos.dev/docs/guides/bloblang/about)). That way, you feed the final fields directly to this plugin without extra logic here.

## Dynamic Configuration Support

The OPC UA output plugin supports interpolated strings for the following fields:

* `nodeId` - Dynamically determine which OPC UA node to write to based on message content
* `dataType` - Dynamically determine the OPC UA data type based on message content

This enables use cases such as:

* Routing writes to different nodes based on message content
* Dynamically selecting data types based on message metadata
* Building flexible, reusable configurations

***

**Basic Configuration**

```yaml
output:
  opcua:
    # endpoint, username, password, securityMode, securityPolicy, serverCertificateFingerprint, clientCertificate
    # see OPC UA Input for more information

    # Static configuration example
    nodeMappings:
      - nodeId: "ns=2;s=MySetpoint"
        valueFrom: "setpoint"   # The JSON field to write
        dataType: "Int32"       # OPC UA data type (required)
      - nodeId: "ns=2;s=MyEnableFlag"
        valueFrom: "enable_flag"
        dataType: "Boolean"     # OPC UA data type (required)

    handshake:
      enabled: true               # enable read-back
      readbackTimeoutMs: 2000     # how long to wait for the new value to appear

      maxWriteAttempts: 3         # how many times to retry if the write fails
      timeBetweenRetriesMs: 1000  # time (ms) between write retries
```

**Supported Data Types**

The OPC UA output plugin supports the following commonly used data types for writing to OPC UA servers:

* `Boolean`: True/false values
* `Byte`: 8-bit unsigned integer (0 to 255)
* `SByte`: 8-bit signed integer (-128 to 127)
* `Int16`: 16-bit signed integer
* `UInt16`: 16-bit unsigned integer
* `Int32`: 32-bit signed integer
* `UInt32`: 32-bit unsigned integer
* `Int64`: 64-bit signed integer
* `UInt64`: 64-bit unsigned integer
* `Float`: 32-bit floating-point number
* `Double`: 64-bit floating-point number
* `String`: UTF-8 encoded string
* `DateTime`: Date and time values

**Note:** When selecting a data type, ensure it matches the expected type on the OPC UA server. Mismatched types may cause write operations to fail or data to be interpreted incorrectly.

**Fields:**

| Field                              | Description                                                                                                                                                                                                                                                                                                                                                                              |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **nodeMappings** (array)           | <p>List of nodes to write to, each with:<br>- <code>nodeId</code>: The OPC UA Node ID, e.g., <code>"ns=2;s=MyVariable"</code><br>- <code>valueFrom</code>: The JSON field name (string) in your message containing the final value.<br>- <code>dataType</code>: The OPC UA data type for the value (e.g., <code>"Boolean"</code>, <code>"Int32"</code>, <code>"Double"</code>, etc.)</p> |
| **handshake**                      | (Optional) A sub-config for read-back validation.                                                                                                                                                                                                                                                                                                                                        |
| **handshake.enabled**              | Default `true` (for safe writes). If `true`, the plugin attempts to read the node(s) after writing. If the new value doesn’t match, the write is considered failed.                                                                                                                                                                                                                      |
| **handshake.readbackTimeoutMs**    | How long to wait for the server to show the updated value. If it times out, the plugin fails (Benthos will not ACK the message).                                                                                                                                                                                                                                                         |
| **handshake.maxWriteAttempts**     | Number of write attempts if the server fails (default 1).                                                                                                                                                                                                                                                                                                                                |
| **handshake.timeBetweenRetriesMs** | Delay between write attempts (default 1000 ms).                                                                                                                                                                                                                                                                                                                                          |

***

**Usage Example**

**Incoming Message** (e.g., from a Benthos input or pipeline):

```json
{
  "setpoint": 123,
  "enable_flag": true
}
```

**Plugin Configuration**:

```yaml
output:
  opcua:
    endpoint: "opc.tcp://192.168.0.10:4840"
    nodeMappings:
      - nodeId: "ns=2;s=MySetpoint"
        valueFrom: "setpoint"
        dataType: "Int32"
      - nodeId: "ns=2;s=MyEnableFlag"
        valueFrom: "enable_flag"
        dataType: "Boolean"

    handshake:
      enabled: true
      readbackTimeoutMs: 2000
      maxWriteAttempts: 3
      timeBetweenRetriesMs: 1000
```

**Behavior**:

1. The plugin connects to the OPC UA server at `opc.tcp://192.168.0.10:4840`.
2. It **writes** the field `setpoint=123` to node `ns=2;s=MySetpoint` and `enable_flag=true` to `ns=2;s=MyEnableFlag`.
3. Immediately after, it **reads** these nodes back:
   * If the server now reports `MySetpoint=123` and `MyEnableFlag=true`, the write is considered successful.
   * If the read times out after 2 s or shows a different value, the plugin fails the write. Benthos will *not* ACK the message upstream, so it can be retried or routed to an error output.
4. Assuming success, the message is **acknowledged** and removed from the pipeline.

If, for example, the `ns=2;s=MyEnableFlag` node is read-only or the server rejects the update, the read-back will fail, causing the plugin to retry up to 3 times (`maxWriteAttempts`). If all attempts fail, Benthos escalates the failure.

***

**Handshake & Acknowledgment**

By default, the plugin **reads back** each node it wrote to confirm the new value appears. This ensures:

1. **Benthos Message ACK**: If the read-back fails or times out, the output plugin fails. Benthos will *not* acknowledge the message upstream, and you can configure fallback or retry strategies.
2. **Consistent Setpoints**: If the OPC UA server discards or modifies the value, you’ll see an immediate error.

> **Disable** the handshake by setting `handshake.enabled: false` if you prefer no read-back check (faster, but less safe).

**Example**: If you disable the handshake:

```yaml
handshake:
  enabled: false
```

The plugin will write to the OPC UA server but **not** confirm. It will “succeed” as soon as the write request is sent.

***

## Dynamic Configuration Example

The OPC UA output plugin supports dynamic node IDs using interpolated fields:

```yaml
output:
  opcua:
    endpoint: "opc.tcp://localhost:4840"

    # Dynamic node mapping based on message content
    nodeMappings:
      - nodeId: "${! json(\"target_node\") }"     # Node ID from message field
        valueFrom: "value"
        dataType: "${! json(\"dataType\") }"      # Data type from message field
```

**Example with message routing:**

```yaml
pipeline:
  processors:
    - mapping: |
        # Route to different nodes based on tag type
        root.target_node = match this.tag_type {
          "color" => "ns=2;s=[default]/PaintRoom1/Spraytan1/Color",
          "speed" => "ns=2;s=[default]/PaintRoom1/Spraytan1/Speed",
          "temp"  => "ns=2;s=[default]/PaintRoom1/Spraytan1/Temperature"
        }
        # Set appropriate data type based on tag type
        root.target_datatype = match this.tag_type {
          "color" => "String",
          "speed" => "Double", 
          "temp"  => "Float"
        }
        root.value = this.tag_value

output:
  opcua:
    endpoint: "opc.tcp://plc.factory.local:4840"
    nodeMappings:
      - nodeId: "${! json(\"target_node\") }"
        valueFrom: "value"
        dataType: "${! json(\"target_datatype\") }"
```

This enables building reusable configurations that adapt to different environments and message types without hardcoding node paths.

***

**Implementation Details & Future Outlook**

For many industrial use cases, you might need more than just writing a value and reading it back:

1. **De-duplication**: If you re-send the same “command” multiple times, do you want the PLC to ignore duplicates?
   * *Now*: Implement a unique command ID (UUID) in your message and let the PLC store/ignore duplicates. Or handle it in your Benthos pipeline (e.g., a “dedupe” processor).
   * *Future*: We may add a built-in “ActionUUID” handshake, which compares a known ID in another read node.
2. **Time-Window Checks**: Only accept a command if it arrives before a certain expiration.
   * *Now*: Use a preceding nodered\_js processor or Bloblang to drop the message if `timestamp_now - msg.timestamp > threshold`.
   * *Future*: We might add plugin-level config like `rejectOlderThanMs` if demand arises.
3. **Separate Acknowledgment Node**: Some PLCs use a separate ack node (e.g., `CommandAck`) that signals the command was *processed*.
   * *Now*: Implement in the PLC + a custom “double read” with a second plugin instance (or a separate input that waits for the ack).
   * *Future*: We may add an advanced handshake config that reads a different node (rather than the same node) and checks for a specific “ACK” value.

With Benthos, the “at least once” acknowledgment ensures that if writing fails, the message can be retried or routed. This plugin’s minimal default handshake (read-back from the same node) is a strong start for safer OPC UA setpoints, and we’ll grow it over time if more advanced scenarios are needed.


# TimescaleDB Historian (Output)

Saves one UNS data contract into TimescaleDB using the UMH Historian schema. The plugin owns the schema bootstrap, the value/attribute writes, metadata de-duplication, and the datatype/conflict guards, so a bridge write flow is just an input and this output. No JavaScript processor or hand-written `sql_raw` is needed.

## Prerequisites

* PostgreSQL 16+ with the TimescaleDB and `ltree` extensions available (16+ so `ltree` labels accept hyphens).
* A non-superuser owner role, created once before the bridge starts (the bridge logs in as this role and cannot create it itself). It creates and owns the dedicated `umh` schema via the database-level grant, so no privilege on `public` is needed:

  ```sql
  CREATE ROLE umh_owner WITH LOGIN PASSWORD 'change-me';
  GRANT CREATE, CONNECT ON DATABASE umh TO umh_owner;
  ```

## Configuration

| Field                                | Required | Default     | Description                                                                                                                                                                                                                                                                                          |
| ------------------------------------ | -------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `host`                               | yes      | —           | TimescaleDB/Postgres host.                                                                                                                                                                                                                                                                           |
| `port`                               | no       | `5432`      | Port.                                                                                                                                                                                                                                                                                                |
| `database`                           | no       | `umh`       | Database name.                                                                                                                                                                                                                                                                                       |
| `username`                           | no       | `umh_owner` | Login role.                                                                                                                                                                                                                                                                                          |
| `password`                           | yes      | —           | Role password (plaintext in config; redacted in logs).                                                                                                                                                                                                                                               |
| `sslmode`                            | no       | `require`   | `require` \| `disable` \| `verify-full`.                                                                                                                                                                                                                                                             |
| `sslrootcert` / `sslcert` / `sslkey` | no       | `""`        | TLS cert paths inside the container.                                                                                                                                                                                                                                                                 |
| `data_contract_name`                 | yes      | —           | Bare lowercase contract name, e.g. `pump`; no leading `_`, no `_vN` suffix. Stored in `umh.tag.data_contract_name` in its UNS form with a leading underscore (`_pump`), matching the topic's data-contract segment.                                                                                  |
| `metadata_keys_all`                  | no       | `true`      | Store every metadata key except structural/high-churn keys and any `metadata_keys_exclude` match.                                                                                                                                                                                                    |
| `metadata_keys`                      | no       | `[]`        | Allowlist used only when `metadata_keys_all=false`.                                                                                                                                                                                                                                                  |
| `metadata_keys_exclude`              | no       | `[]`        | Blacklist applied only when `metadata_keys_all=true`. Each entry is an exact key name or a trailing-`*` prefix (e.g. `opcua_*`); matches are dropped on top of the built-in exclusions. A bare `*` drops everything. Ignored in allowlist mode.                                                      |
| `compress_after`                     | no       | `168h`      | Compress chunks older than this. **Applied once at first bootstrap** — changing it in the bridge afterward has no effect (see [Changing compression or retention](#changing-compression-or-retention)).                                                                                              |
| `retention`                          | no       | `""`        | Drop chunks older than this; empty keeps data forever. **Applied once at first bootstrap** — changing it in the bridge afterward has no effect (see [Changing compression or retention](#changing-compression-or-retention)).                                                                        |
| `batching`                           | no       | —           | benthos batch policy (`count` / `period` / `byte_size`). The whole batch is written in one transaction, so larger batches raise throughput; e.g. `count: 1000`, `period: 1s`.                                                                                                                        |
| `max_in_flight`                      | no       | `8`         | Batches written to the database concurrently. Throughput scales with this and with batch size (see Throughput below).                                                                                                                                                                                |
| `write_timeout`                      | no       | `""`        | Per-batch write timeout as a Go duration (e.g. `30s`). Empty/`0s` means no timeout (a write hung on a lock or half-open connection blocks until the context is cancelled). When set, a timed-out batch is held for retry (NACK), never dropped; set it above the largest expected batch commit time. |

## What it writes

All objects live in a dedicated `umh` schema. For `data_contract_name: pump`, the plugin creates and writes two hypertables:

* **`umh.value_pump`** — one row per `(topic_id, ts)`, where `ts` is a `timestamptz`. Numbers and booleans land in `value_num`, strings and JSON in `value_text`.
* **`umh.attribute_pump`** — the message metadata as a JSON object, queryable via `attribute->>'key'` and `attribute @> '{...}'`.

`umh.get_topic_id(location_path, virtual_path, data_contract, tag_name)` resolves a tag to its `topic_id` for ad-hoc and Grafana queries.

> **Note on the contract name.** You configure the bare form (`pump`), which is used verbatim in the table names (`umh.value_pump`, `umh.attribute_pump`). The `umh.tag.data_contract_name` *column*, however, stores the UNS form with a leading underscore (`_pump`) to match the topic's data-contract segment. This mirrors the ManagementConsole Historian template, so a database written by either resolves identically through `get_topic_id`.

## Reading the data

The value table stores a surrogate `topic_id`, not the location/tag names. To go from a value row back to its identity, join through `umh.topic` to `umh.tag` and `umh.location`:

```
umh.value_pump (topic_id, ts, value_num, value_text)
       │ topic_id
       ▼
umh.topic (topic_id, location_id, tag_id)
       │ tag_id            │ location_id
       ▼                   ▼
umh.tag (tag_id, name,   umh.location (location_id, path)
         virtual_path,
         data_contract_name)
```

Two things trip up hand-written queries:

* The value timestamp column is **`ts`** (a `timestamptz`), not `timestamp` or `time`.
* A tag with no virtual path stores `virtual_path` as the **empty string `''`**, never `NULL`. Passing `NULL` to `get_topic_id` matches nothing and returns an empty result silently.

`umh.get_topic_id(location_path, virtual_path, data_contract, tag_name)` hides that join for single-tag lookups. Its `data_contract` argument is forgiving — `pump`, `_pump`, and `_pump_v1` all resolve to the same tag — so you don't have to remember the exact underscore/version form.

```sql
-- Latest value of one tag. Use '' (not NULL) when the tag has no virtual path.
SELECT ts, value_num, value_text
FROM   umh.value_pump
WHERE  topic_id = umh.get_topic_id('enterprise.site.area.line', '', 'pump', 'temperature')
ORDER  BY ts DESC
LIMIT  1;

-- Values of one tag over a time window (drop-in for a Grafana panel; the
-- WHERE ts line is what Grafana's $__timeFilter(ts) macro expands to).
SELECT ts, value_num
FROM   umh.value_pump
WHERE  topic_id = umh.get_topic_id('enterprise.site.area.line', '', 'pump', 'temperature')
  AND  ts BETWEEN now() - INTERVAL '1 hour' AND now()
ORDER  BY ts;

-- Current value of every tag in the contract, with names resolved.
SELECT DISTINCT ON (v.topic_id)
       l.path::text AS location, g.virtual_path, g.name AS tag, v.ts, v.value_num, v.value_text
FROM   umh.value_pump v
JOIN   umh.topic    t ON t.topic_id    = v.topic_id
JOIN   umh.tag      g ON g.tag_id      = t.tag_id
JOIN   umh.location l ON l.location_id = t.location_id
ORDER  BY v.topic_id, v.ts DESC;

-- Resolve a tag to its numbers: value row → topic → tag/location.
SELECT l.path::text AS location, g.name AS tag, v.ts, v.value_num
FROM   umh.value_pump v
JOIN   umh.topic    t ON t.topic_id    = v.topic_id
JOIN   umh.tag      g ON g.tag_id      = t.tag_id
JOIN   umh.location l ON l.location_id = t.location_id
WHERE  g.name = 'temperature'
ORDER  BY v.ts DESC;
```

> **`DISTINCT ON` and high tag counts.** The "current value of every tag" query above scans the history of every topic to find each one's newest row. That is fine for hundreds of tags but gets expensive as the tag count and history grow. For a dashboard that refreshes it often, back it with a TimescaleDB continuous aggregate holding `last(value_num, ts)` per `topic_id` and query that instead.

## Behavior

* **Startup check.** `Connect()` verifies the server version and bootstraps the schema, so an unreachable, too-old, or misconfigured database fails the bridge at startup rather than writing to a misconfigured database unnoticed.
* **Idempotent replays.** An identical value at the same `(tag, ts)` is absorbed.
* **Topic resolution is read-first.** A topic already in the database is resolved with a lookup that assigns no new id, so the internal surrogate ids advance only when a genuinely new topic is created — not per message, and restarts do not bump them.
* **Conflict and datatype guards drop the offending row.** A *different* value at the same `(tag, ts)`, or a tag whose datatype flips (numeric ↔ text), is rejected by the database and the row is dropped rather than overwriting history — the rest of the batch is still written (see [Error handling](#error-handling)). This includes a tag emitting two distinct values within one millisecond, which the millisecond UNS timestamp cannot distinguish from a real conflict, so this contract is unsuitable for tags that emit distinct values faster than 1 kHz.
* **Malformed messages are dropped, not nacked.** A wrong `data_contract`, an absent or invalid `umh_topic` (validated by the canonical topic parser), a non-finite number, or an unparseable timestamp drop the message and increment the `historian_messages_dropped` metric (labelled by `reason`), so one bad message never stalls the stream.
* **Metadata de-duplication.** An attribute row is rewritten only when its key set changes, via an in-process, LRU-bounded fingerprint cache. The cache is process-local and cleared on restart, so the plugin re-emits at most one attribute row per topic per restart: the first post-restart message lands at a new timestamp, so its identical-metadata row is written as a new `(topic_id, ts)` row rather than being absorbed by the conflict guard.

## Error handling

A write failure is handled by *what caused it*, so a single bad tag never stalls the stream:

* **Transient** (connection loss, serialization/deadlock, lock contention, operator intervention, and any error without a SQLSTATE) — the batch is retried until it succeeds. A DB restart mid-stream loses nothing: held messages replay and identical `(topic_id, ts)` rows are absorbed.
* **Standing fault** (missing table privilege, disk full, an unrecognized error) — retried too (good data is never dropped over a fixable problem), but logged at error level. The bridge does not progress until an operator fixes the cause, then resumes losslessly.
* **Poison** (a value that can never be written: an append-only conflict, a datatype flip, a constraint violation) — the offending row is dropped and counted on `historian_rows_poisoned` (labelled by `sqlstate` and `phase`), with an error log naming the tag. The rest of the batch is written. Retrying a poison row can never succeed, so dropping it is what keeps every other tag flowing.

Only poison rows are ever dropped on a write error. Oversized text is a separate case: a `value_text` longer than the row limit is clipped and counted on `historian_values_truncated` (previously silent).

`Connect` also verifies the login role can `INSERT` into the contract's tables (a `has_table_privilege` check). A role that reaches the database but cannot write to it fails the bridge at startup with a named error, instead of connecting and then stalling on every write.

## Runbook: poisoned tags

**Find them.** A non-zero `historian_rows_poisoned` counter means rows are being dropped. The error log names each one: `dropped poison row at <phase> for contract=… location=… virtual_path=… tag=… (sqlstate=…)`. `phase=resolve` with `sqlstate=P0001` is almost always a **datatype flip**; `phase=value` with `P0001` is an **append-only conflict** (two different values at the same millisecond).

**Datatype flip / accidental first type.** A tag's type is fixed by its first stored value: one stray string (e.g. `"N/A"`) locks the tag to text, and later numeric readings are then rejected. This only arises on generic contracts like `_historian` that carry no upstream type validation; a modelled contract validates types before the historian ever sees them. Confirm the established type, then decide:

```sql
-- what type is this tag locked to?
SELECT value_type FROM umh.tag
WHERE name = 'temperature' AND virtual_path = '' AND data_contract_name = '_historian';
```

To reset a tag that was locked to the wrong type, delete its stored value history and its tag row so the next message re-establishes the type (this discards that tag's history for the contract — take a copy first if you need it):

```sql
-- resolve the topic, delete its values, then remove the topic + tag so the type is no longer pinned
WITH tid AS (SELECT umh.get_topic_id('enterprise.site.area.line', '', 'historian', 'temperature') AS id)
DELETE FROM umh.value_historian WHERE topic_id = (SELECT id FROM tid);
-- then delete the matching rows in umh.topic and umh.tag.
```

**Append-only conflict.** The source emitted two different values at the same millisecond timestamp. On bridges the downsampler collapses duplicate timestamps per series before the historian sees them; if you hit this, the source is producing faster than 1 kHz on one tag — not representable by the millisecond UNS timestamp and unsuitable for this contract.

**Prevention.** Pin the intended type on fixed contracts (don't let an accidental first sample define it), and route text or high-precision counters to a text contract rather than mixing types on one tag.

> **Generic contracts (`_historian`/`_raw`).** These deliberately don't pin a type, so a type change is a realistic operational event rather than a defect. How the plugin should treat a type change there — reject as poison (today), tolerate both `value_num` and `value_text`, or promote the tag to text — is an open policy decision tracked separately; today it is dropped as poison like any other flip.

## Throughput

Each batch is written in one transaction: the distinct topics are resolved once, then value and attribute rows are inserted by `topic_id`. Two knobs scale write throughput, and both help independently:

* **`batching`** — a larger batch amortizes the single per-batch commit over more rows. Set a `count` / `period` policy (e.g. `count: 1000`, `period: 1s`); without one the output writes whatever the pipeline delivers per transaction.
* **`max_in_flight`** — more batches written concurrently. Because topics are resolved in short-lived statements (not held for the whole batch), concurrent batches do not serialize on the shared dimension rows, so throughput scales with this.

The defaults (`max_in_flight: 8` and a `count: 1000` / `period: 1s` batch policy) comfortably exceed a typical per-bridge load. Raise `max_in_flight` (and the connection pool with it) or the batch size for higher-throughput streams.

## Metrics

On top of benthos's built-in output metrics (`output_sent`, `output_error`, `output_latency_ns`), the plugin emits:

* `historian_value_rows_written` — value rows upserted (counted after the batch commits).
* `historian_attribute_rows_written` — attribute rows upserted; the gap below the value-row count is metadata de-duplication at work.
* `historian_messages_dropped` (labelled by `reason`) — messages dropped before any write.
* `historian_dedup_cache_size` — current dedup-cache entry count.

## Numeric precision

`value_num` is `DOUBLE PRECISION`. That is exact for sensor floats but loses precision for integer counters above 2^53 (\~9e15) and for exact decimals. Route such tags to a text data contract instead, where the value is stored verbatim in `value_text`.

## Location identity

The location is canonicalized into an `ltree` path: every character outside `[A-Za-z0-9_-]` becomes `_`, each label is truncated to 255 characters, and empty labels are dropped. Hyphens are kept (PostgreSQL 16+ `ltree` labels accept them), so `enterprise.line-1` and `enterprise.line_1` are **distinct** paths, each with its own `topic_id`. Other punctuation still folds: `enterprise.line@1` becomes `enterprise.line_1` and shares its identity. Distinguish sources by their path segments, not by punctuation that folds.

## Schema and compatibility

The plugin owns the schema: it bootstraps the baseline DDL into the `umh` schema idempotently on first connect and **never alters an already-created `umh.value_<contract>` / `umh.attribute_<contract>` table**. A breaking schema change ships as a new contract (new tables), never an in-place migration. (`ltree` stays in `public`, its conventional shared home.)

The baseline is a port of the Management Console TimescaleDB Historian template and writes the same tables. To avoid schema drift, a given contract/database must be written by exactly **one** writer type — the plugin **or** the template, never both.

## Changing compression or retention

`compress_after` and `retention` are applied **once, at first bootstrap**, and are deliberately not re-applied when the bridge restarts. Editing them in the bridge config therefore has **no effect** on a database that already has the tables. This is intentional: a config edit (or a form change in the Management Console) should not silently change how production history is compressed or — for retention — **deleted**. On restart the bridge logs a warning if the applied policy differs from the config, so drift stays visible, but it does not act on it.

To change them on an existing database, update the TimescaleDB policies directly, on **both** hypertables for the contract. For a contract named `pump`:

```sql
-- retention: keep 30 days (repeat for umh.attribute_pump)
SELECT remove_retention_policy('umh.value_pump', if_exists => true);
SELECT add_retention_policy('umh.value_pump', INTERVAL '30 days');

-- compression: compress chunks older than 7 days (repeat for umh.attribute_pump)
SELECT remove_compression_policy('umh.value_pump', if_exists => true);
SELECT add_compression_policy('umh.value_pump', INTERVAL '7 days');
```

To stop dropping data entirely (the `retention: ""` default), remove the retention policy and do not re-add it. Changing either policy takes effect immediately and never needs a bridge restart.

After changing a policy on the database, set the **same** value in the bridge config too. The config value is still what a fresh bootstrap of a new database uses, and matching it silences the drift warning on restart.

## Quick example

```yaml
input:
  uns:
    umh_topics:
      - '^umh\.v1\..*\._pump_v.*'
output:
  historian:
    host: timescaledb.example.com
    password: change-me
    data_contract_name: pump
```

To deploy a bridge against this output from the Management Console, use the **Historian** template in the Add Bridge wizard.