> This is part 1 of 2 of the full documentation (pages 1–100 of 109).
> The content is paginated: fetch every part to see all of it.
> Next part: https://docs.umh.app/llms-full.txt/1
> Page index: https://docs.umh.app/llms.txt

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
This site has an agentic ask interface you may use to query the documentation dynamically by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.umh.app/index.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

---

# Introduction

> **Looking for the old Kubernetes Helm stack?** See the [UMH Core vs UMH Classic FAQ](/umh-core-vs-classic-faq) to understand which edition fits your project and the current migration path.

## What is UMH Core?

UMH Core is a **single Docker container that turns any PC, VM, or edge gateway into an Industrial Data Hub**.

In manufacturing, every device speaks a different language - PLCs use OPC UA, sensors use Modbus, MES systems use REST APIs. Connecting them all creates a tangled mess of point-to-point integrations. Change one thing, break ten others.

UMH Core creates a **Unified Namespace (UNS)** - a central data backbone where all your industrial data lives in one organized, validated place. Instead of 100 devices talking to each other (creating 1000s of connections), they all publish to one place, and consumers subscribe to what they need.

### What's Inside

That one Docker container includes everything you need:

* **Redpanda** – an embedded, Kafka-compatible broker that buffers every message.
* **Benthos-UMH** – a stream-processor engine with 50+ industrial connectors.
* **Agent** – a Go service that reads `config.yaml`, launches pipelines, watches health, and phones home to the Management Console.
* **S6 Supervisor** – keeps every subprocess alive and starts them in the correct order.

### Why teams pick UMH Core

| Benefit         | What it means in practice                                                                                               |
| --------------- | ----------------------------------------------------------------------------------------------------------------------- |
| **Simple**      | PLCs, sensors, ERP/MES, cloud services all talk to **one Unified Namespace (UNS)** instead of point-to-point spaghetti. |
| **Lightweight** | Runs on almost everything                                                                                               |
| **No lock-in**  | 100 % open-source stack: Redpanda, Benthos, S6, and much more                                                           |

### Core Concepts You'll Learn

Through our getting-started guide, you'll understand:

* **Instance** – A running UMH Core container identified by its location path
* **Management Console** – Cloud UI for deploying and managing instances without touching YAML
* **Unified Namespace (UNS)** – The event-driven data backbone that eliminates point-to-point connections
* **Bridge** – The gateway for external data into the UNS (**the ONLY way data enters**)
* **Topic** – How data is addressed: `location.contract.virtual_path.tag_name`
* **Tag** – A time-series data point (like a PLC variable or sensor reading)
* **Virtual Path** – Folder organization within topics for grouping related data
* **Data Model** – Templates that define and validate data structure
* **Data Contract** – Validation rules (`_raw` = no validation, `_modelname_v1` = enforced structure)

Advanced concepts (after getting-started):

* **Stream Processor** – Transforms messages already inside the UNS (e.g., device models → business models)
* **Stand-alone Flow** – Point-to-point when UNS buffering isn't wanted
* **State Machines** – Component lifecycle management (active/idle/degraded states)

### How It Works

```mermaid
flowchart LR
    subgraph Floor["Your Factory Floor"]
        PLC["PLCs (S7, Modbus)"]
        SEN["Sensors (OPC UA)"]
        MES["MES/ERP (REST)"]
    end
    subgraph Core["UMH Core"]
        UNS["Unified Namespace<br/>(organized data)"]
    end
    subgraph Sys["Your Systems"]
        DASH["Dashboards"]
        CLOUD["Cloud/MQTT"]
        DB["Databases"]
    end
    PLC -->|Bridge| UNS
    SEN -->|Bridge| UNS
    MES -->|Bridge| UNS
    UNS -->|Bridge| DASH
    UNS -->|Bridge| CLOUD
    UNS -->|Bridge| DB
```

1. **Bridges** connect your devices to the UNS (50+ protocols supported) - they're the ONLY entry point
2. **Data flows** into organized topics: `enterprise.site.area.line._contract.virtual_path.tag`
3. **Models** validate critical data (optional but recommended for production)
4. **Consumers** subscribe to the data they need
5. **Configure** via Management Console UI or directly edit YAML files

*Every message is buffered, validated, and organized - no data loss, guaranteed structure.*

## Getting Started

Sign up at [management.umh.app](https://management.umh.app) and deploy your first instance in 60 seconds through the UI.

**Prefer step-by-step learning?** Follow our progressive 4-step guide:

1. [**Install UMH Core**](/getting-started) - One Docker command (5 minutes)
2. [**Connect Your First Data**](/getting-started/1-connect-data) - Create a Bridge and see data flow (10 minutes)
3. [**Organize Your Data**](/getting-started/2-organize-data) - Scale from 1 to 1000s of tags automatically (15 minutes)
4. [**Validate Your Data**](/getting-started/3-validate-data) - Add quality control with Data Models (20 minutes)

By the end, you'll have production-ready data pipelines with validation, organization, and monitoring.

## Documentation Structure

* [**Usage Guides**](/usage) - Step-by-step implementation guides
  * [**Instances**](/usage/instances) - Managing and configuring UMH Core deployments
  * [**Unified Namespace**](/usage/unified-namespace) - Core messaging architecture
  * [**Data Flows**](/usage/data-flows) - Connect and process data streams
  * [**Data Modeling**](/usage/data-modeling) - Enterprise data structuring
  * [**Management Console**](/management-console) - Cloud-based control center
* [**Production Deployment**](/production) - Scaling, security, and operations
* [**Reference Documentation**](/reference) - Complete API and configuration reference

## Learn More About UNS

For deeper understanding of the concepts behind UMH Core:

* [**The Unified Namespace Course Series**](https://learn.umh.app/featured/) - 4-chapter comprehensive course
  * [Chapter 1: OT Foundations](https://learn.umh.app/lesson/chapter-1-the-foundations-of-the-unified-namespace-in-operational-technology/) - Automation pyramid challenges
  * [Chapter 2: The Rise of UNS](https://learn.umh.app/lesson/chapter-2-the-rise-of-the-unified-namespace/) - Core architecture principles
  * [Chapter 3: IT Foundations](https://learn.umh.app/lesson/chapter-3-the-foundations-of-the-unified-namespace-in-information-technology/) - Modern IT patterns
* [**Industrial IoT Architecture**](https://learn.umh.app/blog/cloud-native-technologies-on-the-edge-in-manufacturing/) - Edge computing in manufacturing
* [**MQTT vs UNS Comparison**](https://learn.umh.app/blog/what-is-mqtt-why-most-mqtt-explanations-suck-and-our-attempt-to-fix-them/) - Why data contracts matter

## Community & Support

* [**Discord Community**](https://discord.gg/F9mqkZnm8U) - Get help and connect with other users
* [**GitHub Repository**](https://github.com/united-manufacturing-hub/united-manufacturing-hub) - Source code and issue tracking
* [**Management Console**](https://management.umh.app) - Web-based configuration and monitoring
* [**UMH Website**](https://www.umh.app/) - Company and product information


# Getting Started

**60 seconds — that's all it takes to launch UMH Core.**

No kubectl, no Kubernetes setup, just a single Docker container. Almost anything that runs Docker works — from your MacBook to enterprise edge boxes.

> **Note:** This getting-started guide follows the common PLC/sensor path (raw → device models). For ERP integration and other patterns, see the [Data Modeling documentation](/usage/data-modeling).

## System requirements

**Minimum:**

* 2 vCPU
* 4 GB RAM
* 40 GB SSD

**Architecture:**

* x86\_64 (Intel/AMD 64-bit)
* ARM64 (64-bit ARM, but NOT Raspberry Pi - Redpanda requires 48-bit virtual address space)

**CPU health reporting (early access):**

UMH Core runs on any of the above. Its CPU pressure signal also needs Linux pressure stats (PSI) and cgroup v2, which the Red Hat family ships off. Without them CPU health still reports from throttling, steal and headroom, and an instance with no CPU limit reports limited visibility. [CPU Health](/production/cpu-health) covers how to check, how to switch them on, and which flag enables this reporting.

For detailed sizing recommendations, see the [Sizing Guide](/production/sizing-guide).

## Installation using the Management Console

1. **Sign up** at [management.umh.app](https://management.umh.app)
2. Click **"Add a new instance"**

![Instance setup screen](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-f21251cffe665a1b2a2cd0ff48f3651896236c38%2Finstance-setup.png?alt=media)

3. **Configure your instance:**
   * **Name** (1): Give it a cool name (e.g., `my-first-umh`)
   * **Type** (2): Select "Core" from the dropdown
   * **Location** (3): Enter your location path:
     * **Level 0**: `your-company` (required - your enterprise name)
     * **Level 1**: `Your level 1 name` (optional - e.g., site or plant)
     * **Level 2-4**: Leave empty for now (optional)
   * Click **"Add Instance"** (4)
4. **Copy the Docker command** that appears
5. **Run it** on any machine with Docker installed

## 🎉 Success!

Your instance will appear as **"Online"** in the Management Console within seconds.

**Next step:** [Connect your first data source →](/getting-started/1-connect-data)

***

## Troubleshooting

**Volume permission errors** Named volumes automatically handle permissions for the container user (UID 1000). If you still see permission errors like `s6-svscan: warning: unable to stat benthos-dataflow-*`, try removing the container and recreating the volume:

```bash
docker rm -f umh-core && docker volume rm umh-core-data && docker volume create umh-core-data
```

Then re-run the `docker run` command.

For advanced users needing bind mounts (custom data locations), see the [Container Layout reference](/reference/container-layout#advanced-custom-data-location).

**Corporate firewall/proxy issues** If your corporate network intercepts TLS traffic, see [Network Configuration](/production/security/deployment-security/network-configuration#tls-inspection-mitm) to add your CA certificate.

For proxy configuration, see [Network Configuration](/production/security/deployment-security/network-configuration#proxy-configuration) or add these arguments to your docker run:

```bash
-e HTTP_PROXY=http://proxy.company.com:8080 \
-e HTTPS_PROXY=https://proxy.company.com:8080 \
-e NO_PROXY=localhost,127.0.0.1,.local
```

***

## Coming from UMH Classic?

See the [Migration Guide](/production/migration-from-classic) for step-by-step migration instructions, including data contract changes and configuration updates.

***

## Concepts Learned

* **UMH Core** - Single-container Industrial IoT platform
* **Management Console** - Cloud UI for managing instances
* **Instance** - A running UMH Core deployment
* **Location hierarchy** - Organizational structure (enterprise → site → area → line)
* **Docker container** - How UMH Core runs

## Ready to Connect Data?

**You've successfully installed UMH Core!** 🚀

**Next step:** [Connect your first data source →](/getting-started/1-connect-data)


# Step 2: Connect Data

Let's get some data flowing! We'll start with simulated data, then you can connect real devices later.

This guide walks you through connecting your first data source step by step. If you're looking for the complete bridge reference, see [Bridges](/usage/data-flows/bridges).

## Navigate to Data Flows

1. Click **"Data Flows"** in the left menu
2. Go to the "Bridges" tab. Here you find all of your bridges.
3. Click **"Add Bridge"**

![Data Flow](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-b500632337088925b2861a57b0886c4f5af5baae%2Fdata-flow.png?alt=media)

## Choose How to Create the Bridge

The UI will ask you how to start

* **From Scratch**: Start building from one of our curated templates.
* **From Existing Bridge**: Copy the configuration of an existing bridge.

For this tutorial, pick **"From Scratch"**.

![Create a New Bridge](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-3af8ee60ecd571ae0a4755c8e75f6a318cab5359%2Fbridge-create-new.png?alt=media)

## Select the `generate` Template

You'll land on a list of curated templates.

For this tutorial, type `generate` in the search bar and one template remains: **"Generic via Generate"** (Benthos `generate` input). Select it.

> 💡 **`generate`** is one of the inputs that Redpanda provides out of the box.
>
> [Learn more about the `generate` input here.](https://docs.redpanda.com/connect/components/inputs/generate/)

By default, this template produces `hello world` messages, so you don't need a real PLC to finish the tutorial.

![Select a Template](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-721c4bb82c4d804b25a26de13b63ffcadcb6ed91%2Fbridge-select-template.png?alt=media)

## Configure the Bridge - General Tab

The bridge editor has three tabs: **General**, **Read Flow**, and **Write Flow**.

On the **General** tab:

* **Name:** `my-first-generator-bridge`
* **Instance:** Select your UMH Core instance.
* **IP Address / Port:** The device to connect to. We have no real PLC for this tutorial, so use `localhost` and `8080`.
* **Location:** Pre-filled with your instance location. Leave it as is for now. In the screenshot we define a bridge for `enterprise.siteA`.

Once done, click **"Save & Deploy"**. A popup shows deployment progress and any errors or warnings. If everything goes well, you'll be redirected automatically.

![Bridge General Configuration: Entering IP, Port, Name & Instance.](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-e0df150c19e39796a9918faed5842b675061bcfe%2Fbridge-general.png?alt=media)

![Logs of a successful bridge deployment after clicking "Save & Deploy"](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-34a2ac74871b8c943d5fabf45fa2c3b7e75c2e00%2Fbridge-general-deploy.png?alt=media)

Back on the bridge, verify the connection succeeded. Here we see a latency of "0 ms" since we're connecting to localhost.

![Bridge Latency Good](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-5c1a8066b047e8feed359a116de1d8a5e70f8541%2Fbridge-general-latency.png?alt=media)

If connecting to a real PLC fails, the latency indicator turns orange:

![Bridge Latency Bad](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-42a3c04cfed3c64b07762f7d01ffb11510d9d9ba%2Fbridge-general-latency-bad.png?alt=media)

## Configure the Bridge - Read Flow Tab

Because you started from the "Generic via Generate" template, the protocol is already set to **Generate** and pre-filled with a template generator configuration.

In Data Type, make sure that **Time-Series** is selected and that the **Activate Flow** toggle is enabled.

![Bridge Read Input](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-12c5fedfb6b917777abec3cef14ec541302e0532%2Fbridge-read-input.png?alt=media)

* **Input:** The template pre-fills a generator. For real protocols like "Modbus" or "Siemens S7" you'd see protocol-specific settings here. Keep the defaults: generate `hello world` every `1s`.

![Bridge Read Processing](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-51ba1266380db6f020df8db23329d301f366aa64%2Fbridge-read-processing.png?alt=media)

* **Processing:** The "Tag Processor" appears because we selected "Time Series" data type.

Click the **Always** expansion panel to expand the view. The expand shows the processor code.

There are three required fields:

* **location\_path:** Where the data goes (auto-filled from bridge location)
* **data\_contract:** Leave as `_raw` for now (no validation rules)
* **tag\_name:** Name your tag (data point) - we'll use `my_data`

> 💡 **What's a tag?** In industrial systems, a "tag" is a single data point - like a temperature sensor reading, motor speed, or valve position. Think of it as a variable that changes over time.

The **Always** section uses JavaScript to process messages. We're not modifying anything for now, just passing the data through.

> 💡 **Tip:** You can modify data here later (e.g., unit conversions renaming). If you don't know JavaScript, any LLM (ChatGPT, Claude) can help write the code.

![Bridge Read Output](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-fa4972f0804e5aec67c384b6a4b68b593dfa6d71%2Fbridge-read-output.png?alt=media)

The Output section is auto-generated - it sends data to your Unified Namespace.

Click **"Save & Deploy"**.

![Bridge Read Deployed](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-13d419e08110d6e87c866c95b15e950a742cea23%2Fbridge-read-deployed.png?alt=media)

## 🎉 Success!

You should now see:

* **Status:** Active ✅
* **Throughput:** \~1 msg/sec

Your data is flowing!

## Step 3: View Your Data in the Topic Browser

![Topic Browser My Data](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-22c15c615c42c3ef45014a41f101fab59bfeb3b9%2Ftopic-browser-my_data.png?alt=media)

Click **"Topic Browser"** in the left menu. This shows all data in your Unified Namespace.

You'll see your data organized as a **topic**:

* `enterprise` → `siteA` → `_raw` → `my_data`

This is the full topic path: `enterprise.siteA._raw.my_data`

Exactly as we configured it! Click on `my_data` to see details.

* Topic Details shows your bridge configuration and data location
* Last Message shows the most recent `hello world` message with timestamp
* History shows a table of recent messages (would be a chart for numeric data)
* Metadata contains additional information about the data source (we'll use this later)

> 💡 **Only seeing one `hello world` row in your topics?**
>
> That's intended. Put simply, your data is processed by benthos-umh, which drops repeated, unchanged values. This is called **downsampling**.
>
> [Learn more about our benthos-umh downsampler here.](https://docs.umh.app/benthos-umh/processing/downsampler)

## Understanding What You Built

You just created a complete data pipeline:

```mermaid
flowchart LR
    A["Bridge"] --> B["Processing"] --> C["Unified Namespace"] --> D["Topic Browser"]
```

**Key Concepts:**

* **Bridge:** The ONLY way data enters UMH (ensures quality and monitoring)
* **Location Path:** Organizes your data hierarchically (`enterprise.siteA`)
* **Data Contract:** Currently `_raw` (no validation rules)
* **Tag:** Your data point (`my_data`)
* **Topic:** Complete address in the UNS (`enterprise.siteA._raw.my_data`)

## What's Next?

**You have data flowing!** This is already production-ready for many use cases.

**Want to organize better?** → [Continue to Step 3: Organize Your Data](/getting-started/2-organize-data)

**Connect a real device?** Simply change the protocol from "Generate" to:

* **OPC UA** for modern PLCs
* **Modbus** for older equipment
* **MQTT Subscribe** for existing MQTT devices
* [See all 50+ supported protocols →](https://docs.umh.app/benthos-umh/input)

Real hardware uses the address-mapping table instead of the generator. The [Bridges reference](/usage/data-flows/bridges) covers that, along with logs, metrics, and debug logging.

## Concepts Learned

Building on the location path from Step 1, you now understand:

* **Unified Namespace (UNS)** - Event-driven data backbone that eliminates point-to-point connections ([learn more](/usage/unified-namespace))
* **Bridge** - Gateway for external data into the UNS (the only way data enters)
* **Bridge Template** - Pre-configured starting point for a bridge (e.g. "Generic via Generate")
* **Data Flows** - Configuration area for bridges and other data pipelines
* **Protocol** - Connection method (manufacturing protocols like OPC UA, Modbus, S7, or IT protocols like HTTP)
* **Connection** - IP/hostname and port settings with latency monitoring
* **Tag** - A time-series data point (like a PLC variable)
* **Tag Processor** - JavaScript-based transformation for time-series data
* **Topic** - Complete data address: `location_path.data_contract.tag_name` ([topic convention](/usage/unified-namespace/topic-convention))
* **data\_contract** - Data validation rules (`_raw` = no validation)
* **Topic Browser** - UI for viewing all data in the UNS
* **Downsampler** - Drops repeated unchanged values so the UNS stays clean
* **Throughput** - Tag updates per second for time-series data

***

**Pro tip:** Everything you just configured in the UI is stored as YAML. As you get comfortable, you can copy and modify these configurations for faster setup of similar devices.


# Step 3: Organize Data

> **Prerequisite:** You should have data flowing from [Step 2](/getting-started/1-connect-data). If not, go back and complete that first!

## Understanding Messages

When data flows through bridges, each message contains:

* **payload**: The actual data value
* **meta**: Information about the message (source, destination, tag names, etc.)

In the processing code, you'll work with these as `msg.payload` and `msg.meta`.

## The Challenge

Right now you have one measurement going to one place:

```
enterprise.siteA._raw.my_data
```

But real PLCs have:

* Hundreds of data points with various address formats
* Different data types (numbers, strings, booleans)
* Multiple machines on the same line

**Do you need to map each tag manually?** No! Let's connect a real PLC and see how automatic mapping works.

## Part 1: Connect a Real PLC (Or Simulate One)

### Create a New Bridge for a Real PLC

We'll use Siemens S7 as an example, but this works identically with OPC UA, Modbus, or any other protocol.

1. Go to **Data Flows** → **Add Bridge**
2. **Name:** `s7-plc`
3. **Protocol:** Select **Siemens S7** (or your protocol)
4. **Connection:**
   * **IP:** Enter your PLC's IP address (example: 192.168.1.100)
   * **Rack:** 0
   * **Slot:** 1

![Protocol Selection](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-a1a93b5e93c311363bbe5dbdb366806a39fc0445%2F2-protocol.png?alt=media)

### Configure What to Read

In the **Input** section, you'll see protocol-specific settings:

```yaml
s7comm:
    addresses:
        - DB1.DW20      # Example address
        - DB1.S30.10    # Another address
        - DB3.I270      # Third address
    tcpDevice: '{{ .IP }}'
    rack: 0
    slot: 1
```

![S7 Input Configuration](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-04f5cfdb9f9e41ccbb803d24921fc40db57f43fa%2F2-input.png?alt=media)

### Automatic Tag Mapping

In the **Processing** section, look at the **Always** code:

```javascript
// Set location from bridge config
msg.meta.location_path = "{{ .location_path }}";

// Use _raw (no validation - data passes through as-is)
msg.meta.data_contract = "_raw";

// Key concept: Protocol address becomes the tag name automatically
msg.meta.tag_name = msg.meta.s7_address;  // For OPC UA: msg.meta.opcua_tag_name

// Pass the value through
msg.payload = msg.payload;

return msg;
```

**Key line:** `msg.meta.tag_name = msg.meta.s7_address`

Each protocol provides its address in metadata:

* Siemens S7: `msg.meta.s7_address`
* OPC UA: `msg.meta.opcua_tag_name`
* Modbus: `msg.meta.modbus_address`

The addresses automatically become tags - no manual mapping needed!

### Template Variables

When you entered your PLC's IP address in the Connection settings, it automatically became available as `{{ .IP }}` in your configuration code. Template variables connect your Connection settings to the processing code:

* `{{ .IP }}` - The IP address you entered in Connection settings
* `{{ .PORT }}` - The port you entered in Connection settings
* `{{ .location_path }}` - The location from Bridge configuration

This makes configurations reusable across different sites - just change the Connection settings and the template variables update automatically.

Learn more: [Template Variables Reference](/reference/variables)

Click **Save & Deploy**.

### See Your PLC Data

In **Topic Browser**, you now see ALL your PLC data automatically organized:

```
enterprise.sksk._raw.DB1.DW20     [12345]
enterprise.sksk._raw.DB1.S30.10   ["Product ABC"]
enterprise.sksk._raw.DB3.I270     [789]
```

(These are S7 addresses, but OPC UA NodeIDs or Modbus registers would appear the same way)

**Result:** One bridge reads your entire PLC. Every address becomes a tag automatically.

![Topic Browser DB1.DW20](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-f15c2d2c090d093cd0097fcb872ac337acee4e8e%2F2-topic-browser-DB1.DW20.png?alt=media)

## Part 2: Special Handling with Conditions

Some PLC tags need special treatment - scaling, unit conversion, or validation. Let's use **Conditions** to handle specific addresses differently.

### Add Your First Condition

1. Below the **Always** section, click **Add Condition**
2. **If Condition:** `msg.meta.s7_address == "DB1.DW20"` (use the appropriate metadata field for your protocol)
3. **Then Action:**

```javascript
// Scale the value and add engineering unit
msg.payload = parseFloat(msg.payload) * 1.0;
msg.meta.unit = "bar";  // This is pressure in bar
return msg;
```

### Add Another Condition for String Data

1. Click **Add Condition** again
2. **If Condition:** `msg.meta.s7_address == "DB1.S30.10"`
3. **Then Action:**

```javascript
// Clean up string data
msg.payload = msg.payload.trim();
msg.meta.unit = "Text";  // Mark as text data
return msg;
```

![Processing Conditions](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-615a3e5f4969d7fbb64ac42872115f7d3d25a9f2%2F2-processing.png?alt=media)

### What Conditions Do

* **Always section:** Runs for EVERY tag (sets basics)
* **Conditions:** Run ONLY for specific tags (special handling)

You can handle hundreds of tags with just a few conditions for the special cases.

### View the Enhanced Data

In **Topic Browser**, click on `DB1.DW20`. Notice:

* The value might be scaled
* **Metadata** now shows `unit: "bar"`
* Special handling applied ONLY to this tag

![Metadata View](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-60f6321523ec681d7d3d9ef825f28b0a5761fe13%2F2-topic-browser-metadata-raw.png?alt=media)

## Part 3: Organize with Virtual Folders

Right now all tags are at the root level. Let's organize them into logical folders using virtual paths.

### Add Organization with Conditions

1. **Add Condition** for production data
2. **If Condition:** `msg.meta.s7_address.startsWith("DB1")`
3. **Then Action:**

```javascript
// Group all DB1 data under "production" folder
msg.meta.virtual_path = "production";  // Creates organizational folder
return msg;
```

4. **Add Condition** for quality data
5. **If Condition:** `msg.meta.s7_address.startsWith("DB3")`
6. **Then Action:**

```javascript
// Group all DB3 data under "quality" folder
msg.meta.virtual_path = "quality";  // Creates organizational folder
return msg;
```

### The Result: Organized Folders

In **Topic Browser**, your data is now organized:

```
enterprise.sksk._raw.production.DB1.DW20     [12345]
enterprise.sksk._raw.production.DB1.S30.10   ["Product ABC"]
enterprise.sksk._raw.quality.DB3.I270        [789]
```

* All DB1 tags → production folder
* All DB3 tags → quality folder
* Automatically organized by data block

You organize entire data blocks with one condition, not individual tags.

## Part 4: Route to Different Machines

What if different data blocks belong to different machines? Let's route them!

### Dynamic Location Routing

Add conditions to route data blocks to different machines:

1. **Add Condition** for Machine 1
2. **If Condition:** `msg.meta.s7_address.startsWith("DB1")`
3. **Then Action:**

```javascript
// DB1 belongs to machine-1
msg.meta.location_path = "{{ .location_path }}.machine-1";
msg.meta.virtual_path = "sensors";
return msg;
```

4. **Add Condition** for Machine 2
5. **If Condition:** `msg.meta.s7_address.startsWith("DB3")`
6. **Then Action:**

```javascript
// DB3 belongs to machine-2
msg.meta.location_path = "{{ .location_path }}.machine-2";
msg.meta.virtual_path = "quality";
return msg;
```

### Result

**ONE bridge now routes to MULTIPLE machines:**

```
enterprise.sksk.machine-1._raw.sensors.DB1.DW20     [12345]
enterprise.sksk.machine-1._raw.sensors.DB1.S30.10   ["Product ABC"]
enterprise.sksk.machine-2._raw.quality.DB3.I270     [789]
```

## Understanding the Complete Picture

You now control every part of the topic:

`umh.v1.enterprise.sksk.machine-1._raw.sensors.DB1.DW20`

```mermaid
flowchart LR
    subgraph Fixed["fixed"]
        A["umh.v1"]
    end
    subgraph Location["location_path"]
        B["enterprise.sksk.machine-1"]
    end
    subgraph Virtual["virtual_path"]
        D["sensors.DB1"]
    end
    subgraph Tag["tag_name"]
        E["DW20"]
    end
    A --> B --> C["_raw"] --> D --> E
```

* **location\_path**: WHERE the device is (can be dynamic)
* **virtual\_path**: HOW you organize (folders for grouping)
* **tag\_name**: WHAT you're reading (automatic from S7 address!)

## What You've Learned

✅ **Protocol addresses become tag names** - No manual mapping needed! ✅ **Conditions handle special cases** - Not every tag, just the exceptions ✅ **Folders organize automatically** - Group entire data blocks at once ✅ **Dynamic routing is simple** - Send different PLCs to different locations ✅ **Variables make it reusable** - `{{ .IP }}` works for any PLC

## Concepts Learned

Building on previous guides, you now understand:

* **Template variables** - Dynamic configuration using `{{ .IP }}`, `{{ .PORT }}`, `{{ .location_path }}`
* **Message structure** - msg.meta (metadata) and msg.payload (actual value)
* **Conditions** - If-then rules in Tag Processor for special handling
* **virtual\_path** - Additional folder organization within topics
* **Dynamic routing** - One bridge serving multiple locations
* **Automatic tag mapping** - Protocol addresses become tag names automatically
* **Metadata** - Additional context in msg.meta (units, sources, custom fields)

## What's Next?

You can now:

* Connect entire PLCs with one bridge
* Organize hundreds of tags with a few conditions
* Route to different locations dynamically

**But there's still a risk:** What if DB1.DW20 suddenly sends a string instead of a number? Or DB3.I270 goes out of range?

**Ready to add data validation?** → [Step 4: Validate Your Data](/getting-started/3-validate-data)


# Step 4: Validate Data

> **Prerequisite:** You should have multiple tags flowing from [Step 3](/getting-started/2-organize-data). If not, complete that first!

## The Problem

Right now, ALL your data goes into `_raw` - no validation:

* DB1.DW20 could suddenly send "hello" instead of a number
* Critical vibration data might arrive with wrong units
* Typos in tag names create duplicate data streams

**How do you ensure data quality?** Data Models!

## Part 1: Create Your First Data Model

Let's create a model for a CNC machine's vibration data.

### Add a Data Model

1. Go to **Data Models** → **Add Data Model**
2. **Instance:** Select your instance
3. **Name:** `cnc`
4. **Description:** "CNC machine vibration monitoring"

### Define the Structure

In the **Data Model Structure** section, add this YAML:

```yaml
vibration:
  x-axis:
    _payloadshape: timeseries-number
  y-axis:
    _payloadshape: timeseries-number
```

**What this means:**

* Your CNC model has a `vibration` folder
* Inside are two measurements: `x-axis` and `y-axis`
* Both must be numbers (enforced automatically!)

![Data Model Creation](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-7349a5d89c78da881ff9843e51f0005f567d0e1a%2F3-models-cnc.png?alt=media)

Click **Save & Deploy**.

![Data Model List](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-6d1cf79f97951658a6c2cf10f092cefb1cd9d7d5%2F3-models.png?alt=media)

💡 **Behind the scenes:** The system automatically creates a data contract called `_cnc_v1`. This contract will enforce your model's structure. [Learn more about data contracts →](/usage/data-modeling/data-contracts)

## Part 2: Use Your Model in a Bridge

### Find Your Contract Name

1. Go to **Contracts** tab
2. Find `_cnc_v1` - this was auto-generated from your model
3. Note the structure it expects

![Contract View](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-3688ea02b4be8c023fb9a8868ad10a49051ffc92%2F3-contracts.png?alt=media)

### Update Your Bridge

Go back to your S7 bridge from Step 3. In the **Always** section, change:

```javascript
// OLD: Everything goes to _raw
msg.meta.data_contract = "_raw";
```

To:

```javascript
// NEW: Everything goes to our validated model
msg.meta.data_contract = "_cnc_v1";
```

![Incorrect Processing Setup](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-a8d57ab08169c3c903945b697fe979e23b359785%2F3-bridge-cnc-wrong.png?alt=media)

Click **Save & Deploy**.

## Part 3: Experience Validation (It Will Fail!)

![Deployment Failed](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-a3b030d7252081b8315ab7afa08c3d3f91308f3e%2F3-bridge-cnc-deploy-failed.png?alt=media)

**Your deployment fails!** Look at the error:

```
schema validation failed for message with topic 'umh.v1.enterprise.sksk._cnc_v1.DB1.DW20':
Valid virtual_paths are: [vibration.x-axis, vibration.y-axis].
Your virtual_path is: DB1.DW20
Fix: Update the bridge to set meta.virtual_path to 'vibration' and meta.tag_name to 'x-axis' or 'y-axis'
```

Data models ENFORCE structure. Your S7 address `DB1.DW20` doesn't match the expected paths `vibration.x-axis` or `vibration.y-axis`.

**Two new concepts here:**

1. Deployments can fail not only if the connection is bad (learned in Step 2)
2. But also if the bridge throws validation errors

## Part 4: Fix with Smart Routing

Instead of forcing ALL data into the model, let's be selective. Change your **Always** section back:

```javascript
// Most data stays unvalidated
msg.meta.data_contract = "_raw";
msg.meta.tag_name = msg.meta.s7_address;
return msg;
```

Now update your condition for DB1.DW20:

```javascript
// Special handling for DB1.DW20: Route to validated model
msg.payload = parseFloat(msg.payload) * 1.0;
msg.meta.data_contract = "_cnc_v1";  // Use validated model
msg.meta.virtual_path = "vibration";  // Required: matches model structure
msg.meta.tag_name = "x-axis";         // Required: matches model field
msg.meta.unit = "raw";
return msg;
```

![Correct Processing Setup](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-988305672c172545df1432e2f4aecc70f4d266ba%2F3-bridge-cnc-processing-correct.png?alt=media)

**What this does:**

* DB1.DW20 → Validated as `vibration.x-axis` in CNC model
* All other tags → Continue to `_raw` (no validation)

Click **Save & Deploy**. Now it succeeds!

## Part 5: Success! View Your Validated Data

![Topic Browser with Validated Data](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-d70610e56f1c9c5d72d516a21cc3d2dc0df4e81a%2F3-topic-browser-correct.png?alt=media)

In **Topic Browser**, you now see:

```
enterprise.sksk._cnc_v1.vibration.x-axis    [12345]  ✓ Validated
enterprise.sksk._raw.DB1.S30.10             ["text"]  (Unvalidated)
enterprise.sksk._raw.DB3.I270               [789]     (Unvalidated)
```

**How validation works:**

* The CNC model guarantees `x-axis` is always a number
* If someone sends text, it's rejected at the bridge
* Other data flows normally through `_raw`

## Concepts Learned

Building on previous guides, you now understand:

* **Data Models** - Reusable templates defining data structure
* **Payload Shapes** - Type validation (timeseries-number, etc.)
* **Schema Validation** - Automatic enforcement at the bridge
* **Auto-generated Contracts** - Models create contracts like `_cnc_v1`
* **Validation Errors** - Deployments fail if data doesn't match
* **Selective Validation** - Route specific data to validated models

## What's Next?

You now have a complete data pipeline with:

* ✅ Automatic tag discovery (Step 3)
* ✅ Smart organization (Step 3)
* ✅ Data validation (Step 4)

**Ready for production?** Check out:

* [Production Guide](/production) - Sizing, security, monitoring
* [Data Modeling Deep Dive](/usage/data-modeling) - Advanced models
* [Stream Processors](/usage/data-modeling/stream-processors) - One way to transform device data to business KPIs

***

**Congratulations!** You've mastered the fundamentals of UMH Core. Your data is now organized, validated, and production-ready. 🎉


# Usage

This section covers practical implementation of UMH Core, focusing on data flows and unified namespace concepts. Learn how to set up bridges, standalone flows, and stream processors to connect your industrial devices to the unified namespace. Includes detailed guides on data production, consumption, topic conventions, payload formats, real-time topic exploration via the Topic Browser, and storing history in the Historian.


# Instances

## What is an Instance?

An instance is a running UMH Core container - a single Docker container that hosts your entire Unified Namespace. Each instance has a location path like `enterprise.site.area.line`. The location path determines where data is organized in your industrial data infrastructure. An instance is also uniquely identified by its Instance UUID.

## How an instance connects

An instance authenticates with an `AUTH_TOKEN`, not with a user account. The Management Console generates the token while you create the instance and shows it once. Copy it then and pass it to the container as the `AUTH_TOKEN` environment variable on the first start.

The token does two things:

* It proves the instance's identity. The Management Console accepts the instance's status and hands it the configuration meant for it.
* It locks the instance's access to the Management Console. Only the instance can unlock it, because only the instance has the token. The Management Console cannot.

Keep the token somewhere you can find it again, such as your password manager. For storage and rotation on the instance, see [AUTH\_TOKEN in Environment Variable](/production/security/deployment-security#auth_token-in-environment-variable).

## Instance Overview Page

![Instance Overview](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-ec30c3474ace0adbdab0080868120a61efc65f2f%2Finstance-overview.png?alt=media)

The instance overview shows all your UMH instances at a glance:

* **Instance Name**: Your chosen identifier for the instance
* **Type**:
  * **Core**: Single container deployment (UMH Core)
  * **Classic**: Kubernetes-based deployment (legacy)
* **Version**: Container version identifier (e.g., `9e22396`)
* **Data Flows**: Number of configured bridges and stand-alone flows
* **Topics**: Total data points in your Unified Namespace
* **Latency**: Network response time in milliseconds
* **Throughput**: Messages per second flowing through the system

**Status indicators:**

* Green dot: Instance is online and reachable
* Gray dot: Instance is offline or unreachable

**Quick actions:** Click the context menu (⋮) on any instance for:

* Instance Details
* Config File
* Delete

## Instance Details Page

![Instance Details](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-b0af1a9961fd555792b453666d44bf99fd20faa6%2Finstance-detail.png?alt=media)

The instance details page provides comprehensive monitoring and management:

### Agent Panel

The Agent panel shows the instance's identity and its connection to the Management Console:

* **Name**: Instance identifier (e.g., `sk-core-hetzner`)
* **Location path**: Your organizational structure
  * Level 0: Enterprise (required)
  * Level 1-4: Site, Area, Line, etc. (optional)
* **Latency**: Connection health indicator (N/A when healthy)
* **Logs button**: Opens the instance's system logs for diagnostics

### Container Panel

The Container panel shows the resources the container uses:

* **CPU**: Usage percentage of available cores
* **Memory**: Usage of available RAM (GiB)
* **Disk**: Storage usage (GiB)
* **Architecture**: System architecture (e.g., `amd64`)
* **Hardware ID**: Unique container identifier

### Data Flows Panel

The Data Flows panel groups bridges and stand-alone flows by their state. The states come from the [state machines](/reference/state-machines) reference:

* **Active**: Currently processing data (messages flowing)
* **Neutral**: Includes:
  * `idle`: Healthy but no data for 30+ seconds
  * `stopped`: Intentionally disabled
  * `starting`: Initialization in progress
  * Various transition states

Example states:

* `starting_dfc (port is open)`: Connection verified, waiting for Benthos, the stream processor that runs the flow, to start
* `starting: stopping: stopping`: Complex state transition in progress

**Bridges count**: Total configured with active/neutral breakdown

### Redpanda Panel

The Redpanda panel shows the message broker that stores your [Unified Namespace](/usage/unified-namespace):

* **Incoming Throughput**: Data rate into the broker (KiB/s)
* **Outgoing Throughput**: Data rate from the broker (KiB/s)
* **Logs/Metrics buttons**: Direct access to Redpanda diagnostics

### Topic Browser Panel

The Topic Browser panel summarizes the data in the instance's Unified Namespace:

* **Topics count**: Total number of data topics
* Quick access to the [Topic Browser](/usage/unified-namespace/topic-browser) for data exploration

### Release Panel

The Release panel lists the version information for support requests and updates:

* **Version**: Full semantic version (e.g., `v0.43.4`), not the short container version identifier shown on the overview page
* **Channel**: Release channel (`stable`, `beta`, etc.)
* **Software versions**: Individual component versions (S6 Overlay, Benthos, Redpanda)

## Next Steps

* **Configure your instance**: [Edit the config file](/usage/instances/config-file)
* **Connect devices**: [Create bridges](/usage/data-flows/bridges)
* **View your data**: [Topic Browser](/usage/unified-namespace/topic-browser)
* **Understand states**: [State Machines reference](/reference/state-machines)


# Config File

The configuration file editor allows you to directly modify your instance's `/data/config.yaml` file through the Management Console UI.

## Accessing the Config Editor

Two ways to open the configuration editor:

1. **From Instance Details**: Click the "Config File" button in the top-right corner
2. **From Instance Overview**: Click the context menu (⋮) → "Config File"

![Config File Editor](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-1985ecf53618aa07305607db63babbfea13d288b%2Finstance-detail-config-edit.png?alt=media)

## ⚠️ Warning: Direct YAML Editing

**You are editing the config.yaml file directly.** Use this feature with caution:

* Invalid YAML syntax will break your instance
* Recovery requires SSH access to the container
* The agent applies changes via hot-reload - no restart needed

## Editor Controls

### Main Actions

* **Refresh**: Reload the current configuration from the instance
* **Save**: Apply your changes (the agent will hot-reload)

### Additional Options (⋮ menu)

![Local File Sync Options](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-15e85ecd250c4389526992990a69dd1b2a56ecac%2Finstance-overview-config-edit-sync-to-filesystem-to-work-with-LLM-custom-IDE.png?alt=media)

* **Download**: Save a local copy of the configuration
* **Local File Sync**:
  * **Open Local File**: Edit the config in your preferred IDE
  * **Create Local File**: Start a new local file for editing

Local file sync enables:

* Version control with Git
* IDE features (syntax highlighting, validation, autocomplete)
* AI assistant integration (Cursor, GitHub Copilot, Claude)

## Configuration Structure

The editor shows your complete configuration with main sections:

```yaml
templates:
  protocolConverter:
    # Reusable bridge templates

protocolConverter:
  - name: my-bridge
    # Bridge configurations (shown as "Bridges" in UI)

dataFlow:
  - name: my-flow
    # Stand-alone flow configurations
```

Key sections:

* `templates:` - Reusable configuration blocks for templating
* `protocolConverter:` - Bridge definitions (UI shows as "Bridges")
* `dataFlow:` - Stand-alone flow definitions
* `dataContracts:` - Data validation rules
* `dataModels:` - Data structure definitions

For complete documentation, see [Configuration Reference](/reference/configuration-reference).

## UI ↔ YAML Synchronization

UMH Core maintains two-way sync between UI and YAML:

**UI → YAML**: When you create bridges or data models in the UI, they appear here automatically

**YAML → UI**: After saving YAML changes, they're reflected in the UI immediately

This enables flexible workflows:

* **OT teams**: Use the UI for visual configuration
* **IT teams**: Use YAML for infrastructure-as-code
* **Learning**: Create in UI, study the generated YAML
* **Scaling**: Template in YAML for multiple similar devices

## Troubleshooting

**Syntax Errors**: The Save button will show errors if YAML is invalid. Common issues:

* Incorrect indentation (use spaces, not tabs)
* Missing colons after keys
* Unclosed quotes

**Changes Not Applied**: Ensure you clicked Save. Check the agent logs if changes don't take effect.

**Recovery from Bad Config**: If the instance breaks:

1. SSH into the container
2. Edit `/data/config.yaml` directly
3. The agent will auto-reload when valid

## Next Steps

* [Configuration Reference](/reference/configuration-reference) - Full YAML schema
* [Variables](/reference/variables) - Template variables and substitution
* [Bridges](/usage/data-flows/bridges) - Configure data connections


# Unified Namespace

> **Prerequisite:** Complete the [Getting Started guide](/getting-started) to see the UNS in action with real devices and data.

The Unified Namespace (UNS) is where ALL your industrial data lives in one organized, accessible place. Instead of hunting through 50 different systems to find a temperature reading, everything flows through one central hub.

## The Problem: Spaghetti Diagrams

Traditional manufacturing IT looks like this:

```
100 devices × 100 systems = 10,000 point-to-point connections
```

Every new dashboard means updating PLCs. Every new sensor means modifying databases. Every integration breaks when you upgrade. It's a maintenance nightmare that gets worse with scale.

## The Solution: One Central Hub

The UNS flips this architecture:

```mermaid
flowchart LR
    D["100 devices"] --> N(("1 namespace")) --> S["100 systems"]
```

All data flows through one place with consistent structure, validation, and access patterns.

## How It Works

### Publish Regardless

Your PLC doesn't care if anyone is listening. It publishes "pump is running" to the UNS and moves on. When someone needs that data next week, it's already there. No reprogramming required.

### Structured Topics

Every piece of data has an address that answers three questions:

* **WHERE**: `enterprise.site.area.line` - the location path
* **WHAT**: `_pump_v1` - the data contract defining structure
* **WHICH**: `inlet_temperature` - the specific data point

Result: `umh.v1.enterprise.site.area.line._pump_v1.inlet_temperature`

### Bridges Handle All Data Flow

Data enters and exits the UNS exclusively through [bridges](/usage/data-flows/bridges). This ensures every message gets proper context, validation, and organization.

### From Device to Business

Start simple, add complexity as needed:

* **`_raw`** - Mirror device 1:1 for initial exploration and debugging
* **`_pump_v1` and similar** - Apply business names directly in bridges for production
* **`_maintenance_v1` and similar** - From stream processors (aggregating device models) OR directly from ERP/MES systems

The progression: Use `_raw` temporarily to understand your data, then apply device models directly in bridges for production. Business models can be created by aggregating device models via stream processors OR by connecting directly to business systems like ERP/MES.

## Documentation Structure

* [**Topic Convention**](/usage/unified-namespace/topic-convention) - How data is addressed in the namespace
* [**Payload Formats**](/usage/unified-namespace/payload-formats) - Time-series vs relational message structure
* [**Metadata and Tracing**](/usage/unified-namespace/metadata-and-tracing) - Original tags and data lineage
* [**Topic Browser**](/usage/unified-namespace/topic-browser) - Explore your namespace in real-time

## Next Steps

1. **Explore your data**: Open the [Topic Browser](/usage/unified-namespace/topic-browser)
2. **Connect devices**: Create [bridges](/usage/data-flows/bridges)
3. **Model your data**: Define [data models](/usage/data-modeling/data-models)
4. **Create KPIs**: Build [stream processors](/usage/data-modeling/stream-processors)


# Topic Convention

> **Prerequisite:** Understand the [UNS concept](/usage/unified-namespace) and see it in action via [Getting Started](/getting-started).

Every piece of data in the UNS has a unique address that tells you exactly where it came from and what it represents.

## See It In Action

Here's how real PLC data appears in the Topic Browser:

![Topic Browser showing DB1.DW20](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-f15c2d2c090d093cd0097fcb872ac337acee4e8e%2F2-topic-browser-DB1.DW20.png?alt=media)

The topic `umh.v1.demo.DefaultArea.DefaultProductionLine.SIEMENS-S7._raw.DB1.DW20` tells us:

* **WHERE**: demo → DefaultArea → DefaultProductionLine → SIEMENS-S7
* **WHAT**: `_raw` (unprocessed device data)
* **WHICH**: DB1.DW20 (specific PLC register)

## The Pattern

```
umh.v1.<location_path>.<data_contract>[.<virtual_path>].<name>
```

### Breaking It Down

| Part            | Example                              | Purpose                      |
| --------------- | ------------------------------------ | ---------------------------- |
| `umh.v1`        | Always `umh.v1`                      | Version prefix               |
| `location_path` | `enterprise.site.area.line`          | Physical/logical hierarchy   |
| `data_contract` | `_raw` or `_pump_v1`                 | Data structure type          |
| `virtual_path`  | `motor.diagnostics`                  | Optional folder organization |
| `name`          | `temperature` or `work_order.create` | The data point or action     |

## Location Path - WHERE

The location path organizes your physical and logical structure:

* **Level 0 (Required)**: Enterprise - your company
* **Level 1**: Site - physical location
* **Level 2**: Area - department or zone
* **Level 3**: Line - production line or cell
* **Level 4**: Machine - specific equipment

You can use ISA-95, KKS, or any naming standard. The only rule: level 0 is mandatory.

Example: `umh.v1.acme.chicago.packaging.line1.filler._raw.speed`

## Data Contract - WHAT

The contract defines the data structure:

* **`_raw`**: Unvalidated device data for exploration
* **`_devicemodel_v1`**: Validated device models (e.g., `_pump_v1`, `_cnc_v1`)
* **`_businessmodel_v1`**: Business KPIs and aggregations (e.g., `_maintenance_v1`, `_production_v1`)

Contracts always start with underscore. They're your data's "type system."

## Virtual Path - Organization

Optional segments for grouping related data:

```
umh.v1.acme.plant._pump_v1.motor.diagnostics.vibration
                          └─────┬─────┘
                          Virtual path for organization
```

## Name - WHICH

The specific data point or action:

* **Time-series**: `temperature`, `pressure`, `running`
* **Relational**: `work_order.create`, `batch.complete`

## Next Steps

* [Understand payload formats](/usage/unified-namespace/payload-formats) for message structure
* [Connect devices with bridges](/usage/data-flows/bridges)
* [Explore with Topic Browser](/usage/unified-namespace/topic-browser)


# Payload Formats

UMH Core supports two payload formats: **Time-Series** for individual sensor readings and **Relational** for structured business data.

## Quick Reference

| Type            | Structure                             | Use Case                        | Example                       |
| --------------- | ------------------------------------- | ------------------------------- | ----------------------------- |
| **Time-Series** | `{"timestamp_ms": int, "value": any}` | Sensor readings, machine states | Temperature, pressure, status |
| **Relational**  | `{...any JSON object...}`             | Business records, work orders   | Batch data, quality reports   |

## Time-Series Data

Individual sensor values with timestamps. One value per message.

**Format:**

```json
{
  "timestamp_ms": 1733904005123,
  "value": 42.5
}
```

**Requirements:**

* Must have exactly `timestamp_ms` and `value` keys
* `timestamp_ms`: Integer or float without fraction
* `value`: number, boolean, or string (see [Data Model Type Definitions](/reference/data-model-type-definitions))
* No additional keys allowed
* Max size: 1MiB

**Examples:**

```json
// Temperature reading
{"timestamp_ms": 1733904005123, "value": 23.4}

// Machine state
{"timestamp_ms": 1733904005123, "value": "running"}

// Boolean sensor
{"timestamp_ms": 1733904005123, "value": true}
```

## Relational Data

Structured records with multiple fields. Used for business data and aggregated metrics.

**Format:**

```json
{
  "field1": "value1",
  "field2": 123,
  "nested": {
    "field3": true
  }
}
```

**Requirements:**

* Must be a JSON object (not array or primitive)
* Can have any keys and nested structure
* Max size: 1MiB

**Examples:**

```json
// Work order
{
  "order_id": "WO-2024-001",
  "product": "Widget-A",
  "quantity": 1000,
  "timestamp_ms": 1733904005123
}

// Quality inspection
{
  "batch_id": "B-2024-12-01",
  "measurements": {
    "length_mm": 100.2,
    "width_mm": 50.1,
    "weight_g": 250.5
  },
  "passed": true,
  "inspector": "John Doe"
}
```

## The "One Tag, One Topic" Rule

Each sensor value gets its own topic. This simplifies addressing, consumption, and debugging.

**Instead of:**

```
Topic: umh.v1.acme.plant._raw.weather
Payload: {"temperature": 23.4, "humidity": 42.1}
```

**Use:**

```
Topic: umh.v1.acme.plant._raw.weather.temperature
Payload: {"timestamp_ms": 1733904005123, "value": 23.4}

Topic: umh.v1.acme.plant._raw.weather.humidity
Payload: {"timestamp_ms": 1733904005123, "value": 42.1}
```

### Why This Matters

| Problem with Multi-Value Payloads | Real-World Impact                                                          |
| --------------------------------- | -------------------------------------------------------------------------- |
| **Timing issues**                 | Temperature arrives, humidity 100ms later. Do you cache? Wait? How long?   |
| **Merge windows**                 | Two PLC variables change almost simultaneously but arrive separately       |
| **Clock skew**                    | Reading 100 tags takes milliseconds - first may change before last is read |
| **Complex addressing**            | Need 3 dimensions: topic × tag name × JSON path                            |
| **Testing complexity**            | Each edge case (missing key, late field, partial failure) multiplies bugs  |

**Benefits of One Tag Per Topic:**

* **Zero merge logic** - Each value is complete when published
* **Simple subscriptions** - Subscribe only to tags you need
* **Clear debugging** - One topic = one data point
* **Schema simplicity** - Data contract just specifies time-series format

## Next Steps

* [Bridges](/usage/data-flows/bridges) - Send data to UNS
* [Stream Processors](/usage/data-modeling/stream-processors) - Transform time-series to relational
* [Data Models](/usage/data-modeling/data-models) - Define structure for both formats


# Metadata and Tracing

Every message in the UNS carries metadata that preserves its origin and transformation history. This enables troubleshooting and data lineage tracking.

## Understanding Metadata

When data flows through the UNS, metadata accumulates at each step:

```json
{
  "_incomingKeys": "s7_address",
  "_initialMetadata": "{\"s7_address\":\"DB1.DW20\"}",
  "bridged_by": "protocol-converter_pump-bridge",
  "data_contract": "_pump_v1",
  "data_contract_name": "_pump",
  "data_contract_version": "1",
  "kafka_msg_key": "umh.v1.enterprise.chicago.packaging.line-1.pump-01._pump_v1.inlet_temperature",
  "kafka_timestamp_ms": "1758290100065",
  "kafka_topic": "umh.messages",
  "location_path": "enterprise.chicago.packaging.line-1.pump-01",
  "s7_address": "DB1.DW20",
  "name": "inlet_temperature",
  "topic": "umh.v1.enterprise.chicago.packaging.line-1.pump-01._pump_v1.inlet_temperature",
  "umh_topic": "umh.v1.enterprise.chicago.packaging.line-1.pump-01._pump_v1.inlet_temperature"
}
```

## Tracing Data Flow

### Bridge Metadata

Every bridge adds metadata to identify the source:

```yaml
# In Topic Browser or when consuming:
bridged_by: "protocol-converter_pump-bridge"    # Which bridge created this
location_path: "enterprise.chicago.packaging"    # Where it came from
data_contract: "_pump_v1"                       # What model was applied
```

Note: The exact format of bridge metadata may vary by implementation.

## Accessing Metadata

### In Topic Browser

The Management Console shows metadata in the details panel:

1. Select any topic in the tree
2. View "Metadata" section in the right panel
3. See all headers including original tags

For details, see [Topic Browser documentation](/usage/unified-namespace/topic-browser).

**Further metadata**: Each input or processing step adds its own metadata. For complete field documentation, see:

* [Benthos-UMH Input Plugins](https://docs.umh.app/benthos-umh/input)

## Next Steps

* [**Topic Browser**](/usage/unified-namespace/topic-browser) - View metadata interactively
* [**Bridges**](/usage/data-flows/bridges) - How data enters and exits the UNS


# Topic Browser

Real-time exploration of your Unified Namespace data via Management Console or GraphQL API.

## Management Console

Access via `management.umh.app` → **Topic Browser** (not the Classic version).

### Interface Overview

* **Left panel**: Hierarchical topic tree following ISA-95 structure
* **Right panel**: Details for selected topic (metadata, values, history)
* **Auto-aggregation**: Combines data from all your UMH instances
* **Live updates**: Real-time data without manual refresh

### Key Features

**Topic Tree Navigation:**

* Expand/collapse hierarchy levels (Enterprise → Site → Area → Line → Equipment)
* See data contracts (`_raw`, `_pump_v1`) and virtual paths
* Search bar for filtering by topic name or metadata

**Topic Details Panel:** When selecting a topic, view:

* **Location**: Physical hierarchy path
* **Data Contract**: Applied schema (`_raw`, `_pump_v1`, etc.)
* **Current Value**: Latest data with timestamp
* **Metadata**: Including original device tags (see [Metadata and Tracing](/usage/unified-namespace/metadata-and-tracing))
* **History**: Last 100 values as chart (time-series) or table (boolean/string)

![Topic Browser showing DB1.DW20](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-f15c2d2c090d093cd0097fcb872ac337acee4e8e%2F2-topic-browser-DB1.DW20.png?alt=media)

## GraphQL API

Programmatic access at `http://localhost:8090/graphql` (per instance).

{% hint style="info" %}
**Early Access.** We built the GraphQL API for developers who want to query the Unified Namespace programmatically. It's experimental, so it's off by default. See the instructions below and give it a try. Tell us what you think.
{% endhint %}

### Configuration

```yaml
internal:
  topicBrowser:
    desiredState: "active"  # Enable/disable

agent:
  graphql:
    enabled: true           # Enable GraphQL API (required!)
    port: 8090             # API port
    debug: false           # Set true for GraphiQL UI
```

### Basic Query

```graphql
{
  topics(filter: {
    text: "temperature",
    meta: [{ key: "data_contract", eq: "_pump_v1" }]
  }) {
    topic
    metadata {
      key
      value
    }
    lastEvent {
      ... on TimeSeriesEvent {
        producedAt
        numericValue
        scalarType
      }
    }
  }
}
```

### cURL Example

```bash
curl -X POST http://localhost:8090/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ topics(limit: 10) { topic } }"}'
```

## Common Use Cases

### Finding Specific Data

1. Use search bar for partial matches (e.g., "temp" finds all temperature topics)
2. Filter by metadata like data contract or location
3. Check metadata section for original device tags

### Verifying Data Flow

1. Select topic to see current value
2. Check "Produced At" timestamp for freshness
3. Review history chart for patterns
4. Inspect metadata for source bridge

### Debugging Issues

1. Verify topic exists in tree
2. Check metadata for `bridged_by` to identify source
3. Look for original tag names (e.g., `opcua_tag_name`, `s7_address`)
4. Compare timestamps to detect delays

## Performance Notes

* Topic Browser stores last 100 values per topic
* Extremely rapid updates are batched for display
* Large namespaces (>10,000 topics) may load progressively
* Each UMH instance has its own Topic Browser service

## Next Steps

* [**Metadata and Tracing**](/usage/unified-namespace/metadata-and-tracing) - Understanding metadata fields


# Data Flows

> **Prerequisite:** Complete the [Getting Started guide](/getting-started) to understand basic connections and data organization.

Data flows move and transform industrial data in UMH Core. Three types serve different purposes:

## Flow Types

### [Bridges](/usage/data-flows/bridges)

Move data into and out of the Unified Namespace with connection monitoring and automatic location path. Support for 50+ industrial protocols (OPC UA, Modbus, S7) plus IT systems. Write flows coming soon - use stand-alone flows meanwhile.

### [Stand-alone Flows](/usage/data-flows/stand-alone-flow)

Raw Benthos access for custom processing when bridges or stream processors aren't sufficient. Used as fallback for write flows, external integrations, and specialized transformations.

### [Stream Processors](/usage/data-flows/stream-processor)

Transform existing UNS data into different structures. Part of the [data modeling system](/usage/data-modeling/stream-processors) for aggregating device data into business views.

## Quick Start

Most users start with bridges to connect devices:

1. **Data Flows** → **Add Bridge**
2. Select your protocol (OPC UA, Modbus, S7)
3. Configure connection and location
4. Deploy to start data collection

## Learn More

* [Getting Started Guide](/getting-started) - Connect your first device
* [Data Modeling](/usage/data-modeling) - Structure and validate data
* [Unified Namespace](/usage/unified-namespace) - Understanding topics and payloads


# Bridges

> **Prerequisite:** Understand [Data Flow concepts](/usage/data-flows) and complete the [Getting Started guide](/getting-started).

Bridges move data into and out of the Unified Namespace, providing connection monitoring and automatic data organization. While read flows work with any protocol, write flows are not yet implemented - use [stand-alone flows](/usage/data-flows/stand-alone-flow) for writing to external systems.

> **New to UMH?** This page is the reference for setting up real bridges. If you just want to get mock data flowing and learn the basics first, start with [Step 2: Connect Your First Data Source](/getting-started/1-connect-data).

## UI Capabilities

| Action                      | Available | Notes                                                        |
| --------------------------- | --------- | ------------------------------------------------------------ |
| Create bridges              | ✅         | Visual protocol configuration                                |
| Create from existing bridge | ✅         | Copy a healthy bridge's configuration                        |
| Start from a template       | ✅         | 84 pre-configured templates, filtered by vendor and protocol |
| Select protocols            | ✅         | OPC UA, Modbus, S7, MQTT, 50+ more                           |
| Configure location path     | ✅         | ISA-95 levels (0-4)                                          |
| Map addresses to topics     | ✅         | Dropdown of data model fields for custom data contracts      |
| Monitor connection health   | ✅         | Real-time status                                             |
| View logs                   | ✅         | Live log streaming                                           |
| View metrics                | ✅         | Throughput monitoring                                        |
| Edit bridges                | ✅         | Modify and redeploy                                          |
| Delete bridges              | ✅         | Clean removal                                                |
| Advanced mode               | ✅         | Direct YAML editing                                          |

## Creating a Bridge (UI)

### Step 1: Navigate to Data Flows

Go to **Data Flows** and click **Add Bridge**:

![Data Flows Overview](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-f0624dcca1a77106a17dc75bc85492fd0323a632%2Fbridges-overview.png?alt=media)

### Step 2: Choose How to Start

![Create a New Bridge](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-0bbbf9b410571440a9d8966ef4256d810134930f%2Fbridges-create-from-scratch-or-existing.png?alt=media)

* **From Scratch**: start from a pre-configured protocol template, which is functional out of the box, or customize the template to fit your needs.
* **From Existing Bridge**: pick a healthy bridge from any instance and start with a copy of its configuration. Adapt the bridges name, connection, and addresses.

The remaining steps describe the from-scratch path. Starting from an existing bridge skips the template list and lands on the same screens with the fields already filled in.

### Step 3: Select a Template

![Select a Template](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-1b5c1bd99bd63288d7a7af06cb2c2750e541258f%2Fbridges-template-selection.png?alt=media)

Templates carry a working protocol configuration. Search by name, or narrow the list with the **Vendor** and **Protocol** filters.

Behind the templates we support **50+ industrial protocols** through [Benthos-UMH](https://docs.umh.app/benthos-umh/input) plus everything from [Redpanda Connect](https://docs.redpanda.com/redpanda-connect/components/inputs/about/):

Common industrial protocols:

* **Modbus**: For PLCs and RTUs
* **OPC UA**: For industrial servers
* **S7**: For Siemens PLCs
* **Ethernet/IP**: For Allen-Bradley
* **And many more**: See full list at [Benthos-UMH Inputs](https://docs.umh.app/benthos-umh/input)

### Step 4: Configure General Settings

Fill in the basic information and proceed with Step 5.

![General Information](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-80fd8636c868a27c7300b030b607d666e51cb44e%2Fbridges-general-information.png?alt=media)

**General Information:**

* **Name**: Unique identifier for your bridge
* **Instance**: Select your UMH instance
* **Level**: Location path (e.g., enterprise → site → area → line → machine)

![Connection Configuration](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-5a655e091ada41fe496eb3a48af55a9df1695d9c%2Fbridges-connection-configuration.png?alt=media)

**Connection:**

* **IP Address**: Device IP (becomes `{{ .IP }}` variable)
* **Port**: Connection port (becomes `{{ .PORT }}` variable)

The protocol itself needs no selection here. It comes from the template you picked in Step 3, and the **Read** tab opens on that protocol's settings.

### Step 5: Configure Protocol Settings

Each protocol has specific settings. For example, S7:

![Siemens S7 Configuration](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-c9bc488c40e57b6baac27a8236464b4d458fedb1%2Fbridges-read-no-dfc-data-type-selector-for-siemens.png?alt=media)

### Step 6: Map Addresses

Each row in the address mapping table becomes one UNS topic: the protocol address, followed by the data contract, virtual path, and tag name that build the topic.

![Address Mapping Table](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-97d50d62d76f3aed3e06d5770a82bbcca57b94c2%2Fbridges-read-address-table-model-dropdown.png?alt=media)

With `_historian` or `_raw`, Virtual Path and Tag Name are free-text fields. Selecting a custom data contract merges both columns into a single dropdown listing every field the contract's [data model](/usage/data-modeling/data-models) defines, so you pick a valid path instead of typing one. A value the model rejects, for example after switching contracts, stays in the row, is flagged red, and blocks deploy until you correct it. Contracts whose data model references other models keep free-text input.

### Step 7: Deploy

Click **Save & Deploy**. The bridge will start connecting to your device.

## Managing Bridges

### View Status

The main view shows all bridges with connection status and throughput:

![Bridges Overview](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-f0624dcca1a77106a17dc75bc85492fd0323a632%2Fbridges-overview.png?alt=media)

* **Green dot**: Connected and running
* **Yellow dot**: Starting or warning state
* **Grey dot**: Intermediate states (starting, stopping, or stuck in starting state)

### Context Menu

Right-click any bridge for quick actions:

![Bridge Context Menu](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-7696b81d2db23e97b697a62b0f3c49ad56ea3223%2Fbridges-overview-context-menu.png?alt=media)

### View Logs

Monitor bridge activity in real-time:

![Bridge Logs](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-b5cf7bb937cfb071624d100289f48025837b62ae%2Fbridges-logs.png?alt=media)

### View Metrics

Track performance and throughput:

![Bridge Metrics](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-5dfe43f9586f00c204051786c9e9341a9002d917%2Fbridges-metrics.png?alt=media)

### Enable Debug Logging

When troubleshooting connection or data issues, enable debug logging in your bridge configuration:

**Via YAML:**

```yaml
protocolConverter:
  - name: my-plc-bridge
    protocolConverterServiceConfig:
      debug_level: true  # Enable detailed logging
      config:
```

Debug logging provides:

* Detailed protocol communication logs
* Connection state transitions
* Data parsing details
* For OPC UA: Sets `OPC_DEBUG` environment variable for protocol-level debugging

**Remember to disable debug logging** after troubleshooting - it generates significantly more log data.

To view debug logs, go to the bridge and select **Logs**.

## Important: Data Types from Different Protocols

**Industrial protocols (OPC UA, Modbus, S7) only provide time-series data** - snapshots of values at points in time. They cannot directly provide relational records.

**IT protocols (MQTT, HTTP, Kafka) can provide relational data** - when you select "relational" as the data type in the UI, the bridge automatically uses [`nodered_js` processor](https://docs.umh.app/benthos-umh/processing/node-red-javascript-processor) to handle complex business records like work orders or batch reports.

For converting time-series to relational data, see [Stream Processors](/usage/data-flows/stream-processor#time-series-to-relational-challenges) or use [Stand-alone Flows](/usage/data-flows/stand-alone-flow) with state management.

## Configuration (YAML)

Advanced users can edit the YAML directly:

```yaml
protocolConverter:
  - name: my-plc-bridge
    desiredState: active
    protocolConverterServiceConfig:
      debug_level: false  # Direct child (matches standalone flows)
      location:
        2: "production-line"
        3: "plc-01"
      config:
        connection:
          nmap:
            target: "{{ .IP }}"
            port: "{{ .PORT }}"
        dataflowcomponent_read:
          benthos:
            input:
              s7comm:
                addresses:
                  - DB1.DW20
                  - DB3.I270
                tcpDevice: "{{ .IP }}"
                rack: 0
                slot: 1
            pipeline:
              processors:
                - tag_processor:
                    defaults: |
                      msg.meta.location_path = "{{ .location_path }}";
                      msg.meta.data_contract = "_raw";
                      msg.meta.tag_name = msg.meta.s7_address;
                      return msg;
            output:
              uns: {}
      variables:
        IP: "192.168.1.100"
        PORT: "102"
```

### Key Configuration Elements

**Location Path:**

```yaml
location:
  0: enterprise   # Optional: inherits from agent
  1: site        # Optional: inherits from agent
  2: area        # Required: your addition
  3: line        # Required: your addition
  4: cell        # Optional: further subdivision
```

**Template Variables:**

```yaml
variables:
  IP: "192.168.1.100"      # Available as {{ .IP }}
  PORT: "502"              # Available as {{ .PORT }}
  SCAN_RATE: "1s"          # Custom variable {{ .SCAN_RATE }}
```

**Connection Monitoring:**

```yaml
connection:
  nmap:
    target: "{{ .IP }}"    # Uses IP variable
    port: "{{ .PORT }}"    # Uses PORT variable
```

## Next Steps

* Try the [Getting Started guide](/getting-started) to connect your first device
* Learn about [Data Models](/usage/data-modeling/data-models) to structure your data
* Explore [Metadata and Tracing](/usage/unified-namespace/metadata-and-tracing) for debugging
* Use [Stand-alone Flows](/usage/data-flows/stand-alone-flow) for custom processing
* Configure [Stream Processors](/usage/data-flows/stream-processor) to transform data


# Bridges Do Not Start

A bridge that waits shows "Pending Creation" in the Management Console, and its status shows the reason. This page explains what each reason means and what to do.

Before umh-core starts a new bridge, it runs bridge admission. A new bridge waits when one of the instance's resources is degraded, when its resource health is not proven yet, or when the instance has reached its bridge limit. The reason says which of the three it is. Bridge admission is on only while `agent.enableResourceLimitBlocking` is `true`.

## What the reason says

* **CPU degraded: \[message]** — the instance's CPU is short on headroom. The message comes from the CPU health check. The [CPU Health](/production/cpu-health) page explains every status and how to fix it. If the message starts with "CPU not measured", umh-core cannot read a file it needs to measure the CPU. The CPU Health page says how to check that file.
* **Memory degraded: \[message]** — memory is short. Free memory on the host, or raise the container's memory limit.
* **Disk degraded: \[message]** — the disk is short. Free space on the volume, or make the volume larger.
* **Resource health not proven yet** — umh-core has not yet shown the instance's resources to be healthy. A bridge always waits for this proof, so a fresh instance starts no bridges until its first health readings arrive. If the reason stays at "no health reading yet", the instance cannot read its own resources: use the emergency setting below and report it to UMH. The reason "instance not active yet" means the container has not reached the active state; the same advice applies. The reasons "container monitor not available", "container health status unavailable" and "CPU cores not measured yet" also mean umh-core has no reading yet. The same advice applies.
* **Cannot create bridge - limit exceeded (N bridges maximum with X CPU cores, 1 core reserved for Redpanda)** — the instance has reached its bridge limit. The count includes the bridges already running and the waiting bridges listed before this one in config.yaml. The [Sizing Guide](/production/sizing-guide) explains the limit and what raises it.

## What to do

1. Read the reason on the bridge's status.
2. Fix the resource it names. For CPU, follow the [CPU Health](/production/cpu-health) page. For the bridge limit, follow the [Sizing Guide](/production/sizing-guide).
3. Wait. The bridge starts on its own once the resource is healthy again, or once the instance runs fewer bridges than its limit. Nothing needs to be redeployed.

## Emergency settings: Turn off bridge admission

Only if the bridge is needed now, use the emergency setting *Turn off bridge admission* (`agent.enableResourceLimitBlocking: false`). It turns off all three checks, including the bridge limit. Edit the instance's [Config File](/usage/instances/config-file):

```yaml
agent:
  enableResourceLimitBlocking: false
```

The setting takes effect without a restart. While it is `false`, the instance shows as degraded with the reason "Bridge admission is off", so it is not left in place by accident. It is an emergency fallback, not a permanent configuration: while bridge admission is off, nothing stops a new bridge from loading an instance that is already short on resources. Set it back to `true` once the resource problem is fixed.

## After a restart

After a restart, every bridge waits until the instance's resource health is proven. The first bridges in config.yaml order then start, up to the bridge limit. The rest keep waiting with the limit reason, and each starts once the instance runs fewer bridges than its limit, for example after another bridge is removed.


# Stand-alone Flow

> **Prerequisite:** Understand [Data Flow concepts](/usage/data-flows) and [Bridges](/usage/data-flows/bridges).

Stand-alone flows provide raw Benthos access for custom data processing when bridges or stream processors aren't sufficient. They're the fallback mechanism for write flows, external integrations, and specialized transformations.

## When to Use

Use stand-alone flows for:

* Point-to-point data transformation
* External system integration (databases, APIs, files)
* Custom processing that doesn't fit bridge patterns

Use bridges instead for:

* Industrial device connections
* Data that needs UNS organization
* Connections requiring health monitoring

## UI Capabilities

| Action                | Available | Notes                       |
| --------------------- | --------- | --------------------------- |
| Create flows          | ✅         | Via UI with YAML editor     |
| View flows            | ✅         | Listed in Standalone tab    |
| Edit configuration    | ✅         | Direct YAML editing         |
| Control state         | ✅         | Active/Stopped toggle       |
| Use example templates | ✅         | Pre-built samples available |
| Monitor status        | ✅         | Throughput and state        |
| View logs             | ✅         | Real-time log streaming     |
| View metrics          | ✅         | Performance monitoring      |
| Delete flows          | ✅         | Via context menu            |

## Creating a Stand-alone Flow (UI)

### Step 1: Navigate to Data Flows

Go to **Data Flows** → **Standalone** tab and click **Add Standalone Flow**:

![Standalone Flows List](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-6e82e3c2d9c2ecbf57d09b9f235fd425e59fef48%2Fstandalone-overview.png?alt=media)

### Step 2: Configure the Flow

The configuration interface has multiple sections:

![Standalone Flow Configuration](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-cbab451e25aaf34776f1fd2c2e76833442664179%2Fstandalone-detail-overview-1.png?alt=media)

**Key sections:**

* **General**: Name, instance, and state control (Active/Stopped)
* **Input (YAML)**: Configure data source (see [Benthos-UMH inputs](https://docs.umh.app/benthos-umh/input))
* **Processing (YAML)**: Transform data with processors
* **Output (YAML)**: Send to any destination (generic, not auto-generated like bridges)
* **Examples**: Pre-built templates including database bridges

### Step 3: Use Example Templates

Click the **Examples** tab to access pre-built configurations:

![Example Templates](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-291886ef0d3d38943b2388de5ee77d7751a4c02f%2Fstandalone-detail-samples.png?alt=media)

Available examples include:

* **mqtt\_to\_kafka\_historian\_bridge**: MQTT to Kafka with historian pattern
* **kafka\_to\_postgresql\_historian\_bridge**: Write to TimescaleDB (like Classic)
* **opcua\_to\_umh\_core**: OPC UA integration
* **generate\_to\_stdout**: Testing pattern

### Step 4: Control Flow State

Use the state dropdown to control execution:

![State Control](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-346fe3c9b8847792cea53d37b4e97288e95471a7%2Fstandalone-detail-state.png?alt=media)

* **Active**: Flow is running
* **Stopped**: Temporarily disabled (useful for maintenance)

### Managing Stand-alone Flows

Right-click any flow for quick actions:

![Context Menu](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-3e68a5ed1a01b080effb96100e840bfa39ec8d2e%2Fstandalone-overview-context-menu.png?alt=media)

Options include:

* **Standalone Flow**: Edit configuration
* **Logs**: View real-time logs
* **Metrics**: Monitor performance
* **Delete**: Remove the flow

### Enable Debug Logging

For troubleshooting data processing issues in stand-alone flows, enable the `debug_level` configuration option:

**Via YAML:**

```yaml
dataFlow:
  - name: custom-processor
    desiredState: active
    dataFlowComponentConfig:
      debug_level: true  # Enable detailed logging (default: false)
      benthos:
        # ...
```

**What it controls:**

* Log level: DEBUG (when `true`) vs INFO (when `false`, default)
* Detailed operational logging for input/output operations
* Processing step execution details
* Data transformation trace information

**Important:** Debug logging generates significantly more log data and may impact performance. **Use only during active troubleshooting** and disable afterward.

View logs via right-click → **Logs**. For comprehensive debug\_level configuration reference, see [Configuration Reference](/reference/configuration-reference).

## Configuration (YAML)

Stand-alone flows use standard Benthos configuration:

```yaml
dataFlow:
  - name: custom-processor
    desiredState: active
    dataFlowComponentConfig:
      benthos:
        input:
          # Any Benthos input
        pipeline:
          processors:
            # Processing logic
        output:
          # Any Benthos output
```

## Migrating from Benthos

Existing Benthos configurations work with minimal changes:

```yaml
# Original Benthos config
input:
  mqtt:
    urls: ["tcp://broker:1883"]
output:
  stdout: {}

# As UMH Core stand-alone flow
dataFlow:
  - name: migrated-flow
    desiredState: active
    dataFlowComponentConfig:
      benthos:
        input:
          mqtt:
            urls: ["tcp://broker:1883"]
        output:
          stdout: {}
```

## Next Steps

* Learn about [Bridges](/usage/data-flows/bridges) for device connectivity
* Explore [Stream Processors](/usage/data-flows/stream-processor) for data transformation
* See [Benthos-UMH Documentation](https://docs.umh.app/benthos-umh) for all processors


# Stream Processors

> **Prerequisite:** Understand [Data Flow concepts](/usage/data-flows) and complete the [Getting Started guide](/getting-started).

Stream processors transform data already inside the Unified Namespace, aggregating multiple device streams into business KPIs and metrics. They're the third type of data flow, specialized for transforming device models into business models.

## When to Use

Use stream processors for:

* Aggregating data from multiple devices into KPIs
* Creating business metrics from raw sensor data
* Calculating OEE, efficiency, or other derived values
* Reducing data volume for cloud transmission

Use bridges instead for:

* Getting data into the UNS initially
* Single device to model mapping
* Direct protocol connections

## UI Capabilities

| Action               | Available | Notes                                   |
| -------------------- | --------- | --------------------------------------- |
| Create processors    | ✅         | Visual configuration with topic browser |
| Select source topics | ✅         | Browse and multi-select                 |
| Map to model fields  | ✅         | Expression-based mapping                |
| View processors      | ✅         | Listed in Stream tab                    |
| Monitor throughput   | ✅         | Real-time message rates                 |
| Edit processors      | ✅         | Modify configuration                    |
| Delete processors    | ✅         | Remove when not needed                  |
| Code mode            | ✅         | Direct YAML editing                     |

## Creating a Stream Processor

Stream processors are created through the **Data Flows → Stream** tab:

![Stream Processors List](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-03fc326d1c5873a8f831005020af9d79b08bf9f3%2Fstream-processors.png?alt=media)

### Step 1: General Configuration

Configure basic settings and location hierarchy:

![Stream Processor General Configuration](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-dfe614cb5747aaf26c5486ea829185d2e3af00b4%2Fstream-processors-add.png?alt=media)

* **Name**: Unique identifier for your processor
* **Instance**: The UMH Core instance to run on
* **Data Contract**: Select the output model (creates validated structure)
* **Location Levels**: Set where in the hierarchy to output data (Level 0-4)

### Step 2: Topic Selection

Select source topics and they'll automatically get variable names:

![Topic Selection and Variable Names](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-28a43a394aa74c4da5fccd22d44260f7efe554cf%2Fstream-processors-add-topic-expression.png?alt=media)

* Browse available topics in your UNS
* Select multiple sources with checkboxes
* Variables auto-generated (e.g., `_10` for first topic, `_10_2` for second)
* Use "Show Selected Only" to filter view
* Copy variable names for use in expressions

### Step 3: Expression Mapping

Map source variables to model fields using expressions:

![Expression Mapping](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-28a43a394aa74c4da5fccd22d44260f7efe554cf%2Fstream-processors-add-topic-expression.png?alt=media)

* Each field in your data model appears on the right
* Enter expressions using the source variables
* Simple passthrough: `source_alias`
* Calculations: `_10 + _10_2` or more complex formulas
* The expressions support standard mathematical operations

### Step 4: Code Mode (Advanced)

Switch to Code Mode to see or edit the generated YAML configuration:

![Code Mode Configuration](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-d642500a9352c4efd35819c18b1ed78f7fbec1ba%2Fstream-processors-add-code-mode.png?alt=media)

The UI automatically generates YAML with:

* **sources**: Your selected topics with location\_path templates
* **mapping**: Field-to-expression mappings
* Comments explaining the structure

You can switch between UI and Code Mode anytime to fine-tune configurations.

**For detailed configuration instructions**, see the comprehensive [Stream Processors guide](/usage/data-modeling/stream-processors) in Data Modeling.

## How It Works

Stream processors subscribe to multiple UNS topics and output to a single model-based topic:

```mermaid
flowchart LR
    A["Multiple Device Model Topics<br/>(e.g., _pump_v1, _cnc_v1)"] --> B["Stream Processor<br/>(aggregation)"] --> C["Business Model Topic<br/>(e.g., _maintenance_v1)"]
```

Example: Combining temperature and pressure from different sensors into a pump model:

* Input: `enterprise.site._raw.temp_sensor_1`, `enterprise.site._raw.pressure_gauge_2`
* Output: `enterprise.site._pump_v1` with calculated efficiency

The system handles:

* Consumer group management for offset tracking
* Dependency-based processing (waits for all required inputs)
* Guaranteed output structure via data models

## Configuration (YAML)

While the UI is the primary way to create stream processors, they're stored as YAML:

```yaml
streamProcessor:
  - name: pump_efficiency_calc
    desiredState: active
    streamProcessorServiceConfig:
      config:
        model:
          name: pump
          version: v1
        sources:
          inlet_temp: enterprise.site._raw.inlet_temp
          outlet_temp: enterprise.site._raw.outlet_temp
        mapping:
          efficiency: "(outlet_temp - inlet_temp) / inlet_temp * 100"
```

See the [Stream Processors configuration guide](/usage/data-modeling/stream-processors) for detailed YAML structure.

## Key Differences from Other Data Flows

| Data Flow Type        | Purpose                       | Input Source                     | Output           |
| --------------------- | ----------------------------- | -------------------------------- | ---------------- |
| **Bridges**           | Get data INTO the UNS         | External devices (PLCs, sensors) | UNS topics       |
| **Stream Processors** | Transform data WITHIN the UNS | UNS topics                       | UNS model topics |
| **Stand-alone Flows** | Custom processing             | Any source                       | Any destination  |

## Learn More

For comprehensive stream processor configuration and examples, see the [Stream Processors guide](/usage/data-modeling/stream-processors) in Data Modeling documentation.


# Data Modeling

Data modeling in UMH Core transforms device data into business-ready information through structured schemas and validation.

## The Component Chain

```mermaid
flowchart LR
    A["Payload Shapes<br/>(value types)"] --> B["Data Models<br/>(structure)"] --> C["Data Contracts<br/>(enforcement)"] --> D["Data Flows<br/>(execution)"]
```

Each component builds on the previous:

* [**Payload Shapes**](/usage/data-modeling/payload-shapes) define what types of values are allowed
* [**Data Models**](/usage/data-modeling/data-models) use shapes to create hierarchical structure
* [**Data Contracts**](/usage/data-modeling/data-contracts) enforce models at runtime
* [**Data Flows**](/usage/data-flows) execute with or without contracts

## What You Can Model

In any UNS topic, data modeling controls specific portions:

`umh.v1.enterprise.site.area.line._contract.virtual.path.name`

```mermaid
flowchart LR
    subgraph Fixed["fixed"]
        A["umh.v1.enterprise.site.area.line._contract"]
    end
    subgraph Modeled["modeled"]
        B["virtual.path"] --> C["name"]
    end
    A --> B
```

* **Fixed**: Location hierarchy comes from bridge configuration
* **Modeled**: Everything after the contract is defined by your data model
  * Virtual path: Organizational folders (e.g., `motor.electrical`)
  * Name: The actual data point (e.g., `current`)

See [Topic Convention](/usage/unified-namespace/topic-convention) for complete structure.

## Data Flow Patterns

Data can follow these patterns based on your needs:

### Device Language (\_raw)

Start by exploring your equipment data with original naming:

```mermaid
flowchart LR
    A["Device"] --> B["Bridge"] --> C["_raw"] --> D["Topic Browser"]
```

**Example**: `umh.v1.enterprise.chicago.line-1.pump._raw.DB1.DW20`

* Exploration, debugging, quick connectivity
* Site engineers who know the PLC addressing
* Original tags preserved (e.g., `s7_address: "DB1.DW20"`)

### Device Models

Apply business naming directly in bridges:

```mermaid
flowchart LR
    A["Device"] --> B["Bridge + Model<br/>(one step)"] --> C["_pump_v1"] --> D["Applications"]
```

**Example**: `umh.v1.enterprise.chicago.line-1.pump._pump_v1.inlet_temperature`

* Consistent naming across equipment types
* Operations teams, local dashboards
* Most implementations start here

### Business Models

Transform device data into business KPIs:

```mermaid
flowchart LR
    A["Multiple _pump_v1"] --> B["Stream Processor"] --> C["_maintenance_v1"] --> D["Enterprise Apps"]
```

**Example**: `umh.v1.enterprise.chicago._maintenance_v1.work_orders.create`

* Aggregated metrics, business records
* Enterprise systems, management dashboards
* Required when scaling across sites

## The Two-Layer Architecture

**Sites and HQ both need their view of the same data.** This isn't a choice between approaches - it's about enabling both layers to work together:

### Layer 1: Device Models (Data Structure Within Equipment)

* Define WHAT data points exist in equipment (temperature, vibration)
* NOT the organizational structure (that's location\_path)
* Sites maintain control of their data definitions
* Original tags preserved in metadata
* Applied directly in bridges (one step)
* Primarily time-series data
* **Result**: Sites trust the system because they built it

### Layer 2: Business Models (Enterprise Metrics)

Created in TWO ways:

1. **Aggregation**: Stream processors combine device models into KPIs
2. **Direct**: Bridges connect to ERP/MES systems for business data

* Creates consistent metrics across sites
* Doesn't disturb site operations
* Multiple departments can create their own views
* Primarily relational data
* **Result**: Everyone gets the metrics they need

The key: Device models describe equipment internals, business models describe enterprise needs.

### Why Both Layers Matter

**Common failure patterns:**

* **Only device models**: Chaos across multiple sites, no standardization
* **Only business models**: Sites lose control, engineers reject the system

**The success formula:** Sites own their data structure (device models), everyone creates their views (business models). This is why stream processors exist - to bridge these layers without forcing change on sites.

## Key Concepts

### Location Path vs Device Model

* **Location Path**: WHERE equipment sits in your organization
  * Example: `enterprise.chicago.packaging.line-1.pump-01`
  * Defined in bridge configuration
  * This is your factory hierarchy
* **Device Model**: WHAT data points exist within that equipment
  * Example: `_pump_v1.temperature`, `_pump_v1.vibration.x-axis`
  * Defines internal data structure of a single device
  * NOT the organizational structure

### Name vs Tag

* **Name**: The data point identifier in UNS topics
* **Tag**: Industry term for a sensor/data point
* We use "name" for broader applicability (not just time-series)

### Virtual Path

Organizational folders within your data model:

* **Example**: `motor.electrical`, `diagnostics.vibration`
* **Purpose**: Groups related data points logically within a device

### Data Contract vs Data Model

* **Data Model**: Defines structure (template)
* **Data Contract**: Enforces structure (runtime validation)
* **Example**: Creating `pump` model auto-creates `_pump_v1` contract

### Time-Series vs Relational

* **Time-Series**: Single value with timestamp
  * Example: `{"timestamp_ms": 1733904005123, "value": 42.5}`
* **Relational**: Multiple fields in one message
  * Example: Work order with 10 fields

See [Payload Formats](/usage/unified-namespace/payload-formats) for details.

### Processing Methods

**In Bridges:**

* `tag_processor`: For time-series data from PLCs/sensors
* `nodered_js`: For relational data from ERP/MES systems

**In Stream Processors:**

* JavaScript expressions: For aggregating and transforming data
* Example: `total: "sensor1 + sensor2 + sensor3"`

## Choosing Your Data Flow

Based on your data source, choose the appropriate path:

| Source                 | Output         | Bridge Processor | Path                          |
| ---------------------- | -------------- | ---------------- | ----------------------------- |
| PLC/Sensor             | Device Model   | `tag_processor`  | Direct to \_pump\_v1          |
| PLC/Sensor             | Raw            | `tag_processor`  | Direct to \_raw (exploration) |
| ERP/MES                | Business Model | `nodered_js`     | Direct to \_workorder\_v1     |
| Multiple Device Models | Business Model | -                | Via Stream Processor          |

### Data Type Alignment

* **Device Models**: 90% time-series data
  * Temperature, pressure, vibration readings
  * Status indicators, counters, running hours
  * Process with: `tag_processor` in bridges
* **Business Models**: 90% relational data
  * Work orders, maintenance schedules
  * Production batches, quality reports
  * Process with: `nodered_js` in bridges OR aggregate via stream processors

## Implementation Patterns

### Pattern 1: Equipment Monitoring

```mermaid
flowchart LR
    A["PLC tags"] --> B["Bridge + tag_processor"] --> C["Device Model (_pump_v1)<br/>Location: enterprise.site.line.pump-01<br/>Model adds: .temperature, .pressure, .status"]
```

**When**: Connecting industrial equipment with time-series data

### Pattern 2: ERP Integration

```mermaid
flowchart LR
    A["SAP work orders"] --> B["Bridge + nodered_js"] --> C["Business Model (_workorder_v1)"]
```

**When**: Connecting business systems with relational data

### Pattern 3: Multi-Site Aggregation

```mermaid
flowchart LR
    A["site1._pump_v1"] --> SP
    B["site2._pump_v1"] --> SP
    C["site3._pump_v1"] --> SP
    SP["Stream Processor<br/>(JavaScript expressions)"] --> D["_maintenance_v1"]
```

**When**: Creating KPIs from multiple device models

### Pattern 4: Exploration First

```mermaid
flowchart LR
    A["Unknown PLC"] --> B["Bridge + tag_processor"] --> C["_raw"] --> D["Topic Browser"]
    D --> E["Design device model"] --> F["Apply in bridge"]
```

**When**: Understanding new equipment before modeling

## Why Models Are Immutable

**The scenario**: A data scientist builds a dashboard using `_pump_v1` for 500 pumps across 10 sites.

**Without immutability**: Someone modifies `_pump_v1`:

* Dashboard breaks - fields renamed
* Historical queries fail - structure changed
* Other sites stop working
* Weekend ruined

**With immutability**:

1. Create `_pump_v2` with changes
2. Test thoroughly
3. Migrate gradually
4. Deprecate v1 when safe
5. Dashboard keeps working throughout

## Common Questions

### When should I use data models?

Start with `_raw` for exploration. Add models when:

* You need consistent naming across sites
* Multiple applications consume the data
* Validation becomes critical
* You're ready for production

### How do I handle different equipment versions?

Create separate models:

* `_pump_v1` for older pumps
* `_pump_v2` for newer pumps with more sensors
* Stream processors can aggregate both

### What about equipment-specific data?

Use virtual paths to organize:

```
_pump_v1.motor.temperature
_pump_v1.motor.current
_pump_v1.diagnostics.vibration
```

## Next Steps

1. **Define value types**: [Payload Shapes](/usage/data-modeling/payload-shapes)
2. **Create structure**: [Data Models](/usage/data-modeling/data-models)
3. **Enforce validation**: [Data Contracts](/usage/data-modeling/data-contracts)
4. **Transform data**: [Stream Processors](/usage/data-modeling/stream-processors)

## Learn More

* [Getting Started Guide](/getting-started) - See data modeling in action
* [Bridges](/usage/data-flows/bridges) - Apply models during ingestion
* [Topic Convention](/usage/unified-namespace/topic-convention) - Understand topic structure


# Data Models

> This article assumes you've completed the [Getting Started guide](/getting-started) and understand the [data modeling concepts](/usage/data-modeling).

Data models define the hierarchical structure of your industrial data. They create the virtual paths and fields that organize raw data into meaningful information.

## Overview

In the [component chain](/usage/data-modeling#the-component-chain), models provide the structure:

```mermaid
flowchart LR
    A["Payload Shapes"] --> B["Data Models"] --> C["Data Contracts"] --> D["Data Flows"]
    B -.- N["Structure defined here"]
```

When you create a data model, you're defining:

* Virtual paths - organizational folders (e.g., `vibration`, `motor.electrical`)
* Fields - data endpoints with specific types (e.g., `temperature`, `pressure`)
* Relationships - how components nest and reference each other

## UI Capabilities

The Management Console provides full control over data models:

| Feature              | Available | Notes                                    |
| -------------------- | --------- | ---------------------------------------- |
| View model list      | ✅         | Shows all models with versions           |
| Create models        | ✅         | Visual editor with YAML preview          |
| View model details   | ✅         | Inspect structure and configuration      |
| Create new versions  | ✅         | Models are immutable, edit by versioning |
| Reference sub-models | ✅         | Link to other models via `_refModel`     |
| Delete models        | ✅         | Remove unused model versions             |
| Direct editing       | ❌         | Use "New Version" to modify              |

![Data Models List](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-ed7eb5308527c12f86b88c7bf64d7dbde1d50b85%2Fdata-models.png?alt=media)

**What you see in the UI:**

* **Name**: Model identifier (e.g., `cnc`, `pump`, `temperature-sensor`)
* **Instance**: Which UMH Core instance owns the model
* **Description**: Optional description of the model's purpose
* **Latest**: Current version number (v1, v2, etc.)

### Model Actions

Click the three-dot menu (⋮) on any model to access actions:

![Data Model Actions](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-933de38bf55804d55d53cbd890f0dd8ff0de458a%2Fdata-models-context-menu.png?alt=media)

* **Data Model**: View the model's structure and YAML configuration
* **New Version**: Create a new version with modifications (since models are immutable)
* **Delete**: Remove the model (only if not in use by contracts or bridges)

![Data Model Creation](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-572bd67a6437713948a8d4ce2b20888e218d8d80%2Fdata-models-add.png?alt=media)

## Configuration

### Basic Structure

```yaml
datamodels:
  - name: pump                    # Model name
    description: "Pump monitoring" # Optional description
    version:
      v1:                         # Version identifier
        structure:                # Hierarchical definition
          pressure:
            inlet:
              _payloadshape: timeseries-number
            outlet:
              _payloadshape: timeseries-number
```

### How Structure Becomes Topics

Model structure directly maps to UNS topics:

```yaml
structure:
  vibration:        # Creates: .../_pump_v1.vibration
    x-axis:         # Creates: .../_pump_v1.vibration.x-axis
```

**Complete topic path:** `umh.v1.enterprise.site._pump_v1.vibration.x-axis`

```mermaid
flowchart LR
    subgraph Fixed["fixed"]
        A["umh.v1.enterprise.site"]
    end
    subgraph Contract["contract"]
        B["_pump_v1"]
    end
    subgraph Model["from model"]
        C["vibration.x-axis"]
    end
    A --> B --> C
```

## The Three Building Blocks

### Fields

```yaml
temperature:
  _payloadshape: timeseries-number  # Accepts numeric values
```

**Characteristics:**

* Has `_payloadshape` property
* Creates a topic endpoint that accepts data
* References a [payload shape](/usage/data-modeling/payload-shapes) for validation
* Cannot have child elements

### Folders

```yaml
vibration:           # Folder - no _payloadshape
  x-axis:           # Field inside folder
    _payloadshape: timeseries-number
  y-axis:           # Field inside folder
    _payloadshape: timeseries-number
```

**Characteristics:**

* No `_payloadshape` property
* Groups related fields or other folders
* Creates hierarchy in topic path
* Can nest multiple levels deep

### Sub-Models

Define once, use everywhere:

```yaml
# Define reusable motor model
datamodels:
  - name: motor
    version:
      v1:
        structure:
          rpm:
            _payloadshape: timeseries-number
          temperature:
            _payloadshape: timeseries-number

# Reference in pump model
datamodels:
  - name: pump
    version:
      v1:
        structure:
          pressure:
            inlet:
              _payloadshape: timeseries-number
```

Topics created:

* `_pump_v1.pressure.inlet`
* `_pump_v1.motor.rpm`
* `_pump_v1.motor.temperature`

**Benefits:**

* Single source of truth
* Consistent structure across models
* Update once, reflected everywhere

## Version Evolution

Models are immutable once created. To add fields, create a new version:

### Why Immutability?

From the [README](/usage/data-modeling#why-are-models-immutable):

* Models are contracts between teams
* Dashboards depend on stable structure
* Historical data queries must not break

### Evolution Pattern

**Version 1 - Basic:**

```yaml
version:
  v1:
    structure:
      temperature:
        _payloadshape: timeseries-number
```

**Version 2 - Add pressure:**

```yaml
version:
  v2:
    structure:
      temperature:
        _payloadshape: timeseries-number
      pressure:                          # New field
        _payloadshape: timeseries-number
```

**Migration steps:**

1. Create v2 with additions
2. Deploy new bridges using v2
3. Update dashboards to v2
4. Keep v1 running during transition
5. Deprecate v1 when safe

## Relationship to Contracts

Models define structure, but don't enforce it. That's where [data contracts](/usage/data-modeling/data-contracts) come in:

| Component    | Purpose            | Example                       |
| ------------ | ------------------ | ----------------------------- |
| **Model**    | Defines structure  | `pump` model with fields      |
| **Contract** | Enforces structure | `_pump_v1` validates messages |

When you create a model in the UI:

1. Model `pump` version `v1` is created
2. Contract `_pump_v1` is auto-generated
3. Contract becomes available in bridges

Without a contract, a model is just documentation. With a contract, it becomes validation.

## Next Steps

* [Data Contracts](/usage/data-modeling/data-contracts) - Make models mandatory
* [Payload Shapes](/usage/data-modeling/payload-shapes) - Specify data types for fields
* [Stream Processors](/usage/data-modeling/stream-processors) - Transform device models to business models


# Data Contracts

> This article assumes you've completed the [Getting Started guide](/getting-started) and understand the [data modeling concepts](/usage/data-modeling).

Data contracts are the enforcement mechanism that makes data models mandatory. Without a contract, models are just documentation - contracts make validation happen.

## Overview

In the [component chain](/usage/data-modeling#the-component-chain), contracts sit between models and execution:

```mermaid
flowchart LR
    A["Payload Shapes"] --> B["Data Models"] --> C["Data Contracts"] --> D["Data Flows"]
    C -.- N["Enforcement happens here"]
```

When you specify a data contract in a bridge, the UNS output plugin validates every message against the associated model.

## UI Capabilities

The Management Console provides read-only access to contracts:

| Feature            | Available | Notes                                           |
| ------------------ | --------- | ----------------------------------------------- |
| View contract list | ✅         | Shows all contracts with their models           |
| Filter/search      | ✅         | Filter by name, instance, or model              |
| View associations  | ✅         | See which model version each contract enforces  |
| View usage         | ✅         | Shows count of stream processors using contract |
| Create contracts   | ❌         | Auto-created when creating models in UI         |
| Edit contracts     | ❌         | Immutable once created                          |
| Delete contracts   | ❌         | Must be done via config.yaml                    |

![Contracts List](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-d37a206edab5b69c9f7d84bab93c2a336a520fc1%2Fdata-contracts.png?alt=media)

**What you see in the UI:**

* **Name**: Contract identifier (e.g., `_cnc_v1`, `_pump_v2`)
* **Instance**: Which UMH Core instance owns the contract
* **Model**: The model and version being enforced (e.g., `cnc (v1)`)
* **Stream Processors**: Count of processors using this contract

### Auto-Creation via UI

When you create a model in the UI, it automatically generates a matching contract:

1. Create model `pump` version `v1` in UI
2. System auto-creates contract `_pump_v1`
3. Contract immediately available for use in bridges

## Configuration

Access configuration via: Instances → Select instance → `...` → Config File

### Basic Structure

```yaml
datacontracts:
  - name: _pump_v1          # Contract name (used in bridges)
    model:
      name: pump            # References a data model
      version: v1           # Specific version to enforce
```

### Naming Convention

Contracts follow the pattern `_modelname_version`:

* Always start with underscore
* Include model name
* End with version number
* Examples: `_pump_v1`, `_temperature_sensor_v2`, `_workorder_v1`

## Enforcement Mechanism

### Where Validation Happens

```mermaid
flowchart LR
    A["Bridge"] --> B["UNS Output Plugin"] --> C{"Contract Check"} --> D["Kafka/Redpanda"]
    C -.- N["Validation happens here"]
```

The UNS output plugin (`output: uns: {}`) performs validation:

1. **Reads metadata**: Extracts `data_contract` from message
2. **Looks up contract**: Finds the associated model
3. **Validates structure**: Checks topic path matches model
4. **Validates payload**: Ensures data types match shapes
5. **Result**:
   * ✅ Valid → Message published to topic
   * ❌ Invalid → Message rejected, bridge degraded

### Validation Failures

When validation fails:

```
ERROR: schema validation failed for message with topic 'umh.v1.enterprise.site._pump_v1.invalid.path':
Valid virtual_paths are: [pressure, temperature, motor.rpm].
Your virtual_path is: invalid.path
```

Result:

* Message rejected (not published)
* Bridge enters degraded state
* Error logged with details
* Bridge retries with backoff

## Contract Types

### The Special \_raw Contract

```yaml
# No explicit definition needed - always available
msg.meta.data_contract = "_raw";
```

* Accepts any structure
* No validation performed
* Use for exploration and development
* Bridge never goes degraded from data issues

### Model-Based Contracts

```yaml
datacontracts:
  - name: _pump_v1
    model:
      name: pump
      version: v1
```

* Enforces exact model structure
* Validates data types via payload shapes
* Rejects non-conforming messages
* Use for production systems

### Relationship to Data Types

| Data Type       | Common Contracts                    | Validation             |
| --------------- | ----------------------------------- | ---------------------- |
| Raw/Exploration | `_raw`                              | No validation          |
| Device Models   | `_pump_v1`, `_sensor_v1`, `_cnc_v1` | Model-based validation |
| Business Models | `_workorder_v1`, `_maintenance_v1`  | Always strict          |

## Examples

### Simple Device Contract

```yaml
# Model definition
datamodels:
  - name: temperature-sensor
    version:
      v1:
        structure:
          celsius:
            _payloadshape: timeseries-number

# Contract (auto-created or manual)
datacontracts:
  - name: _temperature-sensor_v1
    model:
      name: temperature-sensor
      version: v1

# Usage in bridge
msg.meta.data_contract = "_temperature-sensor_v1";
msg.meta.tag_name = "celsius";
msg.payload = 23.5;
```

## Relationship to Stream Processors

Stream processors don't use contracts directly - they reference models:

```yaml
templates:
  streamProcessors:
    pump_aggregator:
      model:           # Direct model reference
        name: pump
        version: v1
```

However, if a matching contract exists (`_pump_v1`), the stream processor's output will be validated against it automatically.


# Payload Shapes

> This article assumes you've completed the [Getting Started guide](/getting-started) and understand the [data modeling concepts](/usage/data-modeling).

Payload shapes define the structure and validation rules for message payloads. While built-in shapes handle most time-series data, custom shapes enable relational data modeling for complex business records.

## Overview

In the [component chain](/usage/data-modeling#the-component-chain), payload shapes provide the foundation:

```
Payload Shapes → Data Models → Data Contracts → Data Flows
       ↑
  Value types defined here
```

## UI Capabilities

Payload shapes have no UI component:

| Feature         | Available | Notes                                     |
| --------------- | --------- | ----------------------------------------- |
| View shapes     | ❌         | Configuration file only                   |
| Create shapes   | ❌         | Edit config.yaml directly                 |
| Edit shapes     | ❌         | Modify config.yaml                        |
| Built-in shapes | ✅         | Always available, no configuration needed |

**Configuration location**: Select your instance under `Instances` in the menu, press `...`, then `Config File`. Find payload shapes under the `payloadShapes:` section

## Built-in Shapes

UMH provides three built-in shapes that handle 90% of industrial data:

### timeseries-number

For numeric sensor data and measurements.

**Structure:**

```json
{
  "timestamp_ms": 1733904005123,
  "value": 42.5
}
```

**Use cases:** Temperature, pressure, speed, counts, any numeric measurement

### timeseries-string

For text-based status and identifiers.

**Structure:**

```json
{
  "timestamp_ms": 1733904005123,
  "value": "running"
}
```

**Use cases:** Machine states, batch IDs, product codes, operator names

### timeseries-boolean

For on/off process values.

**Structure:**

```json
{
  "timestamp_ms": 1733904005123,
  "value": true
}
```

**Use cases:** Machine running, alarm active, valve open/closed, any true/false state

## When to Use Custom Shapes

Custom shapes are specifically for **relational data** - business records with multiple related fields that must stay together.

**Time-series vs Relational:**

| Data Type                | Shape                  | Example                          |
| ------------------------ | ---------------------- | -------------------------------- |
| Single values over time  | Built-in timeseries-\* | Temperature readings             |
| Complex business records | Custom shape           | Work orders, quality inspections |

Learn more: [Payload Formats](/usage/unified-namespace/payload-formats)

## Configuration

Access configuration via: Instances → Select instance → `...` → Config File

### Defining Custom Shapes

```yaml
payloadShapes:
  work-order:    # Shape name (referenced in models)
    description: "Work order record"
    fields:
      orderId:
        _type: string
      productId:
        _type: string
      quantity:
        _type: number
      status:
        _type: string    # created/in-progress/completed
      operatorId:
        _type: string
      timestamp:
        _type: string    # ISO 8601 format
```

**Key points:**

* Field types are `string`, `number`, `boolean` (see [Data Model Type Definitions](/reference/data-model-type-definitions))
* All fields are required
* Shape names must be unique

### Using Custom Shapes in Models

Reference custom shapes in your data models to create CRUD-like endpoints:

```yaml
dataModels:
  - name: work-order
    version:
      v1:
        structure:
          create:
            _payloadshape: work-order    # Custom shape for new orders
          update:
            _payloadshape: work-order    # Same shape for updates
          delete:
            _payloadshape: work-order    # Or simplified delete shape
```

This creates topics like:

* `enterprise.site._work_order_v1.create` - New work orders
* `enterprise.site._work_order_v1.update` - Update existing orders
* `enterprise.site._work_order_v1.delete` - Delete orders

## Processing Relational Data

### Example: Processing Work Orders

```javascript
// In nodered_js processor for work order creation
msg.meta.data_contract = "_work_order_v1";
msg.meta.tag_name = "create";  // Maps to the 'create' endpoint

// Build work order record
msg.payload = {
  orderId: "WO-" + Date.now(),
  productId: msg.product,
  quantity: msg.qty,
  status: "created",
  operatorId: msg.operator,
  timestamp: new Date().toISOString()
};

return msg;
```

This creates a message at `enterprise.site._work_order_v1.create` with the complete work order data.

## Relationship to Other Components

Payload shapes are referenced by [data models](/usage/data-modeling/data-models), which are then enforced by [data contracts](/usage/data-modeling/data-contracts).

## Next Steps

* **Define structure**: [Data Models](/usage/data-modeling/data-models) - Create hierarchies using shapes
* **Enforce validation**: [Data Contracts](/usage/data-modeling/data-contracts) - Make shapes mandatory
* **Build pipelines**: [Data Flows](/usage/data-flows) - Process data with bridges


# Stream Processors

> This article assumes you've completed the [Getting Started guide](/getting-started) and understand the [data modeling concepts](/usage/data-modeling).

Stream processors are one way to create business models by transforming multiple device model streams into KPIs and metrics.

## Overview

Stream processors aggregate device data into business KPIs:

```mermaid
flowchart LR
    A["Multiple Device Model Topics"] --> B["Stream Processor"] --> C["Business Model Topic"]
    B -.- N["Aggregation happens here"]
```

Stream processors are for:

* **Aggregating** data from multiple devices
* **Creating** business KPIs from raw metrics
* **Calculating** derived values across equipment
* **Reducing** data volume for cloud transmission

## UI Capabilities

The Management Console provides visual stream processor creation:

| Feature             | Available | Notes                                |
| ------------------- | --------- | ------------------------------------ |
| View processor list | ✅         | Shows all processors with throughput |
| Create processors   | ✅         | Visual or code mode                  |
| Topic selection     | ✅         | Browse and select source topics      |
| Expression mapping  | ✅         | Map topics to model fields           |
| Edit processors     | ✅         | Modify existing processors           |
| Delete processors   | ✅         | Remove unused processors             |

![Stream Processors List](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-03fc326d1c5873a8f831005020af9d79b08bf9f3%2Fstream-processors.png?alt=media)

**What you see in the UI:**

* **Name**: Processor identifier
* **Instance**: Which UMH Core instance runs the processor
* **Throughput**: Messages processed per second

## Creating a Stream Processor (UI)

The UI is the primary way to create stream processors. It generates the YAML configuration automatically.

### Step 1: Basic Configuration

![Add Stream Processor](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-dfe614cb5747aaf26c5486ea829185d2e3af00b4%2Fstream-processors-add.png?alt=media)

1. Click **Add Stream Processor**
2. Enter **Name**: Descriptive identifier
3. Select **Instance**: Where to run the processor
4. Choose **Data Contract**: Output structure (e.g., `_production_v1`)
5. Set **Location levels**: Where in the UNS hierarchy (enterprise, site, etc.)

### Step 2: Topic Selection

![Topic Selection](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-28a43a394aa74c4da5fccd22d44260f7efe554cf%2Fstream-processors-add-topic-expression.png?alt=media)

Select source topics to aggregate:

* Browse available topics
* Select multiple sources
* Variable names auto-generated (e.g., `_10` for each topic)

### Step 3: Expression Mapping

Map source data to output fields:

* Use variable names from selected topics
* Write expressions (e.g., `_10 + _10_2` for sum)
* Each field in your model needs a mapping

### Step 4: Code Mode (Advanced)

![Code Mode Configuration](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-d642500a9352c4efd35819c18b1ed78f7fbec1ba%2Fstream-processors-add-code-mode.png?alt=media)

Switch to code mode to see or edit the generated YAML:

```yaml
sources:
  # Define your data sources (specific topics)
  source_alias: "{{ .location_path }}._raw.some_field"
mapping:
  # Map source aliases to data model fields
  field_name: "source_alias"
```

**Note**: The UI generates this YAML configuration. You can switch between visual and code modes.

## Configuration

Access configuration via: Instances → Select instance → `...` → Config File

### How Stream Processors Work

Stream processors use templates for reusability and subscribe to specific topics:

```yaml
templates:
  streamProcessor:
    production_aggregator:
      model:
        name: production-metrics
        version: v1
      sources:  # Specific topics to subscribe to
        cnc1: "{{ .location_path }}.cnc-01._cnc_v1.vibration.x-axis"
        cnc2: "{{ .location_path }}.cnc-02._cnc_v1.vibration.x-axis"
        cnc3: "{{ .location_path }}.cnc-03._cnc_v1.vibration.x-axis"
      mapping:  # Transform to model fields
        total-vibration: "cnc1 + cnc2 + cnc3"
        avg-vibration: "(cnc1 + cnc2 + cnc3) / 3"
        machines-running: "(cnc1 > 0 ? 1 : 0) + (cnc2 > 0 ? 1 : 0) + (cnc3 > 0 ? 1 : 0)"
```

**Key concepts:**

* Each processor gets a unique consumer group (for offset tracking)
* Sources define exact topics to subscribe to (not patterns)
* Outputs go to: `umh.v1.{{ .location_path }}._<model>_<version>.<field>`

### Deploying Stream Processors

```yaml
streamProcessor:
  - name: line_a_metrics
    desiredState: active
    streamProcessorServiceConfig:
      templateRef: "production_aggregator"
      location:
        0: enterprise
        1: site
        2: line-a
```

### Expression Language

Mappings use JavaScript expressions with dependency-based evaluation:

```yaml
mapping:
  # Direct pass-through
  value: "source1"

  # Math operations
  total: "source1 + source2 + source3"
  average: "(source1 + source2) / 2"

  # Conditionals
  status: "value > 100 ? 'HIGH' : 'NORMAL'"

  # Complex logic
  efficiency: |
    var produced = source1;
    var planned = source2;
    return (produced / planned) * 100;
```

**Evaluation behavior:**

* Static mappings (no dependencies) evaluate on every message
* Dynamic mappings only evaluate when their dependencies arrive

## Relationship to Other Components

Stream processors work with:

* [**Data Models**](/usage/data-modeling/data-models): Define output structure via `model` reference
* [**Data Contracts**](/usage/data-modeling/data-contracts): Enforce output validation
* [**Bridges**](/usage/data-flows/bridges): Provide input data (device model topics)

## Next Steps

* **Create device data**: [Bridges](/usage/data-flows/bridges) - Connect devices first
* **Define structures**: [Data Models](/usage/data-modeling/data-models) - Create output models
* **Learn concepts**: [README](/usage/data-modeling) - Understand data modeling approach


# Historian

A historian is the standard industrial term for the system that records process data over time, so operators can go back and ask what a machine was doing last Tuesday. AVEVA PI and GE Proficy fill that role in a classic plant.

Ours stores data from the Unified Namespace in TimescaleDB rather than a proprietary store, so you can read history back with ordinary SQL and build Grafana dashboards on it. The UNS keeps the live value of every tag; the Historian keeps the past.

{% hint style="info" %}
**Early access.** The Historian is not switched on by default. If you want to use it, get in touch with us on [Discord](https://discord.gg/F9mqkZnm8U) or through your usual UMH contact, and we will show you how to get started.
{% endhint %}

## How it fits together

```mermaid
flowchart LR
    SRC["Bridges / Stream Processors"] --> UNS["Unified Namespace"]
    UNS --> H["Historian bridge<br/>(one per contract)"] --> TS["TimescaleDB<br/>(umh schema)"] --> G["Grafana"]
```

You configure the database connection **once per instance**, in the instance's Plugins tab. Every Historian bridge on that instance then reuses it, so no credentials are copied into bridge configs, and changing the connection updates the bridges that reference it.

Three steps get you from a running UNS to a dashboard:

1. Set up the connection (this page). Tell the instance where its database is.
2. [Save data to the Historian](/usage/historian/save-to-historian). Create one bridge per data contract you want to keep.
3. [Query the data](/usage/historian/querying). Copy queries out of the topic browser into Grafana or psql.

## Prerequisites

* **A TimescaleDB database**, reachable from the umh-core instance. If you don't have one, [Recommended UMH Stack](/production/deployment/docker-compose/additional-services/recommended-umh-stack) brings up umh-core, PgBouncer, TimescaleDB, and Grafana together. To add just the database to a running instance, see [TimescaleDB](/production/deployment/docker-compose/additional-services/timescaledb).
* **PostgreSQL 16 or newer**, with the `timescaledb` and `ltree` extensions available. Version 16 is the floor because older `ltree` labels reject hyphens, and location paths such as `line-1` contain them.
* **A login role**, created before the first bridge starts. The bridge logs in as this role and creates the `umh` schema it owns; it cannot create the role itself. A database-level grant is enough, so no privileges on `public` are needed:

  ```sql
  CREATE ROLE umh_owner WITH LOGIN PASSWORD 'change-me';
  GRANT CREATE, CONNECT ON DATABASE umh TO umh_owner;
  ```
* **Data in the UNS**, already flowing. The Historian archives what is in the namespace; it does not read from devices itself.

## Set up the connection

1. Open the instance, go to the **Plugins** tab, and add the **Historian** plugin.
2. Fill in the TimescaleDB connection:

   | Field        | Default     | Notes                                                          |
   | ------------ | ----------- | -------------------------------------------------------------- |
   | **Host**     | required    | Hostname or IP, resolved *from inside the umh-core container*. |
   | **Port**     | `5432`      | Point this at PgBouncer if you use one.                        |
   | **Database** | `umh`       | Must already exist.                                            |
   | **User**     | `umh_owner` | The login role from the prerequisites.                         |
   | **Password** | required    | Stored in `config.yaml`, hidden in the UI, redacted in logs.   |
   | **SSL Mode** | `require`   | `require` or `disable`.                                        |
3. Save. The instance opens a connection and reports the result on the plugin's overview card.

`require` encrypts the connection but does not verify the server certificate. `disable` turns TLS off entirely.

Full certificate verification (`verify-full`) cannot be configured from the Management Console yet; support is coming in a later release. A connection that already sets `sslmode: verify-full` in `config.yaml` keeps working, along with its `sslrootcert`, `sslcert`, and `sslkey` paths.

### Reading the connection status

The overview card is not shown in the plugin list. Go to **Instances** → your instance → **Plugins** → **Historian** and click **Open**. Saving a new connection also leaves you on the plugin list, so that click is needed either way.

The overview card shows the connection settings plus three live fields, refreshed once per second:

| Field         | Meaning                                                                                                                                           |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Reachable** | The endpoint answered. It stays `true` even when the server rejects the credentials. Only a network fault or timeout makes it `false`.            |
| **Auth**      | Whether the server accepted the login role, password, and database name. `unknown` when nothing answered, so authentication could not be checked. |
| **Latency**   | Round-trip time of the check query.                                                                                                               |

The check is a single `SELECT 1` over one pooled connection. It tells you the database is reachable and the credentials work; it says nothing about whether a particular bridge is writing. Per-bridge throughput, health, and errors stay in the **Data Flows** bridge list.

Connections are recycled every five minutes, so a password rotated on the server surfaces as an authentication failure within that window rather than being masked by a long-lived session.

### Editing and removing

Editing the connection updates every bridge that references it, so you don't touch the bridges. Leave the password field blank to keep the stored one; the Management Console never receives it back, so an empty value means "unchanged", not "clear".

Adding a Historian connection is create-only. If one already exists, edit it instead; a repeated add is rejected rather than silently overwriting your settings.

## Where it is stored

The connection lives in the instance's `config.yaml` as a single shared block:

```yaml
historian:
  timescale:
    host: timescaledb.example.com
    port: 5432
    database: umh
    username: umh_owner
    password: change-me
    sslmode: require
```

This block is the one place the credentials are written. Every Historian bridge on the instance reads them from here instead of carrying its own copy.

Bridge templates read it through the reserved `{{ .historian.timescale.* }}` scope: `host`, `port`, `database`, `username`, `password`, `sslmode`, `sslrootcert`, `sslcert`, and `sslkey`. See [Variables](/reference/variables).

A bridge counts as a Historian bridge when its write flow's destination protocol is `historian`. umh-core then targets the bridge's health check at the shared connection rather than at a host and port entered on the bridge, so the check follows the connection whenever you change it.

## Next steps

* [Save data to the Historian](/usage/historian/save-to-historian)
* [Query the Historian](/usage/historian/querying)


# Save data to the Historian

A Historian bridge subscribes to the Unified Namespace and writes one data contract into TimescaleDB. You create one bridge per contract you want to keep.

The bridge is a write flow only: it has no read flow, because its source is the UNS rather than a device.

## Before you start

Set up the instance's [Historian connection](/usage/historian#set-up-the-connection) first. Without it, the bridge has nothing to connect to and deployment fails with an error saying so.

## Create the bridge

1. Go to **Data Flows** → **Add Bridge**, and choose **From Scratch**.
2. In the template list, pick **TimescaleDB (PostgreSQL, auto-connect) — Historian**. The **Vendor** filter narrows the list to the TimescaleDB templates.
3. Select the instance. The connection host and port are prefilled read-only from that instance's Historian connection, so the bridge's health check always targets the configured database.
4. In the write flow's output, set `data_contract_name` to the contract you want to store, written **without the version suffix**: `pump`, not `pump_v1`.
5. Deploy.

There is no read flow to configure and no address table to map, so the protocol and address steps of the general [bridge walkthrough](/usage/data-flows/bridges) don't apply.

{% hint style="warning" %}
**Drop the version, and no dashes.** `data_contract_name` is the bare contract name: no leading underscore, no `_vN` suffix. For the UNS contract `_pump_v1`, write `pump`.

A contract whose name contains a dash cannot be stored by the Historian at all. The name becomes a PostgreSQL table identifier, which allows only lowercase letters, numbers, and underscores.
{% endhint %}

The only value you normally change is `data_contract_name`. Credentials, database, and TLS mode come from the shared connection through `{{ .historian.timescale.* }}`, so they are never entered twice and follow the connection if you later change it.

### Naming the data contract

Every version of a contract shares one set of tables, so `pump` stores `_pump_v1` and `_pump_v2` alike. That is why the version is left off: the tables hold the contract, not one version of it.

The Management Console rejects dashes in new data model names for the table-identifier reason above. Models created before that check keep working everywhere else, but a Historian bridge cannot store them under their own name.

The default is `historian`, the generic UNS time-series contract.

### What the bridge writes

For `data_contract_name: pump`, the bridge creates and fills two hypertables in a dedicated `umh` schema:

| Table                | Contents                                                                                             |
| -------------------- | ---------------------------------------------------------------------------------------------------- |
| `umh.value_pump`     | One row per (tag, timestamp). Numbers and booleans in `value_num`, strings and JSON in `value_text`. |
| `umh.attribute_pump` | The message metadata as a JSON object, rewritten only when the key set changes.                      |

Tag identity lives in the shared dimension tables `umh.topic`, `umh.tag`, and `umh.location`. See [Query the Historian](/usage/historian/querying) for the layout and how to read it back.

The bridge subscribes to the whole UNS by default and drops messages whose contract doesn't match `data_contract_name`, so there is no topic regex to keep in sync. Narrow the source topics only if you want to stop unrelated messages from reaching the output at all.

## Connecting to a different database

The **TimescaleDB (PostgreSQL) — Historian** template is the same output without the shared connection: you enter host and port in the connection step and set the username, password, and SSL mode in the output config yourself. Use it for a database that isn't the instance's Historian, such as a second archive or a customer-managed database with its own role.

Everything below applies to both templates; only where the connection details come from differs.

## Advanced options

The output supports retention, compression, metadata filtering, batching, and timeouts. They are plain YAML fields you add to the output config in the bridge editor. See the [Historian output reference](https://docs.umh.app/benthos-umh/output/historian) for the full field list and defaults.

`compress_after` and `retention` are the exceptions. Both are applied once, when the tables are first created, so editing them later has no effect on an existing database: a config edit must not silently change how production history is compressed or deleted. To change them, update the TimescaleDB policies directly on both hypertables, then set the same value in the bridge config so the drift warning stops.

## Troubleshooting

**The bridge fails immediately after deploying.** The output verifies the connection at startup rather than failing later on the first write. It fails fast on an unreachable host, a PostgreSQL older than 16, a missing `timescaledb` or `ltree` extension, and a role that can connect but cannot `INSERT` into the contract's tables. The error names which one. Check the plugin overview card in the Plugins tab first: if **Auth** is not valid there, fix the connection before looking at the bridge.

**The bridge is running but nothing arrives.** The output logs once when it stores its first message, and once if data is flowing but none of it matches `data_contract_name`. A mismatch between the contract in your topics and the configured name is the usual cause.

**Rows are being dropped.** Two cases can never be written and are dropped rather than retried: a tag whose datatype flips between numeric and text, and two different values at the same millisecond. Both are logged with the tag name and counted on the `historian_rows_poisoned` metric. The [Historian output reference](https://docs.umh.app/benthos-umh/output/historian) has a runbook for resolving them.

Connection loss, deadlocks, a missing grant, and a full disk are all retried rather than dropped. A database restart mid-stream loses nothing: held messages replay, and an identical value at the same timestamp is absorbed.

## Next steps

* [Query the Historian](/usage/historian/querying)


# Query the Historian

Once a [Historian bridge](/usage/historian/save-to-historian) is storing a contract, the data is ordinary TimescaleDB, and any SQL client or Grafana can read it. The fastest route is to copy a query out of the topic browser; the schema further down is for when you write your own.

The Management Console never runs these queries. It generates the SQL and you paste it into Grafana or psql.

## Copy a query from the topic browser

1. Open the **Topic Browser** and select a tag.
2. Choose the **Grafana** or **TimescaleDB** tab.

   The two differ only in the time filter. Grafana emits `$__timeFilter(ts)`, which the dashboard's time picker fills in. TimescaleDB emits a fixed window sized to the resolution, so the query returns roughly 60 points when pasted straight into psql.
3. Adjust the controls:
   * **Aggregate** buckets the data with `time_bucket()` and returns `avg`, `min`, and `max`. Turn it off for the raw rows. Text tags cannot be aggregated, so the toggle is disabled for them and the query always returns raw values.
   * **Resolution** sets the bucket width: `$__interval`, `1 second`, `10 seconds`, `1 minute`, `1 hour`, or `raw`. Grafana starts at `$__interval`, which lets Grafana pick a width from the panel's time range and width; TimescaleDB has no such macro and starts at `1 minute`.
4. Copy it.

The panel is generated from the topic name, not from the database, so it appears for every tag, including ones no Historian bridge has stored yet. The panel says as much. Those queries are valid and return no rows. When the tag's datatype isn't known yet, the query defaults to the numeric column. Switch it to `value_text` if the tag holds strings.

## Schema

Values are stored per contract; identity is shared across contracts.

```mermaid
erDiagram
    "umh.value_pump"     }o--|| "umh.topic" : topic_id
    "umh.attribute_pump" }o--|| "umh.topic" : topic_id
    "umh.topic"          }o--|| "umh.tag" : tag_id
    "umh.topic"          }o--|| "umh.location" : location_id

    "umh.value_pump" {
        bigint      topic_id   PK
        timestamptz ts         PK
        double      value_num
        text        value_text
    }
    "umh.attribute_pump" {
        bigint      topic_id  PK
        timestamptz ts        PK
        jsonb       attribute
    }
    "umh.topic" {
        bigserial topic_id    PK
        bigint    location_id FK
        bigint    tag_id      FK
    }
    "umh.tag" {
        bigserial  tag_id             PK
        text       name
        text       virtual_path
        text       data_contract_name
        value_type value_type
    }
    "umh.location" {
        bigserial location_id PK
        ltree     path
    }
```

`umh.value_pump` and `umh.attribute_pump` are the per-contract tables, named after `data_contract_name`; the three dimension tables are shared. `attribute` is a JSON object, queryable with `attribute->>'key'` and `attribute @> '{...}'`.

`value_type` on `umh.tag` records whether a tag is numeric or text. It is set on first write and cannot change afterwards, which is why a tag that flips datatype is dropped rather than stored.

### Resolving a tag

`umh.get_topic_id(location_path, virtual_path, data_contract, tag_name)` hides that join for single-tag lookups. It is what the generated queries use:

```sql
SELECT ts, value_num
FROM   umh.value_pump
WHERE  topic_id = umh.get_topic_id('enterprise.site.area.line', '', 'pump', 'temperature')
  AND  ts BETWEEN now() - INTERVAL '1 hour' AND now()
ORDER  BY ts;
```

Three things trip up hand-written queries:

* The timestamp column is **`ts`**, a `timestamptz`, not `timestamp` or `time`.
* A tag with no virtual path stores `virtual_path` as the **empty string**, never `NULL`. Passing `NULL` matches nothing and returns an empty result with no error.
* The `data_contract` argument is forgiving: `pump`, `_pump`, and `_pump_v1` all resolve to the same tag.

Location paths are canonicalized into an `ltree`: characters outside `[A-Za-z0-9_-]` become `_`. Hyphens survive, so `line-1` and `line_1` are **different** locations with different `topic_id`s.

### Latest value of every tag

```sql
SELECT DISTINCT ON (v.topic_id)
       l.path::text AS location, g.virtual_path, g.name AS tag, v.ts, v.value_num, v.value_text
FROM   umh.value_pump v
JOIN   umh.topic    t ON t.topic_id    = v.topic_id
JOIN   umh.tag      g ON g.tag_id      = t.tag_id
JOIN   umh.location l ON l.location_id = t.location_id
ORDER  BY v.topic_id, v.ts DESC;
```

This scans each topic's history to find its newest row, which is fine for hundreds of tags. For a dashboard that refreshes often, back the query with a continuous aggregate holding `last(value_num, ts)` per `topic_id` and read that instead.

## Using it from Grafana

Add the database as a PostgreSQL data source, paste the Grafana-flavored query into a panel, and the dashboard's time picker drives `$__timeFilter(ts)`. If you don't have Grafana yet, [Grafana](/production/deployment/docker-compose/additional-services/grafana) covers adding it to a running stack.

Point the data source at PgBouncer rather than TimescaleDB directly if your deployment has one.

## Precision

`value_num` is `DOUBLE PRECISION`, a binary floating-point type. It stores an approximation of the value, which is close enough for sensor readings but wrong for anything that has to come back byte-for-byte: integer counters above 2^53 lose their low digits, and a decimal such as `0.1` is kept as the nearest binary fraction. Route those tags to a text contract, where the value is stored verbatim in `value_text`.

The [Historian output reference](https://docs.umh.app/benthos-umh/output/historian) covers the rest of what the output plugin does: metrics, error classes, throughput tuning, and schema compatibility.


# Management Console

The Management Console at [management.umh.app](https://management.umh.app) is where you create UMH Core instances, configure what they do, and decide who in your company may change it. UMH Core itself runs on your own hardware. The console is the account layer around it.

## How it works together with UMH Core

```mermaid
flowchart LR
    subgraph MC["Management Console (cloud)"]
        ACC["Account, organization, users, licence"]
        CFG["Instance configuration"]
        STAT["Status, logs, metrics, topics"]
    end
    subgraph Site["Your site"]
        CORE["UMH Core (container)"] --> UNS["Unified Namespace<br/>(your data)"]
    end
    CFG -- pull --> CORE
    CORE -- push --> STAT
```

An instance registers itself once, using a token you generate in the console. After that, configuration travels down and status travels up:

* You edit a bridge in the console, the console queues the change, and the instance picks it up on its own schedule.
* The instance reports health, logs, metrics, and the topic data behind the [Topic Browser](/usage/unified-namespace/topic-browser).

The instance opens outbound HTTPS to `management.umh.app`, and by default nothing listens for connections from the internet, see [Deployment Model](/production/security/deployment-security#deployment-model-edge-only-architecture). It needs that outbound connection, so UMH Core does not run air-gapped.

## Signing in

1. Go to [management.umh.app](https://management.umh.app) and enter your email address.
2. Complete the sign-in method your company uses. Which one you get depends on your account and your plan, see [Authentication](/management-console/authentication).
3. Create an instance, or open one that a colleague already created.

![Login Screen](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-ffc48e326af5d07cd32dc2940a70cf6e33dd8164%2FScreenshot%202025-09-19%20at%2011.52.20.png?alt=media)

## What you manage here

| Area                                       | Where                                                              |
| ------------------------------------------ | ------------------------------------------------------------------ |
| Instances, bridges, data models, topics    | [Usage](/usage)                                                    |
| Sign-in methods                            | [Authentication](/management-console/authentication)               |
| Roles, locations, emergency access         | [Users and Permissions](/management-console/users-and-permissions) |
| Threat model, responsibilities, compliance | [Security](/management-console/security)                           |

Companies, membership, licences and editions are not documented yet. Ask your account executive in the meantime.

Everything you build with the console is documented under [Usage](/usage), because it works the same whether you configure it in the console or by editing the instance's config file.

## Account and settings

Open the account menu at the bottom of the sidebar, then **Settings**, for three tabs:

* **Account** shows your name, email, company, and licence state.
* **Settings** holds **Advanced Mode**, which reveals options for power users, and **Early Access Features**, which switches on features that are still being finished. Early access features can change or break without notice.
* **Permissions** (only available for Enterprise) shows your current role and what it allows, see [Users and Permissions](/management-console/users-and-permissions).


# Authentication

Access to the Management Console works in two layers:

* **Identity and company access.** Who you are, and whose instances you are allowed to reach. This page covers that layer.
* **Permissions.** What you may do once you are inside a company. See [Users and Permissions](/management-console/users-and-permissions).

## How you sign in

Sign-in runs through our IdP. The provider stores any passwords, verifies each method below, and adds the protections in [What the identity provider protects against](#what-the-identity-provider-protects-against). It also lets us tailor access per company: each company can be linked to its own organization with a default connection, and Enterprise customers can connect their own identity provider through [single sign-on](/management-console/authentication/enterprise-sso).

| Method                                                      | Who it applies to                                          |
| ----------------------------------------------------------- | ---------------------------------------------------------- |
| An email code                                               | The default for all Community users                        |
| Community username and password                             | Accounts that were set up with a password in the past      |
| Enterprise username and password                            | The default for Enterprise accounts without the SSO add-on |
| Single sign-on through your company's own identity provider | Enterprise, as an add-on that is paid for separately       |

The sign-in screen also offers **Continue with Google** and **Continue with LinkedIn**, which sign you in with the address held by that provider.

As an Enterprise Plan customer, your account executive will help you get with started with the SSO package. By default, your account's login method will be username and password. Our [Enterprise Single-Sign On](/management-console/authentication/enterprise-sso) page explains this in detail.

Depending on your account, the Management Console can ask you for two different secrets: a password and a passphrase. They do different jobs, and only the password can be reset. See [Passwords and Passphrases](/management-console/authentication/passwords-and-passphrases).

## Which company you sign in to

Companies are separated at sign-in, and one email address can belong to more than one company.

A company can be linked to its own organization with our identity provider. Where that link exists, you can only join the company through that organization, which is checked when your account is created and every time you accept an invitation. A company that is not linked yet, which happens for a while after an upgrade from Community, is not separated this way.

An organization accepts sign-ins through one or more **connections**. A connection is an identity source: your corporate SAML or OIDC provider, a social login, or an email code.

On an Enterprise plan you pick a **default connection** for your team. Everyone signs in through it, and invitations go through it too. Point it at your own identity provider and your usual single sign-on, multi-factor and offboarding rules decide who gets in. Pointing a default connection at your own identity provider is the single sign-on add-on, which is paid for on top of the Enterprise plan. Contact your account executive to enable it.

## What the identity provider protects against

Sign-in itself is handled by our identity provider, which provides:

* Rate limiting and account lockout after repeated failed attempts
* Detection of automated credential stuffing
* Risk-based multi-factor challenges. To require multi-factor from every user, configure it in your own identity provider and connect it with single sign-on
* Storage of passwords, for the accounts that use one

Enterprise customers who connect their own identity provider can configure further controls there, through their usual single sign-on, multi-factor and offboarding policies.


# Email Code

An email code lets you sign in without a password. You enter your email address. Our identity provider emails you a code. You type that code into the sign-in screen.

## Who this applies to

The email code is the default for Community accounts.

Enterprise accounts use a different method. They sign in through the [default connection](/management-console/authentication#which-company-you-sign-in-to) their company picked. Without the [single sign-on add-on](/management-console/authentication/enterprise-sso), that connection is username and password.

Some older Community accounts still use a username and password. When you enter such an email address, the console sends you to that older login instead. The email code becomes available once we have moved your account across. See [Username and Password](/management-console/authentication/username-and-password).

## Signing in

1. Open the Management Console and enter your email address.
2. Choose to receive a code by email. Our identity provider sends an email containing your verification code.
3. Enter the code in the sign-in screen that is still open in your browser.
4. Enter your passphrase. Signing in proves who you are; the passphrase unlocks your access to your instances, and the email code does not replace it. See [Passwords and Passphrases](/management-console/authentication/passwords-and-passphrases#why-every-sign-in-asks-for-the-passphrase).

The first time you sign in with an email address that has no account yet, the console asks you to complete your profile before it asks for a passphrase.

## Why we use email codes as our default

* **Only the latest code counts.** Requesting a new code invalidates every earlier one, and a code stops working the moment it has been used. A code copied from an old email is worthless.
* **Codes are short-lived.** A code is valid for a few minutes. If yours has expired, request a new one.
* **Guessing is cut off.** After three wrong entries the code is rejected and a fresh one has to be requested, so a code cannot be brute-forced.
* **The inbox is the credential.** The code goes to the address you entered and nowhere else, so access to the Management Console follows access to your mailbox. If you cannot read that inbox, use another sign-in method.
* **No password to protect.** No password is stored for this method, so there is none to reset and none to leak. If you receive a code you did not request, reply to that email so we can look into it.
* **Your passphrase stays yours.** The email code proves who you are; the passphrase separately unlocks your access to your instances and never leaves your device. See [Passwords and Passphrases](/management-console/authentication/passwords-and-passphrases).
* **One session per account.** Signing in on another device signs you out everywhere else, so a stale session cannot linger on a machine you left. See [Sessions](/management-console/authentication/sessions).


# Username and Password

You enter your email address and a password to sign in. Two kinds of account use this method, and they differ in where the password is stored and what else it is used for.

## Who this applies to

**Enterprise accounts** sign in with username and password by default. The password is stored by our identity provider, and the sign-in form is the [default connection](/management-console/authentication#which-company-you-sign-in-to) of your company's organization. Enterprise customers with the [single sign-on add-on](/management-console/authentication/enterprise-sso) sign in through their own identity provider instead.

**Community accounts created with a password** keep signing in with that password. The same password is also your [passphrase](/management-console/authentication/passwords-and-passphrases). New Community accounts use an [email code](/management-console/authentication/email-code).

## Signing in

1. Open the Management Console and enter your email address.
2. The console recognises which method your account uses. Enterprise accounts are sent to the sign-in form of their company's organization at our identity provider. Community accounts with a password see the password field directly.
3. Enter your password.
4. Enterprise accounts then enter their passphrase, which unlocks their access to the instances. For Community accounts with a password, the password already did this. See [Why every sign-in asks for the passphrase](/management-console/authentication/passwords-and-passphrases#why-every-sign-in-asks-for-the-passphrase).

Signing in on another device signs you out everywhere else. See [Sessions](/management-console/authentication/sessions).

## Resetting a password

Enterprise passwords are stored by our identity provider and can be reset there from the sign-in form.

For old Community accounts created with a password, the password is also the passphrase that unlocks your access to your instances, so it is handled like a passphrase: only you hold it. Keep it in your password manager. There is no way to reset your password, if you forgot it. See [Passwords and Passphrases](/management-console/authentication/passwords-and-passphrases#the-passphrase).


# Enterprise Single Sign-On

{% hint style="warning" %}
Enterprise Single Sign-On is a paid add-on and is not included in the Community & Enterprise plan. Get in touch with your account executive to learn more.
{% endhint %}

## Who this applies to

This functionality is available for any Enterprise plan customer. The Management Console features an identity provider (IdP) that allows enterprise customers to federate access through their own SSO provider.

## Connecting your identity provider

Enterprise SSO extends your existing identity and access management strategy to the Management Console, so the same governance, provisioning, and offboarding controls your organization already trusts apply here as well. Centralizing authentication in your identity provider reduces the operational overhead of managing separate credentials, strengthens your security posture, and helps satisfy audit and compliance requirements around access control.

Onboarding is a managed, white-glove process. Our team partners with your stakeholders and IT organization to plan the integration, align it with your corporate security policies, and validate it end to end before go-live, minimizing disruption to your users and ensuring a smooth rollout across your enterprise.

## Supported Methods

| Method                                  | Type   |
| --------------------------------------- | ------ |
| SAML 2.0                                | Custom |
| OpenID Connect (OIDC)                   | Custom |
| Okta                                    | Native |
| Google Workspace                        | Native |
| Microsoft Azure AD (Microsoft Entra ID) | Native |
| ADFS                                    | Native |
| LDAP / Active Directory                 | Native |
| Ping Federate                           | Native |

## Multi-factor authentication

When you federate access through your own identity provider, multi-factor authentication (MFA) remains fully under your control, governed by your corporate security policies. We do not layer additional MFA on top of your custom IdP integration, so your organization retains a single, authoritative point of enforcement and a consistent audit trail across every system your identity provider protects.

MFA should not be confused with passphrases, which serve a distinct purpose: unlocking your access to your instances. For how sign-in and the passphrase work together, see [Why every sign-in asks for the passphrase](/management-console/authentication/passwords-and-passphrases#why-every-sign-in-asks-for-the-passphrase).


# Passwords and Passphrases

The Management Console asks you for up to two secrets: a password and a passphrase. Every account has a passphrase. Whether you also have a password depends on how you sign in. The two sound alike but do different jobs, and only the password can be reset.

| Secret         | What it is                                                   | Can it be reset?                                                                               |
| -------------- | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- |
| **Password**   | Proves who you are at sign-in.                               | Yes, except for Community accounts created with a password. See [The password](#the-password). |
| **Passphrase** | Unlocks your access to your instances. Only you can do that. | No                                                                                             |

## The password

The password answers "are you allowed to sign in". It works like any other login password. Our identity provider stores it. If you forget it, you can reset it from the sign-in form. See [Resetting a password](/management-console/authentication/username-and-password#resetting-a-password).

Google, LinkedIn, single sign-on and email codes replace the password with another way to prove who you are. With those methods, you have no password to manage.

Community accounts created with a password use that password as their passphrase too. That password cannot be reset. See [Resetting a password](/management-console/authentication/username-and-password#resetting-a-password).

## The passphrase

The passphrase unlocks your access to your instances. Behind that access is a permission grant, which the Management Console issues to your account. It is stored locked on our servers, and your passphrase is the only thing that unlocks it.

You create the passphrase. It never leaves your device. We never see it, so we cannot unlock your access for you, and no one can reset it.

Keep the passphrase somewhere you can find it again, such as your password manager. If you lose it, no one can unlock your access again, including us.

## Why every sign-in asks for the passphrase

Every sign-in method asks for the passphrase, whichever way you proved your identity. Proving who you are and unlocking your access to your instances are two separate steps. Signing in covers the first. The passphrase covers the second.

An email code involves no password at all. It still prompts for the passphrase to unlock your access to your instances.


# Sessions

Regardless of which [authentication method](/management-console/authentication) your account uses, session timeouts apply the same way.

## Session Durations & Renewal

Your session is managed by the Management Console itself, independently of the identity provider.

| Session rule   | Value                                                                                        |
| -------------- | -------------------------------------------------------------------------------------------- |
| Session length | 14 days                                                                                      |
| Renewal        | While you are active and within 7 days of expiry, the session is extended by another 14 days |
| Hard limit     | 30 days after you first signed in, you sign in again regardless of activity                  |
| Ends on        | Sign-out, the 30 day limit, or removal from the company                                      |

## One session per account

Each account has one active session at a time. When you sign in from another device, the existing session is invalidated and you are signed out of the Management Console everywhere else.

This is intentional. Most work in the Management Console is a one-off change, often to a single umh-core instance. Allowing several sessions at once would let the same account push concurrent, conflicting changes to your umh-core instances.

{% hint style="warning" %}
There is no automatic sign-out after a period of inactivity. A session stays valid until the 14 day token expires or the 30 day limit is reached, whichever comes first. Sign out when you leave your workstation.
{% endhint %}


# Users and Permissions

{% hint style="warning" %}
Management Console continues to evolve on a regular basis. The same applies to our permission system. Depending on when you created your account, you'll see different options.
{% endhint %}

What you may do inside a company depends on the role you hold and where in the company you hold it. Users and instances both have roles.

## How to determine your permission setup

{% hint style="warning" %}
User and Permission Management is part of the **Enterprise Plan**. If you're using the Community Plan, your account can only have one user, which is automatically the Account Owner with full CRUD permissions.
{% endhint %}

Permission models differ between accounts, so the roles you have are not necessarily the roles a colleague at another company has. Open the account menu, click **Settings**, select the **Permissions** tab, and compare what you see:

* **One role for the whole company.** **Current Role** names a single role (`Account Owner`), and a matrix below it ticks off the permission groups that role covers.
* **Roles per location.** Your roles are listed against locations, so you can hold one role at one part of the company and a different role at another.

Role capabilities for both setups are in the [Roles Reference](/management-console/users-and-permissions/roles-reference). Locations, roles and how permissions inherit are covered in [Location-Based Permissions](/management-console/users-and-permissions/location-based-permissions), and inviting, changing and removing people in [Managing Access](/management-console/users-and-permissions/managing-access).

Not every company is on the same setup, and moving between them is something we arrange rather than something you switch on. Ask your account executive which setup you are on and what moving would involve.

### Where these roles apply

Your role governs what you can do in the Management Console, where permissions are managed centrally for your account and for the instances your company connects.

`umh-core` connects to the Management Console through its `AUTH_TOKEN`, and can also run locally. For how instances connect and authenticate, see [Access Control and Authentication](/production/security/deployment-security#access-control-and-authentication).


# Location-Based Permissions

## Locations

A location is a position in your company's tree. Level 0, the enterprise, is the only level you have to use. Everything below it is yours to choose, whether you follow ISA-95, KKS, or your own naming.

Location paths use the same dot-separated format as [topic paths](/usage/unified-namespace/topic-convention):

* `ACME` (enterprise)
* `ACME.Munich` (enterprise, site)
* `ACME.Munich.Assembly` (enterprise, site, area)
* `ACME.Munich.Assembly.Line1` (enterprise, site, area, line)
* `ACME.Munich.Assembly.Line1.Cell5` (enterprise, site, area, line, work cell)

Add more levels if your organization needs them.

## Roles

The full capability list is in the [Roles Reference](/management-console/users-and-permissions/roles-reference).

A user can hold different roles at different locations, for example Admin at `ACME.Munich.Assembly.Line1` and Viewer at `ACME.Munich.Assembly.Line2`.

### Permission inheritance

Permissions inherit downward. Access at `ACME.Munich` also grants access to everything under Munich, including `ACME.Munich.Assembly.Line1.Cell5`.

You can override an inherited permission for a specific location. A user can be a Viewer at `ACME.Munich.Assembly` and an Admin at `ACME.Munich.Assembly.Line1.Cell5` only.


# Managing Access

## Managing users

Any Admin and Account Owner is able to manage users. However, an admin can grant permissions only for locations where they are an admin themselves, which stops anyone from inviting their way to more access than they have.

### Inviting a user

1. The admin enters the email address, the role, and the locations it applies to.
2. The console produces an invite link and, separately, an invite key that only the admin sees.
3. Our identity provider emails the invitation to the address.
4. The admin sends the invite key to the person through another channel.
5. The person opens the link, signs in, and enters the invite key.

An invite key works once.

The invite key that is displayed to you in the Management Console is generated in your web browser and only accessible to you. We do not generate or store invite keys on our servers. This is by design, so that we can never gain access to your instances or Management Console account.

Both halves are needed on purpose. The link proves the person controls the email address. The key, sent another way, proves the admin meant to invite this particular person, so a forwarded invitation email is not enough to join your company.

## Removing access

| What you do                 | What happens                                                                                                                            |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Remove a user               | Their session ends immediately and they can no longer sign in                                                                           |
| Change a user's permissions | Applied while they are signed in, within a few seconds. A user who is offline keeps their previous permissions until they sign in again |
| Remove an instance          | The instance can no longer communicate with the console                                                                                 |

To take access away straight away, remove the user rather than lowering their role, then invite them again with the permissions you want. What they built stays: their bridges, data models and the users they invited belong to the company, not to them.

Sessions cannot be ended individually yet, so removing the user is the only way to end a session you do not control.


# Roles Reference

Roles are not the same in every account. Work out which setup you have in [Users and Permissions](/management-console/users-and-permissions) first, then read the matching section below. There is deliberately no combined table.

## Roles per location

Three roles apply at the locations a user or instance is assigned to.

| Role          | What it allows                                                                                                                                                                         |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Account Owner | Everything Admin allows, and is able to create, update and delete instances.                                                                                                           |
| Admin         | Everything Editor allows, plus inviting users at the locations where they are an admin. See [Managing users](/management-console/users-and-permissions/managing-access#managing-users) |
| Editor        | Create and modify resources, such as bridges and data models. Cannot manage users or instances.                                                                                        |
| Viewer        | Read-only. See resources and their state, change nothing                                                                                                                               |

In addition to setting roles, user access is set on a per-location basis, at the granting person's desired location granularity. Exceptions set at a lower location override what is inherited, see [Permission inheritance](/management-console/users-and-permissions/location-based-permissions#permission-inheritance).

These capabilities apply in the Management Console. They do not apply on an `umh-core` instance, which does not check the role behind a request, see [What roles do not cover](/management-console/users-and-permissions#what-roles-do-not-cover). The Management Console ensures that a user is allowed to talk to an instance instead.

## One role for the whole company

Here your role is not tied to a location. **Current Role** on the **Permissions** tab names it, and the permission matrix underneath shows which groups of actions it covers:

| Permission group | Covers                                                |
| ---------------- | ----------------------------------------------------- |
| Get actions      | Read-only operations for viewing resources and states |
| Edit actions     | Operations that modify existing resources             |
| Delete actions   | Operations that remove resources                      |
| Deploy actions   | Operations for deploying new resources                |
| Test actions     | Operations for testing configurations                 |
| System actions   | System-level operations                               |
| Company actions  | Company-level operations                              |

Expand a group in the matrix to see the individual actions in it. The matrix is generated from the role you hold, so it is the list for your own account rather than a general one. Like the roles above, it covers the console only, see [What roles do not cover](/management-console/users-and-permissions#what-roles-do-not-cover).


# Emergency Access

Admins lose access. People leave, addresses get disabled, permissions get set wrong. Fallback access is what you arrange in advance so that a company is never locked out of its own instances.

## The break-glass account

### What is a break-glass account?

A break-glass account is an account with full permissions that nobody uses for daily work. It exists for one purpose: when every regular admin is locked out, someone signs in with it, repairs the permissions or invites new admins, and signs out again. The name comes from the glass cover on a fire alarm: you break it only in an emergency. Because the account is used rarely, it is kept out of the normal flow of people joining and leaving, its credentials are stored in one controlled place, and its use is a deliberate, visible act rather than routine.

If you already run break-glass accounts in other systems, the Management Console works the same way and you can skip to the practices below.

### The Account Owner is your break-glass account

In the Management Console, the **Account Owner** is the account that created the company. It has Admin access everywhere in the company, and it cannot be transferred or demoted. That is deliberate: it guarantees one account always has full control, whatever happens to the others. Not every company has one. Ask your account executive if you are not sure whether yours does.

The Account Owner cannot be recovered either, so decide before you register which account it will be, and treat it as a break-glass account from day one:

* Use an address your organization owns and can keep, such as `ot-team@example.com`. Not the personal account of an employee who may leave.
* Keep its credentials in your team's password manager rather than passing them around.
* Switch on multi-factor authentication for it.
* Do not work with it day to day. Invite ordinary admin accounts from it and use those.

## Reviewing it

Check your fallback access when you review permissions. Two things drift: the address on the Account Owner stops being monitored. Decide each time whether the access should stay.


# Security

Access to the Management Console is decided in two layers: identity and company access first, then permissions within that company. [Authentication](/management-console/authentication) and [Users and Permissions](/management-console/users-and-permissions) cover how you set both up. This section covers the design behind them, for readers who have to sign off on it.

| Page                                                                        | Answers                                                                                 |
| --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| [Threat Model](/management-console/security/threat-model)                   | Who this is designed to keep out, what it accepts as a risk, and what it does not cover |
| [Shared Responsibility](/management-console/security/shared-responsibility) | Which parts of security are ours and which are yours                                    |
| [Compliance](/management-console/security/compliance)                       | How the controls map to OWASP, NIST and IEC 62443, including where they fall short      |

Security on the instance itself is separate, because the instance runs on your hardware and inside your network. See [umh-core Security](/production/security/deployment-security).


# Threat Model

The Management Console defends against attackers who try to guess credentials, steal a session, reach another company's data, or grant themselves more permissions. Some risks are accepted, and some threats are outside what the Management Console can prevent.

Some measures are ours to run and some are yours. Each table below names which side a measure is on. [Shared Responsibility](/management-console/security/shared-responsibility) has the full split. [Compliance](/management-console/security/compliance) maps the measures to the standards they follow.

* [How each connection is secured](#how-each-connection-is-secured) covers encryption and authentication between the browser, the identity provider and your instances
* [Threat actors](#who-could-attack-the-management-console) lists who might attack and what they could attempt
* [Attack scenarios](#how-to-stop-or-prevent-common-attack-patterns) lists attack patterns and what stops them
* [Threats outside the Management Console](#threats-outside-the-management-console) lists what we cannot prevent

## How each connection is secured

The Management Console has three network connections: to your browser, to our identity provider, and to your instances. The table below shows how our network connections are secured.

| Connection                                       | How it is secured                                                                                                                                                                                                               |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Your browser and the Management Console          | Traffic is encrypted with TLS. After sign-in, the browser holds the session in a cookie that JavaScript cannot read                                                                                                             |
| The Management Console and our identity provider | Sign-in uses OAuth 2.0. The identity provider stores your credentials and runs multi-factor challenges                                                                                                                          |
| The Management Console and your instances        | The instance proves its identity with the `AUTH_TOKEN` environment variable set when its container starts, see [Access Control and Authentication](/production/security/deployment-security#access-control-and-authentication). |

## Who could attack the Management Console

Attackers can be various actors, from outside your company or from inside it. The table below shows what each of them could attempt, how far the Management Console protects you, and what you have to do yourself.

| Attacker                               | What they can attempt                                                                                      | Where you stand                                                                                                                                                                                                                                                                                                                            | What you do                                                                                                                                                                                                                                                   |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Outside attacker                       | Credential stuffing, phishing, brute force                                                                 | **Protected.** Sign-in runs through our identity provider, which rate-limits attempts and checks them for anomalies                                                                                                                                                                                                                        | Use a unique, strong password. With the [single sign-on](/management-console/authentication/enterprise-sso) add-on, your own identity provider can require multi-factor and password rotation for every user                                                  |
| A user whose account is taken over     | Everything that user could do in the Management Console, including on the instances their company connects | **Partly protected.** The backend checks the user's permissions on every request, and removing the user ends their session at once. The instance itself does not check per-user permissions and trusts the Management Console's decision, see [Where these roles apply](/management-console/users-and-permissions#where-these-roles-apply) | Remove the user as soon as you suspect a takeover, see [Removing access](/management-console/users-and-permissions/managing-access#removing-access). Invite them again once the account is secured. Contact your account executive if you need an audit trail |
| A user acting in bad faith from inside | Granting themselves more permissions, intercepting an invitation                                           | **Partly protected.** An admin can only grant permissions where they are already an admin. An invitation needs an invite key that the admin sends separately, see [Inviting a user](/management-console/users-and-permissions/managing-access#inviting-a-user)                                                                             | Send invite keys through a channel other than the invitation email. Review roles and location access periodically, and remove people who leave your company                                                                                                   |
| The Account Owner                      | Taking over the company                                                                                    | **Accepted risk.** The Account Owner cannot be demoted, so that one account keeps full control when every other admin is locked out. Not every company has one                                                                                                                                                                             | Register the Account Owner on an address your company owns and monitors, not on an employee's personal account. Use it only when no other admin can act, see [Emergency Access](/management-console/users-and-permissions/fallback-access)                    |

## How to stop or prevent common attack patterns

Attackers can try different ways to get into an account or a company. The table below lists common attack patterns and what stops or prevents them.

| Attack                             | What stops it                                                                                                                                                                                                                                                                                             |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Guessing or replaying credentials  | Our identity provider rate-limits sign-in attempts and checks them for anomalies. Enterprise customers can require multi-factor for every user through their own identity provider, see [single sign-on](/management-console/authentication/enterprise-sso)                                               |
| Stealing a session                 | JavaScript cannot read the session cookie, so a cross-site scripting bug cannot pass the session to an attacker. All traffic is encrypted with TLS. Optionally, see [One session per account](/management-console/authentication/sessions#one-session-per-account) for context on how we handle sessions. |
| Reaching another company's data    | A sign-in belongs to one company at a time. A company linked to its own organization with our identity provider can only be joined through that organization, see [Which company you sign in to](/management-console/authentication#which-company-you-sign-in-to)                                         |
| Granting yourself more permissions | Editing a request in the browser grants nothing, because the backend checks every grant against what the granting admin holds at that location                                                                                                                                                            |
| Phishing an invitation             | An invitation is issued for one email address, and joining also needs the invite key that the admin sends through another channel. A forwarded invitation email alone is not enough                                                                                                                       |

## Threats outside the Management Console

The Management Console cannot prevent the following threats, because they affect hardware or infrastructure that we do not control.

### Compromised devices and instances

The Management Console cannot tell a compromised device from its real user. Whoever controls the device holds that user's access to your company.

The same applies to instances. An instance runs on hardware outside our control, so we cannot tell whether it has been compromised. See [Threat Model (Simplified)](/production/security/deployment-security#threat-model-simplified).

### Identity provider infrastructure

By default, you sign in through our identity provider using OAuth 2.0. We do not operate the identity provider's infrastructure, so a security incident there is outside what we can prevent.

Such an incident does not by itself grant access to your instances. Signing in also asks for your passphrase, where your account has one. The passphrase unlocks your access to your instances, see [Why every sign-in asks for the passphrase](/management-console/authentication/passwords-and-passphrases#why-every-sign-in-asks-for-the-passphrase).

With the [single sign-on add-on](/management-console/authentication/enterprise-sso), your users sign in through your own identity provider instead. Sign-in security then depends on your setup rather than ours, so monitor your identity provider for suspicious sign-ins. See [Shared Responsibility](/management-console/security/shared-responsibility) for the split.


# Shared Responsibility

Securing the account layer is split between UMH and you. The instance has its own split, see [Shared Responsibility Model](/production/security/deployment-security#shared-responsibility-model).

## Ours

| Area               | What we do                                                                                                                                                   |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Authentication     | Run the identity-provider integration, issue sessions, store instance credentials so that we cannot read them                                                |
| Authorization      | Provide the role system, permission inheritance and location-based access                                                                                    |
| Secure defaults    | The password policy we configure with our identity provider, invitations that need two separate pieces, credential storage the console itself cannot decrypt |
| Platform           | Security of the Management Console application and its API                                                                                                   |
| Audit logging      | Record sign-ins and permission changes                                                                                                                       |
| UMH staff accounts | Identity, authentication and offboarding of the `@umh.app` accounts you invite. UMH staff reach your company only through an invitation you issue            |

{% hint style="info" %}
Audit logging is recorded but not yet available for self-service. Logs with retention settings you control, and forwarding into a SIEM, are planned and not built. Get in touch with your Account Executive if you need an audit trail.
{% endhint %}

## Yours

| Area                   | What you do                                                                                                                   |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| The Account Owner      | Protect the account that cannot be demoted, see [Emergency Access](/management-console/users-and-permissions/fallback-access) |
| Instance tokens        | Store and transmit each instance's `AUTH_TOKEN` safely                                                                        |
| Invitations            | Send invite keys through a channel other than the invitation email                                                            |
| Joiners and leavers    | Remove people when they leave your organization                                                                               |
| Access reviews         | Check periodically who holds what, and where                                                                                  |
| Your identity provider | Configure and secure the single sign-on you connect                                                                           |
| UMH staff              | Decide which UMH people to invite, and remove them when the work is done                                                      |

## Both

| Area              | How it splits                                                                           |
| ----------------- | --------------------------------------------------------------------------------------- |
| Permission design | We provide the role system, you decide who holds which role at which location           |
| Incident response | We watch the platform, you watch for misuse of your own credentials                     |
| Compliance        | We provide the controls, you decide whether your use of them satisfies your obligations |


# Compliance

How the Management Console's access controls map to the standards a security review will ask about, which choices we made on purpose, and what is on the roadmap.

UMH is a product supplier, so the Management Console is assessed as a component under IEC 62443-4-2 (software application). IEC 62443-3-3 applies to the system you build with it, and is yours or your integrator's to assess. Component Requirements (CR) in 4-2 mirror the System Requirements (SR) in 3-3 by number, so CR 1.1 corresponds to SR 1.1.

## Authentication

Relevant standards: NIST SP 800-63B, IEC 62443-4-2 CR 1.1, OWASP Authentication Cheat Sheet.

| Standard                        | Requirement                                                     | How it is met                                                                                                                                                                                                                                        |
| ------------------------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| OWASP Authentication            | Multi-factor, brute force protection, secure session management | Our identity provider provides brute force protection, and multi-factor as described below. Session management is ours, see [Sessions](#sessions)                                                                                                    |
| NIST SP 800-63B AAL2            | Two authentication factors at every sign-in                     | Met for Enterprise companies that connect their own identity provider with multi-factor enforced, or that ask UMH to enforce multi-factor for them. Community accounts use one factor by design, see [Decisions and roadmap](#decisions-and-roadmap) |
| NIST SP 800-63B AAL2            | Reauthentication at most every 24 hours                         | We chose longer sessions, see [Decisions and roadmap](#decisions-and-roadmap)                                                                                                                                                                        |
| IEC 62443-4-2 CR 1.1            | Identify and authenticate human users on all interfaces         | Our identity provider issues a unique identity per user, and every request to the console carries it                                                                                                                                                 |
| IEC 62443-4-2 CR 1.1 RE 1 (SL2) | Unique identification and authentication                        | Met. No shared accounts; each user is a separate identity                                                                                                                                                                                            |
| IEC 62443-4-2 CR 1.1 RE 2 (SL3) | Multi-factor authentication on all interfaces                   | An Enterprise capability. Enterprise companies enforce it through their own identity provider or ask UMH to enforce it. Community accounts use a one-time email code with risk-based challenges from our identity provider                           |

## Authorization

Relevant standards: NIST SP 800-53 AC-3 and AC-6, IEC 62443-4-2 CR 2.1, OWASP Authorization Cheat Sheet.

| Standard                        | Requirement                                                             | How it is met                                                                                                                                                                |
| ------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| OWASP Authorization             | Least privilege, deny by default, validate permissions on every request | Access is granted per location and inherits downward from there. Nothing is granted outside a location you were given. Every request is checked server-side                  |
| NIST SP 800-53 AC-3             | Enforce approved authorizations for logical access                      | The console enforces permissions server-side on every request. Permission data is cached for up to 10 minutes by design, see [Decisions and roadmap](#decisions-and-roadmap) |
| NIST SP 800-53 AC-6             | Least privilege                                                         | Viewer, Editor and Admin, granted at the narrowest location that works                                                                                                       |
| IEC 62443-4-2 CR 2.1 RE 1 (SL2) | Authorization enforcement for all users                                 | Met. The console enforces per user on every request                                                                                                                          |

## Sessions

Relevant standards: OWASP Session Management Cheat Sheet, NIST SP 800-63B reauthentication.

| Standard             | Requirement                                                      | How it is met                                                                                                                                                                                                  |   |
| -------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - |
| OWASP Session        | Secure cookies, server-side sign-out, absolute and idle timeouts | Sessions are held in cookies JavaScript cannot read, sent only over HTTPS, and invalidated server-side on sign-out. Session length is a deliberate choice, see [Decisions and roadmap](#decisions-and-roadmap) |   |
| NIST SP 800-63B AAL2 | Idle timeout of 1 hour or less                                   | We chose no idle timeout, see [Decisions and roadmap](#decisions-and-roadmap)                                                                                                                                  |   |

Our identity provider authenticates. Everything after that, including the session rules above, is managed by the Management Console itself. See [Sessions](/management-console/authentication/sessions).

## Audit

Relevant standards: NIST SP 800-53 AU-2 and AU-12, IEC 62443-4-2 CR 2.8, OWASP Authorization Cheat Sheet (logging).

| Standard             | Requirement                                                | How it is met                                                                                                                                                                                 |
| -------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| IEC 62443-4-2 CR 2.8 | Auditable events for access control and account management | The console records sign-in, sign-up, session invalidation, invitations created, deleted and redeemed, users removed, instances registered, updated and deleted, and permission grant changes |
| NIST SP 800-53 AU-12 | Audit record generation, readable by authorized users      | Company owners and Admins read the log under User Management and export it as CSV. Records are kept; nothing deletes them                                                                     |

## Decisions and roadmap

Some requirements in the tables above are met differently from the letter of the standard. Each is either a choice we made on purpose, or work that is planned. Both are listed here so a reviewer sees them in one place.

### By design

| Standard                            | Requirement                                                      | What we do, and why                                                                                                                                                                                                                                                                                                                |
| ----------------------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| NIST SP 800-63B AAL2                | Two factors at every sign-in                                     | Community accounts sign in with a one-time email code, and our identity provider adds risk-based challenges when a sign-in looks unusual. Enforced multi-factor is part of the Enterprise plan, where it follows your own identity provider's policy or is switched on by UMH for you                                              |
| NIST SP 800-63B AAL2, OWASP Session | Reauthentication within 24 hours, idle timeout of 1 hour or less | A session lasts 14 days from its last renewal and 30 days at most. In return, each account holds exactly one session: signing in elsewhere ends the previous one. Sign out when you leave your workstation. See [Sessions](/management-console/authentication/sessions)                                                            |
| NIST SP 800-53 AC-2(3)              | Disable inactive accounts automatically                          | Last sign-in is recorded and visible to you. Whether an account is dormant is your call, not a timer's: removing people who leave is your side of the [Shared Responsibility](/management-console/security/shared-responsibility) split                                                                                            |
| NIST SP 800-53 AC-3                 | Authorization changes take effect at once                        | The browser checks for role changes every 10 seconds. A change applies at the next check, or at the next sign-in if the user is signed out. Instances follow within one minute. A removed user is signed out at their next CRUD action. This approach ensures the secret that defines the user's access never leaves their device. |
| IEC 62443-4-2 CR 1.1 RE 2 (SL3)     | Multi-factor on all interfaces                                   | Available to Enterprise companies, see [Authentication](#authentication). Our target is SL2, see [Target security level](#target-security-level)                                                                                                                                                                                   |

### On the roadmap

| Standard | Requirement                                 | Where it stands                                                                                                                                                                              |
| -------- | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Audit    | Retention you control, forwarding to a SIEM | Records are kept indefinitely today and exported as CSV. Retention settings and SIEM forwarding are planned, see [Shared Responsibility](/management-console/security/shared-responsibility) |

## Target security level

The Management Console targets **IEC 62443-4-2 Security Level 2** as a software application component. SL2 covers intentional attacks using simple means, by attackers with generic skills, low motivation and few resources, which is the profile that fits standard manufacturing operations. The console meets CR 1.1 RE 1 and CR 2.1 RE 1, the SL2 enhancements in the two areas above.

For critical infrastructure that has to meet SL3 or above, talk to UMH about what is possible.

## References

* [OWASP Authentication Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html)
* [OWASP Authorization Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html)
* [OWASP Session Management Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Session_Management_Cheat_Sheet.html)
* [NIST SP 800-63B Digital Identity Guidelines, Revision 4](https://pages.nist.gov/800-63-4/sp800-63b.html)
* [NIST SP 800-53 Revision 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)
* [ISA/IEC 62443 Series of Standards](https://www.isa.org/standards-and-publications/isa-standards/isa-iec-62443-series-of-standards)
* [IEC 62443-4-2:2019](https://webstore.iec.ch/en/publication/34421)


# Billing and Plans

The Management Console offers two plans, the free Community Plan and a paid Enterprise Plan. Every company starts on the Community Plan by default.

Enterprise customers are upgraded to the Enterprise Edition by the UMH Team. Enterprise licenses are time-limited. The expiry date matches the subscription term in your agreement with the UMH Team. The Enterprise plan is applied on a per-company basis, allowing you to invite users without extra cost.

You can see which plan your company is in three places:

* The sidebar shows Enterprise Edition or Community Edition under the Management Console name.
* Settings → Account has a License field. The field says `Inactive` if you have the Community Plan. As an enterprise customer, that field will show you the expiry date of your Enterprise license.
* Enterprise Only: The License Status card on the home page shows Active or Inactive. For an active license, the card also shows Valid until with the expiry date, which turns red when fewer than 90 days remain.

## What the plans include

On the free Community Plan, your company can register exactly one instance.

The Enterprise Plan lets you register an unlimited amount of instances and includes user management, the Enterprise release channel, and version control features.

You can find every difference between the two plans in the [plan comparison](/management-console/billing-and-plans/plan-comparison).

## How to get an Enterprise license

As part of our white-glove onboarding process, UMH attaches the license to your company for you, and from then on every user in your company is on the Enterprise Plan.

If you are interested in an Enterprise license, [you can get in touch with us through this form](https://www.umh.app/talk-to-team).

### What to do when the Enterprise license expires

After the expiry date, your company automatically gets the Community Plan.

Once your company is back on the Community Plan:

* Your existing instances keep running and stay connected, and nothing is deleted. However, you can't register more instances.
* Your existing users keep their accounts and roles. However, you can't invite new people to your company.
* Every other Enterprise control listed in the [plan comparison](/management-console/billing-and-plans/plan-comparison) is disabled.
* Your data stays yours, regardless of plan. A plan change does not touch the data on your instances.

Our team will get in touch with you prior to your license expiring. If you'd like to proactively discuss your renewal period, reach out to your Account Executive.


# Plan Comparison

The table below lists the available features of our Community and Enterprise plans. Plans are applied to every user and every instance of a company.

To learn more about how plans work at UMH, see [Billing and Plans](/management-console/billing-and-plans). If you are interested in our Enterprise plan, see [How to get an Enterprise license](/management-console/billing-and-plans#how-to-get-an-enterprise-license).

| **Feature**                                            | Community Plan | Enterprise Plan                                                                             |
| ------------------------------------------------------ | -------------- | ------------------------------------------------------------------------------------------- |
| Host your own Unified Namespace                        | ✓              | ✓                                                                                           |
| Management Console Access                              | ✓              | ✓                                                                                           |
| 70+ bridge templates                                   | ✓              | ✓                                                                                           |
| Versioning for Data Models                             | ✓              | ✓                                                                                           |
| Users per company                                      | 1              | ∞                                                                                           |
| Instances per company                                  | 1              | ∞                                                                                           |
| Enterprise Release Channel                             | —              | ✓                                                                                           |
| Roles and location permissions                         | —              | [Admin, Editor and Viewer roles](/management-console/users-and-permissions/roles-reference) |
| Review changes before a deploy with a visual diff view | —              | ✓                                                                                           |
| Instance audit logs                                    | —              | Paid add-on                                                                                 |
| Single sign-on with your own identity provider         | —              | [Paid add-on](/management-console/authentication/enterprise-sso)                            |


# Production

This section provides essential guidance for deploying and maintaining UMH Core in production environments. Topics include system sizing recommendations, key metrics to monitor for optimal performance, update procedures for seamless maintenance, and configuration considerations for corporate firewall environments. Use these resources to ensure reliable, scalable deployments of your industrial data infrastructure.


# Architecture Patterns

Architecture patterns describe how to deploy UMH Core for different operational requirements. Each page explains a topology: what components are involved, how they connect, what the pattern guarantees, and when to use it.

These are not step-by-step guides. They describe architectures so you can choose the right one for your site. For deployment procedures, see [Deployment](/production/deployment). For infrastructure-level recovery (container restarts, storage, node rescheduling), see [High Availability](/production/high-availability).

## Patterns

* [Redundant Data Collection](/production/architecture-patterns/redundant-data-collection) -- two umh-core instances reading from redundant PLCs for zero data loss


# Redundant Data Collection

{% hint style="info" %}
This pattern applies to sites with **redundant PLCs** -- two physical controllers publishing the same data. Redundant PLCs are common in process manufacturing (oil and gas, chemicals, power generation, pharmaceuticals) where the process cannot be safely stopped. If you run standard, non-redundant PLCs -- as most discrete manufacturing sites do -- this page does not apply to you. A single umh-core instance already provides sub-second process recovery via S6, and your container manager handles container-level restarts. See [High Availability](/production/high-availability).
{% endhint %}

## The problem

When a site has redundant PLCs, each controller independently reads the same sensors. If the single umh-core instance collecting data goes down, data is lost until it recovers -- even though a second PLC could have provided the same readings.

## The solution

Run two umh-core instances, each connected to one of the redundant PLCs. Each instance collects into its own [Unified Namespace](/usage/unified-namespace) under a PLC-specific topic:

```mermaid
flowchart LR
    A["umh.v1.enterprise.line.PLC-A._historian<br/>(umh-core 1, from its own bridge)"]
    B["umh.v1.enterprise.line.PLC-B._historian<br/>(umh-core 2, from its own bridge)"]
```

umh-core 2 pushes its data into umh-core 1's UNS. umh-core 1 now has both PLC-specific topics. A [stream processor](/usage/data-flows/stream-processor) then does redundancy resolution: it selects the primary source from the two streams and writes a single consolidated topic:

```mermaid
flowchart LR
    A["umh.v1.enterprise.line.PLC-A._historian"] --> SP["Stream processor<br/>(redundancy resolution)"]
    B["umh.v1.enterprise.line.PLC-B._historian"] --> SP
    SP --> C["umh.v1.enterprise.line._historian<br/>(consolidated, PLC-agnostic)"]
```

The higher-level system (a data lake, an MQTT broker, or another factory-wide umh-core instance) consumes this consolidated topic. It does not need to track which PLC provided each reading, but the original source is always traceable through the PLC-specific topics and message metadata.

![Redundant data collection topology: two umh-core instances each collecting from one PLC, umh-core 2 pushes data to umh-core 1 which does redundancy resolution and forwards to the higher-level system](https://3966587046-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FPQ7UY9eT1SiICDqYFZdA%2Fuploads%2Fgit-blob-c105f9022fbf653077d804eb4255c0b1c8e849cd%2Fdual-umh-core-topology.png?alt=media)

All original data from both collectors is retained in the PLC-specific source topics. Redundancy resolution selects the primary source for downstream processing. It does not delete or modify the original records.

If one instance fails, the other continues collecting into its own UNS. When the failed instance recovers, the bridge resumes pushing data and redundancy resolution continues.

## Prerequisites

* **Two redundant PLCs.** Two physical or logical controllers that publish the same data. A single PLC with two IP addresses does not qualify -- you need hardware designed for redundancy. The redundant PLCs must produce equivalent data: the same sensors, the same values, and ideally synchronized timestamps.
* **High-availability deployment.** Both umh-core instances must be deployed so that a single infrastructure failure does not take both down at the same time. See [High Availability](/production/high-availability) for storage and deployment requirements.
* **A redundancy resolution strategy.** The stream processor needs a rule for selecting the primary source. The strategy depends on the protocol and what timestamps the PLCs provide. OPC UA subscriptions provide a device-level SourceTimestamp. If both PLCs are synchronized to a common time source (for example PTP/IEEE 1588), timestamp-based resolution works well. Clock synchronization between the redundant PLCs is a site prerequisite, not something OPC UA or umh-core provides. Polling-based protocols (Modbus, S7) do not provide device-level timestamps and require a different approach, for example last-value-wins within a time window.

## Recovery behavior

Two terms used in the table below:

* **MTTR** (Mean Time To Recovery): how long until data collection resumes after a failure.
* **RPO** (Recovery Point Objective): how much data is lost during the failure. RPO = 0 means no data is lost.

| Failure                        | MTTR               | RPO | What happens                                                                                                                                                                                                          |
| ------------------------------ | ------------------ | --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Process crash inside container | Seconds to minutes | 0   | S6 restarts the process in sub-second, but data collection resumes only after the bridge reconnects to the PLC (seconds to minutes depending on protocol and number of nodes). The other instance was never affected. |
| Container or pod crash         | 30-60 seconds      | 0   | Container manager restarts the container. The other instance continues.                                                                                                                                               |
| Full instance failure          | 2-5 minutes        | 0   | Infrastructure reschedules the instance (depends on failure detection timeout and storage reattachment). The other instance continues. Data already in each instance's UNS is preserved.                              |

RPO is zero in all three cases because the second instance is always collecting independently. Even during the rescheduling window, one instance is still publishing. This guarantee covers single-instance failure only. Simultaneous failure of both instances (for example, a site-wide power loss) is not covered by this pattern. See [High Availability](/production/high-availability) for infrastructure-level mitigations.

**Note on OPC UA reconnection.** After a connection loss, the OPC UA client must re-establish the TCP connection, create a new session, re-browse the node tree, and re-create subscriptions. This typically takes seconds to minutes depending on the server and number of nodes. This is why the redundant pattern matters: the second instance bridges the gap while the first reconnects.

## When this pattern does not apply

* **Your PLC is not redundant.** Connecting two bridges to a single PLC doubles the load on it without improving availability. Redundant collection requires two separate endpoints publishing the same data. A single PLC with two IP addresses does not qualify -- if the controller fails, both addresses go down. (If your goal is to avoid data gaps during planned updates rather than hardware failure, a single PLC with multiple addresses may be sufficient for a zero-downtime update strategy instead.)
* **You only need process-level or container-level recovery.** A standard single umh-core deployment already provides sub-second process recovery and 30-60 second container recovery. No additional topology is needed. See [High Availability](/production/high-availability).
* **You are in discrete manufacturing.** Automotive assembly lines, packaging lines, and machine tools typically use standard PLCs without redundancy. This pattern adds complexity without benefit in those environments.

## Related

* [High Availability](/production/high-availability) -- infrastructure-level recovery (container restarts, storage, node rescheduling)
* [Architecture Patterns](/production/architecture-patterns) -- overview of all deployment patterns


# Deployment

This section covers how to deploy umh-core in production environments.

## Deployment Options

umh-core is distributed as a container image. We provide instructions for using [Docker](/production/deployment/docker) and [Docker Compose](/production/deployment/docker-compose) to deploy it.


# Docker

This section covers deploying umh-core using the Docker CLI.


# Setup

You can run umh-core without the Management Console. But you'll need to edit configuration files directly.

Find the latest version on the [Releases](https://github.com/united-manufacturing-hub/united-manufacturing-hub/releases) page and replace `<VERSION>` with your selected version.

```bash
docker volume create umh-core-data

docker run -d \
  --name umh-core \
  --restart unless-stopped \
  --env AUTH_TOKEN=your-token \
  --env LOCATION_0=your-location \
  --env RELEASE_CHANNEL=stable \
  --env API_URL=https://management.umh.app/api \
  --volume umh-core-data:/data \
  management.umh.app/oci/united-manufacturing-hub/umh-core:<VERSION>
```


# Updating

To update umh-core, stop the container and start a new one with the latest image.

Find the latest version on the [Releases](https://github.com/united-manufacturing-hub/united-manufacturing-hub/releases) page and replace `<VERSION>` with your selected version.

```bash
docker stop umh-core
docker rm umh-core

docker run -d \
  --name umh-core \
  --restart unless-stopped \
  --volume umh-core-data:/data \
  management.umh.app/oci/united-manufacturing-hub/umh-core:<VERSION>
```

**That's it!** Your data is preserved in the `umh-core-data` volume.

> **Note:** On Linux without Docker group membership, prefix commands with `sudo`.

***

> **Using a custom data folder?** If you manually specified a folder path instead of using a Docker volume, see [Container Layout](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/production/deployment/reference/container-layout.md#advanced-custom-data-location) for upgrade instructions.


# Docker Compose

This documentation describes how to deploy umh-core using Docker Compose as an alternative to the standard `docker run` command mentioned in the [Getting Started section](/getting-started).

## Why Docker Compose?

While umh-core can be started with a single `docker run` command, real-world deployments often require additional services such as databases, visualization tools, and reverse proxies. Docker Compose allows you to define and manage multi-container applications in a single, declarative YAML file. Compared to using the Docker CLI alone, this is easier to set up and maintain.

If you are unfamiliar with Docker Compose, refer to the [official Docker Compose documentation](https://docs.docker.com/compose/) for a comprehensive introduction.

## What is Docker Compose

Docker Compose uses YAML files for the declaration of Docker Resources. These files are called `docker-compose.yaml`.

In these files Docker Volumes, Docker Networks and Docker Containers can all be declared and configured to work with eachother. This makes Docker Compose particularly valuable when additional Services like Grafana or TimescaleDB should be deployed alongside umh-core. But it is also valuable if only umh-core is deployed as it keeps the configuration of the container in a file instead of being lost to a one-off command.

For the configuration we recommend to most deployments, see the [Recommended UMH Stack](/production/deployment/docker-compose/additional-services/recommended-umh-stack).


# Setup

This guide walks you through setting up umh-core with Docker Compose, starting from a minimal configuration you can extend with [additional services](/production/deployment/docker-compose/additional-services).

## What you'll need

* [Docker](https://docs.docker.com/get-started/introduction/get-docker-desktop/) installed on your system
* Basic familiarity with the command line

## Minimal Setup

1. Find the latest version on the [Releases](https://github.com/united-manufacturing-hub/united-manufacturing-hub/releases) page and replace `ENTER_VERSION_HERE` with your selected version.
2. Save this in `docker-compose.yaml`.

> **Hint:** If your desired release is `v0.44.31`, the image tag is: `umh-core:v0.44.31`

```yaml
services:
  umh:
    # TODO: set your desired container version here 
    # e.g. `umh-core:v0.44.31`
    image: management.umh.app/oci/united-manufacturing-hub/umh-core:ENTER_VERSION_HERE
    restart: unless-stopped
    volumes:
      - umh-data:/data
    environment:
      # TODO: Enter your instance's Auth Token.
      # You'll find it in your instance's
      # configuration file on management.umh.app.
      - AUTH_TOKEN=your_auth_token
      # TODO: Change the LOCATION_0 parameter
      # to your desired Level 0 Location name
      - LOCATION_0=your_level_0_location
      # Optional: Define more levels
      # - LOCATION_1=your_level_1
      # - LOCATION_2=your_level_2
      # - LOCATION_3=your_level_3
      # - LOCATION_4=your_level_4
      - RELEASE_CHANNEL=stable
      - API_URL=https://management.umh.app/api

volumes:
  umh-data: {}
```

This achieves the same result as the docker cli commands, but the configuration is now documented in a file that you can version control and extend.

## Starting the Stack

Once the configuration is done you can start the stack:

```bash
docker compose up -d
```

We recommend that you familiarise yourself with the docker commands below. They'll come in handy while working with umh-core.

* `docker compose ps`: check the running state of all services
* `docker compose stats`: check resource usage of all services
* `docker compose pull`: re-pull the image tags currently listed in the `docker-compose.yaml`

> **Note:** `docker compose pull` does not move you to a newer UMH release. The image tags here are pinned to a specific version, so the pull only refreshes that same version. To deploy a newer release, change the tag in `docker-compose.yaml` first. See [Updating](/production/deployment/docker-compose/updating).

For more refer to the [official Docker Compose documentation](https://docs.docker.com/compose/).

## Add More Services

While umh-core can operate standalone, most deployments benefit from persistent storage and visualization capabilities.

* [Recommended UMH Stack](/production/deployment/docker-compose/additional-services/recommended-umh-stack): umh-core, Grafana, PgBouncer, and TimescaleDB in one file
* [TimescaleDB](/production/deployment/docker-compose/additional-services/timescaledb): time-series storage, with PgBouncer in front of it
* [Grafana](/production/deployment/docker-compose/additional-services/grafana): create dashboards that run locally from your data
* [nginx](/production/deployment/docker-compose/additional-services/nginx): reverse proxy and SSL termination


# Additional Services

Docker Compose lets you run more than umh-core. Each page in this section explains how to extend `services:` block of your `docker-compose.yaml`, with a complete configuration you can copy.

Most deployments want a database and dashboards, so start with the [Recommended UMH Stack](/production/deployment/docker-compose/additional-services/recommended-umh-stack), which combines umh-core, Grafana, PgBouncer, and TimescaleDB in one file.

To add services one at a time, see [TimescaleDB](/production/deployment/docker-compose/additional-services/timescaledb), [Grafana](/production/deployment/docker-compose/additional-services/grafana), or [nginx](/production/deployment/docker-compose/additional-services/nginx).


# Recommended UMH Stack

This is the Docker Compose configuration we recommend for most deployments: umh-core, Grafana, PgBouncer, and TimescaleDB in one file. If you are new to running umh-core with Docker Compose, read [Setup](/production/deployment/docker-compose/setup) first.

## Why these four services

umh-core collects and standardizes your data, but it does not keep a long history and it does not draw charts. The other three cover that.

* **TimescaleDB** is PostgreSQL with a time-series extension. Sensor readings, production counts, and machine states all arrive with timestamps, which is exactly what it is built to store and query.
* **Grafana** reads from TimescaleDB and turns it into dashboards. On its own it has nothing to query.
* **PgBouncer** sits in front of TimescaleDB as a connection pooler. umh-core bridges can open many simultaneous connections, and PostgreSQL handles a limited number of them (typically 100 by default). PgBouncer pools hundreds of client connections onto a handful of connections.

TimescaleDB is placed in an internal network that only PgBouncer can reach, so nothing else talks to the database directly.

## The configuration

Copy this into `docker-compose.yaml` and fill in the fields marked `TODO`. Find the latest umh-core version on the [Releases](https://github.com/united-manufacturing-hub/united-manufacturing-hub/releases) page and replace `ENTER_VERSION_HERE`.

> **IMPORTANT:** Change the database and Grafana credentials before using this in production!

```yaml
services:
  umh:
    # TODO: set your desired container version here 
    # e.g. `umh-core:v0.44.31`
    image: management.umh.app/oci/united-manufacturing-hub/umh-core:ENTER_VERSION_HERE
    restart: unless-stopped
    volumes:
      - umh-data:/data
    environment:
      # TODO: Enter your instance's Auth Token. 
      # You'll find it in your instance's configuration
      # file on management.umh.app.
      - AUTH_TOKEN=your_auth_token
      # TODO: Change the LOCATION_0 parameter
      # to your desired Level 0 Location name
      - LOCATION_0=your_level_0_location
      # Optional: Define more levels
      # - LOCATION_1=your_level_1
      # - LOCATION_2=your_level_2
      # - LOCATION_3=your_level_3
      # - LOCATION_4=your_level_4
      - RELEASE_CHANNEL=stable
      - API_URL=https://management.umh.app/api

  grafana:
    image: management.umh.app/oci/grafana/grafana:12.3.0
    restart: unless-stopped
    ports:
      - 3000:3000
    environment:
      # TODO: Set your desired username and password here
      # You'll need these credentials to
      # access your local Grafana instance
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
    healthcheck:
      test: ["CMD-SHELL", "curl --fail http://grafana:3000/api/health"]
      interval: 10s
      timeout: 5s
      retries: 3

  timescaledb:
    image: management.umh.app/oci/timescale/timescaledb:2.24.0-pg17
    restart: unless-stopped
    environment:
      - POSTGRES_DB=umh
      # TODO: Set your postgresDB
      # Password and Username here
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes:
      - timescaledb-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h timescaledb"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - timescaledb-network

  pgbouncer:
    image: management.umh.app/oci/edoburu/pgbouncer:v1.24.1-p1
    restart: unless-stopped
    environment:
      - DB_NAME=umh
      # This has to be the same value as
      # timescaledb.environment.POSTGRES_USER
      - DB_USER=postgres
      # This has to be the same value as
      # timescaledb.environment.POSTGRES_PASSWORD
      - DB_PASSWORD=postgres
      - DB_HOST=timescaledb
      - AUTH_TYPE=scram-sha-256
    ports:
      - 5432:5432
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h pgbouncer"]
      interval: 10s
      timeout: 5s
      retries: 5
    depends_on:
      timescaledb:
        condition: service_healthy
    networks:
      - default
      - timescaledb-network

networks:
  default:
  timescaledb-network:
    internal: true

volumes:
  umh-data: {}
  timescaledb-data: {}
  grafana-data: {}
```

Prefer to add one service at a time? Each page carries the same configuration in smaller pieces: [TimescaleDB](/production/deployment/docker-compose/additional-services/timescaledb), [Grafana](/production/deployment/docker-compose/additional-services/grafana).

To start the stack, see [Starting the Stack](/production/deployment/docker-compose/setup#starting-the-stack).

## VM Sizing

Our [**Sizing Guide**](/production/sizing-guide) covers umh-core on its own, and its recommended box is sized for that. Redpanda inside umh-core already budgets roughly 2 GB per core plus headroom, so running a database and Grafana next to it on the minimum box leaves little room. Give the host more memory and disk than the baseline, and watch actual usage before settling on a size.

## What's next

Starting the stack gives you four running containers. Two things still need configuring before a dashboard shows anything:

1. **Get data into TimescaleDB.** umh-core does not write to the database by itself. You configure a flow that reads from the Unified Namespace and writes to PostgreSQL. See [Stand-alone Flow](/usage/data-flows/stand-alone-flow), which covers the `kafka_to_postgresql_historian_bridge`.
2. **Point Grafana at the database.** Add a PostgreSQL data source in Grafana using host `pgbouncer:5432`, database `umh`, and the credentials you set above. See Grafana's [PostgreSQL data source documentation](https://grafana.com/docs/grafana/latest/datasources/postgres/).


# TimescaleDB

In this example you'll learn how to add TimescaleDB and PgBouncer to a umh-core Docker Compose stack. If you are new to running umh-core with Docker Compose, read [Setup](/production/deployment/docker-compose/setup) first.

[TimescaleDB](https://www.timescale.com/) is PostgreSQL with an extension optimized for time-series data. Manufacturing data is inherently time-series: sensor readings, production counts, and machine states all have timestamps.

PgBouncer is a connection pooler for TimescaleDB. umh-core bridges can create many simultaneous connections to the database, and PostgreSQL handles a limited number of them (typically 100 by default), so without pooling a busy system exhausts the limit and connections start failing. Applications connect to PgBouncer on port 5432 and PgBouncer connects to TimescaleDB internally.

## Complete docker-compose.yaml

umh-core with TimescaleDB and PgBouncer. Copy this into `docker-compose.yaml` and fill in the fields marked `TODO`.

> **IMPORTANT**: Change the template's database credentials before using this in production!

```yaml
services:
  umh:
    # TODO: set your desired container version here 
    # e.g. `umh-core:v0.44.31`
    image: management.umh.app/oci/united-manufacturing-hub/umh-core:ENTER_VERSION_HERE
    restart: unless-stopped
    volumes:
      - umh-data:/data
    environment:
      # TODO: Enter your instance's Auth Token.
      # You'll find it in your instance's configuration
      # file on management.umh.app.
      - AUTH_TOKEN=your_auth_token
      # TODO: Change the LOCATION_0 parameter
      # to your desired Level 0 Location name
      - LOCATION_0=your_level_0_location
      # Optional: Define more levels
      # - LOCATION_1=your_level_1
      # - LOCATION_2=your_level_2
      # - LOCATION_3=your_level_3
      # - LOCATION_4=your_level_4
      - RELEASE_CHANNEL=stable
      - API_URL=https://management.umh.app/api

  timescaledb:
    image: management.umh.app/oci/timescale/timescaledb:2.24.0-pg17
    restart: unless-stopped
    environment:
      - POSTGRES_DB=umh
      # TODO: Set your postgresDB Password and Username here
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes:
      - timescaledb-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h timescaledb"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - timescaledb-network

  pgbouncer:
    image: management.umh.app/oci/edoburu/pgbouncer:v1.24.1-p1
    restart: unless-stopped
    environment:
      - DB_NAME=umh
      # This has to be the same value as
      # timescaledb.environment.POSTGRES_USER
      - DB_USER=postgres
      # This has to be the same value as
      # timescaledb.environment.POSTGRES_PASSWORD
      - DB_PASSWORD=postgres
      - DB_HOST=timescaledb
      - AUTH_TYPE=scram-sha-256
    ports:
      - 5432:5432
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h pgbouncer"]
      interval: 10s
      timeout: 5s
      retries: 5
    depends_on:
      timescaledb:
        condition: service_healthy
    networks:
      - default
      - timescaledb-network

networks:
  default:
  timescaledb-network:
    internal: true

volumes:
  umh-data: {}
  timescaledb-data: {}
```

To start the stack, see [Starting the Stack](/production/deployment/docker-compose/setup#starting-the-stack).

## Already running umh-core?

Add the following to the `docker-compose.yaml` you already have. Your existing `umh:` service stays as it is.

**1. Two new services inside `services:`**

```yaml
  timescaledb:
    image: management.umh.app/oci/timescale/timescaledb:2.24.0-pg17
    restart: unless-stopped
    environment:
      - POSTGRES_DB=umh
      # TODO: Set your postgresDB Password and Username here
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes:
      - timescaledb-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h timescaledb"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - timescaledb-network

  pgbouncer:
    image: management.umh.app/oci/edoburu/pgbouncer:v1.24.1-p1
    restart: unless-stopped
    environment:
      - DB_NAME=umh
      # This has to be the same value as
      # timescaledb.environment.POSTGRES_USER
      - DB_USER=postgres
      # This has to be the same value as
      # timescaledb.environment.POSTGRES_PASSWORD
      - DB_PASSWORD=postgres
      - DB_HOST=timescaledb
      - AUTH_TYPE=scram-sha-256
    ports:
      - 5432:5432
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h pgbouncer"]
      interval: 10s
      timeout: 5s
      retries: 5
    depends_on:
      timescaledb:
        condition: service_healthy
    networks:
      - default
      - timescaledb-network
```

**2. A new top-level `networks:` section**

```yaml
networks:
  default:
  timescaledb-network:
    internal: true
```

**3. One more entry under `volumes:`**

```yaml
  timescaledb-data: {}
```

**4. Apply the changes**

```bash
docker compose up -d
```

See [Starting the Stack](/production/deployment/docker-compose/setup#starting-the-stack) for the other compose commands.

## What this declares

* 2 Services called `pgbouncer` and `timescaledb`: PgBouncer acts as a proxy to TimescaleDB. This means the Credentials have to match between these two Services. The `depends_on` entry with `condition: service_healthy` on `pgbouncer` enforces the startup order; the Healthchecks only report whether a Service is healthy. `timescaledb` is isolated in the Network `timescaledb-network`. `pgbouncer` is both in the `default` and the `timescaledb-network`. This makes PgBouncer the only Service which can talk to TimescaleDB directly.
* 2 Networks called `timescaledb-network` and `default`: Without the Network section the Network `default` is always created by default. Services use the Network `default` if no explicit Network configuration is provided. The `timescaledb-network` is configured to be internal which means Services can't reach the internet or any Service outside this Network if they are only connected through this Network.
* 1 Volume called `timescaledb-data`: This is where TimescaleDB stores its data.

Once the stack is running, PostgreSQL is reachable through PgBouncer at `localhost:5432`.

A database on its own has nothing to show you. Most deployments add [Grafana](/production/deployment/docker-compose/additional-services/grafana) on top of it, which is why both are part of the [**Recommended UMH Stack**](/production/deployment/docker-compose/additional-services/recommended-umh-stack).


# Grafana

In this example you'll learn how to add Grafana to a umh-core Docker Compose stack. If you're not sure about umh-core and Docker Compose stacks, start with [Setup](/production/deployment/docker-compose/setup).

[Grafana](https://grafana.com/) is an open-source visualization platform. It allows you to build dashboards showing real-time and historical data based on your umh-core configuration.

Grafana's admin account defaults to `admin` / `admin`. Port 3000 is published to the host, so set both values before you start the stack. The variable names follow Grafana's `GF_<SECTION>_<KEY>` convention, documented in the [Grafana configuration reference](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/).

## Complete docker-compose.yaml

umh-core with Grafana. Copy this into `docker-compose.yaml` and fill in the fields marked `TODO`.

```yaml
services:
  umh:
    # TODO: set your desired container version here 
    # e.g. `umh-core:v0.44.31`
    image: management.umh.app/oci/united-manufacturing-hub/umh-core:ENTER_VERSION_HERE
    restart: unless-stopped
    volumes:
      - umh-data:/data
    environment:
      # TODO: Enter your instance's Auth Token. 
      # You'll find it in your instance's configuration
      # file on management.umh.app.
      - AUTH_TOKEN=your_auth_token
      # TODO: Change the LOCATION_0 parameter
      # to your desired Level 0 Location name
      - LOCATION_0=your_level_0_location
      # Optional: Define more levels
      # - LOCATION_1=your_level_1
      # - LOCATION_2=your_level_2
      # - LOCATION_3=your_level_3
      # - LOCATION_4=your_level_4
      - RELEASE_CHANNEL=stable
      - API_URL=https://management.umh.app/api

  grafana:
    image: management.umh.app/oci/grafana/grafana:12.3.0
    restart: unless-stopped
    ports:
      - 3000:3000
    environment:
      # TODO: Set your desired username and password here
      # You'll need these credentials to
      # access your local Grafana instance
      - GF_SECURITY_ADMIN_USER=admin     
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
    healthcheck:
      test: ["CMD-SHELL", "curl --fail http://grafana:3000/api/health"]
      interval: 10s
      timeout: 5s
      retries: 3

volumes:
  umh-data: {}
  grafana-data: {}
```

To start the stack, see [Starting the Stack](/production/deployment/docker-compose/setup#starting-the-stack).

## Already running umh-core?

Add the following to the `docker-compose.yaml` you already have. Your existing `umh:` service stays as it is.

**1. One new service inside `services:`**

```yaml
  grafana:
    image: management.umh.app/oci/grafana/grafana:12.3.0
    restart: unless-stopped
    ports:
      - 3000:3000
    environment:
      # TODO: Set your desired username and password here
      # You'll need these credentials to
      # access your local Grafana instance
      - GF_SECURITY_ADMIN_USER=admin     
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-data:/var/lib/grafana
    healthcheck:
      test: ["CMD-SHELL", "curl --fail http://grafana:3000/api/health"]
      interval: 10s
      timeout: 5s
      retries: 3
```

**2. One more entry under `volumes:`**

```yaml
  grafana-data: {}
```

**3. Apply the changes**

```bash
docker compose up -d
```

See [Starting the Stack](/production/deployment/docker-compose/setup#starting-the-stack) for the other compose commands.

## Connecting to Grafana

Once the stack is running, Grafana is reachable at `http://localhost:3000`.

To log into Grafana, use the credentials that you defined in `docker-compose.yaml`.

> 💡 Without a persistent database behind it there is nothing to query, so most deployments run Grafana together with TimescaleDB. We call it the [**Recommended UMH Stack**](/production/deployment/docker-compose/additional-services/recommended-umh-stack).


# nginx

Nginx is a high-performance web server and reverse proxy. In a umh-core deployment, nginx can be used as the proxy for incoming HTTP requests, handling SSL and routing requests to the appropriate services.

## Why a proxy?

umh-core pipelines can expose HTTP endpoints using the `http_server` input. These endpoints allow external systems to send data to umh-core. However, these endpoints should be secured. It is possible to manage certificates in each of those inputs but it is inconvenient to maintain as certificates have to be replaced regularly.

## Adding Nginx in the existing Docker Compose Configuration

Add nginx to your `docker-compose.yaml` within the `services` section:

```diff
  services:
+   nginx:
+     image: management.umh.app/oci/library/nginx:1.27.3-alpine
+     restart: unless-stopped
+     ports:
+       - 80:80   # for http access
+       - 443:443 # for https access
```

## Nginx Configuration File

Refer to the [Nginx documentation](https://nginx.org/en/docs/index.html) for how to create your own configuration.

If you want to customize the `nginx.conf` create an `nginx.conf` file in the same directory as your `docker-compose.yaml` and mount it:

```diff
  services:
    umh:
      # ... existing configuration ...
    nginx:
      image: management.umh.app/oci/library/nginx:1.27.3-alpine
      restart: unless-stopped
      ports:
        - 80:80   # for http access
        - 443:443 # for https access
+     volumes:
+       - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
```

Refer to the [Nginx documentation](https://nginx.org/en/docs/index.html) for what you can do in this configuration.

## Adding a simple http/https forward

```conf
# example of an http/https server configuration for forwarding requests to umh
server {
  listen 80;
  listen 443 ssl;
  server_tokens off;

  # Add CORS headers globally for this server
  add_header 'Access-Control-Allow-Origin' '*' always;
  add_header 'Access-Control-Allow-Methods' 'POST,GET,HEAD,OPTIONS' always;
  add_header 'Access-Control-Allow-Headers' 'content-type' always;

  # Handle OPTIONS requests
  if ($request_method = 'OPTIONS') {
      return 204;
  }

  location /example/route/ {
      # docker exposes the service `umh` as a hostname
      # the port is an example for an http_input that has to be created in umh
      proxy_pass http://umh:8285;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
  }
}
```

## SSL

Obtain certificates from a trusted Certificate Authority (CA) or use Let's Encrypt. Place the `*.crt` and `*.key` file in the `certs` directory and create a `nginx.conf` for nginx which uses those certificates.

The file structure should now look similar to this.

```
.
├── certs
│   ├── example.crt
│   └── example.key
├── docker-compose.yaml
└── nginx.conf
```

The `docker-compose.yaml` has to mount `certs` and `nginx.conf` in appropriate places for nginx. This would look similar to this example:

```diff
  services:
    nginx:
      image: management.umh.app/oci/library/nginx:1.27.3-alpine
      restart: unless-stopped
      ports:
        - 80:80   # for http access
        - 443:443 # for https access
+     volumes:
+       - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
+       - ./certs:/etc/nginx/tls:ro
```


# Updating

To update umh-core, edit the image tag in your `docker-compose.yaml` and restart the stack.

1. Find the latest version on the [Releases](https://github.com/united-manufacturing-hub/united-manufacturing-hub/releases) page and replace `ENTER_VERSION_HERE` with your selected version.
2. Update the umh-core image tag in `docker-compose.yaml`:

```yaml
services:
  umh:
    # make sure to include 'v' in the version string like so: v0.44.31
    image: management.umh.app/oci/united-manufacturing-hub/umh-core:ENTER_VERSION_HERE
```

3. Pull the new image and restart:

```bash
docker compose up -d
```

**That's it!** Your data is preserved in the named volumes.

> **Note:** On Linux without Docker group membership, prefix commands with `sudo`.


# Sizing Guide

## Supported Architectures

| Architecture | Status            | Notes                                                                             |
| ------------ | ----------------- | --------------------------------------------------------------------------------- |
| x86\_64      | ✅ Fully supported | Intel/AMD 64-bit processors                                                       |
| ARM64        | ✅ Fully supported | 64-bit ARM processors (AWS Graviton, Apple Silicon, etc.)                         |
| Raspberry Pi | ❌ Not supported   | Redpanda requires 48-bit virtual address space; Raspberry Pi provides only 38-bit |

**Why Raspberry Pi doesn't work:** Redpanda uses the Seastar framework which requires a 48-bit virtual address space. Raspberry Pi's ARM processors only provide 38-bit VA space, causing SIGABRT on startup. This is a fundamental hardware limitation documented in [Redpanda GitHub Issue #1542](https://github.com/redpanda-data/redpanda/issues/1542).

## Recommended Starting Point

**Start with → 2 vCPU · 4 GB RAM · 40 GB SSD**

#### What that box handles

* **≈ 4 bridges instances** (e.g. OPC UA ➜ Redpanda) **plus one bridge instance** that forwards from the local Redpanda to an external MQTT broker
* **≈ 900 tags at 1 message / second each**
* Keeps **seven days** of replayable raw history under the defaults a new instance starts with (`log_retention_ms = 7 days`, `log_cleanup_policy = delete`)
* Runs comfortably below 70% CPU (automatic throttling protection kicks in above this)

#### Disk usage in practice

Redpanda writes **128 MiB segments**; a segment can be deleted only after it is closed.\
With Snappy compression, a typical 200 B JSON payload shrinks to ≈ 50–70 B (3–4× ratio).\
Allowing a 5 GB safety buffer, a 40 GB SSD gives **≈ 35 GB usable history ≙ \~500–700 million messages**.

Size the disk against your message rate, not your tag count: seven days of raw history is what makes replay possible, and time-based retention has no size ceiling of its own. At 700 messages / second — ≈ 423 million records over seven days — `umh.messages` alone needs roughly **40–55 GB**. Measure the real figure on your instance, since payload size drives it.

*Need more?*\
Shorten retention (either during install with `internal.redpanda.redpandaServiceConfig.topic.defaultTopicRetentionMs` or later on the topic level using `rpk`), cap the topic with `defaultTopicRetentionBytes`, or enlarge the disk. A byte cap protects the disk but can cut the replay window below seven days under heavy load.

#### Memory

| Component           | Rule of thumb                                    |
| ------------------- | ------------------------------------------------ |
| Redpanda            | ≈ 2 GB · cores + 1.5 GB head-room (Seastar rule) |
| Agent + supervision | ≈ 150 MB                                         |
| Each extra pipeline | ≈ 100 MB                                         |

#### CPU

**Theoretical Bridge Limits:**

* We recommend **5 bridges per CPU core** (after reserving 1 core for Redpanda)
* Example: 2 CPU cores = (2-1) × 5 = **5 bridges maximum**
* Example: 4 CPU cores = (4-1) × 5 = **15 bridges maximum**

**Dynamic Resource Protection:** Since every bridge has different resource requirements (OPC UA with 10,000 tags uses more CPU than MQTT with 100 tags), we also monitor actual resource usage:

* **CPU Utilization**: A new bridge waits while CPU usage is above 70%. That is the rule without [CPU Health](/production/cpu-health). With CPU Health turned on (`USE_FSMV2_CPU=true`), usage alone no longer makes a new bridge wait. There, 70% is a last-resort fallback for machine headroom, used only when no CPU limit is set and the kernel publishes no PSI. With CPU Health turned on, if UMH cannot read a file it needs to measure the CPU, CPU health is degraded and a new bridge waits.
* **CPU Throttling**: A new bridge waits while the container is being throttled. Throttling means the system needs brief CPU bursts (e.g., when processing message batches) but hits the CPU limit, causing delays and degraded performance even if average CPU usage looks acceptable
* **Memory Usage**: A new bridge waits while memory use is above 80%
* **Disk Usage**: A new bridge waits while disk use is above 85%

**Redpanda CPU Utilization:** UMH Core runs Redpanda with the `--overprovisioned` flag, which optimizes CPU usage for containerized environments. This disables Seastar's busy-polling reactor model, reducing idle CPU usage from 100% to near-zero when not processing messages. The trade-off is slightly higher latency (microseconds to low milliseconds), which is acceptable for manufacturing data that doesn't require sub-millisecond response times. This is required because UMH Core runs in Docker where CPU pinning doesn't work effectively, and Redpanda shares the container with other processes.

**Automatic Enforcement:** The system will prevent you from deploying new bridges if:

1. You've reached the theoretical limit for your CPU allocation, OR
2. The system detects resource degradation (high CPU, throttling, memory, or disk pressure), OR
3. The instance's resource health is not proven yet, for example right after a restart

This check is bridge admission. The emergency setting *Turn off bridge admission* turns it off, in your `config.yaml`:

```yaml
agent:
  enableResourceLimitBlocking: false  # Start bridges even when a resource is degraded or not proven yet, or the bridge limit is reached
```

`true` is the default, also when `config.yaml` does not contain the key. While it is `false`, the instance shows as degraded. See [Bridges Do Not Start](/usage/data-flows/bridges/bridges-do-not-start) for the full procedure.

While bridge admission is on, it keeps one bridge from overloading the others. If you need more bridges, either:

* Increase CPU allocation (for containerized deployments)
* Upgrade to a larger instance (for VM/bare-metal deployments)
* Optimize existing bridges (reduce polling rates, tag counts, or processing complexity)

#### Resource Limit Error Messages

When a new bridge waits, its status reason says why:

* **Bridge limit**: `Cannot create bridge - limit exceeded (5 bridges maximum with 2.0 CPU cores, 1 core reserved for Redpanda)`
* **High CPU**: `CPU degraded: CPU utilization critical`
* **High Memory**: `Memory degraded: Memory usage at 85%`
* **High Disk**: `Disk degraded: Disk usage at 90%`
* **Health not proven yet**: `Resource health not proven yet`

#### Easy vertical scaling

UMH Core is stateless besides the **`/data`** volume. To grow:

1. Stop the container
2. Move or resize the volume / attach it to a bigger VM
3. Start the same image — no reinstall or re-configuration required

#### High availability

For multi-node failover, mount `/data` on storage accessible from multiple nodes. See [High Availability](/production/high-availability).


# CPU Health

{% hint style="info" %}
**Early access.** This reporting needs UMH Core v0.44.37 or later, started with `-e USE_FSMV2_CPU=true` added to the `docker run` command that starts the container. The flag is read at startup, so changing it means recreating the container. The CPU health check does not need Management Console credentials: it also runs on an instance started without `AUTH_TOKEN` and `API_URL`. Without the flag, an instance is marked degraded whenever its CPU usage stays above 70% of the cores it may use, whether or not work is being delayed.
{% endhint %}

UMH Core reports whether an instance has the CPU it needs. The Management Console shows the result as a CPU status on the instance's detail page, and while that status is degraded UMH will not start another bridge there. High usage on its own does not make an instance degraded: it degrades when its headroom is gone, or when its work is measurably delayed, which UMH reads from throttling, CPU pressure and steal. (For the reasoning, see [why average CPU utilization is the wrong signal](https://www.theocharis.dev/blog/why-we-should-get-rid-of-average-cpu-utilization/).)

What UMH measures headroom against depends on whether the container has a CPU limit (a Docker `--cpus` or a Kubernetes CPU limit). With no limit, the machine is the ceiling: UMH averages how busy the machine is over 60 seconds and reports degraded when less than about one core is free. It also shows the container's own usage, so you can see how much of the machine total is UMH. With a limit, the limit is the ceiling: UMH measures headroom against those cores rather than the machine, and reports degraded once its usage passes 90% of the limit, the last 10% being held in reserve, or when the kernel throttles it.

## What each CPU status means

| Status                              | What it means                                                                                                                                                 | What to do                                                                                                                                                                                                                             |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **CPU healthy**                     | The instance has the CPU it needs. Usage is shown for context, for example "1.2 of 4 cores".                                                                  | Nothing.                                                                                                                                                                                                                               |
| **CPU healthy, limited visibility** | The instance looks fine, but UMH cannot fully measure CPU health here: no CPU limit is set and the operating system is not reporting CPU-pressure statistics. | For full monitoring, set a CPU limit, or [enable CPU pressure stats](#enabling-cpu-pressure-stats).                                                                                                                                    |
| **CPU limited**                     | The instance hit its CPU limit and was paused until the next scheduling cycle, for example in 12% of cycles over the last minute. Work is being delayed.      | Raise the CPU limit, or reduce the load on the instance.                                                                                                                                                                               |
| **CPU contention**                  | Tasks inside the instance spent time waiting for a free CPU core, for example 23% of the last minute.                                                         | Reduce the load, or give the instance more CPU. Workloads sharing the server may be competing for it.                                                                                                                                  |
| **CPU taken by the server**         | Other virtual machines on the same physical server took CPU this instance needed.                                                                             | On your virtualization platform, give this VM more guaranteed CPU, or move the other VMs off the server.                                                                                                                               |
| **CPU running near full**           | There is no room left for the next burst of work, because either the machine is full or the instance is at its CPU limit. The status message says which.      | Add CPU capacity, reduce load, or raise the CPU limit. If the host is full and this container is not the cause, reserve or pin cores for UMH, or reduce what else runs on the machine: a CPU limit caps UMH rather than protecting it. |

If UMH cannot read a file it needs to measure the CPU, the CPU health is degraded and the message names the file, for example `CPU not measured: open /sys/fs/cgroup/cpu.stat: no such file or directory`. To check whether the container can read that file, run `docker exec <container> cat <file>`, for example `docker exec umh-core cat /sys/fs/cgroup/cpu.stat`.

## Where to see the CPU health

The Management Console shows CPU on the instance's detail page: the status, the usage row, and a Technical Details section listing every signal UMH can measure on that machine.

Both the status and Technical Details carry the thresholds. A healthy instance states its remaining headroom, and each Technical Details line gives the reading next to the mark that would change it:

```
CPU healthy. This instance is using 0.3 of 2 cores (15% of its limit) and can use 1.5 more before it is marked degraded.

Technical Details:
Instance headroom 1.5 cores = 2 total - 0.3 used - 0.2 reserved (degrades below 0).
Instance usage not measured (throttling already shows whether this instance is hitting its CPU limit, so this rough estimate is not needed).
Throttling 2% (degrades above 5%).
Pressure 4% (degrades above 20%).
Steal not measured (this instance is not running in a virtual machine, so no other virtual machine can take its CPU).
```

Only the headroom lines are conditional: a machine whose core count is readable also gets a Machine headroom line, and an instance with no CPU limit gets no Instance headroom line. Every other signal keeps its line whether or not this machine can measure it.

A signal that has already fired shows what would clear it instead, for example `Throttling 12% (recovers below 3%)`. A signal this machine cannot measure names the fact that stops it being measured, rather than reading zero.

## Thresholds

Each signal degrades at one value and recovers at another, held apart so the status does not flicker. Which direction is worse depends on the signal: throttling, pressure, steal and usage are worse the higher they read, headroom the lower. All readings are 60-second figures. A reading has to pass a threshold, not merely reach it, except where the table says "at".

| Signal               | Degrades                       | Recovers               | Measured only when                                                                                      |
| -------------------- | ------------------------------ | ---------------------- | ------------------------------------------------------------------------------------------------------- |
| **Throttling**       | above 5% of scheduling periods | below 3%               | a CPU limit is set                                                                                      |
| **CPU pressure**     | above 20% (PSI `avg60`)        | below 12%              | the container can read `cpu.pressure` (see [Enabling CPU pressure stats](#enabling-cpu-pressure-stats)) |
| **CPU steal**        | above 10%                      | below 6%               | the machine is a virtual machine                                                                        |
| **Machine headroom** | less than 1 core free          | 1.5 cores free         | the machine's core count is readable                                                                    |
| **Limit headroom**   | usage past 90% of the limit    | below 85% of the limit | a CPU limit is set                                                                                      |
| **Instance usage**   | at 70%                         | below 60%              | no CPU limit is set and the kernel publishes no PSI (last-resort fallback for machine headroom)         |

Steal uses the 95th percentile once 20 samples are in, and the mean before that, so a fresh instance is judgeable within seconds of starting. Bare metal reports no steal at all, so on a physical machine that signal reads "not measured" with its reason rather than 0%.

## Enabling CPU pressure stats

UMH reads CPU pressure from the container's own `/sys/fs/cgroup/cpu.pressure`. That file exists only if the kernel booted with Pressure Stall Information (PSI) on and the machine runs cgroup v2. Most distributions ship both on. The Red Hat family is the exception: RHEL, Rocky, AlmaLinux and Oracle Linux ship PSI off, and version 8 of each also boots cgroup v1.

Check your own machine:

```bash
docker exec umh-core cat /sys/fs/cgroup/cpu.pressure
```

A line beginning `some avg10=` means there is nothing to do. `No such file or directory` means one or both are missing. Find out which, on the host:

```bash
stat -fc %T /sys/fs/cgroup
```

`cgroup2fs` means only PSI is missing: set `psi=1`. `tmpfs` means the machine runs cgroup v1, where `cpu.pressure` does not exist at all.

On cgroup v1, UMH reads the container's usage, throttling and CPU limit from the v1 controller files, and machine headroom and steal from `/proc/stat`. CPU pressure is the only signal it cannot read, because cgroup v1 has no per-container pressure file.

### Set the kernel parameters

Each operating system sets kernel parameters differently. Follow yours, then reboot:

| Operating system                             | Instructions                                                                                                                                                                                                                                                                                                |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| RHEL, Rocky, AlmaLinux, Oracle Linux, Fedora | [Configuring kernel command-line parameters](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/managing_monitoring_and_updating_the_kernel/configuring-kernel-command-line-parameters_managing-monitoring-and-updating-the-kernel): `sudo grubby --update-kernel=ALL --args="psi=1"` |
| Debian and Ubuntu                            | [GRUB 2 setup](https://help.ubuntu.com/community/Grub2/Setup): add `psi=1` to `GRUB_CMDLINE_LINUX` in `/etc/default/grub`, then run `sudo update-grub`                                                                                                                                                      |

`update-grub` is a Debian and Ubuntu wrapper. On another GRUB machine, edit `GRUB_CMDLINE_LINUX` the same way and run `grub-mkconfig -o /boot/grub/grub.cfg`.

### Confirm UMH can read the file

Run the `docker exec` check again. It reads the same file UMH reads.

The Management Console shows the result on the instance's detail page. The Technical Details `Pressure` line changes from `Pressure not available (not possible).` to a percentage. `Pressure not available (measuring).` means UMH can read the file and is filling its 60-second window, so wait a minute and look again.

If the file is still missing, run `cat /proc/pressure/cpu` on the host. Output means PSI is on and cgroup v2 is what is missing. `No such file or directory` means the kernel ignored `psi=1`, which happens on a kernel built without PSI. If the kernel has no PSI, set a CPU limit on the container instead. UMH then judges the container against its limit. Pressure stays unavailable.

## When a new bridge waits for CPU

While CPU is degraded, UMH will not start an additional bridge on the instance, because it would compete for CPU that is already short. The bridge waits, and its status reason names the resource that stopped it, here the CPU. For every reason a bridge waits and the full procedure, see [Bridges Do Not Start](/usage/data-flows/bridges/bridges-do-not-start). Bridges already running are left alone.

To start bridges anyway in an emergency, use the emergency setting *Turn off bridge admission*. It turns off every check, not only CPU:

```yaml
agent:
  enableResourceLimitBlocking: false
```

While it is `false`, the instance shows as degraded.

This is separate from the bridge limit, the number of bridges a given core count can hold, which the [Sizing Guide](/production/sizing-guide) covers. The bridge limit is an upper bound, not a guarantee: because real CPU use varies per bridge, a bridge can wait on CPU health before you reach it.

## Known limitation

A container pinned to a subset of the machine's CPUs (`cpuset`) with no CPU limit set can print a usage pair measured against different core counts, such as "using 6.0 of 2 cores", beside a healthy verdict. Set a CPU limit on such a container to get a correct reading.

## Glossary

| Term                | Meaning                                                                                                                                                                                                                                                                        |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Throttling**      | The kernel caps a container to its CPU limit in short repeating periods, about 100 ms each. A container that needs more within a period is paused until the next one, so a workload whose average looks fine can still be paused during bursts.                                |
| **CPU pressure**    | How much time tasks spent waiting for a free CPU core, from Linux Pressure Stall Information (PSI). High pressure means CPU is the bottleneck.                                                                                                                                 |
| **CPU steal**       | Time the hypervisor scheduled this machine's CPU elsewhere. High steal usually means the physical server is oversubscribed. A burstable cloud instance whose CPU credits have run out is capped by a different mechanism, but the guest counts that capped time as steal too.  |
| **Host contention** | CPU used by software outside UMH on the same machine. UMH cannot see which processes those are, only that they are using CPU it needs. It is not the **CPU contention** status above: that one is measured inside the instance, as tasks of UMH's own waiting for a free core. |


# High Availability

UMH Core handles high availability without custom failover daemons. Inside the container, [S6](https://skarnet.org/software/s6/) supervises all processes and restarts them on failure (see [Container Layout](/reference/container-layout) for details). Outside the container, your container manager (Docker, Kubernetes, Docker Swarm, Portainer, or any other) handles container restarts and rescheduling.

**Typical recovery times** (actual times depend on configuration and workload):

| Failure                        | Recovery                                                                    |
| ------------------------------ | --------------------------------------------------------------------------- |
| Process crash inside container | Sub-second (S6 restarts the process)                                        |
| Container/pod crash            | 30-60 seconds (depends on restart policy and resource availability)         |
| Node failure                   | 2-5 minutes (depends on failure detection timeout and storage reattachment) |

## How Failover Works

All state lives in the `/data` volume (see [Container Layout](/reference/container-layout) for structure). When a container or node fails, your container manager starts a new container that mounts the same volume. Redpanda recovers from its on-disk data and resumes processing.

## Why Not Custom Failover?

Some industrial software implements custom heartbeat-based failover. This approach predates container orchestration. Today, it duplicates what your container manager already does. It also introduces split-brain risk: the heartbeat network can partition while both nodes remain up, causing both to assume primary ownership.

UMH Core relies on your container manager as the single source of truth for scheduling.

## Storage Requirements

### Single-Node Deployments

If you have one node, storage choice does not affect failover. The node itself is the single point of failure. Redundancy comes from your underlying infrastructure. For example, a VM on vSAN survives host failure, and RAID storage survives disk failure.

### Multi-Node Deployments

For multi-node failover, you need storage accessible from multiple nodes.

The default k3s storage (`local-path`) binds the volume to one node. If that node dies, the container manager cannot reschedule the container elsewhere because the data is stuck on the failed node.

**Compatible storage options** (all support mounting volumes on any node):

* Enterprise SAN or vSAN
* Cloud block storage: EBS, Azure Disk, GCE PD
* Software-defined: Longhorn, Rook-Ceph

**Performance note:** Software-defined storage like Longhorn adds latency due to synchronous replication. For most UMH Core workloads, this is acceptable. For high-throughput scenarios, benchmark your specific workload before committing.

### Longhorn on k3s

If you do not have enterprise storage, Longhorn is a good choice for k3s clusters. See the [Longhorn Quick Installation Guide](https://longhorn.io/docs/latest/deploy/install/) and [K3s-specific configuration](https://longhorn.io/docs/latest/advanced-resources/os-distro-specific/csi-on-k3s/).

## If You Need Zero Data Loss

The standard approach accepts 30-60 second recovery on container failure. If your process cannot tolerate any data loss during a single-instance failure, see [Redundant Data Collection](/production/architecture-patterns/redundant-data-collection) for a topology that provides zero data loss using redundant PLCs and two umh-core instances.

## OPC UA Sessions After Failover

OPC UA sessions must be re-established after any failover. UMH Core reconnects to OPC UA servers automatically, but re-establishment takes a few seconds during which no data is collected from OPC UA sources. The OPC UA protocol requires this.


# Metrics

`http://<device-ip>:8080/metrics` (Prometheus format) exposes:

## Currently Available ✅

* Agent tick & FSM timings (each full reconcile loop < 100 ms by design)
* Per-DFC counters: processed, error, latency, active / idle flag
* Redpanda I/O and disk-utilisation stats
* CPU health evidence (preview)

## CPU health evidence (preview)

Every gauge below is exposed as the series `umh_fsmv2_worker_<gauge name>`, carrying a `hierarchy_path` label that names the worker that published it. A rule written against the bare name matches nothing, and Prometheus reports no error for it.

On a tick that could not measure, every gauge on this page keeps being scraped at its previous value, and nothing marks it stale. The age of `cpu_last_sample_unix` is the only thing that reveals such a freeze.

### Measurements

| Gauge                      | Reports                                                              |
| -------------------------- | -------------------------------------------------------------------- |
| `cpu_avg_usage_cores`      | this container's 60s mean usage, in cores                            |
| `cpu_avg_usage_fraction`   | that mean as a 0..1 fraction of the CPUs this container may run on   |
| `cpu_throttle_ratio`       | the 60s share of scheduling periods that were throttled, 0..1        |
| `cpu_pressure_avg60_ratio` | the kernel's PSI cpu-some avg60 as a 0..1 fraction                   |
| `cpu_host_headroom_cores`  | cores free on the host after the reserve; negative on a full machine |
| `cpu_avg_host_busy_cores`  | the whole machine's 60s mean busy time, in cores                     |
| `cpu_capacity_cores`       | the ceiling CPU health judged against, in cores                      |
| `cpu_reserve_cores`        | the part of that ceiling held back, in cores                         |
| `cpu_host_cpus`            | the machine's CPU count                                              |
| `cpu_last_sample_unix`     | the unix seconds of the last tick that measured                      |

### Flags

| Flag                            | Applies to                                              |
| ------------------------------- | ------------------------------------------------------- |
| `cpu_usage_ring_active`         | `cpu_avg_usage_cores`                                   |
| `cpu_host_busy_ring_active`     | `cpu_avg_host_busy_cores`                               |
| `cpu_host_busy_cores_available` | `cpu_avg_host_busy_cores` and `cpu_host_headroom_cores` |
| `cpu_throttle_signal_ready`     | `cpu_throttle_ratio`                                    |
| `cpu_pressure_signal_ready`     | `cpu_pressure_avg60_ratio`                              |
| `cpu_host_headroom_available`   | `cpu_host_headroom_cores`                               |

Each flag reads 1 or 0. A 0 means the number beside it is not worth acting on this tick. A figure averaged over too few samples and a 0 nobody measured both publish with the flag at 0, and nothing in the exposed metrics tells them apart.

A `_ring_active` flag covers one 60-second window, and reads 0 until that window holds enough samples to trust. A `_signal_ready` flag covers a whole signal, and reads 0 both while the window fills and when the signal could not be read. It also reads 0 for as long as the container runs on a machine with no instrument for that signal, such as a bare-metal host with no cgroup throttle counters. An `_available` flag reads 0 when this tick's sample could not supply the figure. `cpu_host_headroom_available` reads 0 when the container is pinned to a subset of the machine's CPUs, and also when the machine's CPU count could not be read. The first is a deployment choice, the second a read failure worth investigating.

`cpu_host_headroom_cores` has no flag of its own. Trust it only when `cpu_host_headroom_available`, `cpu_host_busy_cores_available` and `cpu_host_busy_ring_active` all read 1.


# Migration from Classic

This guide walks through migrating from UMH Classic to UMH Core. The key changes involve data contracts, configuration syntax, and architectural improvements.

## Migration Strategy

The recommended approach is **side-by-side deployment** to minimize risk:

1. **Deploy Core next to Classic** - Point a Bridge at the Classic UNS topics
2. **Cut over producers/consumers** gradually
3. **Shut down Classic** pods once data is verified

## Key Changes from Classic to Core

### Data Contract Migration

#### Core Change: `_historian` → `_raw`

**UMH Classic (Deprecated):**

```yaml
# Bridge configuration
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.data_contract = "_historian";

# Consumer patterns  
topics: ["umh.v1.+.+.+.+._historian.+"]
```

**UMH Core (Current):**

```yaml
# Bridge configuration - Start simple
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.data_contract = "_raw";

# Consumer patterns
input:
  uns:
    topics: ["umh.v1.+.+.+.+._raw.+"]        # For simple sensor data
    topics: ["umh.v1.+.+.+.+._temperature.+"]  # For structured temperature data
```

### Configuration Changes

#### Bridge Configuration

**UMH Classic:**

```yaml
# Old benthos-style configuration
input:
  opcua: 
    # ... configuration
processors:
  - tag_processor:
      defaults: |
        msg.meta.data_contract = "_historian";
output:
  kafka: {}
```

**UMH Core:**

```yaml
# New protocolConverter (Bridge) configuration
protocolConverter:
  - name: device-bridge
    desiredState: active
    protocolConverterServiceConfig:
      location:
        2: "production-line"
        3: "device-name"
      config:
        dataflowcomponent_read:
          benthos:
            input:
              opcua: 
                # ... configuration
            pipeline:
              processors:
                - tag_processor:
                    defaults: |
                      msg.meta.data_contract = "_raw";
            output:
              uns: {}  # Always use UNS output
```

#### Key Differences

| Aspect                   | UMH Classic                 | UMH Core                                                                          |
| ------------------------ | --------------------------- | --------------------------------------------------------------------------------- |
| **Data Contracts**       | `_historian` only           | `_raw` + explicit contracts                                                       |
| **Configuration**        | Direct Benthos config       | Embedded within Bridges/Flows                                                     |
| **Output**               | Direct Kafka                | UNS output (abstracts Kafka)                                                      |
| **Location**             | Manual topic construction   | Automatic hierarchical path construction (supports ISA-95, KKS, or custom naming) |
| **Data Modeling**        | Single payload format       | Structured models + contracts                                                     |
| **Database Integration** | Automatic with `_historian` | Manual with specific contracts                                                    |

## Important: No Default Database Integration

**⚠️ Breaking Change**: UMH Core has **no default data contract** that automatically writes to TimescaleDB.

**UMH Classic:**

```yaml
# _historian automatically integrated with TimescaleDB
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.data_contract = "_historian";  # Auto-saved to database
```

**UMH Core:**

```yaml
# _raw is for simple data only - NO automatic database integration
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.data_contract = "_raw";  # NOT saved to database automatically

# Database integration requires explicit configuration
dataFlow:
  - name: timescale-sink
    dataFlowComponentConfig:
      benthos:
        input:
          uns:
            topics: ["umh.v1.+.+.+.+._raw.+"]
        pipeline:
          processors:
            - mapping: |
                # Manual transformation for database
                root.timestamp = this.timestamp_ms.ts_unix_milli()
                root.value = this.value
                # Extract location from topic
                let parts = metadata("umh_topic").split(".")
                root.enterprise = parts.1
                root.site = parts.2
                root.tag_name = parts.7
        output:
          sql_insert:
            driver: "postgres"
            dsn: "postgres://user:pass@timescale:5432/manufacturing"
            table: "sensor_readings"
```

**Migration Impact:**

* `_raw` data contracts do **NOT** automatically save to TimescaleDB
* You must create explicit database sink flows for any data you want persisted
* This provides more control but requires manual configuration

## Exporting Benthos Configurations

Export your existing Benthos configs from Classic:

```bash
kubectl get configmap <benthos-config-name> -o yaml > classic-benthos-config.yaml
```

Extract the Benthos pipeline section and paste it into Core's `config.yaml → dataFlow:` section.

## What to Migrate

| Component              | Classic Location | Core Location            | Notes                                 |
| ---------------------- | ---------------- | ------------------------ | ------------------------------------- |
| **Benthos pipelines**  | Individual pods  | `config.yaml → dataFlow` | Export configs from ConfigMaps        |
| **Node-RED flows**     | Node-RED pod     | External container       | Run separately, connect via MQTT/HTTP |
| **Grafana dashboards** | Bundled Grafana  | External Grafana         | Point at TimescaleDB                  |
| **TimescaleDB**        | TimescaleDB pod  | External database        | Use Bridge to forward data            |

### 3. Verify Migration

1. **Check topic creation**: Ensure new `_raw` topics are being created
2. **Validate consumers**: Confirm all consumers are receiving data from `_raw` topics
3. **Monitor metrics**: Watch for any data loss or processing errors
4. **Test integrations**: Verify external systems still receive expected data

## Side-by-Side Deployment

Run Core alongside Classic during transition:

```yaml
# Core Bridge reading from Classic UNS
protocolConverter:
  - name: classic-bridge
    desiredState: active
    protocolConverterServiceConfig:
      config:
        dataflowcomponent_read:
          benthos:
            input:
              kafka:
                addresses: ["classic-kafka:9092"]
                topics: ["umh.v1.+.+.+.+._historian.+"]
            pipeline:
              processors:
                - mapping: |
                    # Convert _historian to _raw
                    root = this
                    meta.data_contract = "_raw"
            output:
              uns: {}
```

## Getting Help

* **Migration Issues**: See [Troubleshooting Guide](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/production/troubleshooting.md)
* **Configuration Questions**: Check [Configuration Reference](/reference/configuration-reference)
* **Community Support**: Visit [UMH Community Forum](https://community.umh.app)

## Next Steps

After completing the basic migration:

1. [**Data Modeling**](/usage/data-modeling) - Implement structured data contracts
2. [**Stream Processors**](/usage/data-flows/stream-processor) - Transform raw data to business models
3. [**Production Deployment**](/production) - Scale and secure your UMH Core deployment

### Tag Groups Migration

**UMH Classic (Tag Groups in Payload):**

```yaml
# Classic bridge configuration
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name = "head";  # Groups appear inside payload

# Classic payload with tag groups
{
  "timestamp_ms": 1670001234567,
  "pos": { 
    "x": 12.5,
    "y": 7.3,
    "z": 3.2
  },  
  "temperature": 50.0,
  "collision": false
}

# Classic database storage
# head$pos$x, head$pos$y, head$pos$z, head$temperature, head$collision
```

**UMH Core (Virtual Paths in Topics):**

```yaml
# Core data model with virtual paths
datamodels:
  - name: CNCHead
    version: v1
    structure:
      pos:
        x: { type: timeseries }
        y: { type: timeseries }
        z: { type: timeseries }
      temperature: { type: timeseries }
      collision: { type: timeseries }

# Core stream processor
streamprocessors:
  - name: cnc_head_sp
    contract: _cnc_head:v1
    sources:
      x_raw: "umh.v1.corp.plant.line.cnc._raw.head_pos_x"
      y_raw: "umh.v1.corp.plant.line.cnc._raw.head_pos_y"
      z_raw: "umh.v1.corp.plant.line.cnc._raw.head_pos_z"
      temp_raw: "umh.v1.corp.plant.line.cnc._raw.head_temperature"
      collision_raw: "umh.v1.corp.plant.line.cnc._raw.head_collision"
    mapping:
      pos.x: "x_raw"
      pos.y: "y_raw"  
      pos.z: "z_raw"
      temperature: "temp_raw"
      collision: "collision_raw"

# Result: Individual topics with virtual paths
# umh.v1.corp.plant.line.cnc._cnc_head.pos.x
# umh.v1.corp.plant.line.cnc._cnc_head.pos.y
# umh.v1.corp.plant.line.cnc._cnc_head.pos.z
# umh.v1.corp.plant.line.cnc._cnc_head.temperature
# umh.v1.corp.plant.line.cnc._cnc_head.collision
```

**Migration Benefits:**

* **Granular subscriptions**: Subscribe to just `pos.*` for positioning data
* **Independent scaling**: Different retention for different sensor groups
* **Type safety**: Each topic has explicit schema validation
* **Cross-site consistency**: Same virtual path structure across all CNCs

**Migration Steps:**

1. **Identify tag groups** in Classic payloads (nested JSON objects)
2. **Create data models** with folder structures matching groups
3. **Define stream processors** to map raw inputs to virtual paths
4. **Update consumers** to subscribe to specific virtual path patterns


# Security

This directory contains security documentation for the United Manufacturing Hub platform.

## Component Scope

The UMH platform consists of two main security domains:

### umh-core (Edge Gateway Container)

**Documentation**: `umh-core/deployment-security.md`

Security scope:

* Instance-level authentication (AUTH\_TOKEN)
* Container security (non-root execution, process isolation)
* Edge gateway security architecture
* Protocol converter and data flow security
* Network security for edge deployment
* Supply chain security (vulnerability scanning, dependencies)
* Industrial protocol handling (OPC UA, Modbus, S7, MQTT)

### ManagementConsole (Cloud Platform)

**Documentation**: [Management Console security](/management-console/security)

Security scope:

* User authentication and multi-factor authentication (MFA)
* Role-based access control (RBAC) for users
* User-level audit trails and action logging
* Cloud security and API protection
* Session management and user permissions
* Organization and team access controls

## Security Responsibility Boundary

**umh-core** handles edge security - authenticating the instance, securing the container, and protecting data flows at the factory edge.

**ManagementConsole** handles user security - authenticating users, controlling access, and securing the cloud platform.

Together they provide defense-in-depth: instance authentication (umh-core) + user authentication (ManagementConsole) + customer infrastructure security.


# umh-core Security

## Shared Responsibility Model

### We are responsible for:

* **Software supply chain** (container images, SBOM, vulnerability scanning via Aikido/FOSSA)
* **Secure defaults** (non-root execution, TLS enabled by default, no default passwords)
* **Clear documentation** of protocol limitations and security considerations
* **Regular security updates** via our Docker registry and documented release process

### You are responsible for:

* **Infrastructure and runtime** (Docker/Kubernetes configuration, host OS security, network architecture)
* **Secrets lifecycle** (AUTH\_TOKEN storage, rotation, access controls)
* **Volume permissions** (ensuring `/data` is writable by UID 1000)
* **Monitoring and incident response** (log aggregation, security monitoring, forensics)
* **Deployment security** (capabilities, AppArmor/SELinux, resource limits, network policies)
* **Network segmentation and zone placement** (deploy at Purdue Level 3 per IEC 62443-3-3 zone architecture, firewall rules for OT/IT boundaries)
* **PLC polling rate configuration** (configure polling intervals per device specifications to prevent overload)
* **Physical security** (secure deployment locations and restrict physical access per IEC 62443-3-3 SR 5.1)
* **OT safety systems** (umh-core must not be integrated into safety-instrumented systems; see IEC 61508/61511)
* **Backup and disaster recovery** (configuration backups, persistent volume snapshots, tested restore procedures per business continuity requirements)
* **High availability**: use storage accessible from multiple nodes. See [High Availability](/production/high-availability).
* **Security event monitoring** (SIEM integration if required, intrusion detection systems)
* **Corporate CA certificate management** (adding certificates for TLS inspection scenarios)
* **Reading this documentation** - we provide secure software, you must deploy it securely

**This aligns with cloud vendor models** - we secure the software, you secure the deployment environment.

**For detailed OWASP/CIS compliance guidance**, see:

* [OWASP Docker Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html)
* [CIS Docker Benchmark](https://www.cisecurity.org/benchmark/docker)
* [Kubernetes Security Best Practices](https://kubernetes.io/docs/concepts/security/)

***

## Security Capabilities

This section documents umh-core's security features across container security, access control, network architecture, cryptography, supply chain integrity, and industrial protocol handling. Each capability is mapped to applicable industry standards (NIST, IEC 62443, OWASP) with implementation details and known limitations.

## Container Security

**Applicable Standards**: OWASP Docker Security Cheat Sheet requirement 2 (non-root user), NIST SP 800-190 (least privilege principle for containers), IEC 62443-4-2 CR 2.1 (authorization enforcement and least privilege at Security Level 2)

**Implementation**: All umh-core processes run as non-root user umhuser with UID 1000, preventing privilege escalation attacks. Each component operates in a separate process namespace isolated from the host, with limited filesystem access restricted to explicitly mounted paths only. The minimal Alpine Linux base image reduces the attack surface by excluding unnecessary system utilities and libraries.

**Deployment Considerations**: All bridges and [data flows](/usage/data-flows) run as the same Linux user within the container, with no per-component user isolation. This design prioritizes non-root security over internal process isolation, as user switching requires root capabilities unavailable in non-root containers.

**Note**: Mounted volumes must be writable by UID 1000. See [Filesystem Access](#filesystem-access).

⚠️ **Safety System Warning**: umh-core is NOT SIL-rated and must not be used in safety-instrumented systems. Refer to IEC 61508/61511 for safety requirements.

***

## Logging and Audit Trail

**Applicable Standards**: NIST SP 800-53 AU-2 and AU-3 (comprehensive audit logging with timestamps, event types, and outcomes), NIST SP 800-92 (log generation, storage, and protection), IEC 62443-4-2 CR 2.8 (auditable events for security-relevant actions)

**Implementation**: All services write structured logs to /data/logs/ with S6 supervision, with an ISO 8601 timestamp on every line. FSM state transitions are tracked for all components, with rolling log rotation to manage storage. Logs capture configuration changes, component lifecycle events, and connection status for all industrial protocols and data flows.

**Deployment Considerations**: Current logging tracks system events and component states but does not capture individual user actions performed through the ManagementConsole interface. User-level audit trails for configuration changes are planned for future releases.

***

## Access Control and Authentication

**Applicable Standards**: NIST SP 800-53 AC-6 (least privilege access control), IEC 62443-4-2 CR 1.2 (software process and device identification and authentication), NIST SP 800-171 IA-2 (identification and authentication of organizational users)

**Implementation**: Each umh-core instance authenticates to the ManagementConsole using a unique AUTH\_TOKEN shared secret combined with a per-instance InstanceUUID. The AUTH\_TOKEN is user-configured with no default values, and ManagementConsole provides the user authentication layer for operator access. TLS encryption protects authentication tokens during transmission to management.umh.app.

**Deployment Considerations**: The current AUTH\_TOKEN is instance-level and shared across all components within the container. Future releases will implement per-message authentication to enable fine-grained access control and authorization.

***

## Network Security

**Applicable Standards**: NIST SP 800-82 (defense in depth for OT/ICS environments and network segmentation), IEC 62443-3-3 SR 5.1 (zone and conduit architecture for edge gateway deployment), IEC 62443-4-2 CR 5.1 (network segmentation support)

**Implementation**: umh-core follows an edge-only architecture designed for deployment at the OT/IT boundary (Purdue Level 3 (DMZ between OT and IT networks)), with outbound HTTPS connections only to management.umh.app for configuration synchronization. Typical deployment: umh-core in DMZ, outbound connections to cloud, inbound connections from OT networks only. No direct internet exposure. By default, no services listen for inbound internet connections. However, customers can configure data flows with HTTP server inputs that expose ports within the container. Network isolation is enforced through separate network namespaces unless explicitly configured otherwise.

Deploy umh-core in a DMZ with firewalls on both OT and IT boundaries. See IEC 62443-3-3 for zone architecture.

**Customer-Configured HTTP Servers**: Customer-configured HTTP servers require coordination between application teams (who configure bridges via ManagementConsole) and security/infrastructure teams (who must explicitly expose container ports). This two-team requirement ensures managed attack surface. umh-core does not provide built-in authentication for HTTP inputs. Deploy behind your standard reverse proxy (nginx, HAProxy) for authentication and TLS.

**Deployment Considerations**: The edge-only design requires outbound internet access to function and is not suitable for air-gapped or fully disconnected environments. Network segmentation and firewall rules must be configured properly by the customer to achieve defense in depth.

***

## Cryptography and TLS

**Applicable Standards**: NIST SP 800-52 (TLS 1.2+ configuration with modern cipher suites), NIST SP 800-53 SC-8 (transmission confidentiality and integrity), IEC 62443-4-2 CR 4.3 (use of cryptography conforming to applicable standards and regulations)

**Implementation**: All connections to management.umh.app use TLS 1.2 or higher with modern cipher suites including AES-GCM and ChaCha20-Poly1305. Certificate validation is enabled by default using Go's standard library crypto packages, and all cryptographic operations follow current industry standards for secure communications. This is the secure default configuration.

**Deployment Considerations**: Certificate validation is always enforced; there is no configuration option to disable it, and the minimum TLS version cannot be lowered. Corporate environments with TLS inspection should add their corporate CA certificate to the container's trusted certificates instead. See [Network Configuration](/production/security/deployment-security/network-configuration) for details.

***

## Supply Chain Security

**Applicable Standards**: OWASP Docker Security Cheat Sheet requirement 0 (regular image updates and vulnerability scanning), NIST SP 800-161 (SBOM generation, vulnerability management, and supply chain risk controls), IEC 62443-4-1 SR-5 (product defect management and security vulnerability tracking)

**Implementation**: All container images undergo automated vulnerability scanning with Aikido. SBOM and OSS License Compliance is tracked through FOSSA. The compliance dashboard at trust.umh.app tracks standards alignment (such as ISO27001 or NIST) through Vanta.

**Deployment Considerations**: Supply chain security depends on timely updates from upstream dependencies and proper image verification during deployment.

***

## Industrial Protocol Security

**Applicable Standards**: OWASP OT Top 10 number 9 (legacy protocol security limitations and risks), IEC 62443-3-3 SR 5.1 (network segmentation for operational technology protocols), IEC 62443-4-2 CR 3.1 (communication integrity requirements)

**Implementation**: umh-core supports industrial protocols including OPC UA (with certificate-based security), MQTT (with TLS support), Modbus TCP, and S7 communication. Connections are managed through protocol-specific bridges with configurable security parameters where the underlying protocol provides security features.

**Deployment Considerations**: Modbus TCP and S7 protocols lack native encryption capabilities due to inherent protocol design limitations. Compensating controls include network segmentation to isolate OT zones from IT and internet networks, physical security requirements for deployment locations, and firewall rules to restrict protocol access by IP address. These compensating controls must be implemented by the customer as part of their overall IEC 62443-3-3 zone architecture.

***

## PLC Integration Limits

**Warning**: Aggressive polling can overwhelm PLCs and cause CPU overload or crashes. Each PLC model has different limits for concurrent connections, polling rates, and tag counts. Consult your device manual for specifications and test polling rates in non-production before deployment. S7 protocol is particularly sensitive to connection overload.

***

## Threat Model (Simplified)

umh-core primarily protects against unintentional compromise of external industrial systems due to vulnerabilities in our software. The non-root execution model prevents privilege escalation, the minimal network attack surface reduces exposure, and TLS is enabled by default for all external communications. Supply chain risks are mitigated through vulnerability scanning and dependency tracking. The pull-based deployment model (see Deployment Model below) prevents misconfiguration that could expose industrial protocols to the internet.

umh-core does not protect against malicious operators or accidental misconfiguration. An operator with access to the ManagementConsole UI or direct filesystem access to config.yaml can deploy bridge configurations that connect to external industrial systems, exfiltrate the AUTH\_TOKEN via outbound network requests, or read sensitive data from mounted volumes. Accidental misconfiguration poses similar risks - operators copying configuration snippets from the internet (blog posts, forums, examples) may inadvertently expose ports, enable insecure settings, or introduce vulnerable patterns without understanding the security implications. Similarly, the system cannot protect against compromise of the underlying container runtime, host operating system, or Kubernetes control plane.

This model aligns with industry-standard edge gateway security - we secure our software, you secure your infrastructure.

***

## Deployment Model: Edge-Only Architecture

umh-core is designed for edge-only deployment with a specific network architecture. The system requires outbound HTTPS connections to management.umh.app for configuration synchronization and supports outbound connections to data sources including MQTT brokers, OPC UA servers, Modbus devices, and APIs. No services are designed for inbound internet connections.

The typical deployment location is on the factory floor, behind corporate firewall, on-premises at customer sites. This architecture reduces the attack surface by eliminating services that listen for inbound internet connections. The ManagementConsole queues configuration changes that umh-core instances pull on their own schedule for execution. This aligns with network segmentation best practices where umh-core sits between OT networks and IT infrastructure at the boundary layer.

The system is not designed for air-gapped environments. umh-core requires outbound internet access to management.umh.app to function and cannot operate in fully disconnected deployments.

***

## What umh-core Accesses and Why

### Filesystem Access

**Required**: `/data` directory (persistent storage)

* Configuration files (config.yaml with AUTH\_TOKEN)
* Logs (rolling logs for all services)
* Redpanda data (message broker storage)

**Optional**: Customer-defined mounts for file-based inputs

* CSV/JSON/XML data files from network shares
* Log files from other systems
* Production reports from local disks

**Why**: Bridges need to read data files and persist configuration across container restarts.

**Access pattern**: Access depends on mount configuration (e.g., `:ro` for read-only, `:rw` for read-write).

#### Volume Permission Requirements

The `/data` directory must be writable by UID 1000 (umhuser). Set ownership before starting:

```bash
sudo chown -R 1000:1000 /path/to/data
```

***

### Network Access

**Required outbound**: HTTPS to `management.umh.app` for configuration sync and status reporting.

See [Network Configuration](/production/security/deployment-security/network-configuration) for details on proxy settings and TLS inspection.

**Data source connections**: Customer-defined connections to industrial devices and data sources.

**Inbound**: No services designed for internet exposure (edge-only deployment).

***

### Process Model

All components run as single non-root user with UID 1000 named umhuser. This includes the main umh-core agent, all bridges implemented as benthos-umh instances, all data flows, the Redpanda broker, and internal services.

See [Container Security](#container-security) above for details on non-root execution model, process isolation, and filesystem access restrictions.

***

## Deployment Considerations

### AUTH\_TOKEN in Environment Variable

**Category**: Deployment Consideration (cannot fix in single-container architecture)

**Issue**: AUTH\_TOKEN shared secret flow:

* **Input**: Environment variable (`AUTH_TOKEN=xxx`) during first container start
* **Persistent storage**: Written to `/data/config.yaml` automatically
* **Subsequent starts**: Read from config.yaml (environment variable no longer required)

**Why this design**: Ensures configuration persists across container restarts without requiring environment variables every time. Once set via environment variable or ManagementConsole, AUTH\_TOKEN is stored in `/data/config.yaml` on the persistent volume.

**Security implication**: Both storage locations are readable by all processes running as umhuser with UID 1000. This includes all bridges handling protocol converters, data flows, and stream processors, any process started within the container, and any code executed via bridge configurations.

**Risk**: Malicious bridge configuration could exfiltrate AUTH\_TOKEN via outbound network requests.

**What you should do**:

1. **Understand the security model**: This follows standard container secret management patterns used by Docker and Kubernetes. All processes within the container share the secret, which is industry-standard for single-container architectures. For enhanced secret isolation, enterprise deployments can integrate with HashiCorp Vault or similar secret management systems.
2. **Monitor** network connections for unexpected outbound traffic (exfiltration attempts)
3. **Rotate if compromised**:
   * Create new instance in ManagementConsole
   * Copy new AUTH\_TOKEN to deployment configuration
   * Update container environment variable or config.yaml
   * Remove old instance from ManagementConsole

***

### No User/Process Isolation Between Bridges

**Category**: Accepted Risk (design trade-off)

**What this means**: All bridges (protocol converters, data flows, stream processors) run as the same Linux user (UID 1000, umhuser). There is no user-level or process-level isolation between different bridges within the container.

### No Per-User Access Control Within Instance

**Category**: Deployment Consideration (planned for future releases)

**Issue**: AUTH\_TOKEN authorizes the umh-core instance to communicate with ManagementConsole. Once authorized, any user within that organization can perform all actions on this instance through the ManagementConsole UI. Per-user and per-action access restrictions at the instance level are not currently implemented but are planned for future releases.

### Logs Accessible to All Processes

**Category**: Deployment Consideration (non-root container design)

**Issue**: All processes running as umhuser (UID 1000) can read and potentially modify log files in /data/logs/. This is a consequence of the non-root container design. Customers requiring tamper-proof logs should implement log forwarding to external SIEM systems with append-only storage.

**What this does not mean**: Bridges do not interfere with each other's data processing, as Redpanda isolates message flows by topic. Bridges are not resource-limited together; each bridge can have separate CPU and memory limits configured via s6-softlimit. Data is not shared between bridges, as each benthos instance maintains separate configuration and state.

**Technical constraint**: Per-bridge user isolation is not possible in non-root containers. Process-level user switching requires CAP\_SETUID and CAP\_SETGID capabilities, which are only available to root processes.

**Design decision**: Non-root container security prioritized over per-bridge user isolation.

**Security implications**:

Because all bridges run as the same user, they share access within the container. All bridges can read the /data/config.yaml file containing AUTH\_TOKEN, access the same mounted directories, view each other's environment variables, and read each other's configuration files.

The container boundary remains enforced despite shared user access. Bridges cannot access the host filesystem except through explicitly mounted paths, cannot see host processes, and network isolation applies unless explicitly disabled with host networking mode.

Non-root execution provides security benefits that justify this trade-off. The container cannot escalate to root privileges even if a bridge is compromised. This follows the standard Docker security model with defense in depth. The design is compatible with restricted Kubernetes environments that do not permit privileged containers or special permissions.

***

## Security Best Practices

### What to Mount

**Required**: `/data` for persistent storage (configs, logs, certificates)

**Common additional mounts**:

* Data files: Network shares with CSV/JSON/XML production data
* Log files: From PLCs or other systems
* Use read-only mounts when bridges only need to read

### What NOT to Mount

* Entire host filesystem (`/`)
* Docker socket (`/var/run/docker.sock`)
* Host `/etc` or `/var` directories

***

## Enterprise Security Features

The following security capabilities and documentation require an enterprise license:

* Security testing reports and penetration test results
* Software Bill of Materials (SBOM) access and vulnerability disclosure timelines
* Incident response playbooks and operational runbooks
* End-of-life policy, extended support, and long-term security update commitments
* Service Level Agreements (SLAs) for security patch response times
* Compliance attestations and audit support documentation

Enterprise license provides compliance documentation and SLA-backed support, not additional security features.

For enterprise licensing information, contact the United Manufacturing Hub sales team.


# Network Configuration

Network requirements and configuration for umh-core edge deployments.

## Outbound Connections

umh-core requires outbound connectivity to the Management Console:

| Destination          | Protocol | Port | Purpose                                                |
| -------------------- | -------- | ---- | ------------------------------------------------------ |
| `management.umh.app` | HTTPS    | 443  | Configuration sync, status reporting, action retrieval |

No inbound connections are required from the internet.

## Corporate Firewall Configuration

### TLS Inspection (MITM)

If your corporate firewall performs TLS inspection, you may see certificate errors. Add your corporate CA certificate to the container's trusted certificates to resolve this.

## Proxy Configuration

If your network requires a proxy:

```bash
docker run \
  -e HTTP_PROXY=http://proxy.company.com:8080 \
  -e HTTPS_PROXY=https://proxy.company.com:8080 \
  -e NO_PROXY=localhost,127.0.0.1,.local \
  management.umh.app/oci/united-manufacturing-hub/umh-core:<VERSION>
```

Supported environment variables: `HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY` (and their lowercase variants).

### Authenticated Proxies

Include credentials in the proxy URL:

```bash
-e HTTP_PROXY=http://username:password@proxy.company.com:8080
```

Supported proxy types: HTTP and HTTPS.

## Common Configuration

In most corporate environments, proxy usage and TLS inspection go together. If you need to configure a proxy, you'll likely also need to add your corporate CA certificate to handle TLS inspection.


# Reference

This section provides comprehensive reference documentation for the United Manufacturing Hub (UMH) Core components. Here you'll find detailed technical specifications, configuration options, and structural documentation to help you implement, configure, and troubleshoot your UMH deployment.

The reference materials include:

* **Configuration Reference**: Complete documentation of the central configuration file (`/data/config.yaml`), including all available fields, data types, and examples
* **Container Layout**: Detailed overview of the container's file system structure, data organization, and persistent volume layout
* **Template Variables**: Comprehensive glossary of all available template variables for protocol converter configurations, including connection, location, and system variables
* **State Machines**: Reference documentation for state machine implementations and workflows
* **HTTP API**: GraphQL API reference for Topic Browser queries and programmatic access

Navigate through these pages to find specific configuration parameters, understand system architecture, or look up technical details for your UMH implementation.


# Configuration Reference

This is the reference for the central config `/data/config.yaml`

> **File location:** The container mounts `/data` as a writable volume.\
> **Hot-reload:** The Agent polls the file every tick; valid changes are applied automatically.\
> **UI Terminology:** In the Management Console UI, `protocolConverter:` is shown as "Bridges" and `dataFlow:` is shown as "Stand-alone Flows"

## YAML vs UI: Choose Your Workflow

UMH Core supports both **direct YAML editing** and **Management Console UI** for configuration. Each approach has distinct advantages:

### YAML Configuration Benefits

* **Version Control**: Every change is tracked with Git — see who changed what, when, and why
* **GitOps Integration**: Automate deployments, rollbacks, and multi-environment promotion
* **AI/LLM Integration**: Tools like Cursor and ChatGPT can generate, modify, and optimize configurations
* **Templating Power**: Create and connect hundreds of machines with a few keystrokes using YAML anchors
* **IDE Support**: Syntax highlighting, validation, autocomplete, and refactoring tools

### UI Configuration Benefits

* **Zero Code**: Any OT person or business user can click and configure without YAML knowledge
* **Input Validation**: Real-time validation prevents configuration errors before deployment
* **Visual Guidance**: Wizards and forms guide users through complex protocol configurations
* **Live Preview**: See the impact of changes before applying them

### The Best of Both Worlds

**UMH Core provides two-way synchronization:** changes made in the UI are reflected in the YAML file, and YAML edits are shown in the UI. This means:

* **OT teams** can use the UI for day-to-day operations
* **IT teams** can use YAML for infrastructure-as-code workflows
* **Both approaches** maintain full version control and auditability
* **Scaling operations** benefit from templating while **individual adjustments** benefit from UI convenience

> **Pro Tip**: Start with the UI to understand the configuration structure, then switch to YAML for advanced templating and automation.

## Terminology Reference

| Current Term (UMH Core) | Legacy Term (UMH Classic) | YAML Key             | Description                                             |
| ----------------------- | ------------------------- | -------------------- | ------------------------------------------------------- |
| Bridge                  | Protocol Converter        | `protocolConverter:` | Connects external devices to UNS with health monitoring |
| Stand-alone Flow        | Data Flow Component (DFC) | `dataFlow:`          | Point-to-point data processing pipelines                |
| Stream Processor        | Stream Processor          | `dataFlow:`          | Processes data within UNS (upcoming feature)            |

> **Note:** YAML configuration keys retain legacy names for backward compatibility. The Management Console UI uses the current terminology.

### Agent - Runtime & Device identity

| Field                           | Type               | Default                                                    | Purpose                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| ------------------------------- | ------------------ | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `metricsPort`                   | `int`              | **9102**                                                   | Exposes Prometheus metrics for the container.                                                                                                                                                                                                                                                                                                                                                                                                |
| `location`                      | map `int → string` | –                                                          | Hierarchical location path (level0-4+) that identifies this instance. **Level 0 (enterprise) is mandatory**. Can follow ISA-95, KKS, or any organizational naming standard.                                                                                                                                                                                                                                                                  |
| `communicator.apiUrl`           | `string`           | – (console-managed)                                        | HTTPS endpoint of the Management Console.                                                                                                                                                                                                                                                                                                                                                                                                    |
| `communicator.authToken`        | `string`           | –                                                          | API Key issued by the console. Can be set via `AUTH_TOKEN` env-var.                                                                                                                                                                                                                                                                                                                                                                          |
| `communicator.allowInsecureTLS` | `bool`             | `false`                                                    | Accepted and ignored. Certificate verification is always enforced. Leaving this key in place is harmless.                                                                                                                                                                                                                                                                                                                                    |
| `enableResourceLimitBlocking`   | `bool`             | `true`, also when config.yaml does not contain the setting | Whether bridge admission is on. While it is `true`, a new bridge waits when a resource is degraded, when resource health is not proven yet, or when the bridge limit is reached. `false` is the emergency setting *Turn off bridge admission*: new bridges start anyway, and the instance shows as degraded while it is `false`. See [Bridges Do Not Start](/usage/data-flows/bridges/bridges-do-not-start). Takes effect without a restart. |

**Location levels**

| Index | Generic Level | ISA-95 Example     | KKS Example     | Other Examples  |
| ----- | ------------- | ------------------ | --------------- | --------------- |
| `0`   | Enterprise    | `enterprise`       | `powerplant`    | `acme-inc`      |
| `1`   | Site/Region   | `site`/`plant`     | `unit-group`    | `cologne-plant` |
| `2`   | Area/Zone     | `area`/`line`      | `unit`          | `cnc-line`      |
| `3+`  | Work Cell+    | `work-cell`, `plc` | `component-grp` | `plc123`        |

```yaml
agent:
  metricsPort: 9102
  communicator:
    apiUrl: "https://api.management.umh.app"
    authToken: "${AUTH_TOKEN}"
  location:
    0: acme-inc
    1: plant1
    2: press-shop
    
```

### DataFlow - Stand-alone Flows

A list of all [stand-alone flows](/usage/data-flows/stand-alone-flow) (UI: "Stand-alone Flows", YAML: `dataFlow:`). Each entry spins up one Benthos-UMH instance.

| Field                                 | Type                  | Required | Description                                                                                                                                                                                                                                                                                                                            |
| ------------------------------------- | --------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                                | `string`              | ✔        | Unique within the file.                                                                                                                                                                                                                                                                                                                |
| `desiredState`                        | `active` \| `stopped` | ✔        | Agent will converge to this state.                                                                                                                                                                                                                                                                                                     |
| `dataFlowComponentConfig.debug_level` | `bool`                | ✗        | Enable verbose logging. When `true`, enables DEBUG log level and detailed operational logging (e.g., input/output operations, processing steps, data transformations). When `false` (default), uses INFO log level. **Use only for troubleshooting** - debug logging generates significantly more log data and may impact performance. |
| `dataFlowComponentConfig.benthos`     | object                | ✔        | Inline Benthos config (inputs, pipeline, outputs).                                                                                                                                                                                                                                                                                     |

```yaml
dataFlow:
- name: opcua-to-uns
  desiredState: active
  dataFlowComponentConfig:
    debug_level: false  # Enable detailed logging for troubleshooting
    benthos:
      input:
        opcua:
          endpoint:  "opc.tcp://192.168.0.50:4840"
          nodeIDs:   ["ns=2;s=FolderNode"]
      pipeline:
        processors:
          - tag_processor:
              defaults: |
                msg.meta.location_path = "acme.plant1.press-shop.plc1"
                msg.meta.data_contract = "_raw"
                msg.meta.tag_name      = "value"
                return msg;
      output:
        uns: {}           # write into embedded Redpanda
```

**Quick field map**

| Path                  | Notes                                                                          |
| --------------------- | ------------------------------------------------------------------------------ |
| `input.*`             | Any supported Benthos-UMH input (OPC UA, S7comm, Modbus, MQTT, Kafka, …).      |
| `pipeline.processors` | Standard Benthos processors **plus** `tag_processor` & `nodered_js`.           |
| `output.*`            | Usually `uns: {}` for Unified Namespace. MQTT, HTTP, SQL, etc. also available. |

*For complete input/output syntax see* [*Benthos-UMH Documentation*](https://docs.umh.app/benthos-umh)

### Bridge – Bridge from Device to UNS

> 🚧 **Roadmap Item**: Bridges are under active development. Current functionality includes connection monitoring and basic read/write flows.

A [**bridge**](/usage/data-flows/bridges) (UI: "Bridges", YAML: `protocolConverter:`) ingests data from a field device (e.g. OPC UA server, Siemens S7, Modbus controller) and pushes it into the Unified Namespace (UNS), and vice versa.\\

It combines a **connection probe** and two **stand-alone data flows (one for reading and one for writing)** under a single name and lifecycle.

```yaml
protocolConverter:
- name: press-opcua
  desiredState: active
  protocolConverterServiceConfig:
    debug_level: false  # Enable detailed logging for troubleshooting
    location:
      2: press1
    config:
      connection:
        nmap:
          target: "{{ .IP }}"
          port: "{{ .PORT }}"
      dataflowcomponent_read:
        benthos:
          input:
            opcua:
              endpoint: "opc.tcp://{{ .IP }}:{{ .PORT }}"
              nodeIDs: ["ns=2;s=MachineFolder"]
          pipeline:
            processors:
              - tag_processor:
                  defaults: |
                    msg.meta.location_path = "{{ .location_path }}"
                    msg.meta.data_contract = "_raw"
                    msg.meta.tag_name      = msg.meta.opcua_tag_name
                    return msg;
          output:
            uns: {}
    variables:
      # Optional templating support
      IP: "192.168.0.50"
      PORT: "4840"
```

#### Key fields

| Field                                        | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                                       | Unique ID for this converter instance                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `desiredState`                               | `active` to run immediately, or `stopped` to keep it defined but off.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| `protocolConverterServiceConfig.location`    | Appended to the global `agent.location`. Optional, but useful to identify per-machine data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `protocolConverterServiceConfig.debug_level` | Enable verbose logging for troubleshooting. When `true`, enables DEBUG log level and protocol-specific debugging (e.g., OPC UA packet inspection via `OPC_DEBUG=1` environment variable, detailed connection diagnostics). When `false` (default), uses INFO log level. **Use only for troubleshooting** - debug logging generates significantly more log data and may impact performance. Disable after resolving issues. **Note:** Same structure as stand-alone flows - `debug_level` is a direct child of the service config, NOT inside `config:` wrapper. |
| `template.connection.nmap`                   | TCP liveness check to decide if the device is reachable.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| `template.dataflowcomponent_read.benthos`    | Benthos pipeline to pull and forward data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `template.dataflowcomponent_write.benthos`   | Benthos pipeline to push and forward data.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `variables`                                  | Optional Go-template variables, referenced via `{{ .VARNAME }}` in the template. See [Template Variables Reference](/reference/variables#variables) for details.                                                                                                                                                                                                                                                                                                                                                                                                |

### Benthos-UMH Processors

UMH Core includes specialized processors for industrial data:

#### tag\_processor 🚧

> **🚧 Roadmap Item**: The current `tag_processor` implementation follows the benthos-umh pattern with tag names in payloads. With the next UMH Core release, `tag_processor` will be updated to align with the new data model where:
>
> * Tag names are only in topics (not in payloads)
> * Metadata is not included in message payloads
> * Output follows standard [timeseries payload format](/usage/unified-namespace/payload-formats)

For current implementation details, see [Benthos-UMH Tag Processor Documentation](https://docs.umh.app/benthos-umh/processing/tag-processor).

```yaml
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.location_path = "{{ .location_path }}";
          msg.meta.data_contract = "_raw";
          msg.meta.tag_name = "temperature";
          return msg;
```

**Data Contract Guidelines:**

* Use `_raw` for simple sensor data and initial device integration
* Use explicit contracts (e.g., `_temperature`, `_pump`) with [data models](/usage/data-modeling) for structured enterprise data
* **Migration from UMH Classic:** See [Migration from UMH Classic to UMH Core](/production/migration-from-classic) for `_historian` contract migration instructions

For detailed documentation, see [Benthos-UMH Tag Processor](https://docs.umh.app/benthos-umh/processing/tag-processor).

#### nodered\_js 🚧

Process data using Node-RED-style JavaScript:

```yaml
processors:
  - nodered_js:
      code: |
        msg.payload = msg.payload * 1.8 + 32;  // Convert C to F
        msg.topic = "temperature_fahrenheit";
        return msg;
```

For complete documentation, see [Benthos-UMH Node-RED JavaScript Processor](https://docs.umh.app/benthos-umh/processing/node-red-javascript-processor).

### Industrial Input Protocols

UMH Core supports 50+ industrial protocols via Benthos-UMH. For complete, up-to-date configuration examples, see [Benthos-UMH Input Documentation](https://docs.umh.app/benthos-umh/input/).

**Popular protocols include:**

* [OPC UA](https://docs.umh.app/benthos-umh/input/opc-ua-input) - Industry standard automation
* [Modbus](https://docs.umh.app/benthos-umh/input/modbus) - TCP/RTU serial communication
* [Siemens S7](https://docs.umh.app/benthos-umh/input/siemens-s7) - Direct PLC access
* [Ethernet/IP](https://docs.umh.app/benthos-umh/input/ethernet-ip) - Allen-Bradley devices

**UMH Core Integration Pattern:**

```yaml
protocolConverter:
  - name: device-bridge
    protocolConverterServiceConfig:
      config:
        dataflowcomponent_read:
          benthos:
            input:
              # ANY benthos-umh protocol input
            pipeline:
              processors:
                - tag_processor: { /* UNS metadata setup */ }
            output:
              uns: {}
```

### GraphQL API - Topic Browser 🚧

GraphQL API for querying Unified Namespace topics.

| Field         | Type       | Default | Description                               |
| ------------- | ---------- | ------- | ----------------------------------------- |
| `enabled`     | `bool`     | `false` | Enable GraphQL API (Early Access; opt in) |
| `port`        | `int`      | `8090`  | HTTP port for GraphQL endpoint            |
| `debug`       | `bool`     | `false` | Enable GraphiQL playground at `/`         |
| `corsOrigins` | `[]string` | `[]`    | CORS origins (empty = allow all)          |

```yaml
agent:
  graphql:
    enabled: false   # Default: false; set true to opt in (Early Access)
    port: 8090
    debug: false
    corsOrigins: 
      - "http://localhost:3000"
      - "https://your-app.com"
```

**Endpoints:**

* **GraphQL API:** `POST /graphql`
* **GraphiQL Playground:** `GET /` (debug mode only)
* **CORS Preflight:** `OPTIONS /graphql`

For complete API reference, see [Topic Browser GraphQL API](/reference/http-api/topic-browser-graphql).

### Internal - Built-In Services (expert)

UMH Core injects this section automatically.

```yaml
internal:
  redpanda:
    desiredState: active
    redpandaServiceConfig:
      topic:
        defaultTopicRetentionMs: 604800000   # 7 days
        defaultTopicCleanupPolicy: delete    # keep 7 days of raw records
      resources:
        maxCores: 1
        memoryPerCoreInBytes: 2147483648
  topicbrowser:
    desiredState: active  # Topic Browser service (auto-enabled)
```

> **Do not edit** unless instructed by UMH support; invalid settings can brick the stack.

#### Topic retention on new instances

On its very first boot — when there is no `config.yaml` in `/data` yet — an instance writes `defaultTopicCleanupPolicy: delete` and `defaultTopicRetentionMs: 604800000` into the file it creates. Together these keep seven days of raw records in `umh.messages`, so a new consumer group can replay the full history; the Historian and any other late-joining consumer depend on this.

Two consequences to be aware of:

* **Disk usage scales with throughput, not tag count.** Seven days of raw records is far more than a compacted snapshot. See the [Sizing Guide](/production/sizing-guide) before running a high-throughput instance, and cap the topic with `defaultTopicRetentionBytes` if disk is tighter than the retention window.
* **The Topic Browser tree is rebuilt from the retained window.** After a restart, a tag that has not published for more than seven days is missing from the tree until it publishes again.

Instances that already have a `config.yaml` keep the policy they run on today, which is `compact` unless it was set explicitly — upgrading never changes their retention. To move an existing instance to seven-day replay, set `defaultTopicCleanupPolicy: delete` yourself. Records that compaction already collapsed cannot be recovered.

### Validation & tips

* **YAML anchors** are supported – useful for re-using input or output snippets.
* The Agent logs schema errors and refuses to activate a malformed DFC.

## Related Documentation

* [**Bridges**](/usage/data-flows/bridges) - Device connectivity patterns
* [**Stand-alone Flows**](/usage/data-flows/stand-alone-flow) - Custom data processing
* [**Data Modeling**](/usage/data-modeling) - Structure your industrial data
* [**State Machines**](/reference/state-machines) - Component lifecycle management

## External References

* [**Benthos-UMH Documentation**](https://docs.umh.app/benthos-umh) - Complete protocol and processor reference
* [**Management Console**](https://management.umh.app) - Web-based configuration interface


# Data Model Type Definitions

> This is a reference document. Use it to look up the types that a data model field accepts and the errors the validator may raise.

A field's value type is declared at one of two levels. A data model field references a type with `_payloadshape` (a named payload shape) or `_refModel` (another data model). Inside a payload shape, each field declares a scalar directly with `_type`.

## Field Types (`_type`)

A [payload shape](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/reference/payload-shapes.md) field's `_type` is one of:

| `_type`   | Meaning                             | Example value |
| --------- | ----------------------------------- | ------------- |
| `string`  | Text                                | `"running"`   |
| `number`  | Any numeric value, decimal or whole | `42.5`, `42`  |
| `boolean` | True / false                        | `true`        |

Any other value is rejected: `unsupported UMH type: <type>`.

## Timeseries payload shapes are inherently typed

When defining the `_payloadshape` to be timeseries data, the name of the payload shape already states the type. As such, no additional type definition is needed.

| Payload shape        | `value` type | Use for                         |
| -------------------- | ------------ | ------------------------------- |
| `timeseries-number`  | `number`     | Numeric readings (temperature…) |
| `timeseries-string`  | `string`     | Text status values              |
| `timeseries-boolean` | `boolean`    | On/off, true/false state        |

Each built-in timeseries shape also carries `timestamp_ms` (a `number`) next to `value`. Both fields come from the built-in shape definition, so you never declare `timestamp_ms` yourself. See [Built-in Shapes](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/reference/payload-shapes.md#built-in-shapes) for the full payload structure.

A `_payloadshape` reference must name one of the three built-ins above or a shape defined under `payloadShapes:`. Any other name fails validation with `referenced payload shape '<name>' does not exist`.

```yaml
temperature:
  _payloadshape: timeseries-number
status:
  _payloadshape: timeseries-string
```

## Examples

Data model with several timeseries fields:

```yaml
dataModels:
  - name: temperature
    version:
      v1:
        structure:
          temperature:
            _payloadshape: timeseries-number
          unit:
            _payloadshape: timeseries-string
```

Nested folders, and a field that references another data model (`_refModel`):

```yaml
dataModels:
  - name: complex-model
    version:
      v1:
        structure:
          sensor: # folder here
            temp_reading:
              _payloadshape: timeseries-number
            temp_unit:
              _refModel:
                name: temperature
                version: v1
          metadata:
            _refModel:
              name: device-info
              version: v1
```

Multiple versions of one model:

```yaml
dataModels:
  - name: sensor-data
    version:
      v1:
        structure:
          value:
            _payloadshape: timeseries-number
      v2:
        structure:
          value:
            _payloadshape: timeseries-number
          status:
            _payloadshape: timeseries-string
```

Define a custom [payload shape](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/reference/payload-shapes.md) (top-level `payloadShapes:`), then reference it:

```yaml
payloadShapes:
  work-order:
    description: Work order record
    fields:
      orderId:
        _type: string
      quantity:
        _type: number
      price:
        _type: number
      active:
        _type: boolean

dataModels:
  - name: orders
    version:
      v1:
        structure:
          order:
            _payloadshape: work-order
```

Rules a data model field must follow:

* A leaf field references its type with `_payloadshape` or `_refModel`, never both.
* A folder (a field with subfields) has neither `_payloadshape` nor `_refModel`.

## Validation Errors

| Error message                                                      | Cause                                                              | Fix                                                             |
| ------------------------------------------------------------------ | ------------------------------------------------------------------ | --------------------------------------------------------------- |
| `referenced payload shape '<name>' does not exist`                 | `_payloadshape` names a shape that is not built-in and not defined | Use a built-in, or define the shape first                       |
| `unsupported UMH type: <type>`                                     | A field `_type` is not a supported type                            | Use `string`, `number`, or `boolean`                            |
| `field cannot have both _payloadshape and _refModel`               | A leaf field sets both keys                                        | Keep one                                                        |
| `leaf nodes must contain _payloadshape, _relational, or _refModel` | A leaf field has no value-type key                                 | Add `_payloadshape` or `_refModel`, or give the field subfields |
| `non-leaf nodes (folders) cannot have _payloadshape`               | A field with subfields also sets `_payloadshape`                   | Remove `_payloadshape` from the folder                          |

## Related

* [Data Modeling](/reference) - Concepts and the component chain
* [Payload Shapes](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/reference/payload-shapes.md) - Built-in and custom shapes
* [Data Models](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/reference/data-models.md) - Structure, `_refModel`, versions
* [Data Contracts](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/reference/data-contracts.md) - Enforcement at ingress
* [Payload Formats](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/docs/unified-namespace/payload-formats.md) - UNS payload structure

> ### Note: the `integer` type
>
> The YAML validator also accepts `_type: integer` inside a custom payload shape defined in an instance's config file. However, there is no built-in `timeseries-integer` shape and the data-model editor does not offer it, so most models never need it. Use `number` for numeric values. `integer` is only relevant when a custom payload shape must reject fractional values (for example a discrete count or an ID).


# Container Layout

```
/data
 ├─ config.yaml           # See also configuration reference
 ├─ config-backups/       # Timestamped config.yaml backups (last 100 retained)
 ├─ logs/                 # Rolling logs for agent, every data flow, Redpanda …
 ├─ redpanda/             # Redpanda data & WALs (backup-worthy)
 ├─ services/             # S6 service directories (only when S6_PERSIST_DIRECTORY=true)
 └─ hwid                  # Device fingerprint sent to the console

/tmp/umh-core-services/   # S6 service directories (default, cleared on restart)

/run/service/             # S6 scan directory (contains symlinks to service directories)
```

Mount **one persistent volume** (e.g. `umh-core-data`) to `/data` and you're done.

#### Advanced: Custom Data Location

If you need control over the exact data location (e.g., for compliance or backup requirements), you can use a custom folder instead of a Docker volume:

```bash
mkdir -p /path/to/umh-core-data
# Ensure container user has write access
chown -R 1000:1000 /path/to/umh-core-data
docker run -d --name umh-core -v /path/to/umh-core-data:/data:z ...
```

On SELinux systems (RHEL, Rocky), the `:z` flag allows Docker to relabel the directory. It's harmlessly ignored on other systems.

#### Upgrading Custom Folders to v0.44+

Version 0.44+ runs as a non-root user. If upgrading from an older version with a custom data folder, fix permissions first:

```bash
docker stop umh-core
docker rm umh-core

# Fix permissions for non-root container
sudo chown -R 1000:1000 /path/to/umh-core-data

docker run -d \
  --name umh-core \
  --restart unless-stopped \
  -v /path/to/umh-core-data:/data:z \
  management.umh.app/oci/united-manufacturing-hub/umh-core:<VERSION>
```

#### Migrating to Docker Volumes (Optional)

If you want to switch from a custom folder to a Docker volume:

```bash
docker stop umh-core
docker rm umh-core

# Create volume and copy data
docker volume create umh-core-data
docker run --rm \
  -v /path/to/umh-core-data:/source:ro,z \
  -v umh-core-data:/target \
  alpine sh -c "cp -av /source/. /target/"

# Fix permissions and start
docker run --rm -v umh-core-data:/data alpine chown -R 1000:1000 /data

docker run -d \
  --name umh-core \
  --restart unless-stopped \
  -v umh-core-data:/data \
  management.umh.app/oci/united-manufacturing-hub/umh-core:<VERSION>
```

Keep your old folder as backup for 24-48 hours before deleting.

### /config.yaml

See also [Configuration Reference](/reference/configuration-reference)

### /logs

Every process that S6 supervises writes to its own directory, `/data/logs/<service>/`. The agent writes to `/data/logs/umh-core/`. The read flow of a bridge writes to `/data/logs/benthos-dataflow-read-protocolconverter-<bridge name>/`. Each log line starts with an ISO 8601 timestamp in the container's local time.

| File/dir                                 | What it is for                                                                                                   | When it appears                                                                                                     |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **`current`**                            | The file S6 is *actively* appending log lines to. Keep an eye on this one with `tail -f`.                        | Always – it is recreated immediately after every rotation. ([Skarnet](https://skarnet.org/software/s6/s6-log.html)) |
| **`previous`**                           | A temporary name used *during* a rotation; disappears once rotation completes.                                   | Only while a rotation is in flight. ([Skarnet](https://skarnet.org/software/s6/s6-log.html))                        |
| **`@<timestamp>.s`**                     | An archived log that was rotated *cleanly*. The timestamp is the moment the rotation occurred, in TAI64N format. | After every successful rotation. ([Skarnet](https://skarnet.org/software/s6/s6-log.html))                           |
| **`@<timestamp>.u`**                     | A “unfinished” archive – it was the `current` file when the container was killed.                                | Only after an un-clean shutdown. ([Skarnet](https://skarnet.org/software/s6/s6-log.html))                           |
| `lock`, `state`, `processed`, `newstate` | Book-keeping files S6-log uses while rotating or while a post-processor runs.                                    | Internal – you normally ignore them. ([Skarnet](https://skarnet.org/software/s6/s6-log.html))                       |

#### The life-cycle in practice

1. **Normal running** – all services write to their own `current` file.
2. **Size hits the limit** – S6 atomically renames `current` to `@<timestamp>.s`, then immediately creates a fresh empty `current`. The agent's log rotates at 10 MB. Other services rotate at a size umh-core sets per service, or at s6-log's default of 99,999 bytes. ([Skarnet](https://skarnet.org/software/s6/s6-log.html))
3. **Prune** – S6 deletes the oldest archives. The agent keeps the newest 5, and every other service keeps the newest 20. ([Skarnet](https://skarnet.org/software/s6/s6-log.html))
4. **You read logs** – use:

   ```bash
   # live stream
   tail -f /data/logs/<service>/current

   # inspect an old file (the '@…s' ones are plain text)
   less /data/logs/<service>/@<timestamp>.s
   ```

### /redpanda

The Redpanda data directory.

### HWID

A unique identifier for that UMH Core installation. Useful for troubleshooting.

### S6 Service Directories

UMH Core uses S6 overlay for service supervision. Service directories contain the runtime state and configuration for each managed service (Benthos, Redpanda, monitors, etc.).

By default, these directories are created in `/tmp/umh-core-services/` which is **cleared on container restart**, ensuring a clean state. This prevents issues from stale supervisor state files.

For debugging purposes, you can set `S6_PERSIST_DIRECTORY=true` to use `/data/services/` instead, which persists across container restarts. This allows inspection of S6 supervisor state files when troubleshooting service startup issues.

The `/run/service/` directory contains symlinks pointing to the actual service directories, which S6's scanner monitors for changes.

See the [Environment Variables](/reference/environment-variables) reference for more details on `S6_PERSIST_DIRECTORY`.


# State Machines

**State machines are the core orchestration mechanism in UMH Core.** Every component is managed by a finite state machine (FSM) with clearly defined states and transitions. This provides predictable, observable behavior and enables reliable error handling and recovery.

## How It Works

UMH Core uses hierarchical state machines where components build upon each other:

* **Bridge** = Connection + Read Flow + Write Flow
* **Flow** = Benthos instance with lifecycle management
* **Benthos Flow** = Individual Benthos process with detailed startup phases
* **Connection** = Network probe service (typically nmap-based)

Each component inherits lifecycle states (`to_be_created`, `creating`, `removing`, `removed`) and adds operational states specific to its function. The Agent continuously reconciles desired vs actual state, triggering appropriate transitions based on observed conditions.

## 1 — Redpanda Service

| State           | What it means                                                                         | How it is entered                                     | How it leaves                                                                                                                                           |
| --------------- | ------------------------------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **stopped**     | `redpanda` process not running.                                                       | *stop\_done* from **stopping**, or initial create.    | *start* → **starting**                                                                                                                                  |
| *starting*      | S6 launching broker; health checks pending.                                           | *start* event from **stopped**.                       | <p><em>start\_done</em> → <strong>idle</strong><br><em>start\_failed</em> → <strong>stopped</strong></p>                                                |
| **idle**        | Broker healthy, **no data for 30 s** (default idle window).                           | *start\_done* or *no\_data\_timeout* from **active**. | <p><em>data\_received</em> → <strong>active</strong><br><em>degraded</em> → <strong>degraded</strong><br><em>stop</em> → <strong>stopping</strong></p>  |
| **active**      | Broker healthy & `BytesIn/OutPerSec` > 0.                                             | *data\_received* from **idle**.                       | <p><em>no\_data\_timeout</em> → <strong>idle</strong><br><em>degraded</em> → <strong>degraded</strong><br><em>stop</em> → <strong>stopping</strong></p> |
| ⚠️ **degraded** | Broker running but ≥1 health‑check failing (`disk-space-low`, `cpu-saturated`, etc.). | *degraded* from **idle/active**.                      | <p><em>recovered</em> → <strong>idle</strong><br><em>stop</em> → <strong>stopping</strong></p>                                                          |
| *stopping*      | Graceful shutdown (draining clients).                                                 | *stop* from any running state.                        | *stop\_done* → **stopped**                                                                                                                              |

***

## 2 — Container Monitor

| State                | Meaning                                    | Enter trigger                                                    | Exit trigger                                       |
| -------------------- | ------------------------------------------ | ---------------------------------------------------------------- | -------------------------------------------------- |
| **active**           | CPU < 85 %, RAM < 90 %, Disk < 90 %.       | *metrics\_all\_ok* after monitor start **or** from **degraded**. | *metrics\_not\_ok* → **degraded**                  |
| ⚠️ **degraded**      | One of the above limits breached for 15 s. | *metrics\_not\_ok*                                               | *metrics\_all\_ok* → **active**                    |
| monitoring\_stopped  | Watchdog disabled.                         | *stop\_monitoring\_done*                                         | *start\_monitoring* → monitoring\_starting         |
| monitoring\_starting | Monitor service booting.                   | *start\_monitoring*                                              | *start\_monitoring\_done* → **degraded** (initial) |
| monitoring\_stopping | Monitor shutting down.                     | *stop\_monitoring*                                               | *stop\_monitoring\_done* → monitoring\_stopped     |

***

## 3 — Agent Monitor

| State                | Meaning                                      | Enter                    | Exit                                               |
| -------------------- | -------------------------------------------- | ------------------------ | -------------------------------------------------- |
| **active**           | Agent connected & internal tasks OK.         | *metrics\_all\_ok*       | *metrics\_not\_ok* → **degraded**                  |
| ⚠️ **degraded**      | Cloud unreachable / auth error / task panic. | *metrics\_not\_ok*       | *metrics\_all\_ok* → **active**                    |
| monitoring\_stopped  | Agent health monitor off.                    | *stop\_monitoring\_done* | *start\_monitoring* → monitoring\_starting         |
| monitoring\_starting | Starting health checks.                      | *start\_monitoring*      | *start\_monitoring\_done* → **degraded** (initial) |
| monitoring\_stopping | Halting checks.                              | *stop\_monitoring*       | *stop\_monitoring\_done* → monitoring\_stopped     |

***

## 4 — Bridge

### Aggregate Bridge FSM

| State                              | Meaning                                       | Status Reason Examples                           | Enter                                              | Exit                                                                                                                                                               |
| ---------------------------------- | --------------------------------------------- | ------------------------------------------------ | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **stopped**                        | All sub‑services stopped.                     | `"stopped"`                                      | *stop\_done* or after create.                      | *start* → **starting\_connection**                                                                                                                                 |
| *starting\_connection*             | Waiting for connection to establish.          | `"starting: waiting for connection"`             | *start*                                            | *start\_connection\_up* → **starting\_redpanda**                                                                                                                   |
| *starting\_redpanda*               | Connection up, waiting for message broker.    | `"starting: redpanda not healthy"`               | *start\_connection\_up*                            | *start\_redpanda\_up* → **starting\_dfc**                                                                                                                          |
| *starting\_dfc*                    | Connection + Redpanda up, waiting for flow.   | `"starting: flow not running"`                   | *start\_redpanda\_up*                              | <p><em>start\_dfc\_up</em> → <strong>idle</strong><br><em>start\_failed\_dfc\_missing</em> → <strong>starting\_failed\_dfc\_missing</strong></p>                   |
| **starting\_failed\_dfc**          | Flow component failed to start.               | `"starting failed: flow in error state"`         | *start\_failed\_dfc*                               | Manual retry or removal                                                                                                                                            |
| **starting\_failed\_dfc\_missing** | No flow configured.                           | `"starting failed: no flows configured"`         | *start\_failed\_dfc\_missing*                      | *start\_retry* (when flow added) or removal                                                                                                                        |
| **idle**                           | All healthy, **no data for 30 s**.            | `"idling: no messages processed in 60s"`         | *start\_dfc\_up*, *no\_data\_timeout*, *recovered* | <p><em>data\_received</em> → <strong>active</strong><br><em>degraded</em> events → <strong>degraded\_</strong>\*<br><em>stop</em> → <strong>stopping</strong></p>  |
| **active**                         | Processing data through flows.                | `""` (empty when fully healthy)                  | *data\_received*                                   | <p><em>no\_data\_timeout</em> → <strong>idle</strong><br><em>degraded</em> events → <strong>degraded\_</strong>\*<br><em>stop</em> → <strong>stopping</strong></p> |
| ⚠️ **degraded\_connection**        | Connection lost/flaky after successful start. | `"connection degraded: probe timeout after 30s"` | *connection\_unhealthy*                            | <p><em>recovered</em> → <strong>idle</strong><br><em>stop</em> → <strong>stopping</strong></p>                                                                     |
| ⚠️ **degraded\_redpanda**          | Message broker issues after successful start. | `"redpanda degraded: not responding"`            | *redpanda\_degraded*                               | <p><em>recovered</em> → <strong>idle</strong><br><em>stop</em> → <strong>stopping</strong></p>                                                                     |
| ⚠️ **degraded\_dfc**               | Flow component issues after successful start. | `"flow degraded: benthos service not running"`   | *dfc\_degraded*                                    | <p><em>recovered</em> → <strong>idle</strong><br><em>stop</em> → <strong>stopping</strong></p>                                                                     |
| ⚠️ **degraded\_other**             | Inconsistent component states detected.       | `"other degraded: inconsistent states"`          | *degraded\_other*                                  | <p><em>recovered</em> → <strong>idle</strong><br><em>stop</em> → <strong>stopping</strong></p>                                                                     |
| *stopping*                         | Stopping all components.                      | `"stopping"`                                     | *stop*                                             | *stop\_done* → **stopped**                                                                                                                                         |

### 4.1 Connection Service FSM

| State           | Meaning                         |
| --------------- | ------------------------------- |
| *starting*      | Probe service launching.        |
| **up**          | Target reachable.               |
| **down**        | Target unreachable.             |
| ⚠️ **degraded** | Flaky / intermittent responses. |
| *stopping*      | Probe shutting down.            |
| **stopped**     | Probe disabled.                 |

### 4.2 Benthos Flow (Source /Sink)

| State                                                  | Meaning                                                |
| ------------------------------------------------------ | ------------------------------------------------------ |
| **stopped**                                            | Service file present, process not running.             |
| *starting*                                             | S6 launched process.                                   |
| *starting\_config\_loading*                            | Benthos parsing YAML pipeline.                         |
| *starting\_waiting\_for\_healthchecks*                 | Pipeline loaded; waiting for plugin health.            |
| *starting\_waiting\_for\_service\_to\_remain\_running* | Stability grace period.                                |
| **idle**                                               | Flow running, no msgs for idle window.                 |
| **active**                                             | Processing messages.                                   |
| ⚠️ **degraded**                                        | Flow running but error state (e.g., endpoint retries). |
| *stopping*                                             | Graceful SIGTERM underway.                             |

> **Idle/Active timeout:** default 30 s (`BRIDGE_IDLE_WINDOW`).

***

## 5 — Topic Browser Service

The Topic Browser service manages real-time topic discovery and caching.

| State                     | Description             | Enter Trigger                 | Exit Trigger                                 |
| ------------------------- | ----------------------- | ----------------------------- | -------------------------------------------- |
| **stopped**               | Service not running     | Initial state or *stop\_done* | *start* → **starting**                       |
| *starting*                | Service initialization  | *start*                       | *benthos\_started* → **starting\_benthos**   |
| *starting\_benthos*       | Benthos starting        | *benthos\_started*            | *redpanda\_started* → **starting\_redpanda** |
| *starting\_redpanda*      | Redpanda connection     | *redpanda\_started*           | *start\_done* → **idle**                     |
| **idle**                  | Healthy, no active data | *start\_done* or *recovered*  | *data\_received* → **active**                |
| **active**                | Processing topic data   | *data\_received*              | *no\_data\_timeout* → **idle**               |
| ⚠️ **degraded\_benthos**  | Benthos degraded        | *benthos\_degraded*           | *recovered* → **idle**                       |
| ⚠️ **degraded\_redpanda** | Redpanda degraded       | *redpanda\_degraded*          | *recovered* → **idle**                       |
| *stopping*                | Graceful shutdown       | *stop*                        | *stop\_done* → **stopped**                   |

**Default:** Active (runs automatically)\
**Transitions:** idle ↔ active based on topic activity\
**Recovery:** Automatic from degraded states when underlying services recover

***

### Quick Defaults

| Parameter              | Default | Source Const / Env     |
| ---------------------- | ------- | ---------------------- |
| Idle window (Redpanda) | 30 s    | `REDPANDA_IDLE_WINDOW` |
| Idle window (Bridge)   | 30 s    | `BRIDGE_IDLE_WINDOW`   |
| Container CPU limit    | 85 %    | `CONTAINER_CPU_LIMIT`  |
| Container RAM limit    | 90 %    | `CONTAINER_RAM_LIMIT`  |
| Container Disk limit   | 90 %    | `CONTAINER_DISK_LIMIT` |


# Environment Variables

This is the reference for all environment variables used by umh-core. These variables override or supplement configuration from `config.yaml`.

| Name                    | Values                                                                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| ----------------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| LOGGING\_LEVEL          | ERROR, WARN, INFO, DEBUG                                                   | Controls log verbosity. DEBUG shows detailed internal operations                                                                                                                                                                                                                                                                                                                                                                             |
| AUTH\_TOKEN             | (Base64 encoded token)                                                     | Management Console authentication token. Overrides auth token from config.yaml                                                                                                                                                                                                                                                                                                                                                               |
| API\_URL                | <https://management.umh.app/api>, <https://staging.management.umh.app/api> | Management Console API endpoint. Use staging for testing environments                                                                                                                                                                                                                                                                                                                                                                        |
| RELEASE\_CHANNEL        | enterprise, stable, nightly                                                | Auto-update channel. Enterprise = most stable, nightly = latest features                                                                                                                                                                                                                                                                                                                                                                     |
| S6\_PERSIST\_DIRECTORY  | true/1/TRUE/True, (unset/false/other)                                      | Controls S6 service directory persistence. Truthy values (true, 1, TRUE, etc.) = /data/services (persist across container restarts for debugging). Default/false = /tmp/umh-core-services (cleared on container restart for fresh state)                                                                                                                                                                                                     |
| USE\_CANONICALIZE\_FAST | 0, 1, false, true (default: 1)                                             | Compares a bridge's desired and running configuration by walking the values directly. Set to 0 and restart the container to fall back to the slower YAML round-trip, the reference implementation, if a configuration is ever reported as changed on every tick. The value is read as a boolean, so `1`/`0`, `true`/`false` and `t`/`f` all work in any capitalization; anything else falls back too, and logs a warning naming what was set |
| LOCATION\_<0-6>         | (string values)                                                            | Sets Agent location path levels 0-6. Example: LOCATION\_0=factory, LOCATION\_1=line1                                                                                                                                                                                                                                                                                                                                                         |


# TLS and Certificates

This page covers the TLS connections to the Management Console: which hosts your browser and your umh-core instances connect to, who issues the certificates, and what happens when a corporate firewall inspects the traffic. The certificates that umh-core manages on the instance, and the access to your instances that a [passphrase](/management-console/authentication/passwords-and-passphrases) unlocks, are a different topic and are not covered here.

## Hosts and certificates

| Host                      | Who connects                                                                                             | Certificate                                                                  |
| ------------------------- | -------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| `management.umh.app`      | Your browser for the console, umh-core for its API under `/api`, and Docker for image pulls under `/oci` | Issued by a publicly trusted certificate authority and renewed automatically |
| `auth.management.umh.app` | Your browser during sign-in                                                                              | Issued by a publicly trusted certificate authority and renewed automatically |

Both certificates chain to public roots that ship with every current browser and operating system. There is nothing to install and nothing to pin on your side. The leaf certificate changes at every renewal and the intermediate certificate can change with it, so allowlist the two hostnames in your firewall, not certificate fingerprints.

## Supported TLS versions

| Version           | `management.umh.app` | `auth.management.umh.app` |
| ----------------- | -------------------- | ------------------------- |
| TLS 1.3           | Accepted             | Accepted                  |
| TLS 1.2           | Accepted             | Accepted                  |
| TLS 1.1 and older | Refused              | Refused                   |

Both hosts send HTTP Strict Transport Security headers, so a browser that has visited them once refuses plain HTTP afterwards. Both hosts also offer HTTP/2 and HTTP/3 to browsers.

umh-core connects with TLS 1.2 or higher and validates the certificate against the trust store of its container. See [Cryptography and TLS](/production/security/deployment-security#cryptography-and-tls).

## Certificate authorities

The Management Console does not run a certificate authority for its own endpoints. Trust comes from the public roots in the browser or operating system, and in the umh-core container from the `ca-certificates` package installed in the container image.

Certificate authorities inside umh-core, such as the one that signs certificates for your bridges and users, are unrelated to the TLS connection to the console.

## Corporate TLS inspection

A firewall that inspects TLS terminates the connection itself and presents a certificate signed by your corporate certificate authority instead of the public one.

* **Browser.** Sign-in and the console keep working as long as the corporate certificate authority is in the operating system's trust store, which your IT usually manages. This applies to `auth.management.umh.app` as well.
* **umh-core.** The container trusts only the public roots by default, so it refuses the corporate certificate. Add your corporate certificate authority to the container's trust store. If that is not possible, `ALLOW_INSECURE_TLS` is the fallback described below. See [Network Configuration](/production/security/deployment-security/network-configuration#tls-inspection-mitm) for both options and for proxy settings, which usually go together with inspection.

## ALLOW\_INSECURE\_TLS

`ALLOW_INSECURE_TLS=true` makes umh-core accept any certificate for its connection to `management.umh.app`, and lowers the minimum TLS version for that connection to TLS 1.0. It has no effect on your browser.

| Where to set it      | Value                                       |
| -------------------- | ------------------------------------------- |
| Environment variable | `ALLOW_INSECURE_TLS=true`                   |
| `config.yaml`        | `agent.communicator.allowInsecureTLS: true` |

With validation disabled, anyone between the instance and the console can read and change the traffic, including the `AUTH_TOKEN`. Use it only behind a firewall that you trust to be the only party inspecting the connection, and prefer adding the corporate certificate authority. See [TLS Certificate Validation Can Be Disabled](/production/security/deployment-security#tls-certificate-validation-can-be-disabled) for the risk assessment, and the [Configuration Reference](/reference/configuration-reference) for the setting itself.


# Variables

## Overview

UMH Core uses a three-tier variable system for protocol converter templates that enables flexible configuration and runtime customization:

* **User Variables**: Defined in the `variables:` section of your YAML configuration, flattened to top-level access
* **Internal Variables**: Runtime-injected system metadata and computed values
* **Global Variables**: Fleet-wide settings (**NOT YET IMPLEMENTED**)

## Variable Precedence

Variables follow a clear precedence hierarchy:

1. **User Variables** - Flattened to the top level. Some names are reserved: see [User-Defined Variables](#user-defined-variables).
2. **Internal Variables** - System-generated values
3. **Global Variables** - Fleet-wide defaults (**NOT YET IMPLEMENTED**)

## Available Variables

### Connection Variables

These variables provide device connection information, typically auto-injected from Connection configurations or defined by users.

| Variable      | Description                   | Example Value   | Source          | Status  |
| ------------- | ----------------------------- | --------------- | --------------- | ------- |
| `{{ .IP }}`   | Device IP address or hostname | `192.168.1.100` | User/Connection | Current |
| `{{ .PORT }}` | Device communication port     | `4840`          | User/Connection | Current |

**Usage Example:**

```yaml
protocolConverter:
  config: |
    input:
      opcua:
        endpoint: "opc.tcp://{{ .IP }}:{{ .PORT }}/OPCUA/SimulationServer"
```

### Location Variables

These variables provide hierarchical location information for data organization and topic routing.

| Variable               | Description                       | Example Value                                | Source   | Status  |
| ---------------------- | --------------------------------- | -------------------------------------------- | -------- | ------- |
| `{{ .location_path }}` | Dot-separated hierarchical path   | `factory.line1.machine1`                     | Internal | Current |
| `{{ .location }}`      | Location as numbered map          | `{"0":"factory","1":"line1","2":"machine1"}` | Internal | Current |
| `{{ .location.0 }}`    | Enterprise level (first location) | `factory`                                    | Internal | Current |
| `{{ .location.1 }}`    | Site level (second location)      | `line1`                                      | Internal | Current |
| `{{ .location.2 }}`    | Area level (third location)       | `machine1`                                   | Internal | Current |

**Usage Example:**

```yaml
protocolConverter:
  config: |
    output:
      mqtt:
        topic: "umh/v1/{{ .location_path }}/data"
        # Or access individual levels:
        # topic: "umh/v1/{{ .location.0 }}/{{ .location.1 }}/data"
```

### Internal System Variables

These variables provide runtime metadata and system integration information.

| Variable                     | Description                                | Example Value                     | Source   | Status                     |
| ---------------------------- | ------------------------------------------ | --------------------------------- | -------- | -------------------------- |
| `{{ .internal.bridged_by }}` | Bridge identifier that created this config | `protocol-converter-node1-pc123`  | Internal | Current                    |
| `{{ .internal.id }}`         | Protocol converter instance ID             | `pc-123`                          | Internal | Current                    |
| `{{ .internal.umh_topic }}`  | UMH topic for write flows                  | `umh/v1/factory/line1/_historian` | Internal | Current (write flows only) |

**Usage Example:**

```yaml
protocolConverter:
  config: |
    output:
      mqtt:
        topic: "{{ .internal.umh_topic }}"
        metadata:
          bridged_by: "{{ .internal.bridged_by }}"
```

### Global Variables (Not Yet Implemented)

These variables will provide fleet-wide configuration when implemented.

| Variable                   | Description                | Example Value    | Source | Status              |
| -------------------------- | -------------------------- | ---------------- | ------ | ------------------- |
| `{{ .global.cluster_id }}` | Multi-cluster identifier   | `cluster-west-1` | Global | **NOT IMPLEMENTED** |
| `{{ .global.* }}`          | Other fleet-wide variables | varies           | Global | **NOT IMPLEMENTED** |

## Deprecated Variables

The following variables should be replaced in your configurations:

| Deprecated Variable  | Status       | Replacement   | Action Required               |
| -------------------- | ------------ | ------------- | ----------------------------- |
| `{{ .HOST }}`        | Deprecated   | `{{ .IP }}`   | Replace in all configurations |
| `{{ .DEVICE_IP }}`   | Inconsistent | `{{ .IP }}`   | Standardize usage             |
| `{{ .DEVICE_PORT }}` | Inconsistent | `{{ .PORT }}` | Standardize usage             |

## User-Defined Variables

You can define custom variables in the `variables:` section of your configuration. These are flattened to top-level access. Do not name a variable `location`, `location_path`, `historian`, `global` or `internal`. umh-core reserves these names for its own values.

**Common patterns:**

* `{{ .SCAN_RATE }}` - Polling intervals
* `{{ .TAG_PREFIX }}` - Tag naming prefixes
* `{{ .USERNAME }}` - Authentication credentials
* `{{ .PASSWORD }}` - Authentication credentials

**Example:**

```yaml
protocolConverter:
  variables:
    SCAN_RATE: "1000ms"
    TAG_PREFIX: "PLC1_"
  config: |
    input:
      opcua:
        endpoint: "opc.tcp://{{ .IP }}:{{ .PORT }}/OPCUA/SimulationServer"
        poll_interval: "{{ .SCAN_RATE }}"
        tag_prefix: "{{ .TAG_PREFIX }}"
```

## Variable Sources

Understanding where variables come from helps with troubleshooting:

* **Connection Variables** (`{{ .IP }}`, `{{ .PORT }}`): Auto-injected from Connection configurations attached to Bridges
* **Location Variables** (`{{ .location_path }}`, `{{ .location }}.*`): Computed from agent location + bridge location by BuildRuntimeConfig
* **Internal Variables** (`{{ .internal.* }}`): Runtime-injected system metadata
* **User Variables**: Explicitly defined in `variables:` section
* **Global Variables**: Fleet-wide settings (**NOT YET IMPLEMENTED**)

## Troubleshooting

### Variable Not Found

If a variable is not resolving:

1. Check spelling and case sensitivity
2. Verify the variable is defined in the `variables:` section (for user variables)
3. Ensure Connection is properly attached (for `.IP`/`.PORT`)
4. Check that location path is configured (for location variables)

### Deprecated Variable Warnings

Replace deprecated variables with their current equivalents:

* `{{ .HOST }}` → `{{ .IP }}`
* `{{ .DEVICE_IP }}` → `{{ .IP }}`
* `{{ .DEVICE_PORT }}` → `{{ .PORT }}`

For questions about variables not listed here, check the [configuration reference](/reference/configuration-reference) or consult the UMH Core documentation.


# HTTP API Reference

{% hint style="info" %}
**Early Access.** The GraphQL API lets developers query the Unified Namespace programmatically. It's experimental and off by default. To turn it on, set `agent.graphql.enabled: true` in the `agent:` block of your `umh-core` instance configuration. See [Configuration](/reference/http-api/topic-browser-graphql#configuration) for all options. Tell us what you think.
{% endhint %}

## GraphQL API

* **Default:** Disabled; enable with `agent.graphql.enabled: true` (port 8090)
* **Endpoint:** `POST /graphql`
* **GraphiQL:** Available at `/` when debug enabled
* **Purpose:** Query Unified Namespace topics

## Topic Browser GraphQL

* [Complete GraphQL API Reference](/reference/http-api/topic-browser-graphql)
* [Schema Documentation](/reference/http-api/topic-browser-graphql#schema)
* [Query Examples](/reference/http-api/topic-browser-graphql#examples)

## Monitoring

* **Metrics:** See [Production Metrics](/production/metrics)
* **Health:** Via FSM state monitoring


# Topic Browser GraphQL API

{% hint style="info" %}
**Early Access.** The GraphQL API lets developers query the Unified Namespace programmatically. It's experimental and off by default. To turn it on, set `agent.graphql.enabled: true` in the `agent:` block of your `umh-core` instance configuration. See [Configuration](#configuration) for all options. Tell us what you think.
{% endhint %}

## Overview

Query Unified Namespace topics and metadata in real-time.

* **Endpoint:** `http://localhost:8090/graphql`
* **Default:** Disabled; opt in with `agent.graphql.enabled: true`
* **Authentication:** None (open by default)

## Schema

### Queries

```graphql
type Query {
  topics(filter: TopicFilter, limit: Int): [Topic!]!
  topic(topic: String!): Topic
}
```

### Types

```graphql
type Topic {
  topic: String!
  metadata: [MetadataEntry!]!
  lastEvent: Event    # Latest event only
}

union Event = TimeSeriesEvent | RelationalEvent

type TimeSeriesEvent {
  producedAt: String!
  scalarType: String!
  numericValue: Float
  stringValue: String
  booleanValue: Boolean
}

type RelationalEvent {
  producedAt: String!
  json: String!
}

input TopicFilter {
  text: String
  meta: [MetaExpr!]
}

input MetaExpr {
  key: String!
  eq: String!
}
```

## Example Queries

### All topics:

```graphql
{ topics { topic metadata { key value } } }
```

### Filter by text:

```graphql
{ topics(filter: { text: "temperature" }) { topic } }
```

### Filter by metadata:

```graphql
{
  topics(filter: { 
    meta: [{ key: "data_contract", eq: "_pump" }] 
  }) {
    topic
    lastEvent {
      ... on TimeSeriesEvent {
        producedAt
        numericValue
      }
    }
  }
}
```

### Single topic:

```graphql
{
  topic(topic: "umh.v1.acme.plant1.line4.sensor1._raw.temperature") {
    metadata { key value }
    lastEvent {
      ... on TimeSeriesEvent {
        producedAt
        numericValue
        scalarType
      }
    }
  }
}
```

## Using curl

### Basic query:

```bash
curl -X POST http://localhost:8090/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ topics(limit: 3) { topic } }"}'
```

### Filter query:

```bash
curl -X POST http://localhost:8090/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ topics(filter: { text: \"pump\" }) { topic lastEvent { ... on TimeSeriesEvent { numericValue } } } }"}'
```

## Limitations

* **History:** Latest event only (no historical data)
* **Subscriptions:** Not supported (queries only)
* **Rate Limiting:** None (use responsibly)
* **Scope:** Single UMH instance only

## Multi-Instance Behavior

* Each UMH Core instance has its own Topic Browser
* GraphQL API returns topics from local instance only
* Management Console aggregates all instances into unified view

## Configuration

```yaml
agent:
  graphql:
    enabled: false   # Default: false; set true to opt in
    port: 8090      # Default: 8090
    debug: false    # Set true for GraphiQL UI
    corsOrigins: [] # Default: allows all origins
```

## Security

* **Authentication:** Currently open (no tokens required)
* **CORS:** Configurable via `corsOrigins` setting
* **Network Security:** Recommended for production deployments
* **Port Access:** Ensure port 8090 is appropriately secured


# UMH Core vs UMH Classic

## Why a single Docker container?

* **Enterprise deployment reality**
  * *In most enterprises*: **tight operating system images, firewalls, and network hoops** turn Kubernetes setup into friction. Custom load-balancers, network policies, storage classes, and custom upgrade paths create endless variables.
  * *UMH Core*: **almost anything that runs Docker works** — from MacBook terminals to edge boxes to enterprise K8s clusters.
* **Deployment complexity**
  * *UMH Classic*: many pods, sidecars, service meshes → a forest of YAML.
  * *UMH Core*: **one image, one command**, sub-second startup.
* **Observability & recovery**
  * *Classic*: error clues spread across pod logs, `kubectl` events and load-balancers. Kubernetes states (`Pending`, `Running`, …) are usually not what the user would expect—`Running` does **not** guarantee the pod is healthy.
  * *Core*: **each component has well-defined states** and is recovered if the state is not healthy (e.g., incl. watching out for error or warning logs, metrics checks, etc.); S6 restarts failed processes instantly.
* **Real-time responsiveness**
  * *Classic*: waiting thirty seconds for a new pipeline to launch due to Kubernetes overhead
  * *Core*: **ticks every 100 milliseconds**, so configuration changes feel instant
* **Version management**
  * *Classic*: drift between Node-RED plug-ins, broker, bridges.
  * *Core*: *bump one line in the Dockerfile → ship vX.Y* — everything already integration-tested.
* **Edge resource footprint**
  * *Classic*: edge boxes fight the kube-control-plane for CPU/RAM.
  * *Core*: no kubelet, no sidecars, no HiveMQ ⇒ **lighter RAM/CPU usage**.

We collapse the stack into a single supervised image, giving enterprises less integration pain and users deterministic behaviour. Inside that image the **Agent, Bridges, Redpanda, and Benthos** still run as separate S6-managed processes, so you keep modularity without the network overhead.

## Isn't that just going back to a monolith? *(Micro-services vs Monolith)*

A "modular monolith" is the sweet spot for most factories:

* **You don't split a five-piece toolkit into hundreds of micro-services** – yet one big binary (like some protocol converters out there) is risky. If one protocol converter fails, it takes all the rest with it.
* **UMH Core** packs the essentials (Agent, Benthos pipelines, Redpanda buffer) in one container but launches **every single part as its own S6-supervised process**. Crash one, the others stay green.
* **Scale when it hurts:** Need more juice later for select pipelines? Break that process out into its own container.
* **Micro-services tax** – separate repos, CI/CD pipelines, service discovery, network latency – only pays off when you reach FAANG-scale org size.

***

## "So you ditched Kubernetes? I thought you were cloud-native!"

We didn't ditch it—**we de-coupled from it**.

* **The enterprise Kubernetes reality:** Organizations run their K8s clusters with custom load balancers, network policies, custom storage classes, and their own upgrade paths. Building one product that survives every combination and ships an SLA is a nightmare.
* **Our solution:** **UMH Core is&#x20;*****more*****&#x20;k8s-friendly** because it's self-contained. Drop the same image into k3s, OpenShift, EKS or bare Docker; no hidden assumptions about CNIs, StorageClasses, or LBs.
* With one container, those variables disappear. **Kubernetes becomes an optional scheduler, not a hard dependency.**

***

## Do we lose Kubernetes advantages?

No – the two big ones are still there:

**Scalability**: Deploy one UMH Core instance per production line, each forwarding to a central site instance. This ISA-95 style hierarchy naturally distributes workload across your infrastructure.

**Failover**: Your container manager restarts crashed containers (30-60 seconds) and reschedules after node failure (2-5 minutes). See [High Availability](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/umh-core/production/high-availability.md).

## Why is Kafka (Redpanda) embedded instead of its own container?

The UNS needs a **policy wall**. Every byte passes schema validation before it hits the log. Exposing raw Redpanda ports would let clients inject malformed data, bypassing Bridges. Embedding gives us:

* On-the-fly topic/ACL/retention re-config from the Agent.
* Enforcement of Matthew Parris' **portal / bridge / firewall** model.

***

## Where is MQTT?

1. **Bring-your-own broker** (HiveMQ CE, Mosquitto) in a side-car, then connect via a Bridge.
2. **UMH MQTT plug-in** (road-map) – a slim container that wraps Redpanda with an MQTT façade plus schema enforcement.

## Where is Benthos running?

Each Bridge launches its own Benthos pipeline **inside** the image; S6 supervises it. If that flow crashes, S6 restarts just that pipeline, leaving the rest untouched.

## Where is Node-RED?

Outside the image, on purpose. Node-RED plug-ins break often; letting plant IT run their preferred image means **we don't accidentally nuke custom flows** during an update.

***

## Why not ship "just a Helm chart"?

Kubernetes distros vary wildly (OpenShift SCCs, Rancher CNIs, air-gapped k3s, …). One universal chart devolves into a spaghetti of `if/else` we can't certify. **A single image sidesteps 90 % of those edge-case failures.**

***

## Why not let users own the Helm chart (self-managed)?

Our value prop is **curated integration + SLA**. If every customer patches their own chart, swaps Grafana versions, tweaks sidecars, we lose traceability and can't guarantee hot-fixes.

***

## How do I migrate from UMH Classic?

1. **Side-by-side:** deploy `umh-core` next to Classic, point a new Bridge at the old UNS topics.
2. **Cut-over:** switch producers to the new UNS, retire Classic components incrementally (Node-RED flows, Grafana, Timescale).
3. **Clean-up:** decommission the Helm stack once data is stable in UMH Core.

## What happened to Grafana, Timescale (Historian), and other "extras"?

**Short answer:** anything that needs heavy customization or third-party plug-ins now lives **outside** the core image, so you keep full version control.

* **Grafana + TimescaleDB (Historian)**\
  Deployed as a separate Docker or Helm package—or via the upcoming *UMH Historian* plug-in. Many plants already run their own Grafana stack or rely on community plug-ins; we don't want to auto-upgrade and break dashboards.
* **Node-RED**\
  Lives in its own container for the same reason: community plug-ins change fast, and site teams often pin specific versions.
* **MQTT broker**\
  Bring your own HiveMQ CE, Mosquitto, etc., or use the soon-to-ship *UMH MQTT* plug-in, which wraps Redpanda with an MQTT façade plus schema enforcement.

**Rule of thumb:** if a service demands lots of plug-ins or customization, we keep it out of the core image and provide a starter Compose/Helm file instead.

## How does UMH Core track health if it's not using Kubernetes pod states?

* **Remember Kubernetes?** A green "Running" pod in Kubernetes only means the container has started — not that your application is healthy.
* **In UMH Core, green actually means healthy.** Every component (Bridge, Benthos pipeline, Redpanda, etc.) is wrapped in a **finite-state machine** managed by the Agent: `starting → active → idle → degraded → stopped`.
* The Agent evaluates **metrics + logs + live config** every 100 ms. A Bridge stuck in a retry loop goes to `degraded`, a pipeline with zero throughput stays `idle`, etc. It fires warnings if anything drifts, and restarts a pipeline the moment a new config lands.
* States and counters are exported via `/metrics`, so you can alert on "connection *degraded* for > 30 s", not just pod restarts.

This offers **finer-grained insight** than Kubernetes' generic `Running/CrashLoopBackOff`.

## How do I connect my MQTT devices now?

Run your own broker (Mosquitto, HiveMQ CE, EMQX, etc.), then create a Bridge in UMH Core that subscribes to the topics you care about and inserts the messages into the Unified Namespace.\
This keeps MQTT exactly where most plants like it—outside the UNS as a "portal" layer—while UMH Core enforces schema and ACLs before data lands in the clean store, and before you forward it to other nodes.

## I don't want Kafka / Redpanda — I want MQTT to be **my** UNS

No problem.\
UMH Core's internal Redpanda is just a **hidden workhorse** for buffering and exactly-once processing; you don't interact with it directly.

1. **Keep (or spin up) your own MQTT broker** — Mosquitto, HiveMQ CE, EMQX, or the upcoming UMH MQTT plug-in.
2. **Create a Bridge** in UMH Core that subscribes to—or publishes from—the topics you choose.
3. From an application point of view, your Unified Namespace *is* MQTT.\
   Redpanda just ensures messages are durable, ordered, and schema-checked before they reach consumers.

You never have to operate or even "see" Redpanda; UMH Core embeds and manages it automatically.

## How do I connect two UMH Core instances?

**Short answer:** create a **Bridge** on each side that speaks the upcoming *umh-core-API* output/input plug-in.

> 🚧 **Roadmap**: The *umh-core-API* plugin is under development to enable direct Core-to-Core communication.

* It's the most firewall-friendly protocol in enterprise networks.
* SSL/TLS is easy to terminate and audit.
* A mature tool-chain (proxies, WAFs, tracing) already exists.
* With idempotent writes we achieve MQTT-style **QoS 1** (at-least-once, de-duplication on the receiving side).

*(Until this feature ships, you can already link cores via Kafka Bridges.)*

## How can I **view** or **query** my Unified Namespace?

* **Topic Browser** (road-map) – a point-and-click tree inside the Management Console to inspect live values.
* **REST API** – two endpoints:
  1. `POST /api/v1/ingest` – push messages into UMH Core (used by the core-to-core Bridge).
  2. `GET /api/v1/uns/tree` – fetch the namespace tree, including latest values and metadata.

These endpoints are secured by mTLS or JWT and respect the same ACLs the Bridges enforce.

***

## What is open-source now?

* **Was:** only the Helm chart was OSS; the Agent and most internals were closed-source.
* **Now:** with UMH Core **everything except the Management Console is Apache-2**.
  * Agent, Bridges, S6 service definitions, Redpanda integration, FSM health logic – all public.
  * The Management Console UI remains free-to-use (Community Edition)


# Introduction

benthos-umh is a specialized extension of Benthos (now known as Redpanda Connect) developed by the [United Manufacturing Hub (UMH)](https://www.umh.app). Tailored for the manufacturing industry, benthos-umh integrates additional manufacturing protocols such as OPC UA, Siemens S7, and Modbus.

## 📚 Documentation

**If you're viewing this on GitHub, please visit our full documentation at** [**docs.umh.app/benthos-umh**](https://docs.umh.app/benthos-umh)**.**


# Input

This section covers Benthos input plugins for collecting data from various industrial systems and protocols. Input plugins allow you to ingest data from PLCs, sensors, and other industrial devices into your Benthos data pipelines.

## Available Input Plugins

* [**Sparkplug B Input**](/benthos-umh/input/sparkplug-b-input) - Ingests data from MQTT brokers using the Sparkplug B specification. Acts as a Host (Secondary or Primary) to consume Sparkplug B messages from Edge Nodes and converts them to UMH-Core format with automatic hierarchy mapping.
* [**OPC UA Input**](/benthos-umh/input/opc-ua-input) - Connects to OPC UA servers to browse and subscribe to nodes. Supports various data types, authentication methods, and provides comprehensive metadata for each message.
* [**Modbus**](/benthos-umh/input/modbus) - Communicates with Modbus devices supporting coils, discrete inputs, holding registers, and input registers. Features register optimization, device-specific workarounds, and extensive configuration options.
* [**ifm IO-Link Master / SensorConnect**](/benthos-umh/input/ifm-io-link-master-sensorconnect) - Integrates with ifm electronic's IO-Link Master devices (AL1350, AL1352) and EIO404 Bluetooth mesh stations. Processes sensor data using IODD files for human-readable output.
* [**Beckhoff ADS (experimental)**](/benthos-umh/input/beckhoff-ads) - Reads Beckhoff PLCs over ADS with change notifications or interval polling, addressing variables by symbol name. Covers TwinCAT 2 and TwinCAT 3, with automatic route registration.
* [**Siemens S7**](/benthos-umh/input/siemens-s7) - Connects to Siemens S7 PLCs via the native S7 protocol, enabling cyclic or on-demand reads of data blocks, inputs/outputs, and markers with configurable polling interval.
* [**Ethernet/IP**](/benthos-umh/input/ethernet-ip) - Interfaces with Rockwell and other Ethernet/IP compatible controllers, mainly supporting the CIP protocol.
* [**UNS Input**](/benthos-umh/input/uns-input) - Connects to UMH Core's Unified Namespace to consume and filter messages using regex patterns against UMH topic keys.
* [**More Input Plugins**](https://docs.redpanda.com/redpanda-connect/components/inputs/about/) - Additional built-in input plugins available in Benthos/Redpanda Connect for various data sources.

## Choosing the Right Input Plugin

* Use **Sparkplug B Input** when consuming data from existing Sparkplug B Edge Nodes or integrating with Sparkplug B-enabled devices
* Use **OPC UA Input** for modern industrial systems that support the OPC UA standard
* Use **Modbus** for legacy industrial devices and PLCs that communicate via Modbus protocol
* Use **ifm IO-Link Master** when working with ifm electronic's IO-Link infrastructure and sensors
* Use **Beckhoff ADS** specifically for Beckhoff PLC systems and TwinCAT environments
* Use **Siemens S7** for industrial Siemens devices that communicate via Siemens protocol
* Use **Ethernet/IP** for CompactLogix, ControlLogix, Micro800er series, that communicate via CIP protocol
* Use **UNS Input** when working within UMH Core to consume and filter messages from the Unified Namespace
* Explore **additional input plugins** for other protocols like HTTP, MQTT, databases, file systems, and more


# Sparkplug B (Input)

## Overview

The **Sparkplug B Input plugin** allows the United Manufacturing Hub (UMH) to ingest data from MQTT brokers using the Sparkplug B specification. It subscribes to Sparkplug B MQTT topics (e.g., device birth/data/death messages) and converts the incoming Protobuf payloads into UMH-compatible messages. It maintains the stateful context required by Sparkplug B – tracking device birth certificates, metric alias mapping, and sequence numbers – so that incoming data is interpreted correctly.

This input plugin is designed to seamlessly integrate Sparkplug-enabled edge devices into the UMH **Unified Namespace**. It automatically decodes Sparkplug messages and enriches them with metadata (such as metric names, types, and timestamps) to fit the UMH-Core data model.

## Sparkplug B in UMH Architecture

### UMH's Modified Parris Method

UMH implements a **Modified Parris Method** that distributes hierarchy across both `device_id` and `metric_name` fields instead of cramming everything into `GroupID`. This approach provides significant advantages for multi-site deployments (e.g., a single UMH instance can ingest data from multiple sites):

**Key Innovation**:

* **Location Hierarchy** → `device_id`: `"enterprise.site.area.line"` → `"enterprise:site:area:line"`
* **Virtual Path Hierarchy** → `metric_name`: Supports multiple separators:
  * Colons: `"motor:diagnostics:temperature"` → virtual\_path=`"motor.diagnostics"` + tag\_name=`"temperature"`
  * Slashes: `"motor/diagnostics/temperature"` → virtual\_path=`"motor.diagnostics"` + tag\_name=`"temperature"`
  * Dots: `"motor.diagnostics.temperature"` → virtual\_path=`"motor.diagnostics"` + tag\_name=`"temperature"`

### Architecture Roles

**Sparkplug B Input Plugin** (this plugin):

* **Role**: Host (Secondary Host by default, Primary Host optional)
* **Function**: Consumes Sparkplug B messages from external systems
* **Output**: Converts to UMH-Core format for UNS integration

**Sparkplug B Output Plugin** ([see output documentation](/benthos-umh/output/sparkplug-b-output)):

* **Role**: Edge Node
* **Function**: Publishes UMH-Core data as Sparkplug B messages
* **Input**: Receives UMH-Core format from UNS

### Integration with UMH Unified Namespace

The Sparkplug B plugins integrate seamlessly with the UMH UNS architecture:

**Data Ingestion Flow**:

External Sparkplug B Systems → Sparkplug B Input Plugin (Host) → [tag\_processor](/benthos-umh/processing/tag-processor) → [UNS Output](/benthos-umh/output/uns-output) → UNS

**Data Publication Flow**: UNS → [UNS Input](/benthos-umh/input/uns-input) → UMH-Core Format → [Sparkplug B Output Plugin](/benthos-umh/output/sparkplug-b-output) (Edge Node) → External Systems

### Why Modified Parris Method Matters

Unlike the original Parris Method which creates separate state management per GroupID, UMH's approach enables unified state management across all organizational levels by preserving hierarchy in `device_id` and `metric_name` fields, allowing scalable multi-enterprise/multi-site data ingestion without state explosion.

## Quick Start

Most users should use this simple configuration to read Sparkplug B data:

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "DeviceLevelTest"
    # role: "secondary_passive" is default - safest for brownfield deployments

pipeline:
  processors:
    - tag_processor:
        defaults: |
          // ============================================================
          // AUTOMATIC CONVERSION (works for 95% of cases)
          // ============================================================
          // SparkplugB fields are auto-converted to UMH format:
          // • Separators: colons/slashes → dots (priority: colon > slash > dot)
          // • Device identifiers → location_path: "Plant:Area" → "Plant.Area"
          // • Metric name → tag_name: "sensors/temp/value" → "value"
          // • Metric path → virtual_path: "sensors/temp/value" → "sensors.temp"

          msg.meta.location_path = msg.meta.umh_location_path;  // Auto-converted location
          msg.meta.data_contract = "_historian";                 // Time-series storage
          msg.meta.tag_name = msg.meta.umh_tag_name;            // Extracted tag name
          msg.payload = msg.payload.value;                      // SparkplugB metric value
          msg.meta.timestamp_ms = msg.meta.spb_timestamp;       // Native SparkplugB timestamp

          // Only set virtual_path if present (Benthos cannot store empty strings)
          if (msg.meta.umh_virtual_path) {
            msg.meta.virtual_path = msg.meta.umh_virtual_path;
          }

          return msg;

output:
  uns: {}
```

This configuration reads Sparkplug B messages from the configured group and converts them to UMH-Core format. The default `secondary_passive` role is read-only and won't interfere with existing Sparkplug infrastructure, making it safe to run multiple instances for load balancing and redundancy.

> `identity.group_id` is also the subscription filter. By default the plugin subscribes to `spBv1.0/<group_id>/#`, so the example above ingests only the `DeviceLevelTest` group. To listen to multiple groups, use `subscription.groups: ["GroupA", "GroupB"]`. To subscribe to every group on the broker, use the MQTT wildcard: `subscription.groups: ["+"]`. The subscribed topics are printed at startup (for example, `Operating as secondary_passive - subscribing to: [spBv1.0/DeviceLevelTest/#]`).

**To publish data as Sparkplug B**: After processing in the UNS, use the [Sparkplug B Output plugin](/benthos-umh/output/sparkplug-b-output) to convert UMH-Core data back to Sparkplug B format for external systems.

### Sparkplug B to UMH-Core Mapping

Here's how a Sparkplug B message maps to UMH-Core using the Modified Parris Method:

**Input Sparkplug B Message:**

* **Topic**: `spBv1.0/FactoryA/DDATA/EdgeNode1/enterprise:factory:line1:station1`
* **Metric Name**: `sensors:ambient:temperature` (or `sensors/ambient/temperature` or `sensors.ambient.temperature`)
* **Payload**: Protobuf with metric alias, value 23.5, timestamp

**↓ Results in Structured JSON Message:**

**Payload:**

```json
{
  "name": "sensors:ambient:temperature",
  "alias": 42,
  "value": 23.5
}
```

**Metadata:**

```json
{
  "spb_group_id": "FactoryA",
  "spb_edge_node_id": "EdgeNode1",
  "spb_device_id": "enterprise:factory:line1:station1",
  "spb_metric_name": "sensors:ambient:temperature",
  "spb_message_type": "DDATA",
  "umh_location_path": "enterprise.factory.line1.station1",
  "umh_virtual_path": "sensors.ambient",
  "umh_tag_name": "temperature"
}
```

**Key Transformations:**

1. **Device ID to Location Path**: `enterprise:factory:line1:station1` → `location_path: "enterprise.factory.line1.station1"` (colons → dots)
2. **Metric Name Parsing**: Splits on last separator (priority: colon > slash > dot)
   * `sensors:ambient:temperature` → `virtual_path: "sensors.ambient"` + `tag_name: "temperature"`
   * `sensors/ambient/temperature` → `virtual_path: "sensors.ambient"` + `tag_name: "temperature"`
   * `sensors.ambient.temperature` → `virtual_path: "sensors.ambient"` + `tag_name: "temperature"`
3. **Sparkplug Protobuf**: Metric value and alias → Structured JSON format `{"name": "...", "alias": X, "value": Y}`
4. **Topic Components**: Group/EdgeNode from MQTT topic used for `spb_group_id` and `spb_edge_node_id` metadata

**Reverse Transformation**: The [Sparkplug B Output plugin](/benthos-umh/output/sparkplug-b-output) performs the inverse transformation to convert UMH-Core messages back to Sparkplug B format.

## Configuration Reference

### MQTT Section

| Field                       | Type       | Default                     | Description              |
| --------------------------- | ---------- | --------------------------- | ------------------------ |
| `mqtt.urls`                 | `[]string` | **required**                | List of MQTT broker URLs |
| `mqtt.client_id`            | `string`   | `"benthos-sparkplug-input"` | MQTT client identifier   |
| `mqtt.credentials.username` | `string`   | `""`                        | MQTT username            |
| `mqtt.credentials.password` | `string`   | `""`                        | MQTT password            |
| `mqtt.qos`                  | `int`      | `1`                         | MQTT QoS level           |
| `mqtt.keep_alive`           | `duration` | `"60s"`                     | MQTT keep alive interval |
| `mqtt.connect_timeout`      | `duration` | `"30s"`                     | Connection timeout       |
| `mqtt.clean_session`        | `bool`     | `true`                      | MQTT clean session flag  |

### Identity Section

| Field                   | Type     | Default      | Description                                                                                                                                                     |
| ----------------------- | -------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `identity.group_id`     | `string` | **required** | Sparkplug B Group ID. Also acts as the default subscription filter (`spBv1.0/<group_id>/#`). Override via `subscription.groups`.                                |
| `identity.edge_node_id` | `string` | `""`         | Required for the `primary` role, where it is used as the Sparkplug v3.0 `host_id` in the STATE topic (`spBv1.0/STATE/<host_id>`). Optional for secondary roles. |

### Role Section

| Field  | Type     | Default               | Description                                     |
| ------ | -------- | --------------------- | ----------------------------------------------- |
| `role` | `string` | `"secondary_passive"` | Operating role for the Sparkplug B input plugin |

**Available Roles:**

* `"secondary_passive"` (default): Read-only consumer, no rebirth commands sent
* `"secondary_active"`: Consumer that can request rebirths when needed
* `"primary"`: Full Primary Host with STATE publishing and session management

### Rebirth Configuration

| Field                      | Type       | Default | Description                                                                                                                                                                                                                                                                       |
| -------------------------- | ---------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `request_birth_on_connect` | `bool`     | `true`  | Send REBIRTH when DATA arrives from a node with no prior BIRTH on this bridge. Typical after a bridge restart. Ignored under `secondary_passive`. Controls only the discovery path; sequence-gap and unresolved-aliases recovery always run for `secondary_active` and `primary`. |
| `birth_request_throttle`   | `duration` | `"1s"`  | Minimum time between REBIRTH commands to the same node, shared across every rebirth reason. Collapses simultaneous discovery, sequence-gap, and unresolved-aliases signals into one broker command per window. Set to `0` to disable throttling.                                  |

### Hierarchy Section

| Field                           | Type   | Default | Description                                                                                                                                                               |
| ------------------------------- | ------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `include_edge_node_in_location` | `bool` | `false` | Nest device-level data under its Sparkplug edge node. When `true`, device data maps to `location_path = <edge_node>.<device>`; node-level data (no device) is unaffected. |

**When to enable.** The default decode treats `device_id` as the full location path (UMH's Modified Parris Method, where `enterprise:site:area` → `enterprise.site.area` and the edge node is only a session identity). That is correct for UMH-published streams, but a **native / brownfield** Sparkplug producer uses `Group / EdgeNode / Device` as genuine nested levels — the `device_id` is just a device name. With the default, such device data lands at the top of the hierarchy and identically-named devices on different edge nodes collide. Set `include_edge_node_in_location: true` to put each device under the edge node that owns it:

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "FactoryA"
    include_edge_node_in_location: true
```

| Topic                                 | `include_edge_node_in_location: false` (default) | `true`                |
| ------------------------------------- | ------------------------------------------------ | --------------------- |
| `spBv1.0/g/DDATA/Line1/IO Controller` | `IO_Controller`                                  | `Line1.IO_Controller` |
| `spBv1.0/g/DDATA/Line2/IO Controller` | `IO_Controller` (collides with Line1)            | `Line2.IO_Controller` |
| `spBv1.0/g/NDATA/Line1` (node-level)  | `Line1`                                          | `Line1` (unchanged)   |

Leave it `false` for Parris-encoded publishers, whose `device_id` already carries the full location path — enabling it there would prepend the edge node and corrupt the path.

> **No-rebuild workaround:** on the default template you can achieve the same nesting in the bridge's `tag_processor` by setting `location_path` from `spb_edge_node_id_sanitized` and `virtual_path` from `spb_device_id_sanitized`.

### Extension Decoding

Sparkplug B lets publishers carry custom data in proto2 extensions of two messages: `Payload.MetaData` (per-metric metadata) and `Payload.MetricValueExtension` (the metric value). The standard decode keeps those bytes but cannot name them, because they are not in the built-in schema. Supply the extension definitions and the plugin decodes them per metric.

| Field                          | Type     | Default | Description                                                                                     |
| ------------------------------ | -------- | ------- | ----------------------------------------------------------------------------------------------- |
| `decode_extensions.extensions` | `string` | `""`    | Inline proto2 schema declaring the extensions to decode. Empty (the default) disables decoding. |

Write only the `package` and `extend` blocks, plus any custom message types or well-known-type imports (for example `google/protobuf/timestamp.proto`). The plugin compiles the snippet as proto2 and adds the Sparkplug import, so do **not** write a `syntax` line or import the Sparkplug schema yourself. The extendees must use these fully-qualified names:

* `org.eclipse.tahu.protobuf.Payload.MetaData`
* `org.eclipse.tahu.protobuf.Payload.MetricValueExtension`

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "DeviceLevelTest"
    decode_extensions:
      extensions: |
        package example;
        extend org.eclipse.tahu.protobuf.Payload.MetaData {
          optional int64 extra_value = 9;
        }
```

For each metric that carries an extension, the plugin attaches:

* `spb_ext_<field>` — one metadata key per scalar extension (for example, `spb_ext_extra_value`), usable directly in a `tag_processor` with no protobuf handling.
* `spb_metric_decoded` — the full decoded metric as a JSON string, where extensions appear as `[package.field]` keys. Use it for message-typed extensions: `JSON.parse(msg.meta.spb_metric_decoded)`.

Metrics without an extension are emitted unchanged. The snippet compiles once at startup; an unparseable snippet, a snippet with no extensions, or two scalar extensions whose names map to the same `spb_ext_*` key fails the bridge at startup with the offending line. Extensions are a proto2-only feature, so the snippet must be proto2; the device's own implementation language is irrelevant, since the wire format is identical.

***

## Technical Details

### Operating Roles Explained

The Sparkplug B input plugin offers three operating roles that automatically configure all necessary settings. These roles provide clear choices for different deployment scenarios while eliminating the complexity of understanding Sparkplug B roles.

#### Role Overview

| Role                | Name                     | Description                             | Safe for Brownfield |
| ------------------- | ------------------------ | --------------------------------------- | ------------------- |
| `secondary_passive` | Secondary Host (Muted)   | Read-only consumer, no rebirth commands | ✅ Yes (Default)     |
| `secondary_active`  | Secondary Host (Unmuted) | Consumer that can request rebirths      | ✅ Yes               |
| `primary`           | Primary Host             | Full host with STATE publishing         | ⚠️ May conflict     |

#### Role: `secondary_passive` (Default)

**What it does:**

* Operates as a read-only Secondary Host
* **Does NOT send rebirth commands** (fully passive)
* Does NOT publish STATE messages
* Safe to run multiple instances for scalability
* Consumes Sparkplug B messages from the configured group without interfering (set `subscription.groups: ["+"]` to consume every group)

**When to use:**

* ✅ **Default choice** - Safest option for any deployment
* ✅ **Brownfield deployments** - Existing infrastructure remains undisturbed
* ✅ **Multi-consumer environments** - Prevents rebirth storms
* ✅ **Uncertain scenarios** - When you're not sure about the infrastructure

**Configuration:**

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "FactoryA"
    # role: "secondary_passive" is the default
```

**Technical behavior:** Pure consumer role with no command publishing capabilities.

#### Role: `secondary_active`

**What it does:**

* Operates as an active Secondary Host
* **Can send rebirth commands** when needed
* Does NOT publish STATE messages
* Can run multiple instances (though rebirth storms possible)
* Actively manages alias resolution through rebirth requests

**When to use:**

* ✅ **Single consumer deployments** - You're the only Sparkplug B consumer
* ✅ **Controlled environments** - You understand the rebirth implications
* ✅ **Active data management needed** - You need to request fresh BIRTH certificates

**Configuration:**

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "FactoryA"
    role: "secondary_active"
```

**Technical behavior:** Secondary Host with NCMD/DCMD publishing for rebirth requests.

#### Role: `primary`

**What it does:**

* Operates as the Primary Host per Sparkplug B specification
* Publishes STATE messages for Edge Node coordination
* Monitors sequence numbers and manages sessions
* Single instance only (multiple Primary Hosts conflict)
* Full control over Edge Node behavior

**When to use:**

* ✅ **Greenfield deployments** - UMH is your only Sparkplug B application
* ✅ **Full control needed** - You want Edge Nodes to buffer data when offline
* ✅ **Spec compliance required** - Strict Sparkplug B v3.0 compliance

**Configuration:**

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "FactoryA"
      edge_node_id: "UMH_Primary"  # Used as the Sparkplug v3.0 host_id
    role: "primary"
```

> For the `primary` role, `identity.edge_node_id` is the Sparkplug v3.0 `host_id`. With the config above, the plugin publishes STATE messages on `spBv1.0/STATE/UMH_Primary`. The startup logs make this explicit: `Primary Host: using identity.edge_node_id='UMH_Primary' as Sparkplug v3.0 host_id for STATE topic spBv1.0/STATE/UMH_Primary`. The field is named `edge_node_id` for consistency with the secondary roles, but in `primary` mode it identifies the host.

**Technical behavior:** Full Primary Host with STATE publishing and session management.

#### Understanding Rebirth Storms

**What is a rebirth storm?** A cascading effect that occurs when multiple Secondary Hosts simultaneously request rebirths from Edge Nodes, potentially overwhelming the MQTT infrastructure with redundant BIRTH messages.

**Scenario:**

1. Edge Node publishes DDATA with aliases
2. Multiple consumers don't have the alias mappings
3. All consumers simultaneously request rebirth
4. Edge Node publishes NBIRTH/DBIRTH for each request
5. Network and broker become saturated

**Prevention:**

* Use `secondary_passive` role (default) in multi-consumer environments
* Only use `secondary_active` when you're the sole consumer
* Coordinate rebirth requests if multiple active consumers are necessary

> **How throttling prevents rebirth storms.** This plugin rate-limits REBIRTH commands per node via `birth_request_throttle` (default `1s`). Without it, the alias-recovery path would loop until the next BIRTH arrives: every DATA in the round-trip window references the same uncached aliases and triggers another rebirth. Suppressed rebirths log at `info` when the throttle has been active longer than 100ms; same-dispatch co-fires (multiple reasons triggered by one DATA) log at `debug`. Throttling is an implementation choice, not Sparkplug B v3.0 behavior; the spec uses MAY for the Host's rebirth obligation and does not specify pacing. Set `birth_request_throttle: 0` to disable.

#### Choosing the Right Role

**Start with `secondary_passive` if:**

* You have any existing Sparkplug B infrastructure
* Multiple applications consume the same data
* You're unsure about the deployment environment
* You want the safest, most compatible option

**Consider `secondary_active` if:**

* You're the only Sparkplug B consumer
* You need to actively request BIRTH certificates
* You understand and can manage rebirth timing
* Network bandwidth isn't a concern

**Use `primary` only if:**

* You're building a greenfield Sparkplug B system
* No other Primary Hosts exist in your infrastructure
* You need Edge Nodes to respect your online/offline state
* You require full Sparkplug B specification compliance

### Advanced Configuration Options

For users who need to override the default subscription behavior, `subscription.groups` lets you listen to multiple groups or to every group on the broker:

```yaml
input:
  sparkplug_b:
    mqtt:
      urls: ["tcp://localhost:1883"]
    identity:
      group_id: "FactoryA"
      edge_node_id: "CustomHost"  # Optional for secondary role
    role: "secondary_passive"

    # Override the default subscription filter
    subscription:
      groups: ["FactoryA", "FactoryB"]   # Listen to several groups
      # groups: ["+"]                      # Or: subscribe to every group (MQTT wildcard)

    # Future options (planned):
    # include_data_contract_in_device_id: true  # Add data contract to device ID
```

When `subscription.groups` is omitted or empty, the plugin filters to `identity.group_id` (`spBv1.0/<group_id>/#`). Set this field only when you need to listen to groups other than your identity, or to opt back into the all-groups behavior with `["+"]`.

### STATE Topic Behavior (primary role only)

When using `primary` role, the plugin publishes STATE messages according to Sparkplug B v3.0:

```
spBv1.0/STATE/<host_id>
```

* The `edge_node_id` is used as the `host_id`
* STATE topics do NOT include `group_id` (per specification)
* This allows Edge Nodes across all groups to detect the Primary Host

### Deployment Considerations

#### Multiple Instance Support

**`secondary_passive` role (default)**: ✅ **Safe for multiple instances** - No STATE conflicts, no rebirth storms, load balancing friendly

**`secondary_active` role**: ✅ **Can run multiple instances** - But be aware of potential rebirth storms

**`primary` role**: ⚠️ **Single instance only** - Publishes STATE messages for host arbitration

### Metadata Enrichment

The plugin attaches comprehensive Sparkplug-specific metadata fields to each output message. These are organized into **primary fields** (commonly used) and **secondary fields** (for advanced use cases):

#### Primary Metadata Fields

These are the main metadata fields that most users will need for processing Sparkplug messages:

* `spb_message_type`: The Sparkplug message type (e.g., "NBIRTH", "NDATA", "NDEATH", "DBIRTH", "DDATA", "DDEATH")
* `spb_group_id`: The Sparkplug Group ID of the source message
* `spb_edge_node_id`: The Edge Node ID (equipment or gateway name)
* `spb_device_id`: The Device ID (for metrics from devices under an edge node, empty for node-level messages)
* `spb_device_key`: Combined device identifier in format "group\_id/edge\_node\_id" or "group\_id/edge\_node\_id/device\_id"
* `spb_topic`: The original MQTT topic the message was received from
* `spb_metric_name`: The Sparkplug metric name (either from name field or alias\_X format)

#### Secondary Metadata Fields (Advanced)

These fields provide additional Sparkplug context and are primarily for debugging or advanced processing:

* `spb_group`: Same as `spb_group_id` (alternative field name)
* `spb_edge_node`: Same as `spb_edge_node_id` (alternative field name)
* `spb_device`: Same as `spb_device_id` (alternative field name)
* `spb_sequence`: The sequence number of the Sparkplug message
* `spb_bdseq`: The birth-death sequence number of the session
* `spb_timestamp`: The timestamp (in epoch ms) of the metric. The per-metric timestamp is used when present, falling back to the payload-level timestamp otherwise. This preserves each sample's own capture time — important for buffered or historical metrics that share one payload but were sampled at different moments.
* `spb_datatype`: The Sparkplug data type of the metric (e.g. "Int32", "Double", "Boolean")
* `spb_alias`: The alias number of the metric (for debugging alias resolution)
* `spb_is_historical`: Set to "true" if the metric was flagged as historical

#### Special Message Types

For STATE messages, the plugin sets:

* `event_type`: "state\_change"
* `node_state`: The state value ("ONLINE" or "OFFLINE")

For NDEATH/DDEATH messages, the plugin sets:

* `event_type`: "device\_offline"

#### UMH Conversion Metadata (Optional)

When UMH conversion is successful, additional metadata is added:

* `umh_conversion_status`: "success", "failed", "skipped\_insufficient\_data", or "failed\_no\_value"
* `umh_location_path`: Converted UMH location path (dots format, sanitized)
* `umh_tag_name`: UMH tag name extracted from metric name (sanitized)
* `umh_data_contract`: UMH data contract (e.g., "\_raw", "\_historian")
* `umh_virtual_path`: UMH virtual path if present in metric name (sanitized)
* `umh_conversion_error`: Error message if conversion failed

**Automatic Sanitization**: The plugin handles Sparkplug metric names to ensure UMH compatibility through a clear architectural boundary at the format conversion layer:

1. **Input Processing**: Sparkplug messages are received with their original metric names intact (no preprocessing)
2. **Format Conversion**: The format converter parses and sanitizes during the conversion to UMH format:
   * Splits metric names to extract virtual paths (priority: colon > slash > dot)
   * Example: `vpath:segment:metric` → virtual\_path=`vpath.segment`, tag\_name=`metric`
   * Trims leading/trailing separators before parsing
3. **Sanitization**: Applied only at the conversion boundary to preserve data integrity

Sanitization rules (applied during conversion):

* Hierarchy separators (`/`, `:`) are converted to dots (`.`) to preserve structure
* Invalid characters are replaced with underscores (`_`)
* Valid characters are: `a-z`, `A-Z`, `0-9`, `.`, `_`, `-`
* Multiple consecutive dots are collapsed into a single dot
* Leading and trailing dots are removed

Examples:

* `Refrigeration/Tower1/Pumps/chemHOA` → `Refrigeration.Tower1.Pumps.chemHOA`
* `Device@Name#123` → `Device_Name_123`
* `Area/Zone@1/Device#2` → `Area.Zone_1.Device_2`
* `Path//with///slashes` → `Path.with.slashes` (double dots prevented)
* `/hello123/test/` → virtual\_path=`hello123`, tag\_name=`test` (slashes trimmed, then split)
* `vpath:segment:metric` → virtual\_path=`vpath.segment`, tag\_name=`metric` (colons to dots)

When sanitization occurs, the plugin preserves the original values in metadata:

* `spb_original_metric_name`: The original metric name before sanitization (if different from spb\_metric\_name)
* `spb_original_device_id`: The original device ID before sanitization (if different from spb\_device\_id)

The sanitized values are available in the UMH metadata fields:

* `umh_location_path`: The sanitized location path (from device ID)
* `umh_virtual_path`: The sanitized virtual path (from metric name)
* `umh_tag_name`: The sanitized tag name (from metric name)

This ensures that messages with non-compliant Sparkplug names are automatically converted to valid UMH topics while preserving the original values for reference.

**Usage Recommendation**: Use the **primary metadata fields** for most processing logic. The alternative `spb_` prefixed fields are provided for consistency and advanced debugging scenarios.

### Message Processing

The Sparkplug B input plugin **always splits metrics** into individual messages to ensure UMH-Core format compliance. Each Sparkplug metric becomes a separate Benthos message for downstream processing.

*Note: This behavior is required for UMH-Core format and cannot be disabled.*

## Stateless Architecture Considerations

### Understanding bdSeq (Birth-Death Sequence) in Sparkplug B

The Sparkplug B input plugin processes **bdSeq** values from incoming Edge Node messages. Understanding bdSeq behavior is important for monitoring Edge Node session lifecycle:

**Specification-Compliant Edge Nodes:**

* bdSeq should increment by +1 for each new MQTT session
* Example: Session 1: bdSeq=0 → Session 2: bdSeq=1 → Session 3: bdSeq=2

**Stateless Edge Nodes (like Benthos Sparkplug B Output):**

* bdSeq may reset to 0 on Edge Node component restart
* This is common in container-based or stateless Edge Node implementations
* Still compliant within individual component lifecycles

### Impact on Input Processing

**What to Expect:**

* Edge Nodes may send bdSeq=0 after restarts (not necessarily the first session)
* bdSeq jumps or resets indicate Edge Node restarts or different implementations
* This is normal behavior for stateless architectures

**Recommendation:** The input plugin handles both persistent and stateless Edge Node bdSeq patterns correctly. No special configuration is needed - the plugin automatically adapts to different Edge Node implementations and their bdSeq behaviors.


# OPC UA (Input)

The plugin browses each configured NodeID and subscribes to the variables below it. It follows all hierarchical references, including `Organizes`, `HasComponent` and `HasProperty`, down to 25 levels. It browses into both Objects and Variables, and subscribes only to Variables.

Tags whose DataType is null are not subscribed.

**Datatypes**

The plugin has been rigorously tested with an array of datatypes, both as single values and as arrays. The following datatypes have been verified for compatibility:

* `Boolean`
* `Byte`
* `DateTime`
* `Double`
* `Enumeration`
* `ExpandedNodeId`
* `Float`
* `Guid`
* `Int16`
* `Int32`
* `Int64`
* `Integer`
* `LocalizedText`
* `NodeId`
* `Number`
* `QualifiedName`
* `SByte`
* `StatusCode`
* `String`
* `UInt16`
* `UInt32`
* `UInt64`
* `UInteger`
* `ByteArray`
* `ByteString`
* `Duration`
* `LocaleId`
* `UtcTime`
* `Variant`
* `XmlElement`

There are specific datatypes which are currently not supported by the plugin and attempting to use them will result in errors. These include:

* Two-dimensional arrays
* UA Extension Objects
* Variant arrays (Arrays with multiple different datatypes)

**Authentication and Security**

In benthos-umh, we design security and authentication to be as robust as possible while maintaining flexibility. The software automates the process of selecting the highest level of security offered by an OPC-UA server for the selected Authentication Method, but the user can specify their own Security Policy / Security Mode if they want (see below at Configuration options)

**Supported Authentication Methods**

* **Anonymous**: No extra information is needed. The connection uses the highest security level available for anonymous connections.
* **Username and Password**: Specify the username and password in the configuration. The client opts for the highest security level that supports these credentials.
* **Certificate (Future Release)**: Certificate-based authentication is planned for future releases.

**Metadata outputs**

The plugin provides metadata for each message, that can be used to create a topic for the output, as shown in the example above. The metadata can also be used to create a unique identifier for each message, which is useful for deduplication.

| Metadata                 | Description                                                                                                                                          |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `opcua_tag_name`         | The sanitized ID of the Node that sent the message. This is always unique between nodes                                                              |
| `opcua_tag_path`         | A dot-separated path to the tag, created by joining the BrowseNames.                                                                                 |
| `opcua_tag_group`        | Other name for `opcua_tag_path`                                                                                                                      |
| `opcua_tag_type`         | The data type of the node optimized for benthos, which can be either a number, string or bool. For the original one, check out `opcua_attr_datatype` |
| `opcua_source_timestamp` | The SourceTimestamp of the OPC UA node                                                                                                               |
| `opcua_server_timestamp` | The ServerTimestamp of the OPC UA node                                                                                                               |
| `opcua_attr_nodeid`      | The NodeID attribute of the Node as a string                                                                                                         |
| `opcua_attr_nodeclass`   | The NodeClass attribute of the Node as a string                                                                                                      |
| `opcua_attr_browsename`  | The BrowseName attribute of the Node as a string                                                                                                     |
| `opcua_attr_description` | The Description attribute of the Node as a string                                                                                                    |
| `opcua_attr_accesslevel` | The AccessLevel attribute of the Node as a string                                                                                                    |
| `opcua_attr_datatype`    | The DataType attribute of the Node as a string                                                                                                       |
| `opcua_attr_statuscode`  | The OPC UA quality/status code for the data value (e.g., "Good", "BadNodeIdUnknown"). Indicates the reliability of the value.                        |

**Note:** String values are delivered exactly as the server sends them. However, `tag_processor` auto-detection converts numeric-looking strings (e.g., serial codes) back to lossy numbers. To keep them as strings, set `msg.meta.datatype = "string"` for those tags in the processing section.

Taking as example the following OPC-UA structure:

```
Root
└── ns=2;s=FolderNode
    ├── ns=2;s=Tag1
    ├── ns=2;s=Tag2
    └── ns=2;s=SubFolder
        ├── ns=2;s=Tag3
        └── ns=2;s=Tag4
```

Subscribing to `ns=2;s=FolderNode` would result in the following metadata:

| `opcua_tag_name` | `opcua_tag_group`      |
| ---------------- | ---------------------- |
| `Tag1`           | `FolderNode`           |
| `Tag2`           | `FolderNode`           |
| `Tag3`           | `FolderNode.SubFolder` |
| `Tag4`           | `FolderNode.SubFolder` |

**Configuration Options**

The following options can be specified in the `benthos.yaml` configuration file:

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
    username: 'your-username'  # optional (default: unset)
    password: 'your-password'  # optional (default: unset)
    insecure: false | true # DEPRECATED, see below
    securityMode: None | SignAndEncrypt # optional (default: unset)
    securityPolicy: None | Basic128Rsa15 | Basic256 | Basic256Sha256  # optional (default: unset)
    serverCertificateFingerprint: 'sha3-fingerprint-of-cert' # optional (default: unset)
    clientCertificate: 'your-fixed-base64-encoded-certificate' # optional (default: unset)
    userCertificate: 'base64-encoded-user-PEM-certificate' # optional (default: unset)
    userPrivateKey: 'base64-encoded-user-PEM-private-key' # optional (default: unset)
    subscribeEnabled: false | true # optional (default: false)
    useHeartbeat: false | true # optional (default: false)
    pollRate: 1000 # optional (default: 1000) The rate in milliseconds at which to poll the OPC UA server when not using subscriptions
    autoReconnect: false | true # optional (default: false)
    reconnectIntervalInSeconds: 5 # optional (default: 5) The rate in seconds at which to reconnect to the OPC UA server when the connection is lost
    # Advanced options - only modify if you understand your OPC UA server's behavior
    queueSize: 10 # optional (default: 10) Number of subscription notifications to buffer
    samplingInterval: 0.0 # optional (default: 0.0) Sampling interval in milliseconds for subscriptions
```

**Endpoint**

You can specify the endpoint in the configuration file. Node endpoints are automatically discovered and selected based on the authentication method.

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
```

**Node IDs**

You can specify the node IDs in the configuration file (currently only namespaced node IDs are supported):

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
```

**Username and Password**

If you want to use username and password authentication, you can specify them in the configuration file:

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
    username: 'your-username'
    password: 'your-password'
```

**User Certificate and Private Key**

* **Keys**: `userCertificate`, `userPrivateKey`
* **Description**: Credentials for User certificate-based authentication.
* `userCertificate`: Base64-encoded certificate in either PEM (.pem) or DER (.der) format.
* `userPrivateKey`: Base64-encoded private key in PEM (.pem) format only.
* Certificate-based authentication provides stronger security than username/password for high-security environments.
* Proper protection of the private key and certificate validation on both client and server are essential.
* **Configuration Example**:

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
    securityMode: SignAndEncrypt
    securityPolicy: Basic256Sha256
    userCertificate: 'base64-encoded certificate (.pem or .der)'
    userPrivateKey: 'base64-encoded private key (.pem only)'
```

**Security Options**

> To ensure a fully secure connection, you must explicitly configure all of the following security options. However, if these settings seem overwhelming, you can leave them unspecified. In that case, **benthos-umh** will automatically scan for and connect to available endpoints until it succeeds—and then it will log the recommended security settings for your future configuration.

OPC UA supports various security modes and security policies. These options define how messages are signed or encrypted and which cryptographic algorithms are used. In the configuration, you can specify the following:

* **Security Mode**: Defines the level of security applied to messages.
  * **Key**: `securityMode`
  * **Values**:
    * **None**: No security is applied; messages are neither signed nor encrypted.
    * **Sign**: Messages are signed for integrity and authenticity but not encrypted.
    * **SignAndEncrypt**: The highest level of security where messages are both signed and encrypted.
* **Security Policy**: Specifies the cryptographic algorithms used for signing/encrypting messages.
  * **Key**: `securityPolicy`
  * **Values**:
    * **None**: No security applied.
    * **Basic128Rsa15** (**deprecated**): Insecure due to SHA-1. Often disabled on servers by default.
    * **Basic256** (**deprecated**): Insecure due to SHA-1. Often disabled on servers by default.
    * **Basic256Sha256**: Recommended. Uses SHA-256 and provides stronger security.
* **Server Certificate Fingerprint**:
  * **Key**: `serverCertificateFingerprint`
  * **Description**: A SHA3-512 hash of the server’s certificate, used to verify you are connecting to the correct server.
  * If you specify this field, the client will verify that the server’s certificate matches the given fingerprint. If there’s a mismatch, the connection is rejected.
  * If omitted while **still using encryption** (`Sign` or `SignAndEncrypt`), the client will attempt to connect and then **log** the server’s actual fingerprint. You can copy that fingerprint into your config to be certain you’re connecting to the intended server.
  * In future releases, omitting the fingerprint may become a warning or block deployment in certain environments.
* **Client Certificate**:
  * **Key**: `clientCertificate`
  * **Description**: A Base64‐encoded PEM bundle (certificate + private key).
  * When using encryption (`Sign` or `SignAndEncrypt`), the client must present a certificate to the server. If you **do not** provide one, the system **auto‐generates** a random certificate at startup.
  * The auto‐generated certificate is logged in Base64 so you can copy/paste it into your configuration. This allows the server to trust the same client certificate across restarts instead of generating a new one each time.
  * Whenever a certificate is created, and the OPC UA server's settings do not allow automatic acceptance of client certificates, you will need to manually trust the client certificate in the server's settings. The client's name will be displayed, enabling you to uniquely identify it in the certificate list.

If you want to connect with security options, you will at least have to provide the following sample:

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
    securityMode: SignAndEncrypt
    securityPolicy: Basic256Sha256
    serverCertificateFingerprint: 'sha3-fingerprint-of-cert'
    clientCertificate: 'your-fixed-base64-encoded-certificate' # optional but recommended
```

**Insecure Mode**

This is now deprecated. By default, benthos-umh will now connect via SignAndEncrypt and Basic256Sha256 and if this fails it will fall back to insecure mode.

**Pull and Subscribe Methods**

Benthos-umh supports two modes of operation: pull and subscribe. In pull mode, it pulls all nodes every second, regardless of changes. In subscribe mode, it only sends data when there's a change in value, reducing unnecessary data transfer.

| Method    | Advantages                                                                                                                                                                                                                                         | Disadvantages                                                                                                                         |
| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| Pull      | <p>- Provides real-time data visibility, e.g., in MQTT Explorer.<br>- Clearly differentiates between 'no data received' and 'value did not change' scenarios, which can be crucial for documentation and proving the OPC-UA client's activity.</p> | - Results in higher data throughput as it pulls all nodes at the configured poll rate (default: every second), regardless of changes. |
| Subscribe | - Data is sent only when there's a change in value, reducing unnecessary data transfer.                                                                                                                                                            | - Less visibility into real-time data status, and it's harder to differentiate between no data and unchanged values.                  |

```yaml
input:
  opcua:
    endpoint: 'opc.tcp://localhost:46010'
    nodeIDs: ['ns=2;s=IoTSensors']
    subscribeEnabled: true
```

**UseHeartbeat**

If you are unsure if the OPC UA server is actually sending new data, you can enable `useHeartbeat` by setting it to true. It will automatically subscribe to the OPC UA server time, and will re-connect automatically if it does not receive an update within 10 seconds.

```yaml
input:
  opcua:
    useHeartbeat: true
```

**Browse Hierarchical References (Option until version 0.5.2)**

**NOTE**: This property is removed in version 0.6.0 and made as a standard way to browse OPCUA nodes. From version 0.6.0 onwards, opcua\_plugin will browse all nodes with Hierarchical References.

The plugin offers an option to browse OPCUA nodes by following Hierarchical References. By default, this feature is disabled (`false`), which means the plugin will only browse a limited subset of reference types, including:

* `HasComponent`
* `Organizes`
* `FolderType`
* `HasNotifier`

When set to `true`, the plugin will explore a broader range of node references. For a deeper understanding of the different reference types, refer to the [Standard References Type documentation](https://qiyuqi.gitbooks.io/opc-ua/content/Part3/Chapter7.html).

**Recommendation**: Enable this option (`browseHierarchicalReferences: true`) for more comprehensive node discovery.

```yaml
input:
  opcua:
    browseHierarchicalReferences: true
```

**Auto Reconnect**

If the connection is lost, the plugin will automatically reconnect to the OPC UA server. This is useful if the OPC UA server is unstable or if the network is unstable.

```yaml
input:
  opcua:
    autoReconnect: true
```

**Reconnect Interval**

The interval in seconds at which to reconnect to the OPC UA server when the connection is lost. This is only used if `autoReconnect` is set to true.

```yaml
input:
  opcua:
    reconnectIntervalInSeconds: 5
```

**Advanced Configuration Options**

> **WARNING**: The following options are for advanced users only. Modifying these settings without understanding your OPC UA server's behavior and limitations can lead to performance issues, memory problems, or connection failures. Setting those configuration options, does **not** mean that the server will respect that settings. **Leave these at their default values unless you have specific performance requirements and understand the implications.**

**Queue Size**

The `queueSize` parameter controls how many subscription notifications are buffered internally before being processed.

* **Key**: `queueSize`
* **Default**: `10`
* **Description**: This parameter determines the internal buffer size for handling subscription notifications from the OPC UA server. A larger queue can handle burst notifications but uses more memory.

**Risks of incorrect configuration**:

* **Too small**: May cause notification loss during high-frequency periods
* **Too large**: Excessive memory usage that could impact system performance

```yaml
input:
  opcua:
    queueSize: 20  # Only increase if you experience notification loss
```

**Sampling Interval**

The `samplingInterval` parameter controls how frequently the OPC UA server samples the underlying data source for subscription notifications.

* **Key**: `samplingInterval`
* **Default**: `0.0` (fastest possible sampling)
* **Unit**: Milliseconds
* **Description**: Defines the rate at which the server samples the data source. A value of `0.0` means "as fast as possible" according to the server's capabilities. Higher values reduce sampling frequency.

**Server behavior dependency**:

* Some servers ignore this parameter and use their own internal sampling rates
* The actual sampling rate depends on the server's implementation and capabilities

```yaml
input:
  opcua:
    samplingInterval: 1000.0  # Sample every 1 second instead of as fast as possible
```

## Server Profiles and Performance Tuning

Server profiles automatically optimize OPC UA connection parameters based on detected server type. The system queries the server's manufacturer and product information to select the best profile.

### What Are Server Profiles?

Profiles auto-optimize OPC UA connection parameters based on detected server type. The system queries the server's manufacturer and product information (ServerInfo nodes) to select appropriate tuning.

**Two performance parameters:**

1. **Workers (Browse Phase)**: Concurrent goroutines discovering the OPC UA node tree
   * Example: 10,000 nodes with 5 workers ≈ 2,000 Browse calls per worker vs 10,000 sequential calls
2. **Batch Size (Subscribe Phase)**: Nodes per CreateMonitoredItems call
   * Larger batches = faster subscription setup
   * Too large = server rejection or performance degradation

### Available Profiles

Profiles are automatically detected based on server manufacturer/product information:

* **Auto**: Unknown servers (conservative defaults)
* **High-Performance**: Manual override for known high-capacity infrastructure
* **Ignition**: Inductive Automation Ignition Gateway
* **Kepware**: PTC Kepware KEPServerEX
* **S7-1200**: Siemens S7-1200 PLCs (hardware limit: 1000 monitored items)
* **S7-1500**: Siemens S7-1500 PLCs (hardware limit: 10000 monitored items)
* **Prosys**: Prosys Simulation Server

**Profile values** are defined in [`opcua_plugin/server_profiles.go`](https://github.com/united-manufacturing-hub/benthos-umh/tree/staging/opcua_plugin/server_profiles.go) with vendor documentation citations.

**Key insight**: Profile values are production-safe limits from vendor docs and real-world testing, NOT server-reported theoretical maximums. Example: S7-1200 reports `MaxMonitoredItemsPerCall=1000` but profile uses 100 (values >200 cause 50× performance degradation per [Siemens docs](https://cache.industry.siemens.com/dl/files/846/109755846/att_1163306/v4/109755846_TIA_Portal_OPC_UA_system_limits.pdf)).

### Performance Impact

Examples of profile-specific optimizations:

* **S7-1200**: Batch size limited to 100 (server reports 1000, but >200 causes 50× degradation - [Siemens docs](https://cache.industry.siemens.com/dl/files/846/109755846/att_1163306/v4/109755846_TIA_Portal_OPC_UA_system_limits.pdf))
* **Ignition/Kepware**: Batch size of 1000 enables 10× faster subscription setup compared to conservative 100. But limited browse worker amount as Eclipse Milo, the OPC UA library that Ignition uses, has a limit on concurrent operations per OPC UA session fo by default 64.
* **Prosys**: Batch size of 800 prevents simulation server unresponsiveness with large node counts

### Dynamic Worker Scaling

During the Browse phase, the system automatically adjusts worker concurrency based on measured server response time. This optimization balances performance (faster browsing with more workers) against server load (preventing overload).

**How it works:**

1. **Measurement**: The system samples response times from 5 consecutive Browse operations
2. **Target latency**: 250ms per Browse request (default)
3. **Scaling logic**:
   * If average response > 250ms → reduce workers by 1 (down to profile's MinWorkers)
   * If average response < 250ms → increase workers by 1 (up to profile's MaxWorkers)
   * If average response ≈ 250ms → no adjustment

**Bounds enforcement**: Worker count always respects the ServerProfile's MinWorkers and MaxWorkers limits. These limits are **global across ALL NodeIDs** being browsed - if you configure 3 NodeIDs with MaxWorkers=20, all 3 NodeIDs share a single pool of 20 workers (not 20 workers per NodeID = 60 total). The system cannot scale beyond profile-defined hardware constraints.

**Why this matters:**

* **Performance**: Automatically finds optimal worker count for each server's capability
* **Safety**: Prevents server overload by reducing workers when response times increase
* **Gradual adaptation**: Fine-grained control with ±1 worker adjustments provides smoother scaling
* **Adaptability**: Adjusts to changing server conditions during long Browse operations
* **Global resource management**: MaxWorkers controls total system concurrency regardless of NodeID count, preventing server overload when browsing multiple large folders simultaneously

Example: An S7-1500 profile with MaxWorkers=50 and **2 NodeIDs** might start with 10 workers **shared across both browse operations**. If Browse responses average 100ms (< 250ms target), the shared worker pool gradually increases to 50 maximum **for all browse operations combined**: 10 → 11 → 12 → 13. If responses slow to 400ms, workers reduce **across all active browse operations**: 13 → 12 → 11 to maintain stability.

### Manual Profile Override

Override auto-detection by setting profile explicitly:

```yaml
input:
  opcua:
    endpoint: "opc.tcp://10.0.0.1:4840"
    profile: "high-performance"  # Override auto-detection
```

Use case: Force high-performance profile when server manufacturer string doesn't match known vendors but infrastructure is validated for aggressive batching.

## Metrics

### opcua\_subscription\_failures\_total

Counter tracking OPC UA subscription failures by reason and node ID.

**Labels:**

* `reason`: Failure classification (`filter_not_allowed`, `filter_unsupported`, `node_id_unknown`, `node_id_invalid`, `other`)
* `node_id`: OPC UA NodeID that failed to subscribe

**Common Failure Reasons:**

| Reason               | Description                     | Resolution                                                     |
| -------------------- | ------------------------------- | -------------------------------------------------------------- |
| `filter_not_allowed` | Server rejected deadband filter | Only numeric types (Int, UInt, Float, Double) support deadband |
| `filter_unsupported` | Server doesn't support filters  | Disable deadband: `deadbandType: none`                         |
| `node_id_unknown`    | NodeID doesn't exist            | Verify NodeID configuration                                    |
| `node_id_invalid`    | NodeID syntax error             | Fix NodeID string format                                       |
| `other`              | Other subscription errors       | Check server logs                                              |


# Modbus

The Modbus plugin facilitates communication with various types of Modbus devices. It supports reading from four types of registers: coils, discrete inputs, holding registers, and input registers. Each data item configuration requires specifying the register type, address, and the data type to be read. The plugin supports multiple data types including integers, unsigned integers, floats, and strings across different sizes and formats.

Data reads can be configured to occur at a set interval, allowing for consistent data polling. Advanced features like register optimization and workarounds for device-specific quirks are also supported to enhance communication efficiency and compatibility.

**Metadata Outputs**

For each read operation, the plugin outputs detailed metadata that includes various aspects of the read operation, which can be utilized to effectively tag, organize, and utilize the data within a system. This metadata encompasses identifiers, data types, and register specifics to ensure precise tracking and utilization of the Modbus data.

Below is the extended metadata output schema provided by the plugin:

| Metadata                     | Description                                                                     |
| ---------------------------- | ------------------------------------------------------------------------------- |
| `modbus_tag_name`            | Sanitized tag name, with special characters removed for compatibility.          |
| `modbus_tag_name_original`   | Original tag name, as defined in the device configuration.                      |
| `modbus_tag_datatype`        | Original Modbus data type of the tag.                                           |
| `modbus_tag_datatype_json`   | Data type of the tag suitable for JSON representation: number, bool, or string. |
| `modbus_tag_address`         | String representation of the tag's Modbus address.                              |
| `modbus_tag_length`          | The length of the tag in registers, relevant for string or array data types.    |
| `modbus_tag_register`        | The specific Modbus register type where the tag is located.                     |
| `modbus_tag_slaveid`         | The slave ID where the tag is coming from                                       |
| `modbus_tag_unified_address` | Unified dotted address string (e.g. `temperature.holding.100.INT16`)            |

This enhanced metadata schema provides comprehensive data for each read operation, ensuring that users have all necessary details for effective data management and application integration.

**Configuration Options**

Below are the comprehensive configuration options available in the configuration file for the Modbus plugin. This includes settings for device connectivity, data reading intervals, optimization strategies, and detailed data item configurations.

```yaml
input:
  modbus:
    controller: 'tcp://localhost:502'
    transmissionMode: 'TCP'
    slaveIDs:
      - 1
    timeout: '1s'
    busyRetries: 3
    busyRetriesWait: '200ms'
    timeBetweenReads: '1s'
    optimization: 'none'
    byteOrder: 'ABCD'
    addresses:
      - name: "firstFlagOfDiscreteInput"
        register: "discrete"
        address: 1
        type: "BIT"
        output: "BOOL"
      - name: "zeroElementOfInputRegister"
        register: "input"
        address: 0
        type: "UINT16"
```

**Per-Slave Address Example**

When different slaves expose different registers, you can assign addresses to specific slaves:

```yaml
input:
  modbus:
    controller: 'tcp://192.168.1.100:502'
    slaveIDs: [1, 2, 3]
    addresses:
      - name: "temperature"
        register: "holding"
        address: 100
        type: "INT16"
        slaveID: 1          # Only slave 1
      - name: "pressure"
        register: "holding"
        address: 200
        type: "INT16"
        slaveID: 2          # Only slave 2
      - name: "status"
        register: "coil"
        address: 10
        type: "BIT"
        # No slaveID → read from ALL slaves (1, 2, 3)
```

**Controller**

Specifies the network address of the Modbus controller:

```yaml
input:
  modbus:
    controller: 'tcp://localhost:502'
```

**Transmission Mode**

Defines the Modbus transmission mode. Can be "TCP" (default), "RTUOverTCP", "ASCIIOverTCP":

```yaml
input:
  modbus:
    transmissionMode: 'TCP'
```

**Slave IDs**

Configure the modbus slave IDs :

```yaml
input:
  modbus:
    slaveIDs:
      - 1
      - 2
```

For backward compatibility, there is also `slaveID: 1`, which allows setting only a single Modbus slave.

**Retry Settings & Timeout**

Configurations to handle retries in case of communication failures:

```yaml
input:
  modbus:
    busyRetries: 3
    busyRetriesWait: '200ms'
    timeout: '1s'
```

**Time Between Reads**

Defines how frequently the Modbus device should be polled:

```yaml
input:
  modbus:
    timeBetweenReads: '1s'
```

**Optimization**

The Modbus plugin offers several strategies to optimize data read requests, enhancing efficiency and reducing network load when interacting with Modbus devices. These strategies are designed to adjust the organization and batching of requests based on device capabilities and network conditions.

The available optimization strategies are:

* **none**: This is the default setting where no optimization is applied. The plugin groups read requests according to the defined metrics without further optimization. Suitable for systems with simple setups or minimal performance requirements.
* **max\_insert**: Enhances efficiency by collating read requests across all defined metrics and filling in gaps (non-consecutive registers) to minimize the total number of requests. This strategy is ideal for complex systems with numerous data points, as it significantly reduces network traffic and processing time.
* **shrink**: Reduces the size of each request by stripping leading and trailing fields marked with an omit flag. This can decrease the overall data payload and improve processing times, especially when many fields are optional or conditional.
* **rearrange**: Allows rearranging fields between requests to reduce the number of registers accessed while maintaining the minimal number of requests. This strategy optimizes the order of fields to minimize the spread across registers.
* **aggressive**: Similar to "rearrange" but allows mixing of groups. This approach may reduce the number of requests at the cost of accessing more registers, potentially touching more data than necessary to consolidate requests.

Each strategy can be tailored with parameters such as `OptimizationMaxRegisterFill` to control how aggressively the system attempts to optimize data reads. For example, the `max_insert` option can be configured to limit the number of additional registers filled to reduce gaps:

```yaml
input:
  modbus:
    optimization: 'max_insert'
    optimizationMaxRegisterFill: 10
```

Additional Configuration for Optimization Strategies:

* **OptimizationMaxRegisterFill**: Specifies the maximum number of registers the optimizer is allowed to insert between non-consecutive registers in the `max_insert` strategy.

**Byte Order**

The `byteOrder` configuration specifies how bytes within the registers are ordered, which is essential for correctly interpreting the data read from a Modbus device. Different devices or systems may represent multi-byte data types (like integers and floating points) in various byte orders. The options are:

* **ABCD**: Big Endian (Motorola format) where the most significant byte is stored first.
* **DCBA**: Little Endian (Intel format) where the least significant byte is stored first.
* **BADC**: Big Endian with byte swap where bytes are stored in a big-endian order but each pair of bytes is swapped.
* **CDAB**: Little Endian with byte swap where bytes are stored in little-endian order with each pair of bytes swapped.

```yaml
input:
  modbus:
    byteOrder: 'ABCD'
```

**Modbus Workaround**

The Modbus plugin incorporates specific workarounds to address compatibility and performance issues that may arise with various Modbus devices. These workarounds ensure the plugin can operate efficiently even with devices that have unique quirks or non-standard Modbus implementations.

```yaml

input:
  modbus:
    workarounds:
      pauseAfterConnect: '500ms'
      oneRequestPerField: true
      readCoilsStartingAtZero: true
      timeBetweenRequests: '100ms'
      stringRegisterLocation: 'upper'
```

1. **Pause After Connect**

* **Description**: Introduces a delay before sending the first request after establishing a connection.
* **Purpose**: This is particularly useful for slow devices that need time to stabilize a connection before they can process requests.
* **Default**: `0s`
* **Configuration Example**:

  ```yaml
  pauseAfterConnect: '500ms'
  ```

2. **One Request Per Field**

* **Description**: Configures the system to send each field request separately.
* **Purpose**: Some devices may have limitations that prevent them from handling multiple field requests in a single Modbus transaction. Isolating requests ensures compatibility.
* **Default**: `false`
* **Configuration Example**:

  ```yaml
  oneRequestPerField: true
  ```

3. **Read Coils Starting at Zero**

* **Description**: Adjusts the starting address for reading coils to begin at address 0 instead of 1.
* **Purpose**: Certain devices may map their coil addresses starting from 0, which is non-standard but not uncommon.
* **Default**: `false`
* **Configuration Example**:

  ```yaml
  readCoilsStartingAtZero: true
  ```

4. **Time Between Requests**

* **Description**: Sets the minimum interval between consecutive requests to the same device.
* **Purpose**: Prevents the overloading of Modbus devices by spacing out the requests, which is critical in systems where devices are sensitive to high traffic.
* **Default**: `0s`
* **Configuration Example**:

  ```yaml
  timeBetweenRequests: '100ms'
  ```

5. **String Register Location**

* **Description**: Specifies which part of the register to use for string data after byte-order conversion.
* **Options**:
  * `lower`: Uses only the lower byte of each register.
  * `upper`: Uses only the upper byte of each register.
  * If left empty, both bytes of the register are used.
* **Purpose**: Some devices may place string data only in specific byte locations within a register, necessitating this adjustment for correct string interpretation.
* **Default**: Both bytes used.
* **Configuration Example**:

  ```yaml
  stringRegisterLocation: 'upper'
  ```

**Unified Address Format (`unifiedAddresses`)**

The preferred way to configure Modbus addresses is using the unified `unifiedAddresses` string list. Each address is a single string with the format:

```
name.register.address.type[:key=value]*
```

**Required positional segments** (dot-separated):

1. `name` — tag name (no dots allowed)
2. `register` — one of: `coil`, `discrete`, `holding`, `input`
3. `address` — numeric 0–65535
4. `type` — one of: `BIT`, `INT8L`, `INT8H`, `UINT8L`, `UINT8H`, `INT16`, `UINT16`, `INT32`, `UINT32`, `INT64`, `UINT64`, `FLOAT16`, `FLOAT32`, `FLOAT64`, `STRING`

**Optional key-value pairs** (colon-separated):

* `slaveID=<0-255>` — restrict to specific slave ID (default: 0 = all)
* `length=<n>` — register count, only valid for STRING type
* `bit=<0-15>` — bit number, only valid for BIT type
* `scale=<float>` — scaling factor
* `output=<INT64|UINT64|FLOAT64|STRING|BOOL|UINT16>` — output type

```yaml
input:
  modbus:
    controller: 'tcp://localhost:502'
    slaveIDs: [1]
    unifiedAddresses:
      - "temperature.holding.100.INT16"
      - "motor_status.discrete.1.BIT:bit=3"
      - "serial_number.holding.200.STRING:length=10"
      - "pressure.holding.300.FLOAT32:scale=0.1:output=FLOAT64:slaveID=2"
```

**Migration from `addresses` to `unifiedAddresses`**

| Old format                                                                              | New format                            |
| --------------------------------------------------------------------------------------- | ------------------------------------- |
| `name: "temp"`, `register: "holding"`, `address: 100`, `type: "INT16"`                  | `temp.holding.100.INT16`              |
| `name: "flag"`, `register: "discrete"`, `address: 1`, `type: "BIT"`, `bit: 3`           | `flag.discrete.1.BIT:bit=3`           |
| `name: "serial"`, `register: "holding"`, `address: 200`, `type: "STRING"`, `length: 10` | `serial.holding.200.STRING:length=10` |

> **Note**: The `unifiedAddresses` and `addresses` fields are mutually exclusive. Using both will result in an error.

**Addresses (Deprecated)**

> **Deprecated**: Use the `unifiedAddresses` string list format above instead.

The Modbus plugin provides a highly configurable way to specify which data points (addresses) to read from Modbus devices. Each address configuration allows precise definition of what data to read, how it's interpreted, and how it should be scaled or formatted before use.

```yaml
input:
  modbus:
    addresses:
      - name: "firstFlagOfDiscreteInput"
        register: "discrete"
        address: 1
        type: "BIT"
        output: "BOOL"
      - name: "zeroElementOfInputRegister"
        register: "input"
        address: 0
        type: "UINT16"
```

1. **Name**

* **Description**: Identifier for the data point being configured.
* **Configuration Example**:

  ```yaml
  name: "TemperatureSensor"
  ```

2. **Register**

* **Description**: Specifies the type of Modbus register to query. Options include "coil", "discrete", "holding", or "input".
* **Default**: "holding"
* **Configuration Example**:

  ```yaml
  register: "holding"
  ```

3. **Address**

* **Description**: The Modbus register address from which data should be read.
* **Configuration Example**:

  ```yaml
  address: 3
  ```

4. **Type**

* **Description**: Specifies the data type of the field, which determines how the data read from the register is interpreted. This setting is crucial as it affects how the raw data from Modbus registers is processed and used. The available data types cater to various data resolutions and formats, ranging from single-bit signals to full 64-bit precision, including special formats for strings and floating-point numbers.
* **Options**:
  * `BIT`: Single bit of a register.
  * `INT8L`: 8-bit integer (low byte).
  * `INT8H`: 8-bit integer (high byte).
  * `UINT8L`: 8-bit unsigned integer (low byte).
  * `UINT8H`: 8-bit unsigned integer (high byte).
  * `INT16`: 16-bit integer.
  * `UINT16`: 16-bit unsigned integer.
  * `INT32`: 32-bit integer.
  * `UINT32`: 32-bit unsigned integer.
  * `INT64`: 64-bit integer.
  * `UINT64`: 64-bit unsigned integer.
  * `FLOAT16`: 16-bit floating point (IEEE 754).
  * `FLOAT32`: 32-bit floating point (IEEE 754).
  * `FLOAT64`: 64-bit floating point (IEEE 754).
  * `STRING`: A sequence of bytes converted to a string.

5. **Length**

* **Description**: Number of registers to read, primarily used when the data type is "STRING".
* **Default**: 0
* **Configuration Example**:

  ```yaml
  length: 2
  ```

6. **Bit**

* **Description**: Relevant only for BIT data type, specifying which bit of the register to read.
* **Default**: 0
* **Configuration Example**:

  ```yaml
  bit: 7
  ```

7. **Scale**

* **Description**: A multiplier applied to the numeric data read from the register, used to scale values to the desired range or unit.
* **Default**: 0.0
* **Configuration Example**:

  ```yaml
  scale: 0.1
  ```

8. **Output**

* **Description**: Specifies the data type of the output field. Options include "INT64", "UINT64", "FLOAT64", or "native" (which retains the original data type without conversion).
* **Default**: Defaults to FLOAT64 if "scale" is provided and to the input "type" class otherwise (i.e. INT\* -> INT64, etc).
* **Configuration Example**:

  ```yaml
  output: "FLOAT64"
  ```

9. **Slave ID (per-address)**

* **Description**: Optionally restrict this address to a specific slave ID. When set, only the specified slave will read this address. When omitted or set to 0, all configured slaves (from the top-level `slaveIDs`) will read this address.
* **Default**: 0 (all slaves)
* **Validation**: If non-zero, the value must appear in the top-level `slaveIDs` list.
* **Configuration Example**:

  ```yaml
  slaveID: 2
  ```


# ifm IO-Link Master / "sensorconnect"

The SensorConnect plugin facilitates communication with ifm electronic’s IO-Link Masters devices, such as the AL1350 or AL1352 IO-Link Masters.\\

It also supports EIO404 Bluetooth mesh base stations with EIO344 Bluetooth mesh IO-Link adapters.\
It enables the integration of sensor data into Benthos pipelines by connecting to the device over HTTP and processing data from connected sensors, including digital inputs and IO-Link devices.\
The plugin handles parsing and interpreting IO-Link data using IODD files, converting raw sensor outputs into human-readable data.

It was previously known as [sensorconnect](https://github.com/united-manufacturing-hub/united-manufacturing-hub/tree/staging/golang/cmd/sensorconnect).

**Configuration**

Below is an example configuration demonstrating all available options for the sensorconnect plugin. This includes settings for device connectivity, IODD API URLs, and detailed device-specific configurations.

```yaml
input:
  sensorconnect:
    device_address: '192.168.0.1' # IP address of the IO-Link Master
    iodd_api: 'https://management.umh.app/iodd' # URL of the IODD API
    devices:
      - device_id: 1234
        vendor_id: 5678
        iodd_url: "https://example.com/iodd/device1234.xml"
      - device_id: 2345
        vendor_id: 6789
        iodd_url: "https://example.com/iodd/device2345.xml"
```

**Configuration Options**

**Device Address**

Specifies the IP address of the ifm IO-Link Master device

```yaml
input:
  sensorconnect:
    device_address: '192.168.0.1'
```

**IODD API**

Defines the URL of the IODD API, which is used to fetch IODD files for connected devices. Defaults to `https://management.umh.app/iodd` and should not be changed except for development purposes.

```yaml
input:
  sensorconnect:
    iodd_api: 'https://management.umh.app/iodd'
```

**Devices**

Provides a list of devices to provide for a given device\_id and vendor\_id, a fallback iodd\_url (in case the IODD file is not available via the IODD API).

```yaml
input:
  sensorconnect:
    devices:
      - device_id: 509 # Device ID of the IO-Link device
        vendor_id: 2035 # Vendor ID of the IO-Link device
        iodd_url: "https://yourserver.com/iodd/KEYENCE-FD-EPA1-20230410-IODD1.1.xml" # URL of the IODD file for the device. You might need to download this from the vendors website and self-host it.
```

**Output**

The payload of each message is a JSON object containing the sensor data, structured according to the data provided by the connected device. The exact structure of the payload depends on the specific sensors connected to the SensorConnect device and the data they provide.

Example for a VVB001 vibration sensor:

```json
{
  "Crest": 41,
  "Device status": 0,
  "OUT1": true,
  "OUT2": true,
  "Temperature": 394,
  "a-Peak": 2,
  "a-Rms": 0,
  "v-Rms": 0
}

```

**Metadata Outputs**

For each read operation, the plugin outputs detailed metadata that includes various aspects of the read operation, which can be utilized to effectively tag, organize, and utilize the data within a system.

Below is the extended metadata output schema provided by the plugin:

| Metadata                                 | Description                                                       |
| ---------------------------------------- | ----------------------------------------------------------------- |
| `sensorconnect_port_mode`                | The mode of the port, e.g., digital-input or io-link.             |
| `sensorconnect_port_number`              | The number of the port on the ifm IO-Link Master device.          |
| `sensorconnect_port_iolink_vendor_id`    | The IO-Link vendor ID of the connected device (if applicable).    |
| `sensorconnect_port_iolink_device_id`    | The IO-Link device ID of the connected device (if applicable).    |
| `sensorconnect_port_iolink_product_name` | The product name of the connected IO-Link device (if applicable). |
| `sensorconnect_port_iolink_serial`       | The serial number of the connected IO-Link device.                |
| `sensorconnect_device_product_code`      | The product code of the connected IO-Link device.                 |
| `sensorconnect_device_serial_number`     | The serial number of the connected IO-Link device                 |


# Beckhoff ADS (experimental)

**This protocol is experimental. It has been tested on CX7000 and CX8190 (TwinCAT 3) and CX1020 (TwinCAT 2); behaviour on other controllers may differ.**

This input reads variables from Beckhoff PLCs over ADS by symbol name, either as change notifications pushed by the PLC or as interval polling. It works with TwinCAT 2 and TwinCAT 3, runs from a virtual machine or a container, and can register its own route on the PLC.

This input reads **symbols only**: named variables such as `GVL_ProcessData.nMasterCycleCounter`. Direct addresses (`%MB100`) are not supported.

## Where to go

| Page                                                                         | For                                                                            |
| ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| [Quick start](/benthos-umh/input/beckhoff-ads/quick-start)                   | Getting first values out of a PLC, step by step                                |
| [How it works](/benthos-umh/input/beckhoff-ads/how-it-works)                 | ADS, AMS NetIDs and routes; read this once and the settings explain themselves |
| [Configuration](/benthos-umh/input/beckhoff-ads/configuration)               | Every setting, with complete examples                                          |
| [Symbols and metadata](/benthos-umh/input/beckhoff-ads/symbols-and-metadata) | How to address symbols, and what each message carries                          |
| [Networking](/benthos-umh/input/beckhoff-ads/networking)                     | Containers, VPNs, NAT, and creating routes by hand                             |
| [Troubleshooting](/benthos-umh/input/beckhoff-ads/troubleshooting)           | Symptoms, causes, and what the log lines mean                                  |

## Prerequisites

Have these ready before you start. Everything else has a working default.

|                           | What you need                                                              | How to check                                                                                                                                                       |
| ------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **PLC address**           | The IP address of the PLC, reachable from wherever this runs               | `ping <plc-ip>`                                                                                                                                                    |
| **ADS port open**         | TCP `48898` to the PLC                                                     | `nc -z -w3 <plc-ip> 48898`                                                                                                                                         |
| **Route port open**       | UDP `48899`, only if the plugin should register its own route              | Blocked UDP means creating the route by hand instead                                                                                                               |
| **PLC credentials**       | An administrator user on the PLC, only for automatic route registration    | The same login you would use in TwinCAT to add a route                                                                                                             |
| **This client's address** | The address the PLC will see this client as, needed for `hostIP`           | Usually the VM's IP on the PLC network. Behind a container boundary, VPN or NAT it is something else: see [Networking](/benthos-umh/input/beckhoff-ads/networking) |
| **Symbol names**          | The full name of each variable, e.g. `GVL_ProcessData.nMasterCycleCounter` | Read them from the PLC project in TwinCAT                                                                                                                          |
| **A running PLC runtime** | The runtime must be in RUN, not CONFIG                                     | A stopped runtime accepts the connection and serves nothing                                                                                                        |

Two limits worth knowing up front:

* Beckhoff recommends [no more than **550 notifications per device**](https://infosys.beckhoff.com/content/1033/tc3_grundlagen/115880971.html). Above that, read in fewer, larger pieces (a struct instead of its members) or switch to `readType: interval`, which reads with sum commands.
* A PLC serves **one connection per client address**. A second client from the same address displaces the first. That second client can be another instance of this input, or an engineering station behind the same NAT gateway.


# Quick start

In this quick start we set up a TwinCAT 3 PLC as a data source in a Benthos pipeline. By the end of this section you will know how to:

1. Find the symbols you want to read
2. Write the input configuration
3. Add a tag processor
4. Send the values somewhere
5. Check that values are arriving

Before you start you need:

* A Beckhoff PLC reachable from this host
* An administrator user on that PLC
* The PLC's IP address

You end with a Benthos config file that you run with `benthos -c config.yaml`. The whole thing takes about 15 minutes.

Step 2 writes a route entry to the PLC, so point this at a PLC you are allowed to configure.

The same five steps work for TwinCAT 2, with different symbol names and a different runtime port. There is a complete TwinCAT 2 example at the end of this page.

## 1. Find the symbols you want

Open the PLC project in TwinCAT and note the full name of each variable, for example `GVL_ProcessData.nMasterCycleCounter`. Global variables live under their GVL name; program variables under the program name.

## 2. Write the input

> **Warning:** `username` and `password` below make the plugin register a route on the PLC. That writes a persistent entry to the device's route table, which stays until someone removes it on the device. See [Getting a route onto the PLC](/benthos-umh/input/beckhoff-ads/networking#getting-a-route-onto-the-plc).

`targetAMS` is discovered from the PLC, so you do not need its AMS NetID. `runtimePort` is not: it defaults to `851`, the first TwinCAT 3 runtime, so a TwinCAT 2 PLC has to set `801`. `hostIP` is the address the PLC will see this client as, which is usually the VM's IP on the PLC network:

```yaml
input:
  ads:
    targetAddress: "192.168.1.100"
    hostIP: "192.168.1.50"
    username: "Administrator"              # the PLC's own login, for route registration
    password: "1"                           # this PLC's password, not a default to copy
    unifiedAddress:
      - "GVL_ProcessData.nMasterCycleCounter"
      - "GVL_ProcessData.fGlobalReal"
```

## 3. Add a tag processor

Each message carries the symbol name in `ads_symbol_name`, which is what the tag name should usually be:

```yaml
pipeline:
  processors:
    - tag_processor:
        defaults: |-
          msg.meta.location_path = "enterprise.site.area.line";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name      = msg.meta.ads_symbol_name;
          return msg;
```

## 4. Send the values somewhere

`stdout` prints each value as it arrives, which is enough to prove the PLC side works:

```yaml
output:
  stdout: {}
```

To write to the Unified Namespace instead, swap in the `uns` output, which needs the embedded Redpanda that ships with UMH Core:

```yaml
output:
  uns: {}
```

## 5. Check the values arrive

On the first connect the plugin registers a route on the PLC, subscribes to each symbol, and waits for one value from each before reporting itself connected. In the logs you should see, in order:

```
registering route               (only the first time, or after the PLC forgets the route)
route registration successful
Connected to PLC
Registering notifications succeeded for 2/2 symbols
Input type ads is now active
```

Two different registrations appear there. The route registration writes an entry to the PLC's route table and happens once. The notification registration subscribes to each symbol and happens on every connect.

Values then print one per line. The `tag_processor` step has already set each message's topic to `umh.v1.enterprise.site.area.line._historian.<symbol>`, which is what the `uns` output uses as the message key.

If the log stops after `Connected to PLC`, or symbols register but no values arrive, go to [Troubleshooting](/benthos-umh/input/beckhoff-ads/troubleshooting). The cause is most often a wrong `hostIP`.

## The same config on TwinCAT 2

Global variables carry a leading dot, program variables an uppercase program prefix:

```yaml
input:
  ads:
    targetAddress: "192.168.1.200"
    runtimePort: 801                    # first TwinCAT 2 runtime; the default 851 is TwinCAT 3
    hostIP: "192.168.1.50"
    username: "Administrator"
    password: "1"
    unifiedAddress:
      - ".nMasterCycleCounter"
      - "PRG_DIAGNOSTICS.nInt"
```

## Next

* Reading structs, arrays or strings: [Symbols and metadata](/benthos-umh/input/beckhoff-ads/symbols-and-metadata)
* Running in a container, or across a VPN: [Networking](/benthos-umh/input/beckhoff-ads/networking)
* Tuning how often values arrive: [Configuration](/benthos-umh/input/beckhoff-ads/configuration)


# How it works

This section covers the background that makes the settings on the other pages understandable. It explains how ADS addresses a device, how the PLC decides whether to accept this client, and what the timing options control. None of it is required reading to get data flowing — the [Quick start](/benthos-umh/input/beckhoff-ads/quick-start) is enough for that.

## How ADS works

Four ideas cover almost everything you need to configure this input.

**A router, not a device.** Every TwinCAT system runs an AMS router, and you always talk to the router rather than to the PLC program directly. It forwards each request to one of the devices behind it (the PLC runtime, the I/O layer, the logger), selected by a port number. `851` is the first PLC runtime on TwinCAT 3, `801` on TwinCAT 2.

**Two addresses, not one.** The IP address gets you to the router; an **AMS NetID** (six numbers, usually written like an IP with `.1.1` appended) identifies the system at either end. Your request carries both your own NetID and the target's. The plugin asks the PLC for its NetID on connect, so normally only the IP needs configuring.

**A route authorizes you.** The PLC answers only clients it has a route for: an entry pairing your NetID with the address it expects you to come from. Routes live on the PLC, survive reboots, and can be created by the plugin itself (`username` + `password`) or by hand in TwinCAT. If a request arrives with no matching route, the connection is accepted and then ignored. That is the most common cause of an input that looks connected but reads nothing. See [How a route is matched](#how-a-route-is-matched).

**One connection, two ways to read.** Everything runs over a single outbound TCP connection to port 48898, and normally no inbound port is needed. On that connection you either poll (`readType: interval`) or subscribe and let the PLC push values when they change (`readType: notification`, the default). Use `interval` when the symbol list is long enough to pass the PLC's notification limit, and `notification` when values change less often than you would otherwise poll. Symbols are addressed by name, so `GVL_ProcessData.nMasterCycleCounter` is all you need; the PLC resolves it to a handle.

## Getting the PLC to accept this client

A route has to exist on the PLC before any data flows, and there are three ways to create one: the TwinCAT Connection Manager on a Windows host, a static route added in the TwinCAT System Manager or XAE, or automatic registration by the plugin itself from `username` and `password`. Automatic registration also needs `hostIP`, because that becomes the address stored in the route. [Networking](/benthos-umh/input/beckhoff-ads/networking) has the procedure for each, and [How a route is matched](#how-a-route-is-matched) explains what the PLC checks.

## How a route is matched

An AMS route pairs an AMS NetID with an address, and the runtime uses both. The NetID selects the route, and it must equal the source NetID the client advertises. The address is checked against the **source IP the runtime observes**. If they disagree, the TCP connection stays up and the requests are silently dropped: no error, no reply. Replies to requests that do pass come back over the connection the client opened.

```mermaid
flowchart LR
  C["Client<br/>NetID 192.168.1.50.1.1<br/>IP 192.168.1.50"]
  subgraph PLC["PLC"]
    R["AMS router"]
    T["Route<br/>NetId 192.168.1.50.1.1<br/>Address 192.168.1.50"]
    Q{"Address = source IP<br/>the PLC sees?"}
  end
  OK["Reply on the client's<br/>own connection"]
  NO["Request dropped,<br/>no reply, connection stays up"]
  C -->|"request on TCP 48898"| R
  R -->|"look up by source NetID"| T
  T --> Q
  Q -->|yes| OK
  Q -->|no| NO
  OK --> C
```

`NetId` only has to match what the client advertises (`hostAMS`) and need not resemble the address. `Address` must equal the source IP the PLC observes. Note *observes*, not *can reach*.

Replies and notifications normally arrive over the connection the client opened, so the input works from a Docker container with no inbound path and no listening ports.

A device can instead answer only on a connection it opens back to the client on TCP `48898`. The plugin accepts that connection and reads the replies from it, so there is nothing to configure. Two things still have to hold: `48898` must be free on the client host, and the PLC must be able to reach it. If the port is already taken, the plugin logs a warning that names it, and that device cannot be read.

Routes are bidirectional by default. Leave that alone: with `Unidirectional` ticked, the runtime stops accepting ADS calls from the system at the far end of the route, which is this client, so it refuses the plugin's own requests.

A wrong `Address` produces a session that connects and registers successfully and then fails, usually as repeated request timeouts rather than a connection error.

Automatic registration derives both fields from `hostIP` (`Address` = `hostIP`, `NetId` = `hostIP`

* `.1.1`). Firmwares differ in what they store when `hostIP` is left as `auto`. Some record the source address they observe, others the value the client advertises, so set `hostIP` explicitly or add the route manually.

## Notifications or polling

The `interval` and `notification` read types can produce similar-looking results (periodic data), but they work differently under the hood:

* **`interval`**: The client polls the PLC every `intervalTime`, reading the whole symbol list in one sum command. No PLC notification overhead, and the 550-notification limit does not apply.
* **`notification` + `serverOnChange`**: The PLC pushes data only when a value changes. After one initial sample at the first check, it sends nothing while the value is unchanged. Counts against the 550-notification limit.
* **`notification` + `serverCycle`**: The PLC pushes data at every `cycleTime` interval regardless of changes. Similar result to `interval` but PLC-driven: more precise timing, with no request/response overhead per cycle. Counts against the 550-notification limit.

| Aspect                 | `interval`                       | `notification` + `serverOnChange`  | `notification` + `serverCycle` |
| ---------------------- | -------------------------------- | ---------------------------------- | ------------------------------ |
| Who drives             | Client polls                     | PLC pushes on change               | PLC pushes on timer            |
| Network per cycle      | Request + response               | Push only                          | Push only                      |
| Sends unchanged values | Yes                              | No, after the initial sample       | Yes                            |
| Timing precision       | Subject to network latency       | PLC real-time task                 | PLC real-time task             |
| PLC notification limit | No limit                         | 550 per device                     | 550 per device                 |
| Best for               | Large symbol lists, simple setup | Event-driven data (most use cases) | Precise periodic sampling      |

## Choosing cycleTime and maxDelay

**cycleTime** controls how often the PLC checks the variable:

* `serverCycle` mode: PLC sends a notification every `cycleTime` regardless of value change
* `serverOnChange` mode: PLC checks the value every `cycleTime` and sends a notification only if it changed

**maxDelay** controls how long the PLC can buffer notifications before sending:

* The PLC collects notification events and sends them in a batch when `maxDelay` expires
* This is a network optimization: fewer packets, multiple notifications bundled in one AMS packet

**Practical example**, `cycleTime: 10ms`, `maxDelay: 100ms`, mode `serverOnChange`:

1. PLC checks variable every 10ms
2. If value changed, queues a notification
3. Sends queued notifications at most every 100ms (batched)

**Edge cases:**

* `maxDelay: 0s`: send immediately, no batching
* `cycleTime: 0s`: check as fast as the PLC task cycle allows
* `maxDelay` < `cycleTime`: effectively no batching (fires before next check)

Think of it as:

* **cycleTime** = polling interval (sensor sampling rate)
* **maxDelay** = delivery batch window (network efficiency)

**Important:** If a variable changes faster than `cycleTime`, intermediate values are missed:

```
cycleTime = 1000ms, mode = serverOnChange

Time:    0ms    200ms   400ms   600ms   800ms   1000ms
Value:   5  →   10  →   3   →   7   →   2   →   8
PLC checks:  ↑                                    ↑
Notifies: 5                                       8  (missed 10,3,7,2)
```

The PLC only samples at `cycleTime` intervals. Between checks it is blind. This is not a continuous event stream.

For fast-changing values, set `cycleTime` close to the PLC task cycle time (typically 1–10ms). The trade-off is more CPU load on the PLC and more network traffic. Even at minimum `cycleTime`, there is no guarantee of capturing every value: if a variable changes twice within one PLC scan cycle, the intermediate value is lost. ADS notifications are polling with push delivery, not event capture.

Which of the four transmission modes applies, and what the `2` variants change, is in [Transmission modes](/benthos-umh/input/beckhoff-ads/configuration#transmission-modes).

## What to expect from notifications

### First batch completeness

When `readType: notification`, TwinCAT sends an initial sample for every subscribed symbol in every one of the four transmission modes, including for a symbol whose value never changes: the first check has no previous value to compare against, so it always reports.

That initial sample arrives at the **first check, one `cycleTime` after registration** (plus `maxDelay` where the PLC batches), not at registration itself. `cycleTime` is the interval at which the PLC compares the value, floored by the task cycle, so the wait scales with what you configure:

| `cycleTime`       | First sample arrives after                 |
| ----------------- | ------------------------------------------ |
| `0s`              | one PLC task cycle (\~10ms on a 10ms task) |
| `100ms` (default) | \~120ms                                    |
| `1s`              | \~1.02s                                    |
| `60s`             | \~60.02s                                   |

The plugin waits for these initial samples before returning from `Connect`, so the **first `ReadBatch` returns a complete batch** containing one message per successfully registered symbol, and no separate read or warm-up period is needed to get the current state of all symbols.

The consequence is that a large `cycleTime` delays startup, because `Connect` cannot complete before the slowest symbol's first check. With the defaults that is about 0.2s. The wait is bounded at `cycleTime + maxDelay + 10s` for the slowest configured symbol; if a symbol has still not reported by then, a warning names it and startup continues, and that symbol's data arrives once the PLC does report it.

### The connection heartbeat

With `readType: notification` the plugin subscribes to one extra cyclic notification of its own, on the PLC's symbol-version group. Those beats are the sign that the runtime is still serving notifications. The beat arrives every 2 s by default, carries one byte, and counts as one of the PLC's 550 notifications.

It exists because a runtime restart or a CONFIG toggle can empty the PLC's notification table without dropping the TCP connection, which would otherwise leave the input connected and silent.

`notificationSilenceTimeout` sets how much silence is tolerated, and is converted into a number of beats: 10 s against a 2 s cycle means five missed beats mark the subscriptions dead. What happens then is `heartbeatRecovery`, see [Silent notification delivery](/benthos-umh/input/beckhoff-ads/troubleshooting#silent-notification-delivery). A PLC that refuses the subscription is logged as a warning and still serves data; only the ability to notice a silent death is lost.

`readType: interval` has no heartbeat, and needs none: a runtime that stops answering fails the next read.

### Partial registration failures

If a symbol fails to register (unknown name, PLC-side ADS error), the plugin:

* Logs an **error** through the Benthos logger identifying the symbol and reason
* Continues with the remaining symbols, so data still flows for every symbol that did register
* Does **not** trigger a reconnect for partial failures; only a full failure (zero symbols registered) forces a reconnect

This means a misconfigured symbol name is surfaced immediately in logs without blocking data from the other symbols.

### Interval read: empty batches during reconnect

When `readType: interval`, the first one or two batches after a reconnect may be **empty or partial** while symbol handles are re-resolved. This is normal, Benthos retries the next poll and subsequent batches are complete. No action is needed; the gap is typically one poll interval (default 1 second).


# Configuration

This section covers every setting the Beckhoff ADS input accepts, if they are required or not and the default value if not used, together with two complete example configurations that use them.

## Settings

| Parameter                      | Required | Default          | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| ------------------------------ | -------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **targetAddress**              | Yes      | —                | IP address (and optional port) of the PLC's ADS gateway, as `ip` or `ip:port`. Port defaults to `48898` if omitted                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| **targetAMS**                  | No       | discovered       | AMS net ID of the target runtime. Left empty, the plugin asks the PLC for its own on connect, so it normally does not need setting. Set it only to pin a specific runtime; a mismatch with what the PLC reports is logged as a warning, not an error                                                                                                                                                                                                                                                                                                                                                                       |
| **unifiedAddress**             | Yes      | —                | Symbols to read, in unified address form (see [Symbols Format](/benthos-umh/input/beckhoff-ads/symbols-and-metadata#addressing-symbols) below)                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| **loadSymbols**                | No       | `false`          | Download the PLC's symbol and datatype table on connect. Required to read a struct or array as one value, which arrives as nested JSON. A named member (`GVL.stStatus.fValue`) resolves without it. The download can cause brief real-time jitter on the PLC, so it is off by default                                                                                                                                                                                                                                                                                                                                      |
| **runtimePort**                | No       | `851`            | AMS port of the PLC runtime. `851` is the first TwinCAT 3 runtime and `801` the first TwinCAT 2 one, so a TwinCAT 2 PLC has to set this. `0` asks the PLC for its TwinCAT version and uses the first runtime of that version. A later runtime (`811`, `821` on TwinCAT 2; `852`, `853` on TwinCAT 3) always has to be set explicitly                                                                                                                                                                                                                                                                                       |
| **hostAMS**                    | No       | `auto`           | Host AMS net ID. Usually the IP address + `.1.1`. Must match a route on the PLC. `auto` (or empty) derives it from `hostIP` when that is set, otherwise from the outbound connection's local IP                                                                                                                                                                                                                                                                                                                                                                                                                            |
| **hostPort**                   | No       | `0`              | AMS source port in protocol headers. `0` uses a random port per session (recommended, avoids notification handle conflicts across sessions). Set a fixed value only in firewalled environments with port allow-lists                                                                                                                                                                                                                                                                                                                                                                                                       |
| **readType**                   | No       | `notification`   | Read type for the symbols. `interval` polls at `intervalTime`; `notification` uses PLC push updates (see [Interval vs Notification](/benthos-umh/input/beckhoff-ads/how-it-works#notifications-or-polling)). Accepted values: `notification`, `interval`                                                                                                                                                                                                                                                                                                                                                                   |
| **maxDelay**                   | No       | `100ms`          | Maximum time the PLC batches notifications before sending (e.g. `100ms`, `0s`). All changes are still delivered; this controls delivery latency vs network efficiency. `0s` = send immediately. Only applies when `readType` is `notification`                                                                                                                                                                                                                                                                                                                                                                             |
| **cycleTime**                  | No       | `100ms`          | How often the PLC checks the symbol for changes (e.g. `100ms`, `10ms`, `1s`). Lower = more responsive but more PLC CPU. Values faster than the PLC task cycle are clamped to the task cycle. A symbol's first sample arrives at its first check, so a large value also delays startup, see [First batch completeness](/benthos-umh/input/beckhoff-ads/how-it-works#first-batch-completeness). Only applies when `readType` is `notification`                                                                                                                                                                               |
| **intervalTime**               | No       | `1s`             | Interval time between reads (e.g. `1s`, `500ms`). Only used when `readType` is `interval`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| **requestTimeout**             | No       | `5s`             | Timeout for individual ADS requests (e.g. `5s`, `10s`). Increase for slow PLCs or large symbol tables                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| **transmissionMode**           | No       | `serverOnChange` | Notification transmission mode. Only applies when `readType` is `notification`. Accepted values: `serverOnChange`, `serverCycle`, `serverOnChange2`, `serverCycle2` (see [Transmission modes](#transmission-modes))                                                                                                                                                                                                                                                                                                                                                                                                        |
| **username**                   | No       | `""`             | Username for automatic UDP route registration on the PLC. Both `username` and `password` must be set to activate registration. Requires UDP port 48899 to be reachable (see [Route Registration](/benthos-umh/input/beckhoff-ads/networking#automatic-route-registration))                                                                                                                                                                                                                                                                                                                                                 |
| **password**                   | No       | `""`             | Password for automatic UDP route registration on the PLC                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| **maxReconnectInterval**       | No       | `30s`            | Upper bound on the reconnect backoff, and on the cooldown applied to a connection that keeps flapping. Every reconnect costs the PLC an accepted socket, which the default protects; lower it for a device that resets on a timer anyway and where samples matter more than sockets (see [Reconnection](/benthos-umh/input/beckhoff-ads/troubleshooting#reconnection)). `0s` keeps the library default                                                                                                                                                                                                                     |
| **routeActivationTimeout**     | No       | `10s`            | How long to wait, after registering a route, for the PLC's AMS router to start serving it. The router acknowledges the registration before the entry is necessarily live, and until it is, requests are dropped with no reply. Raise it for a PLC or router under heavy load. `0s` keeps the library default                                                                                                                                                                                                                                                                                                               |
| **notificationSilenceTimeout** | No       | `10s`            | How long the plugin's [connection heartbeat](/benthos-umh/input/beckhoff-ads/how-it-works#the-connection-heartbeat) may be silent before the subscriptions are treated as dead and re-registered. Converted internally into a number of missed beats. Only applies when `readType` is `notification`. `0s` keeps the library default                                                                                                                                                                                                                                                                                       |
| **heartbeatRecovery**          | No       | `immediate`      | What to do when notification delivery goes silent; see [Silent notification delivery](/benthos-umh/input/beckhoff-ads/troubleshooting#silent-notification-delivery). Accepted values: `immediate`, `confirm`, `rebuild`                                                                                                                                                                                                                                                                                                                                                                                                    |
| **hostIP**                     | No       | `auto`           | IP address the PLC associates with the route. It must be **the address the PLC sees this client as**, which behind a NATing gateway is not the client's own address. See [PLC behind a NATing gateway](/benthos-umh/input/beckhoff-ads/networking#plc-behind-a-nating-gateway-or-subnet-router). Required in Docker bridge networking on TwinCAT 2, where replies are routed via this address (set to Docker host's IP); optional on TwinCAT 3. When `hostAMS` is `auto`, the AMS NetID is also derived from this. `auto` (or empty) detects it from the outbound connection (only correct with `host_network` or macvlan) |

ADS library log verbosity follows the pipeline log level, the top-level `logger.level` key in `config.yaml`. There is no separate setting.

## Transmission modes

> **Note:** `transmissionMode` only applies when `readType` is `notification`. When using `readType: interval`, the plugin sends plain ADS Read commands to the PLC at each interval, no notification mechanism is involved, and `transmissionMode` is ignored.

The `transmissionMode` field controls how the PLC's internal notification handler sends updates back to the client. The available modes are:

| Mode              | Value | Description                                                                                                                                                                                                              |
| ----------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `serverOnChange`  | 4     | Default. The PLC checks the value every `cycleTime` and sends a notification only when it changed. The check runs in the target's default task.                                                                          |
| `serverCycle`     | 3     | The PLC sends the current value every `cycleTime`, whether or not it changed. Use for a steady stream or a heartbeat.                                                                                                    |
| `serverOnChange2` | 6     | `OnChangeInContext`: the same check, bound to the task that owns the variable instead of the default task, which makes the timing task-synchronous. Falls back to `serverOnChange` for any symbol whose ContextMask is 0 |
| `serverCycle2`    | 5     | `CyclicInContext`: as `serverCycle`, bound to the task that owns the variable. Falls back to `serverCycle` for any symbol whose ContextMask is 0                                                                         |

**Choosing a mode:**

* Use `serverOnChange` (default) for event-driven data where only changes matter
* Use `serverCycle` for periodic snapshots regardless of changes
* The `2` variants are safe to set on any PLC, because a symbol with ContextMask 0 falls back to the v1 equivalent. They only change behavior for variables local to a PROGRAM POU in a multi-task project; a GVL variable or a single-task project always has ContextMask 0. Beckhoff describes these as [`CyclicInContext` and `OnChangeInContext`](https://infosys.beckhoff.com/content/1033/tcadsnetref/7313078411.html) and advises against them in the default case

## Full example: TwinCAT 3

All fields with TC3 defaults and comments:

```yaml
input:
  ads:
    # Target connection
    targetAddress: "192.168.1.100:48898"   # PLC IP[:port]; port defaults to 48898
    targetAMS: ""                          # empty = ask the PLC for its own NetID
    runtimePort: 851                       # TC3 runtime port (TC2: 801)
    # Host AMS identity
    hostAMS: "auto"                        # auto = derived from hostIP, else the TCP source IP
    hostPort: 0                            # 0 = random per session (recommended)
    hostIP: "192.168.1.50"                 # the address the PLC sees this client as
    # Route registration: both must be set to activate
    username: "Administrator"
    password: "1"
    # Read mode
    readType: "notification"               # notification | interval
    transmissionMode: "serverOnChange"     # default; the 2 variants need a task-bound symbol
    cycleTime: 100ms                       # how often the PLC checks for changes
    maxDelay: 100ms                        # max batching window before the PLC sends
    intervalTime: 1s                       # poll interval (readType: interval only)
    # Symbols
    loadSymbols: true                      # download the symbol table; needed for structs and arrays
    # Timeouts and recovery
    requestTimeout: 5s                     # per-request timeout; raise on slow links
    maxReconnectInterval: 30s              # cap on the reconnect backoff and flap cooldown
    routeActivationTimeout: 10s            # wait for a new route to start being served
    notificationSilenceTimeout: 10s        # silence tolerated before subscriptions count as dead
    heartbeatRecovery: "immediate"         # immediate | confirm | rebuild
    # Symbols to read (list last for readability)
    unifiedAddress:
      - "GVL_ProcessData.nMasterCycleCounter"
      - "GVL_ProcessData.stMachineStatus.stMotor1.fSpeed"
      - "GVL_ProcessData.anCounters[0]"
      - "MAIN.MyTrigger:maxDelay=0s:cycleTime=10ms"  # per-symbol overrides
```

## Full example: TwinCAT 2

All fields with TC2 differences highlighted:

```yaml
input:
  ads:
    # Target connection
    targetAddress: "192.168.1.200:48898"   # PLC IP[:port]; port defaults to 48898
    targetAMS: ""                          # empty = ask the PLC for its own NetID
    runtimePort: 801                       # TC2 runtime port
    # Host AMS identity
    hostAMS: "auto"
    hostPort: 0
    hostIP: "192.168.1.50"                 # the address the PLC sees this client as
    # Route registration, both must be set to activate
    username: "Administrator"
    password: "1"
    # Read mode
    readType: "notification"
    transmissionMode: "serverOnChange"     # the v2 modes would fall back to this anyway
    cycleTime: 100ms
    maxDelay: 100ms
    intervalTime: 1s                       # readType: interval only
    # Symbols
    loadSymbols: true
    # Timeouts and recovery
    requestTimeout: 5s
    maxReconnectInterval: 30s
    routeActivationTimeout: 10s
    notificationSilenceTimeout: 10s
    heartbeatRecovery: "immediate"         # immediate | confirm | rebuild
    # Symbols to read. TC2 globals use a dot prefix, program variables an uppercase prefix
    unifiedAddress:
      - ".nMasterCycleCounter"
      - ".stMachineStatus.stMotor1.fSpeed"
      - "PRG_DIAGNOSTICS.nInt"
      - ".myTrigger:maxDelay=0s:cycleTime=10ms"
```

## Supported hardware

Verified on the following controllers:

| Controller              | TwinCAT                         | Runtime port |
| ----------------------- | ------------------------------- | ------------ |
| CX7000 (TC/RTOS, ARMv7) | 3.1.4026                        | 851          |
| CX8190                  | 3.1.4024                        | 851          |
| CX1020                  | 2.10.1320, 2.10.1341, 2.11.2234 | 801          |

Each device reports its own model, TwinCAT version and NetID over UDP 48899 before any route exists. That is why `targetAMS` can be left empty, and what `runtimePort: 0` uses to pick the first runtime of the reported version.

### TwinCAT 3 (4024 and 4026)

* Notifications from a Docker container with bridge networking (no `host_network`, no published ports)
* Automatic UDP route registration from Docker bridge networking
* Static route with explicit `hostAMS` and no route registration
* Reconnection after network loss, with automatic notification re-subscribe and symbol reload
* Sum/batch commands for read, add notification, and delete notification
* `cycleTime` verification: 50ms, 200ms and 1000ms intervals match the configured values
* Deterministic value verification against the PLC's own cycle-counter formulas
* Recovery from a route whose `Address` the PLC could not reach

### TwinCAT 2 (2.10 and 2.11)

* Read batches, notifications, and a range of `cycleTime` / `maxDelay` combinations
* Datatypes: BOOL, SINT, INT, DINT, UDINT, REAL, LREAL, STRING, TIME, DATE, DT, TOD, enums, structs and arrays
* Automatic fallback from sum commands to individual calls, and from `SumReadEx2` to `SumReadEx`
* Automatic fallback from v2 transmission modes to v1
* Reconnection after network loss with automatic notification re-subscribe
* `cycleTime` verification and deterministic value verification
* TC2 symbol naming: GVL globals with a leading dot, program locals in uppercase

## Migrating from the community plugin

This input replaces the community-supported `ads` plugin. The input name is unchanged, so an existing configuration keeps loading, but several field names changed, and a config using the old ones fails to start:

| Community plugin                                                                                              | This input                                                                     |
| ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| `targetIP` + `targetPort`                                                                                     | `targetAddress`, as `ip` or `ip:port`                                          |
| `routeUsername`                                                                                               | `username`                                                                     |
| `routePassword`                                                                                               | `password`                                                                     |
| `routeHostAddress`                                                                                            | `hostIP`                                                                       |
| `logLevel`                                                                                                    | removed; the pipeline's `logger.level` applies                                 |
| `symbols`                                                                                                     | removed; use `unifiedAddress`                                                  |
| `targetAMS`                                                                                                   | unchanged, and now optional: left empty, the plugin asks the PLC for its NetID |
| `runtimePort`                                                                                                 | unchanged, and still `851` by default, so a TwinCAT 2 PLC still needs `801`    |
| `hostAMS`, `hostPort`, `readType`, `transmissionMode`, `cycleTime`, `maxDelay`, `intervalTime`, `loadSymbols` | unchanged                                                                      |

Metadata field names changed too: `symbol_name` is now `ads_symbol_name`, so a `tag_processor` carrying `msg.meta.tag_name = msg.meta.symbol_name` needs updating. See [Symbols and metadata](/benthos-umh/input/beckhoff-ads/symbols-and-metadata#metadata-outputs) for the full set.


# Symbols and metadata

This section explains how to configure a symbol, how its name becomes a topic, and what metadata each message carries once it has been read.

## Addressing symbols

Symbols are specified as `name[:key=value...]`. Each option names the field it sets.

| Format                               | maxDelay | cycleTime |
| ------------------------------------ | -------- | --------- |
| `MAIN.var`                           | default  | default   |
| `MAIN.var:cycleTime=100`             | default  | 100       |
| `MAIN.var:maxDelay=50`               | 50       | default   |
| `MAIN.var:maxDelay=50:cycleTime=100` | 50       | 100       |

**Rules:**

* Options are `key=value`; the only keys are `maxDelay` and `cycleTime`
* A bare number is ignored with a warning, and the default is kept
* Values without a unit are milliseconds; `cycleTime=10ms` and `cycleTime=10` are the same
* Invalid or omitted values fall back to the plugin-level `maxDelay`/`cycleTime` defaults
* Both options only apply when `readType` is `notification`; with `interval` the plugin polls every `intervalTime` and per-symbol timing is ignored

**Examples:**

* `MAIN.MYBOOL`: uses plugin-level defaults for both
* `MAIN.MYTRIGGER:maxDelay=0:cycleTime=10`: 0ms max delay, 10ms cycle time
* `MAIN.MYSENSOR:cycleTime=10`: default max delay, 10ms cycle time
* `.superDuperInt`: global variable (TC2, must start with `.`)

**TwinCAT 3** qualifies every symbol with the object that declares it. A global variable is `<GVL name>.<variable>`, where `<GVL name>` is whatever the Global Variable List object is called. TwinCAT names a new GVL `GVL`, so `GVL.nCounter` is the common case; a list named `GVL_ProcessData` gives `GVL_ProcessData.nCounter`. Program variables use the POU name: `MAIN.MyVariable`, `PRG_Machine.nUpdates`. There is no fixed `GVL_` prefix and no leading dot.

**TwinCAT 2** puts globals in one flat namespace with no list name, reached by a leading dot: `.nCounter`, `.stProcessData.fValue`. Program variables still carry the POU name and no dot: `MAIN.someVar`.

Symbol names are case-insensitive, so the PLC accepts any casing. The metadata always carries the casing you configured: TwinCAT 3 returns the original casing, and TwinCAT 2 returns uppercase, which the plugin maps back to what you wrote.

## Symbol names in topics

`ads_symbol_name` becomes the last segment of the UMH topic, where a dot separates levels, so the symbol name has to be rewritten to fit. Array indices keep one separator, a TwinCAT 2 global's leading dot is dropped (it marks the namespace and is not part of the name), and runs of underscores collapse:

| PLC symbol                                        | `ads_symbol_name`                                                 |
| ------------------------------------------------- | ----------------------------------------------------------------- |
| `GVL_ProcessData.stMachineStatus.stMotor1.fSpeed` | `GVL_ProcessData_stMachineStatus_stMotor1_fSpeed`                 |
| `PRG_Diagnostics.astSensorHistory[0].fValue`      | `PRG_Diagnostics_astSensorHistory_0_fValue`                       |
| `.afMeasurements[0]` (TC2 global)                 | `afMeasurements_0`                                                |
| `.nMasterCycleCounter` (TC2 global)               | `nMasterCycleCounter`                                             |
| `_internalVar`                                    | `_internalVar` (a leading underscore may be the PLC's own naming) |

The original is always available as `ads_symbol_name_original`, so a symbol can still be traced back to the PLC. Two symbols can in principle collapse to the same tag name (`.a[0]` and a variable actually named `a_0`), so if a program relies on that distinction, use `ads_symbol_name_original` in the `tag_processor` instead.

## Structs and arrays

A named member reads like any other symbol:

```yaml
unifiedAddress:
  - "GVL_ProcessData.stMachineStatus.stMotor1.fSpeed"
  - "GVL_ProcessData.anCounters[0]"
```

Naming the struct itself returns the whole thing as one message, with the members as nested JSON:

```json
{"SMACHINENAME":"TestMachine_Line1","STMOTOR1":{"BENABLED":false,"FSPEED":291,"FTORQUE":29.1}}
```

That form needs `loadSymbols: true`, because the layout comes from the PLC's datatype table. Without it the read fails for the struct and the log says `Batch read produced no value for symbol`. Reading a member by name needs no table either way.

## Metadata outputs

Each symbol produces one message whose payload is the value read from the PLC, so `meta("ads_symbol_name")` is how a following bloblang processor tells them apart.

| Metadata Field             | Description                                                                                                                                                                                                                          |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `ads_symbol_name`          | PLC symbol name as one topic segment; see [Symbol names in topics](#symbol-names-in-topics)                                                                                                                                          |
| `ads_symbol_name_original` | The symbol exactly as the PLC names it, dots and brackets intact (e.g. `PRG_Diagnostics.astSensorHistory[0].fValue`)                                                                                                                 |
| `ads_unified_address`      | The `unifiedAddress` entry this message came from, exactly as configured and including its options (e.g. `GVL_ProcessData.nCounter:cycleTime=10ms`). Always set. For an entry with no options this equals `ads_symbol_name_original` |
| `ads_datatype`             | PLC data type string as reported by the symbol table (e.g. `DINT`, `E_MachineState`, `REAL`). Set after first successful symbol resolution, may be absent on the very first batch after connect.                                     |
| `ads_base_type`            | Resolved IEC 61131-3 primitive underlying the symbol (e.g. `DINT` for an INT-aliased enum). Absent when the type does not resolve to a primitive                                                                                     |
| `ads_data_size`            | Byte length of the symbol as reported by the PLC (e.g. `4` for DINT, `82` for STRING).                                                                                                                                               |
| `ads_tag_type`             | Value shape of the payload: `number`, `bool`, or `string`. Set for every message so downstream processors can branch on payload type without inspecting `ads_datatype`/`ads_base_type`.                                              |
| `timestamp_ms`             | Unix milliseconds for when the value was captured. For `readType: notification`, this is the PLC's sample time for that update; for `readType: interval`, this is the time the plugin performed the read.                            |


# Networking

This section covers what to configure when network is not a straight path to the PLC, for example running from a Docker container, special network routing, or via a VPN. It also shows how to create a route on the PLC by hand, for the cases where the plugin cannot register one itself.

## Where the client runs

Where the client runs decides which address the PLC sees, and that address is what a route has to match. The three cases below cover a container, a virtual machine, and a host on a routed network. This part is written for whoever administers the host and the network path to the PLC.

### Containers and Kubernetes

ADS works from inside Docker containers with default bridge networking. You need no `host_network` and no port forwarding: requests, responses and notifications normally all flow over the single outbound TCP connection the plugin opens to port 48898. One case does need inbound `48898`, a device that answers only on a connection it opens back to the client. See [How a route is matched](/benthos-umh/input/beckhoff-ads/how-it-works#how-a-route-is-matched).

The only other requirement is that the `hostAMS` value matches a route registered on the PLC. When running in Docker with bridge networking:

* **`hostIP` should be set** to the Docker host's IP on the PLC network (e.g. `192.168.1.50`). `auto` detects the container's bridge IP, which the PLC cannot match. See [How a route is matched](/benthos-umh/input/beckhoff-ads/how-it-works#how-a-route-is-matched).
* **`hostAMS` can be set explicitly** to `hostIP` + `.1.1` (e.g. `192.168.1.50.1.1`), or left as `auto`: when route registration is configured with `hostIP`, `auto` will correctly derive the AMS NetID from `hostIP` instead of the container's bridge IP.
* **A route must exist on the PLC** for the `hostAMS` NetID. This can be added manually in TwinCAT System Manager, or automatically via the `username`/`password` config fields.
* **`hostPort` is optional** (default 0 = random per session). It is a logical AMS port used in protocol headers, not a network port.

**Option A: Automatic route registration (recommended)**

The plugin registers a route on the PLC automatically via UDP before connecting. No manual PLC configuration needed:

```yaml
input:
  ads:
    targetAddress: "192.168.1.100"
    hostAMS: "auto"                        # derives NetID from hostIP
    hostIP: "192.168.1.50"                 # Docker host IP (required in bridge networking)
    username: "Administrator"              # triggers automatic route registration
    password: "1"
    unifiedAddress:
      - "MAIN.MyVariable"
```

You can also set `hostAMS` explicitly if you prefer:

```yaml
    hostAMS: "192.168.1.50.1.1"            # explicit: Docker host IP + .1.1
    hostIP: "192.168.1.50"                 # must match
```

**Option B: Static route on PLC**

If you prefer not to use automatic registration, add a static route on the PLC via TwinCAT System Manager pointing to the Docker host's IP. Then configure `hostAMS` to match. No `username`/`password` needed:

```yaml
input:
  ads:
    targetAddress: "192.168.1.100"
    hostAMS: "192.168.1.50.1.1"            # must match the route on the PLC
    unifiedAddress:
      - "MAIN.MyVariable"
```

**Option C: host\_network or macvlan**

When the container has a routable IP on the PLC network, `hostAMS: auto` works without `hostIP`:

```yaml
input:
  ads:
    targetAddress: "192.168.1.100"
    hostAMS: "auto"                        # auto-derive from the container's real IP
    username: "Administrator"              # optional: auto-register route
    password: "1"
    unifiedAddress:
      - "MAIN.MyVariable"
```

### PLC behind a port-forwarding router

When an IT router maps an external port to the PLC's internal ADS port (for example, external `4233` → internal `48898`), set `targetAddress` to the **external** IP and port:

```yaml
input:
  ads:
    targetAddress: "203.0.113.10:4233"   # router's external IP:port, forwarded to PLC port 48898
    targetAMS: "5.3.69.134.1.1"          # set explicitly: discovery uses UDP 48899, which the forward does not carry
    # ...
```

AMS NetID routing is independent of IP and port, so it still resolves correctly once the TCP connection is established.

Automatic route registration uses **fixed UDP port 48899** and cannot follow a port-forwarded path. Behind NAT or a port-forwarding router, register the route **manually on the PLC** (via TwinCAT System Manager → Routes) and leave `username` and `password` empty.

### PLC behind a NATing gateway or subnet router

`hostIP` becomes the address stored in the PLC's route, and the PLC then compares the **source address it observes** against its stored routes. Any gateway that rewrites the source (a VPN subnet router doing SNAT, a NATing firewall) makes every client behind it appear as the gateway's own LAN address. `hostIP` has to be that address, not the client's.

```mermaid
flowchart LR
  P1["<b>Sent by the input</b><br/>IP src 100.93.137.23<br/>AMS src NetID 100.93.137.23.1.1"]
  G{{"Gateway<br/>SNAT, LAN 192.168.3.14"}}
  P2["<b>Seen by the PLC</b><br/>IP src 192.168.3.14 (rewritten)<br/>AMS src NetID 100.93.137.23.1.1 (unchanged)"]
  Q{"Route Address =<br/>192.168.3.14?"}
  NO["No, request dropped"]
  P1 --> G --> P2 --> Q
  Q -->|"hostIP was the client's own address"| NO
```

NAT is an IP-layer operation: it rewrites the IP source and leaves the AMS header alone. So the PLC receives a request whose IP source and AMS source disagree, and it answers on the basis of the AMS one.

The route the client registers stores its own address, but the runtime sees the gateway's, so the two disagree and every request is dropped without an answer. That is the whole failure, and it applies to every runtime: reachability of the stored address is irrelevant, only whether it equals what the runtime observes.

Removing the translation fixes it outright. Where it has to stay, `hostIP` must be the gateway's address, so that what the route stores and what the runtime observes agree.

With `hostIP` left as the client's own address the input appears to start normally (the route registers, the connection is accepted) and then fails. Which way it fails varies from device to device rather than following the runtime version:

* it serves data for a while and the PLC then closes the connection
* it drops shortly after connecting
* it connects and subscribes, then every request times out

What the cases share is more useful than the differences: the failure arrives after a similar interval each time while the data volume varies, and `readType` makes no difference. Both point at configuration rather than the network.

**Finding the address the PLC sees.** No ADS service reports it, so ask something on the PLC's own subnet:

```bash
nc -l 9999                       # on any host in the PLC's subnet, then connect from the Benthos host
conntrack -L | grep 48898        # on the gateway itself, shows the translation
```

Any HTTP service already running on that subnet works too: make one request from the Benthos host and read its access log.

**Or remove the translation.** With SNAT disabled the PLC reaches the client at its own address, and `hostIP` is that address. That needs a return route for the VPN's range on the LAN (for tailscale, `100.64.0.0/10` via the subnet router), unless the subnet router *is* the LAN's gateway, in which case the return path already passes through it and nothing extra is required.

If the translation has to stay, add the route manually. That is the one way to give the PLC a reply address that differs from the source it observes. See [Adding the route manually on the PLC](#adding-the-route-manually-on-the-plc).

### PLC connection check

A health check that port-scans the device periodically can break the ADS session, unless the client runs an AMS router of its own. Whatever the check, do not point it at **48898**: the TwinCAT router drops the live ADS session every time that port is scanned, which looks like a flaky network but is self-inflicted. `443` is the usual choice, `80` if 443 is closed on the device. The port has to be open or the check never passes, and a web server answering there only proves the device is powered, not that the ADS runtime is running.

Deploying from the UMH Core Management Console adds such a check with the port prefilled; leave it as it is.

## Getting a route onto the PLC

Route entries live on the PLC, so this part needs someone who can log in to the device or has its administrator credentials.

### Automatic route registration

The plugin can automatically register a route on the PLC using the Beckhoff UDP route protocol (port 48899). This removes the need to manually add routes in TwinCAT System Manager.

**Activation:** both `username` and `password` must be set.

**How it works:**

1. The TCP connection to port 48898 is established
2. The plugin probes the PLC with a lightweight ADS command to check if a route already exists
3. If the probe succeeds, route registration is skipped (route already present)
4. If the probe fails, the plugin sends a UDP registration packet to port 48899: "Associate AMS NetID X with IP address Y"
5. After registration the TCP connection is re-established (some PLCs close connections from previously-unknown NetIDs)
6. On reconnect after a network loss, the same probe-first logic runs automatically

**Parameters:**

* `username` / `password`: PLC administrator credentials, same as used in TwinCAT System Manager to add routes
* `hostIP`: the address stored in the route, which has to be [what the PLC sees this client as](/benthos-umh/input/beckhoff-ads/how-it-works#how-a-route-is-matched). In Docker with bridge networking that is the Docker host's IP; mandatory on TwinCAT 2, which routes replies via this address. `auto` (or empty) detects it from the outbound connection (only correct with `host_network` or macvlan)

A registered route stays on the PLC. Removing one is done on the device, not from here: see [Create or delete ADS routes manually](https://infosys.beckhoff.com/content/1033/twincat_bsd/12459254539.html) or the route list in the TwinCAT System Manager.

**Network requirements:**

* UDP port 48899 must be reachable on the PLC from the client (for route registration)
* TCP port 48898 must be reachable on the PLC from the client (outbound, works through any NAT)

### Adding the route manually on the PLC

Automatic registration ties `Address` and `NetId` together, since both come from `hostIP`. A manual route does not, which is what makes it the way out when the two must differ: the address the PLC sees the client as, against whatever NetID the client advertises.

Leave `username` and `password` empty when the route is managed this way; the plugin then connects without attempting registration.

On a Windows host the TwinCAT Connection Manager can do this for you: scan for the device and add a connection with the PLC's credentials. Everywhere else, add the route on the device.

Beckhoff documents the click paths, and they differ per tool and firmware:

* [Adding routes in TwinCAT XAE](https://infosys.beckhoff.com/content/1033/tc3_system/5211773067.html): *SYSTEM → Routes → Static Routes*
* [Route Settings in the TwinCAT 2 System Manager](https://infosys.beckhoff.com/content/1033/tcsystemmanager/1086784395.html): the only route UI on an older CX
* [Create or delete ADS routes manually](https://infosys.beckhoff.com/content/1033/twincat_bsd/12459254539.html), TwinCAT/BSD
* [Beckhoff Device Manager: web interface](https://infosys.beckhoff.com/english.php?content=../content/1033/twincat_bsd/12304716299.html\&id=), DEVICE -> Connectivity -> add route, at `https://<plc>/config`

Whichever tool you use, these are the values that matter:

| Field                 | Value                                                                                                   |
| --------------------- | ------------------------------------------------------------------------------------------------------- |
| Route name            | anything; it only labels the entry                                                                      |
| AMS Net ID            | must equal what the client advertises: `hostAMS`, or `hostIP` + `.1.1` when `hostAMS` is `auto`         |
| Address               | the address the PLC sees the client as; prefer an IP over a host name unless DNS is reliable on the PLC |
| Transport type        | `TCP_IP`                                                                                                |
| Static, not temporary | a temporary route is lost on the next restart                                                           |

The AMS Net ID and the Address are independent: the NetID need not resemble the address, which is exactly what makes a manual route usable behind NAT.

One asymmetry to expect: between two TwinCAT systems a route is created on **both** ends. You add it on one side and the peer gets its own entry automatically, which is what the *Remote Route* options in the XAE dialog are for. This plugin is not a TwinCAT system and runs no router, so only the PLC-side entry exists. There is no second route to look for, and *Remote Route* can be left at `None / Server`.

**On the device itself**, routes live in `StaticRoutes.xml`:

| Platform             | Path                                             |
| -------------------- | ------------------------------------------------ |
| TwinCAT 3 on Windows | `C:\TwinCAT\3.1\Target\StaticRoutes.xml`         |
| CX with Windows CE   | `\Hard Disk\TwinCAT\3.1\Target\StaticRoutes.xml` |

Editing it directly is another way to add routes with no engineering station attached. Refer to the official documentation [Create or delete ADS routes manually](https://infosys.beckhoff.com/content/1033/twincat_bsd/12459254539.html):

```xml
<Route>
    <Name>BENTHOSADS-100.97.160.53</Name>
    <Address>10.13.37.1</Address>
    <NetId>100.97.160.53.1.1</NetId>
    <Type>TCP_IP</Type>
    <Flags>64</Flags>
</Route>
```

`NetId` must match the source NetID the client advertises; `Address` is the address the PLC holds for that client. The example is the NAT case written out: the address is a gateway on the PLC's own subnet while the NetID is the client's.

Leave the remaining fields as the tooling wrote them. The plugin only ever sets the NetID, route name, computer name and credentials.


# Troubleshooting

Start from the symptom. Every entry below refers to the plugin's own log output, which Benthos writes to stdout; adding `--log.level DEBUG` to the `benthos -c config.yaml` command adds the per-request detail. With no access to those logs, the one check worth making from the PLC side is whether a route exists for this client and what address it holds: see [Getting a route onto the PLC](/benthos-umh/input/beckhoff-ads/networking#getting-a-route-onto-the-plc).

| Symptom                                                           | Most likely cause                                                                              | Where to look                                                                                 |
| ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| Connects, registers symbols, no values arrive                     | `hostIP` is not the address the PLC sees                                                       | [Which kind of drop happened](#reading-a-disconnect)                                          |
| Log stops after `Connecting to PLC`                               | Wrong address, PLC off, or firewall on 48898                                                   | [Which kind of drop happened](#reading-a-disconnect)                                          |
| `route ... was registered but the PLC did not serve it`           | Route address or credentials                                                                   | [Repeated route failures back off](#why-retries-slow-down)                                    |
| Values arrive in bursts with gaps between                         | Connection dropping and reconnecting                                                           | [Reconnection](#reconnection)                                                                 |
| Values stop but the input keeps running                           | Notification delivery went silent                                                              | [Silent notification delivery](#silent-notification-delivery)                                 |
| The input reconnects every few seconds                            | A health check port-scanning 48898                                                             | [PLC connection check](/benthos-umh/input/beckhoff-ads/networking#plc-connection-check)       |
| Timestamps decades in the past or future                          | The PLC's own clock, used for notification samples                                             | [Symbols and metadata](/benthos-umh/input/beckhoff-ads/symbols-and-metadata#metadata-outputs) |
| `could not bind the inbound AMS port` and every request times out | The device answers on its own connection, and `48898` is taken on this host or blocked inbound | [How a route is matched](/benthos-umh/input/beckhoff-ads/how-it-works#how-a-route-is-matched) |

## Reading a disconnect

```mermaid
flowchart TD
  A["Connect fails, or a session drops"] --> B{"Did the PLC accept the TCP connection?"}
  B -->|no| C["targetAddress, PLC powered, firewall on 48898"]
  B -->|yes| D{"Did it serve any AMS frame?"}
  D -->|no| E["The PLC will not serve this route:<br/>check hostIP against the source address<br/>the PLC sees, and the credentials"]
  D -->|yes| F{"Do the drops arrive after a similar<br/>interval, while frame counts vary?"}
  F -->|yes| G["Check the route Address against the<br/>source address the PLC sees"]
  F -->|no| H["Transport or device-side reset:<br/>network path, or another client evicting this one"]
```

A failed connect logs a `hint` field naming the likely cause, and the PLC closing the connection is reported as one of two different cases:

| Log says                                                  | Meaning                                                    | Where to look                                                                                                                                                        |
| --------------------------------------------------------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| the PLC did not accept a TCP connection                   | Nothing answered on the ADS port                           | `targetAddress`, PLC powered, firewall                                                                                                                               |
| closed it without serving a single AMS frame              | TCP accepted, but the route does not authorize this client | `username`/`password`, `hostIP` (set it explicitly behind NAT or a VPN), another client holding the route from the same host                                         |
| dropped a connection that was already carrying AMS frames | Transport or device-side reset, not a configuration error  | The network path, for VPN or subnet-router flaps. Or another client evicting this one: a Beckhoff AMS router serves one TCP connection per host and closes the older |

If those drops arrive after a similar interval each time while the frame counts vary, suspect configuration rather than the network: most often a route whose `Address` does not match the source the PLC sees. See [PLC behind a NATing gateway](/benthos-umh/input/beckhoff-ads/networking#plc-behind-a-nating-gateway-or-subnet-router).

## Reconnection

A session that is down is logged at error level: nothing is read while it lasts, and with `readType: notification` the samples in the gap are gone. Under UMH Core that also holds the bridge out of active until it recovers.

The plugin automatically reconnects when the TCP connection is lost (e.g. network cable unplugged, PLC restart). TCP keepalive is set to probe after 3 s idle, then every 2 s, five times, so a dead connection is declared after about 13 s. On reconnect, the plugin:

1. Re-establishes the TCP connection, retrying indefinitely on the backoff ramp below
2. Reloads the symbol table from the PLC
3. Re-subscribes all notification handles

No manual intervention is needed.

## A device that keeps dropping

Reconnect delays ramp `1s → 5s → 15s → 30s` and stay at the cap. A connection that drops repeatedly is treated as flapping and gets that cap as a cooldown. A PLC that resets an established connection every few seconds therefore spends most of its time waiting rather than reading. That is deliberate: every reconnect costs the PLC an accepted socket, and the default protects its socket table rather than stream continuity.

Where the samples matter more than the sockets, lower the cap, accepting one more accepted socket on the PLC per reconnect:

```yaml
maxReconnectInterval: 5s
```

The lower tiers are pulled down with it so the ramp stays valid. Check the drop verdict below first: if the drops are the route not being served, a faster reconnect makes it worse.

## Silent notification delivery

Notification delivery can stop **without the TCP connection dropping**, so there is no error and no reconnect. The [connection heartbeat](/benthos-umh/input/beckhoff-ads/how-it-works#the-connection-heartbeat) is what notices: once its beats stop for `notificationSilenceTimeout`, the subscriptions count as dead and `heartbeatRecovery` decides what happens next:

| Value                 | Behavior                                                                                                    | Use when                                                                                                                                                                                                                              |
| --------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `immediate` (default) | Re-subscribes at once: one delete plus one add per symbol, in a burst                                       | The PLC answers reliably; fastest recovery from a genuine subscription death                                                                                                                                                          |
| `confirm`             | Waits for a second consecutive silent window, then re-subscribes                                            | The PLC stalls under load. Twice as slow to notice a real death, but one late beat no longer churns every handle                                                                                                                      |
| `rebuild`             | Drops the session and reconnects from scratch, re-registering every subscription as part of a fresh connect | The PLC ignores the churn. A device that is not answering times out on both the delete and the add, so every symbol costs two `requestTimeout` waits (5 s each by default) and never recovers, where a reconnect takes about a second |

An input that keeps running while delivering nothing is the failure this protects against: with `rebuild` the reconnect is visible in the logs, and the plugin reports itself disconnected so the pipeline restarts the input rather than sitting on dead handles.

## Why retries slow down

The plugin paces failed connects, and how long it waits depends on why the connect failed:

| Failure                                   | Next attempt                                                                 |
| ----------------------------------------- | ---------------------------------------------------------------------------- |
| Ordinary (dial refused, session rejected) | 1s, doubling to a 1 min ceiling; a rebooting PLC is still picked up promptly |
| The PLC will not serve the route          | 30s, doubling to 5 min                                                       |
| Three consecutive route failures          | Route registration is skipped entirely for 5 minutes                         |

The log says which is happening: `Waiting before the next connect attempt` with the delay, and on the third route failure a warning that registration is being skipped and that `hostIP` must be the address the PLC sees. Any successful connect clears all of it.

The pacing exists because retrying cannot fix a route the PLC will not serve. Every attempt costs the PLC a route registration and a socket, and enough of them will wedge its route table.

While registration is skipped the plugin also skips the probe, so the clean "route not served" verdict is unavailable until the window expires. That is the trade for not writing to the device's route table.


# Siemens S7

This input is tailored for the S7 communication protocol, facilitating a direct connection with S7-300, S7-400, S7-1200, and S7-1500 series PLCs.

For more modern PLCs like the S7-1200 and S7-1500 the following two changes need to be done to use them:

1. "Optimized block access" must be disabled for the DBs we want to access
2. In the "Protection" section of the CPU Properties, enable the "Permit access with PUT/GET" checkbox

**Configuration**

```yaml
input:
  s7comm:
    tcpDevice: '192.168.0.1' # IP address of the S7 PLC (optionally with port, e.g., '192.168.0.1:102')
    rack: 0                  # Rack number of the PLC. Defaults to 0
    slot: 1                  # Slot number of the PLC. Defaults to 1
    timeBetweenReads: 1s     # Time waited between two attempts to read data from the device. Defaults to 1s
    batchMaxSize: 480        # DEPRECATED: This field never worked correctly because batches were calculated before connecting to the PLC, so the actual negotiated PDU size was unknown. PDU size is now automatically negotiated during connection.
    timeout: 10              # Timeout in seconds for connections and requests. Default to 10
    disableCPUInfo: false    # Set this to true to not fetch CPU information from the PLC
    addresses:               # List of addresses to read from
      - "DB1.DW20"     # Accesses a double word at location 20 in data block 1
      - "DB1.S30.10"   # Accesses a 10-byte string at location 30 in data block 1
```

**Configuration Parameters**

* **tcpDevice**: IP address or hostname of the Siemens S7 PLC, optionally with port (e.g., `192.168.0.1:102`). If no port is specified, the default S7 port 102 is used.
* **rack**: Identifies the physical location of the CPU within the PLC rack.
* **slot**: Identifies the specific CPU slot within the rack.
* **timeBetweenReads**: Time the input waits between two attempts to read data from the device. Takes a duration string with a unit, such as `500ms`, `1s` or `2m`. The units are `ns`, `us`, `ms`, `s`, `m` and `h`. A bare number such as `1000` is rejected at startup. Defaults to `1s`. No upper limit is enforced. A negative value or zero is not a usable interval: the input logs a warning and falls back to `1s`. Low values can overload the PLC.
* **timeout**: Timeout duration in seconds for connection attempts and read requests.
* **disableCPUInfo**: Set this to true to not fetch CPU information from the PLC. Should be used when you get the error 'Failed to get CPU information'
* **addresses**: List of PLC memory addresses to read. See [Address Format](#address-format) below.

## Address Format

Each address tells benthos-umh **where** to read in the PLC memory and **what data type** to expect.

**For Data Blocks** (block number required):

```
DB<number>.<type><offset>[.<extra>]
```

Example: `DB1.DW20` — double word at offset 20 in data block 1.

**For all other areas** (no block number):

```
<area>.<type><offset>[.<extra>]
```

Example: `PE.W0` — input word at offset 0.

Breaking this down:

| Part     | What it means                 | Example                                   |
| -------- | ----------------------------- | ----------------------------------------- |
| `area`   | Memory area in the PLC        | `DB`, `MK`, `PE`, `PA`, `C`, `T`          |
| `number` | Which data block (DB only)    | `1` in `DB1` (Data Block 1)               |
| `type`   | Data type to read             | `DW`, `X`, `S`, `R`, etc.                 |
| `offset` | Byte position within the area | `20` in `DB1.DW20` (starts at byte 20)    |
| `extra`  | Required for some types only  | Bit number for `X`, string length for `S` |

Only **DB** (Data Blocks) uses a block number.

### Memory Areas

| Area | Name           | Description                            | Example    |
| ---- | -------------- | -------------------------------------- | ---------- |
| `DB` | Data Block     | Main data storage — most commonly used | `DB1.DW20` |
| `PE` | Process Input  | Physical inputs (sensors, switches)    | `PE.B0`    |
| `PA` | Process Output | Physical outputs (actuators, relays)   | `PA.W0`    |
| `MK` | Merker (Flags) | Internal boolean/word flags            | `MK.W0`    |
| `C`  | Counter        | Hardware counters                      | `C.W0`     |
| `T`  | Timer          | Hardware timers                        | `T.W0`     |

### Data Types

| Type | Name           | Size     | Output Type                 | Extra Required?        | Example Address |
| ---- | -------------- | -------- | --------------------------- | ---------------------- | --------------- |
| `X`  | Bit            | 1 bit    | `bool`                      | Yes — bit number (0–7) | `DB1.X5.2`      |
| `B`  | Byte           | 1 byte   | `uint8`                     | No                     | `DB1.B10`       |
| `C`  | Char           | 1 byte   | `string` (single character) | No                     | `DB1.C10`       |
| `W`  | Word           | 2 bytes  | `uint16`                    | No                     | `DB1.W20`       |
| `I`  | Integer        | 2 bytes  | `int16` (signed)            | No                     | `DB1.I20`       |
| `DW` | Double Word    | 4 bytes  | `uint32`                    | No                     | `DB1.DW100`     |
| `DI` | Double Integer | 4 bytes  | `int32` (signed)            | No                     | `DB1.DI100`     |
| `R`  | Real           | 4 bytes  | `float32`                   | No                     | `DB1.R200`      |
| `DT` | Date/Time      | 8 bytes  | `int64` (Unix nanoseconds)  | No                     | `DB1.DT0`       |
| `S`  | String         | variable | `string`                    | Yes — max length (≥ 1) | `DB1.S30.10`    |

### The Extra Parameter

Two data types require the extra parameter (the part after the second `.`):

**Bit (`X`) — specify which bit (0–7) within the byte:**

```
DB1.X5.2
       │ └─ bit 2 (third bit, counting from 0)
       └─── byte offset 5
```

Bit numbering: `0` is the least significant bit, `7` is the most significant.

**String (`S`) — specify the maximum string length:**

```
DB1.S30.10
        │  └─ max 10 characters
        └──── byte offset 30
```

The PLC stores strings with a 2-byte header (max length + actual length), so `DB1.S30.10` reads 12 bytes starting at offset 30.

All other types must **not** have an extra parameter.

### Examples

```yaml
addresses:
  # Data Block reads
  - "DB1.X0.0"      # Bit 0 of byte 0 — a boolean flag
  - "DB1.DW8"       # Unsigned 32-bit double word at offset 8
  - "DB1.R16"       # 32-bit float at offset 16
  - "DB1.S28.50"    # String of up to 50 chars starting at offset 28

  # Process Inputs (PE) — reading from sensors, switches, etc.
  - "PE.X0.0"       # Input bit 0 — e.g., a digital sensor
  - "PE.X0.7"       # Input bit 7
  - "PE.B2"         # Input byte at offset 2
  - "PE.W4"         # Input word at offset 4 — e.g., an analog sensor value

  # Process Outputs (PA) — reading back output states
  - "PA.X0.0"       # Output bit 0 — e.g., a relay state
  - "PA.W0"         # Output word at offset 0

  # Merker / Flags (MK)
  - "MK.X0.0"       # Merker bit — internal boolean flag
  - "MK.W10"        # Merker word at offset 10
```

### Mapping from TIA Portal

When reading addresses from a TIA Portal project, map them like this:

| TIA Portal    | benthos-umh | Notes                                                   |
| ------------- | ----------- | ------------------------------------------------------- |
| `DB1.DBX 5.2` | `DB1.X5.2`  | Data bit — drop "DB" prefix from type                   |
| `DB1.DBB 10`  | `DB1.B10`   | Data byte                                               |
| `DB1.DBW 20`  | `DB1.W20`   | Data word                                               |
| `DB1.DBD 100` | `DB1.DW100` | Data double word (unsigned)                             |
| `DB1.DBD 100` | `DB1.DI100` | Data double word (signed) — same offset, different type |
| `DB1.DBD 200` | `DB1.R200`  | Data real (float)                                       |
| `M 0.0`       | `MK.X0.0`   | Merker bit                                              |
| `MW 10`       | `MK.W10`    | Merker word                                             |
| `I 0.0`       | `PE.X0.0`   | Input bit                                               |
| `IW 0`        | `PE.W0`     | Input word                                              |
| `Q 0.0`       | `PA.X0.0`   | Output bit                                              |
| `QW 0`        | `PA.W0`     | Output word                                             |

**Output**

Similar to the OPC UA input, this outputs for each address a single message with the payload being the value that was read. To distinguish messages, you can use meta("s7\_address") in a following benthos bloblang processor.

**Batching Behavior**

Addresses are automatically split into batches based on S7 protocol constraints:

* **Max 20 items per request**: The S7 protocol limits AGReadMulti to 20 addresses per request
* **PDU size**: The request and response must fit within the negotiated PDU size (typically 240-480 bytes depending on the PLC model)

When more than 20 addresses are configured or the combined data exceeds PDU limits, multiple sequential requests are made. This has implications:

* **Timing**: Addresses in different batches are read at slightly different times, resulting in different timestamps
* **Performance**: More batches means more round-trips to the PLC, increasing total read time


# Ethernet/IP

The plugin is designed to read data from configured tags within Rockwell Automation PLCs that support the CIP protocol over Ethernet/IP, such as **ControlLogix**, **CompactLogix**, and **Micro820**. It uses the [`gologix`](https://github.com/danomagnum/gologix) driver — a native Go implementation modeled after pylogix — to establish and maintain communication with the controller.

Currently, the plugin behaves similarly to the `s7comm` plugin: you must explicitly configure the **tag names** and **data types** you want to read. Automatic browsing or discovery of tags is **not yet implemented**. As such, you should already know the tag names and structures from the PLC program (e.g., via Studio 5000 or Connected Components Workbench).

> ⚠️ Support is limited to modern controllers that use **CIP over Ethernet/IP**. Legacy PLCs such as **PLC-5**, **SLC-500**, or **MicroLogix** models using **PCCC** are *not supported*.

In future versions, support for **browsing** and **listing available tags** directly from the controller may be added to improve usability and reduce manual configuration. **Datatypes**

The plugin is being tested with multiple datatypes, therefore the following datatypes are verified for compatibility:

* `Boolean`
* `Byte`
* `Int8`
* `Int16`
* `Int32`
* `Int64`
* `UInt16`
* `UInt32`
* `UInt64`
* `Float32`
* `Float64`
* `String`
* `Array of Byte`
* `Array of Int8`
* `Array of Int16`
* `Array of Int32`
* `Array of Int64`
* `Array of Uint16`
* `Array of Uint32`
* `Array of Uint64`
* `Array of Float32`
* `Array of Float64`
* `Array of String`

**Metadata outputs**

The plugin provides metadata for each message, that can be used to create a topic for the output, as shown in the example above. The metadata can also be used to create a unique identifier for each message, which is useful for deduplication.

| Metadata       | Description                                                                                                          |
| -------------- | -------------------------------------------------------------------------------------------------------------------- |
| `eip_tag_name` | The Name of the Tag that sent the message. If an alias was specified it will replace the original naming of the tag. |
| `eip_tag_path` | A dot-separated path to the tag, which is usually used for TagSets and Tags.                                         |
| `eip_tag_type` | The data type of the node optimized for benthos, which can be either a number, string or bool.                       |

**Configuration Options**

The following options can be specified in the `benthos.yaml` configuration file:

```yaml
input:
  ethernetip:
    endpoint: '127.0.0.1:44818'
    pollRate: 1000 # optional (default: 1000) The rate in milliseconds at which to poll the EthernetIP plc
    listAllTags: false | true # (currently not supported)
    useMultiRead: false | true # (currently not supported)
    attributes:
      - path: "1-1-1" # specify the path like `Class-Instance-Attribute`
        type: "int16"
        alias: "vendorID" # optional (default: unset)
    tags:
      - name: "tagName"
        type: "bool"
        alias: "tagAlias" # optional (default: unset)
        length: 1 # optional (default: 1) you only need to set this for arrays
```

**Endpoint**

You can specify the endpoint in the configuration file. Node endpoints are automatically discovered and selected based on the authentication method.

```yaml
input:
  ethernetip:
    endpoint: '127.0.0.1:44818'
    tags:
      - name: "tagName"
        type: "bool"
```

**Attributes**

You can specify the Attributes in the configuration file:

```yaml
input:
  ethernetip:
    endpoint: '127.0.0.1:44818'
    attributes:
      - path: "1-1-1" # specify the path like `Class-Instance-Attribute`
        type: "int16"
        alias: "vendorID" # optional (default: unset)
```

* **Key**: `path`
* **Description**: The `path` always consists of this pattern: "Class-Instance-Attribute". Path to the specific attribute you want to read. Attributes are usually used for some device specific information, therefore you have to check your device's manual to check whether a specific attribute exists.
* **Example**: `1-1-1`
* **Key**: `type`
* **Description**: The `type` specifies the attributes type, which has to be set correctly otherwise you will receive an error. You can also get this from your device's manual.
* **Example**: `int16`
* **Key**: `alias`
* **Description**: The `alias` is an option to store your data into a specific name, so you can later easilier access this attribute.
* **Example**: `testAlias`

**Tags**

You can specify the Tags in the configuration file:

```yaml
input:
  ethernetip:
    endpoint: '127.0.0.1:44818'
    tags:
      - name: "testInt16" # specify the path like `Class-Instance-Attribute`
        type: "int16"
        alias: "counter" # optional (default: unset)
```

* **Key**: `name`
* **Description**: The `name` is basically the name of the tag you want to read data from. This is specified in your device's software and you have to know the name of that.
* **Example**: `testInt16`
* **Key**: `type`
* **Description**: The `type` specifies the tags type, which has to be set correctly otherwise you will receive an error. You can also get this from your device's software.
* **Example**: `int16`
* **Key**: `alias`
* **Description**: The `alias` is an option to store your data into a specific name, so you can later easilier access this tags data.
* **Example**: `testAlias`
* **Key**: `length` (only if you read from a `type` e.g. `arrayofint16`)
* **Description**: The `length` is an option to set the length of the data you want to read. This is only needed when reading arrays. Otherwise you will receive an error here.
* **Example**: 3

**Important:** The functionality of reading tag-sets in specific is not yet implemented and will need further investigation.

**List all tags (not yet implemented)**

This is not yet implemented and currently not set to a specific timeline.

**MultiRead (not yet implemented)**

This is not yet implemented and currently not set to a specific timeline, but will improve reading time out of your plc if it supports this service.


# UNS Input

> **Works exclusively with UMH Core** Inside UMH Core you can leave the block empty — `uns: {}`. If you run the plugin elsewhere, at minimum set `broker_address`.

#### 1 Quick-start (99 % of users)

**Single topic pattern:**

```yaml
input:
  uns:                     # nothing else needed on UMH Core
    umh_topic: "umh\\.v1\\..+" # optional regex – here "take everything"
pipeline:
  processors:
    - tag_processor:       # enrich, rename, move to UMH data model
        defaults: |
          msg.meta.location_path = "enterprise.demo.plant1.line1.plc1";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name      = "value";
          return msg;
output:
  uns: {}                  # hand the curated data back to UMH Core
```

**Multiple topic patterns (preferred for multiple filters):**

```yaml
input:
  uns:
    umh_topics:            # list of regex patterns
      - "umh\\.v1\\.acme\\.berlin\\..+"
      - "umh\\.v1\\.acme\\.munich\\..+"
      - "umh\\.v1\\.acme\\.paris\\.production\\..+"
pipeline:
  processors:
    - tag_processor:       # same pipeline as above
        defaults: |
          msg.meta.location_path = "enterprise.demo.plant1.line1.plc1";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name      = "value";
          return msg;
output:
  uns: {}
```

| What               | Default inside UMH Core                                                                                                          |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------- |
| **Broker address** | `localhost:9092` (embedded Redpanda)                                                                                             |
| **Kafka topic**    | **`umh.messages`** (hard-wired)                                                                                                  |
| **Consumer group** | `uns_plugin`                                                                                                                     |
| **Filter regex**   | `.*` → receive all keys; override with `umh_topic:` (single pattern) or `umh_topics:` (multiple patterns) to limit what you pull |

#### 2 Optional overrides

**Single topic pattern:**

```yaml
input:
  uns:
    umh_topic:           "umh\\.v1\\.acme\\.berlin\\..*"   # only Berlin site
    broker_address:  "edge-redpanda:9092"
    consumer_group:  "analytics_reader"
```

**Multiple topic patterns:**

```yaml
input:
  uns:
    umh_topics:          # list of regex patterns
      - "umh\\.v1\\.acme\\.berlin\\..*"
      - "umh\\.v1\\.acme\\.munich\\..*"
    broker_address:  "edge-redpanda:9092"
    consumer_group:  "analytics_reader"
```

| Field            | Purpose & Default                                                                                                              |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `umh_topic`      | **Single regex pattern** against the Kafka key (UMH topic). Default `.*`. Cannot be used with `umh_topics`.                    |
| `umh_topics`     | **List of regex patterns** against the Kafka key (UMH topic). Preferred for multiple filters. Cannot be used with `umh_topic`. |
| `broker_address` | Comma-separated bootstrap list. Default `localhost:9092`.                                                                      |
| `consumer_group` | Consumer-group ID (offset tracking). Default `uns_plugin`.                                                                     |
| `topic`          | **\[DEPRECATED]** Use `umh_topic` instead. Still supported for backward compatibility.                                         |

#### 3 What the plugin does behind the scenes

1. **Poll** `umh.messages` continuously.
2. **Filter** records whose umh\_topic matches the topic regex (fast path if you only need a subset).
3. **Convert** each record into a Benthos message:

* Payload → Value (raw bytes).
* Metadata →
  * `kafka_timestamp_ms` - Kafka record timestamp in milliseconds (when the record was written to Kafka)
  * `kafka_msg_key` - Original Kafka message key (same as umh\_topic)
  * `kafka_topic` - Physical Kafka topic name (e.g., "umh.messages")
  * `umh_topic` - **UMH topic structure** from Kafka message key (e.g., "umh.v1.enterprise.plant1.\_historian.temperature")
  * Every Kafka header becomes `meta(...)` for processors downstream

4. **Batch-safe ACK** – commits offsets only when the whole Benthos batch succeeds.

**Performance notes**

* A very complex regular expression can cost CPU (back-tracking). Stick to anchored, specific patterns such as `umh\\.v1\\.acme\\.(berlin|munich)\\..*` rather than `.*temperature.*` if throughput matters.

#### 4 Typical end-to-end flow

```mermaid
graph LR
  A[Redpanda umh.messages] -->|uns input| B[Tag Processor]
  B -->|curated<br>topic & payload| C[uns output]
  C --> D[Redpanda umh.messages]
```

Use this pattern when you need to tap into the Unified Namespace, enrich / transform the stream, and publish the result back (or to another cluster).

#### 5 FAQs / Troubleshooting

* **"No messages appear"** – your regex under `umh_topic:` or `umh_topics:` filtered everything out. Start with `.*` (or `[".*"]` for umh\_topics), verify flow, then tighten the pattern.
* **"consumer group lag grows forever"** – your pipeline never ACKs. Check downstream processors / outputs for errors; the input only commits on ACK.


# Libraries

This section contains documentation for core libraries and utilities used across UMH components.

These libraries provide shared functionality that is used by multiple plugins and components throughout the system.

## Available Libraries

* [UMH Topic Parser](/benthos-umh/libraries/umh-topic-parser) - High-performance parsing, validation, and construction utilities for UMH topics


# UMH Topic Parser

The UMH Topic Parser is a high-performance Go package that provides parsing, validation, and construction utilities for UMH (Unified Manufacturing Hub) topics. It ensures that all topics conform to the UMH specification and can be safely used as Kafka message keys.

## Overview

UMH topics follow a strict hierarchical structure that represents the physical and logical organization of manufacturing systems. This package centralizes topic handling across all UMH components, providing consistent validation and preventing bad data from entering the UNS.

### Topic Structure

UMH topics follow this format:

```
umh.v1.<location_path>.<data_contract>[.<virtual_path>].<name>
```

**Components:**

* **location\_path**: 1-N hierarchical levels representing physical organization (enterprise.site.area.line...)
* **data\_contract**: Service/schema identifier starting with underscore (\_historian, \_analytics, etc.)
* **virtual\_path**: Optional logical grouping (axis.x.position, diagnostics.\*, etc.)
* **name**: Final identifier for the specific data point

### Validation Rules

The package enforces these fixed rules:

* **Location path levels**: MUST NOT start with underscore, at least 1 level required
* **Data contract**: MUST start with underscore, cannot be just "\_"
* **Virtual path**: CAN start with underscore, optional
* **Name**: CAN start with underscore, required
* **Kafka compatibility**: Only `[a-zA-Z0-9._-]` characters allowed
* **No consecutive dots, leading dots, or trailing dots**

## Examples

### Valid Topics

```
umh.v1.enterprise._historian.temperature
umh.v1.acme.berlin._historian.pressure
umh.v1.factory.line1.station2._raw.motor.diagnostics.vibration
umh.v1.plant._analytics.efficiency._kpi.oee
```

### Invalid Topics

```
umh.v1._enterprise._historian.temp     // location cannot start with _
umh.v1.factory.historian.temp          // data contract must start with _
umh.v1.factory._._historian.temp       // data contract cannot be just _
umh.v1.factory.._historian.temp        // empty location level
umh.v1.factory._historian.temp@ture    // invalid characters
```

## Usage

### Basic Parsing and Validation

```go
import "github.com/united-manufacturing-hub/benthos-umh/pkg/umh/topic"

// Parse and validate a topic string
uns, err := topic.NewUnsTopic("umh.v1.enterprise.site._historian.temperature")
if err != nil {
    log.Fatal(err)
}

// Use the topic as a Kafka key
kafkaKey := uns.AsKafkaKey()

// Access parsed components
info := uns.Info()
fmt.Println("Level0:", info.Level0)                          // "enterprise"
fmt.Println("Location Sublevels:", info.LocationSublevels)   // ["site"]
fmt.Println("Data Contract:", info.DataContract)             // "_historian"
fmt.Println("Name:", info.Name)                              // "temperature"
```

### Programmatic Construction

```go
// Build a simple topic
topic, err := topic.NewBuilder().
    SetLevel0("enterprise").
    SetDataContract("_historian").
    SetName("temperature").
    Build()

// Build a complex topic with all components
complexTopic, err := topic.NewBuilder().
    SetLocationLevels("enterprise", "site", "area", "line").
    SetDataContract("_historian").
    SetVirtualPath("motor.axis.x").
    SetName("position").
    Build()
```

### High-Performance Scenarios

For high-throughput scenarios, reuse builder instances:

```go
builder := topic.NewBuilder()

for _, sensor := range sensors {
    topic, err := builder.Reset().
        SetLocationPath(sensor.Location).
        SetDataContract("_historian").
        SetName(sensor.Name).
        Build()
    
    if err != nil {
        log.Printf("Invalid topic for sensor %s: %v", sensor.Name, err)
        continue
    }
    
    // Process topic...
}
```

### Location Path Handling

```go
// Different ways to set location paths
builder := topic.NewBuilder()

// Method 1: Dot-separated string
builder.SetLocationPath("enterprise.site.area")

// Method 2: Individual levels
builder.SetLocationLevels("enterprise", "site", "area")

// Method 3: Incremental building
builder.SetLevel0("enterprise").
    AddLocationLevel("site").
    AddLocationLevel("area")

// Method 4: Get current path from builder
currentPath := builder.GetLocationPath() // Returns: "enterprise.site.area"
```

## Performance Characteristics

The package is optimized for high-throughput scenarios:

* **Simple topic parsing**: \~656ns per operation
* **Complex topic parsing**: \~1322ns per operation
* **Simple topic construction**: \~751ns per operation
* **Complex topic construction**: \~1436ns per operation
* **Memory allocations**: 3-5 allocations per topic creation
* **Read operations**: <1ns (String(), Info(), AsKafkaKey() are essentially free)

### Thread Safety

* **UnsTopic instances**: Immutable and safe for concurrent use
* **Builder instances**: NOT thread-safe (use separate builders per goroutine)
* **Package-level functions**: Thread-safe

## Integration Patterns

### Message Processing

```go
func processMessage(topicStr string, payload []byte) error {
    topic, err := topic.NewUnsTopic(topicStr)
    if err != nil {
        return fmt.Errorf("invalid topic: %w", err)
    }

    // Use topic for routing, validation, etc.
    return routeMessage(topic, payload)
}
```

### Location-Based Routing

```go
func routeByLocation(topic *topic.UnsTopic) string {
    info := topic.Info()
    totalLevels := 1 + len(info.LocationSublevels) // level0 + sublevels
    switch totalLevels {
    case 1:
        return "enterprise-router"
    case 2:
        return "site-router"
    default:
        return "local-router"
    }
}
```

### Bulk Topic Generation

```go
func createTopics(sensors []Sensor) ([]*topic.UnsTopic, error) {
    builder := topic.NewBuilder()
    topics := make([]*topic.UnsTopic, 0, len(sensors))

    for _, sensor := range sensors {
        topic, err := builder.Reset().
            SetLocationPath(sensor.LocationPath).
            SetDataContract("_historian").
            SetName(sensor.Name).
            Build()
        if err != nil {
            return nil, err
        }
        topics = append(topics, topic)
    }

    return topics, nil
}
```

## Error Handling

The package provides detailed error messages for debugging:

```go
topic, err := topic.NewUnsTopic("umh.v1._enterprise._historian.temp")
if err != nil {
    fmt.Println(err) // "level0 cannot start with underscore"
}
```

Error messages include:

* Specific validation rule that failed
* Position information for parsing errors
* Suggestions for common mistakes

## Migration Guide

### From Manual String Parsing

**Before:**

```go
parts := strings.Split(topicStr, ".")
if len(parts) < 5 || !strings.HasPrefix(parts[0], "umh") {
    return errors.New("invalid topic")
}
// ... manual validation logic
level0 := parts[2]
dataContract := parts[3] // This might be wrong!
```

**After:**

```go
topic, err := topic.NewUnsTopic(topicStr)
if err != nil {
    return err
}
info := topic.Info()
level0 := info.Level0
dataContract := info.DataContract // Correctly parsed
```

### From Existing Validation Code

The topic parser replaces manual validation logic throughout the codebase:

1. **topic\_browser\_plugin**: Replace `topicToUNSInfo()` and manual parsing
2. **uns\_output**: Replace manual topic validation
3. **tag\_processor\_plugin**: Replace `constructUMHTopic()` with Builder
4. **classic\_to\_core\_plugin**: Replace manual topic parsing

## API Reference

### Core Types

```go
// UnsTopic represents a validated UMH topic
type UnsTopic struct { /* ... */ }

// TopicInfo contains parsed topic components (protobuf-generated)
// Located in pkg/umh/topic/proto package
type TopicInfo struct {
    Level0            string   // Enterprise level
    LocationSublevels []string // Additional location levels
    DataContract      string   // Data contract (_historian, etc.)
    VirtualPath       *string  // Optional virtual path (pointer, can be nil)
    Name              string   // Final name segment
    Metadata          map[string]string // Kafka header metadata
}

// Builder provides fluent topic construction
type Builder struct { /* ... */ }
```

### Key Methods

```go
// Parsing and validation
func NewUnsTopic(topic string) (*UnsTopic, error)

// Topic access
func (u *UnsTopic) String() string
func (u *UnsTopic) AsKafkaKey() string  
func (u *UnsTopic) Info() *proto.TopicInfo

// Field access (direct field access, not methods)
info.Level0                // string
info.LocationSublevels     // []string
info.DataContract          // string
info.VirtualPath           // *string (can be nil)
info.Name                  // string
info.Metadata              // map[string]string

// Builder methods
func NewBuilder() *Builder
func (b *Builder) SetLevel0(level0 string) *Builder
func (b *Builder) SetLocationPath(path string) *Builder
func (b *Builder) SetLocationLevels(level0 string, additional ...string) *Builder
func (b *Builder) AddLocationLevel(level string) *Builder
func (b *Builder) SetDataContract(contract string) *Builder
func (b *Builder) SetVirtualPath(path string) *Builder
func (b *Builder) SetName(name string) *Builder
func (b *Builder) Build() (*UnsTopic, error)
func (b *Builder) BuildString() (string, error)
func (b *Builder) GetLocationPath() string
func (b *Builder) Reset() *Builder
```

## Testing

The package includes comprehensive test coverage:

* **100+ test cases** covering valid and invalid topics
* **Performance benchmarks** for all operations
* **Concurrency tests** for thread safety
* **Edge case validation** for boundary conditions

Run tests:

```bash
cd pkg/umh/topic
go test -v
go test -bench=. -benchmem
```


# Processing

This section covers Benthos processors for data transformation and message handling. Processors allow you to modify, filter, and enrich messages as they flow through your data pipelines.

## Available Processors

* [**Tag Processor**](/benthos-umh/processing/tag-processor) - Optimized for processing tags and time series data, converting them to the UMH data model within the `_historian` data contract. Provides automatic message formatting, metadata generation, and structured processing stages.
* [**Classic to Core Processor**](/benthos-umh/processing/classic-to-core-processor) - Converts UMH Historian Data Contract format messages into Core format, following the "one tag, one message, one topic" principle.
* [**Downsampler**](/benthos-umh/processing/downsampler) - Reduces time-series data volume by filtering out insignificant changes using configurable algorithms. Integrates with UMH data pipelines to compress historian data while preserving significant trends.
* [**Topic Browser**](/benthos-umh/processing/topic-browser) - **\[Internal]** Transforms UMH messages into structured data for the Topic Browser interface. Extracts hierarchical topic information and event data to enable real-time topic exploration, metadata search, and debugging in the Management Console.
* [**Stream Processor**](/benthos-umh/processing/stream-processor) - Collects timeseries data from multiple UNS sources and generates transformed messages using JavaScript expressions. Implements dependency-based evaluation where static mappings are evaluated on every message while dynamic mappings only evaluate when their dependencies are received.
* [**Node-RED JavaScript Processor**](/benthos-umh/processing/node-red-javascript-processor) - Provides full control over payload and metadata through custom JavaScript code. Use this processor when you need complex transformations, conditional logic, or custom processing beyond standard tag handling.
* [**More Processors**](https://docs.redpanda.com/redpanda-connect/components/processors/about/) - Additional built-in processors available in Benthos/Redpanda Connect for various data processing needs.

## Which Processor to Choose?

* Use **Tag Processor** when working with structured time series data that needs to conform to the UMH data model
* Use **Classic to Core Processor** when migrating from legacy UMH Historian Data Contract format to modern Core format
* Use **Downsampler** after tag\_processor to reduce data volume while preserving important changes in time-series data
* Use **Topic Browser** (internal) when implementing systems that need to consume and display UMH topic hierarchies and event streams
* Use **Stream Processor** when you need to combine data from multiple UNS sources with state management, perform complex JavaScript transformations, and generate new UMH topics based on a model/version data contract
* Use **Node-RED JavaScript Processor** when you need maximum flexibility and custom processing logic
* Explore the **additional processors** for specific use cases like JSON manipulation, HTTP requests, caching, and more


# Tag Processor

The Tag Processor is designed to prepare incoming data for the UMH data model. It processes messages through three configurable stages: defaults, conditional transformations, and advanced processing, all using a Node-RED style JavaScript environment.

Use the `tag_processor` compared to the `nodered_js` when you are processing tags or time series data and converting them to the UMH data model within the `_historian` data contract. This processor is optimized for handling structured time series data, automatically formats messages, and generates appropriate metadata.

The JavaScript stages use the same environment as the `nodered_js` processor. For the full list of available globals (`msg`, `console`, `cache`), see the [JavaScript API Reference](/benthos-umh/processing/javascript-api).

**Message Formatting Behavior**

The processor automatically formats different input types into a consistent structure with a "value" field:

1. **Simple Values (numbers, strings, booleans)**\
   Input:

```json
42
```

Output:

```json
{
  "value": 42
}
```

Input:

```json
"test string"
```

Output:

```json
{
  "value": "test string"
}
```

Input:

```json
true
```

Output:

```json
{
  "value": true
}
```

2. **Arrays** (converted to JSON string representation)\
   Input:

```json
["a", "b", "c"]
```

Output:

```json
{
  "value": "[\"a\",\"b\",\"c\"]"
}
```

3. **Objects** (preserved as JSON objects)\
   Input:

```json
{
  "key1": "value1",
  "key2": 42
}
```

Output:

```json
{
  "value": "{\"key1\": \"value1\",\"key2\": 42}"
}
```

4. **Numbers** (preserved as numbers)\
   Input:

```json
23.5
```

Output:

```json
{
  "value": 23.5
}
```

Input:

```json
42
```

Output:

```json
{
  "value": 42
}
```

This consistent formatting ensures that:

* All messages have a "value" field
* Simple types (numbers, strings, booleans) are preserved as-is
* Complex types (arrays, objects) are converted to JSON string representations
* Numbers are always preserved as numeric types (integers or floats)
* Numeric-looking strings (e.g., `"2340925"`) stay strings and keep their exact digits; set `msg.meta.datatype = "number"` to convert them explicitly

> **Breaking change in v4.0.0**: Arrays now serialize to JSON format `["a","b","c"]` instead of space-separated format `[a b c]`. This preserves type information and enables array parsing in downstream processors.

**Configuration**

```yaml
pipeline:
  processors:
    - tag_processor:
        defaults: |

          // Set default location hierarchy and datacontract
          msg.meta.location_path = "enterprise.plant1.machiningArea.cnc-line.cnc5.plc123";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name = "value";
          msg.payload = msg.payload; //does not modify the payload
          return msg;
        conditions:
          - if: msg.meta.opcua_node_id === "ns=1;i=2245"
            then: |
              // Set path hierarchy and tag name for specific OPC UA node
              msg.meta.virtual_path = "axis.x.position";
              msg.meta.tag_name = "actual";
              return msg;
        advancedProcessing: |
          // Optional advanced message processing
          // Example: double numeric values
          msg.payload = parseFloat(msg.payload) * 2;
          return msg;
```

**Processing Stages**

1. **Defaults**
   * Sets initial metadata values
   * Runs first on every message
   * Must return a message object
2. **Conditions**
   * List of conditional transformations
   * Each condition has an `if` expression and a `then` code block
   * Runs after defaults
   * Must return a message object
3. **Advanced Processing**
   * Optional final processing stage
   * Can modify both metadata and payload
   * Must return a message object

**How Metadata Works**

The tag processor uses a two-step process for handling metadata:

1. **JavaScript Stage - Working with `msg.meta`**

   In your JavaScript code (defaults, conditions, advancedProcessing), you work with the `msg.meta` object:

   ```javascript
   // In JavaScript processing stages
   msg.meta.location_path = 'enterprise.site.area'
   msg.meta.data_contract = '_historian'
   msg.meta.tag_name = 'temperature'
   msg.meta.virtual_path = 'axis.x.position'
   ```

   At this stage, these are just JavaScript object properties - they're not yet actual Benthos metadata.
2. **Conversion to Benthos Metadata**

   After the JavaScript processing is complete, the tag processor extracts all properties from the `msg.meta` JavaScript object and converts them to actual Benthos message metadata.
3. **Final Message Structure**

   **Important**: Properties set on `msg.meta` become message metadata, **not** part of the payload.

   **Metadata** (accessible via message metadata):

   * `location_path`: "enterprise.site.area"
   * `data_contract`: "\_historian"
   * `tag_name`: "temperature"
   * `virtual_path`: "axis.x.position"
   * `umh_topic`: "umh.v1.enterprise.site.area.\_historian.axis.x.position.temperature" (auto-generated)
   * `topic`: "umh.v1.enterprise.site.area.\_historian.axis.x.position.temperature" (auto-generated, deprecated)

   **Payload** (the actual message content):

   ```json
   {
     "value": 23.5,
     "timestamp_ms": 1647890123456
   }
   ```

**Integration with UNS Output**

The tag processor is typically used together with the `uns` output plugin, which is the standard way to publish data into the United Manufacturing Hub. Here's how metadata flows from tag processor to UNS:

1. **Metadata to UNS Headers**: All metadata fields set by the tag processor become UNS headers, preserving the complete context and routing information.
2. **UMH Topic as Routing Key**: The auto-generated `umh_topic` metadata becomes the UNS routing key, enabling efficient message routing and topic-based subscriptions.
3. **Payload Preservation**: The structured payload (with `value` and `timestamp_ms`) is published as-is to the UNS.

**Complete Flow Example:**

```yaml
pipeline:
  processors:
    - tag_processor:
        defaults: |
          msg.meta.location_path = "enterprise.site.area";
          msg.meta.data_contract = "_historian";
          msg.meta.tag_name = "temperature";
          msg.meta.custom_field = "sensor_data";
          return msg;
  output:
    uns: {}
```

**Input Message**: `23.5`

**UNS Record Result**:

* **UNS Topic**: `umh.messages` (fixed UNS topic)
* **UNS Routing Key**: `umh.v1.enterprise.site.area._historian.temperature` (from `umh_topic` metadata)
* **UNS Value**: `{"value": 23.5, "timestamp_ms": 1647890123456}` (structured payload)
* **UNS Headers**:

  ```
  location_path: "enterprise.site.area"
  data_contract: "_historian"
  tag_name: "temperature"
  umh_topic: "umh.v1.enterprise.site.area._historian.temperature"
  topic: "umh.v1.enterprise.site.area._historian.temperature"
  custom_field: "sensor_data"
  bridged_by: "umh-core"
  ```

**Metadata Fields**

The processor uses the following metadata fields:

**Required Fields:**

* `location_path`: Hierarchical location path in dot notation (e.g., "enterprise.site.area.line.workcell.plc123")
* `data_contract`: Data schema identifier (e.g., "\_historian", "\_analytics")
* `tag_name`: Name of the tag/variable (e.g., "temperature", "pressure")

**Optional Fields:**

* `virtual_path`: Logical, non-physical grouping path in dot notation (e.g., "axis.x.position")
* `datatype`: Forces the output value type (`"string"`, `"number"`, or `"bool"`) instead of auto-detection. Set to `"number"` to convert a numeric string (e.g., `"42"`) into a number; without it, a string value always stays a string

**Generated Fields:**

* `umh_topic`: Automatically generated from the above fields in the format:

  ```
  umh.v1.<location_path>.<data_contract>.<virtual_path>.<tag_name>
  ```

  Empty or undefined fields are skipped, and dots are normalized.

**Message Structure**

Messages in the Tag Processor follow the Node-RED style format:

```javascript
{
  payload: {
    // The message content - can be a simple value or complex object
    "value": 23.5,
    "timestamp_ms": 1733903611000
  },
  meta: {
    // Required fields
    location_path: "enterprise.site.area.line.workcell.plc123",  // Hierarchical location path
    data_contract: "_historian",                                 // Data schema identifier
    tag_name: "temperature",                                     // Name of the tag/variable

    // Optional fields
    virtual_path: "axis.x.position",                            // Logical grouping path

    // Generated field (by processor)
    umh_topic: "umh.v1.enterprise.site.area.line.workcell.plc123._historian.axis.x.position.temperature",

    // Input-specific fields (e.g., from OPC UA)
    opcua_node_id: "ns=1;i=2245",
    opcua_tag_name: "temperature_sensor_1",
    opcua_tag_group: "sensors.temperature",
    opcua_tag_path: "sensors.temperature",
    opcua_tag_type: "number",
    opcua_source_timestamp: "2024-03-12T10:00:00Z",
    opcua_server_timestamp: "2024-03-12T10:00:00.001Z",
    opcua_attr_nodeid: "ns=1;i=2245",
    opcua_attr_nodeclass: "Variable",
    opcua_attr_browsename: "Temperature",
    opcua_attr_description: "Temperature Sensor 1",
    opcua_attr_accesslevel: "CurrentRead",
    opcua_attr_datatype: "Double"
  }
}
```

**Examples**

1. **Basic Defaults Processing**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise.plant1.machiningArea.cnc-line.cnc5.plc123";
    msg.meta.data_contract = "_historian";
    msg.meta.tag_name = "actual";
    return msg;
```

Input:

```json
23.5
```

Output:

```json
{
  "value": 23.5,
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise.plant1.machiningArea.cnc-line.cnc5.plc123._historian.actual`

2. **OPC UA Node ID Based Processing**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise.plant1.machiningArea.cnc-line.cnc5.plc123";
    msg.meta.data_contract = "_historian";
    return msg;
  conditions:
    - if: msg.meta.opcua_attr_nodeid === "ns=1;i=2245"
      then: |
        msg.meta.virtual_path = "axis.x.position";
        msg.meta.tag_name = "actual";
        return msg;
```

Input with metadata `opcua_attr_nodeid: "ns=1;i=2245"`:

```json
23.5
```

Output:

```json
{
  "value": 23.5,
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise.plant1.machiningArea.cnc-line.cnc5.plc123._historian.axis.x.position.actual`

3. **Moving Folder Structures in Virtual Path**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise.plant1";
    msg.meta.data_contract = "_historian";
    msg.meta.virtual_path = msg.meta.opcua_tag_path;
    msg.meta.tag_name = msg.meta.opcua_tag_name;
    return msg;
  conditions:
    # Move the entire DataAccess_AnalogType folder and its children into axis.x
    - if: msg.meta.opcua_tag_path && msg.meta.opcua_tag_path.includes("DataAccess_AnalogType")
      then: |
        msg.meta.location_path += ".area1.machining_line.cnc5.plc123";
        msg.meta.virtual_path = "axis.x." + msg.meta.opcua_tag_path;
        return msg;
```

Input messages with OPC UA tags:

```javascript
// Original tag paths from OPC UA:
// DataAccess_AnalogType
// DataAccess_AnalogType.EURange
// DataAccess_AnalogType.Min
// DataAccess_AnalogType.Max
```

Output UMH topics will be:

```
umh.v1.enterprise.plant1.area1.machining_line.cnc5.plc123._historian.axis.x.DataAccess_AnalogType
umh.v1.enterprise.plant1.area1.machining_line.cnc5.plc123._historian.axis.x.DataAccess_AnalogType.EURange
umh.v1.enterprise.plant1.area1.machining_line.cnc5.plc123._historian.axis.x.DataAccess_AnalogType.Min
umh.v1.enterprise.plant1.area1.machining_line.cnc5.plc123._historian.axis.x.DataAccess_AnalogType.Max
```

This example shows how to:

* Match an entire folder structure using `includes("DataAccess_AnalogType")`
* Move all matching nodes into a new virtual path prefix (`axis.x`)
* Preserve the original folder hierarchy under the new location
* Apply consistent location path for the entire folder structure

4. **Advanced Processing with getLastPayload**

getLastPayload is a function that returns the last payload of a message that was avaialble in Kafka. Remember that you will get the full payload, and might still need to extract the value you need.

**This is not yet implemented, but will be available in the future.**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise.site.area.line.workcell";
    msg.meta.data_contract = "_analytics";
    msg.meta.virtual_path = "work_order";
    return msg;
  advancedProcessing: |
    msg.payload = {
      "work_order_id": msg.payload.work_order_id,
      "work_order_start_time": umh.getLastPayload("enterprise.site.area.line.workcell._historian.workorder.work_order_start_time").work_order_start_time,
      "work_order_end_time": umh.getLastPayload("enterprise.site.area.line.workcell._historian.workorder.work_order_end_time").work_order_end_time
    };
    return msg;
```

Input:

```json
{
  "work_order_id": "WO123"
}
```

Output:

```json
{
  "value": "{\"work_order_id\":\"WO123\",\"work_order_start_time\":\"2024-03-12T10:00:00Z\",\"work_order_end_time\":\"2024-03-12T18:00:00Z\"}",
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise.site.area.line.workcell._analytics.work_order`

4. **Dropping Messages Based on Value**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise";
    msg.meta.data_contract = "_historian";
    msg.meta.tag_name = "temperature";
    return msg;
  advancedProcessing: |
    if (msg.payload < 0) {
      // Drop negative values
      return null;
    }
    return msg;
```

Input:

```json
-10
```

Output: Message is dropped (no output)

Input:

```json
10
```

Output:

```json
{
  "value": 10,
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise._historian.temperature`

5. **Duplicating Messages for Different Data Contracts**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise";
    msg.meta.data_contract = "_historian";
    msg.meta.tag_name = "temperature";
    return msg;
  conditions:
    - if: true
      then: |
        msg.meta.location_path += ".production";
        return msg;
  advancedProcessing: |
    // Create two versions of the message:
    // 1. Original value for historian
    // 2. Doubled value for custom
    let doubledValue = msg.payload * 2;

    msg1 = {
      payload: msg.payload,
      meta: { ...msg.meta, data_contract: "_historian" }
    };

    msg2 = {
      payload: doubledValue,
      meta: { ...msg.meta, data_contract: "_custom", tag_name: msg.meta.tag_name + "_doubled" }
    };

    return [msg1, msg2];
```

Input:

```json
23.5
```

Output 1 (Historian):

```json
{
  "value": 23.5,
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise.production._historian.temperature`

Output 2 (custom):

```json
{
  "value": 47,
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise.production._custom.temperature_doubled`

6. **Processing Full MQTT Message Payload**

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise.area._workorder";
    msg.meta.data_contract = "_workorder";
    msg.meta.virtual_path = "new";
    msg.meta.tag_name = "maintenance";
    return msg;
```

Input:

```json
{
  "maintenanceSchedule": {
    "eventType": "ScheduledMaintenance",
    "eventId": "SM-20240717-025",
    "timestamp": "2024-07-17T13:00:00Z",
    "equipmentId": "InjectionMoldingMachine5",
    "equipmentName": "Engel Victory 120",
    "scheduledDate": "2024-07-22",
    "maintenanceType": "Preventive",
    "description": "Inspection and cleaning of injection unit and mold.",
    "maintenanceDuration": "6 hours",
    "assignedTo": {
      "employeeId": "EMP-5005",
      "name": "Hans Becker"
    },
    "status": "Scheduled",
    "partsRequired": [
      {
        "partId": "NOZZLE-015",
        "description": "Injection Nozzle",
        "quantity": 1
      }
    ],
    "notes": "Replace worn nozzle to prevent defects."
  }
}
```

Output:

```json
{
  "value": "{\"maintenanceSchedule\":{\"eventType\":\"ScheduledMaintenance\",\"eventId\":\"SM-20240717-025\",\"timestamp\":\"2024-07-17T13:00:00Z\",\"equipmentId\":\"InjectionMoldingMachine5\",\"equipmentName\":\"Engel Victory 120\",\"scheduledDate\":\"2024-07-22\",\"maintenanceType\":\"Preventive\",\"description\":\"Inspection and cleaning of injection unit and mold.\",\"maintenanceDuration\":\"6 hours\",\"assignedTo\":{\"employeeId\":\"EMP-5005\",\"name\":\"Hans Becker\"},\"status\":\"Scheduled\",\"partsRequired\":[{\"partId\":\"NOZZLE-015\",\"description\":\"Injection Nozzle\",\"quantity\":1}],\"notes\":\"Replace worn nozzle to prevent defects.\"}}",
  "timestamp_ms": 1733903611000
}
```

UMH Topic: `umh.v1.enterprise.area._workorder.maintenance`

7. **Setting Custom Timestamps**

By default, the tag processor uses the current time. You can set custom timestamps using the `timestamp_ms` metadata field (Unix milliseconds or RFC3339Nano as string):

```yaml
tag_processor:
  defaults: |
    msg.meta.location_path = "enterprise.site.area";
    msg.meta.data_contract = "_historian";
    msg.meta.tag_name = "temperature";

    // Use OPC UA timestamp either as Unix milliseconds
    if (msg.meta.opcua_source_timestamp) {
      msg.meta.timestamp_ms = new Date(msg.meta.opcua_source_timestamp).getTime().toString();
    }
    // or use it directly as RFC3339Nano
    if (msg.meta.opcua_source_timestamp) {
      msg.meta.timestamp_ms = msg.meta.opcua_source_timestamp;
    }
    // Or use Sparkplug B timestamp directly
    if (msg.meta.spb_timestamp) {
      msg.meta.timestamp_ms = msg.meta.spb_timestamp;
    }

    return msg;
```

**Note:** In the `tag_processor`, the resulting payload will always include `timestamp_ms` and one additional key corresponding to the `tag_name`. If you need to fully control the resulting payload structure, consider using the `nodered_js` processor instead. You can set the topic and payload manually, as shown below:

```yaml
pipeline:
  processors:
    - nodered_js:
        code: |
          // set kafka topic manually
          msg.meta.umh_topic = "umh.v1.enterprise.site.area._workorder.new"

          // only take two fields from the payload
          msg.payload = {
            "maintenanceSchedule": {
              "eventType": msg.payload.maintenanceSchedule.eventType,
              "description": msg.payload.maintenanceSchedule.description
            }
          }
          return msg;
```


# Classic to Core Processor

The `classic_to_core` processor converts UMH Historian schema format messages into Core format, following the "one tag, one message, one topic" principle. This processor is essential for migrating from Classic historian schemas to Core data architecture.

## Overview

The Classic to Core Processor transforms single messages containing multiple values and timestamps into separate Core format messages, each containing a single value. It handles nested tag groups by flattening them into intuitive dot-notation paths and reconstructs topics according to Core conventions.

## When to Use

Use the `classic_to_core` processor when you need to:

* **Process Classic Data in Core**: Convert Classic UMH Historian schema messages to Core format for processing in Core-based systems
* **Use Core Processors**: Enable the use of processors that rely on the Core data model (like downsampler)

## Quick Start

```yaml
pipeline:
  processors:
    - classic_to_core: {}
```

### Configuration Options

| Parameter              | Type   | Default | Description                                         |
| ---------------------- | ------ | ------- | --------------------------------------------------- |
| `target_data_contract` | string | `""`    | Target data contract. If empty, uses input's schema |

## Message Transformation

### Basic Conversion

The processor transforms single messages with multiple values into separate Core format messages:

**Input (Historian Schema):**

* Topic: `umh.v1.acme._historian.weather`
* Payload:

```json
{
  "timestamp_ms": 1717083000000,
  "temperature": 23.4,
  "humidity": 42.1
}
```

* target\_data\_contract: "\_raw"

**Output (Core Format):**

Message 1:

* Topic: `umh.v1.acme._raw.weather.temperature`
* Payload: `{"value": 23.4, "timestamp_ms": 1717083000000}`
* Metadata: `umh_topic: umh.v1.acme._raw.weather.temperature`

Message 2:

* Topic: `umh.v1.acme._raw.weather.humidity`
* Payload: `{"value": 42.1, "timestamp_ms": 1717083000000}`
* Metadata: `umh_topic: umh.v1.acme._raw.weather.humidity`

### Tag Groups (Nested Objects)

The processor flattens nested tag groups using dot notation:

**Input (Historian Schema with Tag Groups):**

* Topic: `umh.v1.acme._historian.cnc-mill`
* Payload:

```json
{
  "timestamp_ms": 1670001234567,
  "pos": {
    "x": 12.5,
    "y": 7.3,
    "z": 3.2
  },
  "temperature": 50.0,
  "collision": false
}
```

* target\_data\_contract: "\_raw"

**Output (Core Format with Flattened Tags):**

The processor creates 5 separate messages:

1. Topic: `umh.v1.acme._raw.cnc-mill.pos.x`
   * Payload: `{"value": 12.5, "timestamp_ms": 1670001234567}`
2. Topic: `umh.v1.acme._raw.cnc-mill.pos.y`
   * Payload: `{"value": 7.3, "timestamp_ms": 1670001234567}`
3. Topic: `umh.v1.acme._raw.cnc-mill.pos.z`
   * Payload: `{"value": 3.2, "timestamp_ms": 1670001234567}`
4. Topic: `umh.v1.acme._raw.cnc-mill.temperature`
   * Payload: `{"value": 50.0, "timestamp_ms": 1670001234567}`
5. Topic: `umh.v1.acme._raw.cnc-mill.collision`
   * Payload: `{"value": false, "timestamp_ms": 1670001234567}`

### Array Handling

Arrays are converted to string representation to ensure UMH-Core compliance (scalar values only):

**Input (Historian Schema with Arrays):**

* Topic: `umh.v1.factory._historian.sensors`
* Payload:

```json
{
  "timestamp_ms": 1670001234567,
  "measurements": [23.4, 24.1, 22.8],
  "sensor_ids": ["temp1", "temp2", "temp3"],
  "status_flags": [true, false, true],
  "mixed_data": ["sensor", 42, true, null]
}
```

* target\_data\_contract: "\_raw"

**Output (Core Format with Array Conversion):**

The processor creates 4 separate messages with arrays converted to strings:

1. Topic: `umh.v1.factory._raw.sensors.measurements`
   * Payload: `{"value": "[23.4 24.1 22.8]", "timestamp_ms": 1670001234567}`
2. Topic: `umh.v1.factory._raw.sensors.sensor_ids`
   * Payload: `{"value": "[temp1 temp2 temp3]", "timestamp_ms": 1670001234567}`
3. Topic: `umh.v1.factory._raw.sensors.status_flags`
   * Payload: `{"value": "[true false true]", "timestamp_ms": 1670001234567}`
4. Topic: `umh.v1.factory._raw.sensors.mixed_data`
   * Payload: `{"value": "[sensor 42 true <nil>]", "timestamp_ms": 1670001234567}`

**Array Conversion Rules:**

* Arrays are converted to space-separated string format: `[item1 item2 item3]`
* Empty arrays become `[]`
* Single-item arrays become `[item]`
* All array elements are converted using Go's `fmt.Sprintf("%v", item)` format
* Null values in arrays appear as `<nil>`
* Arrays within nested objects are also converted to strings

## Advanced Configuration

### Using Input Schema

When `target_data_contract` is not specified, the processor uses the input's schema:

```yaml
classic_to_core:
  # target_data_contract not specified - uses input's schema
```

**Input:**

* Topic: `umh.v1.acme._historian.weather`

**Output:**

* Topic: `umh.v1.acme._historian.weather.pressure`
* Maintains the original `_historian` schema

## Topic Transformation

The processor parses Classic topics and reconstructs them for Core format:

| Component     | Classic Example          | Core Example             | Description                    |
| ------------- | ------------------------ | ------------------------ | ------------------------------ |
| Prefix        | `umh.v1`                 | `umh.v1`                 | Unchanged                      |
| Location Path | `enterprise.plant1.area` | `enterprise.plant1.area` | Unchanged                      |
| Schema        | `_historian`             | `_raw` (configurable)    | Updated based on configuration |
| Context       | `weather`                | `weather`                | Becomes virtual\_path          |
| Field Name    | N/A                      | `temperature`            | Added for each field           |

**Example Transformation:**

* **Input Topic**: `umh.v1.enterprise.plant1.area._historian.weather`
* **Output Topic**: `umh.v1.enterprise.plant1.area._raw.weather.temperature`

## Metadata Handling

The processor sets the following metadata fields:

**Generated Metadata Fields:**

* `topic`: The new Core topic
* `umh_topic`: Same as topic (enables direct use with `uns_output`)
* `location_path`: Extracted from original topic
* `data_contract`: The target data contract (or input's schema if not specified)
* `tag_name`: The field name
* `virtual_path`: Original context (if present)

**Metadata Preservation:** Original metadata is always preserved alongside new fields.

## Performance & Reliability

### Built-in Safeguards

The processor includes several safeguards for production use:

* **Recursion Limit**: Maximum recursion depth of 10 levels for flattening nested tag groups
* **Message Size Limit**: Maximum of 1000 tags per message to prevent memory exhaustion
* **Topic Validation**: Strict UMH v1 topic format validation with proper error handling
* **Comprehensive Metrics**: Tracks processing counts, errors, and limit violations for monitoring

### Performance Considerations

* **Message Expansion**: Each input message creates N output messages (N = number of data fields)
* **Memory Usage**: Metadata is copied for each output message
* **Processing Overhead**: Minimal - efficient string parsing with optimized allocations

## Error Handling

The processor includes comprehensive error handling and logging for non-standard messages. Each error condition is logged with detailed information to help diagnose issues:

| Error Condition          | Log Message Example                                          | Behavior                      | Metrics               |
| ------------------------ | ------------------------------------------------------------ | ----------------------------- | --------------------- |
| Invalid JSON             | "failed to parse as structured data: ..."                    | Message skipped, error logged | `messages_errored`    |
| Missing timestamp\_ms    | "timestamp field 'timestamp\_ms' not found in payload"       | Message skipped, error logged | `messages_errored`    |
| Invalid timestamp format | "failed to parse timestamp: ..."                             | Message skipped, error logged | `messages_errored`    |
| Missing topic metadata   | "no topic found in message metadata"                         | Message skipped, error logged | `messages_errored`    |
| Invalid topic format     | "invalid topic structure, expected at least 4 parts: ..."    | Message skipped, error logged | `messages_errored`    |
| Invalid UMH prefix       | "invalid UMH topic prefix, expected 'umh.v1': ..."           | Message skipped, error logged | `messages_errored`    |
| Missing data contract    | "no data contract found in topic: ..."                       | Message skipped, error logged | `messages_errored`    |
| Missing location path    | "missing location path in topic: ..."                        | Message skipped, error logged | `messages_errored`    |
| Recursion depth exceeded | "Maximum recursion depth of 10 reached, stopping flattening" | Flattening stopped at limit   | `recursion_limit_hit` |
| Too many tags            | "Message exceeds maximum tag limit of 1000, truncating"      | Processing stopped at limit   | `tag_limit_exceeded`  |

All error messages include the original message content and specific details about what went wrong, making it easy to identify and fix issues with non-standard messages.

## Metrics

The processor exposes comprehensive metrics for monitoring:

| Metric                | Type    | Description                             |
| --------------------- | ------- | --------------------------------------- |
| `messages_processed`  | Counter | Total input messages processed          |
| `messages_errored`    | Counter | Messages that failed processing         |
| `messages_expanded`   | Counter | Total output messages created           |
| `messages_dropped`    | Counter | Messages dropped due to configuration   |
| `recursion_limit_hit` | Counter | Times recursion depth limit was reached |
| `tag_limit_exceeded`  | Counter | Times tag limit was exceeded            |

These metrics can be used to monitor the health of the processor and identify patterns of non-standard messages that need attention.

## Complete Integration Example

Here's a complete Benthos configuration for migrating from Classic to Core format:

```yaml
input:
  kafka:
    addresses: ['kafka:9092']
    topics: ['umh.v1.+.+.+._historian.+']
    consumer_group: historian-to-core-migration

pipeline:
  processors:
    - classic_to_core: {}

output:
  # Use UNS output for seamless integration
  uns_output: {}
```

**This configuration:**

1. Consumes all Classic `_historian` topics
2. Converts them to Core format using the `_raw` data contract
3. Publishes to individual Core topics via `uns_output`
4. Uses `umh_topic` metadata for automatic topic routing
5. Preserves all metadata for downstream processing

## Troubleshooting

### Common Issues

**No output messages**

* Check that input has valid JSON with timestamp\_ms field
* Verify topic metadata is present
* Ensure input topics follow Classic format: `umh.v1.<location>._historian.<context>`

**Missing fields**

* Check logs for parsing errors
* Ensure fields are not nested beyond maximum recursion depth (10 levels)

**Wrong topics**

* Validate input topic format
* Check `target_data_contract` configuration
* Verify UMH v1 topic structure

**Timestamp errors**

* Check that timestamp\_ms field contains numeric values (int, float, or string numbers)
* Ensure timestamp values are valid Unix timestamps


# Downsampler

### Downsampler in a Nutshell

The UMH **Downsampler** sits between the field bus and your persistence layer and performs one task with provable bounds: **it discards samples that do not contribute new information**.

#### Supported compression modes

* **Dead-band** – forwards every *value change*, suppresses byte-for-byte repeats, and—if `max_time` is set—adds an audit heartbeat.
* **Swinging-Door Trending (SDT)** – maintains a slope-aware envelope; a sample is forwarded only when it would exceed a fixed vertical error bound.

Wildcard overrides let you apply a conservative baseline (`deadband {threshold: 0}`) and refine it only where tighter compression is beneficial.

#### Recommended deployment points

1. **First hop into the Unified Namespace** *Configuration:* `deadband {threshold: 0, max_time: 30m}` *Effect:* Removes pure duplicates while guaranteeing one confirmatory message every 30 min. No latency is introduced.
2. **Noise suppression anywhere in the stream** *Configuration:* `deadband {threshold: 2 × σ_noise, max_time: 1h}` *Effect:* Removes fluctuations beneath the sensor's specified noise floor yet retains all legitimate steps. No latency is introduced.
3. **Final stage before the historian** *Configuration:* `swinging_door {threshold: 2 × σ_noise, min_time: physics_limit, max_time: 1h}` *Effect:* Achieves 95–99 % volume reduction while preserving slope integrity and audit heartbeat. Introduces latency of up to `max_time` as the algorithm needs to buffer the last point before it can be emitted.

Discrete counters and alarms should remain on `deadband {threshold: 0}`; SDT's emit-previous logic could delay a critical state change.

> **Compliance note** – Both algorithms enforce an absolute error bound and maintain a heartbeat, but they still remove data. If your GxP / 21 CFR Part 11 process mandates a full raw stream, archive the unfiltered UMH-core topic in parallel.

***

### 1 Quick-start Configuration

```yaml
processors:
  - downsampler:
      default:                     # safe baseline for every topic
        late_policy: passthrough   # global policy for late arrivals
        deadband:
          threshold: 0             # keep every change
          max_time: "30m"          # 30-minute heartbeat
      overrides:                   # finer control
        - pattern: "*.temperature"
          deadband:
            threshold: 0.1         # 2 × 0.05 °C noise
        - pattern: "*.furnace*"
          late_policy: drop        # can override per pattern
          swinging_door:
            threshold: 0.1
            min_time: "5s"
            max_time: "1h"
```

*Nothing else is required.* Messages that aren't strict UMH-core time-series (`value` + `timestamp_ms`) pass straight through.

**Pattern Matching:** The `pattern` field uses **shell-style glob patterns** (not regex). Supported wildcards:

* `*` matches any sequence of characters (e.g., `"*.temperature"` matches topics ending in "temperature")
* `?` matches any single character
* `[abc]` matches any character in brackets
* Examples: `"*.temperature"`, `"*sensor*"`, `"temp_?"`, `"sensor[12]"`

**Important:** When an override pattern specifies both `deadband` and `swinging_door` algorithms, the more advanced **swinging door** algorithm takes precedence and deadband parameters are ignored.

***

### Algorithm & Parameter Deep Dive

The Downsampler exposes just four parameters; once you understand their purpose you can configure the system with confidence.

| Parameter     | Purpose                                                                                                                   | Typical setting                                                                                                                                | Issue it prevents                                                               |
| ------------- | ------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| `threshold`   | Maximum absolute deviation allowed before a new point must be stored (same units as the signal). Optional, defaults to 0. | **2 × sensor-noise σ** · If σ is unknown, inspect a steady period and take ± peak spread. **Default 0** = keep any change, drop exact repeats. | Removes pure measurement noise without masking genuine step changes.            |
| `min_time`    | Smallest physically realistic interval between meaningful changes. Available only on `swinging_door`                      | Fastest credible process period (e.g. 1 s for furnaces, 50 ms for servo torque). `0` = off                                                     | Suppresses transients caused by bursty drivers or unstable links.               |
| `max_time`    | Heart-beat that forces an output even during flat periods; also flushes any SDT buffer.                                   | 15 min – 1 h (aligns with 21 CFR §11 "system liveness").                                                                                       | Ensures line-flat sensors remain visible and internal buffers stay bounded.     |
| `late_policy` | Action for out-of-order samples.                                                                                          | `passthrough` (default) or `drop`.                                                                                                             | Lets you balance historical accuracy against traffic volume on skewed networks. |

#### Dead-band

* **Rule** Emit when `|v − last| ≥ threshold` or when `max_time` expires.
* **Best suited to** Duplicate removal, discrete states, counters, Boolean flags.
* **Tuning notes**
  * `threshold` is optional and defaults to 0 if not specified.
  * `threshold = 2 × σ_noise` removes sensor noise yet preserves real transitions.
  * `threshold = 0` (default) provides safe, universal de-duplication.
  * Keep `max_time` within 15 – 60 min to satisfy audit requirements.

#### Swinging-Door Trending (SDT)

* **Rule** Maintain rotating upper and lower "doors" that bound the current slope; emit the **previous** point when a new sample would violate the envelope.
* **Additional parameters**
  * `threshold` – vertical tolerance (same intuition as for dead-band).
  * `min_time` – enforces physical plausibility for fast-changing signals.
  * `max_time` – still provides the audit heart-beat and buffer flush.
* **Why choose SDT** Captures long ramps with just two points, eliminating the stair-step pattern created by naïve dead-band on slow trends, while mathematically guaranteeing the reconstruction error remains ≤ `threshold`.

### Edge-case Handling & Internal Behaviour

The Downsampler runs each series in its own finite-state machine and guards two invariants:

1. **No silent loss** A point is acknowledged only after it—or the predecessor that still sits in the buffer—has been forwarded downstream.
2. **Bounded memory** At most **one** candidate is stored per series; an idle-flush watchdog empties it after `max_time`.

***

#### How incoming records are routed

| Payload type            | Processing path                                    | Reasoning                                                                                 |
| ----------------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| *Numeric* (int / float) | Selected algorithm (`deadband` or `swinging_door`) | These are the signals where compression counts.                                           |
| *Boolean, string*       | Change detector (`value ≠ last_value`)             | For states and text, only transitions matter; thresholds are irrelevant.                  |
| Other JSON structures   | **Bypass** (fail-open)                             | Complex objects could hide multiple semantics—forward untouched rather than risk pruning. |

***

#### Late-arrival strategy

*The timestamp of each processed sample is compared with the most recent one already seen for that series.*

* `passthrough` (default) Forward the late sample raw, flagging it with `meta:late_oos=true` (oos = "out of sequence").
* `drop` Discard and increment a metric counter—useful when stale data has no value but bandwidth is critical.

Both modes preserve at-least-once delivery for in-order traffic; only you decide what to do with stragglers.

***

#### Buffered-emit logic (needed by SDT)

1. A candidate point is retained while the doors stay open.
2. When a violation occurs, the **previous** candidate is released and the new sample becomes the fresh candidate.
3. If traffic stops before the doors close, the watchdog flushes the last candidate after `max_time` to guarantee visibility.

The same mechanism ensures a graceful shutdown: during a Benthos drain, every buffered point is emitted before the plug-in confirms closure.

#### Dynamic, per-message compression hints

Instead of hard-coding a rule map inside the *downsampler* you can let each message declare its own compression wishes in **metadata**. The downsampler simply reads the hints, validates them, and uses them in place of the global/default settings. This is useful if you want to configure everything for a single tag in a single place - the tag\_processor.

***

**Metadata keys**

| Meta key         | Type / allowed values                    | Maps to parameter             |
| ---------------- | ---------------------------------------- | ----------------------------- |
| `ds_algorithm`   | `"deadband"` \| `"swinging_door"`        | algorithm switch              |
| `ds_threshold`   | float                                    | `threshold`                   |
| `ds_min_time`    | Go/ISO-style duration string (`"750ms"`) | `min_time` (SDT)              |
| `ds_max_time`    | duration string (`"15m"`, `"1h"`, …)     | `max_time`                    |
| `ds_late_policy` | `"passthrough"` \| `"drop"`              | late-arrival rule             |
| `ds_ignore`      | any non-empty value                      | bypass downsampler completely |

**Precedence**

```
per-message metadata  →  pattern override  →  global default
```

If any field is missing the cascade falls back to the next level, so you can override only what matters.

**Complete bypass with `ds_ignore`**

In UMH deployments, the downsampler is typically enabled by default with conservative settings to provide data compression benefits across all time-series data. However, certain message types require complete bypass of any downsampling logic to ensure data integrity.

The `ds_ignore` metadata key provides this capability and is designed to be used primarily in the **tag\_processor** to selectively bypass downsampling based on message characteristics:

* **Any non-empty value** for `ds_ignore` will cause the message to skip all downsampling logic
* The message passes through unchanged, preserving all original data and metadata
* A `downsampled_by: "ignored"` metadata field is added for traceability
* No series state is created or modified for ignored messages
* Ignored messages are counted in the `messages_ignored` metric

**Common UMH use cases**

**Critical alarms and alerts:**

```yaml
pipeline:
  processors:
    - tag_processor:
        conditions:
          - if: msg.meta.tag_name === "emergency_stop" || msg.meta.tag_name.endsWith("_alarm")
            then: |
              msg.meta.ds_ignore = "true"
              return msg;
    - downsampler: 
        default:
          deadband:
            threshold: 1.0
            max_time: "30m"
```

**High-precision measurements:**

```yaml
pipeline:
  processors:
    - tag_processor:
        conditions:
          - if: msg.meta.virtual_path && msg.meta.virtual_path.includes("calibration")
            then: |
              msg.meta.ds_ignore = "bypass_precision_data"
              return msg;
    - downsampler: 
        default:
          deadband:
            threshold: 0.1
```

**State change events:**

```yaml
pipeline:
  processors:
    - tag_processor:
        conditions:
          - if: msg.meta.data_contract === "_state" || msg.meta.tag_name.startsWith("mode_")
            then: |
              msg.meta.ds_ignore = "preserve_state_changes"
              return msg;
    - downsampler: 
        default:
          deadband:
            threshold: 2.0
```

This approach allows UMH to benefit from automatic downsampling for most data while preserving critical information that requires every data point to be stored without any filtering or compression.

***

**Tag-processor snippet – picking the right algorithm on the fly**

```yaml
pipeline:
  processors:
    - tag_processor:
        defaults: |
          # generic baseline for every tag
          msg.meta.location_path = "enterprise.plant1"
          msg.meta.data_contract = "_historian"

          # safe, zero-risk compression unless told otherwise
          msg.meta.ds_algorithm       = "deadband"
          msg.meta.ds_threshold  = 0
          msg.meta.ds_max_time   = "30m"
          return msg;

        conditions:
          - if: msg.meta.virtual_path.startsWith("furnace.")
            then: |
              msg.meta.ds_algorithm      = "swinging_door"
              msg.meta.ds_threshold = 0.5
              msg.meta.ds_min_time  = "5s"
              msg.meta.ds_max_time  = "1h"
              return msg;
    - downsampler: {}
```

***

#### Numerical edge cases covered in tests

| Scenario                              | Guard rail in code                                                     |
| ------------------------------------- | ---------------------------------------------------------------------- |
| `threshold < 0`                       | Configuration rejected at startup.                                     |
| `threshold = 0`                       | Legal—drops exact repeats, keeps any change.                           |
| `max_time = 0`                        | Treated as *unset* (no heart-beat, but still at-least-once buffering). |
| `min_time > max_time`                 | Validation error to prevent deadlocks.                                 |
| IEEE-754 extremes (`±Inf`, `NaN`)     | Sample is bypassed with a warning; counting metrics record the event.  |
| Clock skew (non-monotonic timestamps) | Logged and routed through late-arrival policy; compression continues.  |

With these guards the Downsampler behaves deterministically across PLC glitches, network jitter and even deliberate fuzz-test assaults—yet still errs on the side of passing the data through rather than dropping it.

#### How Swinging-Door Trending keeps the essentials—step by step

1. **Segment start (point A, green).** From the very first sample in a segment the algorithm opens two straight "doors": an **upper** line and a **lower** line. *Their slopes mark the fastest rise and the slowest rise allowed while the signal stays inside the ± compDev error band.*&#x20;
2. **Each new sample narrows the door.** Every arriving point recalculates the steepest and flattest slopes that would still stay within ± *compDev*. Geometrically the cone can only shrink, so the blue lines in the pictures pivot closer together frame by frame.

   <img src="https://1639535424-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKZgCXLvkfPPIkQr8UfSb%2Fuploads%2Fgit-blob-4b8595ec1795c0e84273a66237769f1886d793f6%2Fimg2.png?alt=media" alt="Door narrows further" width="600">

   &#x20;

   <img src="https://1639535424-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKZgCXLvkfPPIkQr8UfSb%2Fuploads%2Fgit-blob-fcd99671626c30d22afd8612c9c53830fc9fcd7b%2Fimg3.png?alt=media" alt="Almost closed" width="600">
3. **Two working points—P and S.**
   * **P** (black) is simply the *latest* raw sample.
   * **S** (red, "snapshot") is the *candidate* we might archive. Unlike a dead-band filter we do **not** decide about P immediately; first we test whether S can still represent the trend after P tightens the cone.
4. **Door-slam event = keep S, start over.** When the upper-slope line crosses the lower-slope line the cone has closed (see img 4 / final GIF frame).

   * S is now guaranteed to be the last point that still satisfies the error bound → we **emit** S.
   * P becomes the new anchor A of the next segment, and the door re-opens to its full width.

   <img src="https://1639535424-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKZgCXLvkfPPIkQr8UfSb%2Fuploads%2Fgit-blob-17b4d3df9e04ebcbf583bea359b2ef7f5e8d9d3c%2Fimg4.png?alt=media" alt="Door closed, snapshot flushed" width="600">
5. **Heartbeat and physics limits.** *`max_time`* forces an emit even if the cone never closes (flat signals, watchdog audit). *`min_time`* ignores arrivals that come *so* fast the physical process could not have changed—useful for ovens versus millisecond-level servo data.
6. **The GIF puts it all together.** Blue cone lines swing, the red snapshot jumps forward, only a handful of black-× points survive, yet the reconstructed line never drifts more than ± *compDev* from the raw signal.

   <img src="https://1639535424-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FKZgCXLvkfPPIkQr8UfSb%2Fuploads%2Fgit-blob-2b1f706173172cb76086804dcfad3a49ad00ffd0%2Fanimation1.gif?alt=media" alt="GIF" width="600">

Image Source: [emrumo/swingingdoor](https://github.com/emrumo/swingingdoor)

***

**Practical take-away**

* OT engineers still see smooth ramps without the "stair-step" artefacts a naïve dead-band would create.
* IT teams hold up to **99 % fewer rows** and move far less data, with mathematically bounded error and an audit heartbeat that satisfies 21 CFR §11 / GxP record-liveness rules.




---

[Next Page](https://docs.umh.app/llms-full.txt/1)