Documentation

• 1: Getting Started
• 1.1: 0. Understanding the technologies
• 1.2: 1. Installation
• 1.3: 2. Connecting machines and creating dashboards
• 1.4: 3. Using it in production
• 2: Concepts
• 2.1: Node-RED in Industrial IoT: a growing standard
• 2.2: The UMH datamodel / MQTT
• 2.3: Available states for assets
• 2.4: Digital Shadow - track and trace
• 2.5: Integration into Azure
• 2.6: Open source in Industrial IoT: an open and robust infrastructure instead of reinventing the wheel.
• 2.7: An introduction into certificates and secure communication in IoT for normal people
• 2.8: Why we chose timescaleDB over InfluxDB
• 2.9: The UMH datamodel (preview for v1.0.0)
• 2.9.1: Kafka
• 3: Examples
• 3.1: Flame cutting & blasting
• 3.2: Brewery
• 3.3: Semiconductor
• 3.4: Cycle time monitoring in an assembly cell
• 3.5: Quality monitoring in a bottling line
• 3.6: Pharma packaging
• 3.7: Weaving
• 3.8: CNC Milling
• 4: Tutorials
• 4.1: FAQ
• 4.2: Azure
• 4.2.1: Connecting with Azure IoT Hub
• 4.2.2: Setting up Azure IoT Hub
• 4.3: General
• 4.3.1: Setting up the PKI infrastructure
• 4.3.2: Edge networking
• 4.3.3: How to install an operating system from a USB-stick
• 4.3.4: How to connect with SSH
• 4.3.5: How to flash an operating system onto a USB-stick
• 4.3.6: Versioning in IT
• 4.4: k3OS
• 4.4.1: How to add additional SSH keys in k3OS
• 4.4.2: How to enable SSH password authentification in k3OS
• 4.4.3: How to fix certificate not yet valid issues
• 4.4.4: How to install k3OS
• 4.5: Node-RED
• 4.5.1: Fixing broken Node-RED flows
• 4.6: Tulip
• 4.6.1: How to connect the United Manufacturing Hub to Tulip
• 4.7: United Manufacturing Hub
• 4.7.1: Certified devices
• 4.7.1.1: How to use machineconnect
• 4.7.2: Setting up the documentation
• 4.7.3: Working with the system practically
• 4.7.4: How to setup a development network
• 4.7.5: How to update the stack / helm chart
• 4.7.6: How to work with MinIO
• 4.7.7: Working with Kubernetes, Helm and Lens
• 5: Developers
• 5.1: united-manufacturing-hub
• 5.1.1: grafana-plugins
• 5.1.1.1: factoryinput-panel
• 5.1.1.2: umh-datasource
• 5.1.2: barcodereader
• 5.1.3: cameraconnect
• 5.1.4: environment-variables
• 5.1.5: factoryinput
• 5.1.6: factoryinsight
• 5.1.7: grafana-proxy
• 5.1.8: kafka-bridge
• 5.1.9: kafka-to-blob
• 5.1.10: kafka-to-postgresql
• 5.1.11: mqtt-bridge
• 5.1.12: mqtt-kafka-bridge
• 5.1.13: mqtt-to-blob
• 5.1.14: mqtt-to-postgresql
• 5.1.15: Sensorconnect
• 5.2: How to add new MQTT messages to mqtt-to-postgresql
• 5.3: How to publish a new version
• 6: Publications
• 6.1: Development of a methodology for implementing Predictive Maintenance
• 6.2: Design and Evaluation of a Blue Ocean Strategy for an Open-Core IIoT Platform Provider in the Manufacturing Sector"
• 6.3: Industrial image processing for quality control in production lines: Development of a decision logic for the application case specific selection of hardware and software
• 6.4: Deep learning for industrial quality inspection: development of a plug-and-play image processing system
• 6.5: Development of a decision tool to select appropriate solutions for quality control depending on the defects occurring in the manufacturing process in the automobile branch of the technical-textiles industry
• 6.6: Implementation of Time Series Data based Digital Time Studies for Manual Processes within the Context of a Learning Factory

About The Project

The United Manufacturing System is an open source solution for extracting and analyzing data from manufacturing plants and sensors. The Hub includes both software and hardware components to enable the retrofit of productions plants by plug-and-play as well as to integrate existing machine PLCs and IT systems. The result is an end-to-end solution for various questions in manufacturing such as the optimization of production through OEE analysis, preventive maintenance through condition analysis and quality improvement through stop analysis.

• Open. open-source (see LICENSE) and open and well-documented standard interfaces (MQTT, REST, etc.)
• Scalable. Horizontal scaling incl. fault tolerance through Docker / Kubernetes / Helm. Edge devices can be quickly set up and configured in large numbers.
• Flexible. Flexible deployment options, from public cloud (Azure, AWS, etc.) to on-premise server installations to Raspberry Pis, everything is possible. Free choice of programming language and systems to be connected through central message broker (MQTT).
• Tailor-made for production. Pre-built apps for manufacturing. Use of established automation standards (OPC/UA, Modbus, etc.). Quick connection of production assets either by retrofit or by connecting to existing interfaces.
• Community and support. Enterprise support and community for the whole package. Built exclusively on well-documented software components with a large community.
• Information Security & Data Protection. Implementation of the central protection goals of information security. High confidentiality through e.g. end-to-end encryption, flexible provisioning options and principle of least privilege. High integrity through e.g. ACID databases and MQTT QoS 2 with TLS. High availability through e.g. use of Kubernetes and (for SaaS) a CDN.

1 - Getting Started

1.1 - 0. Understanding the technologies

The materials presented below are usually teached in a 2-3 h workshop session on a live production shopfloor at the Digital Capability Center Aachen. You can find the outline further below.

Introduction into IT / OT

The goal of this chapter is to create a common ground on IT / OT technologies and review best-practices for using IIoT technologies. The target group are people coming from IT, OT and engineering.

Introduction

IIoT sits at the intersection of IT and OT.

History: IT & OT were typically seperate silos but are currently converging to IIoT

Operational Technology (OT)

OT connects own set of various technologies to create highly reliable and stable machines

OT is the hardware and software to manage, monitor and control industrial operations. Its tasks range from monitoring critical assets to controlling robots on the shopfloor. It basically keeps machines and factories running and producing the required product.

Typical responsibilities:

• Monitoring processes to ensure best product quality
• Controlling machine parameters
• Automation of mechanical and controlling processes
• Connecting machines and sensors for communication
• Maintenance of machines and assets
• Certifiying machines for safety and compliance
• Retrofitting assets to increase functionality
• And many more…

Typical vendors for Operational Technology:

• Schneider Electric
• Rockwell Automation
• Honeywell
• Mitsubishi Electric
• National Instruments
• Omron
• Siemens

The concepts of OT are close to eletronics and with a strong focus on human and machine safety

OT focuses on handling processes with highest possible safety for machines and operators

High importance in OT:

Of lesser importance:

Nobody wants to build a nuclear reactor using agile “move fast, break things” principles

Typical device architecture and situation for the OT

Typical device architecture and situation for the OT. 1 Programmable logic controller; 2 Human Machine Interface; 3 Supervisory Control and Data Acquisition

Fundamentals 1: Programmable Logic Controller (PLC)

The Programmable Logic Controller (PLC) is the heart of every modern machine, which stores and runs the program. It is a PC with industrial standards and does not require a monitor, keyboard or other devices to function properly. It collects sensor data and calculates complex algorithms to control actuators.

Background:

• Very old machines use only relays (electric switches) to control actuators and sensors
• PLCs were introduced due to being more reliable and flexible than electrical parts
• The logic of simple switches is still very present in the OT space (programming)

Programming languages:

• The various suppliers like Siemens, Rockwell, Bosch etc. offer different programming languages
• PLCs can be programmed with graphical elements or with code
• Machine vendor programs are not always openly accessible and do not allow changes (loss of warranty)

Communication protocols:

• PLC manufacturers have different communication protocols and functional standards which limits interoperability
• Newer protocols like Profinet or Ethernet/IP are easy to connect to an IT network (if open interface). Others like Profibus require additional hardware and implementation effort

Fundamentals 2: PLCs & PLC programming

Fundamentals 3: Process control using PLCs

Information Technology (IT)

IT connects millions of devices and manages their data flows

IT is the hardware and software to connect thousands of devices in a network and manage their exchange of information. The purpose is to enable data storage and its usage for business and operations. Tasks range from connecting simple telephones to managing complex global networks.

Typical responsibilities:

• Setting up phones, PCs, printers and other office hardware
• Monitoring devices and networks for security breaches
• Maintaining local servers
• Configuration of business systems e.g. ERP/SAP
• Updating devices to ensure IT security
• Setting up local networks and WiFi
• Implementing business solutions like automation and
• And many more…

Typical vendors:

• Microsoft
• Cisco
• Apple
• Dell
• AWS

The concepts of IT are focusing on digital data and networks

What is important in IT? What is not important?

High importance in IT:

Of lesser importance:

Nobody wants to build an app for years just so that the end-user removes it within 30 seconds

Fundamentals 1: Networking

Fundamentals 2: Cloud and Microservices

The term cloud refers to servers and the software running on them. These servers can be used to compute data e.g. process a customer order or simulate the weather and at the same time store it. This data can be accessed around the globe simultaneously with high-speed which enables a centralized “single source of truth”

Cloud products:

• Cloud providers offer their capabilities on advanced analytics and machine learning to reduce time for insights generation (Platform as a Service - PaaS)
• Storage and computational power can be booked flexibly and used freely
• Out of the box applications running in the browser without installation

Microservices:

• Small stand alone blocks running only small functions
• Whenever one microservice block crashes the rest is unaffected (high stability)
• One solutions can be designed out of multiple already available microservices

Scalability:

• Microservice blocks can be flexibly turned on and off depending on the user requirements
• Easy scalability allows customers to only pay what they use
• Single solutions can be deployed and accesses globally without installation on each personal computer

Fundamentals 3: How microservices are built: Docker in 100 seconds

Fundamentals 4: How to orchestrate IT stacks: Kubernetes in 100 seconds

Fundamentals 5: Typical network setups in production facilities

Typical network setups in production facilities

Industrial Internet of Things (IIoT)

Whats it’s all about

Why is digital transformation relevant now?

Technology advancements have lowered barriers to industrial IoT to come down. The benefits of IIoT are real and sizable.

How can manufacturing organizations capture value at scale?

A digital transformation in manufacturing requires an orchestrated approach across the dimensions of business, organization and technology. A holistic framework focuses on full value capture through having a return-on-investment, capability building and technical IIoT ecosystem focus

Value created through digital transformation

Following a digital transformation approach cases show great impact in e.g., throughput, production efficiency, gross margin, quality across various industries.

A full digital transformation of manufacturing needs to consider business, technology and organization

Business: Impact Drive Solutions

• Impact comes from a top-down prioritized portfolio of use cases to address highest value first
• Digital transformations need to have a return-on-investment mindse

Organization: New way of working and dedicated approach to skills and capabilities

• Digital transformations are multiyear, company-wide journeys requiring a central transformation engine and a value capture approach
• Innovative digital-capability-building approaches allow the rapid upskilling of thousands of employee

Technology: Cloud enabled platforms and ecosystem (focus of the United Manufacturing Hub)

• Form a comprehensive, secure, affordable and scalable technology infrastructure based on IoT platform architectures
• Build and lead a focused ecosystem of technology partner

IIoT sits at the intersection of IT and OT

IIoT sits at the intersection of IT and OT.

Architecting for scale

Architecting for scale.

Best-practices 1: Avoid common traps in the IIoT space

See also Open source in Industrial IoT: an open and robust infrastructure instead of reinventing the wheel

Avoid lock-in effects

• Use open and freely accessible standards
• Use open-source whenever reasonable
• When acquiring new software, hardware or machines define contracts on data property, security plans and service levels
• Check for and request full and understandable documentation of each device and interface

Use both of best worlds (IT and OT)

Look into the other world to get alternative solutions and inspiration e.g.

• Grafana dashboard instead of a built in HMI
• Industrial PCs instead of Raspberry PIs
• Secure crucial connections with firewalls (e.g. pfSense)

Avoid quick fixes

• Use industry-wide IT/OT standards wherever applicable before developing on your own
• Invest enough time for architecture basics and the big picture to avoid rework in the near future
• Always document your project even when it seem unnecessary at the moment

Best-practices 2: Protocols which allow communication of IT and OT systems

Coming from IT: MQTT

MQTT

• A light weight with low-bandwidth and power requirements which is the leading standard for IoT applications
• All devices connect to a MQTT Broker which saves and distributes information to devices which subscribed to it (similar to social network)

Coming from IT: REST API

REST API

• Standard web application interface which is used on almost every website
• Can be used to request information from the web e.g. weather data or distance between two cities (Google Maps) or request actions e.g. save a file in the database

Coming from OT: OPC/UA

OPC/UA

• Standard interface for automation and machine connectivity
• Highly complex protocol with wide range of capabilities but low in user friendliness

Best-practices 3: Reduce complexity in machine connection with tools like Node-RED

See also Node-RED in Industrial IoT: a growing standard

Best-practices 4: Connect IT and OT securely using a Demilitarized Zone (DMZ)

See also Why are our networks open by default and how do I protect my valuable industrial assets?

Architecture

See also Architecture

Example projects

See also Examples

1.2 - 1. Installation

There are three options to setup a development environment:

1. using a separate device in combination with k3OS and our installation script (preferred). This requires an external device and is a fully automated installation.
2. using minikube (recommended for developers working on the core functionalities of the stack). This method allows you to install the stack on your device and is semi-automated.
3. manual installation (recommended for production environments, if you want to have fine grained control over the installation steps). This can be executed either on an external device or on your device.

The focus of this article is to provide all necessary information to install it in a compressed tutorial. There are footnotes providing additional information on certain steps, that might be new to certain user groups.

Option 1: using a separate device in combination with k3OS and our installation script

Note: this content is also available in a presence workshop with an experienced facilitator guiding the participants through the installation and answering questions. Contact us for more information!

Prerequisites

This installation method requires some previous settings:

• an edge device with x86 architecture. We recommend using the K300 from OnLogic
• the latest version of k3OS ¹ installed on a bootable USB-stick ².
• a computer with SSH / SFTP client ³ and Lens (for accessing the Kubernetes cluster) installed. We recommend a laptop with an Ethernet port or with an Ethernet adapter.
• local LAN (with DHCP) available via atleast two Ethernet cables and access to the internet.⁴.
• a computer monitor connected with the edge device
• a keyboard connected with the edge device

As shown, the Factorycube is an optional device that combines all the required hardware in a rugged industrial gateway for industrial use.

Installation

This step is also available via a step-by-step video: TODO

k3OS

1. Insert your USB-stick with k3OS into your edge device and boot from it ⁵
2. Install k3OS. When asked for a cloud-init file, enter this URL and confirm: https://www.umh.app/development.yaml. If you are paranoid or want to setup devices for production you could copy the file, modify and host it yourself. Here is the template

This process takes around 15 - 20 minutes depending on your internet connection and there will be no further information about the installation status on the output of the device visible (the information on the computer screen).

Getting access to the device

To verify whether the installation worked and to access Grafana (the dashboard) and Node-RED, we will first enable SSH via password authentification, fetch the login details for Kubernetes and then login via Lens.

Step 1: Login

The login console will look like “messed up” due to the logs of the installation process in the steps above.

Immediatly after start. Nothing is messed up yet.

"Messed up" login screen

You can “clean it up” by pressing two times enter.

You can also immediatly proceed with entering the default username rancher (do not forget to press enter) and the default password rancher to login.

Logging into k3OS using the username `rancher`

Logging into k3OS using the password `rancher`

After a successfull login you should see the current IP address of the device on your computer screen.

Successfully logged into k3OS

Step 2: Enable SSH password authentification

Enable SSH password authentification in k3OS ⁶. This is currently not necessary anymore as the automated setup script will do that automatically, therefore this step can be skipped. This paragraph only exists to remind you that this setting is not the default behavior of k3OS and should be deactivated in production environments.

For production environments we recommend using a certificate to authenticate, which is enabled by default. This can be archieved by modifying the cloud-init file and linking to a public key stored on your GitHub account.

Step 3: Connect via SSH

Connect via SSH ³ from your laptop with the edge device. The IP address is shown on the computer screen on your edge device (see also step 1). If it is not available anymore, you can view the current IP address using ip addr or ifconfig eth0 (works with out devices).

Username: rancher Password: rancher

Step 4: Getting Kubernetes credentials

Execute cat /etc/rancher/k3s/k3s.yaml in your SSH session on your laptop to retrieve the Kubernetes credentials. Copy the content of the result into your clipboard.

Execute `cat /etc/rancher/k3s/k3s.yaml`

Copy the content

Connect with the edge device using the software Lens and the Kubernetes credentials from your clipboard.

Add a new cluster in Lens

Select `Paste as text`

Paste from the clipboard

Ensure that you have adjusted the IP in the Kubernetes credentials with the IP of the edge device.

Also make sure that you simply adjust the IP in between. The port that follows the : should remain untouched (e.g. https://XXX.X.X.X:6443 in that case).

Hint: If you get the message ‘certificate not valid’ or something similar when connecting, verify that you entered the correct port before proceeding to the troubleshooting section (/docs/tutorials/k3os/how-to-fix-invalid-certs-due-to-misconfigured-date/).

You have now access to the Kubernetes cluster!

Verifying the installation and extracting credentials to connect with the dashboard

The installation is finished when all Pods are “Running”. You can do that by clicking on Pods on the left side.

Click in Lens on Workloads and then on Pods

Select the relevant namespaces `factorycube-server` and `factorycube-edge`

everything should be running

Some credentials are automatically generated by the system. One of them are the login credentials of Grafana. You can retrieve them by clicking on “Secrets” on the left side in Lens. Then search for a secret called “grafana-secret” and open it. Press “decode” and copy the password into your clipboard.

Press on the left side in Lens on Configuration and then Secret.

Then select grafana-secret

Then click on the eye on the right side of adminpassword to decrypt it

Opening Grafana and Node-RED

Grafana is now accessible by opening the following URL in your browser: http://<IP>:8080 (e.g., http://192.168.1.2:8080). You can login by using the username admin and password from your clipboard.

Node-RED is accessible by opening the following URL in your browser: http://<IP>:1880/nodered.

Once you have access, you can proceed with the second article Connecting machines and creating dashboards.

Option 2: using minikube

This option is only recommended for developers. Therefore, the installation is targeted for them and might not be as detailed as option 1.

Prerequisites

• minikube installed according to the official documentation
• repository cloned using git clone https://github.com/united-manufacturing-hub/united-manufacturing-hub.git or downloaded and extracted using the download button on GitHub.
• helm ⁷ and kubectl ⁸ installed

Steps

1. Start minikube using minikube start. If minikube fails to start, see the drivers page for help setting up a compatible container or virtual-machine manager.
   

   Output of the command `minikube start`

2. If everything went well, kubectl is now configured to use the minikube cluster by default. kubectl version should look like in the screenshot.
   

   Expected output of `kubectl version`

3. Go into the cloned repository and into the folder deployment/united-manufacturing-hub
4. execute: kubectl create namespace united-manufacturing-hub
   

   Expected output of `kubectl create namespace`

   

   Output of curl

5. Install by executing the following command: helm install united-manufacturing-hub . -n united-manufacturing-hub
   

   Output of `helm install`

Now go grab a coffee and wait 15-20 minutes until all pods are “Running”.

Now you should be able to see the cluster using Lens

Option 3: manual installation

For a manual installation, we recommend that you take a look at the installation script and follow these commands manually and adjust them when needed.

1.3 - 2. Connecting machines and creating dashboards

This guide will explain the typical workflow on how an engineer will connect production lines and then create dashboards based on the extracted data. It is split into two parts:

1. Extracting data from the shop floor by connecting with existing PLCs or additional sensors (there will be examples for both)
2. Creating a customized dashboard in Grafana to visualize the data

Prerequisites

Before continuing with this guide, you should have understood the technologies and have installed the United Manufacturing Hub.

Introduction: extracting data from the shop floor

In general, there are three types of systems on the shop floor that you can connect to:

1. PLCs like for example via OPC/UA, Siemens S7 protocol or Modbus
2. Retrofitted sensors (e.g., because there was no access to the PLC). The United Manufacturing Hub uses IO-Link to connect IO-Link sensors and by using converters digital/analog sensors as well.
3. Additional systems like ERP, MES or quality management systems using TCP/IP, REST or MQTT.

For each protocol and type of system there is a microservice, which handles the connection and extracts all data and pushes them into the central MQTT broker.

From there on, the data is converted into the standardized data model and therefore aggregated and contextualised. The conversion process is usually done in Node-RED, but can be done in any programming language as well.

Sometimes Node-RED is used not only to aggregated and contextualise, but also to extract data from the shop floor. More information about the data flow and the programs behind it, can be found in the architecture. We’ve also written a blog article about the industrial usage of Node-RED, which we can highly recommend.

In the following, we will go through two examples of connecting the shop floor:

1. Retrofitting a cutting machine with the help of external sensors;
2. Using OPC/UA to read a warping machine data directly from the PLC.

Retrofitting with external sensors

In this example we will determine the output and the machine condition of a cutting machine step-by-step.

These sensors are used for this purpose:

• Light barrier
• Button bar
• Inductive sensor

Step 1: Physically connect sensors and extract their data

TODO: Picture or illustration of setup

As a first step you should mount the sensors on the production machine in the regions of your interest. The sensors are then connected to an IO-Link gateway, e.g., the ifm AL1350

The IO-Link gateway is then connected to the network, so that the installation of the United Manufacturing Hub has access to it. As soon as it is physically connected and recieves an IP address it will be automatically detected by sensorconnect and the raw data will be pushed into the central MQTT broker.

The IO-Link gateways, to which sensors are connected, are found by the microservice sensorconnect using a network scan. For this purpose, sensorconnect must be configured with the correcy IP-range, so that it can search for the gateway in the correct network and read out the sensors via it.

The IP-range needs to be entered in the CIDR notation. 192.168.1.0/24 means all IP addresses from 192.168.1.1 to 192.168.1.254. To prevent accidently scanning whole IP-ranges it is set to scan localhost by default.

To do this, there are two options.

• Option 1: Change the IP-range in the development.yaml (k3OS cloud-init) file, that you download. In our example the IP of our Factorycube is 192.168.1.131. Accordingly we change the IP-range to 192.168.1.0/24.

• Option 2: Lens. In Lens you need to open your already set up cluster for your edge device. Then, as you can see in the picture, click on “Apps” in the left bar, then on “Releases and open “factorycube-edge” by clicking on it.

Next, click into the code and press the key combination ctrl+F to search for “iprange”. There you have to change the value of the IP-range as shown. In our example the IP of our Factorycube is 192.168.1.131. Accordingly we change the IP-range to 192.168.1.0/24.

Now the microservice can search for the gateway in the correct network to read the sensors.

As the endresult, we have now all sensor data available in the MQTT broker.

To get a quick and easy overview of the available MQTT messages and topics we recommend the MQTT Explorer. If you don’t want to install any extra software you can use the MQTT-In node to subscribe to all available topics by subscribing to # and then direct the messages of the MQTT in nodes into a debugging node. You can then display the messages in the nodered debugging window and get information about the topic and available data points. More information on using Node-RED will come further down.

Step 2: Processing the data by creating a Node-RED flow

As the next step, we need to process the raw sensor data and add supporting information (e.g., new shifts).

Theoretically, you can do it in any programming language by connecting to the MQTT broker, processing the data and returning the results according to the datamodel. However, we recommend to do that with Node-RED.

With Node-RED, it is possible to quickly develop applications, add new data and convert raw sensor data so that for example it can be shown in a Grafana dashboard. For this purpose, so-called nodes are drawn onto a surface and wired together to create sequences. For more information, take a look at the original Node-RED tutorial

In the following, the creation of such Node-RED flows is demonstrated:

1. Adding new planned operator shifts
2. Recording the number of produced pieces with a light barrier,
3. Changing the machine state using buttons on a button bar
4. Measuring the distance of an object using an inductive sensor.

Example 1: adding new planned operator shifts

A shift is defined as the planned production time and is therefore important for the OEE calculation. If no shifts have been added, it is automatically assumed that the machine is not planned to run. Therefore, all stops will be automatically considered “No shift” or “Operator break” until a shift is added.

A shift can be added by sending a /addShift MQTT message as specified in the datamodel. In Node-RED this results in a flow consisting out of at least three nodes:

1. Inject node
2. Function node (in the picture called “add Shift”)
3. MQTT node

The inject node can be used unmodified.

The function node has the following code, in which the MQTT message payload and topic according to the datamodel are prepared:

msg.payload = {
  "timestamp_ms": 1639522810000,
  "timestamp_ms_end": 1639609200000
}
msg.topic = "ia/factoryinsight/dccaachen/docs/addShift"
return msg;

The MQTT node needs to be configured to connect with the MQTT broker (if nothing has been changed from the default United Manufacturing Hub installation, it will be selected automatically).

Now you can deploy the flow and by clicking on the inject node you will generate the MQTT message. A shift will be added for the customer factoryinsight (default value), the asset location dccaachen and the asset name docs (see also topic structure).

The shift will start at 2021-12-14 at 00:00 and will end at 2021-12-15 00:00 (depending on your timezone). You can get that information by translating the UNIX timestamp in millis (e.g., 1639609200000) to a human readable format using for example https://currentmillis.com/

Example 2: recording the number of produced pieces

Now we will process sensor data instead of sending new data as before.

With the light barrier it is possible, for example, to record the number of pieces produced. Also, with a more complicated logic, machine states can be detected directly with a light barrier. For the sake of simplicity, this is not explored and applied in our example.

The first two nodes in this example will be the same for all other remaining examples as well.

First node: MQTT-in

The topic structure is (see also the datamodel): ia/raw/<transmitterID>/<gatewaySerialNumber>/<portNumber>/<IOLinkSensorID>

An example for an ia/raw/ topic is: ia/raw/development/000200410332/X02/310-372. This means that an IO-Link gateway with serial number 000200410332 has connected the sensor 310-372 to the first port X02.

Now all messages coming in (around 20 messages per second by default for one sensor) will start a Node-RED message flow.

Second node: JSON

The payload of all incoming messages will be in the JSON format, so we need to interpret this by using the JSON node.

Third node: function

The third node “function” formats the incoming data by extracting information from the json (timestamp and relevant data point) and arranging it in a usable way for Node-RED (parsing). The code for this node looks like this:

msg.timestamp=msg.payload.timestamp_ms
msg.payload=msg.payload.Distance;
return msg;

Fourth node: trigger

The trigger allows us, in this example in the case of the light barrier, to sort out distances that are irrelevant for our evaluation, i.e. greater than 15 cm. To do this, you just need to enter a 15 in the “Threshold” field.

Fifth node: trigger

In our example we want to count the number of produced parts. As a trolley on a rail is responsible for the transport into a container after the production of the part and moves towards the light barrier, the number of produced parts shall be increased by one as soon as the distance between the trolley and the light barrier is smaller than 15. To do this, we need a function with the following code:

msg.payload = {
  "timestamp_ms": msg.timestamp,
  "count": 1
}
msg.topic = "ia/factoryinsight/dccaachen/docs/count"
return msg;

Sixth node: MQTT-out

To publish messages to a pre-configured topic, the MQTT-out node is used.

The complete Node-RED flow then looks like this:

Example 3: changing the machine state using buttons on a button bar

Third node: function

Same principle as in example 2, just with changed extracted values.

msg.timestamp=msg.payload.timestamp_ms
msg.payload=msg.payload.value_string;
return msg;

Fourth node: filter

The filter blocks the values of a sensor until the value is changed. A change of the property of the node is not necessary.

Fifth node: switch

The switch node is used to distinguish between the different inputs for the following flow and only lets through the values that come through the previously defined input.

With the button bar, these are the individual buttons. You can see which name is assigned to a single button by the number that appears in the debug window when you press a button. For our example these are the numbers “0108”, “0104”, 0120” and “0101”.

Sixth node: function

The switch node is followed by a separate function for each button. In our example different states are transmitted. States can be e.g. active, unknow pause, material change, process etc. and are defined via numbers. The different states can be found here.

For example, the code for the function looks like this:

msg.payload = {
  "timestamp_ms": msg.timestamp,
  "state": 10000
}

msg.topic = "ia/factoryinsight/dccaachen/docs/state"
return msg;

To reach further machine states, only the adaptation of the state number is necessary.

Seventh node: MQTT-out

To publish messages to a pre-configured topic, the MQTT-out node is used.

The complete Node-RED flow then looks like this:

Example 4: Measuring the distance of an object using an inductive sensor

Third node: function

In the third node “function” a timestamp is generated for each capacitive value. The timestamp is stored in the form of a string. This makes it possible, for example, to read out the time at which a item was produced.

The code for this node looks like this:

msg.timestamp=msg.payload.timestamp_ms
msg.payload=msg.payload["Process value"]
return msg;

Fourth node: function

In our example, the following function formats the incoming data from MQTT-IN by extracting information from the json (timestamp and relevant data point) and arranging it in a usable way for Node-RED (parsing). To do this, we need a function with the following code:

msg.payload = {
    "timestamp_ms": msg.timestamp, 
    "process_value": msg.payload
}
msg.topic = "ia/factoryinsight/dccaachen/docs/processValue"
return msg;

The Process value name process_value can be chosen freely.

Fifth node: MQTT-out

To publish messages to a pre-configured topic, the MQTT-out node is used.

The complete Node-RED flow then looks like this:

Extraction of data points via OPC/UA

This chapter will describe in the future how to connect already existing sensors or integrate an already existing interface (OPC/UA) with our system.

Hint: You can use the OPC UA node in Node-RED

Create dashboards with Grafana

So the first step is to open Grafana by opening the following URL in your browser: http://<IP>:8080 (e.g. http://192.168.1.2:8080). You can log in with the username admin and the password acquired during the installation tutorial.

Next, create a new dashboard by clicking on the + on the left side.

After clicking on Add an empty panel you should see the following settings:

Now you can add different panels to your dashboard.

Example panel 1: number of produced pieces acquired by the light barrier

For the light barrier the stat panel is used.

First of all, in “location” and “asset” labels are selected from the topic of the second function in the Node-RED flow. The “Value” needs to be count.

To get the total amount of produced parts a deeper insight into the panel settings is required. Go to “Value-Options” on the right side and change the entries of “Calculation” and “Fields” as shown below.

Example panel 2: machine state acquired by the button bar

For the button bar the Discrete panel is used.

The query parameters from the topic of the second function in the Node-RED flow must be selected in “location” and “asset”. The “value” must be state.

Example panel 3: process values

For the capacitive sensor the Graph (old) panel is used.

The query parameters from the topic of the second function in the Node-RED flow must be selected in “location” and “asset”. The “value” must be process_process_value as specified in the payload message.

Now the dashboard should look like this. In the upper right corner you can set the time span in which the data should be displayed and how often the dashboard should be refreshed.

Final notes

The complete Node-RED flow can be downloaded here

Now go grab a coffee and relax.

1.4 - 3. Using it in production

This article is split up into two parts:

The first part will focus on factorycube-edge and the Industrial Automation world. The second part will focus on factorycube-server and the IT world.

Using it on the edge

The world of Industrial Automation is heavily regulated as very often not only expensive machines are controlled, but also machines that can potentially injure a human being. Here are some information that will help you in setting it up in production (not legal advice!).

If you are unsure about how to setup something like this, you can contact us for help with implementation and/or certified devices, which will ease the setup process!

Hardware & Installation, Reliability

One key component in Industrial Automation is reliability. Hardware needs to be carefully selected according to your needs and standards in your country.

When changing things at the machine, you need to ensure that you are not voiding the warranty or to void the CE certification of the machine. Even just installing something in the electrical rack and/or connecting with the PLC can do that! And it is not just unnecessary regulations, it is actually important:

PLCs can be pretty old and usually do not have much capacity for IT applications. Therefore, it is essential when extracting data to not overload the PLCs capabilities by requesting too much data. We strongly recommend to test the performance and closely watch the CPU and RAM usage of the PLC.

This is the reason we install sometimes additional sensors instead of plugging into the existing ones. And sometimes this is enough to get the relevant KPIs out of the machine, e.g., the Overall Equipment Effectiveness (OEE).

Network setup

To ensure the safety of your network and PLC we recommend a network setup like following:

Network setup having the machines network, the internal network and PLC network seperated from each other

The reason we recommend this setup is to ensure security and reliability of the PLC and to follow industry best-practices, e.g. the “Leitfaden Industrie 4.0 Security” from the VDMA (Verband Deutscher Maschinenbauer) or Rockwell.

Additionally, we are taking more advanced steps than actually recommended (e.g., preventing almost all network traffic to the PLC) as we have seen very often, that machine PLC are usually not setup according to best-practices and manuals of the PLC manufacturer by system integrators or even the machine manufacturer due to a lack of knowledge. Default passwords not changed or ports not closed, which results in unnecessary attack surfaces.

Also updates are almost never installed on a machine PLC resulting in well-known security holes to be in the machines for years.

Another argument is a pretty practical one: In Industry 4.0 we see more and more devices being installed at the shopfloor and requiring access to the machine data. Our stack will not be the only one accessing and processing data from the production machine. There might be entirely different solutions out there, who need real-time access to the PLC data. Unfortunately, a lot of these devices are proprietary and sometimes even with hidden remote access features (very common in Industrial IoT startups unfortunately…). We created the additional DMZ around each machine to prevent one solution / hostile device at one machine being able to access the entire machine park. There is only one component (usually node-red) communicating with the PLC and sending the data to MQTT. If there is one hostile device somewhere it will have very limited access by default except specified otherwise, as it can get all required data directly from the MQTT data stream.

Our certified device “machineconnect” will have that network setup by default. Our certified device “factorycube” has a little bit different network setup, which you can take a look at here.

Other useful commands

Quick setup on k3OS:

1. export VERIFY_CHECKSUM=false && curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 && chmod 700 get_helm.sh && ./get_helm.sh
2. curl -L https://github.com/united-manufacturing-hub/united-manufacturing-hub/tarball/v0.4.2 | tar zx && mv $(find . -maxdepth 1 -type d -name "united-manufacturing-hub*") united-manufacturing-hub
3. helm install factorycube-edge /home/rancher/united-manufacturing-hub/deployment/factorycube-edge --values "/home/rancher/CUSTOM.yaml" --kubeconfig /etc/rancher/k3s/k3s.yaml

factorycube-server

In general the factorycube-server installation is tailored strongly to the environments it is running in. Therefore, we can only provide general guidance on setting it up.

WARNING: THIS SECTION IS STILL IN WORK, PLEASE ONLY USE AS A ROUGH START. WE STRONGLY RECOMMEND CONTACTING US IF YOU ARE PLANNING ON USING IT IN PRODUCTION ENVIRONMENT AND WITH EXPOSURE TO THE INTERNET

Example deployment on AWS EKS

To give you an better idea, this section explains an example production deployment on AWS EKS.

Preparation

General

• Use Cloudflare as DNS and firewall. This will provide an additional security layer on top of all your HTTP/HTTPS applications, e.g., factoryinsight or Grafana

AWS

• Setup a AWS EKS cluster using eksctl
• Setup a S3 bucket and a IAM user
• Add IAM policy to the user (assuming the bucket is called umhtimescaledbbackup)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::umhtimescaledbbackup/*",
                "arn:aws:s3:::umhtimescaledbbackup"
            ]
        }
    ]
}

If you do not add the IAM policy you might get a ACCESS DENIED in pgbackrest pod.

Kubernetes

• Create a namespace called dev2
• Use later the release name dev2. If you are using a different release name, you might need to adjust dev2 in the following aws_eks.yaml file
• Setup nginx-ingress-controller (e.g., using the bitnami helm chart)
• Setup external-dns
• Setup cert-manager and create a certificate issuer called letsencrypt-prod (see also link)
• to enable backup using S3 buckets create a secret called dev2-pgbackrest and enter the following content:

kind: Secret
apiVersion: v1
metadata:
  name: dev2-pgbackrest
  namespace: dev2
data:
  PGBACKREST_REPO1_S3_BUCKET: <redacted>
  PGBACKREST_REPO1_S3_ENDPOINT: <redacted>
  PGBACKREST_REPO1_S3_KEY: <redacted>
  PGBACKREST_REPO1_S3_KEY_SECRET: <redacted>
  PGBACKREST_REPO1_S3_REGION: <redacted>
type: Opaque

aws_eks.yaml

We recommend the following values to get your journey to production started (using release name dev2 and namespace dev2):

### factoryinsight ###
factoryinsight:
  enabled: true
  image: unitedmanufacturinghub/factoryinsight
  replicas: 2
  redis:
    URI1: dev2-redis-node-0.dev2-redis-headless:26379
    URI2: dev2-redis-node-1.dev2-redis-headless:26379
    URI3: dev2-redis-node-2.dev2-redis-headless:26379
  db_host: "dev2-replica"
  db_port: "5433"
  db_password: "ADD_STRONG_PASSWORD_HERE"
  ingress:
    enabled: true
    publicHost: "api.dev2.umh.app"
    publicHostSecretName: "factoryinsight-tls-secret"
    annotations:
      external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
      cert-manager.io/cluster-issuer: "letsencrypt-prod" 
  resources:
    limits:
       cpu: 1000m         
    requests:
       cpu: 200m      


### mqtt-to-postresql ###
mqtttopostgresql:
  enabled: true
  image: unitedmanufacturinghub/mqtt-to-postgresql
  replicas: 2
  storageRequest: 1Gi

### timescaleDB ###
timescaledb-single:
  enabled: true
  replicaCount: 2
  
  image:
    # Image was built from
    # https://github.com/timescale/timescaledb-docker-ha
    repository: timescaledev/timescaledb-ha
    tag: pg12-ts2.0-latest
    pullPolicy: IfNotPresent 
  
  backup:
    enabled: true
  
  persistentVolumes:
    data:
      size: 20Gi 
    wal:
      enabled: true
      size: 5Gi
  
### grafana ###
grafana:
  enabled: true

  replicas: 2

  image:
    repository: grafana/grafana
    tag: 7.5.9
    sha: ""
    pullPolicy: IfNotPresent

  service:
    type: ClusterIP 

  ingress:
    enabled: true
    annotations: 
      kubernetes.io/ingress.class: nginx
      kubernetes.io/tls-acme: "true"
    labels: {}
    path: /

    # pathType is only for k8s > 1.19
    pathType: Prefix

    hosts:
      - dev2.umh.app 

    tls: []

  ## Pass the plugins you want installed as a list.
  ##
  plugins: 
      - grafana-worldmap-panel
      - grafana-piechart-panel
      - aceiot-svg-panel
      - grafana-worldmap-panel
      - natel-discrete-panel
      - isaozler-paretochart-panel
      - williamvenner-timepickerbuttons-panel
      - agenty-flowcharting-panel
      - marcusolsson-dynamictext-panel
      - factry-untimely-panel
      - cloudspout-button-panel 


  ## Grafana's primary configuration
  ## NOTE: values in map will be converted to ini format
  ## ref: http://docs.grafana.org/installation/configuration/
  ##
  grafana.ini:
    paths:
      data: /var/lib/grafana/data
      logs: /var/log/grafana
      plugins: /var/lib/grafana/plugins
      provisioning: /etc/grafana/provisioning
    analytics:
      check_for_updates: true
    log:
      mode: console
    grafana_net:
      url: https://grafana.net
    database:
      host: dev2 
      user: "grafana"
      name: "grafana"
      password: "ADD_ANOTHER_STRONG_PASSWORD_HERE"
      ssl_mode: require
      type: postgres

  ## Add a seperate remote image renderer deployment/service
  imageRenderer:
    # Enable the image-renderer deployment & service
    enabled: true
    replicas: 1

####################### nodered #######################
nodered:
  enabled: true 
  tag: 1.2.9
  port: 1880
  storageRequest: 1Gi
  timezone: Berlin/Europe
  serviceType: ClusterIP
  ingress:
    enabled: true
    publicHost: "nodered.dev2.umh.app"
    publicHostSecretName: "nodered-tls-secret"
    annotations:
      external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
  settings: |-  
    module.exports = {
        // the tcp port that the Node-RED web server is listening on
        uiPort: process.env.PORT || 1880,
        // By default, the Node-RED UI accepts connections on all IPv4 interfaces.
        // To listen on all IPv6 addresses, set uiHost to "::",
        // The following property can be used to listen on a specific interface. For
        // example, the following would only allow connections from the local machine.
        //uiHost: "127.0.0.1",
        // Retry time in milliseconds for MQTT connections
        mqttReconnectTime: 15000,
        // Retry time in milliseconds for Serial port connections
        serialReconnectTime: 15000,
        // The following property can be used in place of 'httpAdminRoot' and 'httpNodeRoot',
        // to apply the same root to both parts.
        httpRoot: '/nodered',
        // If you installed the optional node-red-dashboard you can set it's path
        // relative to httpRoot
        ui: { path: "ui" },
        // Securing Node-RED
        // -----------------
        // To password protect the Node-RED editor and admin API, the following
        // property can be used. See http://nodered.org/docs/security.html for details.
        adminAuth: {
            type: "credentials",
            users: [
                {
                    username: "admin",
                    password: "ADD_NODERED_PASSWORD",
                    permissions: "*"
                }
            ]
        },
        
        functionGlobalContext: {
            // os:require('os'),
            // jfive:require("johnny-five"),
            // j5board:require("johnny-five").Board({repl:false})
        },
        // `global.keys()` returns a list of all properties set in global context.
        // This allows them to be displayed in the Context Sidebar within the editor.
        // In some circumstances it is not desirable to expose them to the editor. The
        // following property can be used to hide any property set in `functionGlobalContext`
        // from being list by `global.keys()`.
        // By default, the property is set to false to avoid accidental exposure of
        // their values. Setting this to true will cause the keys to be listed.
        exportGlobalContextKeys: false,
        // Configure the logging output
        logging: {
            // Only console logging is currently supported
            console: {
                // Level of logging to be recorded. Options are:
                // fatal - only those errors which make the application unusable should be recorded
                // error - record errors which are deemed fatal for a particular request + fatal errors
                // warn - record problems which are non fatal + errors + fatal errors
                // info - record information about the general running of the application + warn + error + fatal errors
                // debug - record information which is more verbose than info + info + warn + error + fatal errors
                // trace - record very detailed logging + debug + info + warn + error + fatal errors
                // off - turn off all logging (doesn't affect metrics or audit)
                level: "info",
                // Whether or not to include metric events in the log output
                metrics: false,
                // Whether or not to include audit events in the log output
                audit: false
            }
        },
        // Customising the editor
        editorTheme: {
            projects: {
                // To enable the Projects feature, set this value to true
                enabled: false
            }
        }
    }


##### CONFIG FOR REDIS #####
redis:
  enabled: true
  cluster:
    enabled: true
    slaveCount: 2
  image:
    pullPolicy: IfNotPresent
    registry: docker.io
    repository: bitnami/redis
    tag: 6.0.9-debian-10-r13
  master:
    extraFlags:
    - --maxmemory 4gb
    persistence:
      size: 8Gi
    resources:
      limits:
        memory: 4Gi
      requests:
        cpu: 100m
        memory: 1Gi
  podDisruptionBudget:
    enabled: true
    minAvailable: 2
  slave:
    persistence:
      size: 8Gi
    resources:
      limits:
        memory: 4Gi
      requests:
        cpu: 100m
        memory: 1Gi

##### CONFIG FOR VERNEMQ #####

vernemq:
  enabled: true
  AclConfig: |-
     pattern write ia/raw/%u/#
     pattern write ia/%u/#
     pattern $SYS/broker/connection/%c/state

     user TESTING
     topic ia/#
     topic $SYS/#
     topic read $share/TESTING/ia/#

     user ia_nodered
     topic ia/#     
  CACert: |-
        ADD CERT
  Cert: |-
        ADD CERT
  Privkey: |-
        ADD CERT
  image:
    pullPolicy: IfNotPresent 
    repository: vernemq/vernemq
    tag: 1.11.0
  replicaCount: 2 
  service:
    annotations:
      prometheus.io/port: "8888"
      prometheus.io/scrape: "true"
      external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
      external-dns.alpha.kubernetes.io/hostname: mqtt.dev2.umh.app
    mqtts:
      enabled: true
      nodePort: 8883
      port: 8883
    mqtt:
      enabled: false
    type: LoadBalancer

Further adjustments

VerneMQ / MQTT

• We recommend setting up a PKI infrastructure for MQTT (see also prerequisites) and adding the certs to vernemq.CAcert and following in the helm chart (by default there are highly insecure certificates there)
• You can adjust the ACL (access control list) by changing vernemq.AclConfig
• If you are using the VerneMQ binaries in production you need to accept the verneMQ EULA (which disallows using it in production without contacting them)

Redis

• The password is generated once during setup and stored in the secret redis-secret

Nodered

• We recommend disabling external access to nodered entirely and spawning a seperate nodered instance for every project (to avoid having one node crashing all flows)
• You can change the configuration in nodered.settings
• We recommend that you set a password for accessing the webinterface in the nodered.settings. See also the official tutorial from nodered

MinIO

We strongly recommend to change all passwords and salts specified in values.yaml

2 - Concepts

High-level architecture

The United Manufacturing Hub consists out of three layers and two packages for installation:

The following sub-chapters will explain the layers and the two packages further. If you want to deep-dive into the actual architecture, you can scroll further down

Lower level: data acquisition

The data sources connected to the edge device provide the foundation for automatic data collection. The data sources can be external sensors (e.g. light barriers, vibration sensors), input devices (e.g. button bars), Auto-ID technologies (e.g. barcode scanners), industrial cameras and other data sources such as machine PLCs. The wide range of data sources allows the connection of all machines, either directly via the machine PLC or via simple and fast retrofitting with external sensors.

Examples:

• sensorconnect (to automatically read out IO-Link Master and their connected sensors)
• cameraconnect (to automatically read out GenICam compatible cameras and push the result into MQTT, in development)
• barcodereader (to connect USB barcodereader and push data into MQTT)
• Node-RED (e.g., for propertiary and / or machine specific protocols)
• PLC4X (in development)

Middle layer: data infrastructure

This layer is the central part of the United Manufacturing Hub. It provides an infrastructure including data models to fulfull all manufacturing needs for data processing and storage.

It starts by making all acquired data accessible in real-time for data processing using either established solutions like Node-RED or your own written software using a microservice approach and MQTT. Therefore, adding new data, processing it or integrating it with other systems on-the-edge is very easy. We recommend to start transforming data into the central data model at this step.

To send the raw and / or processed data to a central place (cloud or on-premise) we use our self-written MQTT bridge. Internet connections or network in general is often unstable in manufacturing environments and therefore one needs to safely buffer messages across internet or electricity downtimes. As existing MQTT bridge solutions were unreliable we developed our own.

Once the data arrives at the server it can be further processed using the same methods as on-the-edge (MQTT microservice, Node-RED, etc.). The real-time data can also integrated into MES or ERP systems.

All processed data is then stored into databases using load-balanced microservices with caching. Therefore, one can archieve high-availability and enourmous scalability through the load-balanced microservices. Furthermore, common requests and operations are cached in redis

Relational data (e.g., data about orders and products) as well as time series data in high resolution (e.g., machine data like temperature) can be stored in the TimescaleDB database (difference between InfluxDB and timescaleDB have been described in detail). Blob data (e.g., camera pictures) will be stored in a blob storage either directly in Minio or using a Minio gateway in a cloud specific storage like AWS S3 or Microsoft Azure Blob Storage.

We do not allow direct access to the databases for performance and security reasons. Instead, we’ve put an additional, self-written, component in front called factoryinsight. factoryinsight provides a REST API to access raw data from the databases as well as processed data in form of KPI’s like ‘OEE losses’ or similar. All requests are load-balanced, cached and executed only on a replica of the database.

To insert data via a REST API we’ve developed two additional services grafana-proxy and factoryinput.

Examples:

• TimescaleDB
• Node-RED
• factoryinput
• factoryinsight
• minio

Higher level: visualization

As a standard dashboarding tool the United Manufacturing Hub uses Grafana in combination with self-written plugins, which allow every user (even without programming knowledge) to quickly and easilyy compose personally tailored dashboards with the help of modular building blocks.

Examples:

• Grafana
• umh-datasource and umh-factoryinput-panel

The right side: deployment options

The entire stack can be deployed using only a configuration file (values.yaml) and the corresponding Helm charts factorycube-server and factorycube-edge.

This allows to deploy the architecture in hybrid setups, from deploying it on-the-edge IIoT gateways to on-premise servers to the cloud (e.g., Azure AKS)

Low-level architecture

If you want to go more into detail, here is the detailled architecture:

2.1 - Node-RED in Industrial IoT: a growing standard

Using Node-RED and UaExpert to extract data from the PLC of a Saw

Most people know Node-RED from the areas of smart home or programming introductions (those workshops where you connect things with microcontrollers). Yet, very few people realize that it is frequently used in manufacturing as well.

For those of you that do not know it yet, here is the official self-description from the Node-RED website:

Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways.

It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.

And the best thing: it is open-source

The project started in early 2013 in IBM’s research centers. In 2016 it was one of the founding projects of the JS Foundation. Since the version release 1.0 in 2019 it is considered safe for production use.

A self-conducted survey in the same year showed that from 515 respondents, 31.5% use Node-RED in manufacturing, and from 868 respondents, 24% said they have created a PLC application using it ¹. Also, 24.2 % of 871 respondents said that they use InfluxDB in combination with Node-RED. The reason we think that TimescaleDB is better suited for the Industrial IoT than InfluxDB has been described in this article.

But how widespread is it really in manufacturing? What are these users doing with Node-RED? Let’s deep dive into that!

Usage of Node-RED in Industry

Gathering qualitative data of industry usage of specific solutions can be hard to almost impossible as very few companies are open about the technologies they use. However, we can still gather quantitative data, which strongly indicates a heavy usage in various industries in data extraction and processing.

First, it is preinstalled on more and more various automation systems like PLCs. Wikpedia has a really good overview here (and it checks out!). Siemens, in particular, is starting to use it more often, see also Node-RED with SIMATICIOT2000 or the Visual Flow Creator

Furthermore, various so-called “nodes” are available that can only be used in manufacturing environments, e.g., to read out data from specific devices. These nodes also have quite impressive download numbers.

Some examples:

• node-red-contrib-opcua had 874 downloads in the last week (2021-07-02)
• node-red-contrib-s7 had 902 downloads in the last week (2021-07-02)
• node-red-contrib-modbus had 4278 downloads in the last week (2021-07-02). However, it should be noted that this node can also be used in other areas

We’ve talked with the former developer of two of these nodes, Klaus Landsdorf from the German company Iniationware, which offers companies support in the topics of OPC-UA, Modbus, BACnet and data modeling.

Klaus confirmed our hypothesis:

We get many requests from German hardware manufacturers that rely on Node-RED and on these industry-specific nodes like OPC-UA. The OPC-UA project was sponsored by just two small companies with round about 5% of the costs for development in the case of the IIoT OPC-UA contribution package. But in view of using the package and testing it across multiple industrial manufacturing environments to ensure a high stability, we had many and also big companies aboard. In education we have a great response from ILS, because they are using the Iniationware package node-red-contrib-iiot-opcua to teach their students about OPC-UA essentials. Unfortunately, just a few companies understand the idea of a commercial backing for open-source software companies by yearly subscriptions, which could safe a lot of money for each of them. Do it once, stable and share the payment in open-source projects! That would bring a stable community and contribution packages for the specific reley on industrial needs like LTS versions. Simplified: it needs a bit money to make money in a long term as well as to provide stable and up to date Node-RED packages.

It is also described in the community as being production-ready and used quite frequently. In a topic discussing the question of production readiness a user with the name of SonoraTechnical says:

Although anecdotal, just Friday, I was speaking to an engineer at a major OPC Software Vendor who commented that they see Node-RED frequently deployed by industrial clients and even use it internally for proving out concepts and technology.

Another one with the name of gemini86 explains the advantages compared with commercial solutions:

I’m also (very much) late to the party on this, but I work in manufacturing and use AB, Siemens, Codesys, etc. I also use Node-RED for SCADA and database bridging. Our site has well pumps in remote areas where data and commands are sent over 900mhz ethernet radios, and Node-RED handles the MQTT <> modbusRTU processing. Node-RED has been as stable and quick, if not quicker than any Siemens or AB install with comparable network functionality. In fact, I struggled to get my S7-1200 to properly communicate with modbusRTU devices at all. I was completely baffled by their lack of documentation on getting it to work. Their answer? “Use profibus/profinet.” So, I myself prefer Node-RED for anything to do with serial or network communications.

Last but not least, it is very frequently used in scientific environments. There are over 3.000 research papers available on Google Scholar on the usage of Node-RED in industrial environments!

Therefore, it is safe to say that it is widespread, with growing numbers of users in industry. But what exactly can you do with it? Let us give some examples of how we are using it!

What you can do with it

The United Manufacturing Hub relies on Node-RED as a tool to

1. Extract data from production machines using various protocols (OPC/UA, Modbus, S7, HTTP, TCP, …)
2. Processing and unifying data points into our standardized data model
3. Customer-specific integrations into existing systems, e.g., MES or ERP systems like SAP or Oracle
4. Combining data from various machines and triggering actions (machine to machine communication or, in short, M2M)
5. Creating small interactive and customer-specific dashboards to trigger actions like specifying stop reasons

Let’s explain each one by going through them step-by-step:

1. Extract data from production machines using various protocols

One central challenge of Industrial IoT is obtaining data. The shopfloor is usually fitted out with machines from various vendors and of different ages. As there is almost little or no standardization in the protocols or semantics, the data extraction process needs to be customized for each machine.

With Node-RED, various protocols are available as so-called “nodes” - from automation protocols like OPC/UA (see earlier) to various IT protocols like TCP or HTTP. For any other automation protocol, you can use PTC Kepware, which supports over 140 various PLC protocols.

2. Processing and unifying data points into our standardized data model

Node-RED was originally developed for

visualizing and manipulating mappings between MQTT topics ² and this is what we are still using it for today. All these data points that have been extracted from various production machines now need to be standardized to match our data model. The machine state needs to be calculated, the machines’ output converted from various formats into a simple /count message, etc.

More information about this can be found in our data model for Industrial IoT.

Example of working with the United Manufacturing Hub. Everything is flow-based.

3. Customer-specific integrations into existing systems

It is not just good for extracting and processing data. It is also very good for pushing this processed data back into other systems, e.g., MES or ERP systems like Oracle or SAP. These systems usually have REST APIs, e.g., here is an example for the REST API for the Oracle ERP.

As the customer implementations of those systems are usually different, the resulting APIs are mostly also different. Therefore, one needs a system that is quick to use to handle those APIs. And Node-RED is perfect for this.

4. Machine to machine communication

The AGV automatically gets the finished products from one machine and brings them to empty stations, which is a good example for M2M

As a result of our data architecture machine to machine communication (M2M) is enabled by default. The data from all edge devices is automatically sent to a central MQTT broker and is available to all connected devices (that have been allowed access to that data).

It is easy to gather data from various machines and trigger additional actions, e.g., to trigger the Automated Guided Vehicle (AGV) to fetch material from the production machine when one station is empty of material.

And the perfect tool to set those small triggers is, as you might have guessed, Node-RED.

5. Creating small interactive and customer-specific dashboards

Example of a dashboard using node-red-dashboard. It features a multi-level stop reason selection and the visualization of production speed.

Sometimes the machine operators need time-sensitive dashboards to retrieve real-time information or to interact with the system. As many companies still do not have a good and reliable internet connection or even network infrastructure, one cannot wait until the website is fully loaded to enter a stop reason. Therefore, sometimes it is crucial to have a dashboard as close to the machine as possible (and not sitting somewhere in the cloud).

For this one, you can use the node-red-dashboard node, which allows you to easily create dashboards and interact with the data via MQTT.

Bonus: What not to do: process control

However, we strongly recommend NOT using it to intervene in the production process, e.g., process control or ensuring safety mechanisms for two reasons:

1. IT tools and systems like Node-RED are not designed to ensure the safety of machines or people, e.g., guaranteed time-sensitive reactions to a triggered safety alert
2. It would also be almost impossible to get that certified and approved due to 1. For these aspects, very good and safe tools, like PLCs or NCs, are already out there in the automation world.

Summary

The slogan: “The best things in life are free” also applies in manufacturing:

Node-RED is on the same level as “professional” closed-source and commercial solutions and is used by thousands of researchers and hundreds of daily users in various manufacturing industries.

It is included and enabled in every installation of the United Manufacturing Hub - in the cloud and on the edge.

More information on how we use the system can be found in our Quick Start.

2.2 - The UMH datamodel / MQTT

Introduction

All events or subsequent changes in production are transmitted via MQTT in the following data model. This ensures that all participants are always informed about the latest status.

The data model in the MQTT Broker can be divided into four levels. In general, the higher the level, the lower the data frequency and the more the data is prepared.

If you do not know the idea of MQTT (important keywords: “broker”, “subscribe”, “publish”, “topic”), we recommend reading the wikipedia article first.

All MQTT messages consist out of one JSON with at least two elements in it:

{
    "timestamp_ms": 1588879689394,
    "valueName": "str"
}

{
    "timestamp_ms": 1588879689394,
    "valueName": 24
}

{
    "timestamp_ms": 1588879689394,
    "valueName": {
        "key1": "some text",
        "key2": 12
    }
}

1st level: Raw data

Data from this level are all raw data, which are not yet contextualized(i.e., assigned to a machine). These are, in particular, all data from sensorconnect and cameraconnect.

Topic: ia/raw/

Topic structure: `ia/raw/.+'

All raw data coming in via sensorconnect.

{
  "timestamp_ms": ..., // int
  "type": "IO-Link", // in case of IO-Link
  "connected": 1, // 1 if connected
  "<value1Name>": <convertedValue1>, //for example: "distance": 16,
  "<value2Name>": <convertedValue2>,
  ...
  "<valueNName>": <convertedValueN>
}

{
  "timestamp_ms": ...,
  "type": "IO-Link", //in case of IO-Link
  "connected": 1, // 1 in case of connected
  "value_string": <unconvertedValue>
}

{
  "timestamp_ms": ...,
  "type": "IO-Link", //in case of IO-Link
  "connected": 0, // 0 in case of not connected, also no value included in JSON
}

Topic: ia/rawImage/

Topic structure: ia/rawImage/<TransmitterID>/<MAC Adress of Camera>

All raw data coming in via cameraconnect.

{
	"timestamp_ms": 214423040823,
	"image":  {
		"image_id": "<MACaddress>_<timestamp_ms>",
		"image_bytes": "3495ask484...",
		"image_height": 800,
		"image_width": 1203,
		"image_channels": 3
	}
}

im_bytes = base64.b64decode(incoming_mqtt_message["image"]["image_bytes"])
im_arr = np.frombuffer(im_bytes, dtype=np.uint8)  # im_arr is a one-dimensional Numpy array
img = cv2.imdecode(im_arr, flags=cv2.IMREAD_COLOR)
cv2.imwrite(image_path, img)

2nd level: Contextualized data

In this level the data is already assigned to a machine.

Topic structure: ia/<customerID>/<location>/<AssetID>/<Measurement> e.g. ia/dccaachen/aachen/demonstrator/count.

An asset can be a step, machine, plant or line. It uniquely identifies the smallest location necessary for modeling the process.

By definition all topic names should be lower case only!

/count

Topic: ia/<customerID>/<location>/<AssetID>/count

Here a message is sent every time something has been counted. This can be, for example, a good product or scrap.

count in the JSON is an integer. scrap in the JSON is an integer, which is optional. It means scrap pieces of count are scrap. If not specified it is 0 (all produced goods are good).

{
    "timestamp_ms": 1588879689394, 
    "count": 1
}

/scrapCount

Topic: ia/<customerID>/<location>/<AssetID>/scrapCount

Here a message is sent every time products should be marked as scrap. It works as follows: A message with scrap and timestamp_ms is sent. It starts with the count that is directly before timestamp_ms. It is now iterated step by step back in time and step by step the existing counts are set to scrap until a total of scrap products have been scraped.

scrap in the JSON is an integer.

{
    "timestamp_ms": 1588879689394, 
    "scrap": 1
}

/barcode

Topic: ia/<customerID>/<location>/<AssetID>/barcode

{
    "timestamp_ms": 1588879689394, 
    "barcode": "16699"
}

/activity

Topic: ia/<customerID>/<location>/<AssetID>/activity

{
    "timestamp_ms": 1588879689394, 
    "activity": True
}

/detectedAnomaly

Topic: ia/<customerID>/<location>/<AssetID>/detectedAnomaly

{
    "timestamp_ms": 1588879689394, 
    "detectedAnomaly": "maintenance"
}

/addShift

Topic: ia/<customerID>/<location>/<AssetID>/addShift

{
    "timestamp_ms": 1588879689394, 
    "timestamp_ms_end": 1588879689395
}

/addOrder

Topic: ia/<customerID>/<location>/<AssetID>/addOrder

A message is sent here each time a new order is started.

{
    "product_id": "Beierlinger 30x15",
    "order_id": "HA16/4889",
    "target_units": 1
}

/addProduct

Topic: ia/<customerID>/<location>/<AssetID>/addProduct

A message is sent here each time a new product is added.

{
    "product_id": "Beierlinger 30x15",
    "time_per_unit_in_seconds": 0.2
}

/startOrder

Topic: ia/<customerID>/<location>/<AssetID>/startOrder

A message is sent here each time a new order is started.

{
    "timestamp_ms": 1588879689394,
    "order_id": "HA16/4889",
}

/endOrder

Topic: ia/<customerID>/<location>/<AssetID>/endOrder

A message is sent here each time a new order is started.

{
    "timestamp_ms": 1588879689394,
    "order_id": "HA16/4889",
}

/processValue

Topic: ia/<customerID>/<location>/<AssetID>/processValue

A message is sent here every time a process value has been prepared. Unique naming of the key.

{
    "timestamp_ms": 1588879689394, 
    "energyConsumption": 123456
}

/productImage

Topic structure: ia/<customer>/<location>/<assetID>/productImage

/productImage has the same data format as ia/rawImage, only with a changed topic.

/productImage can be acquired in two ways, either from ia/rawImage or /rawImageClassification. In the case of /rawImageClassification, only the Image part is extracted to /productImage, while the classification information is stored in the relational database.

{
	"timestamp_ms": 214423040823,
	"image":  {
		"image_id": "<SerialNumberCamera>_<timestamp_ms>",
		"image_bytes": "3495ask484...",
		"image_heigth": 800,
		"image_width": 1203,
		"image_channels": 3
	}
}

/productTag

Topic structure: ia/<customer>/<location>/<assetID>/productTag

/productTag is usually generated by contextualizing a processValue to a product.

{
    "timestamp_ms": 215348452385,
    "AID": "14432504350",
    "name": "torque",
    "value": 2.12
}

See also Digital Shadow for more information how to use this message

/productTagString

Topic structure: ia/<customer>/<location>/<assetID>/productTagString

/productTagString is usually generated by contextualizing a processValue to a product.

{
    "timestamp_ms": 1243204549,
    "AID": "32493855304",
    "name": "QualityClass",
    "value": "Quality2483"
}

See also Digital Shadow for more information how to use this message

/addParentToChild

Topic structure: ia/<customer>/<location>/<assetID>/addParentToChild

/productTagString is usually generated whenever a product is transformed into another product. It can be used multiple times for the same child to model that one product can consists out of multiple parents.

{
    "timestamp_ms": 124387,
    "childAID": "23948723489",
    "parentAID": "4329875"
}

See also Digital Shadow for more information how to use this message

3rd level: production data

This level contains only highly aggregated production data.

/state

Topic: ia/<customerID>/<location>/<AssetID>/state

A message is sent here each time the asset changes status. Subsequent changes are not possible. Different statuses can also be process steps, such as “setup”, “post-processing”, etc. You can find a list of all supported states here

{
    "timestamp_ms": 1588879689394, 
    "state": 10000
}

/cycleTimeTrigger

Topic: ia/<customerID>/<location>/<AssetID>/cycleTimeTrigger

A message should be sent under this topic whenever an assembly cycle is started.

currentStation in the JSON is a string lastStation in the JSON is a string sanityTime_in_s in the JSON is a integer

{
    "timestamp_ms": 1611170736684,
    "currentStation": "1a",
    "lastStation": "1b",
    "sanityTime_in_s": 100
}

/uniqueProduct

Topic: ia/<customerID>/<location>/<AssetID>/uniqueProduct

A message is sent here each time a product has been produced or modified. A modification can take place, for example, due to a downstream quality control.

There are two cases of when to send a message under the uniqueProduct topic:

• The exact product doesn’t already have a UID (-> This is the case, if it has not been produced at an asset incorporated in the digital shadow). Specify a space holder asset = “storage” in the MQTT message for the uniqueProduct topic.
• The product was produced at the current asset (it is now different from before, e.g. after machining or after something was screwed in). The newly produced product is always the “child” of the process. Products it was made out of are called the “parents”.

{
  "begin_timestamp_ms": 1611171012717,
  "end_timestamp_ms": 1611171016443,
  "product_id": "test123",
  "is_scrap": false,
  "uniqueProductAlternativeID": "12493857-a"
}

See also Digital Shadow for more information how to use this message

/scrapUniqueProduct

Topic: ia/<customerID>/<location>/<AssetID>/scrapUniqueProduct

A message is sent here each time a unique product has been scrapped.

{
    "UID": "161117101271788647991611171016443"
}

4th level: Recommendations for action

/recommendations

Topic: ia/<customerID>/<location>/<AssetID>/recommendations

Shopfloor insights are recommendations for action that require concrete and rapid action in order to quickly eliminate efficiency losses on the store floor.

{
    "timestamp_ms": 1588879689394,
    "recommendationUID": 3556,
    "recommendationType": 8996,
    "enabled": True,
    "recommendationValues": 
    {
        "percentage1": 30, 
        "percentage2": 40
    }
}

Explanation of IDs

There are various different IDs that you can find in the MQTT messages. This section is designed to give an overview.

These IDs are linked together in the database.

TimescaleDB structure

Here is a scheme of the timescaleDB structure:

(open the image using the right click for a better resolution)

2.3 - Available states for assets

Introduction

This data model is based on the following specifications:

• Weihenstephaner Standards 09.01 (for filling)
• Omron PackML (for packaging/filling)
• EUROMAP 84.1 (for plastic)
• OPC 30060 (for tobacco machines)
• VDMA 40502 (for CNC machines)

Additionally, the following literature is respected:

• Steigerung der Anlagenproduktivität durch OEE-Management (Focke, Steinbeck)

Abbreviations

• WS –> “TAG NAME”: Valuename (number)
• PackML –> Statename (number)
• EUROMAP –> Statusname (number)
• Tobacco –> ControlModeName (number)

ACTIVE (10000-29999)

The asset is actively producing.

10000: ProducingAtFullSpeedState

The asset is running on full speed.

Examples for ProducingAtFullSpeedState

• WS_Cur_State: Operating
• PackML/Tobacco: Execute

20000: ProducingAtLowerThanFullSpeedState

The asset is NOT running on full speed.

Examples for ProducingAtLowerThanFullSpeedState

• WS_Cur_Prog: StartUp
• WS_Cur_Prog: RunDown
• WS_Cur_State: Stopping
• PackML/Tobacco: Stopping
• WS_Cur_State: Aborting
• PackML/Tobacco: Aborting
• WS_Cur_State: Holding
• WS_Cur_State: Unholding
• PackML/Tobacco: Unholding
• WS_Cur_State: Suspending
• PackML/Tobacco: Suspending
• WS_Cur_State: Unsuspending
• PackML/Tobacco: Unsuspending
• PackML/Tobacco: Completing
• WS_Cur_Prog: Production
• EUROMAP: MANUAL_RUN
• EUROMAP: CONTROLLED_RUN

NOT INCLUDED FOR NOW:

• WS_Prog_Step: all

UNKNOWN (30000-59999)

The asset is in an unspecified state.

30000: UnknownState

We do not have any data for that asset (e.g. connection to PLC aborted).

Examples for UnknownState

• WS_Cur_Prog: Undefined
• EUROMAP: Offline

40000: UnspecifiedStopState

The asset is not producing, but we do not know why (yet).

Examples for UnspecifiedStopState

• WS_Cur_State: Clearing
• PackML/Tobacco: Clearing
• WS_Cur_State: Emergency Stop
• WS_Cur_State: Resetting
• PackML/Tobacco: Clearing
• WS_Cur_State: Held
• EUROMAP: Idle
• Tobacco: Other
• WS_Cur_State: Stopped
• PackML/Tobacco: Stopped
• WS_Cur_State: Starting
• PackML/Tobacco: Starting
• WS_Cur_State: Prepared
• WS_Cur_State: Idle
• PackML/Tobacco: Idle
• PackML/Tobacco: Complete
• EUROMAP: READY_TO_RUN

50000: MicrostopState

The asset is not producing for a short period (typically around 5 minutes), but we do not know why (yet).

MATERIAL (60000-99999)

The asset has issues with materials.

60000: InletJamState

The machine does not perform its intended function due to a lack of material flow in the infeed of the machine detected by the sensor system of the control system (machine stop). In the case of machines that have several inlets, the condition of lack in the inlet refers to the main flow, i.e. to the material (crate, bottle) that is fed in the direction of the filling machine (central machine). The defect in the infeed is an extraneous defect, but because of its importance for visualization and technical reporting, it is recorded separately.

Examples for InletJamState

• WS_Cur_State: Lack

70000: OutletJamState

The machine does not perform its intended function as a result of a jam in the good flow discharge of the machine detected by the sensor system of the control system (machine stop). In the case of machines that have several discharges, the jam in the discharge condition refers to the main flow, i.e. to the good (crate, bottle) that is fed in the direction of the filling machine (central machine) or is fed away from the filling machine. The jam in the outfeed is an external fault 1v, but it is recorded separately “because” of its importance for visualization and technical reporting.

Examples for OutletJamState

• WS_Cur_State: Tailback

80000: CongestionBypassState

The machine does not perform its intended function due to a shortage in the bypass supply or a jam in the bypass discharge of the machine detected by the sensor system of the control system (machine stop). This condition can only occur in machines that have two outlets or inlets and in which the bypass is in turn the inlet or outlet of an upstream or downstream machine of the filling line (packaging and palletizing machines). The jam/shortage in the auxiliary flow is an external fault, but is recorded separately due to its importance for visualization and technical reporting.

Examples for CongestionBypassState

• WS_Cur_State: Lack/Tailback Branch Line

90000: MaterialIssueOtherState

The asset has a material issue, but it is not further specified.

Examples for MaterialIssueOtherState

• WS_Mat_Ready (Information about which material is lacking)
• PackML/Tobacco: Suspended

PROCESS (100000-139999)

The asset is in a stop which is belongs to the process and cannot be avoided.

100000: ChangeoverState

The asset is in a changeover process between products.

Examples for ChangeoverState

• WS_Cur_Prog: Program-Changeover
• Tobacco: CHANGE OVER

110000: CleaningState

The asset is currently in a cleaning process.

Examples for CleaningState

• WS_Cur_Prog: Program-Cleaning
• Tobacco: CLEAN

120000: EmptyingState

The asset is currently emptied, e.g. to prevent mold for food products over the long breaks like the weekend.

Examples for EmptyingState

• Tobacco: EMPTY OUT

130000: SettingUpState

The machine is currently preparing itself for production, e.g. heating up.

Examples for SettingUpState

• EUROMAP: PREPARING

OPERATOR (140000-159999)

The asset is stopped because of the operator.

140000: OperatorNotAtMachineState

The operator is not at the machine.

150000: OperatorBreakState

The operator is in a break. note: different than planned shift as it could count to performance losses

Examples for OperatorBreakState

• WS_Cur_Prog: Program-Break

PLANNING (150000-179999)

The asset is stopped as it is planned to stop (planned idle time).

160000: NoShiftState

There is no shift planned at that asset.

170000: NoOrderState

There is no order planned at that asset.

TECHNICAL (180000-229999)

The asset has a technical issue.

180000: EquipmentFailureState

The asset itself is defect, e.g. a broken engine.

Examples for EquipmentFailureState

• WS_Cur_State: Equipment Failure

190000: ExternalFailureState

There is a external failure, e.g. missing compressed air

Examples for ExternalFailureState

• WS_Cur_State: External Failure

200000: ExternalInterferenceState

There is an external interference, e.g. the crane to move the material is currently unavailable.

210000: PreventiveMaintenanceStop

A planned maintenance action.

Examples for PreventiveMaintenanceStop

• WS_Cur_Prog: Program-Maintenance
• PackML: Maintenance
• EUROMAP: MAINTENANCE
• Tobacco: MAINTENANCE

220000: TechnicalOtherStop

The asset has a technical issue, but it is not specified further.

Examples for TechnicalOtherStop

• WS_Not_Of_Fail_Code
• PackML: Held
• EUROMAP: MALFUNCTION
• Tobacco: MANUAL
• Tobacco: SET UP
• Tobacco: REMOTE SERVICE

2.4 - Digital Shadow - track and trace

Introduction

During the production process a lot of data is generated (e.g., process values like temperature or images), but typically not linked to specific products. When investigating product returns, one needs to gather data from various data sources to understand what exactly happened, which might take so much time that it is not done at all.

The life would be much easier for a quality inspector if he would be able to enter the ID of the defect product and then receive all related information to it - from temperatures during the production to test results to product images.

Solution: We’ve expanded the United Manufacturing Hub so that the end-user only needs to do two things:

1. Connect data sources with MQTT, for example by leveraging barcodereader and sensorconnect
2. Process this raw data with Node-RED and send MQTT messages according to the UMH specification
3. Access the processed data either in a BI tool or using Grafana

To allow this to happen the backend has been modified to support multiple new MQTT message types in mqtt-to-postgresql and to provide more endpoints to fetch Digital Shadow related data from the database factoryinsight.

Dataflow

(right click on the image and open it for a better resolution)

This is the overview of the digital shadow concept. It follows the general design principles of the United Manufacturing Hub by sending all raw sensor data first to an MQTT broker and then continuously processing it.

The following chapters are going through the concept from left to right (from the inputs of the digital shadow to the outputs).

Step 1: Gathering data from the shopfloor

Data sources are connected by sending their data to the central MQTT broker. UMH recommends to stick to the data definition of the UMH datamodel for the topics and messages, but the implementation is client specific and can be modeled for the individual problem.

Example

This example is acquired using barcodereader

Topic: ia/rawBarcode/2020-0102/210-156
Topic structure: ia/rawBarcode/<transmitterID>/<barcodeReaderID>

{
    "timestamp_ms": 1588879689394, 
    "barcode": "1284ABCtestbarcode"
}

Step 2: contextualizing the data using microservices and MQTT

Now the information is available at the MQTT broker because of that to all subscribed services.

In the next step this raw data is contextualized, which means to link it to specific products. To identify a product two different type of IDs are used: AID’s and UID’s (identifiers are later explained in detail).

The raw data from the data sources needs to be converted to four different MQTT message types:

• productTag
• productTagString
• addParentToChild
• uniqueProduct

To do that we recommend writing microservices. You can do that either in Node-RED (our recommendation) or in a programming language of your choice. These microservices convert messages under a raw topic into messages under processValue or processValueString.

This typically only requires resending the message under the appropriate topic or breaking messages with multiple values apart into single ones.

Generating the contextualized messages

The goal is to convert messages under the processValue and the processValueString topics, containing all relevant data, into messages under the topic productTag, productTagString and addParentToChild. The latter messages contain AID’s which hold the contextualization information - they are tied to a single product.

The implementation of the generation of the above mentioned messages with contextualized information is up to the user and depends heavily on the specific process. To help with this we want to present a general logic and talk about the advantages and disadvantages of it:

General steps:

1. Make empty containers for predefined messages to mqtt-to-postgresql when the first production step took place
2. Fill containers step by step when relevant messages come in.
3. If full, send the container.
4. If the message from the first production step for the new product is received before the container is full, send container and set missing fields to null. Also send an error message.

Example process:

1. parent ID 1 scanned (specifically the later explained AID) -> barcode sent under processValueString topic
2. screws fixed -> torque processValue send
3. child AID scanned -> barcode processValueString send
4. parent AID 2 scanned -> barcode processValueString send

Example of generating a message under productTagString topic containing the measured torque value for the Example process:

• when parent AID scanned: make empty container for message because scanning parent AID is first step

{
    "timestamp_ms": 
    "AID":
    "name": "torque",
    "value":
}

• when torque value comes in: fill in value and timestamp

{
    "timestamp_ms": 13498435234,
    "AID":
    "name": "torque",
    "value": 1.458
}

• when child AID comes in: fill it in:

{
    "timestamp_ms": 13498435234,
    "AID": "34258349857",
    "name": "torque",
    "value": 1.458
}

Now the container is full: send it away.

Important: always send the uniqueProduct message first and afterwards the messages for the related productTag/productTagString and messages on the addParentToChild topic.

Advantages and disadvantages of presented process

Identifiers

The explaination of the IDs can be found in the UMH datamodel (especially the definition of the terms AID and UID)

Definition of when to change the UID

If we can move a product from point “A” in the production to point “B” or back without causing problems from a process perspective, the UID of the product should stay the same. (For example if the product only gets transported between point “A” and “B”).

If moving the object produces problems (e.g. moving a not yet tested object in the bin “tested products”), the object should have gotten a new UID on its regular way.

Example 1: Testing

Even though testing a product doesn’t change the part itself, it changes its state in the production process:

• it gets something like a virtual “certificate”
• the value increases because of that

-> Make a new UID.

Example 2: Transport

Monitored Transport from China to Germany (This would be a significant distance: transport data would be useful to include into digital shadow)

• parts value increases
• transport is separately paid
• not easy to revert

-> Make a new UID

Life of a single UID

MQTT messages under the productTag topic should not be used to indicate transport of a part. If transport is relevant, change the UID (-> send a new MQTT message to mqtt-to-postgresql under the uniqueProduct topic).

Example process to show the usage of AID’s and UID’s in the production:

Explanation of the diagram:

Assembly Station 1:

• ProductA and ProductB are combined into ProductC
• Because ProductA and ProductB have not been “seen” by the digital shadow, they get a new UID and asset = “storage” assigned (placeholder asset for unknown/unspecified origin).
• After ProductC is now produced it gets a new UID and as an asset, Assy1, because it is the child at Assembly Station 1
• The AID of the child can always be freely chosen out of the parent AID’s. The AID of ProductA (“A”) is a physical label. Because ProductB doesn’t have a physical Label, it gets a generated AID. For ProductC (child) we can now choose either the AID from ProductA or from ProductB. Because “A” is a physical label, it makes sense to use the AID of ProductA.

MQTT messages to send at Assembly 1:

• uniqueProduct message for ProductA origin, with asset = storage, under the topic: ia/testcustomer/testlocation/storage/uniqueProduct

  {
    "begin_timestamp_ms": 1611171012717,
    "end_timestamp_ms": 1611171016443,
    "product_id": "test123",
    "is_scrap": false,
    "uniqueProductAlternativeID": "A"
  }
  

• uniqueProduct message for ProductB origin, with asset = storage, under the topic: ia/testcustomer/testlocation/storage/uniqueProduct

  {
    "begin_timestamp_ms": 1611171012717,
    "end_timestamp_ms": 1611171016443,
    "product_id": "test124",
    "is_scrap": false,
    "uniqueProductAlternativeID": "B"
  }
  

• uniqueProduct message for ProductC, with asset = Assy1, under the topic: ia/testcustomer/testlocation/Assy1/uniqueProduct

  {
    "begin_timestamp_ms": 1611171012717,
    "end_timestamp_ms": 1611171016443,
    "product_id": "test125",
    "is_scrap": false,
    "uniqueProductAlternativeID": "A"
  }
  

• addParentToChild message describing the inheritance from ProductA to ProductC, under the topic: ia/testcustomer/testlocation/Assy1/addParentToChild

  {
  "timestamp_ms": 124387,
  "childAID": "A",
  "parentAID": "A"
  }
  

• addParentToChild message describing the inheritance from ProductB to ProductC, under the topic: ia/testcustomer/testlocation/Assy1/addParentToChild

  {
  "timestamp_ms": 124387,
  "childAID": "A",
  "parentAID": "B"
  }
  

• productTag message for e.g. a measured process value like the temperature,under the topic: ia/testcustomer/testlocation/Assy1/productTag

  {
  "timestamp_ms": 1243204549,
  "AID": "A",
  "name": "temperature",
  "value": 35.4
  }
  

Now the ProductC is transported to Assembly Station 2. Because it is a short transport, doesn’t add value etc. we do not need to produce a new UID after the transport of ProductA.

Assembly Station 2:

• ProductC stays the same (in the sense that it is keeping its UID before and after the transport), because of the easy transport.
• ProductD is new and not produced at assembly station 2, so it gets asset = “storage” assigned
• ProductC and ProductD are combined into ProductE. ProductE gets a new UID. Both AID’s are physical. We again freely choose the AID we want to use (AID C was chosen, maybe because after the assembly of ProductC and ProductD, the AID Label on ProductD is not accessible while the AID Label on the ProductC is).

Assembly Station 3:

• At Assembly Station ProductE comes in and is turned into ProductF
• ProductF gets a new UID and keeps the AID of ProductE. It now gets the Assy3 assigned as asset.

Note that the uniqueProduct MQTT message for ProductD would not be under the Topic of Assembly2 as asset but for example under storage. The convention is, that every part never seen by digital shadow “comes” from storage even though the UID and the related uniqueProduct message is created at the current station.

Batches of parts

If for example a batch of screws is supplied to one asset with only one datamatrix code (one AID) for all screws together, there will only be one MQTT message under the topic uniqueProduct created for the batch with one AID, a newly generated UID and with the default supply asset storage.

• The batch AID is then used as parent for a MQTT message under the topic addParentToChild. (-> mqtt-to-postgres will repeatedly fetch the same parent uid for the inheritanceTable)
• The batch AID only changes when new batch AID is scanned.

Step 3: mqtt-to-postgresql

The mqtt-to-postgresql microservice now uses the MQTT messages it gets from the broker and writes the information in the database. The microservice is not use-case specific, so the user just needs to send it the correct MQTT messages.

mqtt-to-postgresql now needs to generate UID’s and save the information in the database, because the database uses UID’s to store and link all the generated data efficiently. Remember that the incoming MQTT messages are contextualized with AID’s.

We can divide the task of mqtt-to-postgresql in three (regarding the digital shadow):

1. Use the MQTT message under the Topic uniqueProduct which gives us the AID and the Asset and make an entry in the uniqueProduct table containing the AID and a newly generated UID.

   1. Generate UID (with snowflake: https://en.wikipedia.org/wiki/Snowflake_ID)
   2. Store new UID and all data from uniqueProduct MQTT Message in the uniqueProductTable

2. Use productTag and productTagString topic MQTT messages. The AID and the AssetId is used to look for the uniqueProduct the messages belong to. The value information is then stored with the UID in the TimescaleDB

   1. Look in TimescaleDB, uniqueProductTable for the uniqueProduct with the same Asset and AID from the productTag message (the child)
   2. Get the UID when found from the child (that is why it is important to send the uniqueProduct message before sending productTag/productTagString).
   3. Write value information without AID, instead with the found UID in the uniqueProductTable

3. Use the addParentToChild message. Retrieve the child UID by using the child AID and the Asset. Get the parent UID’s by finding the last time the parents AID’s were stored in the uniqueProductTable.

   1. Look in TimescaleDB, uniqueProductTable for the uniqueProduct with the same Asset and AID as written in the child of the /addParentToChild message
   2. Look in the TimescaleDB, uniqueProductTable for all other assets for the last time the AID of the parent was used and get the UID
   3. Write UID of child and UID of the parent in the productInheritanceTable

Possible Problems:

• The uniqueProduct MQTT message of the child has to be made before we can store productTag or productTagString messages.
• All uniqueProduct of one step at one asset need to be stored before we can process addParentToChild messages. This means we also need to send possible parent uniqueProduct MQTT messages (asset = storage) before.

Step 4: Database and the database model

The structure of the database might be changed in the future.

Four tables are especially relevant:

• uniqueProductTable contains entries with a pair of one UID and one AID and other data.
• productTagTable and productTagStringTable store information referenced to the UID’s in the uniqueProductTable. Stored is everything from individual measurements to quality classes.
• productInheritanceTable contains pairs of child and parent UID’s. The table as a whole thereby contains the complete inheritance information of each individual part. One entry describes one edge of the inheritance graph.

The new relevant tables are dotted, the uniqueProductTable changes are bold in the timescaleDB structure visualization.

Step 5: factoryinsight

To make the relevant data from digital shadow available we need to provide new REST API’s. factoryinsight is the microservice doing that task. It accepts specific requests, accesses the timescale database and returns the data in the desired format.

Implemented functionality for digital shadow

The following function returns all uniqueProducts for that specific asset in a specified time range. One datapoint contains one childUID, AID and all parentAID’s regarding the asset. All uniqueProductTags and uniqueProductTagStrings (value and timestamp) for the childUID are returned to the same datapoint.

get /{customer}/{location}/{asset}/uniqueProductsWithTags from <timestamp1> to <timestamp2> (in RFC 3999 Format).

Example Return with two data points:

{
  "columnNames":
  [
    "UID",
    "AID",
    "TimestampBegin",
    "TimestampEnd",
    "ProductID",
    "IsScrap",
    "torque2",
    "torque1",
    "torque3",
    "torque4",
    "VH_Type123",
    "Gasket_Type123"
  ],
  "datapoints":
  [
    [
      2,
      "57000458",
      1629807326485,
      null,
      15,
      false,
      5.694793469033914,
      5.500782656464146,
      5.868141105450906,
      5.780416969961664,
      "57000458",
      "12000459"
    ],
    [
      6,
      "57000459",
      1629807443961,
      null,
      15,
      false,
      5.835010327979067,
      5.9666619086350945,
      5.425482064635844,
      5.6943075975030535,
      "57000459",
      "12000460"
    ]
  ]
}

Implemented logic of factoryinsight to achieve the functionality

1. Get all productUID’s and AID’s from uniqueProductTable within the specified time and from the specified asset.
2. Get all parentUID’s from the productInheritanceTable for each of the selected UID’s.
3. Get the AID’s for the parentUID’s from the uniqueProductTable.
4. Get all key, value pairs from the productTagTable and productTagStringTable for the in step 1 selected UID’s.
5. Return all parent AID’s under the column name of the corresponding parent productID’s. Return the child AID and UID. Return the productTag and productTagString values under the column name of the corresponding valueNames.

Step 6: SQL Database to connect to Tableau server

This is currently not included in the stack

For the digital shadow functionality we need to give the tableau server access to the data. Because the tableau server can’t directly connect to the REST API, we need to either use a database in between, or a tableau web data connector. We were advised against the tableau web data connector (general info about tableau webdata connectors: https://help.tableau.com/current/pro/desktop/en-us/examples_web_data_connector.htm ).

Because of that we implemented a SQL database in combination with Node-RED. Node-RED requests data from the REST API in regular intervals and pushes it into the SQL database. From there on we can access the data with the Tableau server.

Industry Example

To test the digital shadow functionality and display its advantages we implemented the solution in a model factory.

Long term: planned features

We plan to integrate further functionalities to the digital shadow. Possible candidates are:

• multiple new REST API’s to use the digital shadow more flexible
• detailed performance analysis and subsequent optimization to enable digital shadow for massive production speed and complexity
• A buffer in microservice mqtt-to-postgresql. If productTag/productTagString messages are sent to the microservice before writing the message uniqueProduct in the database the tags should be stored until the uniqueProduct message arrives. A buffer could hold productTag/productTagString messages and regularly try to write them in the database.

2.5 - Integration into Azure

This section is currently in development. Published articles so far:

• How to setup Azure IoT Hub
• Connecting the United Manufacturing Hub with Azure IoT Hub

2.6 - Open source in Industrial IoT: an open and robust infrastructure instead of reinventing the wheel.

Image author: Christopher Burns from Unsplash

How do we keep up with the big players in the industry despite limited resources and small market share? The best way to do this is to break new ground and draw on the collective experience of organizations and their specialists instead of trying to reinvent the wheel.

The collaborative nature of open source enables companies and individuals alike to turn their visions into reality and keep up with established players such as Siemens, Microsoft, and Rockwell, even without a large number of programmers and engineers. This is the path we are taking at United Manufacturing Hub.

Open source software has long since outgrown the insider stage and has become a veritable trend that is becoming the standard in more and more industries. Many, in the IT world, common and intensively used applications (e.g. Kubernetes, TensorFlow, f-prime by NASA ¹) have nowadays emerged in a collaborative approach and are available for free.

Open-Source on Mars: the Mars helicopter Ingenuity, relies heaviliy on open-source components like f-prime. Image author: JPL/NASA

Typically, these applications are not yet ready for production or Industry 4.0 use. Some, such as Grafana, are intended for completely different industries (Observability & Monitoring).

However, the source code of these software projects is freely accessible to everyone and can be individually adapted to specific needs. Thus, the application in the Industrial IoT is also no problem. In part, those applications were programmed over decades ² by several thousand developers and are continuously developed further ³.

The status quo

Today, it is common to develop proprietary Industrial IoT programs and software platforms - the opposite of open source.

A reason behind is, that companies do not want to have foreign code written into their applications and they want to offer the customer a self-made, end-to-end solution.

It is common for a team of over 20 or even 30 people to be assigned to develop a dashboard or IoT gateway, with the focus on a pretty looking (usually self-branded) user interface (UI) and design. Existing open source solutions or automation standards are rarely built upon.

Self-developed, in-house architectures are often strongly influenced by company-specific know-how and therefore usually also favor the company’s own products and services in their interfaces.

The result: the wheel is often reinvented in both the software and hardware areas. The resulting architectures create a lock-in effect that leads to a dependency of the manufacturing companies on their software and hardware suppliers.

Reinventing the wheel: The software world

In our opinion, good examples in the category “reinvented the wheel” from the software world are:

1. Self-developed visualizations such as visualizations from InfluxDB, PI Vision from OSIsoft or WAGO IoT Cloud Visualization (instead of Grafana).

2. Flow-based low code / no code apps such as Wire Graph by Eurotech (instead of node-red)

3. The bulk of Industrial IoT platforms that are claiming to be a “one-stop solution.” Such platforms are trying to cover every aspect from data acquisition, over processing, to visualization with in-house solutions (instead of relying on established technologies and just filling the gaps in the stack).

Both Grafana and node-red are highly professional solutions in their respective fields, which have already been used in various software projects for several years. Orchestrating such specialized applications means that offered and tested solutions can be put to good use.

Reinventing the wheel: The hardware world

There are numerous examples in the Industrial IoT hardware world where there is a conscious or unconscious deviation from established industry standards of the automation industry.

We have particularly noticed this with vendors in the field of Overall Equipment Effectiveness (OEE) and production overviews. Although they usually have very good dashboards, they still rely on self-developed microcontrollers combined with consumer tablets (instead of established automation standards such as a PLC or an industrial edge PC) for the hardware. In this case, the microcontroller, usually called IoT gateway, is considered a black box, and the end customer only gets access to the device in rare cases.

The advantages cannot be denied:

1. the system is easy to use,
2. usually very inexpensive,
3. and requires little prior knowledge.

Unfortunately, these same advantages can also become disadvantages:

1. the house system integrator and house supplier is not able to work with the system, as it has been greatly reduced for simplicity.
2. all software extensions and appearing problems, such as integrating software like an ERP system with the rest of the IT landscape, must be discussed with the respective supplier. This creates a one-sided market power (see also Lock-In).

Another problem that arises when deviating from established automation standards: a lack of reliability.

Normally, the system always need to work because failures lead to production downtime (the operator must report the problem). The machine operator just wants to press a button to get a stop reason or the desired information. He does not want to deal with WLAN problems, browser update or updated privacy policies on the consumer tablet.

The strongest argument: Lock-In

In a newly emerging market, it is especially important for a manufacturing company not to make itself dependent on individual providers. Not only to be independent if a product/company is discontinued but also to be able to change providers at any time.

Particularly pure SaaS (Software-as-a-Service) providers should be handled with caution:

• A SaaS offering typically uses a centralized cloud-based server infrastructure for multiple customers simultaneously. By its very nature, this makes it difficult to integrate into the IT landscape, e.g., to link with the MES system installed locally in the factory.
• In addition, a change of provider is practically only possible with large-scale reconfiguration/redevelopment.
• Lastly, there is a concern regarding the data ownership and security of closed systems and multiple SaaS offerings.

Basically, exaggerating slightly to make the point, it is important to avoid highly sensitive production data with protected process parameters getting to foreign competitors.

One might think that the manufacturing company is initially entitled to all rights to the data - after all, it is the company that “produced” the data.

In fact, according to the current situation, there is no comprehensive legal protection of the data, at least in Germany, if this is not explicitly regulated by contract, as the Verband der deutschen Maschinenbauer (VDMA) (Association of German Mechanical Engineering Companies) admits ⁴.

Even when it comes to data security, some people feel queasy about handing over their data to someone else, possibly even a US startup. Absolutely rightly so, says the VDMA, because companies based in the USA are obliged to allow US government authorities access to the data at any time ⁵.

An open source project can give a very good and satisfactory answer here:

United Manufacturing Hub users can always develop the product further without the original developers, as the source code is fully open and documented.

All subcomponents are fully open and run on almost any infrastructure, from the cloud to a Raspberry Pi, always giving the manufacturing company control over all its data.

Interfaces with other systems are either included directly, greatly simplifying their development, or can be retrofitted themselves without being nailed down to specific programming languages.

Unused potential

In the age of Industry 4.0, the top priority is for companies to operate as efficiently as possible by taking full advantage of their potential.

Open source software, unlike classic proprietary software, enables this potential to be fully exploited. Resources and hundreds of man-hours can be saved by using free solutions and standards from the automation industry.

Developing and offering a proprietary dashboard or IoT gateway that is reliable, stable, and free of bugs is wasting valuable time.

Another hundred, if not a thousand, man-hours are needed until all relevant features such as single sign-on, user management, or logging are implemented. Thus, it is not uncommon that even large companies, the market leaders in the industry, do not operate efficiently, and the resulting products are in the 6-to-7-digit price range.

But the efficiency goes even further:

Open source solutions also benefit from the fact that a community is available to help with questions. This service is rarely available with proprietary solutions. All questions and problems must be discussed with the multi-level support hotline instead of simply Googling the solution.

And so, unfortunately, most companies take a path that is anything but efficient. But isn’t there a better way?

United Manufacturing Hub’s open source approach.

Who says that you have to follow thought patterns or processes that everyone else is modeling? Sometimes it’s a matter of leaving established paths, following your own convictions, and initiating a paradigm shift. That is the approach we are taking.

We cannot compete with the size and resources of the big players. That is why we do not even try to develop in one or two years, with a team of 20 to 30 programmers what large companies have developed in hundreds of thousands of hours.

But that’s not necessary because the resulting product is unlikely to keep up with the open source projects or established automation standards. That is why the duplicated work is not worth the struggle .

The open source software code is freely accessible and thus allows maximum transparency and, at the same time, security. It offers a flexibility that is not reached by programs developed in the traditional way. By using open source software, the United Manufacturing Hub is taking an efficient way of developing. It allows us to offer a product of at least equal value but with considerably fewer development costs.

Example OEE dashboard created in Grafana

Simplicity and efficiency in the age of Industrial IoT.

At United Manufacturing Hub, we combine open source technologies with industry-specific requirements. To do this, we draw on established software such as Docker, Kubernetes or Helm ¹ and create, for example, data models, algorithms, and KPIs (e.g. the UMH data model, the factoryinsight and mqtt-to-postresql components) that are needed in the respective industries.

By extracting all data from machine controls (OPC/UA, etc.), we ensure the management and distribution of data on the store floor. Also, if additional data is needed, we offer individual solutions using industry-specific certified sensor retrofit kits, for example, at a steel manufacturer. More on this in one of the later parts of this series.

Summary

Why should we reinvent the wheel when we can focus our expertise on the areas we can provide the most value to our customers?

Leveraging open source solutions allow us to expose a stable and robust infrastructure that enables our customers to meet the challenges of Industrial IoT.

Because, in fact, manufacturing and Industrial IoT is not about developing new software at the drop of a hat. It is more about solving individual problems and challenges. This is done by drawing on a global network of experts who have developed special applications in their respective fields. These applications allow all hardware and software components to be quickly and easily established in the overall architecture through a large number of interfaces.

2.7 - An introduction into certificates and secure communication in IoT for normal people

Introduction

Newcomers to IT security are often confused about the difference between passwords and certificates. This article gives newcomers to IT security an introduction into the topic of secure communication with IoT devices. The focus is to explain the two fundamental methods to ensure that nobody else can read your communication, namely symmetric (e.g., passwords) and asymmetric encryption (e.g., certificates).

The need to ensure that nobody else than you and the recipient can read messages exist for thousands of years and has always been a battle between people creating encryption methods and so-called code-breakers, that focus on trying to decipher other messages. Wars have been started, won and lost because of a knowledge advantage gained through secure communications (see also ENIGMA or Zimmerman-Telegram).

In today’s world, the topic of secure communication is more relevant than ever: companies want to protect their intellectual knowledge from other companies and want to protect themselves from Ransomware attacks.

In IoT or Industrial IoT it is important as well:

• How to ensure that the data send to the cloud from my microcontroller or PLC ¹ is not read or modified?
• How to push updates onto my PLCs or IoT devices and prevent that someone else modifies or pushes their own updates?
• What are these certificates that are required for a connection to Azure IoT Hub, an MQTT broker ², or any other API?

Let’s have fun and deep dive in!

Symmetric encryption

Let’s start with the simpler method of both approaches: symmetric encryption. In cryptography, the field of encryption, methods are always explained using Alice, Bob, and Mallory. All of them (there are way more, see also the Wikipedia article on Alice and Bob) are fictional characters. Alice usually initiates the communication and wants to send a message to Bob without any third party like Mallory being able to read it.

Alice, Bob, and Mallory

In IoT, Alice could be a Raspberry Pi with a temperature sensor that is trying to send a message to Bob, who is a service running in the cloud. In Industrial IoT, Alice could be A PLC and Bob an Industrial IoT platform.

Alice, Bob, and Mallory practically

Before Alice and Bob can communicate with each other using a symmetric encryption they need to do two things first:

1. align on the encryption method
2. align on the secret, also called password

Let’s explain symmetric encryption using Alice and Bob and one of the oldest, yet, one of the most famous encryptions, Caesar’s cipher. This encryption technique is named after Julius Caesar’s who used it for his private correspondence.

before their first messages, Alice and Bob need to align to use this cipher and then additionally to use a number between 1-26 (which is also called a secret) first.

When Alice wants to send a message to Bob, Alice needs to encrypt the plaintext using the cipher and the secret. For Caesar’s cipher, it works by shifting the alphabet x characters to the right (and x is the chosen number from above, the secret).

Caesar’s cipher

For example, a right shift of 3 is resulting in the replacement of the letter A with D. Alice then sends the encrypted message to Bob. Bob then reverts the encryption by shifting the alphabet x characters back to the left, so a D gets converted back into an A. Now Bob can read the plaintext.

Theoretically, a third party like Mallory could not decrypt the encrypted message because Mallory doesn’t know the secret. You might ask now: “But hey, couldn’t Mallory just try out all numbers?” and you would be right. Having a secret or password that can only be between the numbers 1-26 is considered unsafe, but please keep in mind that this was the technology in ancient Rome.

In the example above we could easily crack the secret by trying out, which is also called brute-forcing. To prevent that more and more encryption methods emerged and almost as many methods to crack them - from good old brute-forcing to deriving parts of the secret by knowing already knowing certain parts of the plaintext (this was how the Allied cracked the German ENIGMA during World War 2).

Today, we have encryptions like AES rooted in advanced mathematics and to which there are currently no shortcuts except brute-forcing. These keys usually have a fixed length, for example AES-128 has a 128-bit key, and are derived from an easy-to-remember password using a key derivation function.

Brute-forcing the keys is not feasible with current or foreseeable hardware. The KDF is calculation intensive, which prevents brute-forcing easy-to-remember passwords (as long as they are reasonable long), so this method is currently approved in the U.S. for government documents of the highest classification.

Small side-topic: to login into systems, for example websites, you need to enter a username and a password. In these cases the password is not stored on the server, it is only stored as a hash, so even if the database of the website gets compromised one cannot derive the password from the hash. If you are interested in this topic, we suggest reading the following article.

Brute-forcing can also be improved by not checking everything randomly, but by e.g. checking the top 1000 common passwords or words from a dictionary (= dictionary attack).

So how do you then create good passwords? Several years ago the best practice was to use smaller randomly generated passwords, which should change frequently. But humans are humans and most of them will just add a ! or 1 at the end of the password. So nowadays the recommendation is to go for lengthy and still easy rememberable passwords. if2s&sd5 is more insecure and way harder to remember than AlexanderIs25YearsOldAndLikesToHaveHisDeskClean.

Obligatory xkcd: https://xkcd.com/936/

But the approach itself, the symmetric encryption with a secret / password, has two fundamental flaws:

1. How can Alice and Bob exchange the secret without Mallory reading it? Meeting in person is nowadays not always an option.
2. How can Bob know that the message is coming from Alice and that nothing was changed or deleted by Mallory?

For this, some very smart people invented the approach of Public-Key-Encryption, also known as asymmetric encryption.

Asymmetric encryption

In 1977 Ronald Rivest, Adi Shamir, and Leonard Adleman published an alternative method, which they called after their first letter in their name: RSA ³

Little fun fact: a similar method had already been developed at the British Agency GCHQ in 1970, but was classified at that time was not published until 1997 ⁴

Public-Key-Encryption

How does it work? For normal people here is a simple explaination from Computerphile on YouTube.

For the video haters out there, here is a small summary in text form:

Instead of using the same key for encrypting and decrypting you find a mathematic formula, that you can use separate keys, that belong together, but one cannot be derived from the other.

In practice, you have one key, the public key, that everyone knows, and which you use to encrypt a message. To decrypt the message you have to use another key, the so-called private key. When Alice generates these keys, she will keep the private key for herself and publish the public key to everyone.

She can also do the reverse: she can encrypt a message with her private key, so everyone will be able to decrypt it with her public key. Therefore, Bob can be sure that the message is coming from Alice (because only she has the private key and encrypt it like that).

If you are interested in the maths behind it, we recommend taking a look at the original paper. Or take a look at this video tutorial

The following text assumes that you have watched the video above (or read the small summary).

These public keys or private keys, that Computerphile is talking about, is what you would call “certificates”. They are usually stored as separate files in many different formats and file extensions. Here are some common ones for IoT:

• .pub, .pem, .crt for public keys
• .key or no file type at all for private keys
• .csr contains no keys at all, but is related (see also Certificate Signing Requests)
• .pfx, .p12, are other formats

What’s most important is the content of the file, so very often you can just change the file extension and load it in (most are PEM encoded, which looks like this). Sometimes the content differs (e.g., .pfx files or files in OpenSSH format are different than the usual PEM encoded .pem files), so you need conversion tools for that. If you are familiar with Google, it should be easy to find the correct tools and commands for that :)

Public Key Infrastructure (PKI)

But how can Alice ensure that the Public Key is actually from Bob and not from Mallory? What Computerphile did not talk about are Certificate Authorities (CA) and Intermediaries.

A CA is basically just a public-key pair that acts as a trusted instance. It can prove that a public key actually belongs to the person, computer, or website it claims to belong to. It does that by encrypting the public key of Bob with the CA’s private key (= Signing). These CA’s are mostly preinstalled on your operating system and help you in identifying whether your bank login information is actually send to the bank and not to someone else.

In some instances, it might make sense to create your own Public Key Infrastructure (PKI). You can find a guide on how to do that for example for MQTT brokers in our documentation

How it works in IoT

Practically, symmetric and asymmetric encryptions are used in parallel and combined with more advanced security topics like for example Diffie-Hellmann-Key-Exchange or Hashing. For more information on these advanced topics we can recommend to watch the YouTube videos from Computerphile (Diffie-Hellmann-Key-Exchange and Hashing)

The main reason for combining both technologies is computing speed. As a compromise, often a secret is generated and then shared between both parties using a (slow) asymmetric encryption. All the remaining messages are then encrypted using (fast) symmetric encryptions.

HTTPS

You are using these technologies already every day when you are visiting HTTPS websites like the one you are currently on (starts with https://). If you are sending it via HTTP, everyone on the internet can read your messages (see screenshot).

Source: Ibrahim Ali Ibrahim Diyeb, Anwar Saif, Nagi Ali Al-Shaibany,"Ethical Network Surveillance using Packet Sniffing Tools: A Comparative Study", International Journal of Computer Network and Information Security(IJCNIS), Vol.10, No.7, pp.12-22, 2018.DOI: 10.5815/ijcnis.2018.07.02 (https://www.researchgate.net/publication/326419957_Ethical_Network_Surveillance_using_Packet_Sniffing_Tools_A_Comparative_Study)

Security overview over our website docs.umh.app at 2021-10-18. Exact algorithms might change based on your browser

Example: MQTT and CA’s

In IoT we are relying on the same algorithms (SSL/TLS) that allow you to do safe online banking, with the exception that we cannot rely on external CA’s, but that we are creating it entirely ourselves.

We have a CA, a intermediate CA (to prevent using the root CA all the time), a server certificate for the MQTT broker and then client certificates for the MQTT clients.

Small recap: a CA is a trusted public-key pair. The intermediate CA is a public-key pair, that has been trusted by the root CA to take over the role of signing for a given time and in a given scope. A root CA is the “head” of the CA chain and can do everything.

The server certificate is the public side of the public-key pair created specifically for the MQTT broker, and where the intermediate CA signed it saying “yup, this is really the server mqtt.umh.app and it belongs to us. It can also act as a server.”.

The client certificates are signed by the intermediate CA as well with the permission to act only as clients (and not as server).

The public key of the root CA is embedded in all of our software, so that these devices know that, even if they never talked to each other before, they are communicating with whom they should communicate (e.g., a client connects to an MQTT broker and not to another client or someone entirely else).

This is the reason you need to copy these certificates onto the devices.

Example: viewing and analyzing the MQTT certificates

Last but not least, let’s take a look at the MQTT Public-Key infrastructure in detail. Specifically, let’s analyze these certificates.

In the folder deployment/factorycube-server/developmentCertificates/pki you find an example PKI infrastructure for factorycube-server. Almost all files are autogenerated with easy-rsa and you can actually ignore most of them. You can create your own PKI by following our tutorial

Let’s deep dive into 4 different certificates, to understand the structure of a PKI better:

1. ca.crt
2. issued/factorycube-server-vernemq.crt
3. issued/TESTING.crt
4. private/TESTING.key

Each certificate will be opened in xca, which is a graphical tool for creating and managing a PKI.

After importing the above certificates xca will look like this:

xca certificate section, after importing the above certificates

xca private key section, after importing the above certificates

ca.crt / “Easy-RSA CA”

This is the public-key of the root certificate (also known as root CA). We see that the certificate in the file ca.crt has the name “Easy-RSA CA”.

When opening it in an editor, it will look like gibberish:

ca.crt in raw format

After importing it to xca and clicking on it we can see all its details:

ca.crt in xca

ca.crt in xca - extensions

This includes:

• Fingerprint (to easily identify a key)
• Expiration (on the bottom)
• That is is a certificate authority (see second screenshot CA:TRUE)
• That it can be used to sign other certificates (see second screenshot X509v3 Key Usage)

In the overview panel in xca we see after importing the other certificates, that the others are shown under this certificate. xca automatically detects that ca.crt is the root CA and that is has signed the other certificates. It will therefore visualize it like that.

factorycube-server-vernemq.crt / “factorycube-server-vernemq”

This is the public-key of the MQTT server key-pair and you can view the same information as in the certificate above with some exceptions:

factorycube-server-vernemq.crt in xca

factorycube-server-vernemq.crt in xca - extensions

1. It is signed by Easy-RSA CA (see Signature)
2. It is not a CA and can only sign (but cannot sign other certificates, see X509v3 Key Usage)
3. The whole purpose is beeing a Webserver for the domain factorycube-server-vernemq (if you access it under a different domain the client will say that the certificate is invalid). You can see it under the points X509v3 Extended Key Usage and X509v3 Subject Alternative Name

TESTING.crt / “TESTING”

This is a client key and is only allowed to be used as a client (see X509v3 Extended Key Usage).

TESTING.crt in xca - extensions

TESTING.key / “TESTING”

This is the private key for the certificate above.

TESTING.key in xca

Outlook

“But is it really secure?”, one might ask. Probably every security expert would answer that with “It depends” (if someone says yes, they are probably no expert), quickly followed by “it is the best that we have so far”. Technologies like these are officially recommended by basically all national and international security agencies, even for critical applications like energy or banking sectors (NIST for the USA, BSI for Germany).

With the current rise of quantum computing the current algorithms might be considered ineffective in some time into the future, but we already have post-quantum algorithms out there that can safely be integrated into the current infrastructures. If you are interested, you can visit the website of the current tournament of the NIST

2.8 - Why we chose timescaleDB over InfluxDB

Introduction

The introduction and implementation of an Industrial IoT strategy is already complicated and tedious. There is no need to put unnecessary obstacles in the way through lack of stability, new programming languages, or more databases than necessary. You need a piece of software that you can trust with your company’s most important data.

We are often asked why we chose timescaleDB instead of InfluxDB. Both are time-series databases suited for large amounts of machine and sensor data (e.g., vibration or temperature).

We started with InfluxDB (probably due to its strong presence in the home automation and Grafana communities) and then ended up with timescaleDB based on three arguments. In this article, we would like to explain our decision and provide background information on why timescaleDB makes the most sense for the United Manufacturing Hub.

Argument 1: Reliability & Scalability

A central requirement for a database: it cannot lose or corrupt your data. Furthermore, as a central element in an Industrial IoT stack, it must scale with growing requirements.

TimescaleDB

TimescaleDB is built on PostgreSQL, which has been continuously developed for over 25 years and has a central place in the architecture of many large companies like Uber, Netflix, Spotify or reddit. This has created a fault-tolerant database that can scale horizontally across multiple servers. In short: it is boring and works.

InfluxDB

In contrast, InfluxDB is a relatively young startup that was funded at 119.9 M USD (as of 2021-05-03) but still doesn’t have 25+ years of expertise to fall back on.

On the contrary: Influx has completely rewritten the database twice in the last 5 years ^{1 2}. Rewriting software can improve fundamental issues or add exciting new features. However, it is usually associated with breaking changes in the API and new unintended bugs. This results in additional migration projects, which take time and risk system downtime or data loss.

Due to its massive funding, we get the impression that they add quite a lot of exciting new features and functionalities (e.g., an own visualization tool). However, after testing, we noticed that the stability suffers under these new features.

In addition, Influx only offers the horizontally scalable version of the database in the paid version, which will scare off companies wanting to use it on a larger scale as you will be fully dependent on the provider of that software (vendor lock-in).

Summary

With databases, the principle applies: Better boring and working than exciting and unreliable.

We can also strongly recommend an article by timescaleDB.

Argument 2: SQL is better known than flux

The second argument refers to the query language, i.e., the way information can be retrieved from the database.

SQL (timescaleDB)

TimescaleDB, like PostgreSQL, relies on SQL, the de facto standard language for relational databases. Advantages: A programming language established for over 45 years, which almost every programmer knows or has used at least once. Any problem? No problem, just Google it, and some smart person has already solved it on Stack Overflow. Integration with PowerBI? A standard interface that’s already integrated!

SELECT time, (memUsed / procTotal / 1000000) as value
FROM measurements
WHERE time > now() - '1 hour';

Example SQL code to get the average memory usage for the last hour.

flux (InfluxDB)

InfluxDB, on the other hand, relies on the homegrown flux, which is supposed to simplify time-series data queries. It sees time-series data as a continuous stream upon which are applied functions, calculations and transformations³.

Problem: as a programmer, you have to rethink a lot because the language is flow-based and not based on relational algebra. It takes some time to get used to it, but it is still an unnecessary hurdle for those not-so-tech-savvy companies who already struggle with Industrial IoT.

From some experience, we can also say that the language quickly reaches its limits. In the past, we worked with additional Python scripts that extract the data from InfluxDB via Flux, then process it and then play it back again.

// Memory used (in bytes)
memUsed = from(bucket: "telegraf/autogen")
  |> range(start: -1h)
  |> filter(fn: (r) =>
    r._measurement == "mem" and
    r._field == "used"
  )

// Total processes running
procTotal = from(bucket: "telegraf/autogen")
  |> range(start: -1h)
  |> filter(fn: (r) =>
    r._measurement == "processes" and
    r._field == "total"
    )

// Join memory used with total processes and calculate
// the average memory (in MB) used for running processes.
join(
    tables: {mem:memUsed, proc:procTotal},
    on: ["_time", "_stop", "_start", "host"]
  )
  |> map(fn: (r) => ({
    _time: r._time,
    _value: (r._value_mem / r._value_proc) / 1000000
  })
)

Example Flux code for the same SQL code.

Summary

In summary, InfluxDB puts unnecessary obstacles in the way of not-so-tech-savvy companies with flux, while PostgreSQL relies on SQL, which just about every programmer knows.

We can also strongly recommend the blog post by timescaleDB on exactly this topic.

Argument 3: relational data

Finally, the argument that is particularly important for production: Production data is more relational than time-series based.

Relational data is, simply put, all table-based data that you can store in Excel in a meaningful way, for example, shift schedules, orders, component lists, or inventory.

Relational data. Author: AutumnSnow, License: CC BY-SA 3.0

TimescaleDB provides this by default through the PostgreSQL base, whereas with InfluxDB, you always have to run a second relational database like PostgreSQL in parallel.

If you have to run two databases anyway, you can reduce complexity and directly use PostgreSQL/timescaleDB.

Not an argument: Performance for time-series data

Often the duel between timescaleDB and InfluxDB is fought on the performance level. Both databases are efficient, and 30% better or worse does not matter if both databases are 10x-100x faster ⁴ than classical relational databases like PostgreSQL or MySQL.

Even if it is not important, there is strong evidence that timescaleDB is actually more performant. Both databases regularly compare their performance against other databases, and InfluxDB never compares itself to timescaleDB. However, timescaleDB has provided a detailed performance guide of influxDB.

Summary

Who do you trust more? The nerdy and boring, or the good-looking accountant, with 25 new exciting tools?

The same goes for databases: Boring is awesome.

2.9 - The UMH datamodel (preview for v1.0.0)

This site is currently still in work

2.9.1 - Kafka

Default settings

By default, the following important settings are used:

Topics

Our Kafka topics are structured as follows:

  ia.CUSTOMER.LOCATION.MACHINE.EVENT

There are two exception to this rule:

• ia.raw.TRANSMITTERID.GATEWAYSERIALNUMBER.PORTNUMBER.IOLINKSENSORID
• ia.rawImage.TRANSMITTERID.CAMERAMACADDRESS

Note: The maximum length of a topic is < 200 characters and the depth is < 10 (Amount of separators .).

Specifications for those can be found on the UMH datamodel page.

All Topics may have suffixes, which might get ignored by the different microservices. For example:

  ia.testCustomer.testLocation.testAsset.processValue.temperature

All names have to match the following regex:

^[a-zA-Z0-9_\-]$

Customer

This is name of the customer (e.g: united-manufacturing-hub). It can also be an abbreviation (e.g: umh) of the customer name.

Location

This is the name of the location, the sender belongs to. It can be a physical location (aachen), an virtual location (rack-1), or any other unique specifier.

Machine

This is the name of the machine, the sender belongs to. It can be a physical machine (printer-1), a virtual machine (vmachine-1), or any other unique specifier.

Event

Our kafka stack currently supports the following events:

• addMaintenanceActivity
• addOrder
• addParentToChild
• addProduct
• addShift
• count
• deleteShiftByAssetIdAndBeginTimestamp
• deleteShiftById
• endOrder
• modifyProducedPieces
• modifyState
• processValue
• processValueFloat64
• processValueString
• productTag
• productTagString
• recommendation
• scrapCount
• startOrder
• state
• uniqueProduct
• scrapUniqueProduct

Further information about these events can be found at the UMH datamodel site.

Routing

Below you can find an example flow of messages.

Edge PC

In this example, we have an edge pc, which is connected to multiple sensors and a camera. It also receives data via MQTT, Node-RED and a barcode reader.

In our dataflow, we handle any IO-Link compatible sensor with sensorconnect, which reads IO-Link data and publishes it to Kafka. Compatible cameras / barcode readers are handled by cameraconnect and barcode reader respectively.

Node-RED can be used to pre-process arbitrary data and the publish it to Kafka.

MQTT-Kafka-Bridge takes MQTT data and publishes it to Kafka.

Once the data is published to the Edge Kafka broker, other microservices can subscribe to the data and produce higher level data, which gets re-published to the Edge Kafka broker.

Kafka bridge

This microservice can sit on either the edge or server and connects two kafka brokers. It has a regex based filter, for sending messages, which can be bi- or uni-directional. Also, it filters out duplicated messages, to prevent loops.

Every bridge adds an entry to the kafka header, to identify the source of the message and all hops taken.

Server

On the server, we have two microservices listening for incoming data.

Kafka-to-blob is a microservice which listens to Kafka and publishes the data to blob storage, in our case Minio. Kafka-to-postgresql is a microservice which listens to Kafka and publishes the data to a PostgreSQL database, with timescale installed.

Guarantees

Our system is build to provide at-least-once delivery guarantees, once a message first enters any kafka broker, except for high throughput data (processValue, processValueFloat64, processValueString).

Every message taken out of the broker, will only get committed to the broker, once it has been processed, or successfully returned to the broker (in case of an error).

For this, we use the following kafka settings:

{
  "enable.auto.commit":       true,
  "enable.auto.offset.store": false,
  "auto.offset.reset":        "earliest"
}

• enable.auto.commit
  • This will auto commit all offsets, in the local offset store, every couple of seconds.
• enable.auto.offset.store
  • We manually store the offsets in the local offset store, once we confirmed, that the message has been processed.
• auto.offset.reset
  • This will return the offset pointer to the earliest unprocessed message, in case of a re-connect.

Note, that we could have gone for disabling “enable.auto.commit”, but in our testing, that was significantly slower.

For in-depth information about how we handle message inside our microservices, please see their documentation:

• Kafka-to-blob
• Kafka-to-postgresql
• Kafka-bridge

3 - Examples

Metalworking industry

Flame cutting & blasting

Retrofitting of 11 flame cutting machines and blasting systems at two locations using sensors, barcode scanners and button bars to extract and analyze operating data.

See also the detailed documentation.

Identification of the optimization potential of two CNC milling machines

Two-month analysis of CNC milling machines and identification of optimization potentials. Automatic data acquisition coupled with interviews of machine operators and shift supervisors revealed various optimization potentials.

See also the detailed documentation.

Textile industry

Cycle time monitoring

See also the detailed documentation.

Retrofitting of weaving machines for OEE calculation

Retrofitting of weaving machines that do not provide data via the PLC to extract operating data. Subsequent determination of the OEE and detailed breakdown of the individual key figures

See also the detailed documentation

Filling & packaging industry

Performance management in a brewery

Retrofit of a bottling line for different beer types. Focus on the identification of microstops causes and exact delimitation of the bottleneck machine.

See also the detailed documentation.

Retrofit of a Japanese pharmaceutical packaging line

Retrofit of a Japanese pharmaceutical packaging line for automatic analysis of microstop causes as well as to relief the machine operators of data recording.

See also the detailed documentation.

Quality monitoring in a filling line

TODO: #69 add short description for DCC quality check

See also the detailed documentation.

Semiconductor industry

Identification of optimization potential in the context of the COVID-19 crisis

Use of the factorycube for rapid analysis of bottleneck stations. The customer was thus able to increase the throughput of critical components for ventilators within the scope of COVID-19.

See also the detailed documentation.

3.1 - Flame cutting & blasting

Profile

Photos

Challenges

Lack of transparency about production processes

• Lead times are unknown
• No comparison between target and actual times
• Duration and causes of machine downtimes are unclear

Heterogeneous machinery and machine controls from different manufacturers

• Only minimal connection of the machines to the ERP system
• Manual input of production data into the ERP system
• Machine controls are partially locked by manufacturers
• Machine controls use different protocols

Reliable production planning and quotation generation not possible

• No data on past and current machine utilization available
• Quotation preparation is done with theoretical target times, as no information about actual times is available

Solution

Integration

TODO: #68 add integration for flame cutting

Installed hardware

factorycube

factorycube sends the collected production data to the server. See also [factorycube].

Gateways

Gateways connect the sensors to the factorycube.

Models:

• ifm AL1352
• ifm AL1350

Light barriers

Light barriers are installed on cutting machines and are activated when the cutting head is lowered and in use. Used to measure machine conditions, cycle times and piece counts.

Models:

• ifm O5D100 (Optical distance sensor)
• ifm O1D108 (Optical distance sensor)

Vibration sensors

Vibration sensors are installed on the beam attachments and detect the machine condition via vibration. Used to measure machine conditions.

Model:

• ifm VTV122 (vibration transmitter)

Button bar

Button bar is operated by the machine operator in case of a machine standstill. Each button is assigned a reason for standstill. Used to identify the causes of machine downtime.

Model:

• Self-made, based on Siemens SIRIUS ACT

Barcode scanner

Barcode scanners are used to scan production orders, which contain the target times. Used to scan target times for target/actual comparison.

Model:

• Datalogic PowerScan PD9531
• Datalogic USB Cable Straight 2m (CAB-438)

Implemented dashboards

The customer opted for our SaaS offering. We created the following dashboards for the client.

Navigation

Default navigation options from Grafana, which we modified to allow custom menus.

1. Customizable menu lets you quickly navigate between dashboards
2. In the time selection you can adjust the times for the current dashboard

Plant-manager dashboard

1. Dashboard for the plant manager / shift supervisor, which gives an overview of the production in the factory
2. For each machine the current machine status
3. For each machine, the overall equipment effectiveness / OEE for the selected time period
4. For each machine, a timeline showing the machine statuses in color
5. Overview of all orders, including target/actual deviation and which stop reasons, including setup times, occurred during the order

Machine deep dive

1. Dashboard for the machine operator / shift supervisor, which displays the details for a machine
2. The current machine status with time stamp
3. The overall equipment effectiveness / OEE for the selected time period, including trend over time
4. An overview of the accumulated duration of each stop reason
5. A timeline where the machine states are color coded
6. A timeline where the shifts become visible
7. A timeline where the orders are displayed
8. Overview of all orders, including target/actual deviation and which stop reasons, including setup times, occurred during the order
9. Overview of the number of individual stop reasons

Cross-factory dashboard

1. Dashboard for the cross-factory manager, who can use this to obtain an overview of the sites
2. The overall equipment effectiveness / OEE for the selected time period for all machines.
3. The minimum overall equipment effectiveness / OEE for the selected time period for machine type A.
4. The average overall equipment effectiveness / OEE for the selected time period for machine type A
5. The maximum overall equipment effectiveness / OEE for the selected period for machine type A
6. Overview of all orders, including target/actual deviation and which stop reasons, including setup times, occurred during the order
7. Export function as .csv

3.2 - Brewery

Profile

Photos

Challenges

Lack of transparency about production processes

• Duration and causes of machine downtimes are unclear
• High proportion of smaller microstops with unknown cause
• Exclusively reactive maintenance, as data on the condition of the components is lacking

Moving bottleneck

• Since the production process is highly interlinked, a stoppage of a single machine can lead to a stoppage of the entire line
• The problem machine “bottleneck machine” is difficult to identify, as it can shift during a shift and is difficult to see with the eye

High effort to collect data as part of the introduction of a continuous improvement process

• Changeover times must be recorded manually with a stop watch and are still not very standardized
• No data on past and current machine utilization available
• Maintenance actions recorded manually, no automatic system to log, store and visualize error codes from machine

Solution

Integration

At the beginning, a “BDE entry program” was carried out together with a lean consulting to identify optimization potentials and to present our solution. For this purpose, the [factorycube] was installed at the filler within a few hours in combination with the tapping of electrical signals from the control system and button strips. A connection of the PLC interfaces was initially out of the question due to time and cost reasons. After the customer decided on a permanent solution, the factorycube was dismounted.

All machines have been equipped with the “Weihenstephaner Standards”, a protocol commonly used in the German brewery industry and were already connected within a machines network. Therefore, the installation was pretty straightforward using our enterprise plugin for that protocol and one central server.

Installed hardware

Server

Implemented dashboards

The customer opted for our SaaS offering. We created the following dashboards for the client.

Navigation

Default navigation options from Grafana, which we modified to allow custom menus.

1. Customizable menu lets you quickly navigate between dashboards
2. In the time selection you can adjust the times for the current dashboard

Plant-manager dashboard

Dashboard for the plant manager / shift supervisor, which gives an overview of the production in the factory

1. For each machine the current machine status
2. For each machine, the overall equipment effectiveness / OEE for the selected time period
3. For each machine, a timeline showing the machine statuses in color

Performance cockpit

Dashboard for the supervisor to get an overview of the machine

1. The current machine status
2. The overall equipment effectiveness / OEE for the selected time period, including trend over time
3. The average changeover time
4. The average cleaning time
5. A timeline where the machine states are color coded
6. A timeline where the shifts become visible
7. A timeline with the machine speed
8. Overview of the number of individual stop reasons, excluding technical defects as they are not relevant for the shift

Maintenance cockpit

Dashboard for the head of maintenance to get an overview of the machine

1. The current machine status
2. The overall equipment effectiveness / OEE for the selected time period, including trend over time
3. The MTTR (mean time to repeair), an important key figure for maintenance
4. The MTBF (mean time between failures), an important key figure for maintenance
5. A timeline where the machine states are color coded
6. A timeline where the process value “bottle lock open/close” is visualized. This helps the manager of the maintenance to isolate the cause of a problem more precisely.
7. A timeline with the machine speed
8. An overview of the accumulated duration of each stop reason, that is relevant for maintenance
9. Overview of the number of individual stop reasons, that is relevant for maintenance

3.3 - Semiconductor

Profile

Photos

Challenges

Increasing demand could not be fulfilled

• the demand for the product, which was required for ventilators, was increasing over 1000% due to the COVID-19 crisis
• the production was struggling to keep up with the ramp up

Production downtime needed to be avoided at all costs

• production downtime would have meant not fulfilling the demand

A quick solution was needed

• to meet the demand, the company needed a quick solution and could not accept months of project time

Solution

Integration

We were given a 2h time slot by the company to install the sensors, from the time we entered the factory until the time we left (including safety briefings and ESD compliance checks). With the help of videos, we got an overview beforehand and created a sensor plan. During this time slot, we used the machine operator’s break to install all the sensors and verified the data during the subsequent machine run. Through VPN we were able to access the device and fine-tune the configuration.

Installed hardware

factorycube

factorycube sends the collected production data to the server. See also [factorycube].

Gateways

Gateways connect the sensors to the factorycube.

Models:

• ifm AL1352

Ultrasonic sensor

picture TODO

The ultrasonic sensor was used to measure whether the robot was currently moving and thus whether the machine was running.

Models:

• TODO

Proximity sensor

Proximity sensors were used to detect if the product was ready for operator removal. Together with the ultrasonic sensors, we were able to measure whether the machine was standing because the machine operator had not removed the product and was therefore not available.

Models:

• ifm KQ6005

Button bar

Button bar is operated by the machine operator in case of a machine standstill. Each button is assigned a reason for standstill. Used to identify the causes of machine downtime.

Model:

• Self-made, based on Siemens TODO

Implemented dashboards

The customer opted for our SaaS offering and a additional analysis of the data.

Dashboard screenshot

The customer opted for SaaS solution and required only a very simple dashboard as most insights were gained from a detailed analysis. The dashboard includes the functionality to export data as .csv.

Additional analysis

The data was exported into .csv and then analyzed in Python & Excel. Together with interviews of the operators and supervisors we could extract multiple insights including optimization potential through alignment of the work processes and improvement of changeovers through Single-minute exchange of die (SMED).

3.4 - Cycle time monitoring in an assembly cell

Profile

An assembly cell was retrofitted to measure and optimize cycle times. Customizable textile wristbands are produced in the assembly cell.

Photos of the machines

Challenges

Lack of information about production performance

• Cycle times are unknown
• Bottleneck of the assembly cell cannot be identified
• No information about productivity of individual employees
• Piece counts are not documented
• No comparison between target and actual performance

Lack of transparency about downtimes

• Frequency and duration of downtimes of the assembly cell are not recorded
• Causes of downtime are often unknown and not documented

Connection of assembly cell to conventional systems not possible

• Sewing machines do not have machine controls that could be connected

Solution

Integration

TODO: #66 Add integration for assembly analytics

Installed hardware

factorycube

factorycube sends the collected production data to the server. See also [factorycube].

Gateways

Gateways connect the sensors to the factorycube.

Models:

• ifm AL1352

Light barriers

Light barriers are installed on the removal bins and are activated when the employee removes material. Used to measure cycle time and material consumption.

Models:

• ifm O5D100 (Optical distance sensor).

Proximity sensor

Proximity sensors on the foot switches of sewing machines detect activity of the process. Used to measure cycle time.

Models:

• ifm KQ6005

Barcode scanner

The barcode scanner is used to scan the wristband at the beginning of the assembly process. Process start and product identification.

Model:

• Datalogic PowerScan PD9531
• Datalogic USB Cable Straight 2m (CAB-438)

Implemented dashboards

The customer opted for a combination of our SaaS offering with the building kit (and thus an on-premise option). The customer decided to go for PowerBI as a dashboard and connected it via the REST API with factoryinsight.

Used node-red flows

With the help of Assembly Analytics Nodes, it is possible to measure the cycle time of assembly cells in order to measure and continuously improve their efficiency in a similar way to machines.

Here is an exemplary implementation of those nodes:

There are 2 stations with a total of 4 cycles under consideration

Station 1 (AssemblyCell1):

1a: Starts with scanned barcode and ends when 1b starts

1b: Starts with a trigger at the pick to light station and ends when station 1a starts

Station 2 (AssemblyCell2):

2a: Starts when the foot switch at the 2nd station is pressed and ends when 2b starts

2b: Starts when the quality check button is pressed and ends when 2a starts.

Assumptions:

• Unrealistically long cycle times are filtered out (cycle times over 20 seconds).
• There is a button bar between the stations to end the current cycle and mark that product as scrap. The upper 2 buttons terminate the cycle of AssemblyCell1 and the lower ones of AssemblyCell2. The aborted cycle creates a product that is marked as a scrap.

Nodes explained:

• Assembly Analytics Trigger: Cycles can be started with the help of the “Assembly Analytics Trigger” software module.

• Assembly Analytics Scrap: With the help of the software module “Assembly Analytics Scrap”, existing cycles can be aborted and that produced good can be marked as “scrap”.

• With the help of the software module “Assembly Analytics Middleware”, the software modules described above are processed into “unique products”.

Here you can download the flow described above

3.5 - Quality monitoring in a bottling line

Profile

A bottling line for filling water bottles was retrofitted with an artificial intelligence quality inspection system. With the help of a camera connected to an ia: factorycube, the bottles are checked for quality defects and sorted out by a pneumatic device in the event of a defect.

Photos of the machines

Challenges

Manual visual inspection causes high costs

• Each individual bottle is checked for quality defects by an employee
• One employee is assigned to each shift exclusively for quality inspection

Customer complaints and claims due to undetected quality defects

• Various quality defects are difficult to detect with the naked eye and are occasionally overlooked

No data on quality defects that occur for product and process improvement

• Type and frequency of quality defects are not recorded and documented
• No data exists that can be analyzed to derive improvement measures for product and process optimization

Solution

Integration

TODO: #67 Add integration for DCC quality check

Installed hardware

factorycube

A machine learning model runs on the factorycube, which evaluates and classifies the images. See also [factorycube].

Gateways

Gateways connect the sensors to the factorycube.

Models:

• ifm AL1352

Light barriers

A light barrier identifies the bottle and sends a signal to the factorycube to trigger the camera.

Models:

• ifm O5D100 (Optical distance sensor)

Camera

A camera takes a picture of the bottle and sends it to the factorycube.

Models:

• Allied Vision (Mako G-223)

Detectable quality defects

Automated action

As soon as a quality defect is detected the defect bottle is automatically kicked out by the machine.

3.6 - Pharma packaging

Profile

TODO: #70 add pharma packaging case

3.7 - Weaving

Profile

TODO: #71 add weaving case

3.8 - CNC Milling

TODO #65

4 - Tutorials

4.1 - FAQ

I cannot login into ia: factoryinsight / Grafana although I’ve entered my credentials correctly several times

The username in Grafana is case-sensitive. That means, if the user is **J**eremy.**T**[email protected] and you enter **j**eremy.**t**[email protected] you will get a failed login message.

4.2 - Azure

4.2.1 - Connecting with Azure IoT Hub

This tutorial is part of a larger series on how to integrate the United Manufacturing Hub into the Azure stack.

Tutorial

In this tutorial we are leveraging the fact that Azure IoT Hub is nothing but a MQTT broker with benefits (e.g., device management or REST API). We will therefore create a connection via MQTT.

Prerequisites

• Basic knowledge about IT / OT, Azure and the difference between symmetric and asymmetric encryption
• You should have one instance of IoT Hub running and one device created with asymmetric encryption. You can follow our tutorial for setting up Azure IoT Hub.
• By default Microsoft recommends using symmetric encryption as it is more easier to implement, but they say themselves that asymmetric encryption is more secure. If you are using symmetric encryption (Username / Password authentification, no certificates) there might be some steps that are different for you
• All information for your Azure IoT Hub device ready:
• Hostname of the Azure IoT Hub: e.g., contoso-test.azure-devices.net
• Device ID: e.g., factory3
• (symmetric) Username: e.g., contoso-test.azure-devices.net/factory3/?api-version=2018-06-30
• (symmetric) Password / SAS Token: e.g., SharedAccessSignature sr=contoso-test……
• (asymmetric) Certificate files
• Furthermore, we recommend having access to Azure IoT Hub using the Azure Device Explorer (download required) to verify whether the connection worked.

Option 1: using Node-RED

Advantages:

• quick
• easy

Disadvantages:

• can be unrealiable, e.g., when the user accidentally adds an unstable external plugin, which takes down Node-RED
• does not buffer data in case of internet outages

We will do this in three steps. You can also download our example flows.json, where you just need to change your authentification parameters (certificates or username / password, host, etc.) and you are ready to go.

Step 1: Subscribing to all messages

MQTT-In node in Node-RED

As a first step we take all relevant data out of the MQTT broker and thereby starting a Node-RED flow. Add a MQTT-In node. The MQTT broker details should already been set if you have used our recommended installation method (see also Getting Started).

If not, you can connect on factorycube-edge with the URL: factorycube-edge-emqxedge-service under the port 1883.

In the Topic section you can specify what messages from which asset you want to send to Azure IoT Hub using wildcards. If you want to send all messages you can use ia/# as a topic. If you just want to send process values of all machines in plant3 of customer factoryinsight you can use:

ia/factoryinsight/plant3/+/processValue

MQTT-In node in Node-RED - detailled view

You can test the flow by adding a debug node and connecting it the MQTT-In node with it.

Step 2: Changing the payload to contain payload and topic

Azure by default does not support the topic structure of the United Manufacturing Hub. As it contains important infortmation, e.g., the asset that the message is about, we will put that into the payload as well.

json and function node in Node-RED

Add a json node AND a function node with the following content and connect it to the MQTT-In node that we previously created:

msg.payload = {
    "topic": msg.topic,
    "payload": msg.payload
}
msg.topic = ""

return msg;

Code pasted into Node-RED

Verify by connecting it with the debug node.

Step 3: Sending the data to Azure IoT Hub

We recommend following the steps in the article Connecting Node-Red to Azure IoT Hub using MQTT nodes from Nikhil Kinkar. He is using symmetric encryption. For asymmetric encryption edit the TLS configuration and upload there your certificates.

MQTT-out node for connecting Node-RED with Azure IoT

You should have the connection parameters (username, host, port, etc.) already available, if not go to Prerequisites

If necessary: as root CA you need to use the Baltimore CyberTrust Root

Do not forget to set the topic as well (see also the tutorial above):

MQTT-out node - topic and server

MQTT-out node - MQTT broker (1/2)

MQTT-out node - MQTT broker (2/2)

Connect it with the previously created function node and test it. If everything went well the messages should now appear in the device explorer in Azure IoT Hub.

The entire flow

Step 4: check whether the messages appear in Azure IoT Hub using the device explorer

Finding the iothubowner SAS token in Azure

Open the Azure IoT Explorer and connect using your iothubowner SAS token (see image above). If everything went well you should see messages coming up (assuming that there are messages in the local MQTT broker. If not, try to send some debug messages to test the connection).

Messages arriving in the Azure IoT Hub Explorer

For more information about how to use the Azure IoT Explorer, take a look into the official documentation

Option 2: using mqtt-bridge

Currently specifying the username, which is required for Azure IoT Hub, is still in development. See also GitHub issue #569 for more information

Advantages:

• most reliable solution

Disadvantages:

• requires slightly more time than option 1

In the setup script or in the Helm chart adjust the following values with the corresponding certificates:

• mqttBridgeCACert
• mqttBridgeCert
• mqttBridgePrivkey

As mqttBridgeCACert you need to use the Baltimore CyberTrust Root

Furthermore, please set username, host, port, etc. accordingly

If you do not know where to get the certificates or the connection parameters, please go back to Prerequisites

What to do now?

Now we send the data to Azure IoT Hub. But what to do now? Here are some ideas:

1. You could install the server component of the United Manufacturing Hub factorycube-server on Azure Kubernetes Service. You would then be able to immediatly start creating dashboard while leveraging Azure’s scalability
2. You could use Azure’s propertiary services to create a full IIoT infrastructure from scratch. However, this is time intensive as you need to build e.g., your own data model, so we only recommend it for large companies with their own IIoT team. A good next step to proceed from here would be the tutorial Add an IoT hub event source to your Azure Time Series Insight environment

4.2.2 - Setting up Azure IoT Hub

This tutorial is part of a larger series on how to integrate the United Manufacturing Hub into the Azure stack.

What is Azure IoT Hub?

You can find the official description on the Microsoft Website. Speaking in IT terms it is basically a message broker (like a MQTT broker) with benefits.

When connecteing with the United Manufacturing Hub we will substitute the VerneMQ broker (see also architecture) in the United Manufacturing Hub with Azure IoT Hub.

Tutorial

Prerequisites

• Basic knowledge about IT / OT, Azure and the difference between symmetric and asymmetric encryption

Step 1: Create the necessary PKI (can be skipped if using symmetric encryption)

We recommend to skip this step if you are new to IT / Public-Key-Infrastructure and just want to see some data in Azure. We strongly advise to not use in production and instead follow the steps mentioned in this step.

There are two options / tutorials to get started:

1. our tutorial on setting up a PKI infrastructure (recommended, as it can be used in production as well). Replace the server URL with your Azure IoT Hub URL and ensure that you use the device IDs as MQTT client id
2. use the official Microsoft tutorial (only for testing)

Step 2: Create Azure IoT Hub

Follow the official guide. You can use the free plan for the beginning, which will be sufficient to test everything.

When using asymmetric encrpytion: when creating a new device select X.509 as authentification type.

What to do now?

We recommend you to connect the United Manufacturing Hub with Azure IoT Hub as a next step.

4.3 - General

4.3.1 - Setting up the PKI infrastructure

Prerequisites

• Installed easy-rsa (https://github.com/OpenVPN/easy-rsa)
• basic knowledge about public key infrastructure. We recommend reading our article first

This tutorial is assuming zou are using ubuntu and have installed easy-rsa using sudo apt-get install easyrsa

Initially setting up the infrastructure

Create a new directory and go into it, e.g.

mkdir ~/mqtt.umh.app/
cd ~/mqtt.umh.app/

Enable batch mode of easyrsa with export EASYRSA_BATCH=1

Setup basic PKI infrastructure with /usr/share/easy-rsa/easyrsa init-pki

Copy the default configuration file with cp /usr/share/easy-rsa/vars.example pki/vars and edit it to your liking (e.g. adjust EASYRSA_REQ_… and CA and cert validity)

Build the CA using export EASYRSA_REQ_CN=YOUR_CA_NAME && /usr/share/easy-rsa/easyrsa build-ca nopass. Replace YOUR_CA_NAME with a name for your certificate authority (CA), e.g., UMH CA

Create the server certificate by using the following commands (exchange mqtt.umh.app with your domain!):

/usr/share/easy-rsa/easyrsa gen-req mqtt.umh.app nopass
/usr/share/easy-rsa/easyrsa sign-req server mqtt.umh.app 

If you need to generate a certificate for an IP instead of a domain name, use the following command instead (exchange 0.0.0.0 with your IP):

/usr/share/easy-rsa/easyrsa --subject-alt-name="IP:0.0.0.0" gen-req 0.0.0.0 nopass
/usr/share/easy-rsa/easyrsa sign-req server 0.0.0.0

Copy the private key pki/private/mqtt.umh.app.key and the public certificate pki/issued/mqtt.umh.app.crt together with the root CA pki/ca.crt to the configuration of the MQTT broker.

Adding new clients

Create new clients with following commands (remember to change TESTING with the planned MQTT client id):

export EASYRSA_REQ_CN=TESTING && /usr/share/easy-rsa/easyrsa gen-req $EASYRSA_REQ_CN nopass && /usr/share/easy-rsa/easyrsa sign-req client $EASYRSA_REQ_CN nopass

4.3.2 - Edge networking

factorycube

The factorycube (featuring the RUT955) consists out of two separate networks:

1. internal
2. external

The internal network connects all locally connected machines, sensors and miniPCs with each other. The external network is “the connection to the internet”. The internal network can access the external network, but not the other way around, except specifically setting firewall rules (“port forwarding”).

Example components in internal network

• Laptop for setting up
• Router
• miniPC
• ifm Gateways
• Ethernet Cameras

Example components in external network

• Router (with its external IP)
• the “Internet” / server

4.3.3 - How to install an operating system from a USB-stick

Prerequisites

• you need a bootable USB-stick (see also flashing-a-operating-system-onto-a-usb-stick)

Steps

1. Plug the USB-stick into the device
2. Reboot
3. Press the button to go into the boot menu. This step is different for every hardware and is described in the hardware manual. If you do not want to look it up you could try smashing the following buttons during booting (the stuff before the operating system is loaded) and hope for the best: F1, F2, F11, F12, delete
4. Once you are in the boot menu, select to boot from the USB-stick

4.3.4 - How to connect with SSH

For Windows

For Windows we recommend MobaXTerm.

Get the free Version of MobaXTerm on https://mobaxterm.mobatek.net/download.html

MobaXTerm Session

After launching the program, open a new session by clicking on “Session” in the upper left corner.

Enter your IP, e.g. 192.168.1.XXX, in the Remote Host field. Select “Specify username”. The specific username is rancher.

Password

Enter the password and press enter. The default password of the auto setup will be rancher. You do not need to save the password, just click No.

Successfully logged in via SSH

For Linux

For Linux you can typically use the inbuilt commands to connect with a device via SSH. Connect using the following command:

ssh <username>@<IP>, e.g., ssh [email protected].

Connect via SSH

There will be a warning saying that the authenticity of the host can’t be established. Enter yes to continue with the connection.

Warning message: The authenticity of host 'xxx' can't be established.

Enter the password and press enter. The default password of the auto setup will be rancher.

Successfully logged in via SSH

4.3.5 - How to flash an operating system onto a USB-stick

Prerequisites

• You need a USB-stick (we recommend USB 3.0 for better speed)
• You need a OS image in the *.iso format. For k3OS you could choose for example this version

Steps

Download balenaEtcher: www.balena.io/etcher/

Insert USB-stick and open balenaEtcher

Select downloaded *.iso by clicking on "Flash from file" (the sceeen might look different based on your operating system)

Select the USB-stick by clicking on "Select target"

Select "Flash"

It will flash the image on the USB-stick

You are done!

These steps are also available as a YouTube tutorial from the user kilObit.

4.3.6 - Versioning in IT

In IT Semantic Versioning has established itself as the standard to describe versions. It consists out of the format MAJOR.MINOR.PATCH, e.g., 1.0.0.

MAJOR is incremented when making incompatible API changes.

MINOR is incremented when you add functionality

PATCH is incremented when you make bug fixes

If the version is followed by a ‘-’ sign, then it means it is a pre-release and not stable yet. Therefore, the latest stable version means the highest version available that is not a pre-release / has no ‘-’ sign.

More information can be found in the specification of Semantic Versioning 2.0.

4.4 - k3OS

4.4.1 - How to add additional SSH keys in k3OS

Prerequisites

• Edge device running k3OS
• SSH access to that device
• SSH / SFTP client
• Public and private key suited for SSH access

Tutorial

1. Access the edge device via SSH
2. Go to the folder /home/rancher/.ssh and edit the file authorized_keys
3. Add there your additional SSH key

4.4.2 - How to enable SSH password authentification in k3OS

DANGER: NOT RECOMMENDED FOR PRODUCTION! USE DEFAULT BEHAVIOR WITH CERTIFICATES INSTEAD

By default, k3OS allows SSH connections only using certificates. This is a much safer method than using passwords. However, we realized that most mechanical engineers and programmers are overwhelmed with the creation of a public key infrastructure. Therefore, it might make sense to enable password authentication in k3OS for development mode.

Prerequisites

• Edge device running k3OS
• Physical access to that device

Tutorial

1. Access the edge device via computer screen and keyboard and login with username rancher and rancher
2. Set the value PasswordAuthentication in the file /etc/ssh/sshd_config to yes and restart the service sshd. You can use the following command:

sudo vim /etc/ssh/sshd_config -c "%s/PasswordAuthentication  no/PasswordAuthentication  yes/g | write | quit" && sudo service sshd restart

4.4.3 - How to fix certificate not yet valid issues

Issue 1

While executing the command export VERIFY_CHECKSUM=false && curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 && chmod 700 get_helm.sh && ./get_helm.sh on k3OS you might get a error message like this:

curl: (60) SSL certificate problem: certificate is not yet valid
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Checking the time with date results in a timestamp from 2017. There are two possible solutions.

Possible solution 1: configure NTP

The time is not configured properly. It can happen that your NTP server is blocked (especially if you are inside a university network).

You can verify that by entering sudo ntpd -d -q -n -p 0.de.pool.ntp.org. If you get a result like this, then it is definitely blocked:

ntpd: '0.de.pool.ntp.org' is 62.141.38.38
ntpd: sending query to 62.141.38.38
Alarm clock

We recommend using the NTP server from the local university or ask your system administrator. For the RWTH Aachen / FH Aachen you can use sudo ntpd -d -q -n -p ntp1.rwth-aachen.de as specified here

Possible solution 2: set hardware clock via BIOS

Go into the BIOS and set the hardware clock of the device manually.

Issue 2

k3os reports errors, due hardware date being in the past. (ex: 01.01.2017). Example: during startup the k3s certificates are generated, however, it is still using the hardware time. Even after setting the time manually with NTP it wont let you connect with k3s as the certificates created during startup are not not valid anymore. Setting the time is not persisted during reboots.

Steps to Reproduce

1. Install k3os, without updating BIOS clock
2. Install UMH
3. helm will fail on Install factorycube-server step, due to outdated certificates.

Possible solution

Load cloudinit with added ntp_servers on OS install. You can use the one at https://www.umh.app/development.yaml

Be careful: you need to host it on a HTTP server (not HTTPS) as you would get other certificate issues while fetching it.

4.4.4 - How to install k3OS

Prerequisites

• edge device with keyboard and computer screen successfully booted from a USB-stick (see also Installation)
• you should see on the computer screen the screen below (it will automatically continue after 10 seconds with the installation, so do not worry if you only see it for some seconds)

boot menu of k3OS

Tutorial

Wait until k3OS is fully started. You should see the screen below:

k3OS installer fully booted

Enter rancher and press enter to login.

k3OS installer fully booted with rancher as username

You should now be logged in.

Pro tip: Execute lsblk and identify your hard drive (e.g. by the size). It will prevent playing “Russian Roulette” on a later step.

Now type in sudo k3os install to start the installation process.

entered `sudo k3os install`

You are now prompted to select what you want to install. Select 1 and press enter or just press enter (the stuff in brackets [] is the default configuration if you do not specify anything and just press enter).

Install to disk

At this step you may be asked to select your hard drive. One of the devices sda or sdb will be your hard drive and the other will be the USB stick you booted from.

If you don not know what your hard drive is, you will have to play “Russian roulette” and choose a device. If you find out later that you installed the hard drive on the USB stick by mistake, repeat the installation process and use the other device.

After that select y when you get asked for a cloud-init file

Configure system with cloud-init file

Now enter the URL of your cloud-init file, e.g. the one mentioned in the Installation guide.

Attention: The default keyboard of k3Os is American. So just google the American keyboard and see where you can find the letters and characters used.

Press enter to continue.

Specify the cloud-init file

example (do not use this URL)

Confirm with y and press enter.

Confirm installation with `y`

Confirm installation with `y`

If the installation fails with not beeing able to fetch the cloud-init file check the URL and the network configuration

If the installation fails with expired or untrusted certificates (curl: (60) SSL certificate problem: certificate is not yet valid or similar), check out this guide.

The device will then reboot. You might want to remove the USB-stick to prevent booting from the USB-stick again.

If the following screen appears you did everything correct and k3OS was successfully installed.

4.5 - Node-RED

4.5.1 - Fixing broken Node-RED flows

Prerequisites

• Node-RED in a crash loop because of one misconfigured node especially azure-iot-hub and python3-function)
• Node-RED installed as part of the United Manufacturing Hub (either as factorycube-edge or factorycube-server)

Tutorial

The solution is to boot Node-RED in safe mode by changing the environment variable NODE_RED_ENABLE_SAFE_MODE to true.

After 0.6.1

1. Open Lens and connect to the cluster (you should know how to do it if you followed the Getting Started guide)
2. Select the namespace, where Node-RED is in (factorycube-edge or factorycube-server)
3. Go to Apps –> Releases and select your Releases
4. Search for NODE_RED_ENABLE_SAFE_MODE and change the value from false to true
5. Save

Using this method you might encounter an error message on the top right while saving RELEASE not found in Lens. We recommend following this guide to fix the RELEASE not found message while updating a RELEASE in Lens

Before 0.6.1

1. Open Lens and connect to the cluster (you should know how to do it if you followed the Getting Started guide)
2. Select the namespace, where Node-RED is in (factorycube-edge or factorycube-server)
3. Select the StatefulSet Node-RED. A popup should appear on the right side.
4. Press the edit button
   

   Press the edit button / pen symbol on top right of the screen

5. Find the line

env:
    - name: TZ
      value: Berlin/Europe

and change it to this:

env:
    - name: TZ
      value: Berlin/Europe
    - name: NODE_RED_ENABLE_SAFE_MODE
      value: "true"

6. Check whether it says "true" and not true (use quotation marks!)
7. Press Save
8. Terminate the pod manually if necessary (or if you are impatient)
9. Node-RED should now start in safe mode. This means that it will boot, but will not execute any flows.
10. Do your changes, fix the Nodes
11. Do steps 1 - 5, but now set NODE_RED_ENABLE_SAFE_MODE to false

4.6 - Tulip

4.6.1 - How to connect the United Manufacturing Hub to Tulip

Prerequisites

• An active Tulip subscription.
• The UMH platform installed and configured.
• Available data extracted from sensors, industrial controllers or other shopfloor IT systems and available in the MQTT broker or Apache Kafka.

About Tulip

Tulip is a frontline operations platform, which is deployed in manufacturing and logistic environments, bridging the interaction between people, machines, devices, and systems.

Tulip’s most widely documented use case (in different and extensive channels such as “Tulip Libraries”, “Tulip University” and “Tulip Community”) focusses on their hardware “I/O Gateway” and “Edge IO” developed for the deployment of work instructions running in the Tulip Cloud. These work instructions can be authorized with additional hardware that connects Tulip approved devices via USB port and sensors to GPIO ports. The main function of the gateway is to send the data to the Tulip app.

Tulip Edge IO with other devices. Source: https://tulip.co/products/edge-io/

Within the Tulip platform, users can create applications using a a no-code interface. The operations are triggered by various sensors and forwarded by the hardware to the Tulip Cloud.

Work instructions in an assembly cell

In addition to apps, users can create “Tulip Analyses” to monitor KPIs and trends and provide insights to the production team. Interactive dashboards can also be created to monitor KPIs of interest.

The United Manufacturing Hub (UMH) provides the necessary IT/OT infrastructure not only to connect real-time manufacturing data to Tulip, but to also share it with other solutions and applications. For example, data coming from sensors, barcode readers, cameras and industrial controllers can now be used for real-time stream or batch processing by other use cases such as predictive maintenance, quality management systems, track & trace / digital shadow, and supervisor dashboards. These use cases can then run different clouds such as AWS or Azure.

Additionally, this allows for adding data sources to Tulip that are yet to be officially supported (e.g., IO-Link sensors). To access local data sources, a Tulip Connector Host and/or opening ports is no longer required, which should align with the IT security department’s requirements.

Approach

In this tutorial we will choose the “Tulip Machine Attributes API” approach. Other method is using the Tulip Connector Host. The approach chosen here has the advantage that it does not requiring opening ports and is therefore more easily integrated into enterprise IT.

Step-by-step tutorial

Perform the following steps as a Tulip administrator. First, in the Tulip platform, go to your profile icon and click Settings. Then select the Bot tab and create one. For this project, only data should be read and displayed in the Tulip environment, therefore the scope must be set to “read-only”.

Tulip Bot showing the different scopes.

The API Key, Secret and Auth Header are displayed. Please store this information in a safe place.

API Key, secret and authentication header is shown.

Then you will need to link this Machine API to a Machine.

Create a Machine Type

Navigate to the tab “Shopfloor” –> “Machines” then change to the tab “Machine Types”. Here, click in the button “Create type”. For now, only change the name of the machine type.

Creating a Machine Type.

Create a machine

Go to the Machine Library tab and click the Create Machine button. Assign a name for your machine. Then select Tulip API as the data source. The Tulip interface will ask you to assign your machine to a station. If multiple stations use the data, we recommend selecting “None”. Finally, select the machine type you just created.

Creating a Machine.

Add Machine Attribute

Return to the Machine Type tab and navigate to the Machine Type you previously created. Click the edit button and scroll down and head to “Add Machine Attribute”.

Adding a Machine Attribute.

Here you are able to create new machine attributes or assign ones you created and used before.

Assign a name and a type to the Machine Attribute.

Having performed the previous steps, you are now able to to find the field mappings for each Machine Attribute. Go to the tab “Shopfloor/Machines”, select the machine name of interest, and click in the Configuration tab. There you’ll discover field mappings for each Machine Attributes. The “Attibute ID” and “Machine ID” will be used for setting up the connection in Node-RED.

Field mappings

Having performed the previous steps, the Tulip configuration is finalized. The next part of the article describes the configuration in Node-RED that is necessary for connecting the data acquisition layer performed in Node-RED with the Tulip platform. 

Connecting to the Tulip Machines Attributes API using Node-RED

In the manufacturer’s environment, UMH installed an electrical cabinet (E-rack) with an Industrial PC (IPC). The IPC was connected to the industrial controller of the machine and reads several data tags such as power time, hydraulic pump time and power consumption, with a flow in Node-RED (see also the blog article mentioned at the beginning of the article)

Automatic saw with E-rack.

To connect to the Tulip Machine Attributes API, two nodes were used, namely a function node and a HTTP response.

Function and HTTP response nodes used.

For the function node, the message must contain the “Attribute ID” and “Machine ID” corresponding to the data tag being acquired from the industrial controller.

Content in the function node.

For the HTTP response node, a configuration with the Tulip platform is required. This information was acquired while creating the bot in the Tulip environment.

HTTP response node configuration.

Having set everything up, the connection between the Node-RED flow and the Tulip platform is now established. On the next and final part of the blog, it will be shown how to use the data for developing it in the Tulip app environment.

Using the data in Tulip app

Data coming from industrial controllers or sensors can be connected, leveraging the Machine Attribute created as any other Tulip device. For this, please navigate to the “Apps” tab and create a new app or select one you already created.

In the App editor, select to create a new trigger.

The trigger menu will appear, and by selecting: When – Machine – Specific Machine The Machine that was created in the previous steps will appear. Select it and in the outputs, select the corresponding Machine attributes. Select the one that concerns you and by doing so you can leverage the data extracted from the machine.

Digital Work instruction app with Machine Attribute API connectivity.

Summary

This approach makes it possible to connect additional data sources to Tulip and combine Tulip with a Unified Namespace. It also reduces implementation time by eliminating the need to open ports in the corporate firewall.

Interested? Then check out also the corresponding blog article

4.7 - United Manufacturing Hub

4.7.1 - Certified devices

4.7.1.1 - How to use machineconnect

Purpose

machineconnect is our certified device used for the connection of PLCs and installed in the switch cabinet of the machine.

The idea behind machineconnect is to protect the PLC and all components with an additional firewall. Therefore, it is not accessible from outside of machineconnect except explicitly configured in the firewall.

Features

• Industrial Edge Computer
  • With DIN rail mounting and 24V
  • Vibration resistent according to IEC 60068-2-27, IEC 60068-2-64/ MIL-STD-810, UNECE Reg.10 E-Mark, EN50155
  • Increased temperature range (-25°C ~ 70°C)
• Open source core installed and provisioned according to customer needs (e.g. MQTT certificates) in production mode (using k3OS)
• Additional security layer for your PLC by using OPNsense (incl. Firewall, Intrusion Detection, VPN)
• 10 years of remote VPN access via our servers included

Physical installation

1. Attach wall mounting brackets to the chassis
2. Attach DIN Rail mounting brackets to the chassis
3. Clip system to the DIN Rail
4. Connect with 24V power supply
5. Connect Ethernet 1 with WAN / Internet
6. Connect Ethernet 3 with local switch (if existing). This connection will be called from now on “LAN”.
7. (optional, see connection to PLC. If skipped please connect the PLC to Ethernet 3) Connect Ethernet 2 with PLC. This connection will be called from now on “PLC network”.

Verify the installation by turning on the power supply and checking whether all Ethernet LEDs are blinking.

Connection to the PLC

There are two options to connect the PLC. We strongly recommend Option 1, but in some cases (PLC has fixed IP and is communicating with engine controllers or HMI and you cannot change the IP adresses there) you need to go for option 2.

Option 1: The PLC gets the IP via DHCP (recommended)

1. Configure the PLC to retrieve the IP via DHCP
2. Configure OPNsense to give out the same IP for the MAC-address of the PLC for LAN. Go to Services –> DHCPv4 –> LAN and add the PLC under “DHCP static mappings for this device”

Option 2: The PLC has a static IP, which cannot be changed

New method

This method should be quicker

1. Add a new interface for the PLC network
2. 
3. Add a new route with the target being the PLC IP and as gateway the automatically created gateway for the PLC (will not be shown by default, need to enter PLC_GW to be shown)
   1. 
   2. 
4. Make sure to activate the gateway:
   1. 
5. Change NAT to “Hybrid outbound NAT rule generation” and add a NAT for PLC, Source the LAN network, Destination the PLC
   1. 
   2. 
6. Test if the PLC is reachable
   1. 
7. That’s it

Old method

1. Adding a new interface for the PLC network, e.g. “S7”.
   
2. Adding a new gateway (see screenshot. Assuming 192.168.1.150 is the IP of the PLC and the above created interface is called “S7”)
   
3. Adding a new route (see screenshot and assumptions of step 2)
   
4. Changing NAT to “Manual outbound NAT rule generation” (see screenshot and assumptions of step 2)
   
5. Add firewall rule to the PLC interface (see screenshot and assumptions of step 2)
   
6. Add firewall rule to LAN allowing interaction between LAN network and PLC network
   

If you are struggling with these steps and have bought a machineconnect from us, feel free to contact us!

Next steps

After doing the physical setup and connecting the PLC you can continue with part 3 of the getting started guide.

4.7.2 - Setting up the documentation

1. Clone the repo
2. Go to /docs and execute git submodule update --init --recursive to download all submodules
3. git init && git add . && git commit -m "test" (yes it is quite stupid, but it works)
4. Startup the development server by using sudo docker-compose up --build

4.7.3 - Working with the system practically

Please ensure that you have read the safety information and manuals before proceeding! Failure to do so can result in damages to the product or serious injuries.

Create a sensor plan

Please create before installing anything a sensor plan. Prior to installing take the layout of your selected line and add:

• Electricity sockets
• Internet connections (bring your own)
• Your planned sensors
• The position of the factorycube
• Your planned cabling

Discuss and install

Setup a meeting with your line engineers and discuss your plan. Then install everything according to plan. Ensure, that all sensors and cables are mounted tightly and that they do not interfere with the production process.

Supply your factorycube with power and turn it on

Plug in the power cable to turn the factorycube on. After a few seconds the ia: factorycube should be lit up.

Connect your factorycube to the internet

If you want to use the cloud dashboard, you must first connect the factorycube to the internet.

You need:

• your credentials
• an Ethernet cable (provided)
• a laptop which is not connected to any VPN

For a network overview of the Factorycube, click here

Instructions to login

Connect the factorycube with your computer via an ethernet cable using the IO-Link port (not Internettest_ on the factorycube.

Open the following website on your browser: http://172.16.x.2 (The X stands for the last number(s) of the serial number. e.g. 2019_0103 -> x=3 or 2019_0111 -> x=11)

Enter your credentials according to the information in the customer area. The username is always “admin”

3 ways to connect to the internet: WiFi, 3G/4G or Ethernet

Further information on how to connect the factorycube with the internet can be found in the official router manual

Instructions to setup WiFi

• Select “Network” → “Wireless/Wlan”. If necessary remove the old Wireless station point
• Click on “Add” next to wireless station mode
• Click on “Start Scan” next to wireless station mode
• Click on the network of your choice
• “join network”
• Afterwards enter your credentials and confirm

The computer should now be connected to the internet.

Instructions to setup 3G/4G

• Insert the SIM-card (if a SIM-card is already provided in the ia: factorycube, skip this step)
• For the installation of a SIM card please contact our experts
• Select “Network” → “Mobile”
• Adjust the settings under the “General” tab as follows:
  
• Save your settings

The computer should now be connected to the internet.

Instructions to set up connection via Ethernet

• Plug the Ethernet cable into the device’s “Internet” port and the other side into the network access port
• Select “Network” –> “WAN”
• Select as wired as your main WAN
  
• Click save

The computer should now be connected to the internet. You can now the entire United Manufacturing Hub Edge Stack. For more information, take a look in the getting started with edge devices.

Outro

Closely monitor the data and verify over the entire duration of the next days, whether the data is plausible. Things that can go wrong here:

• Sensors not mounted properly and not calibrated anymore
• The operators are using the line different from what was discussed before (e. g., doing a changeover and removing the sensors)

4.7.4 - How to setup a development network

4.7.5 - How to update the stack / helm chart

Prerequisites

none

Tutorial

1. Go to the folder deployment/factorycube-server or deployment/factorycube-edge
2. Execute helm upgrade factorycube-server . --values "YOUR VALUES FILE" --kubeconfig /etc/rancher/k3s/k3s.yaml -n YOUR_NAMESPACE

This is basically your installation command, but you exchange install with upgrade. You need to change “YOUR VALUES FILE” with the path of your values.yaml, e.g. /home/rancher/united-manufacturing-hub/deployment/factorycube-server/values.yaml and you need to adjust YOUR_NAMESPACE with the correct namespace name. If you did not specify any namespace during the installation you can use the namespace default. If you are using factorycube-edge instead of factorycube-server you need to adjust that as well.

4.7.6 - How to work with MinIO

Please also take a look on our guide on how to use it in production!

Default settings / development setup

By default MinIO is exposed outside of the Kubernetes cluster using a LoadBalancer. The default credentials are minio:minio123 for the console. When using minikube you can access the LoadBalancer service using minikube tunnel and then accessing the external IP using the selected port.

4.7.7 - Working with Kubernetes, Helm and Lens

How does Kubernetes and Helm work?

Where is my Dockerfile?

One might ask.

Where is my docker-compose.yaml?

In this chapter, we want to explain how you can configure the Kubernetes cluster and therefore the United Manufacturing Hub.

Informal and not exact explaination (from the right to the left): Docker containers are called in Kubernetes lanugage Pods.

There is also other stuff going on in Kubernetes like secret management or ingresses.

You define the final state (“I want to have one application based on this Docker image with this environment variables ….") and Kubernetes will take care of the rest using its magic. This could be seen equal to a docker-compose.yaml

The final state is defined using Kubernetes object descriptions.

Example for a Kubernetes object description:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

You could now create and maintain all of them manually, which is what some companies do.

To do that automatically, you can use Helm.

Helm will take a values.yaml and will automatically create the corresponding Kubernetes object descriptions.

Example for a Helm template values:

### mqttbridge ###

mqttbridge:
  enabled: true
  image: unitedmanufacturinghub/mqtt-bridge
  storageRequest: 1Gi

### barcodereader ###

barcodereader:
  enabled: true
  image: unitedmanufacturinghub/barcodereader
  customUSBName: "Datalogic ADC, Inc. Handheld Barcode Scanner"
  brokerURL: "factorycube-edge-emqxedge-headless"  # do not change, points to emqxedge i
  brokerPort: 1883  # do not change
  customerID: "raw"
  location: "barcodereader"
  machineID: "barcodereader"

There is nothing comparable to Helm in the Docker world.

For this to work, one needs to import a so-called Helm chart first. This is nothing else than a Kubernetes object description with some variables.

Example for a Kubernetes object description template (for Helm):

{{- if .Values.mqttbridge.enabled -}}
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: {{ include "factorycube-edge.fullname" . }}-mqttbridge
  labels:
    {{- include "factorycube-edge.labels.mqttbridge" . | nindent 4 }}
spec:
  serviceName: {{ include "factorycube-edge.fullname" . }}-mqttbridge
  replicas: 1
  selector:
    matchLabels:
      {{- include "factorycube-edge.labels.mqttbridge" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "factorycube-edge.labels.mqttbridge" . | nindent 8 }}
    spec:
      containers:
      - name: {{ include "factorycube-edge.fullname" . }}-mqttbridge
        {{- if .Values.mqttbridge.tag }}
        image: {{ .Values.mqttbridge.image }}:{{ .Values.mqttbridge.tag }}
        {{- else }}
        image: {{ .Values.mqttbridge.image }}:{{ .Chart.AppVersion }}
        {{- end }}

A Helm chart can either be stored in a folder (e.g., when you clone the United Manufacturing Hub repository) or in a Helm repository (which we use in our default installation script).

You might be tempted to just clone the repository and change stuff in the Kubernetes object description template (in the subfolder templates). We consider this a bad idea as these changes are now hard to track and to replicate somewhere else. Furthermore, when updating the Helm chart to a newer version of the United Manufacturing Hub these changes will be lost and overwritten.

In Lens you can now change the underlying Kubernetes object descriptions by e.g., clicking on a Deployment and then selecting edit. However, these changes will be reverted once you apply the Helm chart again.

Therefore, we recommend in production setups to only adjust the values.yaml in the Helm chart.

Changing the configuration / updating values.yaml

There are two ways to change the values.yaml:

using Lens GUI

Note: if you encounter the issue “path not found”, please go to the troubleshooting section further down.

For this open Lens, select Apps on the left side and then click on Releases. A Release is a deployed version of a chart. Now you can not only change the values.yaml, but also update the Helm chart to a newer version.

using CLI / kubectl in Lens

To override single entries in values.yaml you can use the --set command, for example like this: helm upgrade factorycube-edge . --namespace factorycube-edge --set nodered.env.NODE_RED_ENABLE_SAFE_MODE=true

Troubleshooting

RELEASE (factorycube-edge or factorycube-server) not found when updating the Helm chart

There are some modifications needed to be done when using Lens with Charts that are in a Repository. Otherwise, you might get a “RELEASE not found” message when changing the values in Lens.

The reason for this is, that Lens cannot find the source of the Helm chart. Although the Helm repository and the chart is installed on the Kubernetes server / edge device this does not mean that Lens can access it. You need to add the helm repository with the same name to your computer as well.

So execute the following commands on your computer running Lens (on Windows in combination with WSL2):

1. Install Helm

export VERIFY_CHECKSUM=false && curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3  && chmod 700 get_helm.sh && ./get_helm.sh

2. Add the repository

helm repo add united-manufacturing-hub https://repo.umh.app

3. Get repository updates

helm repo update

4. Restart Lens (make sure to also close it in the system tray)

5 - Developers

This repository contains multiple folders and sub-projects:

• /golang contains software developed in Go, especially factoryinsight and mqtt-to-postgresql and their corresponding tests (-environments)
• /deployment contains all deployment related files for the server and the factorycube, e.g. based on Kubernetes or Docker, sorted in seperate folders
• /sensorconnect contains sensorconnect
• /barcodereader contains barcodereader
• /docs contains the entire documentation and API specifications for all components including all information to buy, assemble and setup the hardware

5.1 - united-manufacturing-hub

factoryinsight

factoryinsight is an open source REST API written in Golang to fetch manufacturing data from a timescaleDB database and calculate various manufacturing KPIs before delivering it to a user visualization, e.g. [Grafana] or [PowerBI].

Features:

• OEE (Overall Equipment Effectiveness), including various options to investigate OEE losses (e.g. analysis over time, microstop analytics, changeover deep-dives, etc.)
• Various options to investigate OEE losses further, for example stop analysis over time, microstop analytics, paretos, changeover deep-dives or stop histograms
• Scalable, microservice oriented approach for Plug-and-Play usage in Kubernetes or behind load balancers (including health checks and monitoring)
• Compatible with important automation standards, e.g. Weihenstephaner Standards 09.01 (for filling), Omron PackML (for packaging/filling), EUROMAP 84.1 (for plastic), OPC 30060 (for tobacco machines) and VDMA 40502 (for CNC machines)

The openapi documentation can be found here

mqtt-to-postgresql

the tool to store incoming MQTT messages to the postgres / timescaleDB database

Technical information and usage can be found in the documentation for mqtt-to-postgresql

grafana-auth

Proxies request from grafana to various backend services, while authenticating the grafana user. Technical information and usage can be found in the documentation for grafana-proxy

grafana-plugins

Contains our grafana datasource plugin and our input panel

sensorconnect

This tool automatically finds connected ifm gateways (e.g. the AL1350 or AL1352), extracts all relevant data and pushes the data to a MQTT broker. Technical information and usage can be found in the documentation for sensorconnect

cameraconnect

This tool automatically identifies connected cameras network-wide which support the GenICam standard and makes them utilizable. Each camera requires its own container. The camera acquisition can be triggered via MQTT. The resulting image data gets pushed to the MQTT broker. Technical information and usage can be found in the documentation for cameraconnect

barcodereader

This tool automatically detected connected USB barcode scanners and send the data to a MQTT broker. Technical information and usage can be found in the documentation for barcodereader

mqtt-bridge

This tool acts as an MQTT bridge to handle bad internet connections. Messages are stored in a persistent queue on disk. This allows using the united-manufacturing-hub in remote environments with bad internet connections. It will even survive restarts (e.g. internet failure and then 1h later power failure). We developed it as we’ve tested multiple MQTT brokers and their bridge functionalities (date of testing: 2021-03-15) and could not find a proper solution:

• emqx causes for internet blackouts longer than 3-4 min a DoS attack on our server (messages are send in endless loop, see https://github.com/emqx/emqx-bridge-mqtt/issues/81)
• VerneMQ only stored 10-20% of the data
• mosquitto was working very unreliable

nodered

This tool is used to connect PLC and to process data. See also Getting Started. Or take a look into the official documentation

5.1.1 - grafana-plugins

umh-datasource

Translates REST calls to factoryinsight into grafana datapoints Technical information and usage can be found in the documentation for umh-datasource

umh-factoryinput-panel

Allows creation of REST requests to the factoryinput service Technical information and usage can be found in the documentation for umh-factoryinput-panel

5.1.1.1 - factoryinput-panel

This microservice is still in development and is not considered stable for production use.

Getting started

UMH Factoryinput Panel allows to easily execute MQTT messages inside the UMH stack from the Grafana Panel.

Requirements

• A united manufacturing hub stack
• External IP or URL of the grafana-proxy server.
  • In most cases it is the same IP as your Grafana dashboard

Installation

If you have installed the UMH-Stack as described in our quick start Tutorial, then this plugin is already installed on your Grafana installation

If you want to develop this Panel further, please follow the instructions below

Build from source

0. Goto united-manufacturing-hub/grafana-plugins/umh-factoryinput-panel

1. Install dependencies

yarn install

2. Build plugin in development mode or run in watch mode

yarn dev

3. Build plugin in production mode (not recommended due to Issue 32336)

yarn build

4. Move the resulting dist folder into your grafana plugins directory

• Windows: C:\Program Files\GrafanaLabs\grafana\data\plugins
• Linux: /var/lib/grafana/plugins

5. Rename the folder to umh-factoryinput-panel

6. Enable the enable development mode to load unsigned plugins

7. Restart your grafana service

Usage

Prerequisites

1. Open your Grafana instance
2. Log in
3. Open your Profile and check if your organization name inside Grafana matches the rest of your UMH stack

Creating a new Panel

1. Create a new Dashboard or edit an existing one
2. Click “Add an empty panel”
3. On the right sidebar switch the Visualization to “Button Panel”
4. Fill out the fields inside “REST Integration”
   1. URL
      • http://{URL to your grafana-proxy}/api/v1/factoryinput/
      • Example:
        • http://172.21.9.195:2096/api/v1/factoryinput/
   2. Location
      • Location of your Asset
   3. Asset
      • Name of the Asset
   4. Value
      • MQTT prefix
        • Example prefixes:
          • count
          • addShift
          • modifyShift
   5. Customer
      • Your organization name
   6. Payload
      • JSON encoded payload to send as MQTT message payload
5. Modify any additional options are you like
6. When you are finished customizing, click on “Apply”

Example Panel

Notes

1. Clicking the button will immediately send the MQTT message, through our HTTP->MQTT stack. Please don’t send queries modifying date you would later need !

Common pitfalls

• Pressing the button just changes the cog to an warning sign
  1. Open your network inspector view (Ctrl+Shift+I on Chrome)
  2. Press the button again
  3. If no request appears, then you haven’t filled out all required fields
  4. Your request shows:
     • 403
       • Make sure the customer field is set to your grafana organization name
     • 400
       • Your request was incorrectly formatted
       • Check that the URL is in the format specified above
       • Check if your payload contains valid JSON
         • You can validate your payload here
       • Check that the Value field matches a valid MQTT command

Technical information

Below you will find a schematic of this flow, through our stack

License

• Original Work: cloudspout-button-panel (MIT)
• Icons made by Pixel perfect from www.flaticon.com
• ButtonPayloadEditor highly influenced from gapitio/gapit-htmlgraphics-panel.

5.1.1.2 - umh-datasource

What is United Manufacturing Hub Datasource?

UMH Datasource provides an Grafana 8.X compatible plugin, allowing easy data extraction from the UMH factoryinsight microservice.

Installation

Build from source

0. Clone the datasource repo [email protected]:united-manufacturing-hub/united-manufacturing-hub-datasource.git

1. Install dependencies

yarn install

2. Build plugin in development mode or run in watch mode

yarn dev

3. Build plugin in production mode (not recommended due to Issue 32336)

yarn build

4. Move the resulting dist folder into your grafana plugins directory

• Windows: C:\Program Files\GrafanaLabs\grafana\data\plugins
• Linux: /var/lib/grafana/plugins

5. Rename the folder to umh-datasource

6. You need to enable development mode to load unsigned plugins

7. Restart your grafana service

From Grafana’s plugin store

TODO

Usage

1. Open Grafana and login

2. Open umh-datasource’s settings

3. Configure your customer name & API Key (automatically configured in Helm deployment)

4. Configure your server url:

   URL: URL/IP:Port of grafanaproxy

   http://{URL}/api/v1/factoryinsight/

   e.g:

   http://172.21.9.195:2096/api/v1/factoryinsight/

5. Click “Save & Test”

5.1.2 - barcodereader

Getting started

Here is a quick tutorial on how to start up a basic configuration / a basic docker-compose stack, so that you can develop.

Go to the root folder of the project and execute the following command:

sudo docker build -f deployment/barcodereader/Dockerfile -t barcodereader:latest . && sudo docker run --privileged -e "DEBUG_ENABLED=True" -v '/dev:/dev' barcodereader:latest 

All connected devices will be shown, the used device is marked with “Found xyz”. After every scan the MQTT message will be printed.

Environment variables

This chapter explains all used environment variables.

DEBUG_ENABLED

Description: Deactivates MQTT and only prints the barcodes to stdout

Type: bool

Possible values: true, false

Example value: true

CUSTOM_USB_NAME

Description: If your barcodereader is not in the supported list of devices, you must specify the name of the USB device here

Type: string

Possible values: all

Example value: Datalogic ADC, Inc. Handheld Barcode Scanner

MQTT_CLIENT_ID

Description: The MQTT client id to connect with the MQTT broker

Type: string

Possible values: all

Example value: weaving_barcodereader

BROKER_URL

Description: The MQTT broker URL

Type: string