The United Manufacturing System is an open source solution for extracting and analyzing data from manufacturing plants and sensors. The Hub includes both software and hardware components to enable the retrofit of productions plants by plug-and-play as well as to integrate existing machine PLCs and IT systems. The result is an end-to-end solution for various questions in manufacturing such as the optimization of production through OEE analysis, preventive maintenance through condition analysis and quality improvement through stop analysis.
Open. open-source (see LICENSE) and open and well-documented standard interfaces (MQTT, REST, etc.)
Scalable. Horizontal scaling incl. fault tolerance through Docker / Kubernetes / Helm. Edge devices can be quickly set up and configured in large numbers.
Flexible. Flexible deployment options, from public cloud (Azure, AWS, etc.) to on-premise server installations to Raspberry Pis, everything is possible. Free choice of programming language and systems to be connected through central message broker (MQTT).
Tailor-made for production. Pre-built apps for manufacturing. Use of established automation standards (OPC/UA, Modbus, etc.). Quick connection of production assets either by retrofit or by connecting to existing interfaces.
Community and support. Enterprise support and community for the whole package. Built exclusively on well-documented software components with a large community.
Information Security & Data Protection. Implementation of the central protection goals of information security. High confidentiality through e.g. end-to-end encryption, flexible provisioning options and principle of least privilege. High integrity through e.g. ACID databases and MQTT QoS 2 with TLS. High availability through e.g. use of Kubernetes and (for SaaS) a CDN.
1 - Getting Started
The guide is split into three parts
1.1 - 0. Understanding the technologies
Strongly recommended. This section gives you an introduction into the used technologies. A rough understanding of these technologies is fundamental for installing and working with the system. Additionally, this article provides further learning materials for certain technologies.
The materials presented below are usually teached in a 2-3 h workshop session on a live production shopfloor at the Digital Capability Center Aachen. You can find the outline further below.
Introduction into IT / OT
The goal of this chapter is to create a common ground on IT / OT technologies and review best-practices for using IIoT technologies. The target group are people coming from IT, OT and engineering.
Introduction
IIoT sits at the intersection of IT and OT.
History: IT & OT were typically seperate silos but are currently converging to IIoT
Operational Technology (OT)
OT connects own set of various technologies to create highly reliable and stable machines
OT is the hardware and software to manage, monitor and control industrial operations. Its tasks range from monitoring critical assets to controlling robots on the shopfloor. It basically keeps machines and factories running and producing the required product.
Typical responsibilities:
Monitoring processes to ensure best product quality
Controlling machine parameters
Automation of mechanical and controlling processes
The concepts of OT are close to eletronics and with a strong focus on human and machine safety
Concept
Description
Examples
Process control
Designing a stable process which create the desired output with continuously changing inputs External and internal factors influence the process but are not allowed to change the result
Controlling refrigerator based on internal temperature
Sensor technology
Using various sensors types to measure pressure, force, temperature, velocity, etc. Converting sensor signals to digital outputs, interpreting their signals and generating insights
Light barrier counting parts on a conveyor belt Vibration sensor for process control in CNC machining
Automation
Using hardware and software to automate repetitive or dangerous tasks Reducing reaction time and increasing speed to increase productivity
Robot assembling smartphones
Reliability and safety
Ensuring that neither humans nor machines are damaged in case of unforeseen events Regular checks, maintenance and certification of crucial assets
Emergency stop buttons for worker safety Regular maintenance to prevent machine breakdown
OT focuses on handling processes with highest possible safety for machines and operators
High importance in OT:
Category
Description
Reliability & Safety
Malfunction can result in extreme damages to human and property
Maintainability & standards
Machines typically run between 20-30 years, sometimes even 50+ years
Certifications
Legally required certifications for safety and optional certificates for reliability
Of lesser importance:
Category
Description
User experience
The operator will be trained anyway, therefore intuitive User Interfaces (UI) are not required
Quick development cycles, e.g., agile
Can result in missing out important safety elements and damage workers or machines
IT Security
20+ year old machines are not designed with cyber security in mind
Nobody wants to build a nuclear reactor using agile “move fast, break things” principles
Typical device architecture and situation for the OT
Typical device architecture and situation for the OT.
1 Programmable logic controller;
2 Human Machine Interface;
3 Supervisory Control and Data Acquisition
The Programmable Logic Controller (PLC) is the heart of every modern machine, which stores and runs the program. It is a PC with industrial standards and does not require a monitor, keyboard or other devices to function properly. It collects sensor data and calculates complex algorithms to control actuators.
Background:
Very old machines use only relays (electric switches) to control actuators and sensors
PLCs were introduced due to being more reliable and flexible than electrical parts
The logic of simple switches is still very present in the OT space (programming)
Programming languages:
The various suppliers like Siemens, Rockwell, Bosch etc. offer different programming languages
PLCs can be programmed with graphical elements or with code
Machine vendor programs are not always openly accessible and do not allow changes (loss of warranty)
Communication protocols:
PLC manufacturers have different communication protocols and functional standards which limits interoperability
Newer protocols like Profinet or Ethernet/IP are easy to connect to an IT network (if open interface). Others like Profibus require additional hardware and implementation effort
Fundamentals 2: PLCs & PLC programming
Fundamentals 3: Process control using PLCs
Information Technology (IT)
IT connects millions of devices and manages their data flows
IT is the hardware and software to connect thousands of devices in a network and manage their exchange of information. The purpose is to enable data storage and its usage for business and operations. Tasks range from connecting simple telephones to managing complex global networks.
Typical responsibilities:
Setting up phones, PCs, printers and other office hardware
Monitoring devices and networks for security breaches
Maintaining local servers
Configuration of business systems e.g. ERP/SAP
Updating devices to ensure IT security
Setting up local networks and WiFi
Implementing business solutions like automation and
The concepts of IT are focusing on digital data and networks
Concept
Description
Examples
Data storage and analytics
Data has to be managed and stored in a manner which allows quick access and driving insights to improve business KPIs Terabytes of data without contextualization does not have any business value
Aggregating sales data and calculating KPIs every quarter
Device Management
Remote device management allows the monitoring and updating of devices Blocking and updating devices to reduce security risks and malicious actions
Updating and restarting computer remotely
Network security
Policies, processes and practices like firewalls and two-factor authentication adopted to prevent cyber attacks Limiting risk by limiting the number of accesses and rights of users e.g. not all users are admins, users are only granted access when it is required for their work etc.
Limiting internet access to specific services
Scalability
New software and functionality can be installed and rolled out only with a few clicks Update to existing solutions does not always require new hardware like in OT
Handing out Microsoft Office to all employees
What is important in IT? What is not important?
High importance in IT:
Category
Description
Quick developmentcycles, e.g., agile
Good user experience is more important than a perfectly designed app
Scalability
Apps need to handle millions of users at the same time (e.g., Google, Netflix)
User experience
If something is unintuitive, people tend to not use it
Of lesser importance:
Category
Description
Reliability & Safety
Hardware is redundant, if one fails another can take over Consequences of hardware failures are smaller
Maintainability & standards
Standards are usually best-practices and might change over time. No hard-written norms.
Certifications
Therefore, certifications are not legally required
Nobody wants to build an app for years just so that the end-user removes it within 30 seconds
Fundamentals 1: Networking
Fundamentals 2: Cloud and Microservices
The term cloud refers to servers and the software running on them. These servers can be used to compute data e.g. process a customer order or simulate the weather and at the same time store it. This data can be accessed around the globe simultaneously with high-speed which enables a centralized “single source of truth”
Cloud products:
Cloud providers offer their capabilities on advanced analytics and machine learning to reduce time for insights generation (Platform as a Service - PaaS)
Storage and computational power can be booked flexibly and used freely
Out of the box applications running in the browser without installation
Microservices:
Small stand alone blocks running only small functions
Whenever one microservice block crashes the rest is unaffected (high stability)
One solutions can be designed out of multiple already available microservices
Scalability:
Microservice blocks can be flexibly turned on and off depending on the user requirements
Easy scalability allows customers to only pay what they use
Single solutions can be deployed and accesses globally without installation on each personal computer
Fundamentals 3: How microservices are built: Docker in 100 seconds
Fundamentals 4: How to orchestrate IT stacks: Kubernetes in 100 seconds
Fundamentals 5: Typical network setups in production facilities
Typical network setups in production facilities
Industrial Internet of Things (IIoT)
Whats it’s all about
Why is digital transformation relevant now?
Technology advancements have lowered barriers to industrial IoT to come down. The benefits of IIoT are real and sizable.
How can manufacturing organizations capture value at scale?
A digital transformation in manufacturing requires an orchestrated approach across the dimensions of business, organization and technology. A holistic framework focuses on full value capture through having a return-on-investment, capability building and technical IIoT ecosystem focus
Value created through digital transformation
Following a digital transformation approach cases show great impact in e.g., throughput, production efficiency, gross margin, quality across various industries.
A full digital transformation of manufacturing needs to consider business, technology and organization
Business: Impact Drive Solutions
Impact comes from a top-down prioritized portfolio of use cases to address highest value first
Digital transformations need to have a return-on-investment mindse
Organization: New way of working and dedicated approach to skills and capabilities
Digital transformations are multiyear, company-wide journeys requiring a central transformation engine and a value capture approach
Innovative digital-capability-building approaches allow the rapid upskilling of thousands of employee
Technology: Cloud enabled platforms and ecosystem (focus of the United Manufacturing Hub)
Form a comprehensive, secure, affordable and scalable technology infrastructure based on IoT platform architectures
Build and lead a focused ecosystem of technology partner
IIoT sits at the intersection of IT and OT
IIoT sits at the intersection of IT and OT.
Architecting for scale
Architecting for scale.
Best-practices 1: Avoid common traps in the IIoT space
When acquiring new software, hardware or machines define contracts on data property, security plans and service levels
Check for and request full and understandable documentation of each device and interface
Use both of best worlds (IT and OT)
Look into the other world to get alternative solutions and inspiration e.g.
Grafana dashboard instead of a built in HMI
Industrial PCs instead of Raspberry PIs
Secure crucial connections with firewalls (e.g. pfSense)
Avoid quick fixes
Use industry-wide IT/OT standards wherever applicable before developing on your own
Invest enough time for architecture basics and the big picture to avoid rework in the near future
Always document your project even when it seem unnecessary at the moment
Best-practices 2: Protocols which allow communication of IT and OT systems
Coming from IT: MQTT
MQTT
A light weight with low-bandwidth and power requirements which is the leading standard for IoT applications
All devices connect to a MQTT Broker which saves and distributes information to devices which subscribed to it (similar to social network)
Coming from IT: REST API
REST API
Standard web application interface which is used on almost every website
Can be used to request information from the web e.g. weather data or distance between two cities (Google Maps) or request actions e.g. save a file in the database
Coming from OT: OPC/UA
OPC/UA
Standard interface for automation and machine connectivity
Highly complex protocol with wide range of capabilities but low in user friendliness
Best-practices 3: Reduce complexity in machine connection with tools like Node-RED
This section explains how the system (edge and server) can be setup for development and testing enviroments.
There are three options to setup a development environment:
using a seperate device in combination with k3OS and our installation script (preferred). This requires an external device and is a fully automated installation.
using minikube (recommended for developers working on the core functionalities of the stack). This method allows you to install the stack on your device and is semi-automated.
manual installation (recommended for production environments, if you want to have fine grained control over the installation steps). This can be executed either on an external device or on your device.
The focus of this article is to provide all necessary information to install it in a compressed tutorial. There are footnotes providing additional information on certain steps, that might be new to certain user groups.
Option 1: using a seperate device in combination with k3OS and our installation script
Note: this content is also available in a presence workshop with an experienced facilitator guiding the participants through the installation and answering questions. Contact us for more information!
Prerequisites
This installation method requires some previous settings:
an edge device with x86 architecture. We recommend using the K300 from OnLogic
a computer with SSH / SFTP client 3 and Lens (for accessing the Kubernetes cluster) installed. We recommend a laptop with an Ethernet port or with an Ethernet adapter.
local LAN (with DHCP) available via atleast two Ethernet cables and access to the internet.4.
a computer monitor connected with the edge device
a keyboard connected with the edge device
As shown, the Factorycube is an optional device that combines all the required hardware in a rugged industrial gateway for industrial use.
Installation
This step is also available via a step-by-step video: TODO
k3OS
Insert your USB-stick with k3OS into your edge device and boot from it 5
Install k3OS. When asked for a cloud-init file, enter this URL and confirm: https://www.umh.app/development.yaml. If you are paranoid or want to setup devices for production you could copy the file, modify and host it yourself. Here is the template
This process takes around 15 - 20 minutes depending on your internet connection and there will be no further information about the installation status on the output of the device visible (the information on the computer screen).
Getting access to the device
To verify whether the installation worked and to access Grafana (the dashboard) and Node-RED, we will first enable SSH via password authentification, fetch the login details for Kubernetes and then login via Lens.
Step 1: Login
The login console will look like “messed up” due to the logs of the installation process in the steps above.
Immediatly after start. Nothing is messed up yet.
"Messed up" login screen
You can “clean it up” by pressing two times enter.
You can also immediatly proceed with entering the default username rancher (do not forget to press enter) and the default password rancher to login.
Logging into k3OS using the username `rancher`
Logging into k3OS using the password `rancher`
After a successfull login you should see the current IP address of the device on your computer screen.
Successfully logged into k3OS
Step 2: Enable SSH password authentification
Enable SSH password authentification in k3OS 6. This is currently not necessary anymore as the automated setup script will do that automatically, therefore this step can be skipped. This paragraph only exists to remind you that this setting is not the default behavior of k3OS and should be deactivated in production environments.
For production environments we recommend using a certificate to authenticate, which is enabled by default. This can be archieved by modifying the cloud-init file and linking to a public key stored on your GitHub account.
Step 3: Connect via SSH
Connect via SSH 3 from your laptop with the edge device. The IP address is shown on the computer screen on your edge device (see also step 1). If it is not available anymore, you can view the current IP address using ip addr or ifconfig eth0 (works with out devices).
Username: rancher
Password: rancher
Step 4: Getting Kubernetes credentials
Note
For experts: in the folder /tools you can find a bash script that will do this step automatically for you
Execute cat /etc/rancher/k3s/k3s.yaml in your SSH session on your laptop to retrieve the Kubernetes credentials. Copy the content of the result into your clipboard.
Execute `cat /etc/rancher/k3s/k3s.yaml`
Copy the content
Connect with the edge device using the software Lens and the Kubernetes credentials from your clipboard.
Add a new cluster in Lens
Select `Paste as text`
Paste from the clipboard
Ensure that you have adjusted the IP in the Kubernetes credentials with the IP of the edge device.
Also make sure that you simply adjust the IP in between. The port that follows the : should remain untouched (e.g. https://XXX.X.X.X:6443 in that case).
Hint: If you get the message ‘certificate not valid’ or something similar when connecting, verify that you entered the correct port before proceeding to the troubleshooting section (/docs/tutorials/k3os/how-to-fix-invalid-certs-due-to-misconfigured-date/).
You have now access to the Kubernetes cluster!
Verifying the installation and extracting credentials to connect with the dashboard
The installation is finished when all Pods are “Running”. You can do that by clicking on Pods on the left side.
Click in Lens on Workloads and then on Pods
Select the relevant namespaces `factorycube-server` and `factorycube-edge`
everything should be running
Some credentials are automatically generated by the system. One of them are the login credentials of Grafana. You can retrieve them by clicking on “Secrets” on the left side in Lens. Then search for a secret called “grafana-secret” and open it. Press “decode” and copy the password into your clipboard.
Press on the left side in Lens on Configuration and then Secret.
Then select grafana-secret
Then click on the eye on the right side of adminpassword to decrypt it
Opening Grafana and Node-RED
Grafana is now accessible by opening the following URL in your browser: http://<IP>:8080 (e.g., http://192.168.1.2:8080). You can login by using the username admin and password from your clipboard.
Node-RED is accessible by opening the following URL in your browser: http://<IP>:1880/nodered.
repository cloned using git clone https://github.com/united-manufacturing-hub/united-manufacturing-hub.git or downloaded and extracted using the download button on GitHub.
Start minikube using minikube start. If minikube fails to start, see the drivers page for help setting up a compatible container or virtual-machine manager.
Output of the command `minikube start`
If everything went well, kubectl is now configured to use the minikube cluster by default. kubectl version should look like in the screenshot.
Expected output of `kubectl version`
Go into the cloned repository and into the folder deployment/united-manufacturing-hub
Install by executing the following command: helm install united-manufacturing-hub . -n united-manufacturing-hub
Output of `helm install`
Now go grab a coffee and wait 15-20 minutes until all pods are “Running”.
Now you should be able to see the cluster using Lens
Option 3: manual installation
For a manual installation, we recommend that you take a look at the installation script and follow these commands manually and adjust them when needed.
Can be installed using the following command: export VERIFY_CHECKSUM=false && curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 && chmod 700 get_helm.sh && ./get_helm.sh↩︎
Can be installed on Ubuntu using the following command: sudo apt-get install kubectl↩︎
1.3 - 2. Connecting machines and creating dashboards
This section explains how the United Manufacturing Hub is used practically - from connecting machines to creating OEE dashboards
This guide will explain the typical workflow on how an engineer will connect production lines and then create dashboards based on the extracted data. It is split into two parts:
Extracting data from the shop floor by connecting with existing PLCs or additional sensors (there will be examples for both)
Creating a customized dashboard in Grafana to visualize the data
In general, there are three types of systems on the shop floor that you can connect to:
PLCs like for example via OPC/UA, Siemens S7 protocol or Modbus
Retrofitted sensors (e.g., because there was no access to the PLC). The United Manufacturing Hub uses IO-Link to connect IO-Link sensors and by using converters digital/analog sensors as well.
Additional systems like ERP, MES or quality management systems using TCP/IP, REST or MQTT.
For each protocol and type of system there is a microservice, which handles the connection and extracts all data and pushes them into the central MQTT broker.
From there on, the data is converted into the standardized data model and therefore aggregated and contextualised. The conversion process is usually done in Node-RED, but can be done in any programming language as well.
Sometimes Node-RED is used not only to aggregated and contextualise, but also to extract data from the shop floor. More information about the data flow and the programs behind it, can be found in the architecture. We’ve also written a blog article about the industrial usage of Node-RED, which we can highly recommend.
In the following, we will go through two examples of connecting the shop floor:
Retrofitting a cutting machine with the help of external sensors;
Using OPC/UA to read a warping machine data directly from the PLC.
Retrofitting with external sensors
In this example we will determine the output and the machine condition of a cutting machine step-by-step.
These sensors are used for this purpose:
Light barrier
Button bar
Inductive sensor
Step 1: Physically connect sensors and extract their data
TODO: Picture or illustration of setup
As a first step you should mount the sensors on the production machine in the regions of your interest. The sensors are then connected to an IO-Link gateway, e.g., the ifm AL1350
The IO-Link gateway is then connected to the network, so that the installation of the United Manufacturing Hub has access to it. As soon as it is physically connected and recieves an IP address it will be automatically detected by sensorconnect and the raw data will be pushed into the central MQTT broker.
The IO-Link gateways, to which sensors are connected, are found by the microservice sensorconnect using a network scan. For this purpose, sensorconnect must be configured with the correcy IP-range, so that it can search for the gateway in the correct network and read out the sensors via it.
The IP-range needs to be entered in the CIDR notation. 192.168.1.0/24 means all IP addresses from 192.168.1.1 to 192.168.1.254. To prevent accidently scanning whole IP-ranges it is set to scan localhost by default.
To do this, there are two options.
Option 1: Change the IP-range in the development.yaml (k3OS cloud-init) file, that you download. In our example the IP of our Factorycube is 192.168.1.131. Accordingly we change the IP-range to 192.168.1.0/24.
Option 2: Lens. In Lens you need to open your already set up cluster for your edge device. Then, as you can see in the picture, click on “Apps” in the left bar, then on “Releases and open “factorycube-edge” by clicking on it.
Next, click into the code and press the key combination ctrl+F to search for “iprange”. There you have to change the value of the IP-range as shown. In our example the IP of our Factorycube is 192.168.1.131. Accordingly we change the IP-range to 192.168.1.0/24.
Now the microservice can search for the gateway in the correct network to read the sensors.
As the endresult, we have now all sensor data available in the MQTT broker.
To get a quick and easy overview of the available MQTT messages and topics we recommend the MQTT Explorer. If you don’t want to install any extra software you can use the MQTT-In node to subscribe to all available topics by subscribing to # and then direct the messages of the MQTT in nodes into a debugging node. You can then display the messages in the nodered debugging window and get information about the topic and available data points.
More information on using Node-RED will come further down.
Step 2: Processing the data by creating a Node-RED flow
As the next step, we need to process the raw sensor data and add supporting information (e.g., new shifts).
Theoretically, you can do it in any programming language by connecting to the MQTT broker, processing the data and returning the results according to the datamodel. However, we recommend to do that with Node-RED.
With Node-RED, it is possible to quickly develop applications, add new data and convert raw sensor data so that for example it can be shown in a Grafana dashboard. For this purpose, so-called nodes are drawn onto a surface and wired together to create sequences. For more information, take a look at the original Node-RED tutorial
In the following, the creation of such Node-RED flows is demonstrated:
Adding new planned operator shifts
Recording the number of produced pieces with a light barrier,
Changing the machine state using buttons on a button bar
Measuring the distance of an object using an inductive sensor.
Example 1: adding new planned operator shifts
A shift is defined as the planned production time and is therefore important for the OEE calculation. If no shifts have been added, it is automatically assumed that the machine is not planned to run. Therefore, all stops will be automatically considered “No shift” or “Operator break” until a shift is added.
A shift can be added by sending a /addShift MQTT message as specified in the datamodel. In Node-RED this results in a flow consisting out of at least three nodes:
Inject node
Function node (in the picture called “add Shift”)
MQTT node
The inject node can be used unmodified.
The function node has the following code, in which the MQTT message payload and topic according to the datamodel are prepared:
The MQTT node needs to be configured to connect with the MQTT broker (if nothing has been changed from the default United Manufacturing Hub installation, it will be selected automatically).
Now you can deploy the flow and by clicking on the inject node you will generate the MQTT message. A shift will be added for the customer factoryinsight (default value), the asset location dccaachen and the asset name docs (see also topic structure).
The shift will start at 2021-12-14 at 00:00 and will end at 2021-12-15 00:00 (depending on your timezone). You can get that information by translating the UNIX timestamp in millis (e.g., 1639609200000) to a human readable format using for example https://currentmillis.com/
Example 2: recording the number of produced pieces
Now we will process sensor data instead of sending new data as before.
With the light barrier it is possible, for example, to record the number of pieces produced. Also, with a more complicated logic, machine states can be detected directly with a light barrier. For the sake of simplicity, this is not explored and applied in our example.
The first two nodes in this example will be the same for all other remaining examples as well.
First node: MQTT-in
Note
It can happen that you need to configure the username for the MQTT broker first. For this you should select a MQTT node and go into the settings of the MQTT broker. Then add there the username ia_nodered without any password.
The topic structure is (see also the datamodel): ia/raw/<transmitterID>/<gatewaySerialNumber>/<portNumber>/<IOLinkSensorID>
An example for an ia/raw/ topic is: ia/raw/development/000200410332/X02/310-372. This means that an IO-Link gateway with serial number 000200410332 has connected the sensor 310-372 to the first port X02.
Now all messages coming in (around 20 messages per second by default for one sensor) will start a Node-RED message flow.
Second node: JSON
The payload of all incoming messages will be in the JSON format, so we need to interpret this by using the JSON node.
Third node: function
The third node “function” formats the incoming data by extracting information from the json (timestamp and relevant data point) and arranging it in a usable way for Node-RED (parsing). The code for this node looks like this:
The trigger allows us, in this example in the case of the light barrier, to sort out distances that are irrelevant for our evaluation, i.e. greater than 15 cm. To do this, you just need to enter a 15 in the “Threshold” field.
Fifth node: trigger
In our example we want to count the number of produced parts. As a trolley on a rail is responsible for the transport into a container after the production of the part and moves towards the light barrier, the number of produced parts shall be increased by one as soon as the distance between the trolley and the light barrier is smaller than 15. To do this, we need a function with the following code:
The filter blocks the values of a sensor until the value is changed. A change of the property of the node is not necessary.
Fifth node: switch
The switch node is used to distinguish between the different inputs for the following flow and only lets through the values that come through the previously defined input.
With the button bar, these are the individual buttons. You can see which name is assigned to a single button by the number that appears in the debug window when you press a button. For our example these are the numbers “0108”, “0104”, 0120” and “0101”.
Sixth node: function
The switch node is followed by a separate function for each button. In our example different states are transmitted. States can be e.g. active, unknow pause, material change, process etc. and are defined via numbers. The different states can be found here.
For example, the code for the function looks like this:
In the third node “function” a timestamp is generated for each capacitive value. The timestamp is stored in the form of a string. This makes it possible, for example, to read out the time at which a item was produced.
In our example, the following function formats the incoming data from MQTT-IN by extracting information from the json (timestamp and relevant data point) and arranging it in a usable way for Node-RED (parsing). To do this, we need a function with the following code:
The Process value name process_value can be chosen freely.
Fifth node: MQTT-out
To publish messages to a pre-configured topic, the MQTT-out node is used.
The complete Node-RED flow then looks like this:
Extraction of data points via OPC/UA
This chapter will describe in the future how to connect already existing sensors or integrate an already existing interface (OPC/UA) with our system.
Hint: You can use the OPC UA node in Node-RED
Create dashboards with Grafana
So the first step is to open Grafana by opening the following URL in your browser: http://<IP>:8080 (e.g. http://192.168.1.2:8080). You can log in with the username admin and the password acquired during the installation tutorial.
Next, create a new dashboard by clicking on the + on the left side.
After clicking on Add an empty panel you should see the following settings:
Now you can add different panels to your dashboard.
Example panel 1: number of produced pieces acquired by the light barrier
For the light barrier the stat panel is used.
First of all, in “location” and “asset” labels are selected from the topic of the second function in the Node-RED flow. The “Value” needs to be count.
To get the total amount of produced parts a deeper insight into the panel settings is required. Go to “Value-Options” on the right side and change the entries of “Calculation” and “Fields” as shown below.
Example panel 2: machine state acquired by the button bar
For the button bar the Discrete panel is used.
The query parameters from the topic of the second function in the Node-RED flow must be selected in “location” and “asset”. The “value” must be state.
Example panel 3: process values
For the capacitive sensor the Graph (old) panel is used.
The query parameters from the topic of the second function in the Node-RED flow must be selected in “location” and “asset”. The “value” must be process_process_value as specified in the payload message.
Now the dashboard should look like this. In the upper right corner you can set the time span in which the data should be displayed and how often the dashboard should be refreshed.
This section explains how the system can be setup and run safely in production
This article is split up into two parts:
The first part will focus on factorycube-edge and the Industrial Automation world. The second part will focus on factorycube-server and the IT world.
Using it on the edge
The world of Industrial Automation is heavily regulated as very often not only expensive machines are controlled, but also machines that can potentially injure a human being. Here are some information that will help you in setting it up in production (not legal advice!).
If you are unsure about how to setup something like this, you can contact us for help with implementation and/or certified devices, which will ease the setup process!
Hardware & Installation, Reliability
One key component in Industrial Automation is reliability. Hardware needs to be carefully selected according to your needs and standards in your country.
When changing things at the machine, you need to ensure that you are not voiding the warranty or to void the CE certification of the machine. Even just installing something in the electrical rack and/or connecting with the PLC can do that! And it is not just unnecessary regulations, it is actually important:
PLCs can be pretty old and usually do not have much capacity for IT applications. Therefore, it is essential when extracting data to not overload the PLCs capabilities by requesting too much data. We strongly recommend to test the performance and closely watch the CPU and RAM usage of the PLC.
This is the reason we install sometimes additional sensors instead of plugging into the existing ones. And sometimes this is enough to get the relevant KPIs out of the machine, e.g., the Overall Equipment Effectiveness (OEE).
Network setup
To ensure the safety of your network and PLC we recommend a network setup like following:
Network setup having the machines network, the internal network and PLC network seperated from each other
The reason we recommend this setup is to ensure security and reliability of the PLC and to follow industry best-practices, e.g. the “Leitfaden Industrie 4.0 Security” from the VDMA (Verband Deutscher Maschinenbauer) or Rockwell.
Additionally, we are taking more advanced steps than actually recommended (e.g., preventing almost all network traffic to the PLC) as we have seen very often, that machine PLC are usually not setup according to best-practices and manuals of the PLC manufacturer by system integrators or even the machine manufacturer due to a lack of knowledge. Default passwords not changed or ports not closed, which results in unnecessary attack surfaces.
Also updates are almost never installed on a machine PLC resulting in well-known security holes to be in the machines for years.
Another argument is a pretty practical one: In Industry 4.0 we see more and more devices being installed at the shopfloor and requiring access to the machine data. Our stack will not be the only one accessing and processing data from the production machine. There might be entirely different solutions out there, who need real-time access to the PLC data. Unfortunately, a lot of these devices are proprietary and sometimes even with hidden remote access features (very common in Industrial IoT startups unfortunately…).
We created the additional DMZ around each machine to prevent one solution / hostile device at one machine being able to access the entire machine park. There is only one component (usually node-red) communicating with the PLC and sending the data to MQTT. If there is one hostile device somewhere it will have very limited access by default except specified otherwise, as it can get all required data directly from the MQTT data stream.
Our certified device “machineconnect” will have that network setup by default. Our certified device “factorycube” has a little bit different network setup, which you can take a look at here.
Other useful commands
Note
Deprecated! Please use installation script instead specified in development.yaml
Deprecated! These changes need to be applied not for factorycube-server, but to the new Helm chart united-manufacturing-hub. It may require some additional changes. It is left in as it is still better than nothing.
In general the factorycube-server installation is tailored strongly to the environments it is running in. Therefore, we can only provide general guidance on setting it up.
WARNING: THIS SECTION IS STILL IN WORK, PLEASE ONLY USE AS A ROUGH START. WE STRONGLY RECOMMEND CONTACTING US IF YOU ARE PLANNING ON USING IT IN PRODUCTION ENVIRONMENT AND WITH EXPOSURE TO THE INTERNET
Example deployment on AWS EKS
To give you an better idea, this section explains an example production deployment on AWS EKS.
Preparation
General
Use Cloudflare as DNS and firewall. This will provide an additional security layer on top of all your HTTP/HTTPS applications, e.g., factoryinsight or Grafana
We recommend the following values to get your journey to production started (using release name dev2 and namespace dev2):
### factoryinsight ###factoryinsight:enabled:trueimage:unitedmanufacturinghub/factoryinsightreplicas:2redis:URI1:dev2-redis-node-0.dev2-redis-headless:26379URI2:dev2-redis-node-1.dev2-redis-headless:26379URI3:dev2-redis-node-2.dev2-redis-headless:26379db_host:"dev2-replica"db_port:"5433"db_password:"ADD_STRONG_PASSWORD_HERE"ingress:enabled:truepublicHost:"api.dev2.umh.app"publicHostSecretName:"factoryinsight-tls-secret"annotations:external-dns.alpha.kubernetes.io/cloudflare-proxied:"false"cert-manager.io/cluster-issuer:"letsencrypt-prod"resources:limits:cpu:1000m requests:cpu:200m ### mqtt-to-postresql ###mqtttopostgresql:enabled:trueimage:unitedmanufacturinghub/mqtt-to-postgresqlreplicas:2storageRequest:1Gi### timescaleDB ###timescaledb-single:enabled:truereplicaCount:2image:# Image was built from# https://github.com/timescale/timescaledb-docker-harepository:timescaledev/timescaledb-hatag:pg12-ts2.0-latestpullPolicy:IfNotPresent backup:enabled:truepersistentVolumes:data:size:20Gi wal:enabled:truesize:5Gi### grafana ###grafana:enabled:truereplicas:2image:repository:grafana/grafanatag:7.5.9sha:""pullPolicy:IfNotPresentservice:type:ClusterIP ingress:enabled:trueannotations:kubernetes.io/ingress.class:nginxkubernetes.io/tls-acme:"true"labels:{}path:/# pathType is only for k8s > 1.19pathType:Prefixhosts:- dev2.umh.app tls:[]## Pass the plugins you want installed as a list.##plugins:- grafana-worldmap-panel- grafana-piechart-panel- aceiot-svg-panel- grafana-worldmap-panel- natel-discrete-panel- isaozler-paretochart-panel- williamvenner-timepickerbuttons-panel- agenty-flowcharting-panel- marcusolsson-dynamictext-panel- factry-untimely-panel- cloudspout-button-panel ## Grafana's primary configuration## NOTE: values in map will be converted to ini format## ref: http://docs.grafana.org/installation/configuration/##grafana.ini:paths:data:/var/lib/grafana/datalogs:/var/log/grafanaplugins:/var/lib/grafana/pluginsprovisioning:/etc/grafana/provisioninganalytics:check_for_updates:truelog:mode:consolegrafana_net:url:https://grafana.netdatabase:host:dev2 user:"grafana"name:"grafana"password:"ADD_ANOTHER_STRONG_PASSWORD_HERE"ssl_mode:requiretype:postgres## Add a seperate remote image renderer deployment/serviceimageRenderer:# Enable the image-renderer deployment & serviceenabled:truereplicas:1####################### nodered #######################nodered:enabled:truetag:1.2.9port:1880storageRequest:1Gitimezone:Berlin/EuropeserviceType:ClusterIPingress:enabled:truepublicHost:"nodered.dev2.umh.app"publicHostSecretName:"nodered-tls-secret"annotations:external-dns.alpha.kubernetes.io/cloudflare-proxied:"false"cert-manager.io/cluster-issuer:"letsencrypt-prod"settings:|- module.exports = {// the tcp port that the Node-RED web server is listening onuiPort:process.env.PORT || 1880,// By default, the Node-RED UI accepts connections on all IPv4 interfaces.// To listen on all IPv6 addresses, set uiHost to "::",// The following property can be used to listen on a specific interface. For// example, the following would only allow connections from the local machine.//uiHost:"127.0.0.1",// Retry time in milliseconds for MQTT connectionsmqttReconnectTime:15000,// Retry time in milliseconds for Serial port connectionsserialReconnectTime:15000,// The following property can be used in place of 'httpAdminRoot' and 'httpNodeRoot',// to apply the same root to both parts.httpRoot:'/nodered',// If you installed the optional node-red-dashboard you can set it's path// relative to httpRootui:{path:"ui"},// Securing Node-RED// -----------------// To password protect the Node-RED editor and admin API, the following// property can be used. See http://nodered.org/docs/security.html for details.adminAuth:{type:"credentials",users:[{username:"admin",password:"ADD_NODERED_PASSWORD",permissions:"*"}]},functionGlobalContext:{// os:require('os'),// jfive:require("johnny-five"),// j5board:require("johnny-five").Board({repl:false})},// `global.keys()` returns a list of all properties set in global context.// This allows them to be displayed in the Context Sidebar within the editor.// In some circumstances it is not desirable to expose them to the editor. The// following property can be used to hide any property set in `functionGlobalContext`// from being list by `global.keys()`.// By default, the property is set to false to avoid accidental exposure of// their values. Setting this to true will cause the keys to be listed.exportGlobalContextKeys:false,// Configure the logging outputlogging:{// Only console logging is currently supportedconsole:{// Level of logging to be recorded. Options are:// fatal - only those errors which make the application unusable should be recorded// error - record errors which are deemed fatal for a particular request + fatal errors// warn - record problems which are non fatal + errors + fatal errors// info - record information about the general running of the application + warn + error + fatal errors// debug - record information which is more verbose than info + info + warn + error + fatal errors// trace - record very detailed logging + debug + info + warn + error + fatal errors// off - turn off all logging (doesn't affect metrics or audit)level:"info",// Whether or not to include metric events in the log outputmetrics:false,// Whether or not to include audit events in the log outputaudit:false}},// Customising the editoreditorTheme:{projects:{// To enable the Projects feature, set this value to trueenabled:false}}}##### CONFIG FOR REDIS #####redis:enabled:truecluster:enabled:trueslaveCount:2image:pullPolicy:IfNotPresentregistry:docker.iorepository:bitnami/redistag:6.0.9-debian-10-r13master:extraFlags:- --maxmemory 4gbpersistence:size:8Giresources:limits:memory:4Girequests:cpu:100mmemory:1GipodDisruptionBudget:enabled:trueminAvailable:2slave:persistence:size:8Giresources:limits:memory:4Girequests:cpu:100mmemory:1Gi##### CONFIG FOR VERNEMQ #####vernemq:enabled:trueAclConfig:|- pattern write ia/raw/%u/#
pattern write ia/%u/#
pattern $SYS/broker/connection/%c/state
user TESTING
topic ia/#
topic $SYS/#
topic read $share/TESTING/ia/#
user ia_nodered
topic ia/#CACert:|-ADD CERTCert:|-ADD CERTPrivkey:|-ADD CERTimage:pullPolicy:IfNotPresent repository:vernemq/vernemqtag:1.11.0replicaCount:2service:annotations:prometheus.io/port:"8888"prometheus.io/scrape:"true"external-dns.alpha.kubernetes.io/cloudflare-proxied:"false"external-dns.alpha.kubernetes.io/hostname:mqtt.dev2.umh.appmqtts:enabled:truenodePort:8883port:8883mqtt:enabled:falsetype:LoadBalancer
Further adjustments
VerneMQ / MQTT
We recommend setting up a PKI infrastructure for MQTT (see also prerequisites) and adding the certs to vernemq.CAcert and following in the helm chart (by default there are highly insecure certificates there)
You can adjust the ACL (access control list) by changing vernemq.AclConfig
If you are using the VerneMQ binaries in production you need to accept the verneMQ EULA (which disallows using it in production without contacting them)
Redis
The password is generated once during setup and stored in the secret redis-secret
Nodered
We recommend disabling external access to nodered entirely and spawning a seperate nodered instance for every project (to avoid having one node crashing all flows)
You can change the configuration in nodered.settings
We recommend that you set a password for accessing the webinterface in the nodered.settings. See also the official tutorial from nodered
MinIO
We strongly recommend to change all passwords and salts specified in values.yaml
2 - Concepts
The software of the United Manufacturing Hub is designed as a modular system. Our software serves as a basic building block for connecting and using various hardware and software components quickly and easily. This enables flexible use and thus the possibility to create comprehensive solutions for various challenges in the industry.
High-level architecture
The United Manufacturing Hub consists out of three layers and two packages for installation:
The following sub-chapters will explain the layers and the two packages further. If you want to deep-dive into the actual architecture, you can scroll further down
Lower level: data acquisition
The data sources connected to the edge device provide the foundation for automatic data collection. The data sources can be external sensors (e.g. light barriers, vibration sensors), input devices (e.g. button bars), Auto-ID technologies (e.g. barcode scanners), industrial cameras and other data sources such as machine PLCs. The wide range of data sources allows the connection of all machines, either directly via the machine PLC or via simple and fast retrofitting with external sensors.
Examples:
sensorconnect (to automatically read out IO-Link Master and their connected sensors)
cameraconnect (to automatically read out GenICam compatible cameras and push the result into MQTT, in development)
barcodereader (to connect USB barcodereader and push data into MQTT)
Node-RED (e.g., for propertiary and / or machine specific protocols)
PLC4X (in development)
Middle layer: data infrastructure
This layer is the central part of the United Manufacturing Hub. It provides an infrastructure including data models to fulfull all manufacturing needs for data processing and storage.
It starts by making all acquired data accessible in real-time for data processing using either established solutions like Node-RED or your own written software using a microservice approach and MQTT. Therefore, adding new data, processing it or integrating it with other systems on-the-edge is very easy. We recommend to start transforming data into the central data model at this step.
To send the raw and / or processed data to a central place (cloud or on-premise) we use our self-written MQTT bridge. Internet connections or network in general is often unstable in manufacturing environments and therefore one needs to safely buffer messages across internet or electricity downtimes. As existing MQTT bridge solutions were unreliable we developed our own.
Once the data arrives at the server it can be further processed using the same methods as on-the-edge (MQTT microservice, Node-RED, etc.). The real-time data can also integrated into MES or ERP systems.
All processed data is then stored into databases using load-balanced microservices with caching. Therefore, one can archieve high-availability and enourmous scalability through the load-balanced microservices. Furthermore, common requests and operations are cached in redis
Relational data (e.g., data about orders and products) as well as time series data in high resolution (e.g., machine data like temperature) can be stored in the TimescaleDB database (difference between InfluxDB and timescaleDB have been described in detail). Blob data (e.g., camera pictures) will be stored in a blob storage either directly in Minio or using a Minio gateway in a cloud specific storage like AWS S3 or Microsoft Azure Blob Storage.
We do not allow direct access to the databases for performance and security reasons. Instead, we’ve put an additional, self-written, component in front called factoryinsight. factoryinsight provides a REST API to access raw data from the databases as well as processed data in form of KPI’s like ‘OEE losses’ or similar. All requests are load-balanced, cached and executed only on a replica of the database.
To insert data via a REST API we’ve developed two additional services grafana-proxy and factoryinput.
Examples:
TimescaleDB
Node-RED
factoryinput
factoryinsight
minio
Higher level: visualization
As a standard dashboarding tool the United Manufacturing Hub uses Grafana in combination with self-written plugins, which allow every user (even without programming knowledge) to quickly and easilyy compose personally tailored dashboards with the help of modular building blocks.
The entire stack can be deployed using only a configuration file (values.yaml) and the corresponding Helm charts factorycube-server and factorycube-edge.
This allows to deploy the architecture in hybrid setups, from deploying it on-the-edge IIoT gateways to on-premise servers to the cloud (e.g., Azure AKS)
Low-level architecture
If you want to go more into detail, here is the detailled architecture:
2.1 - Node-RED in Industrial IoT: a growing standard
How an open-source tool is establishing itself in a highly competitive environment against billion dollar companies
Using Node-RED and UaExpert to extract data from the PLC of a Saw
Most people know Node-RED from the areas of smart home or programming introductions (those workshops where you connect things with microcontrollers). Yet, very few people realize that it is frequently used in manufacturing as well.
Node-RED is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways.
It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click.
And the best thing: it is open-source
The project started in early 2013 in IBM’s research centers. In 2016 it was one of the founding projects of the JS Foundation. Since the version release 1.0 in 2019 it is considered safe for production use.
A self-conducted survey in the same year showed that from 515 respondents, 31.5% use Node-RED in manufacturing, and from 868 respondents, 24% said they have created a PLC application using it 1. Also, 24.2 % of 871 respondents said that they use InfluxDB in combination with Node-RED. The reason we think that TimescaleDB is better suited for the Industrial IoT than InfluxDB has been described in this article.
But how widespread is it really in manufacturing? What are these users doing with Node-RED? Let’s deep dive into that!
Usage of Node-RED in Industry
Gathering qualitative data of industry usage of specific solutions can be hard to almost impossible as very few companies are open about the technologies they use. However, we can still gather quantitative data, which strongly indicates a heavy usage in various industries in data extraction and processing.
Furthermore, various so-called “nodes” are available that can only be used in manufacturing environments, e.g., to read out data from specific devices. These nodes also have quite impressive download numbers.
node-red-contrib-modbus had 4278 downloads in the last week (2021-07-02). However, it should be noted that this node can also be used in other areas
We’ve talked with the former developer of two of these nodes, Klaus Landsdorf from the German company Iniationware, which offers companies support in the topics of OPC-UA, Modbus, BACnet and data modeling.
Klaus confirmed our hypothesis:
We get many requests from German hardware manufacturers that rely on Node-RED and on these industry-specific nodes like OPC-UA. The OPC-UA project was sponsored by just two small companies with round about 5% of the costs for development in the case of the IIoT OPC-UA contribution package. But in view of using the package and testing it across multiple industrial manufacturing environments to ensure a high stability, we had many and also big companies aboard. In education we have a great response from ILS, because they are using the Iniationware package node-red-contrib-iiot-opcua to teach their students about OPC-UA essentials. Unfortunately, just a few companies understand the idea of a commercial backing for open-source software companies by yearly subscriptions, which could safe a lot of money for each of them. Do it once, stable and share the payment in open-source projects! That would bring a stable community and contribution packages for the specific reley on industrial needs like LTS versions. Simplified: it needs a bit money to make money in a long term as well as to provide stable and up to date Node-RED packages.
Although anecdotal, just Friday, I was speaking to an engineer at a major OPC Software Vendor who commented that they see Node-RED frequently deployed by industrial clients and even use it internally for proving out concepts and technology.
Another one with the name of gemini86 explains the advantages compared with commercial solutions:
I’m also (very much) late to the party on this, but I work in manufacturing and use AB, Siemens, Codesys, etc. I also use Node-RED for SCADA and database bridging. Our site has well pumps in remote areas where data and commands are sent over 900mhz ethernet radios, and Node-RED handles the MQTT <> modbusRTU processing. Node-RED has been as stable and quick, if not quicker than any Siemens or AB install with comparable network functionality. In fact, I struggled to get my S7-1200 to properly communicate with modbusRTU devices at all. I was completely baffled by their lack of documentation on getting it to work. Their answer? “Use profibus/profinet.” So, I myself prefer Node-RED for anything to do with serial or network communications.
Last but not least, it is very frequently used in scientific environments. There are over 3.000 research papers available on Google Scholar on the usage of Node-RED in industrial environments!
Therefore, it is safe to say that it is widespread, with growing numbers of users in industry. But what exactly can you do with it? Let us give some examples of how we are using it!
What you can do with it
The United Manufacturing Hub relies on Node-RED as a tool to
Extract data from production machines using various protocols (OPC/UA, Modbus, S7, HTTP, TCP, …)
Customer-specific integrations into existing systems, e.g., MES or ERP systems like SAP or Oracle
Combining data from various machines and triggering actions (machine to machine communication or, in short, M2M)
Creating small interactive and customer-specific dashboards to trigger actions like specifying stop reasons
Let’s explain each one by going through them step-by-step:
1. Extract data from production machines using various protocols
One central challenge of Industrial IoT is obtaining data. The shopfloor is usually fitted out with machines from various vendors and of different ages. As there is almost little or no standardization in the protocols or semantics, the data extraction process needs to be customized for each machine.
With Node-RED, various protocols are available as so-called “nodes” - from automation protocols like OPC/UA (see earlier) to various IT protocols like TCP or HTTP. For any other automation protocol, you can use PTC Kepware, which supports over 140 various PLC protocols.
2. Processing and unifying data points into our standardized data model
Node-RED was originally developed for
visualizing and manipulating mappings between MQTT topics 2
and this is what we are still using it for today. All these data points that have been extracted from various production machines now need to be standardized to match our data model. The machine state needs to be calculated, the machines’ output converted from various formats into a simple /count message, etc.
Example of working with the United Manufacturing Hub. Everything is flow-based.
3. Customer-specific integrations into existing systems
It is not just good for extracting and processing data. It is also very good for pushing this processed data back into other systems, e.g., MES or ERP systems like Oracle or SAP. These systems usually have REST APIs, e.g., here is an example for the REST API for the Oracle ERP.
As the customer implementations of those systems are usually different, the resulting APIs are mostly also different. Therefore, one needs a system that is quick to use to handle those APIs. And Node-RED is perfect for this.
4. Machine to machine communication
The AGV automatically gets the finished products from one machine and brings them to empty stations, which is a good example for M2M
As a result of our data architecture machine to machine communication (M2M) is enabled by default. The data from all edge devices is automatically sent to a central MQTT broker and is available to all connected devices (that have been allowed access to that data).
It is easy to gather data from various machines and trigger additional actions, e.g., to trigger the Automated Guided Vehicle (AGV) to fetch material from the production machine when one station is empty of material.
And the perfect tool to set those small triggers is, as you might have guessed, Node-RED.
5. Creating small interactive and customer-specific dashboards
Example of a dashboard using node-red-dashboard. It features a multi-level stop reason selection and the visualization of production speed.
Sometimes the machine operators need time-sensitive dashboards to retrieve real-time information or to interact with the system. As many companies still do not have a good and reliable internet connection or even network infrastructure, one cannot wait until the website is fully loaded to enter a stop reason. Therefore, sometimes it is crucial to have a dashboard as close to the machine as possible (and not sitting somewhere in the cloud).
For this one, you can use the node-red-dashboard node, which allows you to easily create dashboards and interact with the data via MQTT.
Bonus: What not to do: process control
However, we strongly recommend NOT using it to intervene in the production process, e.g., process control or ensuring safety mechanisms for two reasons:
IT tools and systems like Node-RED are not designed to ensure the safety of machines or people, e.g., guaranteed time-sensitive reactions to a triggered safety alert
It would also be almost impossible to get that certified and approved due to 1.
For these aspects, very good and safe tools, like PLCs or NCs, are already out there in the automation world.
Summary
The slogan: “The best things in life are free” also applies in manufacturing:
Node-RED is on the same level as “professional” closed-source and commercial solutions and is used by thousands of researchers and hundreds of daily users in various manufacturing industries.
It is included and enabled in every installation of the United Manufacturing Hub - in the cloud and on the edge.
More information on how we use the system can be found in our Quick Start.
All events or subsequent changes in production are transmitted via MQTT in the following data model
Introduction
All events or subsequent changes in production are transmitted via MQTT in the following data model. This ensures that all participants are always informed about the latest status.
The data model in the MQTT Broker can be divided into four levels. In general, the higher the level, the lower the data frequency and the more the data is prepared.
If you do not know the idea of MQTT (important keywords: “broker”, “subscribe”, “publish”, “topic”), we recommend reading the wikipedia article first.
All MQTT messages consist out of one JSON with at least two elements in it:
Key
Data type/format
Description
timestamp_ms
int
the amount of milliseconds since the 1970-01-01 (also called UNIX timestamp in milliseconds)
<valueName>
int, str, dict
a value that can be int, str, or even in dict format
Some messages might deviate from this format. When this happens it will be noted explicitly.
All topics are to be written in lower case only!
1st level: Raw data
Data from this level are all raw data, which are not yet contextualized(i.e., assigned to a machine). These are, in particular, all data from sensorconnect and cameraconnect.
This means that the transmitter with the serial number 2020-0102 has one ifm gateway connected to it with the serial number 0000005898845. This gateway has the sensor connected to the first port X01.
Example for decoding an image and saving it locally with OpenCV
im_bytes=base64.b64decode(incoming_mqtt_message["image"]["image_bytes"])im_arr=np.frombuffer(im_bytes,dtype=np.uint8)# im_arr is a one-dimensional Numpy arrayimg=cv2.imdecode(im_arr,flags=cv2.IMREAD_COLOR)cv2.imwrite(image_path,img)
2nd level: Contextualized data
In this level the data is already assigned to a machine.
Topic structure: ia/<customerID>/<location>/<AssetID>/<Measurement> e.g. ia/dccaachen/aachen/demonstrator/count.
An asset can be a step, machine, plant or line. It uniquely identifies the smallest location necessary for modeling the
process.
By definition all topic names should be lower case only!
/count
Topic: ia/<customerID>/<location>/<AssetID>/count
Here a message is sent every time something has been counted. This can be, for example, a good product or scrap.
count in the JSON is an integer.
scrap in the JSON is an integer, which is optional. It means scrap pieces of count are scrap. If not specified it is 0 (all produced goods are good).
Here a message is sent every time products should be marked as scrap. It works as follows:
A message with scrap and timestamp_ms is sent. It starts with the count that is directly before timestamp_ms. It is now iterated step by step back in time and step by step the existing counts are set to scrap until a total of scrap products have been scraped.
Important notes
You can specify maximum of 24h to be scrapped to avoid accidents
(NOT IMPLEMENTED YET) If counts does not equal scrap, e.g. the count is 5 but only 2 more need to be scrapped, it will scrap exactly 2. Currently it would ignore these 2. see also #125
(NOT IMPLEMENTED YET) If no counts are available for this asset, but uniqueProducts are available, they can also be marked as scrap. //TODO
scrap in the JSON is an integer.
key
data type
description
scrap
int
Number of item from count that is considered as scrap. When scrap is equal to 0, that means all produced goods are good quality
A message is sent here every time the machine runs or stops (independent whether it runs slow or fast, or which reason the stop has. This is covered in state)
A message is sent here each time a stop reason has been identified automatically or by input from the machine operator (independent whether it runs slow or fast, or which reason the stop has. This is covered in state)
Represents the amount of target units to be produced (in the same unit as count)
Attention
The product needs to be added before adding the order. Otherwise, this message will be discarded
One order is always specific to that asset and can, by definition, not be used across machines. For this case one would need to create one order and product for each asset (reason: one product might go through multiple machines, but might have different target durations or even target units, e.g. one big 100m batch get split up into multiple pieces)
A message is sent here every time a process value has been prepared. Unique naming of the key.
key
data type
description
<valueName>
int or float
Represents a process value, e.g. temperature.
Attention
As <valueName> is a integer or float, booleans like “true” or “false” are not possible. Please convert them to integer, e.g., “True” –> 1, “False” –> 0
/productImage has the same data format as ia/rawImage, only with a changed topic.
/productImage can be acquired in two ways, either from ia/rawImage or /rawImageClassification. In the case of /rawImageClassification,
only the Image part is extracted to /productImage, while the classification information is stored in the relational database.
key
data type
description
image_id
str
a unique identifier for every image acquired (e.g. format:<MACaddress>_<timestamp_ms>)
image_bytes
str
base64 encoded image in JPG format in bytes
image_height
int
height of the image in pixel
image_width
int
width of the image in pixel
image_channels
int
amount of included color channels (Mono: 1, RGB: 3)
/productTagString is usually generated whenever a product is transformed into another product. It can be used multiple times for the same child to model that one product can consists out of multiple parents.
See also Digital Shadow for more information how to use this message
3rd level: production data
This level contains only highly aggregated production data.
/state
Topic: ia/<customerID>/<location>/<AssetID>/state
A message is sent here each time the asset changes status. Subsequent changes are not possible. Different statuses can also be process steps, such as “setup”, “post-processing”, etc. You can find a list of all supported states here
A message is sent here each time a product has been produced or modified. A modification can take place, for example, due to a downstream quality control.
There are two cases of when to send a message under the uniqueProduct topic:
The exact product doesn’t already have a UID (-> This is the case, if it has not been produced at an asset
incorporated in the digital shadow). Specify a space holder asset = “storage” in the MQTT message for the
uniqueProduct topic.
The product was produced at the current asset (it is now different from before, e.g. after machining or after
something was screwed in). The newly produced product is always the “child” of the process. Products it was made
out of are called the “parents”.
Shopfloor insights are recommendations for action that require concrete and rapid action in order to quickly eliminate efficiency losses on the store floor.
key
data type/format
description
recommendationUID
int
Unique ID of the recommendation. Used to subsequently deactivate a recommendation (e.g. if it has become obsolete)
recommendationType
int
The ID / category of the current recommendation. Used to narrow down the group of people
In short: AID. Used to describe a single product of the type product_id in the order order_id. This is the ID that might be written on the product (e.g., with a physical label, lasered, etc.) and is usually the relevant ID for engineers and for production planning. It usually stays the same.
UID
int
Short for unique product ID. Compared to the AID the UID changes whenever a product changes its state. Therefore, a product will change its UID everytime it is placed on a new asset. It is used mainly on the database side to lookup a specific product in a specific state.
These IDs are linked together in the database.
TimescaleDB structure
Here is a scheme of the timescaleDB structure:
(open the image using the right click for a better resolution)
2.3 - Available states for assets
This data model maps various machine states to relevant OEE buckets.
Introduction
This data model is based on the following specifications:
Weihenstephaner Standards 09.01 (for filling)
Omron PackML (for packaging/filling)
EUROMAP 84.1 (for plastic)
OPC 30060 (for tobacco machines)
VDMA 40502 (for CNC machines)
Additionally, the following literature is respected:
Steigerung der Anlagenproduktivität durch OEE-Management (Focke, Steinbeck)
Abbreviations
WS –> “TAG NAME”: Valuename (number)
PackML –> Statename (number)
EUROMAP –> Statusname (number)
Tobacco –> ControlModeName (number)
ACTIVE (10000-29999)
The asset is actively producing.
10000: ProducingAtFullSpeedState
The asset is running on full speed.
Examples for ProducingAtFullSpeedState
WS_Cur_State: Operating
PackML/Tobacco: Execute
20000: ProducingAtLowerThanFullSpeedState
The asset is NOT running on full speed.
Examples for ProducingAtLowerThanFullSpeedState
WS_Cur_Prog: StartUp
WS_Cur_Prog: RunDown
WS_Cur_State: Stopping
PackML/Tobacco: Stopping
WS_Cur_State: Aborting
PackML/Tobacco: Aborting
WS_Cur_State: Holding
WS_Cur_State: Unholding
PackML/Tobacco: Unholding
WS_Cur_State: Suspending
PackML/Tobacco: Suspending
WS_Cur_State: Unsuspending
PackML/Tobacco: Unsuspending
PackML/Tobacco: Completing
WS_Cur_Prog: Production
EUROMAP: MANUAL_RUN
EUROMAP: CONTROLLED_RUN
NOT INCLUDED FOR NOW:
WS_Prog_Step: all
UNKNOWN (30000-59999)
The asset is in an unspecified state.
30000: UnknownState
We do not have any data for that asset (e.g. connection to PLC aborted).
Examples for UnknownState
WS_Cur_Prog: Undefined
EUROMAP: Offline
40000: UnspecifiedStopState
The asset is not producing, but we do not know why (yet).
Examples for UnspecifiedStopState
WS_Cur_State: Clearing
PackML/Tobacco: Clearing
WS_Cur_State: Emergency Stop
WS_Cur_State: Resetting
PackML/Tobacco: Clearing
WS_Cur_State: Held
EUROMAP: Idle
Tobacco: Other
WS_Cur_State: Stopped
PackML/Tobacco: Stopped
WS_Cur_State: Starting
PackML/Tobacco: Starting
WS_Cur_State: Prepared
WS_Cur_State: Idle
PackML/Tobacco: Idle
PackML/Tobacco: Complete
EUROMAP: READY_TO_RUN
50000: MicrostopState
The asset is not producing for a short period (typically around 5 minutes), but we do not know why (yet).
MATERIAL (60000-99999)
The asset has issues with materials.
60000: InletJamState
The machine does not perform its intended function due to a lack of material flow in the infeed of the machine detected by the sensor system of the control system (machine stop). In the case of machines that have several inlets, the condition of lack in the inlet refers to the main flow, i.e. to the material (crate, bottle) that is fed in the direction of the filling machine (central machine). The defect in the infeed is an extraneous defect, but because of its importance for visualization and technical reporting, it is recorded separately.
Examples for InletJamState
WS_Cur_State: Lack
70000: OutletJamState
The machine does not perform its intended function as a result of a jam in the good flow discharge of the machine detected by the sensor system of the control system (machine stop). In the case of machines that have several discharges, the jam in the discharge condition refers to the main flow, i.e. to the good (crate, bottle) that is fed in the direction of the filling machine (central machine) or is fed away from the filling machine. The jam in the outfeed is an external fault 1v, but it is recorded separately “because” of its importance for visualization and technical reporting.
Examples for OutletJamState
WS_Cur_State: Tailback
80000: CongestionBypassState
The machine does not perform its intended function due to a shortage in the bypass supply or a jam in the bypass discharge of the machine detected by the sensor system of the control system (machine stop). This condition can only occur in machines that have two outlets or inlets and in which the bypass is in turn the inlet or outlet of an upstream or downstream machine of the filling line (packaging and palletizing machines). The jam/shortage in the auxiliary flow is an external fault, but is recorded separately due to its importance for visualization and technical reporting.
Examples for CongestionBypassState
WS_Cur_State: Lack/Tailback Branch Line
90000: MaterialIssueOtherState
The asset has a material issue, but it is not further specified.
Examples for MaterialIssueOtherState
WS_Mat_Ready (Information about which material is lacking)
PackML/Tobacco: Suspended
PROCESS (100000-139999)
The asset is in a stop which is belongs to the process and cannot be avoided.
100000: ChangeoverState
The asset is in a changeover process between products.
Examples for ChangeoverState
WS_Cur_Prog: Program-Changeover
Tobacco: CHANGE OVER
110000: CleaningState
The asset is currently in a cleaning process.
Examples for CleaningState
WS_Cur_Prog: Program-Cleaning
Tobacco: CLEAN
120000: EmptyingState
The asset is currently emptied, e.g. to prevent mold for food products over the long breaks like the weekend.
Examples for EmptyingState
Tobacco: EMPTY OUT
130000: SettingUpState
The machine is currently preparing itself for production, e.g. heating up.
Examples for SettingUpState
EUROMAP: PREPARING
OPERATOR (140000-159999)
The asset is stopped because of the operator.
140000: OperatorNotAtMachineState
The operator is not at the machine.
150000: OperatorBreakState
The operator is in a break. note: different than planned shift as it could count to performance losses
Examples for OperatorBreakState
WS_Cur_Prog: Program-Break
PLANNING (150000-179999)
The asset is stopped as it is planned to stop (planned idle time).
160000: NoShiftState
There is no shift planned at that asset.
170000: NoOrderState
There is no order planned at that asset.
TECHNICAL (180000-229999)
The asset has a technical issue.
180000: EquipmentFailureState
The asset itself is defect, e.g. a broken engine.
Examples for EquipmentFailureState
WS_Cur_State: Equipment Failure
190000: ExternalFailureState
There is a external failure, e.g. missing compressed air
Examples for ExternalFailureState
WS_Cur_State: External Failure
200000: ExternalInterferenceState
There is an external interference, e.g. the crane to move the material is currently unavailable.
210000: PreventiveMaintenanceStop
A planned maintenance action.
Examples for PreventiveMaintenanceStop
WS_Cur_Prog: Program-Maintenance
PackML: Maintenance
EUROMAP: MAINTENANCE
Tobacco: MAINTENANCE
220000: TechnicalOtherStop
The asset has a technical issue, but it is not specified further.
Examples for TechnicalOtherStop
WS_Not_Of_Fail_Code
PackML: Held
EUROMAP: MALFUNCTION
Tobacco: MANUAL
Tobacco: SET UP
Tobacco: REMOTE SERVICE
2.4 - Digital Shadow - track and trace
A system of features allowing tracking and tracing of individual parts through the production process. This article explains how it can be applied and how it works.
Introduction
During the production process a lot of data is generated (e.g., process values like temperature or images), but typically not linked to specific products. When investigating product returns, one needs to gather data from various data sources to understand what exactly happened, which might take so much time that it is not done at all.
The life would be much easier for a quality inspector if he would be able to enter the ID of the defect product and then receive all related information to it - from temperatures during the production to test results to product images.
Solution:
We’ve expanded the United Manufacturing Hub so that the end-user only needs to do two things:
Process this raw data with Node-RED and send MQTT messages according to the UMH specification
Access the processed data either in a BI tool or using Grafana
To allow this to happen the backend has been modified to support multiple new MQTT message types in mqtt-to-postgresql and to provide more endpoints to fetch Digital Shadow related data from the database factoryinsight.
Dataflow
(right click on the image and open it for a better resolution)
This is the overview of the digital shadow concept. It follows the general design principles of the United Manufacturing Hub by sending all raw sensor data first to an MQTT broker and then continuously processing it.
The following chapters are going through the concept from left to right (from the inputs of the digital shadow to the outputs).
Step 1: Gathering data from the shopfloor
Data sources are connected by sending their data to the central MQTT broker.
UMH recommends to stick to the data definition of the UMH datamodel for the topics and messages, but the implementation is client specific and can be modeled for the individual problem.
Step 2: contextualizing the data using microservices and MQTT
Now the information is available at the MQTT broker because of that to all subscribed services.
In the next step this raw data is contextualized, which means to link it to specific products. To identify a product two different type of IDs are used: AID’s and UID’s (identifiers are later explained in detail).
The raw data from the data sources needs to be converted to four different MQTT message types:
To do that we recommend writing microservices. You can do that either in Node-RED (our recommendation) or in a programming language of your choice. These microservices convert messages under a raw topic into messages under processValue or processValueString.
This typically only requires resending the message under the appropriate topic or breaking messages with multiple values apart into single ones.
Generating the contextualized messages
The goal is to convert messages under the processValue and the processValueString topics, containing all relevant data,
into messages under the topic productTag, productTagString and addParentToChild. The latter messages contain AID’s
which hold the contextualization information - they are tied to a single product.
The implementation of the generation of the above mentioned messages with contextualized information is up to the user
and depends heavily on the specific process. To help with this we want to present a general logic and talk about the
advantages and disadvantages of it:
General steps:
Make empty containers for predefined messages to mqtt-to-postgresql when the first production step took place
Fill containers step by step when relevant messages come in.
If full, send the container.
If the message from the first production step for the new product is received before the container is full,
send container and set missing fields to null. Also send an error message.
Example process:
parent ID 1 scanned (specifically the later explained AID) -> barcode sent under
processValueString topic
might need a lot of different containers if the number of e.g. productTag messages gets to big
Identifiers
The explaination of the IDs can be found in the UMH datamodel (especially the definition of the terms AID and UID)
Definition of when to change the UID
If we can move a product from point “A” in the production to point “B” or back without causing problems from a process perspective, the UID of the product should stay the same. (For example if the product only gets transported between point “A” and “B”).
If moving the object produces problems (e.g. moving a not yet tested object in the bin “tested products”), the object should have gotten a new UID on its regular way.
Example 1: Testing
Even though testing a product doesn’t change the part itself, it changes its state in the production process:
it gets something like a virtual “certificate”
the value increases because of that
-> Make a new UID.
Example 2: Transport
Monitored Transport from China to Germany (This would be a significant distance: transport data would be useful to include into digital shadow)
MQTT messages under the productTag topic should not be used to indicate transport of a part. If transport is relevant, change the UID (-> send a new MQTT message to mqtt-to-postgresql under the uniqueProduct topic).
Example process to show the usage of AID’s and UID’s in the production:
Explanation of the diagram:
Assembly Station 1:
ProductA and ProductB are combined into ProductC
Because ProductA and ProductB have not been “seen” by the digital shadow, they get a new UID and asset = “storage”
assigned (placeholder asset for unknown/unspecified origin).
After ProductC is now produced it gets a new UID and as an asset, Assy1, because it is the child at Assembly Station 1
The AID of the child can always be freely chosen out of the parent AID’s. The AID of ProductA (“A”) is a physical
label. Because ProductB doesn’t have a physical Label, it gets a generated AID. For ProductC (child) we can now choose
either the AID from ProductA or from ProductB. Because “A” is a physical label, it makes sense to use the AID of
ProductA.
MQTT messages to send at Assembly 1:
uniqueProduct message for ProductA origin, with asset = storage, under the topic:
ia/testcustomer/testlocation/storage/uniqueProduct
Now the ProductC is transported to Assembly Station 2. Because it is a short transport, doesn’t add value etc. we do not need to produce a new UID after the transport of ProductA.
Assembly Station 2:
ProductC stays the same (in the sense that it is keeping its UID before and after the transport), because of the easy
transport.
ProductD is new and not produced at assembly station 2, so it gets asset = “storage” assigned
ProductC and ProductD are combined into ProductE. ProductE gets a new UID. Both AID’s are physical. We again freely
choose the AID we want to use (AID C was chosen, maybe because after the assembly of ProductC and ProductD, the AID
Label on ProductD is not accessible while the AID Label on the ProductC is).
Assembly Station 3:
At Assembly Station ProductE comes in and is turned into ProductF
ProductF gets a new UID and keeps the AID of ProductE. It now gets the Assy3 assigned as asset.
Note that the uniqueProduct MQTT message for ProductD would not be under the Topic of Assembly2 as asset but for example under storage. The convention is, that every part never seen by digital shadow “comes” from storage even though the UID and the related uniqueProduct message is created at the current station.
Batches of parts
If for example a batch of screws is supplied to one asset with only one datamatrix code (one AID) for all screws together, there will only be one MQTT message under the topic uniqueProduct created for the batch with one AID, a newly generated UID and with the default supply asset storage.
The batch AID is then used as parent for a MQTT message under the topic addParentToChild.
(-> mqtt-to-postgres will repeatedly fetch the same parent uid for the inheritanceTable)
The batch AID only changes when new batch AID is scanned.
Step 3: mqtt-to-postgresql
The mqtt-to-postgresql microservice now uses the MQTT messages it gets from the broker and writes the information in the database. The microservice is not use-case specific, so the user just needs to send it the correct MQTT messages.
mqtt-to-postgresql now needs to generate UID’s and save the information in the database, because the database uses UID’s to store and link all the generated data efficiently. Remember that the incoming MQTT messages are contextualized with AID’s.
We can divide the task of mqtt-to-postgresql in three (regarding the digital shadow):
Use the MQTT message under the Topic uniqueProduct which gives us the AID and the Asset and make an entry in the uniqueProduct table containing the AID and a newly generated UID.
Store new UID and all data from uniqueProduct MQTT Message in the uniqueProductTable
Use productTag and productTagString topic MQTT messages. The AID and the AssetId is used to look for the uniqueProduct the messages belong to. The value information is then stored with the UID in the TimescaleDB
Look in TimescaleDB, uniqueProductTable for the uniqueProduct with the same Asset and AID from the productTag message (the child)
Write value information without AID, instead with the found UID in the uniqueProductTable
Use the addParentToChild message. Retrieve the child UID by using the child AID and the Asset. Get the parent
UID’s by finding the last time the parents AID’s were stored in the uniqueProductTable.
Look in TimescaleDB, uniqueProductTable for the uniqueProduct with the same Asset and AID as written in the child of the /addParentToChild message
Look in the TimescaleDB, uniqueProductTable for all other assets for the last time the AID of the parent was used and get the UID
Write UID of child and UID of the parent in the productInheritanceTable
All uniqueProduct of one step at one asset need to be stored before we can process addParentToChild messages.
This means we also need to send possible parent uniqueProduct MQTT messages (asset = storage) before.
Step 4: Database and the database model
The structure of the database might be changed in the future.
Four tables are especially relevant:
uniqueProductTable contains entries with a pair of one UID and one AID and other data.
productTagTable and productTagStringTable store information referenced to the UID’s in the uniqueProductTable.
Stored is everything from individual measurements to quality classes.
productInheritanceTable contains pairs of child and parent UID’s. The table as a whole thereby contains the complete
inheritance information of each individual part. One entry describes one edge of the inheritance graph.
The new relevant tables are dotted, the uniqueProductTable changes are bold in the timescaleDB structure visualization.
Step 5: factoryinsight
To make the relevant data from digital shadow available we need to provide new REST API’s. factoryinsight is the microservice doing that task. It accepts specific requests, accesses the timescale database and returns the data in the desired format.
Implemented functionality for digital shadow
The following function returns all uniqueProducts for that specific asset in a specified time range. One datapoint contains one childUID, AID and all parentAID’s regarding the asset. All uniqueProductTags and uniqueProductTagStrings (value and timestamp) for the childUID are returned to the same datapoint.
get /{customer}/{location}/{asset}/uniqueProductsWithTags
from <timestamp1> to <timestamp2> (in RFC 3999 Format).
Implemented logic of factoryinsight to achieve the functionality
Get all productUID’s and AID’s from uniqueProductTable within the specified time and from the specified asset.
Get all parentUID’s from the productInheritanceTable for each of the selected UID’s.
Get the AID’s for the parentUID’s from the uniqueProductTable.
Get all key, value pairs from the productTagTable and productTagStringTable for the in step 1 selected UID’s.
Return all parent AID’s under the column name of the corresponding parent productID’s. Return the child AID and UID. Return the productTag and productTagString values under the column name of the corresponding valueNames.
Step 6: SQL Database to connect to Tableau server
This is currently not included in the stack
For the digital shadow functionality we need to give the tableau server access to the data. Because the tableau server can’t directly connect to the REST API, we need to either use a database in between, or a tableau web data connector. We were advised against the tableau web data connector (general info about tableau webdata connectors: https://help.tableau.com/current/pro/desktop/en-us/examples_web_data_connector.htm ).
Because of that we implemented a SQL database in combination with Node-RED. Node-RED requests data from the REST API in regular intervals and pushes it into the SQL database. From there on we can access the data with the Tableau server.
Industry Example
To test the digital shadow functionality and display its advantages we implemented the solution in a model factory.
This graphic displays the events and following MQTT messages, mqtt-to-postgresql receives.
Long term: planned features
We plan to integrate further functionalities to the digital shadow.
Possible candidates are:
multiple new REST API’s to use the digital shadow more flexible
detailed performance analysis and subsequent optimization to enable digital shadow for massive
production speed and complexity
2.6 - Open source in Industrial IoT: an open and robust infrastructure instead of reinventing the wheel.
How we are keeping up with the established players in Industrial IoT and why we believe the United Manufacturing Hub is changing the future of Industrial IoT and Industry 4.0 with the help of Open Source.
Image author: Christopher Burns from Unsplash
How do we keep up with the big players in the industry despite limited resources and small market share? The best way to do this is to break new ground and draw on the collective experience of organizations and their specialists instead of trying to reinvent the wheel.
The collaborative nature of open source enables companies and individuals alike to turn their visions into reality and keep up with established players such as Siemens, Microsoft, and Rockwell, even without a large number of programmers and engineers. This is the path we are taking at United Manufacturing Hub.
Open source software has long since outgrown the insider stage and has become a veritable trend that is becoming the standard in more and more industries. Many, in the IT world, common and intensively used applications (e.g. Kubernetes, TensorFlow, f-prime by NASA1) have nowadays emerged in a collaborative approach and are available for free.
Open-Source on Mars: the Mars helicopter Ingenuity, relies heaviliy on open-source components like f-prime. Image author: JPL/NASA
Typically, these applications are not yet ready for production or Industry 4.0 use. Some, such as Grafana, are intended for completely different industries (Observability & Monitoring).
However, the source code of these software projects is freely accessible to everyone and can be individually adapted to specific needs. Thus, the application in the Industrial IoT is also no problem. In part, those applications were programmed over decades 2 by several thousand developers and are continuously developed further 3.
The status quo
Today, it is common to develop proprietary Industrial IoT programs and software platforms - the opposite of open source.
A reason behind is, that companies do not want to have foreign code written into their applications and they want to offer the customer a self-made, end-to-end solution.
It is common for a team of over 20 or even 30 people to be assigned to develop a dashboard or IoT gateway, with the focus on a pretty looking (usually self-branded) user interface (UI) and design. Existing open source solutions or automation standards are rarely built upon.
Self-developed, in-house architectures are often strongly influenced by company-specific know-how and therefore usually also favor the company’s own products and services in their interfaces.
The result: the wheel is often reinvented in both the software and hardware areas. The resulting architectures create a lock-in effect that leads to a dependency of the manufacturing companies on their software and hardware suppliers.
Reinventing the wheel: The software world
In our opinion, good examples in the category “reinvented the wheel” from the software world are:
The bulk of Industrial IoT platforms that are claiming to be a “one-stop solution.” Such platforms are trying to cover every aspect from data acquisition, over processing, to visualization with in-house solutions (instead of relying on established technologies and just filling the gaps in the stack).
Both Grafana and node-red are highly professional solutions in their respective fields, which have already been used in various software projects for several years. Orchestrating such specialized applications means that offered and tested solutions can be put to good use.
Reinventing the wheel: The hardware world
There are numerous examples in the Industrial IoT hardware world where there is a conscious or unconscious deviation from established industry standards of the automation industry.
We have particularly noticed this with vendors in the field of Overall Equipment Effectiveness (OEE) and production overviews. Although they usually have very good dashboards, they still rely on self-developed microcontrollers combined with consumer tablets (instead of established automation standards such as a PLC or an industrial edge PC) for the hardware. In this case, the microcontroller, usually called IoT gateway, is considered a black box, and the end customer only gets access to the device in rare cases.
The advantages cannot be denied:
the system is easy to use,
usually very inexpensive,
and requires little prior knowledge.
Unfortunately, these same advantages can also become disadvantages:
the house system integrator and house supplier is not able to work with the system, as it has been greatly reduced for simplicity.
all software extensions and appearing problems, such as integrating software like an ERP system with the rest of the IT landscape, must be discussed with the respective supplier. This creates a one-sided market power (see also Lock-In).
Another problem that arises when deviating from established automation standards: a lack of reliability.
Normally, the system always need to work because failures lead to production downtime (the operator must report the problem). The machine operator just wants to press a button to get a stop reason or the desired information. He does not want to deal with WLAN problems, browser update or updated privacy policies on the consumer tablet.
The strongest argument: Lock-In
In a newly emerging market, it is especially important for a manufacturing company not to make itself dependent on individual providers. Not only to be independent if a product/company is discontinued but also to be able to change providers at any time.
Particularly pure SaaS (Software-as-a-Service) providers should be handled with caution:
A SaaS offering typically uses a centralized cloud-based server infrastructure for multiple customers simultaneously. By its very nature, this makes it difficult to integrate into the IT landscape, e.g., to link with the MES system installed locally in the factory.
In addition, a change of provider is practically only possible with large-scale reconfiguration/redevelopment.
Lastly, there is a concern regarding the data ownership and security of closed systems and multiple SaaS offerings.
Basically, exaggerating slightly to make the point, it is important to avoid highly sensitive production data with protected process parameters getting to foreign competitors.
One might think that the manufacturing company is initially entitled to all rights to the data - after all, it is the company that “produced” the data.
In fact, according to the current situation, there is no comprehensive legal protection of the data, at least in Germany, if this is not explicitly regulated by contract, as the Verband der deutschen Maschinenbauer (VDMA) (Association of German Mechanical Engineering Companies) admits 4.
Even when it comes to data security, some people feel queasy about handing over their data to someone else, possibly even a US startup. Absolutely rightly so, says the VDMA, because companies based in the USA are obliged to allow US government authorities access to the data at any time 5.
An open source project can give a very good and satisfactory answer here:
United Manufacturing Hub users can always develop the product further without the original developers, as the source code is fully open and documented.
All subcomponents are fully open and run on almost any infrastructure, from the cloud to a Raspberry Pi, always giving the manufacturing company control over all its data.
Interfaces with other systems are either included directly, greatly simplifying their development, or can be retrofitted themselves without being nailed down to specific programming languages.
Unused potential
In the age of Industry 4.0, the top priority is for companies to operate as efficiently as possible by taking full advantage of their potential.
Open source software, unlike classic proprietary software, enables this potential to be fully exploited. Resources and hundreds of man-hours can be saved by using free solutions and standards from the automation industry.
Developing and offering a proprietary dashboard or IoT gateway that is reliable, stable, and free of bugs is wasting valuable time.
Another hundred, if not a thousand, man-hours are needed until all relevant features such as single sign-on, user management, or logging are implemented. Thus, it is not uncommon that even large companies, the market leaders in the industry, do not operate efficiently, and the resulting products are in the 6-to-7-digit price range.
But the efficiency goes even further:
Open source solutions also benefit from the fact that a community is available to help with questions. This service is rarely available with proprietary solutions. All questions and problems must be discussed with the multi-level support hotline instead of simply Googling the solution.
And so, unfortunately, most companies take a path that is anything but efficient. But isn’t there a better way?
United Manufacturing Hub’s open source approach.
Who says that you have to follow thought patterns or processes that everyone else is modeling? Sometimes it’s a matter of leaving established paths, following your own convictions, and initiating a paradigm shift. That is the approach we are taking.
We cannot compete with the size and resources of the big players. That is why we do not even try to develop in one or two years, with a team of 20 to 30 programmers what large companies have developed in hundreds of thousands of hours.
But that’s not necessary because the resulting product is unlikely to keep up with the open source projects or established automation standards. That is why the duplicated work is not worth the struggle .
The open source software code is freely accessible and thus allows maximum transparency and, at the same time, security. It offers a flexibility that is not reached by programs developed in the traditional way. By using open source software, the United Manufacturing Hub is taking an efficient way of developing. It allows us to offer a product of at least equal value but with considerably fewer development costs.
Example OEE dashboard created in Grafana
Simplicity and efficiency in the age of Industrial IoT.
At United Manufacturing Hub, we combine open source technologies with industry-specific requirements. To do this, we draw on established software such as Docker, Kubernetes or Helm 1 and create, for example, data models, algorithms, and KPIs (e.g. the UMH data model, the factoryinsight and mqtt-to-postresql components) that are needed in the respective industries.
By extracting all data from machine controls (OPC/UA, etc.), we ensure the management and distribution of data on the store floor. Also, if additional data is needed, we offer individual solutions using industry-specific certified sensor retrofit kits, for example, at a steel manufacturer. More on this in one of the later parts of this series.
Summary
Why should we reinvent the wheel when we can focus our expertise on the areas we can provide the most value to our customers?
Leveraging open source solutions allow us to expose a stable and robust infrastructure that enables our customers to meet the challenges of Industrial IoT.
Because, in fact, manufacturing and Industrial IoT is not about developing new software at the drop of a hat. It is more about solving individual problems and challenges. This is done by drawing on a global network of experts who have developed special applications in their respective fields. These applications allow all hardware and software components to be quickly and easily established in the overall architecture through a large number of interfaces.
Leitfaden Datennutzung. Orientierungshilfe zur Vertragsgestaltung für den Mittelstand. Published by VDMA in 2019. ↩︎
*Digitale Marktabschottung: Auswirkungen von Protektionismus auf Industrie 4.0 * Published by VDMA’s Impulse Foundation in 2019. ↩︎
2.7 - An introduction into certificates and secure communication in IoT for normal people
This article explains the two fundamental approaches to encrypting your messages with your IoT devices, from passwords (symmetric) to certificates (asymmetric).
Introduction
Newcomers to IT security are often confused about the difference between passwords and certificates. This article gives newcomers to IT security an introduction into the topic of secure communication with IoT devices. The focus is to explain the two fundamental methods to ensure that nobody else can read your communication, namely symmetric (e.g., passwords) and asymmetric encryption (e.g., certificates).
The need to ensure that nobody else than you and the recipient can read messages exist for thousands of years and has always been a battle between people creating encryption methods and so-called code-breakers, that focus on trying to decipher other messages. Wars have been started, won and lost because of a knowledge advantage gained through secure communications (see also ENIGMA or Zimmerman-Telegram).
In today’s world, the topic of secure communication is more relevant than ever: companies want to protect their intellectual knowledge from other companies and want to protect themselves from Ransomware attacks.
In IoT or Industrial IoT it is important as well:
How to ensure that the data send to the cloud from my microcontroller or PLC 1 is not read or modified?
How to push updates onto my PLCs or IoT devices and prevent that someone else modifies or pushes their own updates?
What are these certificates that are required for a connection to Azure IoT Hub, an MQTT broker 2, or any other API?
Let’s have fun and deep dive in!
Symmetric encryption
Let’s start with the simpler method of both approaches: symmetric encryption. In cryptography, the field of encryption, methods are always explained using Alice, Bob, and Mallory. All of them (there are way more, see also the Wikipedia article on Alice and Bob) are fictional characters. Alice usually initiates the communication and wants to send a message to Bob without any third party like Mallory being able to read it.
Alice, Bob, and Mallory
In IoT, Alice could be a Raspberry Pi with a temperature sensor that is trying to send a message to Bob, who is a service running in the cloud. In Industrial IoT, Alice could be A PLC and Bob an Industrial IoT platform.
Alice, Bob, and Mallory practically
Before Alice and Bob can communicate with each other using a symmetric encryption they need to do two things first:
align on the encryption method
align on the secret, also called password
Let’s explain symmetric encryption using Alice and Bob and one of the oldest, yet, one of the most famous encryptions, Caesar’s cipher. This encryption technique is named after Julius Caesar’s who used it for his private correspondence.
before their first messages, Alice and Bob need to align to use this cipher and then additionally to use a number between 1-26 (which is also called a secret) first.
When Alice wants to send a message to Bob, Alice needs to encrypt the plaintext using the cipher and the secret. For Caesar’s cipher, it works by shifting the alphabet x characters to the right (and x is the chosen number from above, the secret).
Caesar’s cipher
For example, a right shift of 3 is resulting in the replacement of the letter A with D. Alice then sends the encrypted message to Bob. Bob then reverts the encryption by shifting the alphabet x characters back to the left, so a D gets converted back into an A. Now Bob can read the plaintext.
Theoretically, a third party like Mallory could not decrypt the encrypted message because Mallory doesn’t know the secret. You might ask now: “But hey, couldn’t Mallory just try out all numbers?” and you would be right. Having a secret or password that can only be between the numbers 1-26 is considered unsafe, but please keep in mind that this was the technology in ancient Rome.
In the example above we could easily crack the secret by trying out, which is also called brute-forcing. To prevent that more and more encryption methods emerged and almost as many methods to crack them - from good old brute-forcing to deriving parts of the secret by knowing already knowing certain parts of the plaintext (this was how the Allied cracked the German ENIGMA during World War 2).
Today, we have encryptions like AES rooted in advanced mathematics and to which there are currently no shortcuts except brute-forcing. These keys usually have a fixed length, for example AES-128 has a 128-bit key, and are derived from an easy-to-remember password using a key derivation function.
Brute-forcing the keys is not feasible with current or foreseeable hardware. The KDF is calculation intensive, which prevents brute-forcing easy-to-remember passwords (as long as they are reasonable long), so this method is currently approved in the U.S. for government documents of the highest classification.
Small side-topic: to login into systems, for example websites, you need to enter a username and a password. In these cases the password is not stored on the server, it is only stored as a hash, so even if the database of the website gets compromised one cannot derive the password from the hash. If you are interested in this topic, we suggest reading the following article.
Brute-forcing can also be improved by not checking everything randomly, but by e.g. checking the top 1000 common passwords or words from a dictionary (= dictionary attack).
So how do you then create good passwords? Several years ago the best practice was to use smaller randomly generated passwords, which should change frequently. But humans are humans and most of them will just add a ! or 1 at the end of the password. So nowadays the recommendation is to go for lengthy and still easy rememberable passwords. if2s&sd5 is more insecure and way harder to remember than AlexanderIs25YearsOldAndLikesToHaveHisDeskClean.
Obligatory xkcd: https://xkcd.com/936/
But the approach itself, the symmetric encryption with a secret / password, has two fundamental flaws:
How can Alice and Bob exchange the secret without Mallory reading it? Meeting in person is nowadays not always an option.
How can Bob know that the message is coming from Alice and that nothing was changed or deleted by Mallory?
For this, some very smart people invented the approach of Public-Key-Encryption, also known as asymmetric encryption.
Asymmetric encryption
In 1977 Ronald Rivest, Adi Shamir, and Leonard Adleman published an alternative method, which they called after their first letter in their name: RSA 3
Little fun fact: a similar method had already been developed at the British Agency GCHQ in 1970, but was classified at that time was not published until 1997 4
Public-Key-Encryption
How does it work? For normal people here is a simple explaination from Computerphile on YouTube.
For the video haters out there, here is a small summary in text form:
Instead of using the same key for encrypting and decrypting you find a mathematic formula, that you can use separate keys, that belong together, but one cannot be derived from the other.
In practice, you have one key, the public key, that everyone knows, and which you use to encrypt a message. To decrypt the message you have to use another key, the so-called private key. When Alice generates these keys, she will keep the private key for herself and publish the public key to everyone.
She can also do the reverse: she can encrypt a message with her private key, so everyone will be able to decrypt it with her public key. Therefore, Bob can be sure that the message is coming from Alice (because only she has the private key and encrypt it like that).
The following text assumes that you have watched the video above (or read the small summary).
These public keys or private keys, that Computerphile is talking about, is what you would call “certificates”. They are usually stored as separate files in many different formats and file extensions. Here are some common ones for IoT:
What’s most important is the content of the file, so very often you can just change the file extension and load it in (most are PEM encoded, which looks like this). Sometimes the content differs (e.g., .pfx files or files in OpenSSH format are different than the usual PEM encoded .pem files), so you need conversion tools for that. If you are familiar with Google, it should be easy to find the correct tools and commands for that :)
Public Key Infrastructure (PKI)
But how can Alice ensure that the Public Key is actually from Bob and not from Mallory? What Computerphile did not talk about are Certificate Authorities (CA) and Intermediaries.
A CA is basically just a public-key pair that acts as a trusted instance. It can prove that a public key actually belongs to the person, computer, or website it claims to belong to. It does that by encrypting the public key of Bob with the CA’s private key (= Signing). These CA’s are mostly preinstalled on your operating system and help you in identifying whether your bank login information is actually send to the bank and not to someone else.
Practically, symmetric and asymmetric encryptions are used in parallel and combined with more advanced security topics like for example Diffie-Hellmann-Key-Exchange or Hashing. For more information on these advanced topics we can recommend to watch the YouTube videos from Computerphile (Diffie-Hellmann-Key-Exchange and Hashing)
The main reason for combining both technologies is computing speed. As a compromise, often a secret is generated and then shared between both parties using a (slow) asymmetric encryption. All the remaining messages are then encrypted using (fast) symmetric encryptions.
HTTPS
You are using these technologies already every day when you are visiting HTTPS websites like the one you are currently on (starts with https://). If you are sending it via HTTP, everyone on the internet can read your messages (see screenshot).
Source: Ibrahim Ali Ibrahim Diyeb, Anwar Saif, Nagi Ali Al-Shaibany,"Ethical Network Surveillance using Packet Sniffing Tools: A Comparative Study", International Journal of Computer Network and Information Security(IJCNIS), Vol.10, No.7, pp.12-22, 2018.DOI: 10.5815/ijcnis.2018.07.02 (https://www.researchgate.net/publication/326419957_Ethical_Network_Surveillance_using_Packet_Sniffing_Tools_A_Comparative_Study)
Security overview over our website docs.umh.app at 2021-10-18. Exact algorithms might change based on your browser
Example: MQTT and CA’s
In IoT we are relying on the same algorithms (SSL/TLS) that allow you to do safe online banking, with the exception that we cannot rely on external CA’s, but that we are creating it entirely ourselves.
We have a CA, a intermediate CA (to prevent using the root CA all the time), a server certificate for the MQTT broker and then client certificates for the MQTT clients.
Small recap: a CA is a trusted public-key pair. The intermediate CA is a public-key pair, that has been trusted by the root CA to take over the role of signing for a given time and in a given scope. A root CA is the “head” of the CA chain and can do everything.
The server certificate is the public side of the public-key pair created specifically for the MQTT broker, and where the intermediate CA signed it saying “yup, this is really the server mqtt.umh.app and it belongs to us. It can also act as a server.”.
The client certificates are signed by the intermediate CA as well with the permission to act only as clients (and not as server).
The public key of the root CA is embedded in all of our software, so that these devices know that, even if they never talked to each other before, they are communicating with whom they should communicate (e.g., a client connects to an MQTT broker and not to another client or someone entirely else).
This is the reason you need to copy these certificates onto the devices.
Example: viewing and analyzing the MQTT certificates
Last but not least, let’s take a look at the MQTT Public-Key infrastructure in detail. Specifically, let’s analyze these certificates.
In the folder deployment/factorycube-server/developmentCertificates/pki you find an example PKI infrastructure for factorycube-server. Almost all files are autogenerated with easy-rsa and you can actually ignore most of them. You can create your own PKI by following our tutorial
Let’s deep dive into 4 different certificates, to understand the structure of a PKI better:
ca.crt
issued/factorycube-server-vernemq.crt
issued/TESTING.crt
private/TESTING.key
Each certificate will be opened in xca, which is a graphical tool for creating and managing a PKI.
After importing the above certificates xca will look like this:
xca certificate section, after importing the above certificates
xca private key section, after importing the above certificates
ca.crt / “Easy-RSA CA”
This is the public-key of the root certificate (also known as root CA). We see that the certificate in the file ca.crt has the name “Easy-RSA CA”.
When opening it in an editor, it will look like gibberish:
ca.crt in raw format
After importing it to xca and clicking on it we can see all its details:
ca.crt in xca
ca.crt in xca - extensions
This includes:
Fingerprint (to easily identify a key)
Expiration (on the bottom)
That is is a certificate authority (see second screenshot CA:TRUE)
That it can be used to sign other certificates (see second screenshot X509v3 Key Usage)
In the overview panel in xca we see after importing the other certificates, that the others are shown under this certificate. xca automatically detects that ca.crt is the root CA and that is has signed the other certificates. It will therefore visualize it like that.
This is the public-key of the MQTT server key-pair and you can view the same information as in the certificate above with some exceptions:
factorycube-server-vernemq.crt in xca
factorycube-server-vernemq.crt in xca - extensions
It is signed by Easy-RSA CA (see Signature)
It is not a CA and can only sign (but cannot sign other certificates, see X509v3 Key Usage)
The whole purpose is beeing a Webserver for the domain factorycube-server-vernemq (if you access it under a different domain the client will say that the certificate is invalid). You can see it under the points X509v3 Extended Key Usage and X509v3 Subject Alternative Name
TESTING.crt / “TESTING”
This is a client key and is only allowed to be used as a client (see X509v3 Extended Key Usage).
TESTING.crt in xca - extensions
TESTING.key / “TESTING”
This is the private key for the certificate above.
TESTING.key in xca
Outlook
“But is it really secure?”, one might ask. Probably every security expert would answer that with “It depends” (if someone says yes, they are probably no expert), quickly followed by “it is the best that we have so far”. Technologies like these are officially recommended by basically all national and international security agencies, even for critical applications like energy or banking sectors (NIST for the USA, BSI for Germany).
With the current rise of quantum computing the current algorithms might be considered ineffective in some time into the future, but we already have post-quantum algorithms out there that can safely be integrated into the current infrastructures. If you are interested, you can visit the website of the current tournament of the NIST
TimescaleDB is better suited for the Industrial IoT than InfluxDB, because it is stable, mature and failure resistant, it uses the very common SQL as a query language and you need a relational database for manufacturing anyway
Introduction
The introduction and implementation of an Industrial IoT strategy is already complicated and tedious. There is no need to put unnecessary obstacles in the way through lack of stability, new programming languages, or more databases than necessary. You need a piece of software that you can trust with your company’s most important data.
We are often asked why we chose timescaleDB instead of InfluxDB. Both are time-series databases suited for large amounts of machine and sensor data (e.g., vibration or temperature).
We started with InfluxDB (probably due to its strong presence in the home automation and Grafana communities) and then ended up with timescaleDB based on three arguments. In this article, we would like to explain our decision and provide background information on why timescaleDB makes the most sense for the United Manufacturing Hub.
Argument 1: Reliability & Scalability
A central requirement for a database: it cannot lose or corrupt your data. Furthermore, as a central element in an Industrial IoT stack, it must scale with growing requirements.
TimescaleDB
TimescaleDB is built on PostgreSQL, which has been continuously developed for over 25 years and has a central place in the architecture of many large companies like Uber, Netflix, Spotify or reddit. This has created a fault-tolerant database that can scale horizontally across multiple servers. In short: it is boring and works.
InfluxDB
In contrast, InfluxDB is a relatively young startup that was funded at 119.9 M USD (as of 2021-05-03) but still doesn’t have 25+ years of expertise to fall back on.
On the contrary: Influx has completely rewritten the database twice in the last 5 years 12. Rewriting software can improve fundamental issues or add exciting new features. However, it is usually associated with breaking changes in the API and new unintended bugs. This results in additional migration projects, which take time and risk system downtime or data loss.
Due to its massive funding, we get the impression that they add quite a lot of exciting new features and functionalities (e.g., an own visualization tool). However, after testing, we noticed that the stability suffers under these new features.
In addition, Influx only offers the horizontally scalable version of the database in the paid version, which will scare off companies wanting to use it on a larger scale as you will be fully dependent on the provider of that software (vendor lock-in).
Summary
With databases, the principle applies: Better boring and working than exciting and unreliable.
We can also strongly recommend an article by timescaleDB.
Argument 2: SQL is better known than flux
The second argument refers to the query language, i.e., the way information can be retrieved from the database.
SQL (timescaleDB)
TimescaleDB, like PostgreSQL, relies on SQL, the de facto standard language for relational databases. Advantages: A programming language established for over 45 years, which almost every programmer knows or has used at least once. Any problem? No problem, just Google it, and some smart person has already solved it on Stack Overflow. Integration with PowerBI? A standard interface that’s already integrated!
Example SQL code to get the average memory usage for the last hour.
flux (InfluxDB)
InfluxDB, on the other hand, relies on the homegrown flux, which is supposed to simplify time-series data queries. It sees time-series data as a continuous stream upon which are applied functions, calculations and transformations3.
Problem: as a programmer, you have to rethink a lot because the language is flow-based and not based on relational algebra. It takes some time to get used to it, but it is still an unnecessary hurdle for those not-so-tech-savvy companies who already struggle with Industrial IoT.
From some experience, we can also say that the language quickly reaches its limits. In the past, we worked with additional Python scripts that extract the data from InfluxDB via Flux, then process it and then play it back again.
// Memory used (in bytes)
memUsed = from(bucket: "telegraf/autogen")
|> range(start: -1h)
|> filter(fn: (r) =>
r._measurement == "mem" and
r._field == "used"
)
// Total processes running
procTotal = from(bucket: "telegraf/autogen")
|> range(start: -1h)
|> filter(fn: (r) =>
r._measurement == "processes" and
r._field == "total"
)
// Join memory used with total processes and calculate
// the average memory (in MB) used for running processes.
join(
tables: {mem:memUsed, proc:procTotal},
on: ["_time", "_stop", "_start", "host"]
)
|> map(fn: (r) => ({
_time: r._time,
_value: (r._value_mem / r._value_proc) / 1000000
})
)
Example Flux code for the same SQL code.
Summary
In summary, InfluxDB puts unnecessary obstacles in the way of not-so-tech-savvy companies with flux, while PostgreSQL relies on SQL, which just about every programmer knows.
Finally, the argument that is particularly important for production: Production data is more relational than time-series based.
Relational data is, simply put, all table-based data that you can store in Excel in a meaningful way, for example, shift schedules, orders, component lists, or inventory.
Relational data. Author: AutumnSnow, License: CC BY-SA 3.0
TimescaleDB provides this by default through the PostgreSQL base, whereas with InfluxDB, you always have to run a second relational database like PostgreSQL in parallel.
If you have to run two databases anyway, you can reduce complexity and directly use PostgreSQL/timescaleDB.
Not an argument: Performance for time-series data
Often the duel between timescaleDB and InfluxDB is fought on the performance level. Both databases are efficient, and 30% better or worse does not matter if both databases are 10x-100x faster 4 than classical relational databases like PostgreSQL or MySQL.
Even if it is not important, there is strong evidence that timescaleDB is actually more performant. Both databases regularly compare their performance against other databases, and InfluxDB never compares itself to timescaleDB. However, timescaleDB has provided a detailed performance guide of influxDB.
Summary
Who do you trust more? The nerdy and boring, or the good-looking accountant, with 25 new exciting tools?
This is name of the customer (e.g: united-manufacturing-hub).
It can also be an abbreviation (e.g: umh) of the customer name.
Location
This is the name of the location, the sender belongs to.
It can be a physical location (aachen), an virtual location (rack-1), or any other unique specifier.
Machine
This is the name of the machine, the sender belongs to.
It can be a physical machine (printer-1), a virtual machine (vmachine-1), or any other unique specifier.
Event
Our kafka stack currently supports the following events:
addMaintenanceActivity
addOrder
addParentToChild
addProduct
addShift
count
deleteShiftByAssetIdAndBeginTimestamp
deleteShiftById
endOrder
modifyProducedPieces
modifyState
processValue
processValueFloat64
processValueString
productTag
productTagString
recommendation
scrapCount
startOrder
state
uniqueProduct
scrapUniqueProduct
Further information about these events can be found at the UMH datamodel site.
Routing
Below you can find an example flow of messages.
Edge PC
In this example, we have an edge pc, which is connected to multiple sensors and a camera.
It also receives data via MQTT, Node-RED and a barcode reader.
In our dataflow, we handle any IO-Link compatible sensor with sensorconnect, which reads IO-Link data and publishes it to Kafka.
Compatible cameras / barcode readers are handled by cameraconnect and barcode reader respectively.
Node-RED can be used to pre-process arbitrary data and the publish it to Kafka.
MQTT-Kafka-Bridge takes MQTT data and publishes it to Kafka.
Once the data is published to the Edge Kafka broker, other microservices can subscribe to the data and produce higher level data,
which gets re-published to the Edge Kafka broker.
Kafka bridge
This microservice can sit on either the edge or server and connects two kafka brokers.
It has a regex based filter, for sending messages, which can be bi- or uni-directional.
Also, it filters out duplicated messages, to prevent loops.
Every bridge adds an entry to the kafka header, to identify the source of the message and all hops taken.
Server
On the server, we have two microservices listening for incoming data.
Kafka-to-blob is a microservice which listens to Kafka and publishes the data to blob storage, in our case Minio.
Kafka-to-postgresql is a microservice which listens to Kafka and publishes the data to a PostgreSQL database,
with timescale installed.
Guarantees
Our system is build to provide at-least-once delivery guarantees, once a message first enters any kafka broker,
except for high throughput data (processValue, processValueFloat64, processValueString).
Every message taken out of the broker, will only get committed to the broker, once it has been processed, or
successfully returned to the broker (in case of an error).
This section is an overview over the various showcases that we already did. It provides for every showcase a quick summary including a picture. More details can be found in the subsequent documents.
Metalworking industry
Flame cutting & blasting
Retrofitting of 11 flame cutting machines and blasting systems at two locations using sensors, barcode scanners and button bars to extract and analyze operating data.
Identification of the optimization potential of two CNC milling machines
Two-month analysis of CNC milling machines and identification of optimization potentials. Automatic data acquisition coupled with interviews of machine operators and shift supervisors revealed various optimization potentials.
Retrofitting of weaving machines for OEE calculation
Retrofitting of weaving machines that do not provide data via the PLC to extract operating data. Subsequent determination of the OEE and detailed breakdown of the individual key figures
Retrofit of a bottling line for different beer types. Focus on the identification of microstops causes and exact delimitation of the bottleneck machine.
Retrofit of a Japanese pharmaceutical packaging line
Retrofit of a Japanese pharmaceutical packaging line for automatic analysis of microstop causes as well as to relief the machine operators of data recording.
Identification of optimization potential in the context of the COVID-19 crisis
Use of the factorycube for rapid analysis of bottleneck stations. The customer was thus able to increase the throughput of critical components for ventilators within the scope of COVID-19.
Duration and causes of machine downtimes are unclear
Heterogeneous machinery and machine controls from different manufacturers
Only minimal connection of the machines to the ERP system
Manual input of production data into the ERP system
Machine controls are partially locked by manufacturers
Machine controls use different protocols
Reliable production planning and quotation generation not possible
No data on past and current machine utilization available
Quotation preparation is done with theoretical target times, as no information about actual times is available
Solution
Integration
TODO: #68 add integration for flame cutting
Installed hardware
factorycube
factorycube sends the collected production data to the server. See also [factorycube].
Gateways
Gateways connect the sensors to the factorycube.
Models:
ifm AL1352
ifm AL1350
Light barriers
Light barriers are installed on cutting machines and are activated when the cutting head is lowered and in use. Used to measure machine conditions, cycle times and piece counts.
Models:
ifm O5D100 (Optical distance sensor)
ifm O1D108 (Optical distance sensor)
Vibration sensors
Vibration sensors are installed on the beam attachments and detect the machine condition via vibration. Used to measure machine conditions.
Model:
ifm VTV122 (vibration transmitter)
Button bar
Button bar is operated by the machine operator in case of a machine standstill. Each button is assigned a reason for standstill. Used to identify the causes of machine downtime.
Barcode scanners are used to scan production orders, which contain the target times. Used to scan target times for target/actual comparison.
Model:
Datalogic PowerScan PD9531
Datalogic USB Cable Straight 2m (CAB-438)
Implemented dashboards
The customer opted for our SaaS offering. We created the following dashboards for the client.
Navigation
Default navigation options from Grafana, which we modified to allow custom menus.
Customizable menu lets you quickly navigate between dashboards
In the time selection you can adjust the times for the current dashboard
Plant-manager dashboard
Dashboard for the plant manager / shift supervisor, which gives an overview of the production in the factory
For each machine the current machine status
For each machine, the overall equipment effectiveness / OEE for the selected time period
For each machine, a timeline showing the machine statuses in color
Overview of all orders, including target/actual deviation and which stop reasons, including setup times, occurred during the order
Machine deep dive
Dashboard for the machine operator / shift supervisor, which displays the details for a machine
The current machine status with time stamp
The overall equipment effectiveness / OEE for the selected time period, including trend over time
An overview of the accumulated duration of each stop reason
A timeline where the machine states are color coded
A timeline where the shifts become visible
A timeline where the orders are displayed
Overview of all orders, including target/actual deviation and which stop reasons, including setup times, occurred during the order
Overview of the number of individual stop reasons
Cross-factory dashboard
Dashboard for the cross-factory manager, who can use this to obtain an overview of the sites
The overall equipment effectiveness / OEE for the selected time period for all machines.
The minimum overall equipment effectiveness / OEE for the selected time period for machine type A.
The average overall equipment effectiveness / OEE for the selected time period for machine type A
The maximum overall equipment effectiveness / OEE for the selected period for machine type A
Overview of all orders, including target/actual deviation and which stop reasons, including setup times, occurred during the order
Export function as .csv
3.2 - Brewery
This document describes the brewery use case
Profile
category
answer
Industry
Brewery
Employees
~150
Number of retrofitted sites
1
Project duration
3 months
Number of retrofitted machines
8
Types of machines retrofitted
Entire filling line (filler, labeler, palletizer, etc.)
Photos
Challenges
Lack of transparency about production processes
Duration and causes of machine downtimes are unclear
High proportion of smaller microstops with unknown cause
Exclusively reactive maintenance, as data on the condition of the components is lacking
Moving bottleneck
Since the production process is highly interlinked, a stoppage of a single machine can lead to a stoppage of the entire line
The problem machine “bottleneck machine” is difficult to identify, as it can shift during a shift and is difficult to see with the eye
High effort to collect data as part of the introduction of a continuous improvement process
Changeover times must be recorded manually with a stop watch and are still not very standardized
No data on past and current machine utilization available
Maintenance actions recorded manually, no automatic system to log, store and visualize error codes from machine
Solution
Integration
At the beginning, a “BDE entry program” was carried out together with a lean consulting to identify optimization potentials and to present our solution. For this purpose, the [factorycube] was installed at the filler within a few hours in combination with the tapping of electrical signals from the control system and button strips. A connection of the PLC interfaces was initially out of the question due to time and cost reasons. After the customer decided on a permanent solution, the factorycube was dismounted.
All machines have been equipped with the “Weihenstephaner Standards”, a protocol commonly used in the German brewery industry and were already connected within a machines network. Therefore, the installation was pretty straightforward using our enterprise plugin for that protocol and one central server.
Installed hardware
Server
Implemented dashboards
The customer opted for our SaaS offering. We created the following dashboards for the client.
Navigation
Default navigation options from Grafana, which we modified to allow custom menus.
Customizable menu lets you quickly navigate between dashboards
In the time selection you can adjust the times for the current dashboard
Plant-manager dashboard
Dashboard for the plant manager / shift supervisor, which gives an overview of the production in the factory
For each machine the current machine status
For each machine, the overall equipment effectiveness / OEE for the selected time period
For each machine, a timeline showing the machine statuses in color
Performance cockpit
Dashboard for the supervisor to get an overview of the machine
The current machine status
The overall equipment effectiveness / OEE for the selected time period, including trend over time
The average changeover time
The average cleaning time
A timeline where the machine states are color coded
A timeline where the shifts become visible
A timeline with the machine speed
Overview of the number of individual stop reasons, excluding technical defects as they are not relevant for the shift
Maintenance cockpit
Dashboard for the head of maintenance to get an overview of the machine
The current machine status
The overall equipment effectiveness / OEE for the selected time period, including trend over time
The MTTR (mean time to repeair), an important key figure for maintenance
The MTBF (mean time between failures), an important key figure for maintenance
A timeline where the machine states are color coded
A timeline where the process value “bottle lock open/close” is visualized. This helps the manager of the maintenance to isolate the cause of a problem more precisely.
A timeline with the machine speed
An overview of the accumulated duration of each stop reason, that is relevant for maintenance
Overview of the number of individual stop reasons, that is relevant for maintenance
3.3 - Semiconductor
This document describes the semiconductor use case
Profile
category
answer
Industry
Semiconductor industry
Employees
>1000
Number of retrofitted sites
1
Project duration
2 months
Number of retrofitted machines
1
Types of machines retrofitted
Dispensing robot
Photos
Challenges
Increasing demand could not be fulfilled
the demand for the product, which was required for ventilators, was increasing over 1000% due to the COVID-19 crisis
the production was struggling to keep up with the ramp up
Production downtime needed to be avoided at all costs
production downtime would have meant not fulfilling the demand
A quick solution was needed
to meet the demand, the company needed a quick solution and could not accept months of project time
Solution
Integration
We were given a 2h time slot by the company to install the sensors, from the time we entered the factory until the time we left (including safety briefings and ESD compliance checks). With the help of videos, we got an overview beforehand and created a sensor plan. During this time slot, we used the machine operator’s break to install all the sensors and verified the data during the subsequent machine run. Through VPN we were able to access the device and fine-tune the configuration.
Installed hardware
factorycube
factorycube sends the collected production data to the server. See also [factorycube].
Gateways
Gateways connect the sensors to the factorycube.
Models:
ifm AL1352
Ultrasonic sensor
picture TODO
The ultrasonic sensor was used to measure whether the robot was currently moving and thus whether the machine was running.
Models:
TODO
Proximity sensor
Proximity sensors were used to detect if the product was ready for operator removal. Together with the ultrasonic sensors, we were able to measure whether the machine was standing because the machine operator had not removed the product and was therefore not available.
Models:
ifm KQ6005
Button bar
Button bar is operated by the machine operator in case of a machine standstill. Each button is assigned a reason for standstill. Used to identify the causes of machine downtime.
Model:
Self-made, based on Siemens TODO
Implemented dashboards
The customer opted for our SaaS offering and a additional analysis of the data.
Dashboard screenshot
The customer opted for SaaS solution and required only a very simple dashboard as most insights were gained from a detailed analysis. The dashboard includes the functionality to export data as .csv.
Additional analysis
The data was exported into .csv and then analyzed in Python & Excel. Together with interviews of the operators and supervisors we could extract multiple insights including optimization potential through alignment of the work processes and improvement of changeovers through Single-minute exchange of die (SMED).
3.4 - Cycle time monitoring in an assembly cell
This document describes the cycle time monitoring use case
Profile
An assembly cell was retrofitted to measure and optimize cycle times. Customizable textile wristbands are produced in the assembly cell.
Photos of the machines
Challenges
Lack of information about production performance
Cycle times are unknown
Bottleneck of the assembly cell cannot be identified
No information about productivity of individual employees
Piece counts are not documented
No comparison between target and actual performance
Lack of transparency about downtimes
Frequency and duration of downtimes of the assembly cell are not recorded
Causes of downtime are often unknown and not documented
Connection of assembly cell to conventional systems not possible
Sewing machines do not have machine controls that could be connected
Solution
Integration
TODO: #66 Add integration for assembly analytics
Installed hardware
factorycube
factorycube sends the collected production data to the server. See also [factorycube].
Gateways
Gateways connect the sensors to the factorycube.
Models:
ifm AL1352
Light barriers
Light barriers are installed on the removal bins and are activated when the employee removes material. Used to measure cycle time and material consumption.
Models:
ifm O5D100 (Optical distance sensor).
Proximity sensor
Proximity sensors on the foot switches of sewing machines detect activity of the process. Used to measure cycle time.
Models:
ifm KQ6005
Barcode scanner
The barcode scanner is used to scan the wristband at the beginning of the assembly process. Process start and product identification.
Model:
Datalogic PowerScan PD9531
Datalogic USB Cable Straight 2m (CAB-438)
Implemented dashboards
The customer opted for a combination of our SaaS offering with the building kit (and thus an on-premise option). The customer decided to go for PowerBI as a dashboard and connected it via the REST API with factoryinsight.
Used node-red flows
With the help of Assembly Analytics Nodes, it is possible to measure the cycle time of assembly cells in order to measure and continuously improve their efficiency in a similar way to machines.
Here is an exemplary implementation of those nodes:
There are 2 stations with a total of 4 cycles under consideration
Station 1 (AssemblyCell1):
1a: Starts with scanned barcode and ends when 1b starts
1b: Starts with a trigger at the pick to light station and ends when station 1a starts
Station 2 (AssemblyCell2):
2a: Starts when the foot switch at the 2nd station is pressed and ends when 2b starts
2b: Starts when the quality check button is pressed and ends when 2a starts.
Assumptions:
Unrealistically long cycle times are filtered out (cycle times over 20 seconds).
There is a button bar between the stations to end the current cycle and mark that product as scrap. The upper 2 buttons terminate the cycle of AssemblyCell1 and the lower ones of AssemblyCell2. The aborted cycle creates a product that is marked as a scrap.
Nodes explained:
Assembly Analytics Trigger: Cycles can be started with the help of the “Assembly Analytics Trigger” software module.
Assembly Analytics Scrap: With the help of the software module “Assembly Analytics Scrap”, existing cycles can be aborted and that produced good can be marked as “scrap”.
With the help of the software module “Assembly Analytics Middleware”, the software modules described above are processed into “unique products”.
This document describes the quality monitoring use case
Profile
A bottling line for filling water bottles was retrofitted with an artificial intelligence quality inspection system. With the help of a camera connected to an ia: factorycube, the bottles are checked for quality defects and sorted out by a pneumatic device in the event of a defect.
Photos of the machines
Challenges
Manual visual inspection causes high costs
Each individual bottle is checked for quality defects by an employee
One employee is assigned to each shift exclusively for quality inspection
Customer complaints and claims due to undetected quality defects
Various quality defects are difficult to detect with the naked eye and are occasionally overlooked
No data on quality defects that occur for product and process improvement
Type and frequency of quality defects are not recorded and documented
No data exists that can be analyzed to derive improvement measures for product and process optimization
Solution
Integration
TODO: #67 Add integration for DCC quality check
Installed hardware
factorycube
A machine learning model runs on the factorycube, which evaluates and classifies the images. See also [factorycube].
Gateways
Gateways connect the sensors to the factorycube.
Models:
ifm AL1352
Light barriers
A light barrier identifies the bottle and sends a signal to the factorycube to trigger the camera.
Models:
ifm O5D100 (Optical distance sensor)
Camera
A camera takes a picture of the bottle and sends it to the factorycube.
Models:
Allied Vision (Mako G-223)
Detectable quality defects
Automated action
As soon as a quality defect is detected the defect bottle is automatically kicked out by the machine.
3.6 - Pharma packaging
This document describes the pharma packaging use case
Profile
category
answer
Industry
pharma industry
Employees
Number of retrofitted sites
Project duration
Number of retrofitted machines
Types of machines retrofitted
TODO: #70 add pharma packaging case
3.7 - Weaving
TODO
Profile
category
answer
Industry
Employees
Number of retrofitted sites
Project duration
Number of retrofitted machines
Types of machines retrofitted
TODO: #71 add weaving case
3.8 - CNC Milling
This document describes the CNC milling use case
TODO #65
4 - Tutorials
This section has tutorials and other documents, that do not fit into the other categories.
4.1 - FAQ
This document gives answers to the frequently asked questions
I cannot login into ia: factoryinsight / Grafana although I’ve entered my credentials correctly several times
This category contains tutorials for using Azure, e.g., Azure IoT Hub
4.2.1 - Connecting with Azure IoT Hub
Azure IoT Hub enabled highly secure and reliable communication between IoT applications and the devices it manages. In this article it is described how one can connect the United Manufacturing Hub with Azure IoT hub.
In this tutorial we are leveraging the fact that Azure IoT Hub is nothing but a MQTT broker with benefits (e.g., device management or REST API). We will therefore create a connection via MQTT.
By default Microsoft recommends using symmetric encryption as it is more easier to implement, but they say themselves that asymmetric encryption is more secure. If you are using symmetric encryption (Username / Password authentification, no certificates) there might be some steps that are different for you
All information for your Azure IoT Hub device ready:
Hostname of the Azure IoT Hub: e.g., contoso-test.azure-devices.net
can be unrealiable, e.g., when the user accidentally adds an unstable external plugin, which takes down Node-RED
does not buffer data in case of internet outages
Warning
Please do not use the official Azure IoT Hub plugin for Node-RED (node-red-contrib-azure-iot-hub). Our customers and we found this plugin to be unrealiable and we therefore recommend for using the MQTT out node.
We will do this in three steps. You can also download our example flows.json, where you just need to change your authentification parameters (certificates or username / password, host, etc.) and you are ready to go.
Step 1: Subscribing to all messages
MQTT-In node in Node-RED
As a first step we take all relevant data out of the MQTT broker and thereby starting a Node-RED flow. Add a MQTT-In node. The MQTT broker details should already been set if you have used our recommended installation method (see also Getting Started).
If not, you can connect on factorycube-edge with the URL: factorycube-edge-emqxedge-service under the port 1883.
In the Topic section you can specify what messages from which asset you want to send to Azure IoT Hub using wildcards. If you want to send all messages you can use ia/# as a topic. If you just want to send process values of all machines in plant3 of customer factoryinsight you can use:
ia/factoryinsight/plant3/+/processValue
MQTT-In node in Node-RED - detailled view
You can test the flow by adding a debug node and connecting it the MQTT-In node with it.
Step 2: Changing the payload to contain payload and topic
Azure by default does not support the topic structure of the United Manufacturing Hub. As it contains important infortmation, e.g., the asset that the message is about, we will put that into the payload as well.
json and function node in Node-RED
Add a json node AND a function node with the following content and connect it to the MQTT-In node that we previously created:
Do not forget to set the topic as well (see also the tutorial above):
MQTT-out node - topic and server
MQTT-out node - MQTT broker (1/2)
MQTT-out node - MQTT broker (2/2)
Connect it with the previously created function node and test it. If everything went well the messages should now appear in the device explorer in Azure IoT Hub.
The entire flow
Step 4: check whether the messages appear in Azure IoT Hub using the device explorer
Finding the iothubowner SAS token in Azure
Open the Azure IoT Explorer and connect using your iothubowner SAS token (see image above). If everything went well you should see messages coming up (assuming that there are messages in the local MQTT broker. If not, try to send some debug messages to test the connection).
Furthermore, please set username, host, port, etc. accordingly
If you do not know where to get the certificates or the connection parameters, please go back to Prerequisites
What to do now?
Now we send the data to Azure IoT Hub. But what to do now? Here are some ideas:
You could install the server component of the United Manufacturing Hub factorycube-server on Azure Kubernetes Service. You would then be able to immediatly start creating dashboard while leveraging Azure’s scalability
You could use Azure’s propertiary services to create a full IIoT infrastructure from scratch. However, this is time intensive as you need to build e.g., your own data model, so we only recommend it for large companies with their own IIoT team. A good next step to proceed from here would be the tutorial Add an IoT hub event source to your Azure Time Series Insight environment
4.2.2 - Setting up Azure IoT Hub
Azure IoT Hub enabled highly secure and reliable communication between IoT applications and the devices it manages. This article explains how to set it up and links to the official Microsoft Azure documentation. It provides additional information that are required for mechanical engineers working in Industrial IoT.
When connecteing with the United Manufacturing Hub we will substitute the VerneMQ broker (see also architecture) in the United Manufacturing Hub with Azure IoT Hub.
Step 1: Create the necessary PKI (can be skipped if using symmetric encryption)
We recommend to skip this step if you are new to IT / Public-Key-Infrastructure and just want to see some data in Azure. We strongly advise to not use in production and instead follow the steps mentioned in this step.
This tutorial is assuming zou are using ubuntu and have installed easy-rsa using sudo apt-get install easyrsa
Initially setting up the infrastructure
Create a new directory and go into it, e.g.
mkdir ~/mqtt.umh.app/
cd ~/mqtt.umh.app/
Enable batch mode of easyrsa with export EASYRSA_BATCH=1
Setup basic PKI infrastructure with /usr/share/easy-rsa/easyrsa init-pki
Copy the default configuration file with cp /usr/share/easy-rsa/vars.example pki/vars and edit it to your liking (e.g. adjust EASYRSA_REQ_… and CA and cert validity)
Build the CA using export EASYRSA_REQ_CN=YOUR_CA_NAME && /usr/share/easy-rsa/easyrsa build-ca nopass. Replace YOUR_CA_NAME with a name for your certificate authority (CA), e.g., UMH CA
Create the server certificate by using the following commands (exchange mqtt.umh.app with your domain!):
/usr/share/easy-rsa/easyrsa gen-req mqtt.umh.app nopass
/usr/share/easy-rsa/easyrsa sign-req server mqtt.umh.app
If you need to generate a certificate for an IP instead of a domain name, use the following command instead (exchange 0.0.0.0 with your IP):
/usr/share/easy-rsa/easyrsa --subject-alt-name="IP:0.0.0.0" gen-req 0.0.0.0 nopass
/usr/share/easy-rsa/easyrsa sign-req server 0.0.0.0
Copy the private key pki/private/mqtt.umh.app.key and the public certificate pki/issued/mqtt.umh.app.crt together with the root CA pki/ca.crt to the configuration of the MQTT broker.
Adding new clients
Create new clients with following commands (remember to change TESTING with the planned MQTT client id):
The UMH stack features a sophisticated system to be integrated into any enterprise network. Additionally, it forces multiple barriers against attacks by design. This document should clear up any confusion.
factorycube
The factorycube (featuring the RUT955) consists out of two separate networks:
internal
external
The internal network connects all locally connected machines, sensors and miniPCs with each other. The external network is “the connection to the internet”. The internal network can access the external network, but not the other way around, except specifically setting firewall rules (“port forwarding”).
Example components in internal network
Laptop for setting up
Router
miniPC
ifm Gateways
Ethernet Cameras
Example components in external network
Router (with its external IP)
the “Internet” / server
4.3.3 - How to install an operating system from a USB-stick
This article explains how to install an operating system from a bootable USB-stick.
Press the button to go into the boot menu. This step is different for every hardware and is described in the hardware manual. If you do not want to look it up you could try smashing the following buttons during booting (the stuff before the operating system is loaded) and hope for the best: F1, F2, F11, F12, delete
Once you are in the boot menu, select to boot from the USB-stick
4.3.4 - How to connect with SSH
This article explains how to connect with an edge device via SSH
There will be a warning saying that the authenticity of the host can’t be established. Enter yes to continue with the connection.
Warning message: The authenticity of host 'xxx' can't be established.
Enter the password and press enter. The default password of the auto setup will be rancher.
Successfully logged in via SSH
4.3.5 - How to flash an operating system onto a USB-stick
There are multiple ways to flash a operating system onto a USB-stick. We will present you the method of using balenaEtcher.
Prerequisites
You need a USB-stick (we recommend USB 3.0 for better speed)
You need a OS image in the *.iso format. For k3OS you could choose for example this version
Steps
Download balenaEtcher: www.balena.io/etcher/
Insert USB-stick and open balenaEtcher
Select downloaded *.iso by clicking on "Flash from file" (the sceeen might look different based on your operating system)
Select the USB-stick by clicking on "Select target"
Select "Flash"
It will flash the image on the USB-stick
You are done!
These steps are also available as a YouTube tutorial from the user kilObit.
4.3.6 - Versioning in IT
This article explains how version numbers are typically structured in IT.
In IT Semantic Versioning has established itself as the standard to describe versions. It consists out of the format MAJOR.MINOR.PATCH, e.g., 1.0.0.
MAJOR is incremented when making incompatible API changes.
MINOR is incremented when you add functionality
PATCH is incremented when you make bug fixes
If the version is followed by a ‘-’ sign, then it means it is a pre-release and not stable yet. Therefore, the latest stable version means the highest version available that is not a pre-release / has no ‘-’ sign.
Go to the folder /home/rancher/.ssh and edit the file authorized_keys
Add there your additional SSH key
4.4.2 - How to enable SSH password authentification in k3OS
This article explains how to enable the classic username / password authentification for SSH in k3os
DANGER: NOT RECOMMENDED FOR PRODUCTION! USE DEFAULT BEHAVIOR WITH CERTIFICATES INSTEAD
By default, k3OS allows SSH connections only using certificates. This is a much safer method than using passwords. However, we realized that most mechanical engineers and programmers are overwhelmed with the creation of a public key infrastructure. Therefore, it might make sense to enable password authentication in k3OS for development mode.
Access the edge device via computer screen and keyboard and login with username rancher and rancher
Set the value PasswordAuthentication in the file /etc/ssh/sshd_config to yes and restart the service sshd. You can use the following command:
sudo vim /etc/ssh/sshd_config -c "%s/PasswordAuthentication no/PasswordAuthentication yes/g | write | quit"&& sudo service sshd restart
4.4.3 - How to fix certificate not yet valid issues
curl might fail and not download helm as the certificate is not yet valid. This happens especially when you are in a restricted network and the edge device is not able fetch the current date and time via NTP.
Issue 1
While executing the command export VERIFY_CHECKSUM=false && curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 && chmod 700 get_helm.sh && ./get_helm.sh on k3OS you might get a error message like this:
curl: (60) SSL certificate problem: certificate is not yet valid
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
Checking the time with date results in a timestamp from 2017. There are two possible solutions.
Possible solution 1: configure NTP
The time is not configured properly. It can happen that your NTP server is blocked (especially if you are inside a university network).
You can verify that by entering sudo ntpd -d -q -n -p 0.de.pool.ntp.org. If you get a result like this, then it is definitely blocked:
ntpd: '0.de.pool.ntp.org' is 62.141.38.38
ntpd: sending query to 62.141.38.38
Alarm clock
We recommend using the NTP server from the local university or ask your system administrator. For the RWTH Aachen / FH Aachen you can use sudo ntpd -d -q -n -p ntp1.rwth-aachen.de as specified here
Possible solution 2: set hardware clock via BIOS
Go into the BIOS and set the hardware clock of the device manually.
Issue 2
k3os reports errors, due hardware date being in the past. (ex: 01.01.2017). Example: during startup the k3s certificates are generated, however, it is still using the hardware time. Even after setting the time manually with NTP it wont let you connect with k3s as the certificates created during startup are not not valid anymore. Setting the time is not persisted during reboots.
Steps to Reproduce
Install k3os, without updating BIOS clock
Install UMH
helm will fail on Install factorycube-server step, due to outdated certificates.
Be careful: you need to host it on a HTTP server (not HTTPS) as you would get other certificate issues while fetching it.
4.4.4 - How to install k3OS
This article explains how to install k3OS on an edge device using the United Manufacturing Hub installation script.
Prerequisites
edge device with keyboard and computer screen successfully booted from a USB-stick (see also Installation)
you should see on the computer screen the screen below (it will automatically continue after 10 seconds with the installation, so do not worry if you only see it for some seconds)
boot menu of k3OS
Tutorial
Wait until k3OS is fully started. You should see the screen below:
k3OS installer fully booted
Enter rancher and press enter to login.
k3OS installer fully booted with rancher as username
You should now be logged in.
Pro tip: Execute lsblk and identify your hard drive (e.g. by the size). It will prevent playing “Russian Roulette” on a later step.
Now type in sudo k3os install to start the installation process.
entered `sudo k3os install`
You are now prompted to select what you want to install. Select 1 and press enter or just press enter (the stuff in brackets [] is the default configuration if you do not specify anything and just press enter).
Install to disk
At this step you may be asked to select your hard drive. One of the devices sda or sdb will be your hard drive and the other will be the USB stick you booted from.
If you don not know what your hard drive is, you will have to play “Russian roulette” and choose a device. If you find out later that you installed the hard drive on the USB stick by mistake, repeat the installation process and use the other device.
After that select y when you get asked for a cloud-init file
Configure system with cloud-init file
Now enter the URL of your cloud-init file, e.g. the one mentioned in the Installation guide.
Attention: The default keyboard of k3Os is American.
So just google the American keyboard and see where you can find the letters and characters used.
Press enter to continue.
Specify the cloud-init file
example (do not use this URL)
Confirm with y and press enter.
Confirm installation with `y`
Confirm installation with `y`
If the installation fails with not beeing able to fetch the cloud-init file check the URL and the network configuration
If the installation fails with expired or untrusted certificates (curl: (60) SSL certificate problem: certificate is not yet valid or similar), check out this guide.
The device will then reboot. You might want to remove the USB-stick to prevent booting from the USB-stick again.
If the following screen appears you did everything correct and k3OS was successfully installed.
4.5 - Node-RED
This category contains tutorials for Node-RED
4.5.1 - Fixing broken Node-RED flows
This tutorial shows how you can recover Node-RED flows that are stuck in an endless loop of crashing.
Check whether it says "true" and not true (use quotation marks!)
Press Save
Terminate the pod manually if necessary (or if you are impatient)
Node-RED should now start in safe mode. This means that it will boot, but will not execute any flows.
Do your changes, fix the Nodes
Do steps 1 - 5, but now set NODE_RED_ENABLE_SAFE_MODE to false
4.6 - Tulip
This category contains tutorials for Tulip (www.tulip.co)
4.6.1 - How to connect the United Manufacturing Hub to Tulip
Combine all shop floor data with work instructions from Tulip (www.tulip.co) using the UMH approach. This approach does not require opening ports and additionally allows to deeply integrate Tulip into the shopfloor IT. For more information on the shared value proposition and the use-case please check out also our blog article
Prerequisites
An active Tulip subscription.
The UMH platform installed and configured.
Available data extracted from sensors, industrial controllers or other shopfloor IT systems and available in the MQTT broker or Apache Kafka.
About Tulip
Tulip is a frontline operations platform, which is deployed in manufacturing and logistic environments, bridging the interaction between people, machines, devices, and systems.
Tulip’s most widely documented use case (in different and extensive channels such as “Tulip Libraries”, “Tulip University” and “Tulip Community”) focusses on their hardware “I/O Gateway” and “Edge IO” developed for the deployment of work instructions running in the Tulip Cloud. These work instructions can be authorized with additional hardware that connects Tulip approved devices via USB port and sensors to GPIO ports. The main function of the gateway is to send the data to the Tulip app.
Tulip Edge IO with other devices. Source: https://tulip.co/products/edge-io/
Within the Tulip platform, users can create applications using a a no-code interface. The operations are triggered by various sensors and forwarded by the hardware to the Tulip Cloud.
Work instructions in an assembly cell
In addition to apps, users can create “Tulip Analyses” to monitor KPIs and trends and provide insights to the production team. Interactive dashboards can also be created to monitor KPIs of interest.
The United Manufacturing Hub (UMH) provides the necessary IT/OT infrastructure not only to connect real-time manufacturing data to Tulip, but to also share it with other solutions and applications. For example, data coming from sensors, barcode readers, cameras and industrial controllers can now be used for real-time stream or batch processing by other use cases such as predictive maintenance, quality management systems, track & trace / digital shadow, and supervisor dashboards. These use cases can then run different clouds such as AWS or Azure.
Additionally, this allows for adding data sources to Tulip that are yet to be officially supported (e.g., IO-Link sensors). To access local data sources, a Tulip Connector Host and/or opening ports is no longer required, which should align with the IT security department’s requirements.
Approach
In this tutorial we will choose the “Tulip Machine Attributes API” approach. Other method is using the Tulip Connector Host. The approach chosen here has the advantage that it does not requiring opening ports and is therefore more easily integrated into enterprise IT.
Perform the following steps as a Tulip administrator. First, in the Tulip platform, go to your profile icon and click Settings. Then select the Bot tab and create one. For this project, only data should be read and displayed in the Tulip environment, therefore the scope must be set to “read-only”.
Tulip Bot showing the different scopes.
The API Key, Secret and Auth Header are displayed. Please store this information in a safe place.
API Key, secret and authentication header is shown.
Then you will need to link this Machine API to a Machine.
Create a Machine Type
Navigate to the tab “Shopfloor” –> “Machines” then change to the tab “Machine Types”. Here, click in the button “Create type”. For now, only change the name of the machine type.
Creating a Machine Type.
Create a machine
Go to the Machine Library tab and click the Create Machine button. Assign a name for your machine. Then select Tulip API as the data source. The Tulip interface will ask you to assign your machine to a station. If multiple stations use the data, we recommend selecting “None”. Finally, select the machine type you just created.
Creating a Machine.
Add Machine Attribute
Return to the Machine Type tab and navigate to the Machine Type you previously created. Click the edit button and scroll down and head to “Add Machine Attribute”.
Adding a Machine Attribute.
Here you are able to create new machine attributes or assign ones you created and used before.
Assign a name and a type to the Machine Attribute.
Having performed the previous steps, you are now able to to find the field mappings for each Machine Attribute. Go to the tab “Shopfloor/Machines”, select the machine name of interest, and click in the Configuration tab. There you’ll discover field mappings for each Machine Attributes. The “Attibute ID” and “Machine ID” will be used for setting up the connection in Node-RED.
Field mappings
Having performed the previous steps, the Tulip configuration is finalized. The next part of the article describes the configuration in Node-RED that is necessary for connecting the data acquisition layer performed in Node-RED with the Tulip platform.
Connecting to the Tulip Machines Attributes API using Node-RED
In the manufacturer’s environment, UMH installed an electrical cabinet (E-rack) with an Industrial PC (IPC). The IPC was connected to the industrial controller of the machine and reads several data tags such as power time, hydraulic pump time and power consumption, with a flow in Node-RED (see also the blog article mentioned at the beginning of the article)
Automatic saw with E-rack.
To connect to the Tulip Machine Attributes API, two nodes were used, namely a function node and a HTTP response.
Function and HTTP response nodes used.
For the function node, the message must contain the “Attribute ID” and “Machine ID” corresponding to the data tag being acquired from the industrial controller.
Content in the function node.
For the HTTP response node, a configuration with the Tulip platform is required. This information was acquired while creating the bot in the Tulip environment.
HTTP response node configuration.
Having set everything up, the connection between the Node-RED flow and the Tulip platform is now established. On the next and final part of the blog, it will be shown how to use the data for developing it in the Tulip app environment.
Using the data in Tulip app
Data coming from industrial controllers or sensors can be connected, leveraging the Machine Attribute created as any other Tulip device. For this, please navigate to the “Apps” tab and create a new app or select one you already created.
In the App editor, select to create a new trigger.
The trigger menu will appear, and by selecting:
When – Machine – Specific Machine
The Machine that was created in the previous steps will appear. Select it and in the outputs, select the corresponding Machine attributes. Select the one that concerns you and by doing so you can leverage the data extracted from the machine.
Digital Work instruction app with Machine Attribute API connectivity.
Summary
This approach makes it possible to connect additional data sources to Tulip and combine Tulip with a Unified Namespace. It also reduces implementation time by eliminating the need to open ports in the corporate firewall.
Interested? Then check out also the corresponding blog article
4.7 - United Manufacturing Hub
This category includes tutorial specific for the United Manufacturing Hub
4.7.1 - Certified devices
This section contains tutorials related to our commercial certified devices.
4.7.1.1 - How to use machineconnect
This document explains how to install and use machineconnect
Purpose
machineconnect is our certified device used for the connection of PLCs and installed in the switch cabinet of the machine.
The idea behind machineconnect is to protect the PLC and all components with an additional firewall. Therefore, it is not accessible from outside of machineconnect except explicitly configured in the firewall.
Features
Industrial Edge Computer
With DIN rail mounting and 24V
Vibration resistent according to IEC 60068-2-27, IEC 60068-2-64/ MIL-STD-810, UNECE Reg.10 E-Mark, EN50155
Increased temperature range (-25°C ~ 70°C)
Open source core installed and provisioned according to customer needs (e.g. MQTT certificates) in production mode (using k3OS)
Additional security layer for your PLC by using OPNsense (incl. Firewall, Intrusion Detection, VPN)
10 years of remote VPN access via our servers included
Physical installation
Attach wall mounting brackets to the chassis
Attach DIN Rail mounting brackets to the chassis
Clip system to the DIN Rail
Connect with 24V power supply
Connect Ethernet 1 with WAN / Internet
Connect Ethernet 3 with local switch (if existing). This connection will be called from now on “LAN”.
(optional, see connection to PLC. If skipped please connect the PLC to Ethernet 3) Connect Ethernet 2 with PLC. This connection will be called from now on “PLC network”.
Verify the installation by turning on the power supply and checking whether all Ethernet LEDs are blinking.
Connection to the PLC
There are two options to connect the PLC. We strongly recommend Option 1, but in some cases (PLC has fixed IP and is communicating with engine controllers or HMI and you cannot change the IP adresses there) you need to go for option 2.
Option 1: The PLC gets the IP via DHCP (recommended)
Configure the PLC to retrieve the IP via DHCP
Configure OPNsense to give out the same IP for the MAC-address of the PLC for LAN. Go to Services –> DHCPv4 –> LAN and add the PLC under “DHCP static mappings for this device”
Option 2: The PLC has a static IP, which cannot be changed
New method
This method should be quicker
Add a new interface for the PLC network
Add a new route with the target being the PLC IP and as gateway the automatically created gateway for the PLC (will not be shown by default, need to enter PLC_GW to be shown)
Make sure to activate the gateway:
Change NAT to “Hybrid outbound NAT rule generation” and add a NAT for PLC, Source the LAN network, Destination the PLC
Test if the PLC is reachable
That’s it
Old method
Adding a new interface for the PLC network, e.g. “S7”.
Adding a new gateway (see screenshot. Assuming 192.168.1.150 is the IP of the PLC and the above created interface is called “S7”)
Adding a new route (see screenshot and assumptions of step 2)
Changing NAT to “Manual outbound NAT rule generation” (see screenshot and assumptions of step 2)
Add firewall rule to the PLC interface (see screenshot and assumptions of step 2)
Add firewall rule to LAN allowing interaction between LAN network and PLC network
If you are struggling with these steps and have bought a machineconnect from us, feel free to contact us!
This document explains how to get started with the documentation locally
Clone the repo
Go to /docs and execute git submodule update --init --recursive to download all submodules
git init && git add . && git commit -m "test" (yes it is quite stupid, but it works)
Startup the development server by using sudo docker-compose up --build
4.7.3 - Working with the system practically
Three step implementation plan for using the factorycube
Please ensure that you have read the safety information and manuals before proceeding! Failure to do so can result in damages to the product or serious injuries.
Create a sensor plan
Please create before installing anything a sensor plan. Prior to installing take the layout of your selected line and add:
Electricity sockets
Internet connections (bring your own)
Your planned sensors
The position of the factorycube
Your planned cabling
Discuss and install
Setup a meeting with your line engineers and discuss your plan. Then install everything according to plan. Ensure, that all sensors and cables are mounted tightly and that they do not interfere with the production process.
Supply your factorycube with power and turn it on
Plug in the power cable to turn the factorycube on. After a few seconds the ia: factorycube should be lit up.
Connect your factorycube to the internet
If you want to use the cloud dashboard, you must first connect the factorycube to the internet.
You need:
your credentials
an Ethernet cable (provided)
a laptop which is not connected to any VPN
For a network overview of the Factorycube, click here
Instructions to login
Connect the factorycube with your computer via an ethernet cable using the IO-Link port (not Internettest_ on the factorycube.
Open the following website on your browser: http://172.16.x.2 (The X stands for the last number(s) of the serial number. e.g. 2019_0103 -> x=3 or 2019_0111 -> x=11)
Enter your credentials according to the information in the customer area. The username is always “admin”
3 ways to connect to the internet: WiFi, 3G/4G or Ethernet
Further information on how to connect the factorycube with the internet can be found in the official router manual
Instructions to setup WiFi
Select “Network” → “Wireless/Wlan”. If necessary remove the old Wireless station point
Click on “Add” next to wireless station mode
Click on “Start Scan” next to wireless station mode
Click on the network of your choice
“join network”
Afterwards enter your credentials and confirm
The computer should now be connected to the internet.
Instructions to setup 3G/4G
Insert the SIM-card (if a SIM-card is already provided in the ia: factorycube, skip this step)
For the installation of a SIM card please contact our experts
Select “Network” → “Mobile”
Adjust the settings under the “General” tab as follows:
Save your settings
The computer should now be connected to the internet.
Instructions to set up connection via Ethernet
Plug the Ethernet cable into the device’s “Internet” port and the other side into the network access port
Select “Network” –> “WAN”
Select as wired as your main WAN
Click save
The computer should now be connected to the internet. You can now the entire United Manufacturing Hub Edge Stack. For more information, take a look in the getting started with edge devices.
Outro
Closely monitor the data and verify over the entire duration of the next days, whether the data is plausible. Things that can go wrong here:
Sensors not mounted properly and not calibrated anymore
The operators are using the line different from what was discussed before (e. g., doing a changeover and removing the sensors)
4.7.4 - How to setup a development network
This article explains how to setup a development network to work with an edge device
4.7.5 - How to update the stack / helm chart
This article explains how to update the helm chart, so that you can apply changes to the configuration of the stack or to install newer versions
Prerequisites
none
Tutorial
Go to the folder deployment/factorycube-server or deployment/factorycube-edge
This is basically your installation command, but you exchange install with upgrade. You need to change “YOUR VALUES FILE” with the path of your values.yaml, e.g. /home/rancher/united-manufacturing-hub/deployment/factorycube-server/values.yaml and you need to adjust YOUR_NAMESPACE with the correct namespace name. If you did not specify any namespace during the installation you can use the namespace default. If you are using factorycube-edge instead of factorycube-server you need to adjust that as well.
4.7.6 - How to work with MinIO
This article explains how you can access MinIO for development and administration purposes
By default MinIO is exposed outside of the Kubernetes cluster using a LoadBalancer. The default credentials are minio:minio123 for the console. When using minikube you can access the LoadBalancer service using minikube tunnel and then accessing the external IP using the selected port.
4.7.7 - Working with Kubernetes, Helm and Lens
This article explains how to work with Kubernetes, Helm and Lens, especially how to enable / disable functionalities, update the configuration or how to do software upgrade
How does Kubernetes and Helm work?
Where is my Dockerfile?
One might ask.
Where is my docker-compose.yaml?
In this chapter, we want to explain how you can configure the Kubernetes cluster and therefore the United Manufacturing Hub.
Informal and not exact explaination (from the right to the left):
Docker containers are called in Kubernetes lanugage Pods.
There is also other stuff going on in Kubernetes like secret management or ingresses.
You define the final state (“I want to have one application based on this Docker image with this environment variables ….") and Kubernetes will take care of the rest using its magic. This could be seen equal to a docker-compose.yaml
The final state is defined using Kubernetes object descriptions.
Example for a Kubernetes object description:
apiVersion:apps/v1kind:Deploymentmetadata:name:nginx-deploymentspec:selector:matchLabels:app:nginxreplicas:2# tells deployment to run 2 pods matching the templatetemplate:metadata:labels:app:nginxspec:containers:- name:nginximage:nginx:1.14.2ports:- containerPort:80
You could now create and maintain all of them manually, which is what some companies do.
To do that automatically, you can use Helm.
Helm will take a values.yaml and will automatically create the corresponding Kubernetes object descriptions.
Example for a Helm template values:
### mqttbridge ###mqttbridge:enabled:trueimage:unitedmanufacturinghub/mqtt-bridgestorageRequest:1Gi### barcodereader ###barcodereader:enabled:trueimage:unitedmanufacturinghub/barcodereadercustomUSBName:"Datalogic ADC, Inc. Handheld Barcode Scanner"brokerURL:"factorycube-edge-emqxedge-headless"# do not change, points to emqxedge ibrokerPort:1883# do not changecustomerID:"raw"location:"barcodereader"machineID:"barcodereader"
There is nothing comparable to Helm in the Docker world.
For this to work, one needs to import a so-called Helm chart first. This is nothing else than a Kubernetes object description with some variables.
Example for a Kubernetes object description template (for Helm):
{{- if .Values.mqttbridge.enabled -}}apiVersion:apps/v1kind:StatefulSetmetadata:name:{{include "factorycube-edge.fullname" . }}-mqttbridgelabels:{{- include "factorycube-edge.labels.mqttbridge" . | nindent 4 }}spec:serviceName:{{include "factorycube-edge.fullname" . }}-mqttbridgereplicas:1selector:matchLabels:{{- include "factorycube-edge.labels.mqttbridge" . | nindent 6 }}template:metadata:labels:{{- include "factorycube-edge.labels.mqttbridge" . | nindent 8 }}spec:containers:- name:{{include "factorycube-edge.fullname" . }}-mqttbridge{{- if .Values.mqttbridge.tag }}image:{{.Values.mqttbridge.image }}:{{ .Values.mqttbridge.tag }}{{- else }}image:{{.Values.mqttbridge.image }}:{{ .Chart.AppVersion }}{{- end }}
You might be tempted to just clone the repository and change stuff in the Kubernetes object description template (in the subfolder templates). We consider this a bad idea as these changes are now hard to track and to replicate somewhere else. Furthermore, when updating the Helm chart to a newer version of the United Manufacturing Hub these changes will be lost and overwritten.
In Lens you can now change the underlying Kubernetes object descriptions by e.g., clicking on a Deployment and then selecting edit. However, these changes will be reverted once you apply the Helm chart again.
Therefore, we recommend in production setups to only adjust the values.yaml in the Helm chart.
Changing the configuration / updating values.yaml
There are two ways to change the values.yaml:
using Lens GUI
Note: if you encounter the issue “path not found”, please go to the troubleshooting section further down.
For this open Lens, select Apps on the left side and then click on Releases. A Release is a deployed version of a chart. Now you can not only change the values.yaml, but also update the Helm chart to a newer version.
using CLI / kubectl in Lens
To override single entries in values.yaml you can use the --set command, for example like this:
helm upgrade factorycube-edge . --namespace factorycube-edge --set nodered.env.NODE_RED_ENABLE_SAFE_MODE=true
Troubleshooting
RELEASE (factorycube-edge or factorycube-server) not found when updating the Helm chart
There are some modifications needed to be done when using Lens with Charts that are in a Repository. Otherwise, you might get a “RELEASE not found” message when changing the values in Lens.
The reason for this is, that Lens cannot find the source of the Helm chart. Although the Helm repository and the chart is installed on the Kubernetes server / edge device this does not mean that Lens can access it. You need to add the helm repository with the same name to your computer as well.
So execute the following commands on your computer running Lens (on Windows in combination with WSL2):
/docs contains the entire documentation and API specifications for all components including all information to buy, assemble and setup the hardware
5.1 - united-manufacturing-hub
The architecture of united-manufacturing-hub
factoryinsight
factoryinsight is an open source REST API written in Golang to fetch manufacturing data from a timescaleDB database and calculate various manufacturing KPIs before delivering it to a user visualization, e.g. [Grafana] or [PowerBI].
Features:
OEE (Overall Equipment Effectiveness), including various options to investigate OEE losses (e.g. analysis over time, microstop analytics, changeover deep-dives, etc.)
Various options to investigate OEE losses further, for example stop analysis over time, microstop analytics, paretos, changeover deep-dives or stop histograms
Scalable, microservice oriented approach for Plug-and-Play usage in Kubernetes or behind load balancers (including health checks and monitoring)
Proxies request from grafana to various backend services, while authenticating the grafana user.
Technical information and usage can be found in the documentation for grafana-proxy
grafana-plugins
Contains our grafana datasource plugin and our input panel
sensorconnect
This tool automatically finds connected ifm gateways (e.g. the AL1350 or AL1352), extracts all relevant data and pushes the data to a MQTT broker. Technical information and usage can be found in the documentation for sensorconnect
cameraconnect
This tool automatically identifies connected cameras network-wide which support the GenICam standard and makes them utilizable. Each camera requires its own container. The camera acquisition can be triggered via MQTT. The resulting image data gets pushed to the MQTT broker. Technical information and usage can be found in the documentation for cameraconnect
barcodereader
This tool automatically detected connected USB barcode scanners and send the data to a MQTT broker. Technical information and usage can be found in the documentation for barcodereader
mqtt-bridge
This tool acts as an MQTT bridge to handle bad internet connections. Messages are stored in a persistent queue on disk. This allows using the united-manufacturing-hub in remote environments with bad internet connections. It will even survive restarts (e.g. internet failure and then 1h later power failure). We developed it as we’ve tested multiple MQTT brokers and their bridge functionalities (date of testing: 2021-03-15) and could not find a proper solution:
Open your Profile and check if your organization name inside Grafana matches the rest of your UMH stack
Creating a new Panel
Create a new Dashboard or edit an existing one
Click “Add an empty panel”
On the right sidebar switch the Visualization to “Button Panel”
Fill out the fields inside “REST Integration”
URL
http://{URL to your grafana-proxy}/api/v1/factoryinput/
Example:
http://172.21.9.195:2096/api/v1/factoryinput/
Location
Location of your Asset
Asset
Name of the Asset
Value
MQTT prefix
Example prefixes:
count
addShift
modifyShift
Customer
Your organization name
Payload
JSON encoded payload to send as MQTT message payload
Modify any additional options are you like
When you are finished customizing, click on “Apply”
Example Panel
Notes
Clicking the button will immediately send the MQTT message, through our HTTP->MQTT stack. Please don’t send queries modifying date you would later need !
Common pitfalls
Pressing the button just changes the cog to an warning sign
Open your network inspector view (Ctrl+Shift+I on Chrome)
Press the button again
If no request appears, then you haven’t filled out all required fields
Your request shows:
403
Make sure the customer field is set to your grafana organization name
400
Your request was incorrectly formatted
Check that the URL is in the format specified above
All connected devices will be shown, the used device is marked with “Found xyz”. After every scan the MQTT message will be printed.
Environment variables
This chapter explains all used environment variables.
DEBUG_ENABLED
Description: Deactivates MQTT and only prints the barcodes to stdout
Type: bool
Possible values: true, false
Example value: true
CUSTOM_USB_NAME
Description: If your barcodereader is not in the supported list of devices, you must specify the name of the USB device here
Type: string
Possible values: all
Example value: Datalogic ADC, Inc. Handheld Barcode Scanner
MQTT_CLIENT_ID
Description: The MQTT client id to connect with the MQTT broker
Type: string
Possible values: all
Example value: weaving_barcodereader
BROKER_URL
Description: The MQTT broker URL
Type: string
Possible values: IP, DNS name
Example value: ia_mosquitto
Example value 2: localhost
BROKER_PORT
Description: The MQTT broker port. Only unencrypted ports are allowed here (default: 1883)
Type: integer
Possible values: all
Example value: 1883
CUSTOMER_ID
Description: The customer ID, which is used for the topic structure
Type: string
Possible values: all
Example value: dccaachen
LOCATION
Description: The location, which is used for the topic structure
Type: string
Possible values: all
Example value: aachen
MACHINE_ID
Description: The machine ID, which is used for the topic structure
Type: string
Possible values: all
Example value: weaving_machine_2
5.1.3 - cameraconnect
This docker container automatically detects cameras in the network and makes them accessible via MQTT. The MQTT output is specified in the MQTT documentation
This microservice is still in development and is not considered stable for production use.
Getting started
Using the Helm chart
By default cameraconnect will be deactivated in united-manufacturing-hub. First, you need to enable it in the united-manufacturing-hub values. Then you need to create a folder on the node in /home/rancher/gentl_producer and move your genTL producer files (*.cti) and all required libaries into that folder. Then apply your settings to the Helm chart with helm upgrade.
Another idea for changing it:
helm install ....... --set 'cameraconnect.enabled=true'
Or overwroite it in the development_values.yaml
Furthermore, you need to adjust the MQTT_HOST to the externally exposed MQTT IP (e.g., the IP of your node). Usually you can use the Kubernetes internal DNS. But cameraconnect needs to be in hostMode = true and hterefore you need to access from external.
Development setup
Here is a quick tutorial on how to start up a basic configuration / a basic docker-compose stack, so that you can develop.
Specify the environment variables, e.g. in a .env file in the main folder or directly in the docker-compose
execute sudo docker-compose -f ./deployment/cameraconnect/docker-compose.yaml up -d --build
Environment variables
This chapter explains all used environment variables.
CUBE_TRANSMITTERID
Description: The unique transmitter id. This will be used for the creation of the MQTT topic. ia/raw/TRANSMITTERID/…
Type: string
Possible values: all
Example value: 2021-0156
MQTT_HOST
Description: The MQTT broker URL
Type: string
Possible values: IP, DNS name
Example value: ia_mosquitto
Example value 2: localhost
MQTT_PORT
Description: The MQTT broker port. Only unencrypted ports are allowed here (default: 1883)
Type: integer
Possible values: all
Example value: 1883
TRIGGER
Description: Defines the option of how the camera is triggered. Either via MQTT or via a continuous time trigger. In production the camera should be triggered via MQTT. The continuous time trigger is just convenient for debugging. If MQTT is selected, the camera will be triggered by any message which arrives via its subscribed MQTT topic. However, if the arriving MQTT message contains a UNIX timestamp in milliseconds with the key “timestamp_ms”, the camera will be triggered at that exact timestamp.
Type: string
Possible values: MQTT, Continuous
Example value: MQTT
ACQUISITION_DELAY
Description: Timeconstant in seconds which delays the image acquisition after the camera has been triggered. This is mostly used, if the camera is triggered with a UNIX timestamp (see variable TRIGGER), to make sure, that the camera is triggered, even if the UNIX timestamps lies in the past. This could be caused by network latencies.
Type: float
Possible values: all
Example value: 0.7
CYCLE_TIME
Description: Only relevant if the trigger is set to “Continuous”. Cycle time gives the time period which defines the frequency in which the camera is triggered. For example: a value of 0.5 would result in a trigger frequency of 2 images per second.
Type: float
Possible values: all
Example value: 1.5
CAMERA_INTERFACE
Description: Defines which camera interface is used. Currently only cameras of the GenICam standard are supported. However, for development of testing you can also use the DummyCam, which simulates a camera and sends a static image via MQTT, when triggered.
Type: String
Possible values: GenICam, DummyCam
Example value: GenICam
EXPOSURE_TIME
Description: Defines the exposure time for the selected camera. You should adjust this to your local environment to achieve optimal images.
Type: int
Possible values: Depends on camera interface. Values between 1 and 80,000 are eligible for most cameras.
Example value: 1000
EXPOSURE_AUTO
Description: Determines if camera automatically adjusts the exposure time. Your settings will only be executed if the camera supports this. You do not have to check if the camera supports this.
Type: String
Possible values: “Off” - No automatic adjustment “Once” - Adjusted once “Continuous” - Continuous adjustment (not recommended, Attention: This could have a big impact on the frame rate of your camera)
Example value: Off
PIXEL_FORMAT
Description: Sets the pixel format which will be used for image acquisition. This module allows you to acquire images in monochrome pixel formats(use: “Mono8”) and RGB/BRG color pixel formats (use:“RGB8Packed” or “BGR8Packed”)
Type: String
Possible values: Mono8, RGB8Packed, BGR8Packed
Example value: Mono8
IMAGE_WIDTH
Description: Defines the horizontal width of the images acquired. If the width values surpasses the maximum capability of the camera
the maximum value is set automatically.
Type: int
Possible values: all except 0
Example value: 1000
IMAGE_HEIGHT
Description: Defines the vertical height of the images acquired. If the height value surpasses the maximum capability of the camera
the maximum value is set automatically.
Type: int
Possible values: all except 0
Example value: 1000
IMAGE_CHANNELS
Description: Number of channels (bytes per pixel) that are used in the array (third dimension of the image data array).You do not have to set this value. If None, the best number of channels for your set pixel format will be used.
Type: int
Possible values: 1 or 3
Example value: 1
MAC_ADDRESS
Description: Defines which camera is accessed by the container. One container can use only one camera. The MAC address can be found on the backside of the camera. The input is not case sensitive. Please follow the example format below.
Type: String
Possible values: all
Example value: 0030532B879C
LOGGING_LEVEL
Description: Defines which logging level is used. Mostly relevant for developers. Use WARNING or ERROR in production.
Type: String
Possible values: DEBUG, INFO, WARNING, ERROR, CRITICAL
This site describes all environment variables used to setup and customize the united manufacturing hub.
Overview
Environment variables are used to setup and customize your system to your needs. The United Manufacturing Hub has a lot of different microservices you can enable or disable via environment variables. Additionally those microservices have settings you can set with those environment variables.
This webpage shows you how to change them and what variables are accessible.
How to change environment variables
The environment variables are specified in a .yaml file. You can either directly change them in the file or use our Management SaaS tool to generate the file according to your needs. The latter option eliminates the risk of invalid .yaml files.
Usually the hostname. This will be used for the creation of the MQTT/Kafka topic. ia/raw/TRANSMITTERID/…
string
all
2021-0156
development
sensorconnect/storageRequest
ToDo
string
1Gi
sensorconnect/ioddfilepath
A persistent path, to save downloaded IODD files at
string
All valid paths
/tmp/iodd
sensorconnect/lowerPollingTime
The fastest time to read values from connected sensors in milliseconds
int
all
100
sensorconnect/upperPollingTime
The slowest time to read values from connected sensors in milliseconds. To disable this feature, set this variable to the same value as lowerPollingTime
int
all
100
sensorconnect/pollingSpeedStepUpMs
The time to add to the actual tick rate in case of a failure (incremental)
int
all
10
sensorconnect/pollingSpeedStepDownMs
The time to subtract from actual tick rate in case of a failure (incremental)
int
all
10
sensorconnect/sensorInitialPollingTimeMs
The tick speed, that sensor connect will start from. Set a bit higher than LOWER_POLLING_TIME_MS to allow sensors to recover from faults easier
int
all
100
sensorconnect/maxSensorErrorCount
After this numbers on errors, the sensor will be removed until the next device scan
int
all
50
sensorconnect/deviceFinderTimeSec
Seconds between scanning for new devices
int
all
10
sensorconnect/deviceFinderTimeoutSec
Timeout per device for scan response
int
all
10
sensorconnect/additionalSleepTimePerActivePortMs
This adds a sleep per active port on an IO-Link master
float
all
0.0
sensorconnect/allowSubTwentyMs
Allows query times below 20MS. DO NOT activate this, if you are unsure, that your devices survive this load.
bool
0, 1
0
sensorconnect/additionalSlowDownMap
A json map of additional slowdowns per device. Identificates can be Serialnumber, Productcode or URL
This chapter explains all used environment variables.
POSTGRES_HOST
Description: Specifies the database DNS name / IP-address for postgresql / timescaleDB
Type: string
Possible values: all DNS names or IP
Example value: united-manufacturing-hub
POSTGRES_PORT
Description: Specifies the database port for postgresql
Type: int
Possible values: valid port number
Example value: 5432
POSTGRES_DATABASE
Description: Specifies the database name that should be used
Type: string
Possible values: an existing database in postgresql
Example value: factoryinsight
POSTGRES_USER
Description: Specifies the database user that should be used
Type: string
Possible values: an existing user with access to the specified database in postgresql
Example value: factoryinsight
POSTGRES_PASSWORD
Description: Specifies the database password that should be used
Type: string
Possible values: all
Example value: changeme
FACTORYINSIGHT_USER
Description: Specifies the admin user for the REST API
Type: string
Possible values: all
Example value: jeremy
FACTORYINSIGHT_PASSWORD
Description: Specifies the password for the admin user for the REST API
Type: string
Possible values: all
Example value: changeme
VERSION
Description: The version of the API used (currently not used yet)
Type: int
Possible values: all
Example value: 1
DEBUG_ENABLED
Description: Enables debug logging
Type: string
Possible values: true,false
Example value: false
REDIS_URI
Description: URI for accessing redis sentinel
Type: string
Possible values: All valids URIs
Example value: united-manufacturing-hub-redis-node-0.united-manufacturing-hub-redis-headless:26379
REDIS_URI2
Description: Backup URI for accessing redis sentinel
Type: string
Possible values: All valids URIs
Example value: united-manufacturing-hub-redis-node-1.united-manufacturing-hub-redis-headless:26379
REDIS_URI3
Description: Backup URI for accessing redis sentinel
Type: string
Possible values: All valids URIs
Example value: united-manufacturing-hub-redis-node-2.united-manufacturing-hub-redis-headless:26379
REDIS_PASSWORD
Description: Password for accessing redis sentinel
Type: string
Possible values: all
Example value: changeme
REST API
5.1.7 - grafana-proxy
Documentation of grafana-proxy
This microservice is still in development and is not considered stable for production use.
Getting started
This program proxies requests to backend services, if the requesting user is logged into grafana and part of the organization he requests.
Either using go run github.com/united-manufacturing-hub/united-manufacturing-hub/cmd/grafana-auth
Or using the Dockerfile
- Open an terminal inside united-manufacturing-hub/deployment/grafana-auth
- Run docker build -f ./Dockerfile ../..
Environment variables
This chapter explains all used environment variables.
FACTORYINPUT_USER
Description: Specifies the user for the REST API
Type: string
Possible values: all
Example value: jeremy
FACTORYINPUT_KEY
Description: Specifies the password for the user for the REST API
Type: string
Possible values: all
Example value: changeme
FACTORYINPUT_BASE_URL
Description: Specifies the DNS name / IP-address to connect to factoryinput
Type: string
Possible values: all DNS names or IP
Example value: http://united-manufacturing-hub-factoryinput-service
FACTORYINSIGHT_BASE_URL
Description: Specifies the DNS name / IP-address to connect to factoryinsight
Type: string
Possible values: all DNS names or IP
Example value: http://united-manufacturing-hub-factoryinsight-service
LOGGING_LEVEL
Description: Optional variable, if set to “DEVELOPMENT”, it will switch to debug logging
Type: string
Possible values: Any
Example value: DEVELOPMENT
Type: string
Possible values: all DNS names or IP
Type: string
Possible values: 0-65535
Example value: 9411
DISABLE_JAEGER
Type: string
Possible values: Any
Example value: 1
Notes
Grafana-Proxy accepts all cors requests.
For authenticated requests, you must send your Origin, else CORS will fail
This microservice is a bridge between two Kafka brokers.
Getting started
Kafka-bridge is a microservice that connects two Kafka brokers,
and forwards messages between them.
It is used, when having one or multiple edge devices and a central server.
Environment Variables
The following environment variables are used by the microservice:
Variable
Type
Description
LOGGING_LEVEL
String
Configures the zap logging level, set to DEVELOPMENT to enable development logging.
KAFKA_TOPIC_MAP
JSON
Configures which topics should be forwarded.
LOCAL_KAFKA_BOOTSTRAP_SERVER
String
Configures the local Kafka bootstrap server.
REMOTE_KAFKA_BOOTSTRAP_SERVER
String
Configures the remote Kafka bootstrap server.
Kafka Topic Map
The KAFKA_TOPIC_MAP has the following JSON schema:
{"$schema":"http://json-schema.org/draft-07/schema","type":"array","title":"Kafka Topic Map","description":"This schema validates valid Kafka topic maps.","default":[],"examples":[[{"name":"HighIntegrity","topic":"^ia\\..+\\..+\\..+\\.(?!processValue).+$","bidirectional":false,"send_direction":"to_remote"}],[{"name":"HighIntegrity","topic":"^ia\\..+\\..+\\..+\\.(?!processValue).+$","bidirectional":true},{"name":"HighThroughput","topic":"^ia\\..+\\..+\\..+\\.(processValue).*$","bidirectional":false,"send_direction":"to_remote"}]],"additionalItems":true,"items":{"$id":"#/items","anyOf":[{"$id":"#/items/anyOf/0","type":"object","title":"Unidirectional Kafka Topic Map with send direction","description":"This schema validates entries, that are unidirectional and have a send direction.","default":{},"examples":[{"name":"HighIntegrity","topic":"^ia\\..+\\..+\\..+\\.(?!processValue).+$","bidirectional":false,"send_direction":"to_remote"}],"required":["name","topic","bidirectional","send_direction"],"properties":{"name":{"$id":"#/items/anyOf/0/properties/name","type":"string","title":"Entry Name","description":"Name of the map entry, only used for logging & tracing.","default":"","examples":["HighIntegrity"]},"topic":{"$id":"#/items/anyOf/0/properties/topic","type":"string","title":"The topic to listen on","description":"The topic to listen on, this can be a regular expression.","default":"","examples":["^ia\\..+\\..+\\..+\\.(?!processValue).+$"]},"bidirectional":{"$id":"#/items/anyOf/0/properties/bidirectional","type":"boolean","title":"Is the transfer bidirectional?","description":"When set to true, the bridge will consume and produce from both brokers","default":false,"examples":[false]},"send_direction":{"$id":"#/items/anyOf/0/properties/send_direction","type":"string","title":"Send direction","description":"Can be either 'to_remote' or 'to_local'","default":"","examples":["to_remote","to_local"]}},"additionalProperties":true},{"$id":"#/items/anyOf/1","type":"object","title":"Bi-directional Kafka Topic Map with send direction","description":"This schema validates entries, that are bi-directional.","default":{},"examples":[{"name":"HighIntegrity","topic":"^ia\\..+\\..+\\..+\\.(?!processValue).+$","bidirectional":true}],"required":["name","topic","bidirectional"],"properties":{"name":{"$id":"#/items/anyOf/1/properties/name","type":"string","title":"Entry Name","description":"Name of the map entry, only used for logging & tracing.","default":"","examples":["HighIntegrity"]},"topic":{"$id":"#/items/anyOf/1/properties/topic","type":"string","title":"The topic to listen on","description":"The topic to listen on, this can be a regular expression.","default":"","examples":["^ia\\..+\\..+\\..+\\.(?!processValue).+$"]},"bidirectional":{"$id":"#/items/anyOf/1/properties/bidirectional","type":"boolean","title":"Is the transfer bidirectional?","description":"When set to true, the bridge will consume and produce from both brokers","default":false,"examples":[true]}},"additionalProperties":true}]}}
This microservice provides at-least-once guarantees, by manually committing the offset of the message that was processed.
5.1.9 - kafka-to-blob
This microservice saves images from kafka to blob storage.
Getting started
Kafka-to-blob is a microservice that saves images from kafka to blob storage.
This is currently used in combination with cameraconnect.
Environment Variables
Variable
Type
Description
KAFKA_BOOSTRAP_SERVER
String
The kafka bootstrap server.
KAFKA_LISTEN_TOPIC
String
The kafka topic to listen to.
KAFKA_BASE_TOPIC
String
Set this, if you want to automatically create the topic.
MINIO_URL
String
The url of the minio server.
MINIO_ACCESS_KEY
String
The access key of the minio server.
MINIO_SECRET_KEY
String
The secret key of the minio server.
BUCKET_NAME
String
The bucket to store the images in.
MINIO_SECURE
Boolean
Whether to use https or http.
Guarantees
Kafka-to-blob does not guarantee delivery of messages.
5.1.10 - kafka-to-postgresql
This microservices consumes messages from a Kafka topic and writes them to a PostgreSQL database.
Getting started
Kafka-to-PostgreSQL is a microservice that consumes messages from a Kafka topic and writes them to a PostgreSQL database.
By default, it sets up, two kafka consumers, one for high throughput and one for high integrity.
High throughput
This kafka listener is usually configured to listen on the processValue topics.
High integrity
This kafka listener is usually configured to listen on all other topics.
Environement variables
The following environment variables are used by the microservice:
Variable
Type
Description
LOGGING_LEVEL
String
Configures the zap logging level, set to DEVELOPMENT to enable development logging.
DRY_RUN
Boolean
If set to true, the microservice will not write to the database.
POSTGRES_HOST
String
The hostname of the PostgreSQL database.
POSTGRES_USER
String
The username to use for PostgreSQL connections.
POSTGRES_PASSWORD
String
The password to use for PostgreSQL connections.
POSTGRES_DATABASE
String
The name of the PostgreSQL database.
POSTGRES_SSLMODE
Boolean
If set to true, the PostgreSQL connection will use SSL.
KAFKA_BOOSTRAP_SERVERS
String
The kafka server to connect to.
KAFKA_HIGH_INTEGRITY_LISTEN_TOPIC
String
The kafka topic to listen to for high integrity messages. (This can be a regex)
KAFKA_HIGH_THROUGHPUT_LISTEN_TOPIC
String
The kafka topic to listen to for high throughput messages. (This can be a regex)
Program flow
The graphic below shows the program flow of the microservice.
Data flow
High Integrity
The graphic below shows the flow for an example High Integrity message.
5.1.11 - mqtt-bridge
This tool acts as an MQTT bridge to handle bad internet connections.
Getting started
Here is a quick tutorial on how to start up a basic configuration / a basic docker-compose stack, so that you can develop.
Here is a quick tutorial on how to start up a basic configuration / a basic docker-compose stack, so that you can develop.
Specify the environment variables, e.g. in a .env file in the main folder or directly in the docker-compose
execute sudo docker-compose -f ./deployment/mqtt-bridge/docker-compose.yaml up -d --build
Environment variables
This chapter explains all used environment variables.
REMOTE_CERTIFICATE_NAME
Description: the certificate name / client id
Type: string
Possible values: all
Example value: 2021-0156
REMOTE_BROKER_URL
Description: the URL to the broker. Can be prepended either with ssl:// or mqtt:// or needs to specify the port afterwards with :1883
Type: string
Possible values: all
Example value: ssl://mqtt.app.industrial-analytics.net
REMOTE_SUB_TOPIC
Description: the remote topic that should be subscribed. The bridge will automatically append a /# to the string mentioned here
Type: string
Possible values: all
Example value: ia/ia
REMOTE_PUB_TOPIC
Description: the remote topic prefix where messages from the remote broker should be send to.
Type: string
Possible values: all
Example value: ia/ia
REMOTE_BROKER_SSL_ENABLED
Description: should SSL be enabled and certificates be used for connection?
Type: bool
Possible values: true or false
Example value: true
LOCAL_CERTIFICATE_NAME
Description: the certificate name / client id
Type: string
Possible values: all
Example value: 2021-0156
LOCAL_BROKER_URL
Description: the URL to the broker. Can be prepended either with ssl:// or mqtt:// or needs to specify the port afterwards with :1883
Type: string
Possible values: all
Example value: ssl://mqtt.app.industrial-analytics.net
LOCAL_SUB_TOPIC
Description: the remote topic that should be subscribed. The bridge will automatically append a /# to the string mentioned here
Type: string
Possible values: all
Example value: ia/ia
LOCAL_PUB_TOPIC
Description: the remote topic prefix where messages from the remote broker should be send to.
Type: string
Possible values: all
Example value: ia/ia
LOCAL_BROKER_SSL_ENABLED
Description: should SSL be enabled and certificates be used for connection?
Type: bool
Possible values: true or false
Example value: true
BRIDGE_ONE_WAY
Description: DO NOT SET TO FALSE OR THIS MIGHT CAUSE AN ENDLESS LOOP! NEEDS TO BE FIXED BY SWITCHING TO MQTTV5 AND USING NO_LOCAL OPTION WHILE SUBSCRIBING. If true it sends the messages only from local broker to remote broker (not the other way around)
Type: bool
Possible values: true or false
Example value: true
Important note regarding topics
The bridge will append /# to LOCAL_SUB_TOPIC and subscribe to it. All messages will then be send to the remote broker. The topic on the remote broker is defined by:
First stripping LOCAL_SUB_TOPIC from the topic
and then replacing it with REMOTE_PUB_TOPIC
5.1.12 - mqtt-kafka-bridge
This docker container transfers messages from mqtt to kafka and vise versa
Getting started
Using the Helm chart
By default, mqtt-kafka-bridge will be deactivated in united-manufacturing-hub.
Environment variables
This chapter explains all used environment variables.
MQTT_BROKER_URL
Description: The MQTT broker URL
Type: string
Possible values: IP, DNS name
Example value: ia_mosquitto
Example value 2: united-manufacturing-hub-vernemq-local-service:1883
MQTT_CERTIFICATE_NAME
Description: Set to NO_CERT to allow non-encrypted MQTT access
Type: string
Possible values: NO_CERT, any string
Example value: NO_CERT
MQTT_TOPIC
Description: MQTT topic to listen on
Type: string
Possible values: any string
Example value: ia/#
KAFKA_BOOSTRAP_SERVER
Description: The Kafka broker URL
Type: string
Possible values: IP, DNS name
Example value: united-manufacturing-hub-kafka:9092
KAFKA_LISTEN_TOPIC
Description: Kafka topic to listen on
Type: string
Possible values: any string
Example value: ia/#
KAFKA_BASE_TOPIC
Description: Kafka base topic (usually the top most part of the listen topic)
Type: string
Possible values: any string
Example value: ia
REDIS_URI
Description: Redis database url
Type: string
Possible values: IP, DNS name
Example value: united-manufacturing-hub-redis-node-0.united-manufacturing-hub-redis-headless:26379
REDIS_PASSWORD
Description: Redis password
Type: string
Possible values: any string
Example value: super secret password
MY_POD_NAME
Description: Name of the docker pod, used for identification at the MQTT broker
Type: string
Possible values: any string
Example value: MQTT-Kafka-Bridge
5.1.13 - mqtt-to-blob
The following guide describes how to catch data from the MQTT-Broker and push them to the MIN.io blob storage
This microservice is still in development and is not considered stable for production use.
MQTT-to-Blob has the function of subscribing to MQTT topics and storing the information into a MIN.io blob storage. The current iteration of MQTT-to-Blob can subscribe to cameraconnect’s MQTT topic and send the image data to MIN.io server’s bucket.
The input format of mqtt-to-blob is in accordance with /productImage JSON format. The information from the JSON is extracted and then stored into an image file in MIN.io with the metadata attached.
Before running the code, the environment variables have to be adjusted for the system. Information regarding the environment variables can be seen in the respective section.
Dependencies
paho-mqtt==1.6.1
numpy==1.21.3
minio==7.1.1
Environment variables
This chapter explains all used environment variables.
Variable
Type
Description
Possible values
Example value
BROKER_URL
str
Specifies the address to connect to the MQTT-Broker
all DNS names or IP address
127.0.0.1
BROKER_PORT
int
Specifies the port for the MQTT-Broker
valid port number
1883
TOPIC
str
MQTT Topic name
Published MQTT Topic name
ia/rawImage/#
MINIO_URL
str
Specifies the database DNS name / IP-address for the MIN.io server
all DNS names or IP address with its port number
play.min.io
MINIO_SECURE
bool
Select True or False to activate HTTPS connection or not
True or False
True
MINIO_ACCESS_KEY
str
Specifies the username for MIN.io server
eligible username
username
MINIO_SECRET_KEY
str
Specifies the password for MIN.io server
password of the user
password
BUCKET_NAME
str
Specifies the Bucket name to store the blob in MIN.io server
Bucket name
Bucket-name
LOGGING_LEVEL
str
Select level for logging messages
DEBUGINFOERRORWARNINGCRITICAL
DEBUG
Note
For MINIO_SECURE param, the boolean value has to have capital letter for the first letter
When using IP address for MINIO_URL, the port number has to be included (e.g. 0.0.0.0:9000)
MQTT Connection Return Code
Return Code
Meaning
0
Successful Connection
1
Connection refused - incorrect protocol version
2
Connection refused - invalid client identifier
3
Connection refused - server unavailable
4
Connection refused - Bad username or password
5
Connection refused - not authorised
6-255
Currently unused
Future work
In the future, MQTT-to-Blob should also be able to store different kind of data such as sound and video
Note
Use the dummy_pic_mqtt_topic.json in the mqtt-to-blob folder manually inject mqtt topic using mqtt explorer
5.1.14 - mqtt-to-postgresql
Documentation of mqtt-to-postgresql
mqtt-to-postgresql subscribes to the MQTT broker (in the stack this is VerneMQ), parses incoming messages on the topic “ia/#” and stores them in the postgresql / timescaleDB database (if they are in the correct datamodel)
Getting started
Here is a quick tutorial on how to start up a basic configuration / a basic docker-compose stack, so that you can develop.
docker-compose -f ./deployment/mqtt-to-postgresql/docker-compose-mqtt-to-postgresql-development.yml –env-file ./.env up -d –build
Message processing flow
Below diagram shows an abstract flow, of an incoming MQTT message.
Environment variables
This chapter explains all used environment variables.
POSTGRES_HOST
Description: Specifies the database DNS name / IP-address for postgresql / timescaleDB
Type: string
Possible values: all DNS names or IP
Example value: united-manufacturing-hub
POSTGRES_PORT
Description: Specifies the database port for postgresql
Type: int
Possible values: valid port number
Example value: 5432
POSTGRES_DATABASE
Description: Specifies the database name that should be used
Type: string
Possible values: an existing database in postgresql
Example value: factoryinsight
POSTGRES_USER
Description: Specifies the database user that should be used
Type: string
Possible values: an existing user with access to the specified database in postgresql
Example value: factoryinsight
POSTGRES_PASSWORD
Description: Specifies the database password that should be used
Type: string
Possible values: all
Example value: changeme
DRY_RUN
Description: Enables dry run mode (doing everything, even “writing” to database, except committing the changes.)
Type: string
Possible values: true,false
Example value: false
REDIS_URI
Description: URI for accessing redis sentinel
Type: string
Possible values: All valids URIs
Example value: united-manufacturing-hub-redis-node-0.united-manufacturing-hub-redis-headless:26379
REDIS_URI2
Description: Backup URI for accessing redis sentinel
Type: string
Possible values: All valids URIs
Example value: united-manufacturing-hub-redis-node-1.united-manufacturing-hub-redis-headless:26379
REDIS_URI3
Description: Backup URI for accessing redis sentinel
Type: string
Possible values: All valids URIs
Example value: united-manufacturing-hub-redis-node-2.united-manufacturing-hub-redis-headless:26379
REDIS_PASSWORD
Description: Password for accessing redis sentinel
Type: string
Possible values: all
Example value: changeme
MY_POD_NAME
Description: The pod name. Used only for tracing, logging and MQTT client id.
Type: string
Possible values: all
Example value: app-mqtttopostgresql-0
MQTT_TOPIC
Description: Topic to subscribe to. Only set for debugging purposes, e.g., to subscribe to a certain message type. Default usually works fine.
Type: string
Possible values: all possible MQTT topics
Example value: $share/MQTT_TO_POSTGRESQL/ia/#
5.1.15 - Sensorconnect
This microservice of the united manufacturing hub automatically detects ifm gateways in the specified network and reads their sensor values in the in the highest possible data frequency (or chosen data frequency). It preprocesses the data and sends it via MQTT or Kafka.
Sensorconnect overview
Sensorconnect provides plug-and-play access to IO-Link sensors connected to ifm gateways. Digital input mode is also supported. A typical setup contains multiple sensors connected to ifm gateways on the production shopfloor. Those gateways are connected via LAN to your server infrastructure. The sensorconnect microservice constantly monitors a given IP range for gateways. Once a gateway is found, it automatically starts requesting, receiving and processing sensordata in short intervals. The received data is preprocessed based on a database including thousands of sensor definitions. And because we strongly believe in open industry standards, Sensorconnect brings the data via MQTT or Kafka to your preferred software solutions, for example, features of the United Manufacturing Hub or the cloud.
Which problems is Sensorconnect solving
Let’s take a step back and think about, why we need a special microservice:
The gateways are providing a rest api to request sensordata. Meaning as long as we dont have a process to request the data, nothing will happen.
Constantly requesting and processing data with high robustness and reliability can get difficult in setups with a large number of sensors.
Even if we use for example node-red flows (flow from node-red, flow from ifm) to automatically request data from the rest api of the ifm gateways, the information is most of the times cryptic without proper interpretation.
Device manufacturers will provide one IODD file (IO Device Description), for every sensor and actuator they produce. Those contain information to correctly interpret data from the devices. They are in XML-format and available at the IODDfinder website. But automatic IODD interpretation is relatively complex and manually using IODD files is not economically feasible.
Installation
Production setup
Sensorconnect comes directly with the united manufacturing hub - no additional installation steps required. We allow the user to customize sensorconnect by changing environment variables. All possible environment variables you can use to customize sensorconnect and how you change them is described in the environment-variables documentation. Sensorconnect is by default enabled in the United Manufacturing Hub. To set your preferred serialnumber and choose the ip range to look for gateways, you can configure either the values.yaml directly or use our managment SaaS tool.
Development setup
Here is a quick tutorial on how to start up a basic configuration / a basic docker-compose stack, so that you can develop.
execute docker-compose -f ./deployment/sensorconnect/docker-compose.yaml up -d --build
Underlying Functionality
Sensorconnect downloads relevant IODD files automatically after installation from the IODDfinder website. If an unknown sensor is connected later, sensorconnect will automatically download the file. We will also provide a folder to manually deposit IODD-files, if the automatic download doesn’t work (e.g. no internet connection).
Rest API POST requests from sensorconnect to the gateways
Sensorconnect scans the ip range for new ifm gateways (used to connect the IO-Link devices to). To do that, sensorconnect iterates through all the possible IP addresses in the specified IP address range (“http://"+url, payload, timeout=0.1). It stores the IP adresses, with the product codes (the types of the gateways) and the individual serialnumbers.
Scanning with following payload: (information sent during a POST request to the ifm gateways)
All port modes of the connected gateways are requested. Depending on the productcode, we can determine the total number of ports on the gateway and iterate through them.
Requesting port modes with following payload: (information sent during a POST request to the ifm gateways)
If the mode == 1: port_mode = “DI” (Digital Input)
If the mode == 2: port_mode = “DO” (Digital output)
If the mode == 3: port_mode = “IO_Link”
All values of accessible ports are requested as fast as possible (ifm gateways are by far the bottleneck in comparison to the networking).
Requesting IO_Link port values with following payload: (information sent during a POST request to the ifm gateways)
Based on the vendorid and deviceid (extracted out of the received data from the ifm-gateway), sensorconnect looks in it’s persistant storage if an iodd file is available. If there is no iodd file stored for the specific sensor, it tries to download a file online and saves it. If this also is not possible, sensorconnect doesn’t preprocess the pdin data entry and forwards it as is.
Data interpretation (key: pdin) with IODD files
The iodd files are in xml format and often contain multiple thousand lines. You can find extensive documentation on the io link website. Especially relevant areas of the iodd files are (for our use-case):
IODevice/DocumentInfo/version: version of IODD files
IODevice/ProfileBody/DeviceIdentity: deviceid, vendorid etc.
IODevice/DeviceFunction/ProcessDataCollection/ProcessData/ProcessDataIn: contains information about the data structure received from the sensor. Datatypes or datatype references are given.
IODevice/DeviceFunction/DatatypeCollection: if used, contains datatypes referenced by the ProcessDataCollection(Point 3.)
IODevice/ExternalTextCollection: contains translations for textId’s.
Digital input data management
The gateways are also able to receive digital input data (not IO-Link). This data is also requested and forwarded.
Data delivery to MQTT or Kafka
For delivery of the data, sensorconnect converts it to a JSON containing the preprocessed data, timestamps, serialnumber etc. and sends it via MQTT to the MQTT broker or via Kafka to the Kafka broker. You can change the environment variables to choose your preferred protocol. The format of those JSON messages coming from sensorconnect is described in detail and with examples on the UMH Datamodel website.
Tested gateways
AL1350
AL1342
Future plans
Support for additional gateway manufacturers.
Support for value ranges and single values specified in the IODD file (giving additional information about currently received sensor value).
5.2 - How to add new MQTT messages to mqtt-to-postgresql
For new developers the internal structure of mqtt-to-postgresql might not be self-explaining. Therefore, this tutorial.
In general MQTT messages in mqtt-to-postgresql are first received (see entrypoint) and then stored in a message specific queue. Per message type: All messages within the last second are then gathered together and written into the database.
If one of the messages fails to get written into the database the entire batch of messages is regarded. In the future we should add here an additional buffer that tries each message separately to first identify the broken messages from the rest of the batch and then repeatedly tries to add the broken messages for like 5 times over a time period of 30 minutes to prevent issues with delayed messages (e.g., creation of a product is stuck somewhere on the edge, but the product itself is already used in a following station).
For testing we recommend spinning up a local instance of the entire united-manufacturing-hub Helm chart and then sending messages with Node-RED.
5.3 - How to publish a new version
The UMH uses the semantic versioning. This article explains how to increase the version number and what steps are needed to take
Create release branch (e.g., v0.6.0) from staging. Docker will automatically build Docker containers with the tag VERSION-prerelease, e.g., v0.6.0-prerelease
Create PR from release branch to main
Update the helm charts united-manufacturing-hub by going into Charts.yaml and changing the version to the next version including a -prerelease
Adjust repo link https://repo.umh.app in docs/static/examples/development.yaml to the deploy-preview of helm-repo, e.g., https://deploy-preview-515--umh-helm-repo.netlify.app/ with 515 beeing the PR created in 2. Additionally, add a –devel flag to both helm install commands, so that helm considers the prerelease as a valid version.
Go into the folder deployment/helm-repo and execute
helm package ../united-manufacturing-hub/
Go into deployment/helm-repo and execute helm repo index --url https://deploy-preview-515--umh-helm-repo.netlify.app/ --merge index.yaml .
Commit changes. Wait for all container and deploy previews to be created. Conduct test on K300 by specifying the new cloud-init file e.g., https://deploy-preview-515--umh-docs.netlify.app/examples/development.yaml (you can create a bit.ly link for that)
Test
Conduct steps 3 - 6 with changed version v0.6.0 (instead of v0.6.0-prerelease) and changed repo index url: https://repo.umh.app
Remove old helm packages from prerelease from repo
Merge PR from staging to main
Add a new release containing a changelog of all changes
6 - Publications
This page contains multiple interesting publications made around Industry 4.0, mainly thesis from the RWTH Aachen University.
6.1 - Development of a methodology for implementing Predictive Maintenance
Because of high costs and effort PdM is only economically viable on machines and components with high revenue losses due to breakdown and where the failure is almost independent from uptime
Abstract
Objective of this thesis: The goal of this thesis is to develop a methodology to implement Predictive Maintenance (PdM) economically viable into a company. The methodology is then validated in the Digital Capability Center (DCC) Aachen.
Solution process: Maintenance strategies and machine learning algorithms are researched together with methods for optimizing productions lines. This knowledge is then summarized and validated in the DCC Aachen.
Key results: Because of high costs and effort PdM is only economically viable on machines and components with high revenue losses due to breakdown and where the failure is almost independent from uptime and wear. In the DCC Aachen the wind up bearing at the warping machine is identified as a component for a PdM implementation, but a combination of machine learning and existing sensors is not enough for a economically viable implementation.
6.2 - Design and Evaluation of a Blue Ocean Strategy for an Open-Core IIoT Platform Provider in the Manufacturing Sector"
Currently, the market is quite opaque, which makes it difficult to compare providers on the market and thus to compete. This thesis is written in cooperation with the Aachen-based startup developing the IIoT platform “United Manufacturing Hub”. Its objective is to set UMH apart from the existing red-ocean market with the development of a blue ocean strategy.
Final blue ocean strategy and competitive positioning of UMH
Introduction
The ongoing trend of digitization does not exclude the manufacturing sector which poses challenges for both involved parties: machine manufacturers and producing companies. While larger corporations and certain industries, such as the automotive industry, may be already well advanced in digitization, small and medium-sized enterprises (SMEs) often still face difficulties. For many producing companies, the added value of digitization is unclear and they also have strong security concerns about making their data available for analyses. On the other hand, machine manufacturers face problems in their transformation from hardware manufacturers to service providers (Bitkom Research & Ernst & Young 2018, p. 21; VDMA & McKinsey 2020, p. 22; Vargo & Lusch 2008, p. 254ff). Due to these difficulties and the currently high implementation costs, IIoT (industrial internet of things) platforms have so far been deployed rather sporadically. However, they can offer great potential for optimization, for example of service processes or the overall equipment effectiveness (OEE) and will play an important role in maintaining competitiveness in the future (VDMA & McKinsey 2020, p. 27ff).
For this reason, more and more startups and third-party providers are currently establishing businesses that are trying to solve the challenges and problems of both sides with a wide variety of approaches. Currently, the market is quite opaque, which makes it difficult to compare providers on the market and thus to compete. This thesis is written in cooperation with the Aachen-based startup developing the IIoT platform “United Manufacturing Hub” (UMH; UMH Systems GmbH). Its objective is to set UMH apart from the existing red-ocean market with the development of a blue ocean strategy. By redistributing the development focus to attributes that are most relevant to customers in the market and reducing efforts in less relevant areas, the goal is to create a new, non-competitive market (Kim & Mauborgne 2015, p. 24ff). UMH has set itself the task of making the digital transformation as easy as possible for machine manufacturers and producing companies as their end customers. To do this, it is important to know the needs and problems of the customers and to obtain their assessment of the solution approaches. As a starting point for market analysis, this thesis focuses on machine manufacturers as customers of the platform.
The research question is divided into sub-questions, which together contribute to answering the primary question (Karmasin & Ribing 2017, p. 24f). While the topic is elaborated on the example of UMH, the underlying questions can be generalized and are not sufficiently addressed in the existing literature. The concepts further described in chapter 2 provide useful insights into IIoT, open-source platforms, as well as blue ocean strategies, but there is limited literature on the linkages between those topics (e.g., Frank et al. 2019; p. 341ff; Shafiq et al. 2018, p. 1076ff) and none describing a blue ocean strategy in an IIoT platform context. Therefore, the primary research question (PQ) is:
PQ: Which blue ocean strategy has the best potential to set industry standards and establish an IIoT platform in the manufacturing sector?
Currently, most machine manufacturers rely on in-house developed IIoT platforms (Bender et al. 2020, p. 10f), although using an external platform would reduce duplication costs and provide access to existing applications and customers (Evans & Schmalensee 2008, p. 673). This suggests that currently available external IIoT platforms do not sufficiently cover customer needs. To better understand machine manufacturers’ needs and their motivation, the first sub-question (SQ) is therefore:
SQ1: What functionalities do the manufacturers’ platforms include and how were they implemented? Why have machine manufacturers decided to develop their own platform?
The four actions framework in the blue ocean literature suggests that product attributes need to be raised or created to increase the customer value and create new demand while others are reduced or eliminated to achieve cost leadership (Kim & Mauborgne 2015, p. 51). To assess and extend UMH’s solution approaches, the second sub-question is:
SQ2: 11 What functions or features are currently missing from existing platforms on the market? Which attributes must be raised to fulfill the desired customer benefits?
Finally, making the core of the software stack open source is a relevant part of UMH’s disruptive business model. Open source reduces costs and dependence and promotes among other things value creation. Dedrick and West (2004, p. 5f) found that the perceived reliability of Linux-operated servers was lower than that of servers with a proprietary operating system, which could also be the case for an open-source IIoT platform. To examine the effects of the open-source approach the third sub-question is:
SQ3: How does an open-source approach affect the value curve and how is it perceived by machine manufacturers?
To answer these research questions, this thesis first reviews the state of research on digitization and IIoT, platforms, and technology adoption of a market. Next, UMH and its open-core concept are presented based on an implemented proof of concept at the Digital Capability Center (DCC) Aachen. UMH’s competitors are then clustered into infrastructure providers, proprietary IIoT platforms, and system integrators, for which value curves are generated that show the current focus of the providers on the market. Hypotheses are formulated about the requirements of the IIoT market based on the literature, a conversation with Bender and Lewandowski (2021; authors of the underlying paper Bender et al. 2020), and an existing market research by UMH (2020). The hypotheses facilitate the preparation of the three-part interview guideline, each dedicated to answering one sub-question. Finally, the interviews with eight development managers at machine manufacturing companies are evaluated. Based on the findings, the hypotheses are assessed and the blue ocean strategy for UMH open core and premium is derived, thus answering the primary research question.
6.3 - Industrial image processing for quality control in production lines: Development of a decision logic for the application case specific selection of hardware and software
To select suitable hardware components, a five-stage decision logic is developed and implemented as a software application, which suggests suitable components to the user depending on the specified use case and prioritizes them according to list price. In a simulative evaluation, this achieves complexity reductions between 73 and 98% and cost savings between 46 and 93%. A decision between Deep Learning and conventional algorithms can be made based on the given development circumstances as well as the complexity of image features.
This publication was made by Michael Müller as a Master Thesis for the “Institut für Textiltechnik der RWTH Aachen University” in cooperation with Kai Müller (ITA / RWTH Aachen ) and us.
Cognex camera connected with the United Manufacturing Hub open-source stack
Abstract
Objective of this thesis: The goal of the work is the development of a decision logic for the application case-specific selection of hardware and software for image processing systems for for quality control in industrial production. On the hardware side the components components camera, lens and illumination system are considered. On software side, it is decided, depending on the application, whether conventional algorithms or ventional algorithms or methods of Deep Learning are more suitable.
Solution process: Within the scope of a literature search, relevant descriptive variables for standardized for the standardized characterization of technologies and use cases. Furthermore, interdependencies between individual components and properties of the use case will be identified. By means of a market research, a database with concrete product product information. Based on these steps, a set of rules for the selection of hardware and software technologies is derived and tested on a use case in the application case at the Digital Capability Center Aachen. The decision-making logic for selecting hardware components will finally be user-friendly computer application.
Key results: To select suitable hardware components, a five-stage decision logic is developed and implemented as a software application, which suggests suitable components to the user depending on the specified use case and prioritizes them according to list price. In a simulative evaluation, this achieves complexity reductions between 73 and 98% and cost savings between 46 and 93%. A decision between Deep Learning and conventional algorithms can be made based on the given development circumstances as well as the complexity of image features.
Key word: Digital quality control, Technical textiles, Mobiltech, Industry 4.0, Technology selection
6.4 - Deep learning for industrial quality inspection: development of a plug-and-play image processing system
The central result is an overall process overview and a microservice architecture, with the help of which an industrial image processing system can be put into operation on the software side only by configuring the camera and entering the environment variables. Currently, cameras of the GenICam standard with GigE Vision interface and Cognex cameras are supported. The open architecture creates a basic platform for the development of further microservices and subsequent processes in the context of industrial image processing.
This publication was made by Patrick Kunz as a Bachelor Thesis for the “Institut für Textiltechnik der RWTH Aachen University” in cooperation with Kai Müller (ITA / RWTH Aachen ) and us.
MQTT is used as a central element in the open-source architecture for image processing systems
Abstract
Objective of this thesis:
The objective of the thesis is the development of a robust and user-friendly software for an industrial image processing system, which applies deep learning methods. The user of this software will be able to quickly and easily put an image processing system into operation due to its plug-and-play capability and standardized interfaces. The system software is based exclusively on royalty-free software products.
Solution process:
For the development of the overall system, relevant standards, interfaces and software solutions are researched and presented. By dividing the sys- tem into sub-processes, functional requirements for the software are derived and implemented in the development with the general requirements in a system architecture. The implementation and subsequent validation is carried out in the model production for textile wristbands at the Digital Capability Center Aachen.
Key results:
The central result is an overall process overview and a microservice architecture, with the help of which an industrial image processing system can be put into operation on the software side only by configuring the camera and entering the environment variables. Currently, cameras of the GenICam standard with GigE Vision interface and Cognex cameras are supported. The open architecture creates a basic platform for the development of further microservices and subsequent processes in the context of industrial image processing.
6.5 - Development of a decision tool to select appropriate solutions for quality control depending on the defects occurring in the manufacturing process in the automobile branch of the technical-textiles industry
The results of this research provide an overview of the problems being faced regarding quality control during the manufacturing processes of technical textile in the automotive industry. In addition, information on the extent to which digital solutions for quality control are established in the industry is analyzed. Moreover, existing digital quality control solutions and measuring principles to tackle the identified problems in the industry are researched and identified.
This publication was made by Aditya Narayan Mishra as a Master Thesis for the “Institut für Textiltechnik der RWTH Aachen University” in cooperation with Kai Müller (ITA / RWTH Aachen ) and us.
Source: https://www.lindenfarb.de/en/
Abstract
Objective of this thesis: The objective of this thesis is to develop a decision tool regarding the quality control in the manufacturing of technical textiles for the automotive industry. The tool shall enable the access to information about the problems being faced and the consequent defects occurring during the manufacturing of technical textiles in the automotive industry. Subsequently, it shall provide an overview of the corresponding solutions and measuring principles for each of the identified problems
Solution process: Firstly, a literature review is carried out to provide a deep profound understanding to the important quality parameters and defects in each of the manufacturing processes of technical textile. Based on the literature review, a questionnaire is created to perform a market analysis in form of expert interviews. With the help of the market analysis, industry insights to the current status and problems associated with the quality control of manufacturing the technical textile fabrics in the automotive industry are addressed. Afterwards, based on the problems acquired through the expert interviews, the solutions and measuring principles are identified and subsequently a concept for the decision tool is designed.
Key results: The results of this research provide an overview of the problems being faced regarding quality control during the manufacturing processes of technical textile in the automotive industry. In addition, information on the extent to which digital solutions for quality control are established in the industry is analyzed. Moreover, existing digital quality control solutions and measuring principles to tackle the identified problems in the industry are researched and identified.
Key word: Digital quality control, Technical textiles, Mobiltech, Industry 4.0, Technology selection
Finished setup of the time studies in the LEAD-Factory
Abstract
The steadily advancing globalization significantly shapes today’s business environment for companies. Therefore, companies are increasingly under immense cost pressure and need to improve their production efficiency and product quality to remain competitive. Industry 4.0 applications that result from the advancing digitization offer great potential for long-term cost savings. Implementing time studies for mechanical activities can identify potential for improvement in the production process and enable them to be rectified. This thesis is concerned with combining the subject areas Industry 4.0 and the implementation of manual time studies. For this purpose, a digital time recording of manual activities was implemented in the LEAD Factory, the learning factory of the Institute of Innovation and Industrial Management at Graz University of Technology. Therefore, a mobile sensor kit and an IoT platform, provided by the factorycube of Industrial Analytics, were used. Using sensors, existing data from an RFID system, and an energy monitoring system, all activities on a selected workstation in the LEAD Factory can be documented and analyzed. This automated time recording enables long-term measurements to analyze working times and possible anomalies. The collected data is stored in so-called time- series databases and processed using various methods. The data is displayed on a dashboard using a visualization program. One focus of the work was the design of the data processing architecture with two different time-series data models, as well as the conception and development of methods for data processing in the context of time studies. A relational and a NoSQL database system were used equally. The use of two very different approaches should show the possibilities of both systems and enable an assessment of the two systems. Based on a utility analysis, both approaches are evaluated and compared using selected criteria. Thus, a clear recommendation can be made for one of the two approaches. Making the results of the work available to an open-source community, they can be used as a basis for the implementation of similar applications. In addition, the work shows through a digital time recording the huge potential to improve productivity in case of using existing data in a production environment.