This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

factorycube-server

The architecture of factorycube-server

factoryinsight

factoryinsight is an open source REST API written in Golang to fetch manufacturing data from a timescaleDB database and calculate various manufacturing KPIs before delivering it to a user visualization, e.g. [Grafana] or [PowerBI].

Features:

  • OEE (Overall Equipment Effectiveness), including various options to investigate OEE losses (e.g. analysis over time, microstop analytics, changeover deep-dives, etc.)
  • Various options to investigate OEE losses further, for example stop analysis over time, microstop analytics, paretos, changeover deep-dives or stop histograms
  • Scalable, microservice oriented approach for Plug-and-Play usage in Kubernetes or behind load balancers (including health checks and monitoring)
  • Compatible with important automation standards, e.g. Weihenstephaner Standards 09.01 (for filling), Omron PackML (for packaging/filling), EUROMAP 84.1 (for plastic), OPC 30060 (for tobacco machines) and VDMA 40502 (for CNC machines)

The openapi documentation can be found here

mqtt-to-postgresql

the tool to store incoming MQTT messages to the postgres / timescaleDB database

Technical information and usage can be found in the documentation for mqtt-to-postgresql

grafana-auth

Proxies request from grafana to various backend services, while authenticating the grafana user. Technical information and usage can be found in the documentation for grafana-proxy

grafana-plugins

Contains our grafana datasource plugin and our input panel

1 - grafana-plugins

Our grafana plugins

umh-datasource

Translates REST calls to factoryinsight into grafana datapoints Technical information and usage can be found in the documentation for umh-datasource

umh-factoryinput-panel

Allows creation of REST requests to the factoryinput service Technical information and usage can be found in the documentation for umh-factoryinput-panel

1.1 - factoryinput-panel

Documentation of factoryinput-panel

This microservice is still in development and is not considered stable for production use.

Getting started

UMH Factoryinput Panel allows to easily execute MQTT messages inside the UMH stack from the Grafana Panel.

Requirements

  • A united manufacturing hub stack
  • External IP or URL of the grafana-proxy server.
    • In most cases it is the same IP as your Grafana dashboard

Installation

If you have installed the UMH-Stack as described in our quick start Tutorial, then this plugin is already installed on your Grafana installation

If you want to develop this Panel further, please follow the instructions below

Build from source

  1. Goto united-manufacturing-hub/grafana-plugins/umh-factoryinput-panel

  2. Install dependencies

yarn install
  1. Build plugin in development mode or run in watch mode
yarn dev
  1. Build plugin in production mode (not recommended due to Issue 32336)
yarn build
  1. Move the resulting dist folder into your grafana plugins directory
  • Windows: C:\Program Files\GrafanaLabs\grafana\data\plugins
  • Linux: /var/lib/grafana/plugins
  1. Rename the folder to umh-factoryinput-panel

  2. Enable the enable development mode to load unsigned plugins

  3. Restart your grafana service

Usage

Prerequisites

  1. Open your Grafana instance
  2. Log in
  3. Open your Profile and check if your organization name inside Grafana matches the rest of your UMH stack

Creating a new Panel

  1. Create a new Dashboard or edit an existing one
  2. Click “Add an empty panel”
  3. On the right sidebar switch the Visualization to “Button Panel”
  4. Fill out the fields inside “REST Integration”
    1. URL
      • http://{URL to your grafana-proxy}/api/v1/factoryinput/
      • Example:
        • http://172.21.9.195:2096/api/v1/factoryinput/
    2. Location
      • Location of your Asset
    3. Asset
      • Name of the Asset
    4. Value
      • MQTT prefix
        • Example prefixes:
          • count
          • addShift
          • modifyShift
    5. Customer
      • Your organization name
    6. Payload
      • JSON encoded payload to send as MQTT message payload
  5. Modify any additional options are you like
  6. When you are finished customizing, click on “Apply”

Example Panel

Notes

  1. Clicking the button will immediately send the MQTT message, through our HTTP->MQTT stack. Please don’t send queries modifying date you would later need !

Common pitfalls

  • Pressing the button just changes the cog to an warning sign
    1. Open your network inspector view (Ctrl+Shift+I on Chrome)
    2. Press the button again
    3. If no request appears, then you haven’t filled out all required fields
    4. Your request shows:
      • 403
        • Make sure the customer field is set to your grafana organization name
      • 400
        • Your request was incorrectly formatted
        • Check that the URL is in the format specified above
        • Check if your payload contains valid JSON
          • You can validate your payload here
        • Check that the Value field matches a valid MQTT command

Technical information

Below you will find a schematic of this flow, through our stack

License

1.2 - umh-datasource

What is United Manufacturing Hub Datasource?

UMH Datasource provides an Grafana 8.X compatible plugin, allowing easy data extraction from the UMH factoryinsight microservice.

Installation

Build from source

  1. Clone the datasource repo [email protected]:united-manufacturing-hub/united-manufacturing-hub-datasource.git

  2. Install dependencies

yarn install
  1. Build plugin in development mode or run in watch mode
yarn dev
  1. Build plugin in production mode (not recommended due to Issue 32336)
yarn build
  1. Move the resulting dist folder into your grafana plugins directory
  • Windows: C:\Program Files\GrafanaLabs\grafana\data\plugins
  • Linux: /var/lib/grafana/plugins
  1. Rename the folder to umh-datasource

  2. You need to enable development mode to load unsigned plugins

  3. Restart your grafana service

From Grafana’s plugin store

TODO

Usage

  1. Open Grafana and login

  2. Open umh-datasource’s settings

  3. Configure your customer name & API Key (automatically configured in Helm deployment)

  4. Configure your server url:

    URL: URL/IP:Port of grafanaproxy

    http://{URL}/api/v1/factoryinsight/

    e.g:

    http://172.21.9.195:2096/api/v1/factoryinsight/

  5. Click “Save & Test”

2 - factoryinput

Documentation of factoryinput

This microservice is still in development and is not considered stable for production use.

This program provides an REST endpoint, to send MQTT messages via HTTP requests. It is typically accessed via grafana-proxy.

Environment variables

This chapter explains all used environment variables.

FACTORYINPUT_USER

Description: Specifies the admin user for the REST API

Type: string

Possible values: all

Example value: jeremy

FACTORYINPUT_PASSWORD

Description: Specifies the password for the admin user for the REST API

Type: string

Possible values: all

Example value: changeme

VERSION

Description: The version of the API to host. Currently, only 1 is a valid value

Type: int

Possible values: all

Example value: 1

CERTIFICATE_NAME

Description: Certificate for MQTT authorization or NO_CERT

Type: string

Possible values: all

Example value: NO_CERT

MY_POD_NAME

Description: The pod name. Used only for tracing, logging and MQTT client id.

Type: string

Possible values: all

Example value: app-factoryinput-0

BROKER_URL

Description: the URL to the broker. Can be prepended either with ssl:// or mqtt:// or needs to specify the port afterwards with :1883

Type: string

Possible values: all

Example value: tcp://factorycube-server-vernemq-local-service:1883

CUSTOMER_NAME_{NUMBER}

Description: Specifies a user for the REST API

Type: string

Possible values: all

Example value: jeremy

CUSTOMER_PASSWORD_{NUMBER}

Description: Specifies the password for the user for the REST API

Type: string

Possible values: all

Example value: changeme

LOGGING_LEVEL

Description: Specifies the logging level. Everything except DEVELOPMENT will be parsed as production (including not set)

Type: string

Possible values: DEVELOPMENT, PRODUCTION

Example value: PRODUCTION

MQTT_QUEUE_HANDLER

Description: Number of queue workers to spawn. If not set, it defaults to 10

Type: uint

Possible values: 0-65535

Example value: 10

Other

  1. Run the program
    • Either using go run github.com/united-manufacturing-hub/united-manufacturing-hub/cmd/factoryinput
    • Or using the Dockerfile
      • Open a terminal inside united-manufacturing-hub
      • Run docker build -f .\deployment\mqtt-to-postgresql\Dockerfile .
      • Look for the image SHA
        • Example: => => writing image sha256:11e4e669d6581df4cb424d825889cf8989ae35a059c50bd56572e2f90dd6f2bc
      • docker run SHA
        • docker run 11e4e669d6581df4cb424d825889cf8989ae35a059c50bd56572e2f90dd6f2bc -e VERSION=1 ....

Rest API

3 - factoryinsight

This document describes the usage of factoryinsight including environment variables and the REST API

Getting started

Here is a quick tutorial on how to start up a basic configuration / a basic docker-compose stack, so that you can develop.

Go to the root folder of the project and execute the following command:

sudo docker build -f deployment/factoryinsight/Dockerfile -t factoryinsight:latest . && sudo docker run factoryinsight:latest 

Environment variables

This chapter explains all used environment variables.

POSTGRES_HOST

Description: Specifies the database DNS name / IP-address for postgresql / timescaleDB

Type: string

Possible values: all DNS names or IP

Example value: factorycube-server

POSTGRES_PORT

Description: Specifies the database port for postgresql

Type: int

Possible values: valid port number

Example value: 5432

POSTGRES_DATABASE

Description: Specifies the database name that should be used

Type: string

Possible values: an existing database in postgresql

Example value: factoryinsight

POSTGRES_USER

Description: Specifies the database user that should be used

Type: string

Possible values: an existing user with access to the specified database in postgresql

Example value: factoryinsight

POSTGRES_PASSWORD

Description: Specifies the database password that should be used

Type: string

Possible values: all

Example value: changeme

FACTORYINSIGHT_USER

Description: Specifies the admin user for the REST API

Type: string

Possible values: all

Example value: jeremy

FACTORYINSIGHT_PASSWORD

Description: Specifies the password for the admin user for the REST API

Type: string

Possible values: all

Example value: changeme

VERSION

Description: The version of the API used (currently not used yet)

Type: int

Possible values: all

Example value: 1

DEBUG_ENABLED

Description: Enables debug logging

Type: string

Possible values: true,false

Example value: false

REDIS_URI

Description: URI for accessing redis sentinel

Type: string

Possible values: All valids URIs

Example value: factorycube-server-redis-node-0.factorycube-server-redis-headless:26379

REDIS_URI2

Description: Backup URI for accessing redis sentinel

Type: string

Possible values: All valids URIs

Example value: factorycube-server-redis-node-1.factorycube-server-redis-headless:26379

REDIS_URI3

Description: Backup URI for accessing redis sentinel

Type: string

Possible values: All valids URIs

Example value: factorycube-server-redis-node-2.factorycube-server-redis-headless:26379

REDIS_PASSWORD

Description: Password for accessing redis sentinel

Type: string

Possible values: all

Example value: changeme

REST API

4 - grafana-proxy

Documentation of grafana-proxy

This microservice is still in development and is not considered stable for production use.

Getting started

This program proxies requests to backend services, if the requesting user is logged into grafana and part of the organization he requests.

Either using go run github.com/united-manufacturing-hub/united-manufacturing-hub/cmd/grafana-auth Or using the Dockerfile - Open an terminal inside united-manufacturing-hub/deployment/grafana-auth - Run docker build -f ./Dockerfile ../..

Environment variables

This chapter explains all used environment variables.

FACTORYINPUT_USER

Description: Specifies the user for the REST API

Type: string

Possible values: all

Example value: jeremy

FACTORYINPUT_KEY

Description: Specifies the password for the user for the REST API

Type: string

Possible values: all

Example value: changeme

FACTORYINPUT_BASE_URL

Description: Specifies the DNS name / IP-address to connect to factoryinput

Type: string

Possible values: all DNS names or IP

Example value: http://factorycube-server-factoryinput-service

FACTORYINSIGHT_BASE_URL

Description: Specifies the DNS name / IP-address to connect to factoryinsight

Type: string

Possible values: all DNS names or IP

Example value: http://factorycube-server-factoryinsight-service

LOGGING_LEVEL

Description: Optional variable, if set to “DEVELOPMENT”, it will switch to debug logging

Type: string

Possible values: Any

Example value: DEVELOPMENT

JAEGER_HOST

Description: Optional variable, Jaeger tracing host

Type: string

Possible values: all DNS names or IP

Example value: http://jaeger.localhost

JAEGER_PORT

Description: Optional variable, Port for Jaeger tracing

Type: string

Possible values: 0-65535

Example value: 9411

DISABLE_JAEGER

Description: Optional variable, disables Jaeger if set to 1 or true

Type: string

Possible values: Any

Example value: 1

Notes

Grafana-Proxy accepts all cors requests. For authenticated requests, you must send your Origin, else CORS will fail

Access-Control-Allow-Headers: content-type, Authorization
Access-Control-Allow-Origin: $(REQUESTING_ORIRING) or *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: *

5 - mqtt-to-blob

The following guide describes how to catch data from the MQTT-Broker and push them to the MIN.io blob storage

This microservice is still in development and is not considered stable for production use.

MQTT-to-Blob has the function of subscribing to MQTT topics and storing the information into a MIN.io blob storage. The current iteration of MQTT-to-Blob can subscribe to cameraconnect’s MQTT topic and send the image data to MIN.io server’s bucket.

The input format of mqtt-to-blob is in accordance with /productImage JSON format. The information from the JSON is extracted and then stored into an image file in MIN.io with the metadata attached.

Before running the code, the environment variables have to be adjusted for the system. Information regarding the environment variables can be seen in the respective section.

Dependencies

  • paho-mqtt==1.6.1
  • numpy==1.21.3
  • minio==7.1.1

Environment variables

This chapter explains all used environment variables.

VariableTypeDescriptionPossible valuesExample value
BROKER_URLstrSpecifies the address to connect to the MQTT-Brokerall DNS names or IP address127.0.0.1
BROKER_PORTintSpecifies the port for the MQTT-Brokervalid port number1883
TOPICstrMQTT Topic namePublished MQTT Topic nameia/rawImage/#
MINIO_URLstrSpecifies the database DNS name / IP-address for the MIN.io serverall DNS names or IP address with its port numberplay.min.io
MINIO_SECUREboolSelect True or False to activate HTTPS connection or notTrue or FalseTrue
MINIO_ACCESS_KEYstrSpecifies the username for MIN.io servereligible usernameusername
MINIO_SECRET_KEYstrSpecifies the password for MIN.io serverpassword of the userpassword
BUCKET_NAMEstrSpecifies the Bucket name to store the blob in MIN.io serverBucket nameBucket-name
LOGGING_LEVELstrSelect level for logging messagesDEBUG INFO ERROR WARNING CRITICALDEBUG

MQTT Connection Return Code

Return CodeMeaning
0Successful Connection
1Connection refused - incorrect protocol version
2Connection refused - invalid client identifier
3Connection refused - server unavailable
4Connection refused - Bad username or password
5Connection refused - not authorised
6-255Currently unused

Future work

  • In the future, MQTT-to-Blob should also be able to store different kind of data such as sound and video

6 - mqtt-to-postgresql

Documentation of mqtt-to-postgresql

mqtt-to-postgresql subscribes to the MQTT broker (in the stack this is VerneMQ), parses incoming messages on the topic “ia/#” and stores them in the postgresql / timescaleDB database (if they are in the correct datamodel)

Getting started

Here is a quick tutorial on how to start up a basic configuration / a basic docker-compose stack, so that you can develop.

docker-compose -f ./deployment/mqtt-to-postgresql/docker-compose-mqtt-to-postgresql-development.yml –env-file ./.env up -d –build

Message processing flow

Below diagram shows an abstract flow, of an incoming MQTT message.

MQTT-Flow

Environment variables

This chapter explains all used environment variables.

POSTGRES_HOST

Description: Specifies the database DNS name / IP-address for postgresql / timescaleDB

Type: string

Possible values: all DNS names or IP

Example value: factorycube-server

POSTGRES_PORT

Description: Specifies the database port for postgresql

Type: int

Possible values: valid port number

Example value: 5432

POSTGRES_DATABASE

Description: Specifies the database name that should be used

Type: string

Possible values: an existing database in postgresql

Example value: factoryinsight

POSTGRES_USER

Description: Specifies the database user that should be used

Type: string

Possible values: an existing user with access to the specified database in postgresql

Example value: factoryinsight

POSTGRES_PASSWORD

Description: Specifies the database password that should be used

Type: string

Possible values: all

Example value: changeme

DRY_RUN

Description: Enables dry run mode (doing everything, even “writing” to database, except committing the changes.)

Type: string

Possible values: true,false

Example value: false

REDIS_URI

Description: URI for accessing redis sentinel

Type: string

Possible values: All valids URIs

Example value: factorycube-server-redis-node-0.factorycube-server-redis-headless:26379

REDIS_URI2

Description: Backup URI for accessing redis sentinel

Type: string

Possible values: All valids URIs

Example value: factorycube-server-redis-node-1.factorycube-server-redis-headless:26379

REDIS_URI3

Description: Backup URI for accessing redis sentinel

Type: string

Possible values: All valids URIs

Example value: factorycube-server-redis-node-2.factorycube-server-redis-headless:26379

REDIS_PASSWORD

Description: Password for accessing redis sentinel

Type: string

Possible values: all

Example value: changeme

MY_POD_NAME

Description: The pod name. Used only for tracing, logging and MQTT client id.

Type: string

Possible values: all

Example value: app-mqtttopostgresql-0

MQTT_TOPIC

Description: Topic to subscribe to. Only set for debugging purposes, e.g., to subscribe to a certain message type. Default usually works fine.

Type: string

Possible values: all possible MQTT topics

Example value: $share/MQTT_TO_POSTGRESQL/ia/#