Device-Connectivity MQTT Reference

This guide shows how to enable MQTT (3.1.1) communication support for a Murano Product. Once the MQTT protocol is enabled, the example code provided shows how a simulated device connects and activates with the Murano MQTT endpoint.

IMPORTANT: Your hardware & Library MUST be compatible with Murano connectivity TLS Specs., make sure you read it first.

Table of Contents


Overview

Exosite’s MQTT offering supports bi-directional device communication with the Murano Platform using the MQTT protocol. Devices can provision with a Murano Product, publish data to its resources, and receive updates about changes made to its resources.

Communication between the device and the Murano MQTT endpoint is secured by TLS and made available by default on port 8883 or on port 443 as an option.

Murano MQTT communication support translates MQTT communication into native Murano commands, allowing MQTT clients to connect and communicate with the Murano Platform. While the translation layer provides most features of MQTT 3.1.1 there are some special considerations to make for Murano integration.

Messaging Limitations

Murano supports standard MQTT messaging with a few exceptions. Those exceptions are detailed below. MQTT messages or message features not listed below are fully compliant to the MQTT 3.1.1 specification.

CONNECT message "Will Flag", "Will QoS" and "Will Retain" bits

Not supported. The primary purpose of communication from devices to Murano is the live reporting of (typically) sensory data. A last will type of pre-stored message does not make much sense for such applications.

CONNECT message "Clean Session" flag

Ignored, acting as if set to 0. By definition, Murano will sync any resource value that had been updated while the device was offline, back to the device when it connects. This behavior essentially mimics that of when Clean Session is set to 0. Also, since subscriptions are also hardcoded by definition, the device will not have to resubscribe when it reconnects which is also inline with "Clean Session 0" behavior.

CONNACK message "Session Present" flag

Always set to 1. Because Murano devices are auto-subscribed and Murano will hold resource state for devices even before their initial connection and because "Clean Session" is always considered 0, the "Session Present" flag will always be set to 1.

PUBLISH message "Retain" flag

Ignored, acting as if set to 0. Device initiated PUBLISH messages are processed by Murano real time by applying the published value to the correspond device resource state. This will initiate an internal message to application Lua scripts running at the time for processing. Should new application scripts be launched later, the message will not be sent to them regardless of whether the device specified 0 or 1 for the "Retain" flag, essentially acting as if the device had set it to 0.

PUBLISH message "QoS" flags

Fully supports QoS-0 and QoS-1 but ignores QoS-2. If it is set to indicate QoS-2, Murano will act as if QoS-0 was specified.

PUBLISH message "DUP" flag

Ignored. Since it only has meaning for QoS-2 which Murano currently does not support, this flag is ignored by Murano. However, Murano will disconnect a device if it misuses the flag by setting it to 1 while specifying QoS-0.

PUBACK, PUBREC, PUBREL and PUBCOMP messages

Not supported. These messages are not supported because QoS-1 or QoS-2 are not supported. QoS-1 is supported device-to-Murano but not from Murano-to-device, hence PUBACK is supported Murano-to-device. If a device sends a one of these messages to Murano, it will disconnect the device immediately.

SUBSCRIBE message

Limited support. By definition, Murano will sync resource state changes back to devices. This does not require explicit subscription from the device. Murano will send the device a PUBLISH message with QoS-0 whenever there is a state change to one of its resources.

If a device sends a SUBSCRIBE message with a topic filter that it is entitled to, Murano will ack it by returning a SUBACK message with QoS-0, even if the device specifies a higher QoS level. The following are valid topics to subscribe to: "#", "$content", "$provision/{device id}", "$resource/#", "$resource/+" and "$resource/{alias}".

If a device sends a SUBSCRIBE message with a topic filter that does not exist or the device is not entitled to receive messages for, Murano will return a SUBACK with the return code 'failure'.

UNSUBSCRIBE message

Ignored. By definition, Murano will sync resource state changes back to devices using PUBLISH messages. Devices may not opt out of receiving these messages by unsubscribing. Hence, the UNSUBSCRIBE message is accepted, responded to by UNSUBACK but functionally ignored by Murano. That is, after an unsubscription message exchange, the device will continue to receive state updates from Murano.

Topics Support and Isolation

The Murano MQTT implementation is not a full MQTT broker. Rather, it is a protocol level implementation with some auto-subscribe functionality and topics support.

Device connectivity and data exchange from and to Murano, is implemented using standard MQTT messaging and using a predefined set of topics as follows: "$content", "$provision", "$provision/{device id}", "$resource", "$resource/{alias}" and "$resource.batch". Devices are not allowed to publish to other topics and doing so will result in the immediate disconnection of the device.

Anonymous access to the $provision topic is provided to facilitate activating an MQTT device. The processing of activation supplies the device with the credentials (i.e., MQTT password) it needs to authenticate future sessions with Murano.

Murano will use device authentication (token, certificate etc.) to identify a device and its state. It's recommended to leave MQTT ClientId empty because a mismatch with Murano device authentication will make the device unable to connect.

Topics are specially handled in Murano. Topics are not public. Access control isolates an activated device to publishing/subscribing only to that device’s topics even though multiple devices will have subscriptions to identically named topics. A device is not allowed to subscribe to another device's topics. A topic called "$resource/temperature", for example, will represent an isolated data stream for the authenticated device.

Anonymous clients, by contrast, can only publish to the $provision endpoint and can only subscribe to that endpoint’s activation reply topic, "$provision/{device id}", which is unique to each activation request. When connecting a device to a Murano Product for the first time, it must anonymously subscribe to the $provision topic in order to receive its MQTT password via the subscription reply. The password the device receives in the reply represents the "activation" of the client and is what enables successive future connections.

The "$content" topic is shared across devices belonging to the same IoT-Connector. That is, if two different devices of the same IoT-Connector publish to the "$content" topic, both will receive the same list of available contents to download. If, however, two devices belonging to two separate IoT-Connectors, publish to "$content", they will each receive a different list, unique to each IoT-Connector - despite the topic name being same.


MQTT APIs

Provision authentication credentials for a device

To provision a device identity to Murano, you have to send a PUBLISH message with the topic:

$provision/<device_id>

Device ID must conform to the identity format specification defined in the Settings tab of the Product UI.

Note:

  1. This is only for token and password authentication types.
  2. Only password authentication needs to provide the credentials (at least 20 characters) as the request payload. For token authentication, leave the payload empty since the token will be generated by Murano.

Response message with the secret token (When a Product is configured for token authentication):

eY4f3tyrE4Jqe3HsLGnPYf7cACKZlb0uvKVatFxX

Received Data

To retrieve all meta information (name, length, mime, url) for all contents, you have to send a PUBLISH message with the topic:

$content

Leave the request payload empty.

Response message with a list of JSON objects, each one containing the name, length, mime and url keys. For example::

[
  {
    "name": "logo.gif",
    "length": 16214,
    "mime": "image/gif",
    "url": "https://s3.something.something.logo.gif"
  }
]

The values for name, length, mime and url keys are:


Report data

to multiple resources

To report data, you have to send a PUBLISH message with the topic:

$resource/

The request payload should be a JSON object, containing the resource alias as key and value of the data point. For example:

{"state": "on", "temperature": 25}

Murano will assign a timestamp in microseconds.

to specific resource

To report data to a specific resource, you have to send a PUBLISH message with the topic:

$resource/<alias>

The request payload must follow the format specification defined in the Resources Settings. Murano will assign a timestamp in microseconds.

to historical timestamps

To report historical data to multiple resources, you have to send a PUBLISH message with the topic:

$resource.batch

The request payload must be a list of JSON objects, each one containing the timestamp and values keys. For example:

[{"timestamp": 1531111131679000, "values": {"state": "on", "temperature": 25}}, {"timestamp": 1531111131689665, "values": {"temperature": 30}}]

Accepted values for timestamp and values keys are:


Getting Started

MQTT client libraries are readily available. Exosite requires that the library supports TLS and requires that the TLS support is modern enough that it includes Server Name Indication (SNI). Please contact Exosite support if a preferred MQTT client library fails either criteria.

This tutorial will use Python and Eclipse Paho™ MQTT Python Client to connect to Murano. For an example in C using Eclipse Paho™ MQTT C Client, take a look at https://github.com/exosite-garage/mqtt_example.

Requirements

Hardware Setup

A development computer or laptop.

Software Setup

To complete this guide, download and install the following on the development machine:

You must have a Murano account and have created a Product within it. For more information on how to create and account and a Product in the account, visit the Create a Product article for more information.


Setup A Sandbox

This guide will require files to be created on the development machine. Throughout this guide it will be assumed that all commands and files will be run from the following directory:

mkdir ~/murano-mqtt-client
cd ~/murano-mqtt-client

Configure MuranoCLI

In order to proceed, the MuranoCLI tool needs to be initialized and configured to the correct Product.

The following command is interactive. It will prompt you for a username and password as well as the desired Murano Business. It also creates some local default assets for things like hosted applications (which this guide does not use) if they don't exist already.

cd ~/murano-mqtt-client
murano init

You can verify that the correct Product ID is selected by navigating to the Product in a web browser and comparing the Product ID in the web UI with the output of the following command:

murano config product.id

Configuration

Enable MQTT

Enable the MQTT protocol on your Murano Product.

murano setting write Gateway.protocol name mqtt

Verify the current setting is "mqtt":

> murano setting read Gateway.protocol
{:name=>"mqtt", :devmode=>false, :port=>443}

Alternatively you can set it in the web UI:

image alt text

Create a Resource

Edit the file specs/resources.yaml in the sandbox directory (i.e., ~/murano-mqtt-client) to the following:

# ~/murano-mqtt-client/specs/resources.yaml
---
  temp:
      format: number

Now use MuranoCLI to create the resource specified in the above spec file:

murano syncup

Activate Your Device

The default Product settings are such that devices are allowed to register their own identities. In some provisioning models it is required or advantageous to have a list of pre-authorized device identities registered with Murano. This is also known as whitelisting. At this point, make sure that the default setting is applied and saved. Navigate to the SETTINGS tab in your Product web UI and verify that "Allow devices to register their own identity" is selected.

Another way to set or check this is to use Murano CLI:

murano setting write Gateway.provisioning presenter_identity --bool yes
  1. Download Murano Service Root certificate from Murano supported TLS Certificate

Save it under ~/murano-mqtt-client/Murano_Root_CA.cer:

  1. Save the following code into a file called ~/murano-mqtt-client/activate.py:
from paho.mqtt import client as mqtt
import ssl
import logging

# logging.basicConfig(level=logging.DEBUG)

pid = input("Product ID? ")
did = input("Device ID? ")
host = pid + ".m2.exosite.io"
open("product_id.txt", "w").write(pid)
cert = "./Murano_Root_CA.cer"

def on_connect(client, userdata, flags, rc):
    provision_str = "$provision/" + did
    client.publish(provision_str, None, qos=0)

def on_message(client, userdata, msg):
    print("Activation succeeded!")
    token = msg.payload.decode()
    print("Token: ", token)
    open('token.txt', "w").write(token)
    client.disconnect()

def on_disconnect(client, userdata, rc):
    if rc != 0:
        print("Disconnected with error", rc)
        exit()

client = mqtt.Client(client_id="")
logger = logging.getLogger(__name__)
client.enable_logger(logger)

# see https://github.com/eclipse/paho.mqtt.python/blob/1.1/README.rst#tls_set
client.tls_set(
    ca_certs=cert,
    cert_reqs=ssl.CERT_REQUIRED
)

client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_message = on_message

client.connect(host, 443)
client.loop_forever()

Now execute the above script by running the following:

python3 activate.py
  1. Provide the Product ID and a device identity of your choice when the activate.py script prompts you for them.
$ python activate.py
Product ID? x2lmj5npsktbuik9
Device ID? 12345

Activation succeeded!
Token: b7b34f55e948b94841820ea50868a2490632d78f

A successful result, as shown above, activates the device, prints the credential, and saves it to a file called ~/murano-mqtt-client/token.txt for subsequent sessions. Notice that in the Product UI the device has been "activated". Save the device ID for later usage (e.g., 12345

NOTE: The client connected anonymously and then provisioned itself using the provided device identity (e.g., 12345) as the MQTT client ID.


Publish Data

Next, use the returned credentials to reconnect and publish data to the new device in the Murano Product. The data will be published to the temp resource defined a few steps above.

Save the following code into the file ~/murano-mqtt-client/publish.py:

from paho.mqtt import client as mqtt
import ssl
import logging

# logging.basicConfig(level=logging.DEBUG)

pid = open("product_id.txt").read()
print("Using Product ID: ", pid)
host = pid + ".m2.exosite.io"
cert = "./Murano_Root_CA.cer"

def on_connect(client, userdata, flags, rc):
    res = "temp"
    resource = "$resource/" + res
    print("Publishing to Resource: ", res)
    client.publish(resource, input("Value? "), qos=0)
    print("Done. Disconnecting...")
    client.disconnect()

def on_message(client, userdata, msg):
    print("Value set, previous was: ", msg.payload.decode())

def on_disconnect(client, userdata, rc):
    if rc != 0:
        print("Disconnected with error", rc)
        exit()

client = mqtt.Client(client_id="")
logger = logging.getLogger(__name__)
client.enable_logger(logger)

# see https://github.com/eclipse/paho.mqtt.python/blob/1.1/README.rst#tls_set
client.tls_set(
    ca_certs=cert,
    cert_reqs=ssl.CERT_REQUIRED
)

# read auth token from file
token = open("./token.txt").read()
print("Using Token: ", token)
client.username_pw_set("", token)

client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_message = on_message

client.connect(host, 443)
client.loop_forever()

Next, execute the script with the following command:

python3 publish.py

The script prompts the user for the data to send. The device’s resources are represented as topics "$resource/" (e.g., $resource/temp”).

Below is some example output of the script prompting the user for data and then publishing:

$ python3 publish.py
Using Product ID: d23kegyeoxb280000
Using Token: XzE3KU2Zhs9ZDl0cSz0Lf8Xp5Ez7rR0cUa1rO4qE
Publishing to Resource: temp
Value? 23
Done. Disconnecting...

The device’s temp resource value will reflect the value published by the script.

You can check the value by either using the Murano web UI or reading the state from the command line using Murano CLI.

An example of checking the value using Murano CLI is provided below:

$ murano device read 12345 temp
+-------+----------+-----+------------------+
| Alias | Reported | Set | Timestamp        |
+-------+----------+-----+------------------+
| temp  | 72       | 72  | 1505394662994644 |
+-------+----------+-----+------------------+

Example of checking using the web UI:

image alt text


Receiving data

Copy the following code to a file called ~/murano-mqtt-client/subscribe.py:

from paho.mqtt import client as mqtt
import ssl
import logging

# logging.basicConfig(level=logging.DEBUG)

pid = open("product_id.txt", "r").read()
print("Using Product ID: ", pid)
host = pid + ".m2.exosite.io"
cert = "./Murano_Root_CA.cer"

def on_message(client, userdata, msg):
    print("Received: ", msg.payload.decode())

def on_connect(client, userdata, flags, rc):
    print("Waiting for messages")

def on_disconnect(client, userdata, rc):
    if rc != 0:
        print("Disconnected with error", rc)
        exit()

client = mqtt.Client(client_id="")
logger = logging.getLogger(__name__)
client.enable_logger(logger)

# see https://github.com/eclipse/paho.mqtt.python/blob/1.1/README.rst#tls_set
client.tls_set(
    ca_certs=cert,
    cert_reqs=ssl.CERT_REQUIRED
)

token = open("token.txt", "r").read()
print("Using Token: ", token)
client.username_pw_set("", token)

client.on_connect = on_connect
client.on_disconnect = on_disconnect
client.on_message = on_message

client.connect(host, 443)
client.loop_forever()

Executing the script (command provided below) will subscribe to all changes to the Murano device with the token recieved with the activate.py script. The subscribe.py script will print all messages sent to the device.

python3 subscribe.py

Now, either using Murano Product web UI or by using the Murano CLI, change the state of the temp resource:

murano device write 12345 temp=68

Or using the web UI: if you directly enter values in the browser, the device will also receive those changes as messages:

image alt text

After running the command, above, in a separate terminal, the subscribe.py script should print something like the following:

$ python3 subscribe.py
Using Product ID:  n110e3xbifmfk0000
Using Token:  eY4f3tyrE4Jqe3HsLGnPYf7cACKZlb0uvKVatFxX
Waiting for messages
Received:  68

Authentication Types

Field\Auth Type Token Password Certificate
ClientID Empty/DeviceID Empty/DeviceID Empty
Username Empty/DeviceID DeviceID Empty
Password Token Credential Password Credential Empty

Password Authentication

You need to change the provisioning/auth_type to password first. You can use the previous examples; you need to do minor changes.

Activation

Read/publish


Client Certificate

Activation

from paho.mqtt import client as mqtt
import os
import ssl
import logging

# logging.basicConfig(level=logging.DEBUG)

pid = input("Product ID? ")
host = pid + ".m2.exosite.io"
open("product_id.txt", "w").write(pid)
cert = "./Murano_Root_CA.cer"

def on_connect(client, userdata, flags, rc):
    print("Activation succeeded!")
    client.disconnect()

def on_disconnect(client, userdata, rc):
    if rc != 0:
        print("Disconnected with error", rc)
        exit()

client = mqtt.Client(client_id="")
logger = logging.getLogger(__name__)
client.enable_logger(logger)

certfile = "./certs/" + "cert.pem"
keyfile  = "./certs/" + "key.pem"
print("Current dir: " + os.getcwd() + " Certificate: " + certfile + ", Keyfile: " + keyfile)
client.tls_set(
    ca_certs=cert,
    certfile=certfile,
    keyfile=keyfile,
    cert_reqs=ssl.CERT_REQUIRED
)

client.on_connect = on_connect
client.on_disconnect = on_disconnect

client.connect(host, 443)
client.loop_forever()

Now execute the above script by running the following:

python3 certificate_activate.py
  1. Provide the Product ID of your choice when the certificate_activate.py script prompts you for them. The device ID does not need to be set; it will be set to the common name in the certificate automatically
$ python3 certificate_activate.py
Product ID? x2lmj5npsktbuik9

Activation succeeded!

After the success message nothing need to be done you only need to use the same cert/key for any further communication.

Read/publish

from paho.mqtt import client as mqtt
import os
import ssl
import logging

# logging.basicConfig(level=logging.DEBUG)

pid = open("product_id.txt").read()
print("Using Product ID: ", pid)
host = pid + ".m2.exosite.io"
cert = "./Murano_Root_CA.cer"

def on_connect(client, userdata, flags, rc):
    resource = "$resource/" + input("Resource ID? ")
    client.publish(resource, input("Value? "), qos=0)
    client.disconnect()

def on_message(client, userdata, msg):
    print("Value set, previous was: ", msg.payload.decode())

def on_disconnect(client, userdata, rc):
    if rc != 0:
        print("DisConnected with error", rc)
        exit()

client = mqtt.Client(client_id="")
logger = logging.getLogger(__name__)
client.enable_logger(logger)

certfile = "./certs/" + "cert.pem"
keyfile  = "./certs/" + "key.pem"
print("Current dir: " + os.getcwd() + " Certificate: " + certfile + ", Keyfile: " + keyfile)
client.tls_set(
    ca_certs=cert,
    certfile=certfile,
    keyfile=keyfile,
    cert_reqs=ssl.CERT_REQUIRED
)

client.on_connect = on_connect
client.on_disconnect = on_disconnect

client.connect(host, 443)
client.loop_forever()

Next, execute the script with the following command:

python3 certificate_publish.py

The script prompts the user for the data to send. The device’s resources are represented as topics "$resource/" (e.g., $resource/temp”).

Below is some example output of the script prompting the user for data and then publishing:

$ python3 certificate_publish.py
Using Product ID: d23kegyeoxb280000
Publishing to Resource: temp
Value? 23
Done. Disconnecting...

The device’s temp resource value will reflect the value published by the script.


Summary

This guide showed how to configure a Murano Product to use the MQTT internet protocol to provision and activate a simulated device with a Python script, as well as publishing to device resources and subscribing to changes to device resources.

Supported MQTT Hardware

Next steps