Distributed sensor measurements next level

Note	This article is still work in progress!

In our previous project on distributed measurements, we encountered some limitations that needed to be addressed. Specifically, we wanted to:

Easily access data from multiple measurement locations.
This is a requirement if you want to implement a dew point controlled ventilation and therefore need to receive dew point data of outside sensors.
Simplify the maintenance of our sensor networks.
Maintaining the RaspberryPis was quite some effort. In the end most of the measurement locations have been updated to use RaspberryPi Zero Ws which were pretty stable, but the OS still needed to be managed (Using Salt helped to keep them up to date. Still there were some manual staps involved setting up SD cards as a replacement. Faulty drives has been the main reason of failure over the time).
Update: Unfortunately Salt has stopped supporting 32-bit ARM targets so it is not possible right now to update to the most recent Debian without losing Salt support.

To overcome these challenges, we explored new hardware and software solutions.

Hardware Selection

After researching various options, we decided to use the Adafruit QT Py ESP32-S2 as our next-generation sensor platform. This board offers several advantages:

Lower power consumption compared to Raspberry Pi-based solutions.
Easier maintenance due to its embedded system nature.
Improved scalability and replaceability.

We also used the Adafruit CircuitPython drivers, which we were already familiar with from our previous project. To get started, we created a new project repository, integrating some existing codebases.

MQTT Integration

To enable easy data exchange between our sensor networks, we implemented MQTT (Message Queuing Telemetry Transport) as our communication protocol of choice. This allowed us to:

Decouple our sensors from the data storage systems.
Simplify integration with other smart home systems like OpenHAB.

We also created a separate repository for the MQTT gateway component, which uses Rust as its programming language. This ensures that our bridge service is reliable and performant.

Configuration and Operation

To configure the operation of our MQTT bridge, we use a YAML file. This file defines:

The incoming channels (e.g., sensor data). The outgoing storage systems (e.g., InfluxDB or TimescaleDB).

After some weeks of testing and improving the ESP32 modules and the software they are running quite stable and usually have no problem with repeated power outages. The modules are easily configurable with a simple configuration file like this:

WIFI_SSID = "SSDF"
WIFI_PASSWORD = "XZCXCZXC"
MQTT_HOST="mqtt"
MQTT_PREFIX="sensors/test"
ELEVATION="530"

and installing the newest software is automated by using the ./install.py script from the repository mentioned above.

For places with bad WLAN reception there is a version of the ESP32 board which can use an external WLAN antenna but in the end the setup is easier to maintain compared to the RaspberryPi version we used before (Some of the old RasberryPis are still in use and the klimalogger package has been updated to be able to use MQTT for data delivery as well).

Missing software components

MQTT to time series database

Component to store data from MQTT into a timeseries database.

We updated to InfluxDB 2 but also set up an evaluation system with Timescale. For now, we keep using InfluxDB as the Grafana integration for Timescale seemed not up to the level we have reached with InfluxDB.

In order to put all the data being distributed via MQTT to those time series targets we created a separate compontent https://github.com/wuan/mqtt-gateway which uses Rust as a programming language. This should be a good fit for a reliable and performant bridge service.

In order to have time series data from other components like the Shelly relays, plugs and switches we use or our OpenDTU based receiver for our photovoltaic micro inverter we added support and a configuration file base flexibility to the project.

The operation of the bridge (incoming channels and outgoing storage systems) can be configured easily through one YAML file like:

mqttUrl: "mqtt://<hostname>:1883"
mqttClientId: "sensors_gateway"
sources:
  - name: "Sensor data"
    type: "sensor"
    prefix: "sensors"
    targets:
      - type: "influxdb"
        host: "<influx host>"
        port: 8086
        database: "sensors"
      - type: "postgresql"
        host: "<postgres host>"
        port: 5432
        user: "<psql username>"
        password: "<psql password"
        database: "sensors"
  - name: "Shelly data"
    type: "shelly"
    prefix: "shellies"
    targets:
      - type: "influxdb"
        host: "<influx host>"
        port: 8086
        database: "shelly"
      - type: "postgresql"
        host: "<postgres host>"
        port: 5433
        user: "<psql username>"
        password: "<psql password>"
        database: "shelly"
  - name: "PV data"
    type: "opendtu"
    prefix: "solar"
    targets:
      - type: "influxdb"
        host: "<influx host>"
        port: 8086
        database: "solar"

Basically you just need to declare the MQTT prefix for the different kinds of data and then define a list of target storage systems (currently supporting InfluxDB and TimescaleDB).

Integration with OpenHAB

In order to use sensor data for tasks like fan control we also started to integrate the MQTT data with the local running OpenHAB server. OpenHAB allows to configure MQTT items using configuration files like in the following examples.

First we define OpenHAB "Things" using a file AirSensors.things:

Bridge mqtt:broker:local "Local Broker" [ host="<hostname/IP>", secure=false ] {
   ...
  Thing topic Livingroom_Air "Air Livingroom" @ "Livingroom" {
    Channels:
      Type number : Temperature  [ stateTopic="sensors/Livingroom/temperature", transformationPattern="JSONPATH:$.value" ]
      Type number : Humidity  [ stateTopic="sensors/Livingroom/relative humidity", transformationPattern="JSONPATH:$.value" ]
      Type number : Dewpoint  [ stateTopic="sensors/Livingroom/dew point", transformationPattern="JSONPATH:$.value" ]
      Type number : CO2  [ stateTopic="sensors/Livingroom/CO2", transformationPattern="JSONPATH:$.value" ]
      Type number : VOCIndex  [ stateTopic="sensors/Livingroom/VOC index", transformationPattern="JSONPATH:$.value" ]
  }
   ...
}

And as a second step we define OpenHAB "Items" using a file AirSensors.items:

...
Number:Temperature Livingroom_Air_Temperature "Livingroom Temperature [%.1f °C]" (Livingroom) {channel="mqtt:topic:local:Livingroom_Air:Temperature"}
Number             Livingroom_Air_Humidity    "Livingroom Humidity [%.1f %%]"    (Livingroom) {channel="mqtt:topic:local:Livingroom_Air:Humidity"}
Number:Temperature Livingroot_Air_DewPoint    "Livingroom Dewpoint [%.1f °C]"    (Livingroom) {channel="mqtt:topic:local:Livingroom_Air:Dewpoint"}
Number             Livingroom_Air_CO2         "Livingroom CO2 [%.1f ppm]"        (Livingroom) {channel="mqtt:topic:local:Livingroom_Air:CO2"}
Number             Livingroom_Air_VOCIndex    "Livingroom VOC Index"             (Livingroom) {channel="mqtt:topic:local:Livingroom_Air:VOCIndex"}
...

After that you can use the Air sensor data for example to create a fan controlling rule.

Hardware Selection#

MQTT Integration#

Configuration and Operation#

Missing software components#

MQTT to time series database#

Integration with OpenHAB#