I want to cache all measurements inside Core Module and, from time to time, sends all of them using only one/two radio packet(s). Actual time of measurement (from RTC) will be cached together with measured value to preserve correct timestamps of each measurement in database storage.
I’m using typical BC stack:
Core->Radio->Dongle->bcg->Mosquitto->mqtt2influxdb->InfluxDB->Grafana
Questions:
- How do you propagate timestamps from Core Module to InfluxDB? Does mqtt2influxdb supports timestamp translation? I suppose not
.
- Does the radio stack supports sending timestamps with each measurement?
- Is there any project using this approach? I could’n find any on github.
- What is your recommendation?
Thank you.
Hi,
seems like mqtt2influx does not support any kind of external timestamp. Even if it did, you would need to assign each sample different timestamp. This seems like you would need to code your own script.
You can also use node-red and use function node to juggle with bytes and use InfluxDB plugin to write data.
Will the samples be periodic, or do you need precise timestamp with each sample?
In our NB-IoT sensor CHESTER we send with packet this information when we send aggergated data:
- unix timestamp of the first sample (uin32_t)
- period between samples (uint16_t in milliseconds)
- number of samples (uint16_t)
- buffer of float/uint16/32 values
This way we save some payload data, because the sampling is periodic and we can calculate timestamp of other samples.
All this information I would convert to array of bytes and send over radion in a single packet:
bool bc_radio_pub_buffer (void *buffer, size_t length)
Then in python/node-red you do the opposite and reconstruct timestamps and create a object with timestamps which you can simply pass to InfluxDB node that will write values with correct timestamps.
On server we unpack the data and put them back in nice JSON array:
"sensor": {
"counter1": {
"samples": [
{
"timestamp": 1588934459,
"count": 855
},
{
"timestamp": 1588934519,
"count": 856
},
{
"timestamp": 1588934579,
"count": 857
},
{
"timestamp": 1588934639,
"count": 858
},
...
We can help you with the format if you tell more info:
- what values do you sample (temperature?)
- how precise you like values to be? (0.1°C? or 0.5°C?)
- what is a sampling period?
- what is a transmission period?
OK, so there is no support for sending timestamps in SDK/stack yet.
I’ve changed this proposal every time I dig deeper inside your code. Now I understand it enough to think about enhancing your BC stack as follows:
- node (using SDK function) will pack multiple values of the same type with timestamps into one radio packet.
- Structure can be defined by Hardwario, something like you proposed:
[<timestamp>;<period>;<count>;<buffer...>;].bool bc_radio_pub_temperature_buffer(uint8_t channel, uint32_t timetamp, uint16_t period, float *celsius_buf, uint8_t count)
{
uint8_t buffer[3 // header
+ sizeof(uint32_t) // timestamp
+ sizeof(uint16_t) // period
+ sizeof(uint8_t) // samples count
+ sizeof(*celsius)*count]; // samples values
buffer[0] = BC_RADIO_HEADER_PUB_BUFFER; // message type identification
buffer[1] = BC_RADIO_HEADER_PUB_TEMPERATURE;// one sample type identification
buffer[2] = channel;
...
return bc_radio_pub_queue_put(buffer, sizeof(buffer));
}
- bcf-gateway-usb-dongle will:
- unpack data into separate measurements and calculate timestamp for each one
- publish all measurement to the
bcg in json format
- with proper timestamp assigned as part of payload
- to different topic ‘…/temperature/cached’ (because of different payload structure)
- bcg will publish raw payload to mqtt broker like it does now
- mqtt2influxdb will use timestamp provided in mqtt payload and force timestamp of inserted measurement into influxdb
-
Configuration may look like this:
points:
- measurement: temperature
topic: node/+/thermometer/+/temperature/cached
fields:
value: $.payload.value
tags:
id: $.topic[1]
channel: $.topic[3]
timestamp: $.payload.timestamp
Few notes:
-
I think this feature will keep mqtt2influxdb as flexible as it is now and adds one more new feature.
-
I find adding new message format between node and dongle to the SDK useful not only for NB applications.
-
As far as I know, it is only up to you (hardwario) to define what data and in what format will be transferred from node to bcg and to which topics will dongle publish that data.
-
This change will not break any backward compatibility.
-
The only drawback will be the difference in topic between single measurement and multiple (cached/buffered) measurements because of timestamps. But I don’t see any advantage in subscribing both topics and mixing actual and historical data…
-
Another approach I’ve considered was to send timestamp from bcg as MQTT MessageID to preserve usage of same topic like single measurement. In this case mqtt2infuxdb.yml should be like:
points:
- measurement: temperature
...
timestamp: $.id
But ID has only 16bits
-
Maybe bc_data_stream can be used to provide collection of measurements?
bool bc_radio_pub_temperature_from_stream(uint8_t channel, uint32_t timetamp, uint16_t period, bc_data_stream_t stream);
Bc_data_stream documentation
- Temperature
- 0.1 °C
- Sampling period alters between two states:
- After trigger event I want to sample every 10 seconds for about 10 minutes.
- If no trigger within 10 minutes then sample only once per 5 minutes.
- Transmission period is based on two conditions, whichever happens first:
- Once buffer is full (buffer size based on maximum TX packet size).
- At least one time per hour.
The way you extend SDK is another possible approach. You create new type of packet in node firmware, define receiving function in Radio Dongle firmware and send over USB unpacked values with timestamps that propagate to MQTT. However to get it right with timestamps to Influx you also need to edit mqtt2influx
This might be doable and I do not want you to discourage but more complex that I previously suggested.
What I’ve sugested is to keep SDK and dongle firmware completely as is. And in node in you application.c use function to send buffer to send bytes.
bool bc_radio_pub_buffer (void *buffer, size_t length)
So you wouldn’t edit SDK and wouldn’t create new bc_radio_pub_temperature_buffer function, but you do the serialization in you application logic. Please try this function with some array of bytes, you’ll see that the bytes nicely arrives to MQTT as an JSON array.
Then in node-red you decode these bytes and reconstruct the structure that will have multiple measurements and each mesurement will have timestamp and temperature, then you just pass this JSON to InfluxDB node that will write them correctly in a single transaction to the DB.
Data with multiple measurements:
This is how you need to format two values to write them with multiple timestamps.
msg.payload = [
{
measurement: "weather_sensor",
fields: {
temp: 19.6,
},
tags:{
location:"garden"
},
timestamp: 1589269377
},
{
measurement: "weather_sensor",
fields: {
temp: 19.5
},
tags:{
location:"garden"
},
timestamp: 1589269378
}
];
return msg;
Yes, I’ve understood your suggested solution already on 8.5. And no, that’s not what I’m looking for. 
Reasons:
- Node-RED or “my own script” is just another piece of software which:
- I have to care about (keep running 24/7, apply monitoring on it, do updates with crossed fingers, and so on…)
- may fail and I’ll loose my data
- is good quick workaround, but it should’n be final solution
- I can use
bc_radio_pub_buffer() and Node-RED also for all the other measurements I’m already collecting now (without timestamps)…
- …but you have prepared bunch of functions inside SDK to make this process easier…
- …so sending measurements with timestamps should be also as easy as without it.
- You have no support for timestamps inside SDK yet and I’m just curious why:
- Timestamped measurements does not fits into BC’s philosophy?
- Timestamps was not required yet, but there is a plan to add support when needed. If so, what’s BC’s idea?
- BC team has not make any considerations about sending timestamps yet.
Hello,
#1 ok, I understand that if you implement it in SDK you don’t have to deal with more glue-scripts
#3 Nobody till now really needed this functionality. Basic temperature measurement people done did not needed more data than single sample every 15 minutes.
Timestamps was not required yet, but there is a plan to add support when needed. If so, what’s BC’s idea?
Here are some ideas what we need to deal with to make it universal and efficient for everyone:
The idea is that to make it right it would need to go to begining. First the RTC needs to be sychronized with universal time. This needs to be designed, but it will never be perfect for everyone (sleeping nodes…, but we can use bc_radio_set_rx_timeout_for_sleeping_node, search for this function on this forum). You would need somehow to react to current time request and get system time and send it back - script? or even integrate that to bcg?
Then the current screams needs to be refactored to add timestamp (optinally, which is hard in embedded world and do not break or fill RAM). Does it need to have timestamp on every sample, or just first and then period? This adds 4bytes timestamp to every sample. Then you need to plan about RAM utilization to make it most efficient. But if you run just the measuring samples (and not use LCD Module with LED strip…) then you are fine.
Then sending. You need to somehow fit it in around 50-60 bytes which is radio packet. Every user/use case will be different:
- which values to send in a single stream?
- will we make possible to send more streams in a single packet?
- how to encode values to efficient fill the radio buffer so you could fit more samples? If someone needs just room temperature +10 to +40°C and 0.5°C resolution then you can save bits and fill more samples in a single radio packet. Or do you use 4byte float when you really need to send 8 bit value?
and lots of other ideas 
My suggestion is that easiest implementation will be do not use streams, create an array of structures with {uint32_t timestamp, float temperature} and pack it into the buffer and send with your own new function bc_radio_pub_temperature_buffer and unpack it by custom code in Radio Dongle firmware to your custom MQTT topic and format. We can help you with details how to integrate that if you get stuck somewhere.