Receiving message with generic JSON payload

Hi,
I need to send generic JSON object to core module, like it is used in generic node firmware for controlling LCD or LED strip (GitHub - hardwario/twr-generic-node: Firmware for HARDWARIO Generic Node), for example:

mosquitto_pub -t "node/{id}/lcd/-/text/set" -m '{"x": 5, "y": 40, "text": "HARDWARIO", "font": 28}'

But in SDK, there is no TWR_RADIO_SUB_PT_JSON available in twr_radio_sub_pt_t (see Firmware SDK: twr_radio). In generic node firmware source there is nothing at all regarding communication setup (from what I’ve seen, all communication in your firmwares is handled somewhere inside the SDK).

So, what can I do to be able to receive generic JSON payload in my firmware?

Thanks
Mixi

Hi,
the JSONs to control LCD Module or LED strips are not going over radio nor are decoded in Core Module. The bcg gateway (or Dongle, not sure now) transforms them to byte-level radio packets.

I would not recommend to send data in JSON because it is really data hungry format. The command has to fit into one radio packet which is somewhere around 50 bytes/characters.

You can use TWR_RADIO_SUB_PT_STRING to send string and parse it like comma separated values. But this is also inefficient. You can send JSON there, but very short to fit in those 50 bytes.

Can you explain what exactly would you like to send over radio and control? With that knowledge, we would suggest best solution.

I have firmware for PWM LED control, where I have 7 configuration parameters per PWM channel at this moment (there will be probably even more parameters in future). Making individual topic for each of them is quite overkill - for 16 channels that would be 119 topics (including wildcard topics setting parameter for all channels). I would also prefer setting those parameters in atomic operation, to prevent undefined behavior during configuration provisioning. And I want possibility to update only some of the parameters.

Using TWR_RADIO_SUB_PT_STRING is one of things I tried, but as bcg expects it to be JSON string value (including "), it requires quite a lot of escaping to pass JSON through it (I would like to prevent such hacky workarounds to keep software generating messages clean).

BTW, why is the JSMN JSON parser part of the SDK then (twr-sdk/lib/jsmn)?

Hm, 119 parameters is really not possible to set in a single message. You would need to divide that somehow and if possible, use binary protocol instead of text, to fit the most data to a single packet.

jsmn library from SDK is now used just in bcf-gateway firmware, where we get JSONs over USB/UART converter.

Oh, I really don’t want to send 119 parameters in one message. :wink:
All I want is possibility to send 1 to 7 parameters, where each PWM channel have own topic. My original idea was something like this:

set full config to all channels:

led-pwm/-/config/set: {toBase: 90, toMax: 1800, toStep: 10, trOn: 1.5, trOff: 5, trCh: 0.5, br: 1}

set lower timeouts on 2nd channel:

led-pwm/2/config/set: {toMax: 360, toStep: 5}

set brightness to 20% on 3rd channel:

led-pwm/3/config/set: {br: 0.2}

I’m aware first example is too long to fit into the message, so some more optimalization would be required or full config would have to be split into multiple messages.

In this case, all values are float, so I guess I can just put it into string as is (string values inside JSON would require some escaping which I’m not sure how would be handled by bcg).

Take this post as my stream of thoughts, I started to thing about shortening JSON but then realized this is not the way, bu I keep all my thoughts flow here so you can learn…

JSON isn’t really ideal format or needed in this usage. I checked that twr_radio_sub_pt_t unfortunatelly does not have plain binary format. for the other way from node to dongle there is option to send raw buffer which arrives as array of numbers in MQTT twr_radio_pub_buffer.

So we would need to use string to send settings, but do it more efficiently.

Addressing

since you would like to configure one or many channels by single message, I would suggest just single topic for all data, let’s say led-pwm/-/config/set.

And in the string I would use by bit masking which of eight PWM channels this received configuration applies. Lets name it mask. So for 8 channels it will be 8 bit number and when you send value 0b00000001 then only first channel will be set with incoming configuration, when the value is 0b10000001 = 129 decimal then then first and eight channel will be configured by received parameter.

led-pwm/-/config/set: {mask: 129,toBase: 90, toMax: 1800, toStep: 10, trOn: 1.5, trOff: 5, trCh: 0.5, br: 1}

The address will be a decimal number 0-255 in your message, which will be explained below.

Efficient string encoding

JSON is really terrible. I see that you would like send different parameters, so you would like to have possibility to send only few attributes, but you will need to really shorten them.

led-pwm/-/config/set: {toBase: 90, toMax: 1800, toStep: 10, trOn: 1.5, trOff: 5, trCh: 0.5, br: 1}
  • get rid of spaces, for example here toBase: 90 is extra space, also after every , coma is space.
  • shorten names to less letters
  • make sure that if you set the highest number value to all parameters (worst-case), you will fit into that 45-50 bytes of radio message.

So your message can be

led-pwm/-/config/set: {tBs:90,tMx:1800,tSt:10,trOn:1.5,trOff:5,trCh:0.5,br:1}

Also it does not makes sense for me to send the {} symbols

76 letters becomes 53… which is still too much. So I would suggest completely get rid of JSONs

Comma separated values & positional parameters

if you have 7 parameters and some of them are optional we can use comas to separate them and if the parameter should not be changed, just place two commas ,,.

So this (let say we wouldn’t like to set toMax parameter)
{mask: 129,toBase: 90, /*toMax: 1800*/, toStep: 10, trOn: 1.5, trOff: 5, trCh: 0.5, br: 1}
could look like this

129,90,,10,1.5,5,0.5,1
notice the two commas above ,, for the missing toMax parameter.

This way we have 22 characters.

So you use strchar() function in loop to find all the , symbols and if there is some string in between , you parse them from string to numbers atoi() or atof().

Since I don’t know the range of each value, you have to be careful whether you send big float values like 2 000 000 000, if you send them 7x then you have 70 characters.

If you send some big values, you can also give them coefficients, so you send lets say 10 times smaller value and in the node you multiply it again. It depends how much precision you need.

Another improvement could be that you encode the numbers to hexadecimal, which takes less characters. So uint16_t with biggest value 65535 will have FFFF. Bit since I don’t know the ranges of your parameters it might not help much.

You can also send completely binary data in string, using base64 encoding. You can use twr_base64_decode function in SDK. But this might be more complicated.

You can read my article of efficient coding of values. It gets to the protobuffers, but the first part is generic for any transfer and can help you with some decisions.
https://www.hardwario.com/cs/blog/2021-05-12-protobuffers/

I was meanwhile thinking more about the configuration and remote setup in general and came to conclusion I would like to generalize it and make it much more efficient than JSON allows to. I basically went through similar thought process as you and decided to make some configuration service which would store and manage configuration of all devices and use some efficient binary communication protocol (encoded by base64 as you suggest) able to split payload to several packets. This is mostly because I would want to use same mechanism for configuring devices with displays where I might need to send texts and even bitmaps (icons).

Regarding this binary protocol, I came up with idea of something very similar to protobuffers (thanks a lot for mentioning this). I also wanted it to be “streamable”, so when sending multiple packets, I can decode them as they come and don’t need to store them all in one large buffer (I guess protobuffers might allow this).

As this will require quite a lot of time to get going, I will put it in my “backlog” and for now I will keep using hardcoded configuration.