Enforce a standard format for decoded sensor data

[pablojimpas]

Summary

The main explanation of this issue has already been described in a previous one: #237

However, this one is about making a detailed data model specification for the sensors in devices.

Why do we need this?

There's value in going a step further than just giving best practices, actually enforcing a strict format has benefits.

For example, suppose that someone wants to create an app that scans the QR code of a LoRaWAN device, automatically registers it in The Things Stack and starts displaying data from the device's various sensors. Currently, it's impossible to build such an app in a manufacturer-agnostic way, completely decoupled from the device specifics.

Having a known data model/format will allow building integrations/apps using data coming from The Things Stack without caring about what device it's providing that data, the units used…just caring about the capabilities/sensors that it has.

What is already there? What do you see now?

Previous related issue: #237

What is missing? What do you want to see?

A strict data model specification, its implementation in every current decoder, and its enforcement in new devices.

How do you propose to implement this?

• In the spirit of simplicity, I would enforce all units to be in SI, that's one less problem to care about when defining the data model.
• Of course, this enforced data model it's only for sensor data that the devices send, manufacturers would still be able to send all the additional data that they want in whatever format they wish.

Can you do this yourself and submit a Pull Request?

I can help to provide requirements for the spec and maybe migrate current decoders to comply with the format.

Work in progress normalized format

quantity	sensor type	units	data type	validity
temperature	temperature, surface temperature	degrees Celsius	number
humidity	humidity, moisture	percentage	number	[0.0, 100.0]
pressure	barometer, pressure, vapour pressure	hectopascal	number
conductivity	conductivity	microsiemens per cm	number
pH	soil pH, water pH	pH scale	number	[0.0, 14.0]
time	time	second	number
direction	wind direction	degree	number	[0.0, 360.0]
velocity	velocity, wind speed	metre per second	number
acceleration	accelerometer	metre per second squared	number
length	altitude, distance	metre	number
mass		kilogram	number
density	co, co2	kilogram per cubic metre	number
current		ampere	number
voltage		volt	number
power		watt	number
solar radiation	solar radiation	watt per square metre	number
luminous flux		lumen	number
luminous intensity		candela	number
luminance		candela per square metre	number
illuminance		lux	number

[johanstokking]

@pablojimpas thanks for your suggestion.

I think this can be very useful indeed. In my view, there would be a common JSON schema for decoded payload; the normalized payload. Then, there would be another function that maps the output from the uplink decoder to this new schema.

I would like to keep that as a separate, optional step, after the uplink decoder. The decoded uplink is preserved in upstream messages in The Things Stack, and there will be a new field with normalized fields. The reason for this is that this standard JSON schema will never fully cover all device specific fields.

We already track sensors so what's really to be done is drafting a JSON schema with properties for each of these sensor types, that clearly define their data type, unit, description and validity (max/min, scale, patterns, enum values etc).

When we have that, we can define a new function signature maybe normalizeUplink(data) that simply takes the output from decodeUplink() and returns the normalized fields.

The Things Stack would make this available on normalized_payload (next to decoded_payload).

What do you think?

[pablojimpas]

@johanstokking I like your approach! Having this as a separate schema will allow us to draft the solution more quickly and then, since it's an optional step, the transition period will be graceful. Each manufacturer will be able to implement the normalizeUplink(data) function at its own pace.

Now it's just a matter of adding the new normalized_payload field in The Things Stack and defining a good data model for each sensor type. Even that could be done incrementally, a first solution could be made with the most common/easier sensor types like temperature, humidity, pressure, GPS, moisture…

I may be able to contribute to this if I'm pointed in the right direction, since I want to see this happen promptly, but I don't promise anything, looking forward to your comments.

[johanstokking]

We can very well use some help here. Are you proficient with drafting JSON schema?

What we need is to extend the schema with:

1. An uplinkNormalizer (see uplinkDecoder) that takes as input decodeOutput (that's the output from uplinkDecoder) and outputs a new schema normalizedPayload
2. New schema normalizedPayload with properties for each sensor type. This needs a definition of the data types, units, descriptions and validations

If you're not proficient with JSON schema you can also just start with a table defining the sensor types in terms of unit and validity info. We can get that drafted in a JSON schema.

What we may also consider is allowing multiple measurements or an indication of the actual sensor. There may be more than one sensor of the same type, like buttons. CayenneLPP supports numbered channels for this.

[pablojimpas]

We can very well use some help here. Are you proficient with drafting JSON schema?

Unfortunately, I've minimal experience with JSON schemas.

If you're not proficient with JSON schema you can also just start with a table defining the sensor types in terms of unit and validity info. We can get that drafted in a JSON schema.

I'll take care of it. I'll start such a table with all the current sensor types in the first comment of this topic so that it is visible, I will start with just the basics, but I encourage input from all who are interested.

[pablojimpas]

@johanstokking how do you think generic sensor types should be handled? For example, a sensor type of analog input could be a temperature reading or a humidity reading or whatever else.

I also see an issue with the current list of sensor types, there's for example barometer, pressure and vapor pressure sensor types, what's the difference? In theory, they all measure the same physical quantity and should produce the same kind of output.

I think right now it's a bit messy mixing types of sensor with quantities, types of sensor it's a more consumer-facing concept but the actual quantity being measured it's what developers/integrators are interested in. So, maybe the normalized output should only care about types of measurements and not about types of sensors, what do you think?

[johanstokking]

So maybe the normalized output should only care about types of measurements and not about types of sensors, what do you think?

Yes I agree. We should probably define the quantities and the units. Which sensor produced a measurement can be defined in some sort of channel or sensor index.

[pablojimpas]

Yes I agree. We should probably define the quantities and the units. Which sensor produced a measurement can be defined in some sort of channel or sensor index.

So, something like this:

"normalized_payload": {
        "temperature": [
                {
                        "value": 17.5,
                        "unit": "celsius",
                        "sensor": "surface temperature"
                }
        ],
        "pressure": [
                {
                        "value": 1022,
                        "unit": "hectopascal",
                        "sensor": "vapor pressure"
                },
                {
                        "value": 1013.25,
                        "unit": "hectopascal",
                        "sensor": "barometer"
                }
        ]
}

Well, maybe the unit field could be removed, since it will already be specified in the documentation of the format and will always be the same for the same quantity.

[pablojimpas]

@johanstokking I've started working on a JSON schema for the format that I described in my previous comment. I'm only making required the value and sensor properties because the unit property it's not really needed, as I stated earlier, since we're making units standard for this format.

I'll post a PR with my early progress soon, but first let me know if you've something against this proposed format, please.

[johanstokking]

Thanks, see review comments in the PR.

Indeed we don't need units.

Maybe we need to support both simple readings and per-sensor readings?

Like

{
  "temperature": 21.2,
  "humidity": 37.5,
  "sensors": {
    "test": {
      "temperature": 23.5
    },
    "other": {
      "humidity": 41.1,
      "windSpeed": 11.9
    }
  }
}

This way, we can do the following:

1. 90% of sensors will only use the top-level properties as they have at most one sensor of each type

2. We can add multiple sensors of the same type under sensors

3. We can have multiple sensor groups under sensors

The way this would work is that there's a pattern properties within sensors that has the same object type.

[pablojimpas]

Maybe we need to support both simple readings and per-sensor readings?

I agree that would make most common payloads smaller and simpler while still retaining the flexibility to model complex scenarios.

However, I see a flaw with your proposed format. Suppose that an integrator wants to make use of this normalized payload and gets:

{
  "temperature": 21.2,
  "humidity": 37.5
}

How will the end-user app know if temperature it's coming from an ambient sensor or a soil sensor, for example?

I like the simpler approach, but I think we have to keep some context in the data to be useful. Some quantities are used in many situations, and knowing which type of sensor made the measurement gives the context to figure out those situations.

[johanstokking]

How will the end user app know if temperature it's coming from an ambient sensor or a soil sensor for example?

I like the simpler approach but I think we have to retain some context in the data to be useful, some quantities are used in a lot of different scenarios and generally speaking knowing which type of sensor produced the measurement gives the necessary context to infer those scenarios.

Right, so then we have two options:

{
  "soilTemperature": -3.1,
  "airTemperature": 9.4
}

Or:

{
  "temperature": [
    {
      "value": -3.1,
      "source": "soil"
    },
    {
      "value": 9.4,
      "source": "air"
    }
  ]
}

Here, anyone can access temperature[0].value for the first reading, which is often enough. However, when using arrays, the order matters, and that may become problematic and may require the application layer to enumerate the readings and recognize the source. In this example, I made source an arbitrary, sensor-specific identifier.

Therefore I would prefer the first scenario: it's very explicit. There can be different temperatures: soil, water, air, for thermostats also the target temperature etc.

Things like soil moisture levels on different depths (i.e. % at 25/50/75/100 cm deep) wouldn't be normalized this way; we would still need arrays for that. So we can also do both:

{
  "soilMoisture": [
    {
      "value": 29.4,
      "source": "-100cm"
    },
    {
      "value": 36.5,
      "source": "-75cm"
    },
    ...
  ]
}

This way, most applications will just use soilMoisture[0].value but if the device and application works with multiple soil moisture values, it can be modeled.

What do you think?