[CrateDB] Add support for data acquisition and data export - #148
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #148 +/- ##
==========================================
+ Coverage 78.45% 78.88% +0.43%
==========================================
Files 56 59 +3
Lines 3021 3192 +171
==========================================
+ Hits 2370 2518 +148
- Misses 651 674 +23 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
f395199 to
f774a90
Compare
amotl
changed the title
Grafana instant dashboardsAbout8abe55d added baseline support for producing Grafana instant dashboards, and 9c663b2 now improves it by using proper time bucketing within the standard SQL statement template, based on emulating Reference documentation
ThanksThank you for the guidance, @seut and @hammerhead. |
a609bbb to
9c663b2
Compare
amotl
changed the title
fcd4379 to
2a2ec79
Compare
| def record_from_dict(item): | ||
| record = OrderedDict() | ||
| record.update({"time": item["time"]}) | ||
| record.update(item["tags"]) | ||
| record.update(item["fields"]) | ||
| return record |
There was a problem hiding this comment.
I was looking for a function to merge those two objects, tags and fields, into a single record, but have not been able to discover it. Only now, I luckily discovered the concat(object, object) function in @proddata's tutorial 1, and that it can also operate on objects, effectively merging those.
I think the two reasons why I have not been able to discover this function were:
a) That both sets of functions operating on container data types 23 have been on the "Scalar functions" page, and I did not expect to find them there.
b) That the search term "merge" did not occur in the corresponding documentation section of the concat(object, object) function, contrary to the documentation of the array_unique() function.
The
concat(object, object)function combines two objects into a new object.
Thearray_unique(array, array, ...)function merges two arrays into one array with unique elements.
Please let me know if you think this could be improved on the CrateDB documentation.
Footnotes
There was a problem hiding this comment.
Ah. concat() does not completely do what I am aiming at here.
cr> select time, concat(tags, fields) from mqttkit_2_itest.foo_bar_sensors;
+---------------+------------------------------------------+
| time | concat(tags, fields) |
+---------------+------------------------------------------+
| 1687469154383 | {"humidity": 83.1, "temperature": 42.84} |
+---------------+------------------------------------------+
It is nice that it will merge two objects, but now, I would like to destructure the top-level attributes of that single object into individual fields again.
Is there any chance to do this, or, if not, would submitting a corresponding feature request make sense?
There was a problem hiding this comment.
I do not think we can break down the object into fields at the moment, but maybe using object_keys and then referencing the fields one by one could be sufficient for what you are trying to do?
| def write(self, meta, data): | ||
| """ | ||
| Format ingress data chunk and store it into database table. | ||
|
|
||
| TODO: This dearly needs efficiency improvements. Currently, there is no | ||
| batching, just single records/inserts. That yields bad performance. | ||
| """ |
There was a problem hiding this comment.
It should not be forgotten. When wrapping up all review comments, put this TODO item into the backlog for a subsequent iteration.
There was a problem hiding this comment.
Most probably, we should look at using the improvement from crate/crate-python#553 here, if that would be applicable.
| class TimezoneAwareCrateJsonEncoder(json.JSONEncoder): | ||
| epoch_aware = datetime(1970, 1, 1, tzinfo=pytz.UTC) | ||
| epoch_naive = datetime(1970, 1, 1) | ||
|
|
||
| def default(self, o): | ||
| if isinstance(o, Decimal): | ||
| return str(o) | ||
| if isinstance(o, datetime): | ||
| if o.tzinfo: | ||
| delta = o - self.epoch_aware | ||
| else: | ||
| delta = o - self.epoch_naive | ||
| return int(delta.microseconds / 1000.0 + | ||
| (delta.seconds + delta.days * 24 * 3600) * 1000.0) | ||
| if isinstance(o, date): | ||
| return calendar.timegm(o.timetuple()) * 1000 | ||
| return json.JSONEncoder.default(self, o) | ||
|
|
||
|
|
||
| # Monkey patch. | ||
| # TODO: Submit upstream. | ||
| crate.client.http.CrateJsonEncoder = TimezoneAwareCrateJsonEncoder |
There was a problem hiding this comment.
This monkeypatch should be submitted upstream to the Python driver on behalf of the crate Python package. It may resolve https://github.com/crate/crate-python/issues/361.
Effectively, it is only this change:
if o.tzinfo:
delta = o - self.epoch_aware
else:
delta = o - self.epoch_naive| # TODO: Add querying by tags. | ||
| tags = {} | ||
| # tags = CrateDBAdapter.get_tags(data) |
There was a problem hiding this comment.
Do not forget about implementing this, querying by tags. It hasn't been implemented for InfluxDB, but that does not mean it should stay like this.
| { | ||
| "alias": "{{ alias }}", | ||
| "format": "table", | ||
| "resultFormat": "time_series", | ||
| "tags": {{ tags }}, | ||
| "groupByTags": [], | ||
| "measurement": "{{ measurement }}", | ||
| "rawQuery": true, | ||
| "rawSql": "SELECT $__timeGroupAlias(time, $__interval), MEAN(fields['{{ name }}']) AS {{ alias }} FROM {{ table }} WHERE $__timeFilter(time) GROUP BY time ORDER BY time" | ||
| } |
There was a problem hiding this comment.
@hammerhead helped me to discover the right solution for the SQL query here, and he also told me that the DATE_BIN() function, which @seut recommended to use, does not yet understand Grafana's interval values.
This issue is already being tracked at crate/crate#14211. After it has been resolved, adjust the SQL statement template here, to use DATE_BIN() instead of $__timeGroupAlias(time, $__interval).
There was a problem hiding this comment.
By using the new cratedb-grafana-plugin, this will provide a much better usability. Let's check if that needs any adjustments to the data source and dashboard management code.
cratedb-grafana-plugincrate/crate-clients-tools#275- Internal customer-experience testing of Grafana plugin crate/crate-clients-tools#331
Thank you @florinutz. 🚀
| "groupByTags": [], | ||
| "measurement": "{{ measurement }}", | ||
| "rawQuery": true, | ||
| "rawSql": "SELECT $__timeGroupAlias(time, $__interval), MEAN(fields['{{ name }}']) AS {{ alias }} FROM {{ table }} WHERE $__timeFilter(time) GROUP BY time ORDER BY time" |
There was a problem hiding this comment.
On behalf of a subsequent iteration, we may also want to demonstrate advanced downsampling on a secondary panel, using the largest triangle three buckets (LTTB) algorithm, as presented by @hlcianfagna 1, when possible.
Footnotes
There was a problem hiding this comment.
It looks like CrateDB can do LTTB natively today.
| - Extremely fast distributed query execution. | ||
| - Auto-partitioning, auto-sharding, and auto-replication. | ||
| - Self-healing and auto-rebalancing. | ||
| - User-defined functions (UDFs) can be used to extend the functionality of CrateDB. |
There was a problem hiding this comment.
User-defined functions are not mentioned on the canonical CrateDB README at all. It should be added.
There was a problem hiding this comment.
| .. code-block:: sql | ||
|
|
||
| -- An SQL DDL statement defining a custom schema for holding sensor data. | ||
| CREATE TABLE iot_data ( | ||
| timestamp TIMESTAMP WITH TIME ZONE, | ||
| sensor_data OBJECT (DYNAMIC) AS ( | ||
| temperature FLOAT, | ||
| humidity FLOAT, | ||
| location OBJECT (DYNAMIC) AS ( | ||
| latitude DOUBLE PRECISION, longitude DOUBLE PRECISION | ||
| ) | ||
| ) | ||
| ); |
There was a problem hiding this comment.
That's the only SQL DDL statement within the "query examples" section. Adding a few more, including use of other CrateDB special data types, may be sensible. Do you have any suggestions in your toolboxes?
There was a problem hiding this comment.
Any chance of using GEO_POINT maybe?
| CREATE TABLE IF NOT EXISTS {tablename} ( | ||
| time TIMESTAMP WITH TIME ZONE DEFAULT NOW() NOT NULL, | ||
| tags OBJECT(DYNAMIC), | ||
| fields OBJECT(DYNAMIC) | ||
| ); |
There was a problem hiding this comment.
Does it make sense to use such a DDL from the very beginning, as partitioning by year seems to be a general useful approach to be used as a reasonable default, @hlcianfagna?
| CREATE TABLE IF NOT EXISTS {tablename} ( | |
| time TIMESTAMP WITH TIME ZONE DEFAULT NOW() NOT NULL, | |
| tags OBJECT(DYNAMIC), | |
| fields OBJECT(DYNAMIC) | |
| ); | |
| CREATE TABLE IF NOT EXISTS {tablename} ( | |
| time TIMESTAMP WITH TIME ZONE DEFAULT NOW() NOT NULL, | |
| tags OBJECT(DYNAMIC), | |
| fields OBJECT(DYNAMIC), | |
| year TIMESTAMP GENERATED ALWAYS AS DATE_TRUNC('year', time) | |
| ) PARTITIONED BY (year); |
There was a problem hiding this comment.
I see you went for this, I think it is a good idea, yes
| self.db_client = client.connect( | ||
| self.host_uri, username=self.username, password=self.password, pool_size=20, | ||
| ) |
There was a problem hiding this comment.
Verify and demonstrate connecting also to CrateDB Cloud, maybe on behalf of a subsequent iteration.
| into tables. Tables are grouped into schemas, which is equivalent to the concept of hosting | ||
| multiple databases on the same server instance. |
There was a problem hiding this comment.
@proddata mentioned that, at least from a PostgreSQL perspective, databases are more like catalogs. Thanks.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Apply time bucketing over interval obtained from Grafana's date range picker. Emulate `GROUP BY DATE_BIN()` by using Grafana's `$__timeGroupAlias` macro for casting `$__interval` values, until CrateDB's `DATE_BIN()` function understands Grafana's native interval values.
2 participants
About
Migrating to InfluxDB version 2 would mean to leave SQL behind 1. While using the Flux query language is intriguing, and I will not reject bringing in support for InfluxDB2 and its successor IOx, supporting an SQL-based timeseries database makes sense for me, and this time maybe even a more capable one than InfluxDB in terms of broader support for data types and SQL operations.
So, I think viable alternatives are both CrateDB and TimescaleDB 2, which may even share parts of their corresponding adapter implementations, because both are building upon PostgreSQL standards. This patch makes a start by adding support for CrateDB, let's have a look at TimescaleDB later.
Documentation
https://kotori--148.org.readthedocs.build/database/cratedb.html
Backlog
max_byandmin_bycould also be utilized in a sensible way.Footnotes
https://docs.influxdata.com/influxdb/v2.7/query-data/flux/ ↩
With the drawback that TimescaleDB also changed the license for parts of their code to non-FOSS, see https://github.com/timescale/timescaledb/blob/main/tsl/LICENSE-TIMESCALE. ↩