-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Grafana can not present downsampled data from thanos-query #8567
Description
Thanos, Prometheus and Golang version used:
Prometheus: 3.2.1
thanos --version
thanos, version 0.37.2 (branch: HEAD, revision: 18291a7)
build user: root@ae668ee2d250
build date: 20241211-09:08:24
go version: go1.23.4
platform: linux/amd64
tags: netgo
Object Storage Provider:
Ceph Object Store (RadosGW)
What happened:
Grafana connected to the Thanos data source only shows data without resolution reduction. Thanos-Query shows the full data set when we send a query directly via the webUI interface.
What you expected to happen:
Grafana should read all data correctly.
How to reproduce it (as minimally and precisely as possible):
No reproduction is necessary as this is a persistent state.
Full logs to relevant components:
Anything else we need to know:
thanos configuration parameters:
THANOS_COMPACT_OPTS='--http-address=XX.XX.XX.107:10902 \
--consistency-delay=30m \
--retention.resolution-raw=15d \
--retention.resolution-5m=30d \
--retention.resolution-1h=60d \
--compact.concurrency=2 \
--downsample.concurrency=2 \
--block-files-concurrency=8 \
--compact.enable-vertical-compaction \
--debug.max-compaction-level=4 \
--deduplication.replica-label="label1" \
--delete-delay=24h \
--data-dir=/opt/thanos/thanos-cache/tcompact/ \
--objstore.config-file=/opt/monitoring/ceph-bucket.yml \
--web.disable-cors \
--wait \
--log.level=debug'
THANOS_QUERY_OPTS='--http-address=XX.XX.XX.107:9090
--grpc-address=XX.XX.XX.107:19091
--query.replica-label=label2
--query.replica-label=label1
--query.partial-response
--query.auto-downsampling
--query.promql-engine=thanos
--query.timeout=5m
--endpoint=XX.XX.XX.110:19091
--endpoint=XX.XX.XX.109:19091
--endpoint=XX.XX.XX.107:30901
--web.disable-cors
--log.level=debug'