Skip to content

Grafana can not present downsampled data from thanos-query #8567

@ppietka-bp

Description

@ppietka-bp

Thanos, Prometheus and Golang version used:
Prometheus: 3.2.1
thanos --version
thanos, version 0.37.2 (branch: HEAD, revision: 18291a7)
build user: root@ae668ee2d250
build date: 20241211-09:08:24
go version: go1.23.4
platform: linux/amd64
tags: netgo

Object Storage Provider:
Ceph Object Store (RadosGW)

Image

What happened:
Grafana connected to the Thanos data source only shows data without resolution reduction. Thanos-Query shows the full data set when we send a query directly via the webUI interface.
What you expected to happen:
Grafana should read all data correctly.
How to reproduce it (as minimally and precisely as possible):
No reproduction is necessary as this is a persistent state.
Full logs to relevant components:

Anything else we need to know:
thanos configuration parameters:

THANOS_COMPACT_OPTS='--http-address=XX.XX.XX.107:10902 \
--consistency-delay=30m \
--retention.resolution-raw=15d \
--retention.resolution-5m=30d \
--retention.resolution-1h=60d \
--compact.concurrency=2 \
--downsample.concurrency=2 \
--block-files-concurrency=8 \
--compact.enable-vertical-compaction \
--debug.max-compaction-level=4 \
--deduplication.replica-label="label1" \
--delete-delay=24h \
--data-dir=/opt/thanos/thanos-cache/tcompact/ \
--objstore.config-file=/opt/monitoring/ceph-bucket.yml \
--web.disable-cors \
--wait \
--log.level=debug'


THANOS_QUERY_OPTS='--http-address=XX.XX.XX.107:9090
    --grpc-address=XX.XX.XX.107:19091
    --query.replica-label=label2
    --query.replica-label=label1
    --query.partial-response
    --query.auto-downsampling
    --query.promql-engine=thanos
    --query.timeout=5m
    --endpoint=XX.XX.XX.110:19091
    --endpoint=XX.XX.XX.109:19091
    --endpoint=XX.XX.XX.107:30901
    --web.disable-cors
    --log.level=debug'

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions