Surrounding logs feature doesn't work with non-OTEL schema

[sukhada369]

I use HyperDx for Log Management usecase and have a custom schema to handle the non-otel semi-structured data at the moment. The schema on the high level looks like this:

CREATE TABLE logdb.logs_local
(
    `EventTime` DateTime64(3) CODEC(DoubleDelta, ZSTD(1)),
    `EventTimeTTL` DateTime CODEC(Delta(4), ZSTD(1)),
    `ModuleName` LowCardinality(String) DEFAULT '' CODEC(ZSTD(1)),
    `FilePath` String DEFAULT '' CODEC(ZSTD(1)),
    `HostName` LowCardinality(String) DEFAULT '' CODEC(ZSTD(1)),
    `type` LowCardinality(String) DEFAULT '' CODEC(ZSTD(1)),
    `log_timestamp` String DEFAULT '' CODEC(ZSTD(1)),
    `ApplicationName` LowCardinality(String) DEFAULT '' CODEC(ZSTD(1)),
    `LogLevel` LowCardinality(String) DEFAULT '' CODEC(ZSTD(1)),
    `Logger` LowCardinality(String) DEFAULT '' CODEC(ZSTD(1)),
    `Message` String DEFAULT '' CODEC(ZSTD(3)),
    .... other fields....
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}')
PARTITION BY toStartOfDay(EventTime)
ORDER BY (ModuleName, ApplicationName, HostName, toUnixTimestamp(EventTime))
TTL EventTimeTTL + toIntervalDay(30)
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1, min_rows_for_wide_part = 100000, min_bytes_for_wide_part = 5242880, parts_to_delay_insert = 8000, parts_to_throw_insert = 10000

When I open a log for Surrounding logs, all the previously applied filters on ModuleName (equivalent to Service in OTEL) and HostName (equivalent to host/instance in OTEL) are not considered and I see only "Custom" in the filtering on Surrounding logs window. This means that now I have to again apply the filters and find the specific log line for further debugging.

Sample of Surrounding logs from non-otel schema

Sample of Surrounding logs from otel schema

[MikeShi42]

Thanks for the issue @sukhada369 - I'm wondering in this case, what are you expecting to be able to see in the surrounding context? Is it some way to customize the different attributes to match on for surrounding context?

[sukhada369]

@MikeShi42 , example if I have 10 applications sending logs to clickhouse and I use filters and select for 1 application for debugging. When I open surrounding logs, I expect that the selected filter continues in the surrounding logs panel as well.

This means that my selection for that 1 application should retain. But in non-otel scenario, I have to again find the application using filters among those all 10 applications.

Here, if my Service Name Expression is "ModuleName", I want this to retain at least

--- Update to the previous comment:
If we can have all the filters applied on the main screen to carry over as it is to the surrounding logs, that will be a perfect scenario!