Loki Patterns returns incomplete and inaccurate results in DrillDown Logs

[kadjoudi]

Describe the bug
Over a 3-hour time window, Loki's pattern detection produces inaccurate and incomplete results:

• Pattern counts don't match log totals : the count per pattern does not add up to the total number of logs in the selected time range.
• Pattern detection is not representative : only one or very few patterns are identified, even when the log stream contains many distinct patterns. The patterns shown are not representative of the full log volume.

To Reproduce
Have a log stream
Open Grafana Drilldown → Logs → Patterns tab
Set the time range to ~3 hours
Observe that:
Pattern count per row does not sum to the total log count
Only 1–2 patterns are shown despite the UI indicating 51+ patterns exist
Filtering by log level (e.g. error) shows 0 patterns despite logs being present

Expected behavior
Pattern counts should be consistent with the total number of logs in the selected time range

Pattern detection should identify a representative set of patterns across the full time window, not just a narrow sample.

Environment:

• Infrastructure: Cloud / AWS/ GCP

Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.

[trevorwhitney]

This could happen if the customer has many patterns in their log lines, but each with few hits. As a reminder, Loki will only store 300 patterns per stream and log level, and if a pattern gets evicted we lose all previous counts (since it's stored in memory). Loki will also only return patterns with more than 30 hits for the selected time period. You can see how these can negatively effect each other. If you're constantly churning patterns, and losing counts, you'll never have enough hits to meet the threshold for the response.

Here are the highlights I see for this customer

only one or very few patterns are identified, even when the log stream contains many distinct patterns.

so this might be the customer's main problem, the log stream may contain too many patterns. the pattern ingesters will store 300 patterns per stream, per level. If the number of patterns is more than this than patterns are evicted based on LRU. Their eviction rate is not high, but it's non-zero, so there will be patterns lost by this alone. Depending on the frequency of certain patterns in the logs, we could be losing patterns before they hit the minimum cluster size of 30 necessary to be included in the response. This could further account for seeing so few patterns.

Pattern counts don't match log totals

How are you determining this? This is unlikely to ever be exact, due to the same reasons mentioned above.

In addition, we're seeing ~12% of the customer's log lines being dropped by the tokenization process, either because of too few tokens (must be > 4), too many tokens (must be < 80), or the log line is too long.

So, here's where I'd start troubleshooting:

1. Are all the streams successfully getting a detected_level field? This is import to increase distribution, as mentioned above
2. We could try to create a few more streams to better group similar items, in hopes to reduce the number of patterns per stream
3. Fix the ~12% of lines being rejected due to tokenization (thought this also may increase the # of patterns)

Looking at the provided screenshot there is not a lot of tokenization in that line. The less tokenization, the fewer log lines that will "hit" a pattern, and the more patterns you'll end up with