You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: buffering: document memrb storage type and improve coverage (#2374)
- Add memory ring buffer (memrb) as third buffering mode section
- Update intro from "two modes" to "three modes"
- Add memrb as accepted value for per-input storage.type
- Add memrb configuration examples in YAML and classic formats
- Document memrb_dropped_chunks and memrb_dropped_bytes metrics
- Add note about storage.inherit and global storage.type usage
- Sort per-input settings table alphabetically by key
- Clarify mem_buf_limit applies to both memory-only and memrb modes
- Add storage.inherit configuration example with YAML and classic formats
- fix Log_Level capitalization in classic format examples
Fixes#2373
Signed-off-by: Eric D. Schabell <eric@schabell.org>
Copy file name to clipboardExpand all lines: pipeline/buffering.md
+72-6Lines changed: 72 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -45,7 +45,7 @@ After marking a chunk as irrecoverable, Fluent Bit logs an error message and the
45
45
46
46
## Buffering modes
47
47
48
-
Fluent Bit offers two modes for storing buffered data. Both modes store buffered data in memory, but filesystem buffering is a hybrid method that stores an additional copy of buffered data in the filesystem.
48
+
Fluent Bit offers three modes for storing buffered data. Memory-only and memory ring buffer modes store data exclusively in memory, while filesystem buffering is a hybrid method that stores an additional copy of buffered data in the filesystem.
49
49
50
50
You can set the buffering mode for each active [input plugin](#per-input-settings).
51
51
@@ -61,6 +61,14 @@ When filesystem buffering is enabled, Fluent Bit stores each chunk of buffered d
61
61
62
62
This buffering method is less efficient than memory-only buffering, and uses more system overhead, but is less prone to data loss.
63
63
64
+
### Memory ring buffer buffering
65
+
66
+
When memory ring buffer (`memrb`) buffering is enabled, Fluent Bit stores buffered data in a fixed-size memory ring buffer. Unlike memory-only buffering, when the ring buffer is full, Fluent Bit automatically drops the oldest chunks to make room for new data instead of pausing the input plugin. Use this mode in scenarios where you want to keep the most recent data and can tolerate loss of older data.
67
+
68
+
When chunks are dropped, Fluent Bit tracks the number of dropped chunks and bytes through the `memrb_dropped_chunks` and `memrb_dropped_bytes` metrics. You can monitor these metrics to understand how much data is being discarded.
69
+
70
+
This buffering method prioritizes continuous data ingestion over data completeness, making it suitable for high-throughput scenarios where pausing the input isn't acceptable.
71
+
64
72
## Configuration settings
65
73
66
74
{% hint style="info" %}
@@ -73,17 +81,66 @@ Use the information in this section to configure buffering settings in Fluent Bi
73
81
74
82
In the [`service` section](../administration/configuring-fluent-bit/yaml/service-section.md) of Fluent Bit configuration files, several settings related to buffering are stored in the [`storage` key](../administration/configuring-fluent-bit/yaml/service-section.md#storage-configuration). These are global settings that affect all input and output plugins.
75
83
84
+
To set a default buffering mode for all input plugins, configure `storage.type` and enable `storage.inherit` in the `service` section. Input plugins that don't explicitly set their own `storage.type` will inherit the global value. The following example sets filesystem buffering as the default for all inputs, while allowing individual inputs to override the default:
85
+
86
+
{% tabs %}
87
+
{% tab title="fluent-bit.yaml" %}
88
+
89
+
```yaml
90
+
service:
91
+
flush: 1
92
+
log_level: info
93
+
storage.path: /var/log/flb-storage/
94
+
storage.type: filesystem
95
+
storage.inherit: on
96
+
storage.sync: normal
97
+
storage.max_chunks_up: 128
98
+
99
+
pipeline:
100
+
inputs:
101
+
- name: cpu
102
+
# Inherits storage.type: filesystem from service
103
+
104
+
- name: mem
105
+
storage.type: memory # Overrides the inherited default
106
+
```
107
+
108
+
{% endtab %}
109
+
{% tab title="fluent-bit.conf" %}
110
+
111
+
```text
112
+
[SERVICE]
113
+
flush 1
114
+
Log_Level info
115
+
storage.path /var/log/flb-storage/
116
+
storage.type filesystem
117
+
storage.inherit on
118
+
storage.sync normal
119
+
storage.max_chunks_up 128
120
+
121
+
[INPUT]
122
+
name cpu
123
+
# Inherits storage.type: filesystem from SERVICE
124
+
125
+
[INPUT]
126
+
name mem
127
+
storage.type memory # Overrides the inherited default
128
+
```
129
+
130
+
{% endtab %}
131
+
{% endtabs %}
132
+
76
133
### Per-input settings
77
134
78
135
You can configure buffering settings for any input plugin by using these configuration parameters:
79
136
80
137
| Key | Description | Default |
81
138
| :--- | :--- | :--- |
82
-
|`storage.type`| Specifies the buffering mechanism to use for this input plugin. To enable filesystem buffering, a global [`storage.path`](../administration/configuring-fluent-bit/yaml/service-section.md#storage-configuration) value must be set in the `service` section of your configuration file. Accepted values: `memory`, `filesystem`. |`memory`|
83
-
|`mem_buf_limit`| If memory-only buffering is enabled, sets a limit for how much buffered data the plugin can write to memory. After this limit is reached, the plugin will pause until more memory becomes available. This value must follow [unit size](../administration/configuring-fluent-bit.md#unit-sizes) specifications. If unspecified, no limit is enforced. |`0`|
139
+
|`mem_buf_limit`| Sets a limit for how much buffered data the plugin can write to memory. With memory-only buffering, the plugin pauses until more memory becomes available after this limit is reached. With memory ring buffer (`memrb`) buffering, the oldest chunks are dropped to make room for new data instead of pausing. This value must follow [unit size](../administration/configuring-fluent-bit.md#unit-sizes) specifications. If unspecified, no limit is enforced. |`0`|
84
140
|`storage.pause_on_chunks_overlimit`| If filesystem buffering is enabled, specifies how the input plugin should behave after the global `storage.max_chunks_up` limit is reached. When set to `off`, the plugin will stop buffering data to memory but continue buffering data to the filesystem. When set to `on`, the plugin will stop both memory buffering and filesystem buffering until more memory becomes available. Possible values: `on`, `off`. |`off`|
141
+
|`storage.type`| Specifies the buffering mechanism to use for this input plugin. To enable filesystem buffering, a global [`storage.path`](../administration/configuring-fluent-bit/yaml/service-section.md#storage-configuration) value must be set in the `service` section of your configuration file. When set to `memrb`, Fluent Bit uses a memory ring buffer that automatically drops the oldest chunks when the buffer is full to make room for new data. Accepted values: `memory`, `filesystem`, `memrb`. |`memory`|
85
142
86
-
The following configuration example sets global settings in `service` to support filesystem buffering, then configures one input plugin with filesystem buffering and one input plugin with memory-only buffering:
143
+
The following configuration example sets global settings in `service` to support filesystem buffering, then configures input plugins with filesystem buffering, memory-only buffering, and memory ring buffer buffering:
0 commit comments