Skip to content

Commit 7bf02b6

Browse files
cjc7373cjc7373
authored
chore: supports pulsar 4.0.6 (#2447)
1 parent e381e3d commit 7bf02b6

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+4394
-114
lines changed

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ KubeBlocks add-ons.
3131
| orioledb | orioledb-16.4.0 | OrioleDB is a new storage engine for PostgreSQL, bringing a modern approach to database capacity, capabilities and performance to the world's most-loved database platform. | 1aal |
3232
| polardbx | polardbx-cdc-2.3.0<br>polardbx-cn-2.3.0<br>polardbx-dn-2.3.0<br>polardbx-gms-2.3.0 | PolarDB-X is a cloud native distributed SQL Database designed for high concurrency, massive storage, complex querying scenarios. | Vettal Wu |
3333
| postgresql | postgresql-12.14.0<br>postgresql-12.14.1<br>postgresql-12.15.0<br>postgresql-12.22.0<br>postgresql-14.18.0<br>postgresql-14.7.2<br>postgresql-14.8.0<br>postgresql-15.13.0<br>postgresql-15.7.0<br>postgresql-16.4.0<br>postgresql-16.9.0<br>postgresql-17.5.0<br>postgresql-18.1.0 | A PostgreSQL (with Patroni HA) cluster definition Helm chart for Kubernetes | ldming dingshun-cmss |
34-
| pulsar | pulsar-bookies-recovery-2.11.2<br>pulsar-bookies-recovery-3.0.2<br>pulsar-bookkeeper-2.11.2<br>pulsar-bookkeeper-3.0.2<br>pulsar-broker-2.11.2<br>pulsar-broker-3.0.2<br>pulsar-proxy-2.11.2<br>pulsar-proxy-3.0.2<br>pulsar-zookeeper-2.11.2<br>pulsar-zookeeper-3.0.2 | Apache Pulsar is an open-source, distributed messaging and streaming platform built for the cloud. | caiq1nyu |
34+
| pulsar | pulsar-bookies-recovery-2.11.2<br>pulsar-bookies-recovery-3.0.2<br>pulsar-bookies-recovery-4.0.6<br>pulsar-bookkeeper-2.11.2<br>pulsar-bookkeeper-3.0.2<br>pulsar-bookkeeper-4.0.6<br>pulsar-broker-2.11.2<br>pulsar-broker-3.0.2<br>pulsar-broker-4.0.6<br>pulsar-proxy-2.11.2<br>pulsar-proxy-3.0.2<br>pulsar-proxy-4.0.6<br>pulsar-zookeeper-2.11.2<br>pulsar-zookeeper-3.0.2 | Apache Pulsar is an open-source, distributed messaging and streaming platform built for the cloud. | caiq1nyu |
3535
| qdrant | qdrant-1.10.0<br>qdrant-1.13.4<br>qdrant-1.15.4<br>qdrant-1.5.0<br>qdrant-1.7.3<br>qdrant-1.8.1<br>qdrant-1.8.4 | High-performance, massive-scale Vector Database for the next generation of AI. | iziang |
3636
| rabbitmq | rabbitmq-3.10.25<br>rabbitmq-3.11.28<br>rabbitmq-3.12.14<br>rabbitmq-3.13.7<br>rabbitmq-3.8.34<br>rabbitmq-3.9.29<br>rabbitmq-4.0.9<br>rabbitmq-4.1.6<br>rabbitmq-4.2.1 | RabbitMQ is a reliable and mature messaging and streaming broker. | xuriwuyun |
3737
| redis | redis-5.0.12<br>redis-6.2.14<br>redis-6.2.17<br>redis-6.2.18<br>redis-7.0.6<br>redis-7.2.10<br>redis-7.2.11<br>redis-7.2.12<br>redis-7.2.4<br>redis-7.2.7<br>redis-7.4.2<br>redis-7.4.5<br>redis-7.4.6<br>redis-7.4.7<br>redis-8.0.1<br>redis-8.0.3<br>redis-8.0.4<br>redis-8.0.5<br>redis-8.2.1<br>redis-8.2.2<br>redis-8.2.3<br>redis-8.4.0<br>redis-cluster-5.0.12<br>redis-cluster-6.2.14<br>redis-cluster-6.2.17<br>redis-cluster-6.2.18<br>redis-cluster-7.0.6<br>redis-cluster-7.2.10<br>redis-cluster-7.2.11<br>redis-cluster-7.2.12<br>redis-cluster-7.2.4<br>redis-cluster-7.2.7<br>redis-cluster-7.4.2<br>redis-cluster-7.4.5<br>redis-cluster-7.4.6<br>redis-cluster-7.4.7<br>redis-cluster-8.0.1<br>redis-cluster-8.0.3<br>redis-cluster-8.0.4<br>redis-cluster-8.0.5<br>redis-cluster-8.2.1<br>redis-cluster-8.2.2<br>redis-cluster-8.2.3<br>redis-cluster-8.4.0<br>redis-sentinel-5.0.12<br>redis-sentinel-6.2.14<br>redis-sentinel-6.2.17<br>redis-sentinel-6.2.18<br>redis-sentinel-7.0.6<br>redis-sentinel-7.2.10<br>redis-sentinel-7.2.11<br>redis-sentinel-7.2.12<br>redis-sentinel-7.2.4<br>redis-sentinel-7.2.7<br>redis-sentinel-7.4.2<br>redis-sentinel-7.4.5<br>redis-sentinel-7.4.6<br>redis-sentinel-7.4.7<br>redis-sentinel-8.0.1<br>redis-sentinel-8.0.3<br>redis-sentinel-8.0.4<br>redis-sentinel-8.0.5<br>redis-sentinel-8.2.1<br>redis-sentinel-8.2.2<br>redis-sentinel-8.2.3<br>redis-sentinel-8.4.0<br>redis-twemproxy-0.5.0 | Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps. | ApeCloud wangyelei |

‎addons-cluster/pulsar/templates/NOTES.txt‎

Lines changed: 1 addition & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,6 @@ APP VERSION: {{ .Values.version }}
44

55
** Please be patient while the chart is being deployed **
66

7-
Connection Credentials:
8-
9-
echo "Username : $(kubectl get secret --namespace {{ .Release.Namespace }} {{ include "kblib.clusterName" . }}-conn-credential -o jsonpath="{.data.username}" | base64 -d)"
10-
echo "Password : $(kubectl get secret --namespace {{ .Release.Namespace }} {{ include "kblib.clusterName" . }}-conn-credential -o jsonpath="{.data.password}" | base64 -d)"
11-
echo "Pulsar Endpoint: $(kubectl get secret --namespace {{ .Release.Namespace }} {{ include "kblib.clusterName" . }}-conn-credential -o jsonpath="{.data.pulsarEndpoint}" | base64 -d)"
12-
echo "HTTP Endpoint : $(kubectl get secret --namespace {{ .Release.Namespace }} {{ include "kblib.clusterName" . }}-conn-credential -o jsonpath="{.data.httpEndpoint}" | base64 -d)"
13-
echo "Kafka Endpoint : $(kubectl get secret --namespace {{ .Release.Namespace }} {{ include "kblib.clusterName" . }}-conn-credential -o jsonpath="{.data.kafkaEndpoint}" | base64 -d)"
14-
15-
Reference Pulsar Quick Start Guides: https://pulsar.apache.org/docs/2.11.x/
7+
Reference Pulsar Quick Start Guides: https://pulsar.apache.org/docs/
168

179
Reference Kafka Quick Start Guides: https://kafka.apache.org/documentation/#quickstart

‎addons-cluster/pulsar/templates/cluster.yaml‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ spec:
6363
targetPort: 2181
6464
{{- end }}
6565
componentSpecs:
66-
{{- if or (eq .Values.mode "pulsar-enhanced-cluster") (and (not $topologyEnable) .Values.proxy.enabled) }}
66+
{{- if or (hasPrefix "pulsar-enhanced" .Values.mode) (and (not $topologyEnable) .Values.proxy.enabled) }}
6767
- name: proxy
6868
{{- if not $topologyEnable }}
6969
componentDef: pulsar-proxy
@@ -76,16 +76,16 @@ spec:
7676
{{- if .limits }}
7777
limits:
7878
cpu: {{ .limits.cpu | quote }}
79-
memory: {{ .limits.memory | quote }}
79+
memory: {{ print .limits.memory "Gi" | quote }}
8080
{{- end }}
8181
{{- if .requests }}
8282
requests:
8383
cpu: {{ .requests.cpu | quote }}
84-
memory: {{ .requests.memory | quote }}
84+
memory: {{ print .requests.memory "Gi" | quote }}
8585
{{- end }}
8686
{{- end }}
8787
{{- end }}
88-
{{- if or (eq .Values.mode "pulsar-enhanced-cluster") ( and (not $topologyEnable) .Values.bookiesRecovery.enabled) }}
88+
{{- if or (hasPrefix "pulsar-enhanced" .Values.mode) ( and (not $topologyEnable) .Values.bookiesRecovery.enabled) }}
8989
- name: bookies-recovery
9090
{{- if not $topologyEnable }}
9191
componentDef: pulsar-bookies-recovery
@@ -98,12 +98,12 @@ spec:
9898
{{- if .limits }}
9999
limits:
100100
cpu: {{ .limits.cpu | quote }}
101-
memory: {{ .limits.memory | quote }}
101+
memory: {{ print .limits.memory "Gi" | quote }}
102102
{{- end }}
103103
{{- if .requests }}
104104
requests:
105105
cpu: {{ .requests.cpu | quote }}
106-
memory: {{ .requests.memory | quote }}
106+
memory: {{ print .requests.memory "Gi" | quote }}
107107
{{- end }}
108108
{{- end }}
109109
{{- end }}
@@ -130,12 +130,12 @@ spec:
130130
{{- if .limits }}
131131
limits:
132132
cpu: {{ .limits.cpu | quote }}
133-
memory: {{ .limits.memory | quote }}
133+
memory: {{ print .limits.memory "Gi" | quote }}
134134
{{- end }}
135135
{{- if .requests }}
136136
requests:
137137
cpu: {{ .requests.cpu | quote }}
138-
memory: {{ .requests.memory | quote }}
138+
memory: {{ print .requests.memory "Gi" | quote }}
139139
{{- end }}
140140
{{- end }}
141141
- name: bookies
@@ -154,12 +154,12 @@ spec:
154154
{{- if .limits }}
155155
limits:
156156
cpu: {{ .limits.cpu | quote }}
157-
memory: {{ .limits.memory | quote }}
157+
memory: {{ print .limits.memory "Gi" | quote }}
158158
{{- end }}
159159
{{- if .requests }}
160160
requests:
161161
cpu: {{ .requests.cpu | quote }}
162-
memory: {{ .requests.memory | quote }}
162+
memory: {{ print .requests.memory "Gi" | quote }}
163163
{{- end }}
164164
{{- end }}
165165
{{- if .Values.bookies.persistence.enabled }}

‎addons-cluster/pulsar/values.schema.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,8 @@
1515
"default": "pulsar-enhanced-cluster",
1616
"enum": [
1717
"pulsar-basic-cluster",
18-
"pulsar-enhanced-cluster"
18+
"pulsar-enhanced-cluster",
19+
"pulsar-enhanced-with-kb-zk"
1920
]
2021
},
2122
"bookies": {

‎addons-cluster/pulsar/values.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ bookies:
3737
replicaCount: 3
3838
# supported values: selfVerifying, generic
3939
# refer to https://bookkeeper.apache.org/docs/admin/bookies
40-
mode: generic
40+
mode: selfVerifying
4141
resources:
4242
# We usually recommend not to specify default resources and to leave this as a conscious
4343
# choice for the user. This also increases chances charts run on environments with little

‎addons/pulsar/config/4.0.6/bookies-config.tpl‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
{{- $phy_memory := default 0 $.PHY_MEMORY | int }}
2+
3+
journalDirectories=/pulsar/data/bookkeeper/journal
4+
compactionRateByBytes=52428800
5+
useTransactionalCompaction=true
6+
7+
## for bookies storage configuration
8+
9+
{{- if gt $phy_memory 0 }}
10+
{{- $phy_memory_mb := div $phy_memory ( mul 1024 1024 ) }}
11+
## 25% of the available direct memory
12+
dbStorage_writeCacheMaxSizeMb={{ div $phy_memory_mb 4 }}
13+
dbStorage_readAheadCacheMaxSizeMb={{ div $phy_memory_mb 4 }}
14+
{{- end }}
15+
16+
dbStorage_readAheadCacheBatchSize=100
17+
dbStorage_rocksDB_blockSize=268435456
18+
dbStorage_rocksDB_writeBufferSizeMB=64
19+
dbStorage_rocksDB_sstSizeInMB=64
20+
dbStorage_rocksDB_blockSize=65536
21+
dbStorage_rocksDB_bloomFilterBitsPerKey=10
22+
dbStorage_rocksDB_numLevels=-1
23+
dbStorage_rocksDB_numFilesInLevel0=10
24+
dbStorage_rocksDB_maxSizeInLevel1MB=256
25+
26+
## for disk utilization
27+
# For each ledger dir, maximum disk space which can be used. Default is 0.95f. i.e. 95% of disk can be used at most after which nothing will be written to that partition. If all ledger dir partions are full, then bookie will turn to readonly mode if 'readOnlyModeEnabled=true' is set, else it will shutdown. Valid values should be in between 0 and 1 (exclusive).
28+
diskUsageThreshold=0.95
29+
# The disk free space low water mark threshold. Disk is considered full when usage threshold is exceeded. Disk returns back to non-full state when usage is below low water mark threshold. This prevents it from going back and forth between these states frequently when concurrent writes and compaction are happening. This also prevent bookie from switching frequently between read-only and read-writes states in the same cases.
30+
diskUsageWarnThreshold=0.95
31+
# Set the disk free space low water mark threshold. Disk is considered full when usage threshold is exceeded. Disk returns back to non-full state when usage is below low water mark threshold. This prevents it from going back and forth between these states frequently when concurrent writes and compaction are happening. This also prevent bookie from switching frequently between read-only and read-writes states in the same cases.
32+
diskUsageLwmThreshold=0.9
33+
# Disk check interval in milliseconds. Interval to check the ledger dirs usage.
34+
diskCheckInterval=10000
35+
36+
## for Journal settings
37+
journalDirectories=/pulsar/data/bookkeeper/journal
38+
#journalDirectory=/pulsar/data/bookkeeper/txn
39+
journalFormatVersionToWrite=5
40+
journalMaxSizeMB=2048
41+
journalMaxBackups=5
42+
journalPreAllocSizeMB=16
43+
journalWriteBufferSizeKB=64
44+
journalRemoveFromPageCache=true
45+
journalSyncData=true
46+
journalAdaptiveGroupWrites=true
47+
journalMaxGroupWaitMSec=2
48+
journalBufferedWritesThreshold=524288
49+
journalFlushWhenQueueEmpty=false
50+
journalAlignmentSize=512
51+
# Maximum entries to buffer to impose on a journal write to achieve grouping.
52+
journalBufferedEntriesThreshold=0
53+
journalFlushWhenQueueEmpty=false
54+
journalQueueSize=10000
55+
56+
# TODO fix: For persisiting explicitLac, journalFormatVersionToWrite should be >= 6and FileInfoFormatVersionToWrite should be >= 1
57+
# fileInfoFormatVersionToWrite=1
58+
59+
# how long to wait, in seconds, before starting autorecovery of a lost bookie.
60+
# TODO: set to 0 after opsRequest for rollingUpdate supports hooks
61+
lostBookieRecoveryDelay=300
62+
httpServerEnabled=true
63+
httpServerPort=8000
64+
ledgerDirectories=/pulsar/data/bookkeeper/ledgers
65+
# statsProviderClass ref: https://bookkeeper.apache.org/docs/admin/metrics#stats-providers
66+
statsProviderClass=org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider
67+
enableStatistics=true
68+
useHostNameAsBookieID=true
69+
70+
zkLedgersRootPath=/ledgers
71+
72+
{{- $autoRecoveryDaemonEnabled := "true" }}
73+
{{- range $i, $e := $.cluster.spec.componentSpecs }}
74+
{{- if index $e "componentDef" }}
75+
{{- if eq $e.componentDef "bookies-recovery" }}
76+
{{- $autoRecoveryDaemonEnabled = "false" }}
77+
{{- end }}
78+
{{- end }}
79+
{{- end }}
80+
autoRecoveryDaemonEnabled={{ $autoRecoveryDaemonEnabled }}
81+
82+
zkServers={{ .ZOOKEEPER_SERVERS }}

‎addons/pulsar/config/4.0.6/bookies-recovery-config.tpl‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
httpServerEnabled=true
2+
httpServerPort=8000
3+
prometheusStatsHttpPort=8000
4+
useHostNameAsBookieID=true
5+
# how long to wait, in seconds, before starting autorecovery of a lost bookie.
6+
# TODO: set to 0 after opsRequest for rollingUpdate supports hooks
7+
lostBookieRecoveryDelay=300
8+
9+
zkServers={{ .ZOOKEEPER_SERVERS }}

0 commit comments

Comments
 (0)