Skip to content

Commit 989a314

Browse files
fatih-acarclaude
andcommitted
Add Redis Sentinel support to prefect-redis
Build on the Redis Cluster URL groundwork merged in #22211 to add native Redis Sentinel support. Cluster URLs stay gated (detection-only) exactly as in #22211; this change enables the redis+sentinel:// and rediss+sentinel:// schemes across every connection point: the server messaging client, the RedisLockManager, the RedisDatabase block, and the Docket services URL. - New prefect_redis.connection module: a shared URL parser (parse_redis_url) and client builder (build_redis_client) that resolve the current master through the listed Sentinel daemons and follow failover automatically. Data-node connections get pinned TCP keepalive so a silently-dead master is noticed promptly while Sentinel completes failover. - client.py dispatches redis+sentinel:// URLs to the Sentinel builder while leaving #22211's cluster NotImplementedError gate intact. - RedisLockManager and RedisDatabase accept a connection_url that is authoritative over the scalar host/port fields; RedisDatabase round-trips Sentinel URLs through from_connection_string and block_info. - Document the Sentinel schemes on PREFECT_SERVER_DOCKET_URL and bump the pydocket floor to >=0.22.0 for its Sentinel URL support. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent f6efa62 commit 989a314

14 files changed

Lines changed: 1225 additions & 17 deletions

File tree

docs/integrations/prefect-redis/index.mdx

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,81 @@ Register the block types in the `prefect-redis` module to make them available fo
3030
prefect block register -m prefect_redis
3131
```
3232

33+
## Connecting with a URL (including Redis Sentinel)
34+
35+
Every `prefect-redis` connection point — the server messaging client, the
36+
`RedisLockManager`, and the `RedisDatabase` block — accepts a full connection URL in
37+
addition to the individual `host`/`port`/`db`/`username`/`password`/`ssl` fields. When
38+
a URL is provided it is authoritative and the scalar fields are ignored.
39+
40+
The following schemes are supported:
41+
42+
| Scheme | Behavior |
43+
| --- | --- |
44+
| `redis://` | Single-node TCP connection |
45+
| `rediss://` | Single-node TLS connection |
46+
| `redis+sentinel://` | [Redis Sentinel](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/) discovery (TCP data nodes) |
47+
| `rediss+sentinel://` | Redis Sentinel discovery (TLS data nodes) |
48+
49+
The Sentinel schemes accept a comma-separated list of Sentinel members and a master
50+
group name, and the resulting client resolves the current master through the Sentinel
51+
daemons and follows failover automatically:
52+
53+
```
54+
redis+sentinel://[user:pass@]sentinel-a:26379,sentinel-b:26379/<master-name>[/<db>][?params]
55+
```
56+
57+
Sentinel-specific query parameters configure the connections to the Sentinel daemons
58+
separately from the data nodes: `sentinel_username`, `sentinel_password`, `sentinel_ssl`,
59+
`sentinel_tls_insecure`, and `sentinel_tls_ca_file`. Single-node TLS URLs additionally
60+
accept `tls_insecure` and `tls_ca_file`.
61+
62+
### Server messaging (events / streams)
63+
64+
Point the Prefect server's Redis messaging at a Sentinel topology with a single
65+
environment variable:
66+
67+
```bash
68+
export PREFECT_REDIS_MESSAGING_URL="redis+sentinel://sentinel-a:26379,sentinel-b:26379/mymaster"
69+
```
70+
71+
The same setting also accepts single-node `redis://` / `rediss://` URLs.
72+
73+
### Background services (Docket)
74+
75+
Multi-server deployments also point `PREFECT_SERVER_DOCKET_URL` at the same shared
76+
Redis so the server's background services coordinate through it. That URL is
77+
resolved by [Docket](https://github.com/chrisguidry/docket) rather than
78+
`prefect-redis`, so Sentinel URLs require a `pydocket` release that includes Redis
79+
Sentinel support:
80+
81+
```bash
82+
export PREFECT_SERVER_DOCKET_URL="redis+sentinel://sentinel-a:26379,sentinel-b:26379/mymaster/0"
83+
```
84+
85+
### Lock manager
86+
87+
{/* pmd-metadata: notest */}
88+
```python
89+
from prefect_redis import RedisLockManager
90+
91+
lock_manager = RedisLockManager(
92+
connection_url="redis+sentinel://sentinel-a:26379,sentinel-b:26379/mymaster"
93+
)
94+
```
95+
96+
### `RedisDatabase` block
97+
98+
{/* pmd-metadata: notest */}
99+
```python
100+
from prefect_redis import RedisDatabase
101+
102+
block = RedisDatabase.from_connection_string(
103+
"redis+sentinel://sentinel-a:26379,sentinel-b:26379/mymaster"
104+
)
105+
block.save("BLOCK_NAME")
106+
```
107+
33108
## Resources
34109

35110
Refer to the [SDK documentation](/integrations/prefect-redis/api-ref/prefect_redis-blocks) to explore all the capabilities of `prefect-redis`.

docs/v3/advanced/self-hosted.mdx

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,14 @@ For Redis instances that require SSL/TLS:
189189
export PREFECT_SERVER_DOCKET_URL="rediss://redis-host:6379/0"
190190
```
191191

192+
For a Redis deployment fronted by [Redis Sentinel](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/), use the `redis+sentinel://` scheme with a comma-separated list of Sentinel daemons and the master group name. Docket discovers the current master through Sentinel and follows failover automatically:
193+
194+
```bash
195+
export PREFECT_SERVER_DOCKET_URL="redis+sentinel://sentinel-a:26379,sentinel-b:26379,sentinel-c:26379/mymaster/0"
196+
```
197+
198+
Use the `rediss+sentinel://` scheme for SSL/TLS. This requires `pydocket>=0.22.0`.
199+
192200
<Note>
193201
The Docket URL can use the same Redis instance as the messaging configuration above, but you may use a different database number (e.g., `/1` instead of `/0`) to keep the data separate.
194202
</Note>

docs/v3/api-ref/settings-ref.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1691,7 +1691,7 @@ The name of the Docket instance.
16911691
`PREFECT_SERVER_DOCKET_NAME`
16921692

16931693
### `url`
1694-
The URL of the Redis server to use for Docket.
1694+
The URL of the Redis server to use for Docket. Supports the memory:// (single-server only), redis://, rediss://, redis+sentinel:// and rediss+sentinel:// schemes; the Sentinel schemes discover the current master through the listed Sentinel daemons and follow failover automatically (requires pydocket>=0.22.0).
16951695

16961696
**Type**: `string`
16971697

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ dependencies = [
8383
"whenever>=0.7.3,<0.11.0; python_version>='3.13'",
8484
"semver>=3.0.4",
8585
"pluggy>=1.6.0",
86-
"pydocket>=0.19.0",
86+
"pydocket>=0.22.0",
8787
]
8888
[project.urls]
8989
Changelog = "https://github.com/PrefectHQ/prefect/releases"

schemas/settings.schema.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1502,7 +1502,7 @@
15021502
},
15031503
"url": {
15041504
"default": "memory://",
1505-
"description": "The URL of the Redis server to use for Docket.",
1505+
"description": "The URL of the Redis server to use for Docket. Supports the memory:// (single-server only), redis://, rediss://, redis+sentinel:// and rediss+sentinel:// schemes; the Sentinel schemes discover the current master through the listed Sentinel daemons and follow failover automatically (requires pydocket>=0.22.0).",
15061506
"supported_environment_variables": [
15071507
"PREFECT_SERVER_DOCKET_URL"
15081508
],

src/integrations/prefect-redis/prefect_redis/blocks.py

Lines changed: 54 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
"""Redis credentials handling"""
22

3-
from typing import Any, Dict, Optional, Union
3+
from typing import Any, Dict, Optional, Union, cast
4+
from urllib.parse import urlsplit
45

56
import redis
67
import redis.asyncio
@@ -9,6 +10,11 @@
910
from redis.asyncio.connection import parse_url
1011

1112
from prefect.filesystems import WritableFileSystem
13+
from prefect_redis.connection import (
14+
SCHEME_SENTINEL,
15+
build_redis_client,
16+
parse_redis_url,
17+
)
1218

1319
DEFAULT_PORT = 6379
1420

@@ -59,10 +65,24 @@ class RedisDatabase(WritableFileSystem):
5965
username: Optional[SecretStr] = Field(default=None, description="Redis username")
6066
password: Optional[SecretStr] = Field(default=None, description="Redis password")
6167
ssl: bool = Field(default=False, description="Whether to use SSL")
68+
connection_url: Optional[SecretStr] = Field(
69+
default=None,
70+
description=(
71+
"Full Redis connection URL, authoritative over the scalar connection fields "
72+
"when set. Supports the redis://, rediss://, redis+sentinel:// and "
73+
"rediss+sentinel:// schemes; the Sentinel schemes resolve the current master "
74+
"through the listed Sentinel daemons and follow failover automatically, e.g. "
75+
"redis+sentinel://sentinel-a:26379,sentinel-b:26379/mymaster."
76+
),
77+
)
6278

6379
def block_initialization(self) -> None:
6480
"""Validate parameters"""
6581

82+
if self.connection_url is not None:
83+
# Fail fast on a malformed URL rather than at first connection.
84+
parse_redis_url(self.connection_url.get_secret_value())
85+
return
6686
if not self.host:
6787
raise ValueError("Missing hostname")
6888
if self.username and not self.password:
@@ -100,8 +120,16 @@ def get_client(self) -> redis.Redis:
100120
"""Get Redis Client
101121
102122
Returns:
103-
An initialized Redis async client
123+
An initialized Redis client
104124
"""
125+
if self.connection_url is not None:
126+
return cast(
127+
redis.Redis,
128+
build_redis_client(
129+
parse_redis_url(self.connection_url.get_secret_value()),
130+
asynchronous=False,
131+
),
132+
)
105133
return redis.Redis(
106134
host=self.host,
107135
port=self.port,
@@ -117,6 +145,14 @@ def get_async_client(self) -> redis.asyncio.Redis:
117145
Returns:
118146
An initialized Redis async client
119147
"""
148+
if self.connection_url is not None:
149+
return cast(
150+
redis.asyncio.Redis,
151+
build_redis_client(
152+
parse_redis_url(self.connection_url.get_secret_value()),
153+
asynchronous=True,
154+
),
155+
)
120156
return redis.asyncio.Redis(
121157
host=self.host,
122158
port=self.port,
@@ -135,18 +171,27 @@ def from_connection_string(
135171
Supports the following URL schemes:
136172
- `redis://` creates a TCP socket connection
137173
- `rediss://` creates a SSL wrapped TCP socket connection
174+
- `redis+sentinel://` / `rediss+sentinel://` discover the master through
175+
Redis Sentinel and follow failover automatically
138176
139177
Args:
140178
connection_string: Redis connection string
141179
142180
Returns:
143-
`RedisCredentials` instance
181+
`RedisDatabase` instance
144182
"""
145-
connection_kwargs = parse_url(
183+
raw_connection_string = (
146184
connection_string
147185
if isinstance(connection_string, str)
148186
else connection_string.get_secret_value()
149187
)
188+
189+
# Sentinel URLs cannot be flattened to scalar host/port fields, so they are
190+
# retained verbatim and resolved through the Sentinel daemons at connect time.
191+
if urlsplit(raw_connection_string).scheme.lower() in SCHEME_SENTINEL:
192+
return cls(connection_url=SecretStr(raw_connection_string))
193+
194+
connection_kwargs = parse_url(raw_connection_string)
150195
ssl = connection_kwargs.get("connection_class") == redis.asyncio.SSLConnection
151196
return cls(
152197
host=connection_kwargs.get("host", "localhost"),
@@ -174,4 +219,9 @@ def as_connection_params(self) -> Dict[str, Any]:
174219
else:
175220
data.pop("password", None)
176221

222+
if self.connection_url is not None:
223+
data["connection_url"] = self.connection_url.get_secret_value()
224+
else:
225+
data.pop("connection_url", None)
226+
177227
return data

src/integrations/prefect-redis/prefect_redis/client.py

Lines changed: 47 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
import asyncio
22
import functools
33
import warnings
4-
from typing import Any, Callable, Optional, Union
4+
from typing import Any, Callable, Optional, Union, cast
55
from urllib.parse import urlparse, urlunparse
66

77
from pydantic import Field, model_validator
@@ -12,6 +12,11 @@
1212
PrefectBaseSettings,
1313
build_settings_config, # type: ignore[reportPrivateUsage]
1414
)
15+
from prefect_redis.connection import (
16+
SCHEME_SENTINEL,
17+
build_redis_client,
18+
parse_redis_url,
19+
)
1520

1621
_UNSET: Any = object()
1722

@@ -39,8 +44,12 @@ class RedisMessagingSettings(PrefectBaseSettings):
3944
default=None,
4045
description=(
4146
"Full Redis URL (e.g. redis://user:pass@host:6379/0 or "
42-
"rediss://… for TLS). When set, host/port/db/username/"
43-
"password/ssl are ignored."
47+
"rediss://… for TLS). Also supports redis+sentinel:// and "
48+
"rediss+sentinel:// for Redis Sentinel; the Sentinel schemes "
49+
"accept a comma-separated list of members and a master group "
50+
"name, e.g. "
51+
"redis+sentinel://sentinel-a:26379,sentinel-b:26379/mymaster. "
52+
"When set, host/port/db/username/password/ssl are ignored."
4453
),
4554
)
4655
host: str = Field(default="localhost")
@@ -108,6 +117,11 @@ def is_cluster_url(url: str) -> bool:
108117
return url.partition("://")[0] in {"redis+cluster", "rediss+cluster"}
109118

110119

120+
def is_sentinel_url(url: str) -> bool:
121+
"""Return True if the URL uses a Redis Sentinel scheme."""
122+
return urlparse(url).scheme in SCHEME_SENTINEL
123+
124+
111125
def normalize_cluster_url(url: str) -> str:
112126
"""Return a redis-py compatible URL for Redis Cluster connections."""
113127
parsed = urlparse(url)
@@ -200,11 +214,14 @@ def get_async_redis_client(
200214
201215
When a standalone `url` is provided (or configured via
202216
`PREFECT_REDIS_MESSAGING_URL`), `Redis.from_url` is used and
203-
the discrete host/port/… arguments are ignored. Redis Cluster
204-
URLs are detected but intentionally not enabled yet.
217+
the discrete host/port/… arguments are ignored. `redis+sentinel://`
218+
and `rediss+sentinel://` URLs resolve the current master through the
219+
listed Sentinel daemons and follow failover automatically. Redis
220+
Cluster URLs are detected but intentionally not enabled yet.
205221
206222
Args:
207-
url: Full Redis URL (e.g. `redis://localhost:6379/0`).
223+
url: Full Redis URL (e.g. `redis://localhost:6379/0` or
224+
`redis+sentinel://sentinel-a:26379,sentinel-b:26379/mymaster`).
208225
host: The host location.
209226
port: The port to connect to the host with.
210227
db: The Redis database to interact with.
@@ -235,6 +252,20 @@ def get_async_redis_client(
235252

236253
url = url or settings.url
237254
if url:
255+
if is_sentinel_url(url):
256+
return cast(
257+
Redis,
258+
build_redis_client(
259+
parse_redis_url(url),
260+
asynchronous=True,
261+
health_check_interval=health_check_interval
262+
or settings.health_check_interval,
263+
decode_responses=decode_responses,
264+
socket_timeout=resolved_socket_timeout,
265+
socket_connect_timeout=resolved_socket_connect_timeout,
266+
protocol=resolved_protocol,
267+
),
268+
)
238269
if is_cluster_url(url):
239270
_raise_cluster_not_supported()
240271
return Redis.from_url(
@@ -275,6 +306,16 @@ def async_redis_from_settings(
275306
}
276307

277308
if settings.url:
309+
if is_sentinel_url(settings.url):
310+
return cast(
311+
Redis,
312+
build_redis_client(
313+
parse_redis_url(settings.url),
314+
asynchronous=True,
315+
health_check_interval=settings.health_check_interval,
316+
**options,
317+
),
318+
)
278319
if is_cluster_url(settings.url):
279320
_raise_cluster_not_supported()
280321
return Redis.from_url(

0 commit comments

Comments
 (0)