Synchronize state between multiple replicas of Gateway Controller

[VirajSalaka]

Problem

Gateway Controller needs to run multiple replicas for the sake of high availability and all those replicas should have the same XDS cache state populated.

Solution

Abstract

Introduce new tables for maintaining states and events for each category. The data won't be linked to any available data via foreign keys. When api deploy request is received, gateway controller updates the tables and process ends. There is a separate go routine running in each gateway controller which polls on the states table. And if what received from state table for APIs is different compared to what it already knows, then it would do a lookup on Events table based on timestamp.

Key Characteristics

• Event-Driven: Changes are recorded as events and propagated to other instances
• Eventually# Consistent: All instances converge to the same state given time
• Database-Backed: Uses shared database (SQLite/PostgreSQL) as source of truth
• Poll-Based: Instances periodically check for state changes (no pub/sub required)
• Multi-Entity Support: Handles APIs, Certificates, and LLM Provider Templates
• Multi-Tenancy Ready: Organization-based isolation built in

DDL

-- States table: tracks version for API entity type
CREATE TABLE IF NOT EXISTS entity_states (
    entity_type TEXT NOT NULL PRIMARY KEY,  -- 'API'
    version_id TEXT NOT NULL,               -- UUID updated on every change
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

-- API Events table
CREATE TABLE IF NOT EXISTS api_events (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    processed_timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    originated_timestamp TIMESTAMP NOT NULL,
    action TEXT NOT NULL CHECK(action IN ('CREATE', 'UPDATE', 'DELETE')),
    entity_id TEXT NOT NULL,
    event_data TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_api_events_lookup ON api_events(processed_timestamp);

Synchronization Flow

Write Path (Instance A modifies an API)

User/API Request → REST API Handler
                        ↓
                 SyncAwareStorage.UpdateConfig()
                        ↓
            ┌───────────┴───────────┐
            │   BEGIN TRANSACTION   │
            └───────────┬───────────┘
                        ↓
            ┌───────────┴───────────┐
            │ Underlying Storage    │
            │ (save to apis table)  │
            └───────────┬───────────┘
                        ↓
            ┌───────────┴───────────┐
            │ EventStore.RecordEvent│
            │ (save to api_events)  │
            └───────────┬───────────┘
                        ↓
            ┌───────────┴───────────┐
            │ StateManager.UpdateState│
            │ (new UUID in entity_states)│
            └───────────┬───────────┘
                        ↓
            ┌───────────┴───────────┐
            │   COMMIT TRANSACTION  │
            └───────────────────────┘

Read Path (Instance A/B detects and applies change)

SyncPoller (every 5s + jitter)
            ↓
    Poll entity_states
            ↓
    ┌───────┴────────┐
    │ Version Match? │
    └───────┬────────┘
            │
         NO │ (version mismatch detected)
            ↓
    GetEventsSince(lastApplied)
            ↓
    ┌───────┴────────┐
    │ Events Found?  │
    └───────┬────────┘
            │
        YES │
            ↓
    ProcessAPIEvents(events)
            ↓
    ┌───────┴────────────┐
    │ For each event:    │
    │ - Deserialize JSON │
    │ - ConfigStore.Add/ │
    │   Update/Delete    │
    └───────┬────────────┘
            ↓
    SnapshotManager.UpdateSnapshot()
            ↓
    Update knownVersions & lastApplied

Cleanup Path

Hourly Timer
      ↓
CleanupOldEvents(time.Now() - 24h)
      ↓
DELETE FROM api_events WHERE processed_timestamp < cutoff
DELETE FROM certificate_events WHERE processed_timestamp < cutoff
DELETE FROM llm_template_events WHERE processed_timestamp < cutoff

Alternative Approaches

• Broadcast to others using headless services (k8s)/ dnsrr (docker-compose)

Issue here is this becomes heavily dependent on network.

• Bring in a message broker

Then we increase the number of items required for running api-platform.

[VirajSalaka]

Sequence Diagrams:

%%{init: {'theme':'base', 'themeVariables': { 'actorTextColor':'#ff0000', 'noteBkgColor':'rgb(220,255,220)', 'noteTextColor':'#000000'}}}%%
sequenceDiagram
    participant Client
    participant APIServer
    participant DeploymentService
    participant EventHub
    participant EventListener
    participant PolicyManager
    participant XDSManager

    rect rgb(200,220,255)
    Note over Client,XDSManager: Single-replica (enableReplicaSync=false)
    Client->>APIServer: CreateAPI / UpdateAPI
    APIServer->>DeploymentService: UpdateAPIConfiguration
    DeploymentService->>DeploymentService: BuildStoredPolicyFromAPI
    DeploymentService->>PolicyManager: updatePolicyConfiguration
    DeploymentService->>XDSManager: triggerXDSSnapshotUpdate (sync)
    XDSManager-->>DeploymentService: snapshot updated
    DeploymentService-->>APIServer: result
    APIServer-->>Client: 200 OK
    end

    rect rgb(220,255,220)
    Note over Client,EventHub: Multi-replica (enableReplicaSync=true)
    Client->>APIServer: CreateAPI / UpdateAPI
    APIServer->>DeploymentService: UpdateAPIConfiguration
    DeploymentService->>DeploymentService: BuildStoredPolicyFromAPI
    DeploymentService->>EventHub: PublishEvent (API_CREATE/UPDATE/DELETE)
    EventHub-->>DeploymentService: ack/persisted
    DeploymentService-->>APIServer: result
    APIServer-->>Client: 200 OK

    par Async replication processing
        EventHub->>EventListener: deliver event (via subscription)
        EventListener->>EventListener: processAPIEvents -> handleAPICreateOrUpdate / handleAPIDelete
        EventListener->>PolicyManager: apply/remove policy (if configured)
        EventListener->>XDSManager: triggerXDSSnapshotUpdate (async, with timeout)
    end
    end

Loading

%%{init: {'theme':'base', 'themeVariables': { 'actorTextColor':'#ff0000', 'noteBkgColor':'rgb(220,255,220)', 'noteTextColor':'#000000'}}}%%
sequenceDiagram
    participant API as API Handler
    participant DS as APIDeploymentService
    participant EH as EventHub
    participant EL as EventListener
    participant DB as Database

    rect rgb(200, 220, 255)
    note over API,EL: Multi-Replica Mode (enableReplicaSync=true)
    API->>DS: UpdateAPIConfiguration(params)
    DS->>DS: Parse & Validate Config
    DS ->> DB: Save Configuration
    DS->>EH: PublishEvent(API_CREATE/UPDATE)
    EH->>EL: Event (buffered channel)
    DS->>DS: Return Success
    end

Loading

%%{init: {'theme':'base', 'themeVariables': { 'actorTextColor':'#ff0000', 'noteBkgColor':'rgb(220,255,220)', 'noteTextColor':'#000000'}}}%%
sequenceDiagram
    participant DS as APIDeploymentService
    participant EH as EventHub
    participant EL as EventListener
    participant PM as PolicyManager
    participant XDS as XDS Manager
    participant DB as Database

    rect rgb(200, 220, 255)
    note over EL: Async Event Processing
    EL->>DS: handleAPICreateOrUpdate(event)
    DS->>DB: Fetch API from DB
    DS->>DS: Update  in memory DB
    DS->>PM: Build & Apply Policy
    PM->>PM: Update PolicyConfig
    DS->>XDS: Trigger Snapshot Update
    XDS->>XDS: Update Configuration
    end

Loading

[VirajSalaka]

Update: Architecture Document.

Design Proposal: Event-Based Multi-Replica Synchronization for Gateway Controller

---

Section 1: Problem Statement

What is the problem?

In production deployments, the Gateway Controller often runs as multiple replicas for high availability and load distribution. Currently, when an API configuration is created or updated via one replica, other replicas have no mechanism to learn about these changes. This leads to configuration drift where different replicas serve inconsistent xDS snapshots to Envoy proxies, causing unpredictable routing behavior and potential service disruptions.

From the operator's perspective, deploying multiple Gateway Controller replicas should provide seamless high availability without requiring external synchronization tooling or manual intervention.

Why should it be solved now?

1. High-Availability: We can have redundency for gateway controller.

---

Section 2: Solution

2.1 Architecture Overview

The solution introduces an EventHub abstraction with a pluggable backend model that enables publish-subscribe event delivery between Gateway Controller replicas. The default implementation uses SQLite with a polling mechanism, making it suitable for deployments sharing a common database.

┌────────────────────────────────────────────────────────────────────────────┐
│                 Gateway Controller Replica 1 (Request Flow)                │
│                                                                            │
│  ┌──────────────┐    ┌────────────────────┐    ┌─────────────────────────┐ │
│  │  REST API    │───▶│ DeploymentService  │───▶│      EventHub           │ │
│  │  (Port 9090) │    │                    │    │  (Publish to SQLite)    │ │
│  └──────────────┘    └────────────┬───────┘    └───────────┬─────────────┘ │
│                                                                            │
└──────────────────────────────────────────────────┬─────────────────────────┘
                                                   │
                                                   ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                            SQLite Database (Shared)                         │
│  ┌───────────────────────────┐    ┌─────────────────────────────────────┐   │
│  │    organization_states    │    │              events                 │   │
│  │  ─────────────────────    │    │  ────────────────────────────────── │   │
│  │  organization | version   │    │  org_id | event_type | action | ... │   │
│  └───────────────────────────┘    └─────────────────────────────────────┘   │
└──────────────────────────────────┬──────────────────────────────────────────┘
                                   │
                    ┌──────────────┴──────────────┐
                    │                             │
                    ▼                             ▼
┌──────────────────────────────────┐  ┌──────────────────────────────────┐
│  Replica 1 (Async EventListener) │  │  Replica 2 (Async EventListener) │
│  ┌────────────────────────────┐  │  │  ┌────────────────────────────┐  │
│  │  EventListener (polling)   │  │  │  │  EventListener (polling)   │  │
│  │         ▼                  │  │  │  │         ▼                  │  │
│  │  Fetch Config from DB      │  │  │  │  Fetch Config from DB      │  │
│  │         ▼                  │  │  │  │         ▼                  │  │
│  │  Update In-Memory Store    │  │  │  │  Update In-Memory Store    │  │
│  │         ▼                  │  │  │  │         ▼                  │  │
│  │  Trigger xDS Update        │  │  │  │  Trigger xDS Update        │  │
│  └────────────────────────────┘  │  │  └────────────────────────────┘  │
└──────────────────────────────────┘  └──────────────────────────────────┘

3.2 Component Design

EventHub Interface

The EventHub provides a broker-agnostic interface for event publishing and subscription: This is to simulate the broker function.

type EventHub interface {
    Initialize(ctx context.Context) error
    RegisterOrganization(organizationID string) error
    PublishEvent(ctx context.Context, organizationID string, eventType EventType,
        action, entityID, correlationID string, eventData []byte) error
    Subscribe(organizationID string, eventChan chan<- []Event) error
    CleanUpEvents(ctx context.Context, timeFrom, timeEnd time.Time) error
    Close() error
}

Event Types

const (
    EventTypeAPI         EventType = "API"
    EventTypeCertificate EventType = "CERTIFICATE"
    EventTypeLLMTemplate EventType = "LLM_TEMPLATE"
)

Backend Abstraction

The EventhubImpl interface allows pluggable backends:

type EventhubImpl interface {
    Initialize(ctx context.Context) error
    RegisterOrganization(ctx context.Context, orgID string) error
    Publish(ctx context.Context, orgID string, eventType EventType,
        action, entityID, correlationID string, eventData []byte) error
    Subscribe(orgID string, eventChan chan<- []Event) error
    Unsubscribe(orgID string, eventChan chan<- []Event) error
    Cleanup(ctx context.Context, olderThan time.Time) error
    CleanupRange(ctx context.Context, from, to time.Time) error
    Close() error
}

Supported backend types (extensible):

• sqlite (default) - SQLite with polling
• postgres (future) - Postgres with polling

EventSource Abstraction and EventHub Adapter

To decouple the EventListener from the specific EventHub implementation and enable testability, an EventSource interface is introduced:

type EventSource interface {
    Subscribe(ctx context.Context, organizationID string, eventChan chan<- []Event) error
    Unsubscribe(organizationID string) error
    Close() error
}

The EventHubAdapter implements the EventSource interface and acts as a bridge between the EventHub (which uses eventhub.Event types) and the EventListener (which uses generic Event types):

type EventHubAdapter struct {
    eventHub eventhub.EventHub
    logger   *zap.Logger
    activeSubscriptions sync.Map // tracks active subscriptions
}

Key responsibilities of the EventHub Adapter:

1. Organization Registration: Automatically registers organizations with the EventHub when subscriptions are created
2. Type Conversion: Converts eventhub.Event to generic Event types via a bridge goroutine
3. Subscription Management: Tracks active subscriptions and prevents duplicates
4. Event Forwarding: Receives events from EventHub's bridge channel and forwards them to EventListener's channel
5. Lifecycle Management: Handles cleanup of subscriptions and bridge goroutines

This adapter pattern enables:

• Testability: EventListener can be tested with mock event sources (e.g., MockEventSource)
• Flexibility: Future event sources (Kafka, NATS, RabbitMQ) can be added by implementing the EventSource interface
• Decoupling: EventListener remains independent of EventHub implementation details

3.3 Configuration

A new configuration option EnableReplicaSync controls the behavior:

server:
  enableReplicaSync: true  # default: false

When enabled:

• Requires persistent storage (SQLite)
• EventHub is initialized and starts background polling
• EventListener subscribes to organization events
• API changes are published as events rather than triggering synchronous xDS updates

3.4 Database Schema Changes

Two new tables are added (schema version 6):

-- Tracks version per organization for efficient change detection
CREATE TABLE organization_states (
    organization TEXT PRIMARY KEY,
    version_id TEXT NOT NULL DEFAULT '',
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

-- Stores all entity change events
CREATE TABLE events (
    organization_id TEXT NOT NULL,
    processed_timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    originated_timestamp TIMESTAMP NOT NULL,
    event_type TEXT NOT NULL,
    action TEXT NOT NULL CHECK(action IN ('CREATE', 'UPDATE', 'DELETE')),
    entity_id TEXT NOT NULL,
    correlation_id TEXT NOT NULL,
    event_data TEXT NOT NULL,
    PRIMARY KEY (correlation_id)
);

CREATE INDEX idx_events_org_time ON events(organization_id, processed_timestamp);

3.5 Alternatives Considered

Approach	Pros	Cons	Decision
SQLite Polling (Chosen)	No external dependencies, simple deployment, works with existing SQLite database	Higher latency (poll interval), not suitable for very large deployments	Default backend for simplicity
NATS JetStream	Real-time delivery, durable subscriptions, horizontal scaling	Requires external infrastructure	Future consideration if we should rely on broker
Redis Pub/Sub	Low latency, widely used	No durability, messages lost on subscriber disconnect	Not suitable
PostgreSQL LISTEN/NOTIFY	Real-time, database-native	Requires PostgreSQL migration	Future consideration

---

Section 4: Challenges and Constraints

4.1 Security Considerations

• Event Data Privacy: Events contain API configuration metadata but not sensitive credentials. Credentials are stored separately and referenced by ID.
• Correlation ID Tracing: All events include correlation IDs for audit trails and debugging.

4.2 Performance Characteristics

Metric	Value	Notes
Poll Interval	5 seconds (configurable)	Trade-off between latency and database load
Event Retention	1 hour (configurable)	Automatic cleanup via background goroutine
Event Channel Buffer	10 events	Prevents blocking publishers
xDS Update Timeout	10 seconds	Prevents hung updates

4.3 Failure Handling

• Failed Event Delivery: Events remain in database until successfully delivered to all subscribers
• Subscriber Channel Full: Events are logged and retried on next poll cycle
• Database Unavailable: EventHub initialization fails, controller falls back to single-replica mode

4.4 Backward Compatibility

• Default Disabled: enableReplicaSync: false preserves existing single-replica behavior
• Database Migration: Schema version 6 adds tables without modifying existing data
• API Compatibility: No changes to REST API contracts

4.5 Trade-offs

Trade-off	Decision	Rationale
Consistency vs. Latency	Eventual consistency (5s poll interval)	Acceptable for API configuration updates; real-time not required
Simplicity vs. Performance	SQLite polling over message broker	Reduces operational complexity for most deployments
Synchronous vs. Asynchronous	Async xDS updates in multi-replica mode	Faster API response times; xDS updates can tolerate brief delays

---

Appendix: Key Files

Component	File Path
EventHub Interface	pkg/eventhub/types.go
EventHub Implementation	pkg/eventhub/eventhub.go
SQLite Backend	pkg/eventhub/sqlite_backend.go
Backend Interface	pkg/eventhub/backend.go
EventSource Interface	pkg/eventlistener/event_source.go
EventHub Adapter	pkg/eventlistener/eventhub_adapter.go
Mock Event Source	pkg/eventlistener/mock_event_source.go
EventListener	pkg/eventlistener/listener.go
API Event Processor	pkg/eventlistener/api_processor.go
Database Schema	pkg/storage/gateway-controller-db.sql