Multi Channel RAG Assistant

A demo TypeScript/Express backend showcasing a retrieval-augmented generation (RAG) assistant built on Google's Gemini API, exposed over six different communication channels that all share the same underlying chat/voice logic and knowledge base.

Features

• Live voice calls over a WebSocket (/ws/audio) — duplex audio streamed to/from Gemini's Live API, with barge-in support.
• Text chat (POST /chat) — stateful per session, backed by Gemini's chat/tool-calling loop.
• WhatsApp text messaging — the same text-chat behavior, driven by Meta's webhook.
• WhatsApp voice calls — the same live-voice behavior, fronted by a WebRTC peer connection instead of a raw WebSocket.
• Messenger text & voice messaging — the same text-chat behavior, driven by Meta's Messenger Platform webhook; voice notes are downloaded and passed to Gemini as audio, not just text.
• Instagram DM text & voice messaging — the same text-chat behavior, driven by Meta's Messenger Platform webhook for Instagram (sent via graph.instagram.com); voice notes are handled the same way as Messenger's.
• RAG-based lookups — a simple in-process embedding store (no external vector DB) searched via cosine similarity, fed by a CSV of domain data.

How the six channels connect

All six channels are just different transports wrapped around two shared cores, so a change to the underlying assistant logic or knowledge base automatically applies everywhere:

• WS /ws/audio and the WhatsApp call webhook both drive one GeminiLiveBridge instance — the same live-voice session logic, just fed audio from a raw WebSocket in one case and from a WebRTC peer connection in the other.
• POST /chat, the WhatsApp message webhook, the Messenger webhook, and the Instagram webhook all drive one ChatSession instance — the same text chat/tool-calling loop, just addressed by a client-supplied session id, phone number, Messenger PSID, or Instagram IGSID respectively.
• Every channel resolves domain questions the same way: through a shared tool function that queries the in-process RAG store, so all six surfaces answer from the exact same embedded dataset.

Prerequisites

• Node.js 18+ (native fetch is used for all Graph API calls)
• A Gemini API key
• (Optional, for WhatsApp features) a Meta WhatsApp Business API setup: verify token, access token, and phone number ID
• (Optional, for Messenger) a Meta Messenger Platform setup: verify token and page access token
• (Optional, for Instagram) a Meta Instagram API setup: verify token and access token

Setup

npm install
cp .env.example .env   # then fill in GEMINI_API_KEY and, if needed, the WhatsApp vars
npm run ingest          # embeds the CSV dataset into data/embeddings.json
npm run dev             # hot-reload dev server (tsx watch src/server.ts)

npm run ingest isn't strictly required before first boot — the server calls vectorStore.ensureIngested() at startup and will run ingestion automatically if data/embeddings.json doesn't exist yet — but it's the explicit way to re-embed after editing the CSV.

Environment variables

See .env.example for the full list. Key ones:

Variable	Purpose
GEMINI_API_KEY	Google Gemini API key
GEMINI_LIVE_MODEL	Model used for live voice sessions
GEMINI_TEXT_MODEL	Model used for text chat
GEMINI_EMBEDDING_MODEL	Model used to embed CSV rows and queries
RESTAURANT_DATA_CSV / EMBEDDINGS_JSON_PATH	Paths for the RAG source data and its embeddings cache
CORS_ORIGINS	Comma-separated browser origins allowed to call the HTTP API
WHATSAPP_VERIFY_TOKEN / WHATSAPP_ACCESS_TOKEN / WHATSAPP_PHONE_NUMBER_ID	Meta WhatsApp Business API credentials
MESSENGER_VERIFY_TOKEN / MESSENGER_PAGE_ACCESS_TOKEN	Meta Messenger Platform credentials
INSTAGRAM_VERIFY_TOKEN / INSTAGRAM_ACCESS_TOKEN	Meta Instagram API credentials

Commands

npm run dev      # tsx watch src/server.ts — hot-reload dev server
npm run build    # tsc — compiles to dist/
npm start        # node dist/server.js — run the compiled build
npm run ingest   # tsx scripts/ingestData.ts — (re)generate data/embeddings.json from the CSV

There is no test suite or lint script configured in this project.

Project structure

express-server/
├── .env.example              # Template for required environment variables
├── package.json
├── tsconfig.json
├── data/
│   ├── restaurant_services.csv   # Source rows (id, category, title, content) for RAG
│   └── embeddings.json           # Generated cache: CSV rows + their embeddings (gitignored)
├── scripts/
│   └── ingestData.ts         # `npm run ingest` entry point — calls vectorStore.ingest()
└── src/
    ├── server.ts              # Express app, WebSocket server, all route handlers, session maps
    ├── config.ts              # Env-derived constants (models, paths, CORS origins, per-channel tokens)
    ├── geminiClient.ts         # Single shared GoogleGenAI client instance
    ├── prompts.ts              # System instruction strings (live vs. text personas)
    ├── tools.ts                # RAG lookup tool + hand-written Gemini FunctionDeclaration
    ├── vectorStore.ts          # In-process RAG store: CSV ingestion, embeddings cache, cosine search
    ├── chatSession.ts          # ChatSession class — shared by /chat, WhatsApp text, Messenger, Instagram
    ├── geminiLiveBridge.ts     # GeminiLiveBridge class — shared by /ws/audio and WhatsApp calls
    ├── webrtcGeminiBridge.ts   # GeminiCallBridge/GeminiAudioTrack — WhatsApp call WebRTC transport (werift)
    ├── webrtcEcho.ts           # EchoCall — standalone WebRTC transport smoke test (not wired into server.ts)
    ├── whatsapp.ts             # WhatsApp text webhook parsing + sending (Graph API)
    ├── whatsappCalls.ts        # WhatsApp call webhook parsing + call actions (Graph API)
    ├── messenger.ts            # Messenger webhook parsing + sending + audio-attachment download (Graph API, graph.facebook.com)
    └── instagram.ts            # Instagram DM webhook parsing + sending + audio-attachment download (Graph API, graph.instagram.com)