Skip to content

Commit afed110

Browse files
jcscottiiijcscottiii
authored
docs: add architecture documentation (#2277)
* [Feb 2026] Update GEMINI.md with new information Updates the GEMINI.md file to include the latest information about the project, including new features, bug fixes, and improvements. This commit ensures that the documentation is up-to-date and provides accurate information to users and contributors. * docs: add architecture documentation Uses Mermaid to create diagrams that illustrate the architecture of the project. This documentation will help new contributors understand the structure and components of the project more quickly.
1 parent 71c055d commit afed110

File tree

4 files changed

+253
-1
lines changed

4 files changed

+253
-1
lines changed

‎.devcontainer/devcontainer.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,8 @@
7373
"figma.figma-vscode-extension",
7474
"mike-lischke.vscode-antlr4",
7575
"ms-vsliveshare.vsliveshare",
76-
"Google.gemini-cli-vscode-ide-companion"
76+
"Google.gemini-cli-vscode-ide-companion",
77+
"MermaidChart.vscode-mermaid-chart"
7778
]
7879
}
7980
},

‎DEVELOPMENT.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,9 @@ everything pre-installed.
2828
- jsonschema/: JSON schemas for data validation.
2929
- antlr: Description of search grammar
3030
- lib/gen: Output of generated code
31+
- docs/ARCHITECTURE.md: Architecture diagrams
32+
33+
For a visual overview of how these components interact in the cloud and during local development, see the [Architecture Diagrams](./docs/ARCHITECTURE.md).
3134

3235
## Running locally
3336

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ described in a [OpenAPI](./openapi/backend/openapi.yaml) document.
2323
The tool provides a [frontend](./frontend/) dashboard written in Typescript to
2424
display the data.
2525

26+
To understand how these components interact in the cloud and during local development, please refer to the [Architecture Diagrams](./docs/ARCHITECTURE.md).
27+
2628
## Search Syntax
2729

2830
webstatus.dev provides a powerful search feature to help you find the

‎docs/ARCHITECTURE.md‎

Lines changed: 246 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
1+
# Webstatus.dev Architecture & Workflows
2+
3+
This document contains high-level architectural diagrams for the webstatus.dev ecosystem. These diagrams are intended to help new developers understand the system boundaries, data flows, and local development environment.
4+
5+
_Note: The nodes in these diagrams are clickable! Click on a component to jump to its source code or relevant GCP documentation._
6+
7+
---
8+
9+
## 1. High-Level Architecture (Multi-Project Setup)
10+
11+
To limit the scope of the `allUsers` IAM permission (which makes the frontend and API public), the infrastructure is split across three GCP projects. They are networked together via a Shared VPC in the Host project.
12+
13+
```mermaid
14+
flowchart TD
15+
subgraph Host_Project ["Project: Host (VPC)"]
16+
VPC["Shared VPC / Cloud Armor"]
17+
end
18+
19+
subgraph Public_Project ["Project: Public (allUsers Access)"]
20+
FE["Frontend (Cloud Run)"]
21+
BE["Backend API (Cloud Run)"]
22+
end
23+
24+
subgraph Internal_Project ["Project: Internal (Data & Jobs)"]
25+
direction TB
26+
DB[(Cloud Spanner)]
27+
Cache[(Valkey/Memorystore)]
28+
PS[[Pub/Sub]]
29+
Workflows["Ingestion Jobs (Cloud Run Jobs)"]
30+
Workers["Notification Workers"]
31+
GCS[(GCS State Buckets)]
32+
end
33+
34+
%% Interaction Flows
35+
User((User/Browser)) --> FE
36+
User --> BE
37+
FE -.-> VPC
38+
BE -.-> VPC
39+
VPC -.-> DB
40+
VPC -.-> Cache
41+
BE --> DB
42+
BE --> Cache
43+
44+
Workflows --> DB
45+
Workers --> DB
46+
Workers --> PS
47+
Workers --> GCS
48+
49+
%% Clickable Links
50+
click VPC "https://cloud.google.com/vpc/docs/shared-vpc" "GCP Shared VPC Docs"
51+
click FE "https://github.com/GoogleChrome/webstatus.dev/tree/main/frontend/" "Go to Frontend Source"
52+
click BE "https://github.com/GoogleChrome/webstatus.dev/tree/main/backend/" "Go to Backend Source"
53+
click DB "https://cloud.google.com/spanner/docs" "GCP Spanner Docs"
54+
click Cache "https://cloud.google.com/memorystore/docs/valkey" "GCP Memorystore for Valkey"
55+
click PS "https://cloud.google.com/pubsub/docs" "GCP Pub/Sub Docs"
56+
click Workflows "https://github.com/GoogleChrome/webstatus.dev/tree/main/workflows/" "Go to Workflows Source"
57+
click Workers "https://github.com/GoogleChrome/webstatus.dev/tree/main/workers/" "Go to Workers Source"
58+
click GCS "https://cloud.google.com/storage/docs" "GCP Cloud Storage Docs"
59+
```
60+
61+
---
62+
63+
## 2. Public-to-Internal Request Flow
64+
65+
This diagram illustrates how a user's request travels from the browser, hits the public-facing API, and securely queries the internal database using the Spanner Adapter pattern.
66+
67+
```mermaid
68+
sequenceDiagram
69+
participant U as User / Browser
70+
box "Public Project" #f9f9f9
71+
participant FE as Frontend (Lit)
72+
participant BE as Backend (Go API)
73+
end
74+
box "Internal Project" #e1f5fe
75+
participant C as Valkey Cache
76+
participant DB as Spanner DB
77+
end
78+
79+
U->>FE: Load Dashboard
80+
FE->>BE: GET /v1/features
81+
BE->>C: Check Cache
82+
alt Cache Hit
83+
C-->>BE: Return JSON
84+
else Cache Miss
85+
BE->>DB: Query via Spanner Adapter
86+
DB-->>BE: Return Rows
87+
BE->>C: Update Cache
88+
end
89+
BE-->>FE: HTTP 200 JSON
90+
FE-->>U: Render Web Components
91+
```
92+
93+
_(Note: Sequence diagrams in Mermaid currently have limited support for external hyperlinks on participants, so we rely on the flowchart diagrams for deep-linking.)_
94+
95+
---
96+
97+
## 3. Data Ingestion Pipeline (Internal Project)
98+
99+
Webstatus.dev relies heavily on external data. Cloud Scheduler triggers jobs that download, parse, and synchronize this data into our Spanner database.
100+
101+
```mermaid
102+
flowchart LR
103+
subgraph External ["External Sources"]
104+
WPT["WPT.fyi API"]
105+
BCD["MDN BCD Repo"]
106+
WF["Web-Features Repo"]
107+
end
108+
109+
subgraph Internal ["Project: Internal"]
110+
direction TB
111+
Sched["Cloud Scheduler"] --> Job["Ingestion Job (Go)"]
112+
113+
subgraph Job_Logic ["Job Internal Flow"]
114+
DL["pkg/data/downloader.go"] --> P["pkg/data/parser.go"]
115+
P --> Adp["Spanner Adapter"]
116+
end
117+
118+
Adp --> DB[(Cloud Spanner)]
119+
end
120+
121+
WPT --> DL
122+
BCD --> DL
123+
WF --> DL
124+
125+
%% Clickable Links
126+
click WPT "https://wpt.fyi/" "Web Platform Tests API"
127+
click BCD "https://github.com/mdn/browser-compat-data" "MDN BCD Repository"
128+
click WF "https://github.com/web-platform-dx/web-features" "Web-Features Repository"
129+
click Sched "https://cloud.google.com/scheduler/docs" "GCP Cloud Scheduler"
130+
click Job "https://github.com/GoogleChrome/webstatus.dev/tree/main/workflows/steps/services/" "Go to Workflow Services Source"
131+
click Adp "https://github.com/GoogleChrome/webstatus.dev/tree/main/lib/gcpspanner/spanneradapters/" "Go to Spanner Adapters"
132+
click DB "https://cloud.google.com/spanner" "GCP Spanner"
133+
```
134+
135+
---
136+
137+
## 4. Notification System Architecture (Event-Driven)
138+
139+
When data changes (via ingestion workflows) or users update their saved searches, an event-driven architecture processes those changes to deliver email and push notifications.
140+
141+
```mermaid
142+
flowchart TD
143+
subgraph Internal ["Project: Internal"]
144+
direction TB
145+
DB[(Spanner: Subscriptions & Channels)]
146+
EP["Event Producer Worker"] --> PS[[Pub/Sub: feature-diffs]]
147+
PS --> PDW["Push Delivery Worker"]
148+
PS --> EW["Email Worker"]
149+
150+
PDW --> DB
151+
EW --> Temp["lib/email (Templates)"]
152+
EW --> Chime["Chime (email service)"]
153+
154+
EP --> GCS[(GCS: Snapshot State)]
155+
end
156+
157+
DB -- Trigger Change --> EP
158+
159+
%% Clickable Links
160+
click DB "https://github.com/GoogleChrome/webstatus.dev/tree/main/lib/gcpspanner/" "Go to Spanner Schema/Mappers"
161+
click EP "https://github.com/GoogleChrome/webstatus.dev/tree/main/workers/event_producer/" "Go to Event Producer Source"
162+
click PS "https://cloud.google.com/pubsub" "GCP Pub/Sub"
163+
click PDW "https://github.com/GoogleChrome/webstatus.dev/tree/main/workers/push_delivery/" "Go to Push Delivery Worker Source"
164+
click EW "https://github.com/GoogleChrome/webstatus.dev/tree/main/workers/email/" "Go to Email Worker Source"
165+
click Temp "https://github.com/GoogleChrome/webstatus.dev/tree/main/lib/email/" "Go to HTML Email Templates"
166+
click GCS "https://cloud.google.com/storage" "GCS Blob Storage"
167+
```
168+
169+
---
170+
171+
## 5. Local Development & E2E Testing Environment
172+
173+
Understanding the local dev loop is crucial. We use Skaffold and Minikube to orchestrate live services alongside GCP emulators. The `Makefile` provides targets to populate these emulators with either _fake data_ (for deterministic E2E testing) or _live data_ (for manual workflow testing).
174+
175+
```mermaid
176+
flowchart TD
177+
subgraph Dev_Machine ["Developer Machine / CI Pipeline"]
178+
direction LR
179+
Make["Makefile"]
180+
Playwright{"Playwright (E2E Tests)"}
181+
end
182+
183+
subgraph Minikube ["Minikube Local Cluster"]
184+
direction TB
185+
186+
subgraph Live_Services ["Skaffold Live Services"]
187+
FE["Frontend (Lit/Nginx)"]
188+
BE["Backend API (Go)"]
189+
end
190+
191+
subgraph Emulators ["GCP Emulators & Mocks"]
192+
Spanner[(Spanner Emulator)]
193+
DS[(Datastore Emulator)]
194+
Auth["Firebase Auth Emulator"]
195+
WM["Wiremock
196+
(Mock GitHub API)"]
197+
Valkey[(Valkey Cache)]
198+
PS[[Pub/Sub Emulator]]
199+
end
200+
201+
FE <-->|API Calls| BE
202+
FE <-->|Firebase Login| Auth
203+
BE <-->|Read/Write| Spanner
204+
BE <-->|Cache| Valkey
205+
BE <-->|Fetch Profile/Emails
206+
on Login| WM
207+
end
208+
209+
subgraph Data_Population ["Data Population Strategies"]
210+
direction TB
211+
FakeUsers["make dev_fake_users
212+
(util/cmd/load_test_users)"]
213+
FakeData["make dev_fake_data
214+
(util/cmd/load_fake_data)"]
215+
RealFlows["make dev_workflows
216+
(util/run_job.sh)"]
217+
end
218+
219+
%% Makefile connections
220+
Make -.->|Triggers| FakeUsers
221+
Make -.->|Triggers| FakeData
222+
Make -.->|Triggers| RealFlows
223+
224+
%% Population flows
225+
FakeUsers -->|Seeds Test Users| Auth
226+
FakeData -->|Seeds Predictable Entities| Spanner
227+
FakeData -->|Seeds Predictable Entities| DS
228+
RealFlows -->|Ingests Live Data| Spanner
229+
230+
%% Testing flows
231+
Playwright -.->|Requires| FakeData
232+
Playwright -.->|Requires| FakeUsers
233+
Playwright -->|Runs Tests Against| FE
234+
235+
%% Clickable Links
236+
click Make "https://github.com/GoogleChrome/webstatus.dev/tree/main/Makefile" "Go to Makefile"
237+
click Playwright "https://github.com/GoogleChrome/webstatus.dev/tree/main/e2e/tests/" "Go to E2E Playwright Tests"
238+
click FE "https://github.com/GoogleChrome/webstatus.dev/tree/main/frontend/src/" "Go to Frontend Source"
239+
click BE "https://github.com/GoogleChrome/webstatus.dev/tree/main/backend/pkg/httpserver/" "Go to Backend Handlers"
240+
click Spanner "https://github.com/GoogleChrome/webstatus.dev/tree/main/.dev/spanner/" "Spanner Emulator Setup"
241+
click WM "https://github.com/GoogleChrome/webstatus.dev/tree/main/.dev/wiremock/" "Wiremock Configuration"
242+
click Auth "https://github.com/GoogleChrome/webstatus.dev/tree/main/.dev/auth/" "Firebase Auth Emulator Setup"
243+
click FakeUsers "https://github.com/GoogleChrome/webstatus.dev/tree/main/util/cmd/load_test_users/" "Go to load_test_users Source"
244+
click FakeData "https://github.com/GoogleChrome/webstatus.dev/tree/main/util/cmd/load_fake_data/" "Go to load_fake_data Source"
245+
click RealFlows "https://github.com/GoogleChrome/webstatus.dev/tree/main/util/run_job.sh" "Go to run_job.sh"
246+
```

0 commit comments

Comments
 (0)