Skip to content

Commit 079f3f0

Browse files
committed
chore: Update documentation and add configuration files
- Enhanced CHANGELOG with recent additions, including datamart enhancements and improved export functionality. - Added .gitignore and trunk.yaml files to manage project configurations and ignore unnecessary files. - Updated various documentation files for improved readability and consistency, including architecture diagrams and guides. - Introduced new configuration files for shellcheck and yamllint to standardize code quality checks across the project.
1 parent 8e68d72 commit 079f3f0

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

51 files changed

+301
-168
lines changed

.trunk/.gitignore

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
*out
2+
*logs
3+
*actions
4+
*notifications
5+
*tools
6+
plugins
7+
user_trunk.yaml
8+
user.yaml
9+
tmp

.trunk/configs/.shellcheckrc

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
enable=all
2+
source-path=SCRIPTDIR
3+
disable=SC2154
4+
5+
# If you're having issues with shellcheck following source, disable the errors via:
6+
# disable=SC1090
7+
# disable=SC1091

.trunk/configs/.yamllint.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
rules:
2+
quoted-strings:
3+
required: only-when-needed
4+
extra-allowed: ["{|}"]
5+
key-duplicates: {}
6+
octal-values:
7+
forbid-implicit-octal: true

.trunk/trunk.yaml

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# This file controls the behavior of Trunk: https://docs.trunk.io/cli
2+
# To learn more about the format of this file, see https://docs.trunk.io/reference/trunk-yaml
3+
version: 0.1
4+
cli:
5+
version: 1.25.0
6+
# Trunk provides extensibility via plugins. (https://docs.trunk.io/plugins)
7+
plugins:
8+
sources:
9+
- id: trunk
10+
ref: v1.7.4
11+
uri: https://github.com/trunk-io/plugins
12+
# Many linters and tools depend on runtimes - configure them here. (https://docs.trunk.io/runtimes)
13+
runtimes:
14+
enabled:
15+
- go@1.21.0
16+
- node@22.16.0
17+
- python@3.10.8
18+
# This is the section where you manage your linters. (https://docs.trunk.io/check/configuration)
19+
lint:
20+
enabled:
21+
- actionlint@1.7.10
22+
- checkov@3.2.499
23+
- git-diff-check
24+
- markdownlint@0.47.0
25+
- prettier@3.8.1
26+
- shellcheck@0.11.0
27+
- shfmt@3.6.0
28+
- sqlfluff@4.0.0
29+
- taplo@0.10.0
30+
- trufflehog@3.92.5
31+
- yamllint@1.38.0
32+
actions:
33+
disabled:
34+
- trunk-announce
35+
- trunk-check-pre-push
36+
- trunk-fmt-pre-commit
37+
enabled:
38+
- trunk-upgrade-available

CHANGELOG.md

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -116,13 +116,17 @@ All notable changes to this project will be documented in this file.
116116
### Added
117117

118118
- **CI/CD Testing Infrastructure**: Added local CI testing scripts and improved test workflows
119-
- **Datamart Enhancements**: Added new tracking columns and metrics to datamartUsers and datamartCountries
120-
- **Enhanced Export Functionality**: Improved CSV and JSON export processes with incremental updates and conflict resolution
119+
- **Datamart Enhancements**: Added new tracking columns and metrics to datamartUsers and
120+
datamartCountries
121+
- **Enhanced Export Functionality**: Improved CSV and JSON export processes with incremental updates
122+
and conflict resolution
121123

122124
### Changed
123125

124-
- **Documentation**: Comprehensive documentation updates including standardized links, metadata sections, and Mermaid diagrams
125-
- **ETL Process**: Enhanced ETL execution with improved phase handling and closed dimension date updates
126+
- **Documentation**: Comprehensive documentation updates including standardized links, metadata
127+
sections, and Mermaid diagrams
128+
- **ETL Process**: Enhanced ETL execution with improved phase handling and closed dimension date
129+
updates
126130
- **Database**: Updated SQL scripts to use public schema and improved query optimization
127131
- **Performance**: Optimized SQL queries, export processes, and added locking mechanisms
128132
- **Code Quality**: Standardized code formatting, improved error handling, and enhanced logging

LICENSE

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,33 @@
55
Everyone is permitted to copy and distribute verbatim copies
66
of this license document, but changing it is not allowed.
77

8+
-------------------------------------------------------------------------------
9+
10+
Copyright (C) 2025 Andres Gomez Casanova (AngocA), miembros de OSM Notes Latam
11+
12+
This program is free software: you can redistribute it and/or modify
13+
it under the terms of the GNU General Public License as published by
14+
the Free Software Foundation, either version 3 of the License, or
15+
(at your option) any later version.
16+
17+
This program is distributed in the hope that it will be useful,
18+
but WITHOUT ANY WARRANTY; without even the implied warranty of
19+
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
20+
GNU General Public License for more details.
21+
22+
You should have received a copy of the GNU General Public License
23+
along with this program. If not, see <https://www.gnu.org/licenses/>.
24+
25+
-------------------------------------------------------------------------------
26+
27+
Documentation License
28+
29+
All documentation files (including but not limited to `.md` files in the
30+
`docs/` directory, README files, and other documentation) are licensed
31+
under the GNU General Public License v3, the same license as the code.
32+
33+
-------------------------------------------------------------------------------
34+
835
Preamble
936

1037
The GNU General Public License is a free, copyleft license for

README.md

Lines changed: 23 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,9 @@ For complete navigation by role, see
114114

115115
## Architecture Decision Records
116116

117-
Architecture Decision Records (ADRs) document important architectural decisions made in this project:
117+
Architecture Decision Records (ADRs) document important architectural decisions made in this
118+
project:
119+
118120
- [View all ADRs](docs/adr/)
119121
- [Central ADR Index](https://github.com/OSM-Notes/OSM-Notes-Common/blob/main/docs/adr/README.md)
120122

@@ -145,7 +147,8 @@ Architecture Decision Records (ADRs) document important architectural decisions
145147

146148
## Requirements
147149

148-
For complete installation instructions, see **[Installation and Dependencies Guide](docs/Installation_Dependencies.md)**.
150+
For complete installation instructions, see
151+
**[Installation and Dependencies Guide](docs/Installation_Dependencies.md)**.
149152

150153
### Application Requirements
151154

@@ -1214,10 +1217,14 @@ For more information about OSM licensing, see:
12141217
For shared documentation of the complete ecosystem, see:
12151218

12161219
- **[OSM Notes Ecosystem](https://github.com/OSM-Notes/OSM-Notes)** - Ecosystem landing page
1217-
- **[Global Glossary](https://github.com/OSM-Notes/OSM-Notes-Common/blob/main/docs/Glossary.md)** - Terms and definitions
1218-
- **[Complete Installation Guide](https://github.com/OSM-Notes/OSM-Notes-Common/blob/main/docs/Installation.md)** - Step-by-step installation of all projects
1219-
- **[End-to-End Data Flow](https://github.com/OSM-Notes/OSM-Notes-Common/blob/main/docs/Data_Flow.md)** - Complete data flow
1220-
- **[Decision Guide](https://github.com/OSM-Notes/OSM-Notes-Common/blob/main/docs/Decision_Guide.md)** - Which project do I need?
1220+
- **[Global Glossary](https://github.com/OSM-Notes/OSM-Notes-Common/blob/main/docs/Glossary.md)** -
1221+
Terms and definitions
1222+
- **[Complete Installation Guide](https://github.com/OSM-Notes/OSM-Notes-Common/blob/main/docs/Installation.md)** -
1223+
Step-by-step installation of all projects
1224+
- **[End-to-End Data Flow](https://github.com/OSM-Notes/OSM-Notes-Common/blob/main/docs/Data_Flow.md)** -
1225+
Complete data flow
1226+
- **[Decision Guide](https://github.com/OSM-Notes/OSM-Notes-Common/blob/main/docs/Decision_Guide.md)** -
1227+
Which project do I need?
12211228

12221229
---
12231230

@@ -1241,7 +1248,8 @@ OSM-Notes/
12411248
└── OSM-Notes-Data/ # JSON data files (GitHub Pages)
12421249
```
12431250

1244-
**All 8 projects are independent repositories**, working together to provide a complete OSM Notes analysis solution.
1251+
**All 8 projects are independent repositories**, working together to provide a complete OSM Notes
1252+
analysis solution.
12451253

12461254
### How Projects Work Together
12471255

@@ -1250,27 +1258,27 @@ graph TB
12501258
subgraph External["External Sources"]
12511259
OSM[OSM Planet/API]
12521260
end
1253-
1261+
12541262
subgraph Base["Base Project"]
12551263
INGESTION[OSM-Notes-Ingestion<br/>Base project]
12561264
end
1257-
1265+
12581266
subgraph Processing["Processing Layer"]
12591267
ANALYTICS[OSM-Notes-Analytics<br/>ETL → Data Warehouse<br/>this repo]
12601268
WMS[OSM-Notes-WMS<br/>WMS layers]
12611269
end
1262-
1270+
12631271
subgraph Delivery["Delivery Layer"]
12641272
DATA[OSM-Notes-Data<br/>JSON files<br/>GitHub Pages]
12651273
API[OSM-Notes-API<br/>REST API<br/>reads from Analytics DWH]
12661274
VIEWER[OSM-Notes-Viewer<br/>Consumes JSON from Data]
12671275
end
1268-
1276+
12691277
subgraph Support["Support Layer"]
12701278
MONITORING[OSM-Notes-Monitoring<br/>Monitors all projects]
12711279
COMMON[OSM-Notes-Common<br/>Shared libraries<br/>submodule]
12721280
end
1273-
1281+
12741282
OSM -->|Downloads| INGESTION
12751283
INGESTION -->|Base Tables| ANALYTICS
12761284
INGESTION -->|Same Database| WMS
@@ -1284,7 +1292,7 @@ graph TB
12841292
COMMON -.->|Used by| ANALYTICS
12851293
COMMON -.->|Used by| WMS
12861294
COMMON -.->|Used by| MONITORING
1287-
1295+
12881296
style OSM fill:#ADD8E6
12891297
style INGESTION fill:#90EE90
12901298
style ANALYTICS fill:#FFFFE0
@@ -1355,7 +1363,8 @@ When setting up the complete ecosystem, install projects in this order:
13551363
8. **[OSM-Notes-Data](https://github.com/OSM-Notes/OSM-Notes-Data)**
13561364
- JSON data files exported from Analytics
13571365
- Served via GitHub Pages
1358-
- **Requires**: OSM-Notes-Analytics (generates and publishes the data via `exportAndPushJSONToGitHub.sh`)
1366+
- **Requires**: OSM-Notes-Analytics (generates and publishes the data via
1367+
`exportAndPushJSONToGitHub.sh`)
13591368
- **Consumed by**: Viewer (primary consumer), API (optional)
13601369

13611370
### External Resources

docs/Architecture_Diagram.md

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@ project: "OSM-Notes-Analytics"
1212
status: "active"
1313
---
1414

15-
1615
# Architecture Diagram
1716

1817
This document provides architecture diagrams for the OSM-Notes-Analytics system, following the C4
@@ -94,7 +93,8 @@ graph TB
9493
- **OSM-Notes-Viewer**: Downstream system consuming JSON exports from Data
9594
- **OSM-Notes-WMS**: Uses same database as Ingestion (geographic visualization)
9695
- **OSM-Notes-Monitoring**: Monitors all ecosystem components
97-
- **OSM-Notes-Common**: Shared library used by multiple systems (Ingestion, Analytics, WMS, Monitoring)
96+
- **OSM-Notes-Common**: Shared library used by multiple systems (Ingestion, Analytics, WMS,
97+
Monitoring)
9898

9999
**User Types:**
100100

@@ -406,23 +406,23 @@ sequenceDiagram
406406
```mermaid
407407
graph TD
408408
DB[PostgreSQL Database: osm_notes]
409-
409+
410410
subgraph Public["Schema: public<br/>managed by OSM-Notes-Ingestion"]
411411
NOTES[notes]
412412
COMMENTS[note_comments]
413413
COMMENTS_TEXT[note_comments_text]
414414
USERS[users]
415415
COUNTRIES[countries]
416416
end
417-
417+
418418
subgraph DWH["Schema: dwh<br/>managed by OSM-Notes-Analytics"]
419419
subgraph Facts["facts partitioned by year"]
420420
F2013[facts_2013]
421421
F2014[facts_2014]
422422
FDOTS[...]
423423
F2025[facts_2025]
424424
end
425-
425+
426426
DIM_USERS[dimension_users]
427427
DIM_COUNTRIES[dimension_countries]
428428
DIM_DAYS[dimension_days]
@@ -434,34 +434,34 @@ graph TD
434434
DIM_SEASONS[dimension_seasons]
435435
DIM_AUTO[dimension_automation_level]
436436
DIM_EXP[dimension_experience_levels]
437-
437+
438438
DM_USERS[datamartusers]
439439
DM_COUNTRIES[datamartcountries]
440440
DM_GLOBAL[datamartglobal]
441441
end
442-
442+
443443
subgraph Staging["Schema: staging<br/>temporary, managed by ETL"]
444444
STAGE_2013[facts_2013 per-year staging]
445445
STAGE_2014[facts_2014 per-year staging]
446446
STAGE_DOTS[...]
447447
end
448-
448+
449449
DB --> Public
450450
DB --> DWH
451451
DB --> Staging
452-
452+
453453
Public --> NOTES
454454
Public --> COMMENTS
455455
Public --> COMMENTS_TEXT
456456
Public --> USERS
457457
Public --> COUNTRIES
458-
458+
459459
DWH --> Facts
460460
Facts --> F2013
461461
Facts --> F2014
462462
Facts --> FDOTS
463463
Facts --> F2025
464-
464+
465465
DWH --> DIM_USERS
466466
DWH --> DIM_COUNTRIES
467467
DWH --> DIM_DAYS
@@ -476,11 +476,11 @@ graph TD
476476
DWH --> DM_USERS
477477
DWH --> DM_COUNTRIES
478478
DWH --> DM_GLOBAL
479-
479+
480480
Staging --> STAGE_2013
481481
Staging --> STAGE_2014
482482
Staging --> STAGE_DOTS
483-
483+
484484
style DB fill:#90EE90
485485
style Public fill:#E0F6FF
486486
style DWH fill:#FFFFE0

docs/Atomic_Validation_Export.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
---
22
title: "Atomic Validation Strategy for JSON Export"
3-
description: "Implementation guide for atomic validation in the JSON export process, ensuring data integrity during file generation"
3+
description:
4+
"Implementation guide for atomic validation in the JSON export process, ensuring data integrity
5+
during file generation"
46
version: "1.0.0"
57
last_updated: "2026-01-25"
68
author: "AngocA"
@@ -13,7 +15,6 @@ project: "OSM-Notes-Analytics"
1315
status: "active"
1416
---
1517

16-
1718
# Atomic Validation Strategy for JSON Export
1819

1920
## Overview

docs/Business_Glossary.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@ project: "OSM-Notes-Analytics"
1212
status: "active"
1313
---
1414

15-
1615
# Business Glossary
1716

1817
This document provides business definitions for terms, concepts, and metrics used in the

0 commit comments

Comments
 (0)