Skip to content

Commit 95dbb4b

Browse files
committed
refactor: Update SQL script versions and improve documentation
- Renamed and updated SQL scripts for datamartCountries and datamartUsers to reflect correct versioning. - Adjusted references in various documentation files to align with the new script names and versions. - Enhanced clarity in the README and other documentation regarding the structure and purpose of SQL scripts. - Ensured consistency across the project by updating related scripts and comments to match the new naming conventions.
1 parent 5c82aa9 commit 95dbb4b

File tree

78 files changed

+466
-352
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

78 files changed

+466
-352
lines changed

CHANGELOG.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -407,10 +407,10 @@ All notable changes to this project will be documented in this file.
407407

408408
### Files Modified
409409

410-
- `sql/dwh/datamartCountries/datamartCountries_12_createDatamarCountriesTable.sql`
411-
- `sql/dwh/datamartCountries/datamartCountries_13_createProcedure.sql`
412-
- `sql/dwh/datamartUsers/datamartUsers_12_createDatamartUsersTable.sql`
413-
- `sql/dwh/datamartUsers/datamartUsers_13_createProcedure.sql`
410+
- `sql/dwh/datamartCountries/datamartCountries_11_createDatamarCountriesTable.sql`
411+
- `sql/dwh/datamartCountries/datamartCountries_12_createProcedure.sql`
412+
- `sql/dwh/datamartUsers/datamartUsers_11_createDatamartUsersTable.sql`
413+
- `sql/dwh/datamartUsers/datamartUsers_12_createProcedure.sql`
414414
- `tests/unit/bash/datamart_high_priority_metrics.test.bats` (new)
415415
- `docs/Dashboard_Analysis.md`
416416
- `docs/Metric_Definitions.md`

CONTRIBUTING.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -310,10 +310,10 @@ declare -r ERROR_INVALID_ARGUMENT=242
310310

311311
#### File Naming Convention
312312

313-
- **Process files**: `processAPINotes_21_createApiTables.sql`
313+
- **Process files**: `processAPINotes_20_createApiTables.sql`
314314
- **ETL files**: `ETL_11_checkDWHTables.sql`
315-
- **Function files**: `functionsProcess_21_createFunctionToGetCountry.sql`
316-
- **Drop files**: `processAPINotes_12_dropApiTables.sql`
315+
- **Function files**: `functionsProcess_20_createFunctionToGetCountry.sql`
316+
- **Drop files**: `processAPINotes_10_dropApiTables.sql`
317317

318318
#### SQL Code Standards
319319

@@ -598,9 +598,9 @@ OSM-Notes-Analytics/
598598

599599
Follow the naming pattern: `<Component>_<Phase><Step>_<Description>.sql`
600600

601-
- **ETL scripts**: `ETL_22_createDWHTables.sql`, `ETL_25_populateDimensionTables.sql`
602-
- **Staging scripts**: `Staging_31_createBaseStagingObjects.sql`, `Staging_61_loadNotes.sql`
603-
- **Datamart scripts**: `datamartCountries_31_populateDatamartCountriesTable.sql`
601+
- **ETL scripts**: `ETL_20_createDWHTables.sql`, `ETL_25_populateDimensionTables.sql`
602+
- **Staging scripts**: `Staging_31_createBaseStagingObjects.sql`, `Staging_60_loadNotes.sql`
603+
- **Datamart scripts**: `datamartCountries_30_populateDatamartCountriesTable.sql`
604604
- **Phase numbers**: 1x=validation, 2x=creation, 3x=population, 4x=constraints, 5x=finalization,
605605
6x=incremental
606606

ToDo/Progress_Tracker.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@
9595
- Problem: Procedure used `opened_dimension_id_country` which doesn't exist in dwh.facts table
9696
- Solution: Changed to use `dimension_id_country` with `action_comment = 'opened'` filter
9797
- Files modified:
98-
- sql/dwh/datamartCountries/datamartCountries_13_createProcedure.sql (lines 993, 1003, 1015, 1027)
98+
- sql/dwh/datamartCountries/datamartCountries_12_createProcedure.sql (lines 993, 1003, 1015, 1027)
9999
- Affected calculations: applications_used, most_used_application_id, mobile_apps_count,
100100
desktop_apps_count
101101
- Created verification script: sql/dwh/improvements/verify_datamart_calculations.sql

bin/dwh/ETL.sh

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -184,15 +184,15 @@ declare -r POSTGRES_12_DROP_DATAMART_OBJECTS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/E
184184
declare -r POSTGRES_13_DROP_DWH_OBJECTS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_13_removeDWHObjects.sql"
185185

186186
# Create DWH tables.
187-
declare -r POSTGRES_22_CREATE_DWH_TABLES="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_22_createDWHTables.sql"
187+
declare -r POSTGRES_22_CREATE_DWH_TABLES="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_20_createDWHTables.sql"
188188
# Create fact partitions.
189-
declare -r POSTGRES_22A_CREATE_FACT_PARTITIONS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_22a_createFactPartitions.sql"
189+
declare -r POSTGRES_22A_CREATE_FACT_PARTITIONS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_21_createFactPartitions.sql"
190190
# Populates regions per country.
191-
declare -r POSTGRES_23_GET_WORLD_REGIONS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_23_getWorldRegion.sql"
191+
declare -r POSTGRES_23_GET_WORLD_REGIONS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_22_getWorldRegion.sql"
192192
# Add functions.
193-
declare -r POSTGRES_24_ADD_FUNCTIONS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_24_addFunctions.sql"
193+
declare -r POSTGRES_24_ADD_FUNCTIONS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_23_addFunctions.sql"
194194
# Populate ISO country codes.
195-
declare -r POSTGRES_24A_POPULATE_ISO_CODES="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_24a_populateISOCodes.sql"
195+
declare -r POSTGRES_24A_POPULATE_ISO_CODES="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_24_populateISOCodes.sql"
196196
# Populate dimension tables.
197197
declare -r POSTGRES_25_POPULATE_DIMENSIONS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_25_populateDimensionTables.sql"
198198
# Update dimension tables.
@@ -207,19 +207,19 @@ declare -r POSTGRES_31_CREATE_BASE_STAGING_OBJECTS="${SCRIPT_BASE_DIRECTORY}/sql
207207
declare -r POSTGRES_32_CREATE_STAGING_OBJECTS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_32_createStagingObjects.sql"
208208
# Create initial facts base objects.
209209
declare -r POSTGRES_33_CREATE_FACTS_BASE_OBJECTS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_33_initialFactsBaseObjects.sql"
210-
declare -r POSTGRES_33_CREATE_FACTS_BASE_OBJECTS_SIMPLE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_33_initialFactsBaseObjects_Simple.sql"
210+
declare -r POSTGRES_33_CREATE_FACTS_BASE_OBJECTS_SIMPLE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_33a_initialFactsBaseObjects_Simple.sql"
211211
# Create initial facts load.
212212
declare -r POSTGRES_34_CREATE_FACTS_YEAR_LOAD="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_34_initialFactsLoadCreate.sql"
213213
# Execute initial facts load.
214214
declare -r POSTGRES_35_EXECUTE_FACTS_YEAR_LOAD="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_35_initialFactsLoadExecute.sql"
215-
declare -r POSTGRES_35_EXECUTE_FACTS_YEAR_LOAD_SIMPLE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_35_initialFactsLoadExecute_Simple.sql"
216-
declare -r POSTGRES_35_EXECUTE_FACTS_YEAR_LOAD_PHASE2="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_35_initialFactsLoadExecute_Phase2.sql"
217-
declare -r POSTGRES_35B_UPDATE_CLOSED_DIMENSION_ID_DATE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_35b_updateClosedDimensionIdDate.sql"
218-
declare -r POSTGRES_34_INITIAL_FACTS_LOAD_PARALLEL="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_34_initialFactsLoadCreate_Parallel.sql"
215+
declare -r POSTGRES_35_EXECUTE_FACTS_YEAR_LOAD_SIMPLE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_35a_initialFactsLoadExecute_Simple.sql"
216+
declare -r POSTGRES_35_EXECUTE_FACTS_YEAR_LOAD_PHASE2="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_35b_initialFactsLoadExecute_Phase2.sql"
217+
declare -r POSTGRES_35B_UPDATE_CLOSED_DIMENSION_ID_DATE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_35c_updateClosedDimensionIdDate.sql"
218+
declare -r POSTGRES_34_INITIAL_FACTS_LOAD_PARALLEL="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_34a_initialFactsLoadCreate_Parallel.sql"
219219
# Drop initial facts load.
220220
declare -r POSTGRES_36_DROP_FACTS_YEAR_LOAD="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_36_initialFactsLoadDrop.sql"
221221
# Add constraints, indexes and triggers.
222-
declare -r POSTGRES_41_ADD_CONSTRAINTS_INDEXES_TRIGGERS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_41_addConstraintsIndexesTriggers.sql"
222+
declare -r POSTGRES_41_ADD_CONSTRAINTS_INDEXES_TRIGGERS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_40_addConstraintsIndexesTriggers.sql"
223223
# Create automation detection system.
224224
declare -r POSTGRES_50_CREATE_AUTOMATION_DETECTION="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_50_createAutomationDetection.sql"
225225
# Create experience levels system.
@@ -233,7 +233,7 @@ declare -r POSTGRES_53A_ENHANCE_DATAMARTS_HASHTAGS="${SCRIPT_BASE_DIRECTORY}/sql
233233
# Create specialized hashtag indexes.
234234
declare -r POSTGRES_53B_CREATE_HASHTAG_INDEXES="${SCRIPT_BASE_DIRECTORY}/sql/dwh/improvements/13_create_hashtag_indexes.sql"
235235
# Unify facts.
236-
declare -r POSTGRES_54_UNIFY_FACTS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_51_unify.sql"
236+
declare -r POSTGRES_54_UNIFY_FACTS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_50_unify.sql"
237237
# Create note current status table and procedures (ETL-003, ETL-004)
238238
declare -r POSTGRES_55_CREATE_NOTE_CURRENT_STATUS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_55_createNoteCurrentStatus.sql"
239239
declare -r POSTGRES_56_GENERATE_ETL_REPORT="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_56_generateETLReport.sql"
@@ -243,7 +243,7 @@ declare -r POSTGRES_57_VALIDATE_ETL_INTEGRITY="${SCRIPT_BASE_DIRECTORY}/sql/dwh/
243243
declare -r POSTGRES_63_COMPLETE_HASHTAG_ANALYSIS="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamarts/63_completeHashtagAnalysis.sql"
244244

245245
# Load notes staging.
246-
declare -r POSTGRES_61_LOAD_NOTES_STAGING="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_61_loadNotes.sql"
246+
declare -r POSTGRES_61_LOAD_NOTES_STAGING="${SCRIPT_BASE_DIRECTORY}/sql/dwh/Staging_60_loadNotes.sql"
247247

248248
# Setup Foreign Data Wrappers for incremental processing.
249249
declare -r POSTGRES_60_SETUP_FDW="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_60_setupFDW.sql"
@@ -257,7 +257,7 @@ declare -r DATAMART_COUNTRIES_SCRIPT="${SCRIPT_BASE_DIRECTORY}/bin/dwh/datamartC
257257
declare -r DATAMART_USERS_SCRIPT="${SCRIPT_BASE_DIRECTORY}/bin/dwh/datamartUsers/datamartUsers.sh"
258258
declare -r DATAMART_GLOBAL_SCRIPT="${SCRIPT_BASE_DIRECTORY}/bin/dwh/datamartGlobal/datamartGlobal.sh"
259259
# Create datamart performance log table.
260-
declare -r POSTGRES_DATAMART_PERFORMANCE_CREATE_TABLE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartPerformance/datamartPerformance_11_createTable.sql"
260+
declare -r POSTGRES_DATAMART_PERFORMANCE_CREATE_TABLE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartPerformance/datamartPerformance_10_createTable.sql"
261261
# Create logs table for debugging (only needed when Ingestion and Analytics databases are different).
262262
declare -r POSTGRES_CREATE_LOGS_TABLE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/ETL_54_createLogsTable.sql"
263263

bin/dwh/Entry_Points.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -238,7 +238,7 @@ error handling and logging.
238238

239239
```bash
240240
# SQL scripts - should not be executed directly
241-
psql -d osm_notes -f sql/dwh/ETL_22_createDWHTables.sql # WRONG (use ETL.sh instead)
241+
psql -d osm_notes -f sql/dwh/ETL_20_createDWHTables.sql # WRONG (use ETL.sh instead)
242242
```
243243

244244
## Workflow Examples

bin/dwh/README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -55,15 +55,15 @@ flowchart TD
5555

5656
Key scripts:
5757

58-
- `sql/dwh/ETL_22_createDWHTables.sql`
59-
- `sql/dwh/ETL_41_addConstraintsIndexesTriggers.sql`
58+
- `sql/dwh/ETL_20_createDWHTables.sql`
59+
- `sql/dwh/ETL_40_addConstraintsIndexesTriggers.sql`
6060
- `sql/dwh/ETL_25_populateDimensionTables.sql`
6161
- `sql/dwh/ETL_26_updateDimensionTables.sql`
6262
- `sql/dwh/Staging_31_createBaseStagingObjects.sql`
6363
- `sql/dwh/Staging_32_createStagingObjects.sql`
6464
- `sql/dwh/Staging_34_initialFactsLoadCreate.sql`
6565
- `sql/dwh/Staging_35_initialFactsLoadExecute.sql`
66-
- `sql/dwh/Staging_51_unify.sql`
66+
- `sql/dwh/Staging_50_unify.sql`
6767

6868
### Schema Overview
6969

bin/dwh/datamartCountries/Parallel_Processing.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -491,6 +491,6 @@ Parallel processing with work queue is the standard implementation. There is no
491491

492492
- **Main file:** `bin/dwh/datamartCountries/datamartCountries.sh`
493493
- **Population SQL:**
494-
`sql/dwh/datamartCountries/datamartCountries_31_populateDatamartCountriesTable.sql`
495-
- **Procedure:** `sql/dwh/datamartCountries/datamartCountries_13_createProcedure.sql`
494+
`sql/dwh/datamartCountries/datamartCountries_30_populateDatamartCountriesTable.sql`
495+
- **Procedure:** `sql/dwh/datamartCountries/datamartCountries_12_createProcedure.sql`
496496
- **Related:** `bin/dwh/datamartUsers/PARALLEL_PROCESSING.md` (different approach)

bin/dwh/datamartCountries/datamartCountries.sh

Lines changed: 15 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -87,35 +87,35 @@ if [[ -z "${PROCESS_TYPE:-}" ]]; then
8787
fi
8888

8989
# Name of the SQL script that contains the objects to create in the DB.
90-
declare -r CHECK_OBJECTS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_11_checkDatamartCountriesTables.sql"
90+
declare -r CHECK_OBJECTS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_10_checkDatamartCountriesTables.sql"
9191

9292
# Name of the SQL script that contains the tables to create in the DB.
93-
declare -r CREATE_TABLES_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_12_createDatamarCountriesTable.sql"
93+
declare -r CREATE_TABLES_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_11_createDatamarCountriesTable.sql"
9494

9595
# Name of the SQL script that contains the procedures to create in the DB.
96-
declare -r CREATE_PROCEDURES_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_13_createProcedure.sql"
96+
declare -r CREATE_PROCEDURES_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_12_createProcedure.sql"
9797

9898
# Generic script to add years.
99-
declare -r ADD_YEARS_SCRIPT="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_21_alterTableAddYears.sql"
99+
declare -r ADD_YEARS_SCRIPT="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_23_alterTableAddYears.sql"
100100

101101
# Name of the SQL script that contains the ETL process.
102-
declare -r POPULATE_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_31_populateDatamartCountriesTable.sql"
102+
declare -r POPULATE_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_30_populateDatamartCountriesTable.sql"
103103

104104
# Last year activities script.
105105
declare -r LAST_YEAR_ACTITIES_SCRIPT="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamarts_lastYearActivities.sql"
106106

107107
# Performance optimization scripts (applied automatically on setup)
108108
declare -r OPTIMIZE_INDEXES_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_14_optimize_indexes.sql"
109-
declare -r INCREMENTAL_YEAR_PROCESSING_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_15_incremental_year_processing.sql"
109+
declare -r INCREMENTAL_YEAR_PROCESSING_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_13_incremental_year_processing.sql"
110110
declare -r CONSOLIDATE_METRICS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_17_consolidate_basic_metrics.sql"
111-
declare -r CONSOLIDATE_RANKINGS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_18_consolidate_user_rankings.sql"
112-
declare -r CONSOLIDATE_HASHTAGS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_19_consolidate_hashtags.sql"
113-
declare -r CONSOLIDATE_YEAR_ACTIVITY_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_20_consolidate_year_activity.sql"
111+
declare -r CONSOLIDATE_RANKINGS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_20_consolidate_user_rankings.sql"
112+
declare -r CONSOLIDATE_HASHTAGS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_21_consolidate_hashtags.sql"
113+
declare -r CONSOLIDATE_YEAR_ACTIVITY_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_22_consolidate_year_activity.sql"
114114
declare -r CONSOLIDATE_DATES_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_21_consolidate_dates_metrics.sql"
115-
declare -r CONSOLIDATE_WORKING_HOURS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_22_consolidate_working_hours.sql"
116-
declare -r CONSOLIDATE_APPLICATIONS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_23_consolidate_applications.sql"
117-
declare -r CONSOLIDATE_COMMENTS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_24_consolidate_comments.sql"
118-
declare -r CONSOLIDATE_NOTES_AGE_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_25_consolidate_notes_age.sql"
115+
declare -r CONSOLIDATE_WORKING_HOURS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_24_consolidate_working_hours.sql"
116+
declare -r CONSOLIDATE_APPLICATIONS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_25_consolidate_applications.sql"
117+
declare -r CONSOLIDATE_COMMENTS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_26_consolidate_comments.sql"
118+
declare -r CONSOLIDATE_NOTES_AGE_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartCountries/datamartCountries_27_consolidate_notes_age.sql"
119119

120120
###########
121121
# FUNCTIONS
@@ -575,6 +575,8 @@ function __processNotesCountriesParallel {
575575
__logi "Starting ${adjusted_threads} parallel worker threads..."
576576
for ((thread_num = 1; thread_num <= adjusted_threads; thread_num++)); do
577577
(
578+
# Disable exit on error to handle errors explicitly in this subshell
579+
set +e
578580
local thread_processed=0
579581
local thread_failed=0
580582
local country_id

bin/dwh/datamartGlobal/datamartGlobal.sh

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -74,13 +74,13 @@ if [[ -z "${PROCESS_TYPE:-}" ]]; then
7474
fi
7575

7676
# Name of the SQL script that contains the objects to create in the DB.
77-
declare -r CHECK_OBJECTS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartGlobal/datamartGlobal_11_checkTables.sql"
77+
declare -r CHECK_OBJECTS_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartGlobal/datamartGlobal_10_checkTables.sql"
7878

7979
# Name of the SQL script that contains the tables to create in the DB.
80-
declare -r CREATE_TABLES_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartGlobal/datamartGlobal_12_createTable.sql"
80+
declare -r CREATE_TABLES_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartGlobal/datamartGlobal_11_createTable.sql"
8181

8282
# Name of the SQL script that contains the ETL process.
83-
declare -r POPULATE_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartGlobal/datamartGlobal_31_populate.sql"
83+
declare -r POPULATE_FILE="${SCRIPT_BASE_DIRECTORY}/sql/dwh/datamartGlobal/datamartGlobal_30_populate.sql"
8484

8585
###########
8686
# FUNCTIONS

bin/dwh/datamartUsers/Parallel_Processing.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -540,7 +540,7 @@ batch_size=1000
540540

541541
- **Main file:** `bin/dwh/datamartUsers/datamartUsers.sh`
542542
- **Population SQL:** `sql/dwh/datamartUsers/datamartUsers_32_populateDatamartUsersTable.sql`
543-
- **Procedure:** `sql/dwh/datamartUsers/datamartUsers_13_createProcedure.sql`
543+
- **Procedure:** `sql/dwh/datamartUsers/datamartUsers_12_createProcedure.sql`
544544

545545
## Version History
546546

0 commit comments

Comments
 (0)