|
1 | | -# Log Processor Automation Module |
| 1 | +# Automation (Log Processor) |
2 | 2 |
|
3 | | -## 1. Overview |
| 3 | +This module installs and runs the end‑to‑end data pipeline on a schedule: collect logs → process to CSV → time‑window filter → upload to S3 (or queue if offline). |
4 | 4 |
|
5 | | -This module automates the entire log processing pipeline, from data collection to uploading. It is designed to run as a reliable, unattended background service on a Linux server. |
6 | | -The core of the automation is built using **systemd**, Linux's native service manager. This was chosen over other methods like `cron` because it offers superior logging via the system journal, better service dependency management, and more flexible timer configurations. |
| 5 | +Why systemd? It provides reliable scheduling (timer units), dependency handling, robust logging via journalctl, and runs without an interactive session. |
7 | 6 |
|
8 | | ---- |
| 7 | +Key features |
9 | 8 |
|
10 | | -## 2. Key Features ✨ |
| 9 | +- Scheduled runs via systemd timer (daily/weekly/monthly/custom; hourly for Castle logs) |
| 10 | +- Offline‑first: queues uploads in 00_DATA/00_UPLOAD_QUEUE and flushes next time online |
| 11 | +- Guided configuration with AWS bucket discovery and a live test upload |
| 12 | +- Status dashboard covering timer, queue, connectivity, and AWS identity |
| 13 | +- Dual logging to journalctl and /var/log/v5_log_processor/automation.log |
11 | 14 |
|
12 | | -- **Scheduled Execution**: Automatically runs the data pipeline on a configurable schedule using a dedicated `systemd` timer. |
13 | | -- **Intelligent Offline Queueing**: If the internet is unavailable, processed files are automatically placed in an `00_DATA/00_UPLOAD_QUEUE` directory. The system flushes this queue by uploading all pending files the next time it runs with an active internet connection. |
14 | | -- **Advanced Interactive Menu**: A user-friendly menu (enhanced with `whiptail` if available) for installing, checking status, and configuring the automation. |
15 | | -- **Dynamic & Validated Configuration**: The configuration script dynamically discovers available S3 buckets and subfolders. Before saving, it performs a **live upload test** to validate that the chosen settings and AWS credentials are correct. |
16 | | -- **Comprehensive Status Dashboard**: The status script provides a full health check, including timer status, queue contents, network connectivity, and AWS identity verification. |
17 | | -- **Dual Logging**: All output is simultaneously logged to both the systemd journal (viewable with `journalctl`) and a persistent file at `/var/log/v5_log_processor/automation.log` for robust troubleshooting. |
18 | | -- **Automatic AWS Region Detection**: The upload scripts automatically detect the correct AWS region for the target S3 bucket, eliminating the need for manual configuration. |
| 15 | +Prerequisites |
19 | 16 |
|
20 | | ---- |
| 17 | +- Linux with systemd, bash, Python 3 |
| 18 | +- AWS CLI installed and configured for the service user (typically pi) |
| 19 | +- Python dependencies for processors (see requirements.txt) |
21 | 20 |
|
22 | | -## 3. How It Works ⚙️ |
| 21 | +Components |
23 | 22 |
|
24 | | -The automation follows a clear, multi-stage workflow triggered by a system timer. |
| 23 | +- main.sh — Menu entrypoint for Install, Status, Configure |
| 24 | +- install.sh — Creates service/timer and the wrapper at /usr/local/bin/run_v5_log_processor.sh |
| 25 | +- configure.sh — Writes config/automation.conf; discovers buckets/subfolders; validates with live test upload; sets schedule |
| 26 | +- runner.sh — Orchestrates the pipeline (collect → process → filter → upload/queue) and auto‑detects S3 bucket region |
| 27 | +- status.sh — Health/status report: timer/service, queue contents, connectivity, AWS identity, last logs |
| 28 | +- flush_queue.sh — Uploads any queued CSVs using per‑bucket region detection |
| 29 | +- filter_time_based.py — Builds final CSVs for hourly/daily/weekly; automation calls process_csv.py for monthly |
| 30 | +- /usr/local/bin/run_v5_log_processor.sh — Wrapper used by systemd to run runner.sh and tee output to both log and journal |
25 | 31 |
|
26 | | -1. **Timer Triggers**: At the scheduled time, the `v5-log-processor.timer` unit tells `systemd` to start the service. The schedule is managed via a robust override file in `/etc/systemd/system/v5-log-processor.timer.d/`. |
27 | | -2. **Service Runs**: The `v5-log-processor.service` unit executes the main wrapper script located at `/usr/local/bin/run_v5_log_processor.sh`. |
28 | | -3. **Pipeline Initiated**: The wrapper script is responsible for setting up the environment. It navigates to the project directory and executes the main pipeline script, `scripts/data/automation/runner.sh`. It also uses the `tee` command to pipe all output to both the journal and the log file. |
29 | | -4. **Configuration Loaded**: The `runner.sh` script sources its settings from `config/automation.conf`, loading the correct Server Version, S3 Bucket, etc. |
30 | | -5. **The Data Workflow (executed by `runner.sh`)**: |
31 | | - - **Collect**: The script collects raw server logs based on the `SERVER_VERSION` setting. |
32 | | - - **Process**: The collected logs are cleaned and transformed into a `summary.csv` by the appropriate Python processor (`log.py`, `logv2.py`, or `castle.py`), which is selected based on the `SERVER_VERSION` and `PYTHON_SCRIPT` variables. |
33 | | - - **Filter & Finalize**: The `summary.csv` is further processed by `process_csv.py` to generate the final, month-specific CSV file. |
34 | | - - **Upload or Queue**: The script checks for internet connectivity. |
35 | | - - If **online**, it first runs the `flush_queue.sh` logic to upload any previously queued files, then uploads the newly generated file. |
36 | | - - If **offline**, it copies the new file to the `00_DATA/00_UPLOAD_QUEUE` directory to be uploaded later. |
| 32 | +Configuration (config/automation.conf) |
37 | 33 |
|
38 | | ---- |
| 34 | +Written by configure.sh and kept inside the repo so the automation can run from the project directory. |
39 | 35 |
|
40 | | -## 4. Usage and Management |
| 36 | +- SERVER_VERSION: v1 (Server v4/Apache) or v2 (Server v5/OC4D) |
| 37 | +- PYTHON_SCRIPT: oc4d or cape_coast_d (only v2) |
| 38 | +- DEVICE_LOCATION: short label used in folder names and output filenames |
| 39 | +- S3_BUCKET: s3://bucket‑name |
| 40 | +- S3_SUBFOLDER: optional prefix under the bucket; files go to subfolder/RACHEL/ |
| 41 | +- SCHEDULE_TYPE: hourly (castle only), daily, weekly, monthly, or custom |
| 42 | +- RUN_INTERVAL: for custom schedules (seconds, >= 300) |
41 | 43 |
|
42 | | -To manage the automation, navigate to the project root and run the menu script. |
| 44 | +Data flow |
43 | 45 |
|
44 | | -```bash |
45 | | -./main.sh |
| 46 | +1. Collect |
46 | 47 |
|
| 48 | +- v1/v4: copies /var/log/apache2 access.log\* into 00_DATA/LOCATION_logs_YYYY_MM_DD |
| 49 | +- v2/v5: copies /var/log/oc4d oc4d-_.log, capecoastcastle-_.log, and \*.gz (excludes exceptions) |
| 50 | +- Gzip files are decompressed |
47 | 51 |
|
48 | | ---- |
| 52 | +1. Process |
49 | 53 |
|
50 | | -## ⚙️ Automation |
| 54 | +- Chooses processor based on SERVER_VERSION and PYTHON_SCRIPT: |
| 55 | + - v1/v4 → scripts/data/process/processors/log.py |
| 56 | + - v2/v5 (oc4d) → scripts/data/process/processors/logv2.py |
| 57 | + - v2/v5 (cape_coast_d) → scripts/data/process/processors/castle.py |
| 58 | +- Produces 00_DATA/00_PROCESSED/RUN_FOLDER/summary.csv |
51 | 59 |
|
52 | | -This project includes a powerful automation module for scheduling the entire log processing pipeline. It can be installed as a systemd service to run automatically, even when no user is logged in. |
| 60 | +1. Filter to final CSV |
53 | 61 |
|
54 | | -**[For full details, see the Automation Module Documentation](./scripts/data/automation/README.md)** |
| 62 | +- hourly/daily/weekly → filter_time_based.py selects the last completed window and writes a device‑named CSV in the same folder; it prints the filename to stdout |
| 63 | +- monthly → scripts/data/upload/process_csv.py runs in filename mode to create LOCATION_MM_YYYY.csv and prints the filename |
| 64 | + |
| 65 | +1. Upload or queue |
| 66 | + |
| 67 | +- Connectivity check: if online, first flush queued files, then upload the new one using per‑bucket region detection; if offline, copy the new file to 00_DATA/00_UPLOAD_QUEUE |
| 68 | + |
| 69 | +Scheduling |
| 70 | + |
| 71 | +- Timer unit: v5-log-processor.timer |
| 72 | +- Service unit: v5-log-processor.service |
| 73 | +- Schedule is applied via override at /etc/systemd/system/v5-log-processor.timer.d/override.conf |
| 74 | +- Modes: |
| 75 | + - hourly (only when PYTHON_SCRIPT=cape_coast_d) |
| 76 | + - daily, weekly, monthly |
| 77 | + - custom interval via OnUnitActiveSec=RUN_INTERVAL |
| 78 | +- Timers are Persistent=true, so missed runs trigger shortly after boot |
| 79 | + |
| 80 | +Where things live |
| 81 | + |
| 82 | +- Config: config/automation.conf (chmod 600, owned by the service user) |
| 83 | +- Raw runs: 00_DATA/<DEVICE_LOCATION>\_logs_YYYY_MM_DD/ |
| 84 | +- Processed: 00_DATA/00_PROCESSED/<DEVICE_LOCATION>\_logs_YYYY_MM_DD/ |
| 85 | +- Queue: 00_DATA/00_UPLOAD_QUEUE/ |
| 86 | +- Logs: /var/log/v5_log_processor/automation.log and journalctl -u v5-log-processor.service |
| 87 | + |
| 88 | +Commands |
| 89 | + |
| 90 | +- Install: sudo ./scripts/data/automation/install.sh |
| 91 | +- Configure: sudo ./scripts/data/automation/configure.sh |
| 92 | +- Status: ./scripts/data/automation/status.sh |
| 93 | +- Manual run (wrapper): sudo /usr/local/bin/run_v5_log_processor.sh |
| 94 | + |
| 95 | +Troubleshooting |
| 96 | + |
| 97 | +- Use ./scripts/data/automation/status.sh to quickly see timer state, next/last run, queue, connectivity, and AWS identity |
| 98 | +- If uploads fail, runner auto‑detects bucket region but credentials/policies can still block PUT; run Configure and use the live test or aws sts get-caller-identity |
| 99 | +- “No new entries matched the time period” means summary.csv existed but didn’t contain rows in the last completed window |
| 100 | +- If config can’t be read, ensure config/automation.conf exists and is readable; Configure will recreate it with correct ownership and perms |
| 101 | +- To disable, sudo systemctl stop v5-log-processor.timer && sudo systemctl disable v5-log-processor.timer |
0 commit comments