|
| 1 | +# Monitoring SQL Warehouse |
| 2 | + |
| 3 | +A small, dedicated serverless SQL warehouse for scheduled Databricks Alerts, monitoring queries, and ad-hoc health checks. Optimized for bursty, sub-second workloads where idle time would otherwise dominate the bill. |
| 4 | + |
| 5 | +## When to use this |
| 6 | + |
| 7 | +Use a dedicated warehouse like this when: |
| 8 | + |
| 9 | +- You run scheduled Databricks Alerts (every minute, every five minutes, every hour) whose underlying SQL completes in well under a second. |
| 10 | +- The default warehouse those queries land on has a long auto-stop window (10 minutes is the platform default for Pro/Classic; serverless allows 1 minute). |
| 11 | +- You see the warehouse staying warm for the full auto-stop window after a single sub-second query, inflating monthly cost. |
| 12 | + |
| 13 | +A separate `2X-Small` serverless warehouse with `auto_stop_mins: 1` keeps cost proportional to actual query time: the warehouse starts on the Alert run, executes in under a second, and stops one minute after the last query. |
| 14 | + |
| 15 | +## What this asset installs |
| 16 | + |
| 17 | +| Path | Purpose | |
| 18 | +|---|---| |
| 19 | +| `<target_dir>/<warehouse_resource_key>.sql_warehouse.yml` | The DABs SQL warehouse resource definition. | |
| 20 | +| `docs/monitoring-sql-warehouse/README.md` | This file. | |
| 21 | + |
| 22 | +Default install paths: |
| 23 | + |
| 24 | +- Resource: `resources/monitoring_sql_warehouse.sql_warehouse.yml` |
| 25 | +- Docs: `docs/monitoring-sql-warehouse/README.md` |
| 26 | + |
| 27 | +## Bundle integration |
| 28 | + |
| 29 | +Most generated bundles already include `resources/*.yml` from `databricks.yml`: |
| 30 | + |
| 31 | +```yaml |
| 32 | +include: |
| 33 | + - resources/*.yml |
| 34 | +``` |
| 35 | +
|
| 36 | +If your bundle uses that pattern and you accepted the default `target_dir`, the warehouse is picked up automatically. No `databricks.yml` change is needed. |
| 37 | + |
| 38 | +If you installed to a custom subdirectory (e.g., `resources/sql_warehouses/`), add the matching glob to `databricks.yml`: |
| 39 | + |
| 40 | +```yaml |
| 41 | +include: |
| 42 | + - resources/sql_warehouses/*.yml |
| 43 | +``` |
| 44 | + |
| 45 | +## Deploy |
| 46 | + |
| 47 | +```bash |
| 48 | +databricks bundle validate -t <your-target> |
| 49 | +databricks bundle deploy -t <your-target> |
| 50 | +``` |
| 51 | + |
| 52 | +After deploy, the warehouse appears in the workspace SQL Warehouses list under the display name you chose at install time. |
| 53 | + |
| 54 | +## Referencing the warehouse from other bundle resources |
| 55 | + |
| 56 | +Other resources (Alerts, jobs, dashboards) can pin to this warehouse by ID: |
| 57 | + |
| 58 | +```yaml |
| 59 | +${resources.sql_warehouses.<warehouse_resource_key>.id} |
| 60 | +``` |
| 61 | + |
| 62 | +For example, an Alert resource: |
| 63 | + |
| 64 | +```yaml |
| 65 | +resources: |
| 66 | + alerts: |
| 67 | + my_alert: |
| 68 | + warehouse_id: ${resources.sql_warehouses.monitoring_sql_warehouse.id} |
| 69 | + # ... rest of the alert definition |
| 70 | +``` |
| 71 | + |
| 72 | +## The `auto_stop_mins: 1` nuance |
| 73 | + |
| 74 | +The proven configuration was verified by deploying this asset to a real workspace (Databricks SQL channel v2026.10). The notes below capture behavior that contradicts or refines parts of the public API reference. |
| 75 | + |
| 76 | +| Setting | Behavior | |
| 77 | +|---|---| |
| 78 | +| `auto_stop_mins: 0` | Disables auto-stop entirely. The warehouse runs until manually stopped. Does NOT mean "stop immediately." | |
| 79 | +| `auto_stop_mins: 1` | Valid and effective on serverless via DABs/API. The workspace UI shows "After 1 minute of inactivity." This is the recommended value for Alerts and monitoring workloads. | |
| 80 | +| `auto_stop_mins: 0.5` (or any sub-minute value) | Not supported. The field is modeled as an integer; the bundle CLI emits an integer-coercion warning and the workspace silently reverts to the platform default (120 minutes). | |
| 81 | +| `auto_stop_mins: 10+` | Per the public [Warehouses API reference][create-warehouse-api], the documented contract is `auto_stop_mins` must be `0` or `>= 10`. This floor applies to Pro/Classic warehouses. Safe upper bound for serverless too. | |
| 82 | + |
| 83 | +The public [Warehouses API reference][create-warehouse-api] states that `auto_stop_mins` must be `0` or `>= 10`. That floor applies to Pro/Classic warehouses; the serverless path accepts `1` end-to-end via DABs and via the CLI (confirmed by deploying this resource and inspecting the workspace UI). |
| 84 | + |
| 85 | +The manual UI editor for an existing warehouse may enforce a higher floor on serverless than 1 minute. If you see that, fall back to the DABs path or the `databricks warehouses edit` CLI path, both of which apply `1` on serverless. |
| 86 | + |
| 87 | +[create-warehouse-api]: https://docs.databricks.com/api/workspace/warehouses/create |
| 88 | +[edit-warehouse-api]: https://docs.databricks.com/api/workspace/warehouses/edit |
| 89 | + |
| 90 | +## Applying the same tuning to a warehouse created outside DABs |
| 91 | + |
| 92 | +If you have an existing warehouse that you do not want to recreate as a DABs resource, you can edit it in place to the same auto-stop behavior. |
| 93 | + |
| 94 | +### Via the Databricks CLI |
| 95 | + |
| 96 | +```bash |
| 97 | +databricks warehouses edit <warehouse-id> --auto-stop-mins 1 |
| 98 | +``` |
| 99 | + |
| 100 | +The full flag set is visible via `databricks warehouses edit --help`. For serverless, also ensure `--enable-serverless-compute` is on and `--warehouse-type PRO` is set. |
| 101 | + |
| 102 | +### Via the REST API |
| 103 | + |
| 104 | +Use the [Edit Warehouse endpoint][edit-warehouse-api] (`POST /api/2.0/sql/warehouses/{id}/edit`) with `auto_stop_mins: 1`. For serverless, include `enable_serverless_compute: true` and `warehouse_type: PRO` in the request body. The endpoint reference still documents the `0 or >= 10` contract; the serverless `1` value is accepted in practice and is what DABs and the CLI emit. |
| 105 | + |
| 106 | +```bash |
| 107 | +curl -X POST \ |
| 108 | + -H "Authorization: Bearer $DATABRICKS_TOKEN" \ |
| 109 | + -H "Content-Type: application/json" \ |
| 110 | + https://$DATABRICKS_HOST/api/2.0/sql/warehouses/<warehouse-id>/edit \ |
| 111 | + -d '{ |
| 112 | + "auto_stop_mins": 1, |
| 113 | + "enable_serverless_compute": true, |
| 114 | + "warehouse_type": "PRO" |
| 115 | + }' |
| 116 | +``` |
| 117 | + |
| 118 | +### Via the workspace UI |
| 119 | + |
| 120 | +The "Edit warehouse" form may enforce a higher minimum than 1 minute for serverless. If the UI rejects `1`, use the CLI or DABs path. |
| 121 | + |
| 122 | +## Cost intuition |
| 123 | + |
| 124 | +A `2X-Small` serverless warehouse billed per second of running time, kept warm by a 1-minute auto-stop after each sub-second query, costs a small constant per scheduled run plus the query duration itself. Compared to a 10-minute auto-stop window on the same size warehouse, the per-run dwell time drops by an order of magnitude. Apply this to dozens of scheduled Alerts and the monthly delta is meaningful. |
| 125 | + |
| 126 | +## References |
| 127 | + |
| 128 | +1. [Databricks Docs: Configure SQL warehouse settings (auto-stop)](https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-behavior) |
| 129 | +2. [Declarative Automation Bundles: `sql_warehouses` resource reference](https://docs.databricks.com/aws/en/dev-tools/bundles/resources.html#sql_warehouse) |
| 130 | +3. [Databricks API: Create Warehouse][create-warehouse-api] (`auto_stop_mins` contract) |
| 131 | +4. [Databricks API: Edit Warehouse][edit-warehouse-api] (in-place edits to an existing warehouse) |
| 132 | +5. [Databricks CLI: `databricks warehouses edit`](https://docs.databricks.com/aws/en/dev-tools/cli/reference/warehouses-commands.html#edit) |
| 133 | +6. [Databricks Alerts overview](https://docs.databricks.com/aws/en/sql/user/alerts/) |
0 commit comments