Skip to content

Conversation

@lennartkats-db
Copy link
Contributor

Changes

This extends apps-mcp with jobs & pipeline initialization:

  • Moved most of theinit_template.go into init_template/app.go
  • Added init_template/{job,pipeline,empty}.go and support for respective templates
  • Added generic AGENTS.md/CLAUDE.md for the new templates
  • Updated associated prompts

Why

We want to generalize apps-mcp to other resource types

Tests

  • Acceptance tests
  • Manual and agent-based experiental validation

lennartkats-db and others added 5 commits December 24, 2025 15:15
Refactors init-template into dedicated subcommands with comprehensive
L2 guidance for each resource type:

- Add job subcommand using default-python template
- Add pipeline subcommand using lakeflow-pipelines template
- Add empty subcommand using default-minimal template
- Create L2 templates: target_jobs, target_pipelines, target_mixed
- Add AGENTS.tmpl for scaffolded projects
- Parameterize default catalog in discover flow
- Update detector to show "mixed" guidance for non-app-only projects
- Move implementation to cmd/init_template/ subpackage
- Make databricks_discover description more forceful (MANDATORY)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
…y configMap conditionals

- Add acceptance tests for job, pipeline, empty, and app init-template subcommands
- Remove unnecessary 'if catalog != ""' checks (templates handle empty values)
- Fix logdiag double-initialization in bundle detector
- Simplify FormatProjectScaffoldSuccess output formatting
- Remove undefined listAllSkills parameter from generateDiscoverGuidance call
- Fix test expectation: jobs-only projects now include "mixed" target type

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@lennartkats-db lennartkats-db requested a review from a team as a code owner December 31, 2025 08:51
@lennartkats-db lennartkats-db changed the title [draft] apps-mcp: add templates for Lakeflow jobs & pipelines templates apps-mcp: add templates for Lakeflow jobs & pipelines templates Dec 31, 2025
lennartkats-db and others added 2 commits January 3, 2026 11:52
This makes the terminology more accurate for empty and single-resource
bundles, and makes resource detection more maintainable.

Changes:
- Rename target_mixed.tmpl to target_bundle.tmpl
- Use AllResources() iterator instead of manually checking each type
- Simplify isAppOnly logic to: hasApps && len(TargetTypes) == 1
- Add default catalog logic to empty.go for consistency
- Add test coverage for empty bundles

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
@lennartkats-db lennartkats-db requested a review from igrekun January 5, 2026 14:13
@lennartkats-db lennartkats-db added this pull request to the merge queue Jan 6, 2026
Merged via the queue into main with commit b02beb4 Jan 6, 2026
26 checks passed
@lennartkats-db lennartkats-db deleted the lakeflow-mcp branch January 6, 2026 12:16
@eng-dev-ecosystem-bot
Copy link
Collaborator

Commit: b02beb4

Run: 20748036010

Env ❌​FAIL 🟨​KNOWN 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🟨​ aws linux 15 5 2 417 652 352:25
🟨​ aws windows 14 6 2 419 650 51:13
🟨​ aws-ucws linux 11 9 2 586 523 314:45
🟨​ aws-ucws windows 16 4 2 588 521 113:10
🟨​ azure linux 10 5 3 417 651 310:29
❌​ azure windows 2 15 3 417 649 80:45
🟨​ azure-ucws linux 10 2 5 3 580 522 352:06
❌​ azure-ucws windows 2 15 3 582 520 110:01
🟨​ gcp linux 10 5 3 397 660 57:34
❌​ gcp windows 2 15 3 397 658 50:14
28 interesting tests: 21 KNOWN, 6 FAIL, 1 SKIP
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/deployment/bind/alert 🙈​S 🙈​S 🙈​S 🙈​S 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
❌​ TestAccept/bundle/integration_whl/interactive_cluster ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f ❌​F ✅​p ✅​p
❌​ TestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f ❌​F ✅​p ✅​p
❌​ TestAccept/bundle/integration_whl/interactive_single_user ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F
❌​ TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F
🟨​ TestAccept/bundle/resources/alerts/basic 🟨​K 💚​R 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 💚​R 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 💚​R 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/alerts/with_file 🟨​K 💚​R 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 💚​R 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/alerts/with_file/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 💚​R 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
❌​ TestAccept/bundle/resources/clusters/run/spark_python_task ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ✅​p ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p ✅​p ✅​p ✅​p ❌​F ✅​p ✅​p ✅​p ✅​p
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🟨​K 🟨​K 🟨​K 🟨​K 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestExport 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestExportWithFileFlag 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestImportDir 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestImportDirDoesNotOverwrite 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestImportDirWithOverwriteFlag 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestImportFileFormatAuto 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestImportFileFormatSource 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
Top 50 slowest tests (at least 2 minutes):
duration env testname
14:01 gcp windows TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
12:24 gcp linux TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
11:52 aws-ucws linux TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
11:52 azure linux TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
10:47 aws-ucws windows TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
10:40 aws-ucws windows TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
10:31 azure windows TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
10:07 aws-ucws windows TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
10:00 aws-ucws linux TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=terraform
9:57 aws-ucws linux TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
9:55 aws-ucws linux TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
9:35 aws-ucws linux TestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=terraform
9:27 azure-ucws linux TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
9:15 azure-ucws linux TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
9:08 aws-ucws windows TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION
9:02 gcp windows TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION
8:50 gcp linux TestAccept/bundle/run/app-with-job/DATABRICKS_BUNDLE_ENGINE=terraform
8:45 aws-ucws linux TestAccept/bundle/resources/database_instances/single-instance/DATABRICKS_BUNDLE_ENGINE=terraform
8:40 aws-ucws linux TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=USER_ISOLATION
8:38 azure-ucws linux TestAccept/bundle/resources/permissions/factcheck/DATABRICKS_BUNDLE_ENGINE=terraform
8:31 gcp windows TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:27 aws linux TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
8:24 aws-ucws windows TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION
8:19 azure-ucws windows TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
8:18 gcp windows TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION
8:17 aws-ucws windows TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
8:17 azure windows TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
8:16 aws-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
8:15 aws windows TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
8:15 aws windows TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION
8:07 aws windows TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
8:06 aws-ucws linux TestAccept/bundle/resources/model_serving_endpoints/running-endpoint/DATABRICKS_BUNDLE_ENGINE=direct
8:05 gcp linux TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=USER_ISOLATION
8:04 aws-ucws windows TestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=direct
8:03 aws-ucws windows TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=direct/DATA_SECURITY_MODE=SINGLE_USER
8:00 aws-ucws windows TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
7:55 aws linux TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
7:54 azure-ucws windows TestAccept/bundle/integration_whl/interactive_single_user/DATABRICKS_BUNDLE_ENGINE=terraform
7:49 gcp windows TestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=terraform
7:49 gcp linux TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
7:49 gcp linux TestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=direct
7:46 aws linux TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
7:44 aws-ucws linux TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER
7:44 gcp windows TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=direct
7:43 gcp linux TestAccept/bundle/resources/clusters/run/spark_python_task/DATABRICKS_BUNDLE_ENGINE=terraform
7:42 azure linux TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
7:42 aws windows TestAccept/bundle/integration_whl/base/DATABRICKS_BUNDLE_ENGINE=terraform
7:41 gcp windows TestAccept/bundle/integration_whl/interactive_cluster/DATABRICKS_BUNDLE_ENGINE=direct
7:40 azure-ucws linux TestSparkJarTaskDeployAndRunOnVolumes/Databricks_Runtime_14.3_LTS
7:39 azure windows TestAccept/bundle/integration_whl/interactive_cluster_dynamic_version/DATABRICKS_BUNDLE_ENGINE=terraform/DATA_SECURITY_MODE=SINGLE_USER

github-merge-queue bot pushed a commit that referenced this pull request Jan 12, 2026
…4199)

## Changes

This adds a **skills system** to apps-mcp that provides domain-specific
implementation guides to AI agents. It follows up on
#4183. Skills are organized by
resource type and delivered through the discover tool. They work for any
agent that supports MCP.

Detailed overview:
* Skills live in `lib/skills/{apps,jobs,pipelines,...}` and include a
standard `SKILL.md` file with a skill front matter.
* The list of skills is shared via the `databricks_discover` tool.
* Skills can be read using the `read_skill_file` tool (which has a
precedent in many other MCPs an in our first-party agent).
* Adding useful skills is considered followup work. The present PR
includes one sample skill for doing auto-CDC with pipelines.

- [x] **Stacked PR**: this branch is based on [the `lakeflow-mcp`
branch](#4183); before merging it
needs to be based on `main` instead

## Why

* Supporting skills enables us to extend the set of domain-specific
capabilities of agents
* We want skills to be the universal format for these capabilities
across different agent implementations.
* Not all agents natively support skills (Codex, Cursor); the
`read_skill_file` tool helps these agents use skills.

## Tests

- Extended detector tests to validate resource type detection
- Updated acceptance tests for all init-template commands
- Manual validation with agents using the skills system

---------

Co-authored-by: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants