feat(beacon): add Beacon probe discovery, registration, and flashing#19
feat(beacon): add Beacon probe discovery, registration, and flashing#19JohnBaumb merged 24 commits intoJohnBaumb:devfrom
Conversation
…code-review hardening, and bridge flash fixes Closes JohnBaumb#10, JohnBaumb#14, JohnBaumb#15, JohnBaumb#16. Incorporates PR JohnBaumb#13. Summary: This release adds AVR microcontroller flashing, overhauls the installer into a modular and robust pipeline, introduces health and update-check endpoints with UI indicators, hardens the backend with proper logging/locking/timeouts from a code review pass, extracts a correct Katapult wire-protocol module, and fixes several critical flash-path bugs for CAN bridges and Linux MCU devices. Features: - AVR flashing support: Flash AVR-based boards via avrdude. Batch operations now handle non-Katapult devices that skip the CAN reboot phase. Firmware path resolution falls back to .elf for AVR targets. (Closes JohnBaumb#15) - Health endpoint (/api/health): Reports install-time issues (missing binaries, broken venv, etc.). UI shows a green check or orange warning icon on the Update button. - Update-check endpoint (/api/update-check): Compares local HEAD against the remote branch and returns how many commits behind. UI displays a badge with the count. - Commit hash and branch display: The sidebar footer now shows the running commit hash (right-aligned) and the active branch name when not on main. - Kalico firmware-extras auto-discovery: Automatically runs Kalico's find-firmware-extras.sh before Kconfig parsing so extra board definitions are available. - Katapult protocol module: Extracted CAN/serial Katapult communication into backend/katapult_protocol.py with correct wire format, replacing ad-hoc implementations. - Installer overhaul: install.sh refactored with set -Eeuo pipefail, structured logging helpers (log_info/log_warn/log_error), and an ERR trap that prints the failing line. Moonraker integration, Mainsail navi.json setup, and sudoers configuration are now handled by dedicated Python scripts under install_scripts/. System dependencies are read from system-dependencies.json (single source of truth). (PR JohnBaumb#13) - Moonraker declarative dependency management: New setup_moonraker.py writes the [update_manager klipperfleet] block idempotently, and system-dependencies.json declares runtime packages for Moonraker's dependency resolver. - Mainsail navi redirect shim: Navi link now points to a local HTML redirect (klipperfleet.html) served by Nginx, preserving the user's hostname/IP instead of hardcoding it. - Self-healing startup: On startup, the backend auto-repairs missing sudoers files (for main-to-dev upgrades where install.sh was never re-run), migrates Moonraker config, and ensures system dependencies. Bug Fixes: - CAN bridge flash-path correction: Bridges with a /dev/serial/by-id path are now auto-corrected from CAN to serial flash method, preventing silent flash failures. (Closes JohnBaumb#16) - CAN bridge serial fallback after reboot: After a CAN bridge reboots into Katapult, the flash manager switches to the serial path for the actual flash instead of attempting CAN (which is down). (Closes JohnBaumb#16) - CAN bridge serial dedup: Fixed serial attach creating duplicate fleet entries by deduplicating on serial_id. - CAN bridge reboot fallback: When can0 is down, bridge reboot now falls back to serial instead of failing. - Linux MCU flash on Ubuntu: Fixed flash failure on Ubuntu where the Linux MCU service name differs from Raspberry Pi OS. (Closes JohnBaumb#14) - Linux MCU post-flash service ordering: klipper-mcu service is now restarted after firmware install, with correct service start ordering. - AVR firmware path resolution: build_manager now resolves .elf output for AVR architectures instead of only looking for .bin. (Closes JohnBaumb#15) - UART flashing fixes: Multiple fixes for UART-based upload paths (double-write, JSON serialization, hostname resolution). (Closes JohnBaumb#10) - Stale navi entries: setup_mainsail_navi.py now removes old entries by href (not just title), fixing UNKNOWN items left by earlier installs. - Stale profile in localStorage: UI now validates the cached profile name against the server on startup; clears it if the profile no longer exists. - Kconfiglib install recognition: Fixed glob patterns and detection logic for the bundled Klipper kconfiglib vs. the pip-installed version. - Traceback leak in /config/tree: Replaced traceback.format_exc() with str(e) to avoid leaking internal stack traces to the client. - is_katapult model default: Fixed default from False to True to match .get() fallbacks elsewhere. - Profile download validation: Added validate_profile_name to /download endpoint. - Private-notes gitignore: Added private-notes/ to .gitignore. Code Quality and Hardening: - Replaced all print() calls with proper logger in flash_manager, kconfig_manager, and build_manager. - Converted FleetManager from threading.Lock to asyncio.Lock (all callers are async). - Added build timeout: 10 min total, 2 min stall detection in run_build. - Narrowed os.chdir scope in kconfig_manager to reduce race window. - Normalized all 'Profile not found' error messages to a consistent format. - Replaced past-tense string hack with a dict lookup. - Trimmed bloated docstrings and over-narrated comments across flash/kconfig managers. - Fixed duplicate type annotation in FleetManager.save_device. - Profile rename now handles .elf, .hex, and .build_info.json artifacts. - Added set -Eeuo pipefail to update.sh. - Removed attribution comment from install.sh. UI/UX: - Self-update confirmation dialog reworded to clearly state it will reinstall dependencies and restart the service. - Version bumped to v1.2.0-alpha in the sidebar. Files Changed: 25 files, +2650 / -347 lines
Moonraker's update_manager API does not return the 'path' field for git_repo entries. Add fallback to parse moonraker.conf directly for the beacon_klipper repo path.
| baudrate: 250000, // Default baudrate for serial (Katapult default) | ||
| notes: '', | ||
| is_katapult: dev.id.toLowerCase().includes('katapult') || dev.id.toLowerCase().includes('canboot'), | ||
| is_katapult: !isBeacon && (dev.id.toLowerCase().includes('katapult') || dev.id.toLowerCase().includes('canboot')), |
There was a problem hiding this comment.
beacon is only usb - this should possibly be reverted.
| magic_baud_tested: false, | ||
| use_magic_baud: false, | ||
| exclude_from_batch: false | ||
| exclude_from_batch: isBeacon |
There was a problem hiding this comment.
feels a bit weird doing this and maybe the intention is to default to excluded and let the user enable it?
|
Very nice! I didn't even consider this as an option. I always see my beacon probe in my discovered but never even thought of handling updating it through KF. Beacon's update process seems pretty robust already, but I like this, I'll test it out and let you know this week. |
Replace moonraker.conf parsing fallback with Moonraker's /server/config API endpoint to discover the beacon_klipper repo path. This is the approach recommended by the Moonraker maintainer and requires no upstream changes.
Beacon path discovery: switched from
|
Yeah, I thought, why not. How hard can it be... Using a digital twin gets the ideas out of the mind and into reality. I'm not too happy with the ui changes where I made comments. My frontend understanding is lacking there. A "beacon" type could possibly be more generic as a future refactor. Just a pity the beacon firmware is solid for flash testing 😉 Agree that having a card for it and not in the "unknown" list is a bonus. |
Address PR JohnBaumb#19 maintainer feedback: 1. Filter beacon devices (Beacon_Beacon_Rev) from serial device discovery so users can't accidentally add them as managed devices — beacon should only appear via auto-detection. 2. Fix version comparison for beacon devices: - Add method and remote_version to version API response - Extract remote_version from Moonraker update_manager status - versionMatch/versionColor/versionTitle now compare beacon against its own remote_version instead of system Klipper - Tooltip shows "Beacon: v2.0.0-28 | Latest: v2.0.0-28" instead of confusing Klipper version pairing
The beacon hardware firmware (e.g. Beacon 2.1.0) is independent of the beacon_klipper Python repo version (e.g. v2.0.0-30). KlipperFleet now: - Gets live_version from Klipper MCU query (mcu beacon.mcu_version) instead of Moonraker update_manager (which returns repo version) - Sets remote_version to None (no remote FW version API exists) - Stores repo info as repo_version/repo_remote_version for display - Shows blue (informational) instead of green/orange since FW update status cannot be determined via API - Tooltip: "Beacon FW: 2.1.0 | Klipper plugin: v2.0.0-30"
|
@JohnBaumb - thanks for the feedback.
Verified live on V2.4, all 14 tests pass. Wait a minute - firmware is 2.1.0 and not the repo version - they are independent. Digging further.
|
The available beacon firmware version is determined from the beacon_klipper repo's firmware/ directory git history using a 3-step fallback chain: 1. Extract semver from latest DFU commit message (e.g. "firmware: version 2.1.0 release" → 2.1.0) 2. Closest git tag to that commit (e.g. v2.0.0 → 2.0.0) 3. Short commit hash as fallback (e.g. git-b30f733) UI strips "Beacon " prefix from MCU-reported firmware version before comparing against remote_version. Shows green/orange for semver matches, blue for hash-only fallback. Adds 4 tests covering all fallback scenarios.
|
The "Flash Beacon" on the card should stay or be removed? |
|
I think it can stay flash beacon, I'll try this out tonight. |
|
The beacon software is a "monorepo" that has the firmware files that are released very rarely, and the python code (code_base) that causes the moonraker plugin to show "update", as it is changed more regularly. The beacon code change in the last commit, identifies the firmware_version state versus the code_base state. If the firmware_version matches the installed_firmware_version, the "Flash Beacon" button should be disabled/changed to show it is on the latest. This is very similar to "klipper/kalico" where not every update to the repo requires devices to be flashed. There is no definitive, this file changes therefore the MCUs need to be flashed. The user has to kinda guess if they need to flash or not. Not entirely true; they are prompted to flash if there is a version mismatch, but that's a runtime warning. Perhaps we can identify this? The "Flash Beacon" button should only be active if there is a firmware_version mismatch. Otherwise, button should be deactivated and the text changed to "On Latest Version". I'll fix this up and send a patch for you to review. Fetch beacon code? I'm leaning towards let the user decide when to do updates via moonraker. Moonraker does a great job there already and is a known workflow. |
- Remove beacon option from transport dropdown; show static teal badge instead (beacon is USB-only, method set at registration) - Hide exclude_from_batch checkbox for beacon devices entirely (server enforces it; no need for disabled checkbox with explanation) - Disable "Flash Beacon" button when firmware is up to date, changing text to "On Latest Firmware" via new beaconUpToDate() helper
… layout - Filter beacon devices from serial discovery by resolving ttyACM* real paths against beacon by-id symlinks (ttyACM0 was leaking through) - Add _is_beacon_device() helper for consistent beacon path detection - Fix Beacon Probes discovery div: add overflow-hidden wrappers, truncate on name/id, wrap "+" button in flex-none container to match Serial/CAN
All 5 discovery sections (Serial, CAN, DFU, Linux, Beacon) now hide the "+" button when dev.managed is true, preventing duplicate fleet registration.
…id refactor, UI symlink fix)
|
Something else I've noticed - not sure if these changes caused this or if it is existing behaviour... But the firmware version being flashed is adding the My devices should be in "green".
# klippy/util.py
# 202:def get_git_version(from_file=True):
# method uses the following command to produce the version number
$ git describe --always --tags --long --dirty
v2026.03.00-0-g27902226c
# from klippy.log
$ grep -E "^Git|^Load|^Bran" ~/printer_data/logs/klippy.log|sort -u
Branch: (HEAD detached at v2026.03.00)
Git version: 'v2026.03.00-0-g27902226'
Loaded MCU 'beacon' 45 commands ( Beacon 2.1.0 / )
Loaded MCU 'ebb36' 133 commands (Kalico v2026.03.00-0-g27902226 / gcc: (15:12.2.rel1-1) 12.2.1 20221205 binutils: (2.40-2+18+b1) 2.40)
Loaded MCU 'mcu' 146 commands (Kalico v2026.03.00-0-g27902226 / gcc: (15:12.2.rel1-1) 12.2.1 20221205 binutils: (2.40-2+18+b1) 2.40)
Loaded MCU 'rpi' 134 commands (Kalico v2026.03.00-0-g27902226 / gcc: (Debian 12.2.0-14+deb12u1) 12.2.0 binutils: (GNU Binutils for Debian) 2.40)Note to self: http://v24:8321/firmware/version is the "System: " {"version":"v2026.03.00","commit":"27902226cab6","date":"2026-02-26 22:51:50 +0000"}and uses $ git describe --always --tags --dirty
v2026.03.00The async def get_klipper_version(self) -> Dict[str, str]:
"""Gets the current Klipper git version info."""
version_info: Dict[str, str] = {"version": "unknown", "commit": "unknown", "date": "unknown"}
try:
# Get the short commit hash
process: Process = await asyncio.create_subprocess_exec(
"git", "describe", "--always", "--tags", "--dirty",
cwd=self.klipper_dir, |
|
_is_beacon_device() is defined but isn't called anywhere; use or remove. The git fallback for beacon versions is quite complex, can this be simplified? maybe caching results? Too many API calls per refresh. Cache the beacon path at startup and move git lookups to a background timer? Then it won't have to do it on every refresh. |
- Remove _is_beacon_device() static method that was never called; beacon exclusion from serial discovery is already handled inline via glob patterns - Add _beacon_klipper_path cache to FlashManager instance to reduce redundant HTTP calls to Moonraker /server/config API - Add async refresh_beacon_path() method and modify get_beacon_klipper_path() to return cached value on subsequent calls
- Add module-level cache for beacon remote_version with 3600s TTL (firmware files change rarely) - Extract 3-step git fallback logic into _get_beacon_remote_version() helper function with clear sequencing: semver from commit message → closest tag → short hash - Call flash_mgr.refresh_beacon_path() at app startup to resolve path before any request hits, avoiding repeated HTTP calls to Moonraker - On subsequent version refreshes, check cache before running git subprocesses - Add test isolation fixture to reset cache between test runs
Add autouse fixture to reset beacon cache before each test to ensure test results are not affected by cache state from previous tests.
…JohnBaumb#24) - Add stripGitSuffix() helper that strips -<N>-g<hash>[-dirty] suffix from version strings, where N is any commit count (not just 0) - Allows versions like v0.13.0-572-g88a71c3c and v2026.03.00-0-g27902226c to match their release version and display green when on same semantic version regardless of commit distance - Applies to Klipper, Kalico, and other semantic-versioned firmware - Remove redundant !isBeacon guard from is_katapult check - Add comment to exclude_from_batch explaining beacon uses its own flash path
- Fix misleading comment on exclude_from_batch: beacon is always excluded, not user-configurable (UI hides toggle, backend enforces on every POST/update) - Add 5s timeout to git subprocesses in _get_beacon_remote_version() to prevent hanging on slow/NFS beacon_klipper repos during cache miss - Decouple remote_version population from live_version check: always populate remote_version for beacon devices regardless of whether live_version was found. This ensures version comparison works even if Moonraker provides live_version through unexpected means.
… background refresh Backend changes: - Enrich discovered devices with fleet names for managed devices - Display 'Beacon 2' instead of generic 'Beacon RevH' in discovery list UI/UX improvements: - Optimistic updates: '+' button disappears instantly when adding device - Optimistic updates: card removed instantly, '+' reappears instantly (no 10-20s wait) - Background refresh: device list updates in background without blocking UI - Beacon array initialization: add missing 'beacon: []' for Vue reactivity - Refresh discovery after add/remove: keeps both lists synchronized Result: Device add/remove now feels instant instead of waiting for network calls
Restore original UI state if add/remove fails: - Save fleet and devices state before optimistic updates - On error, restore saved state so user doesn't see stale UI - Show clearer error message with recovery instructions Fixes: 'failed to fetch' errors leaving UI in inconsistent state
SummaryFixed device add/remove UX to provide instant feedback instead of waiting 10-20s for network round-trips. Optimized ChangesBackend (main.py)
Frontend (ui/index.html)
ResultDevice add/remove operations now feel instant with optimistic UI updates. Network refreshes happen silently in the Before
After
Testing
There is a spurious "Error: Failed to fetch" which just popped up. Digging into it too. |
- Only call fetchFleet() and discoverDevices() if add/remove succeeds - Run refreshes in parallel with Promise.all to avoid double spinner - On error, restore UI state without fetching stale data Fixes: Stale data reappearing after failed operation, double querying spinner
|
New commit fixes the double spinner fetching in parallel and resolves the Failed fetch. |
Add immediate optimistic UI update to show new fleet card while awaiting server response, improving perceived responsiveness. Maintains rollback on error with restored fleet and device states.
|
Optimistic Fleet Card Addition Added optimistic UI update when adding devices to the fleet. The fleet card now appears immediately after clicking the "+" Implementation:
UX improvement: Instant visual feedback that the device was added, with graceful error recovery if the server request |
|
@bassco sorry for the delay, Post-flash status is wrong. generate_beacon() sets status to "ready" after a successful flash. But services get restarted right after, so the beacon goes back to running Klipper. Status should be "service" to match what check_device_status("beacon") returns. Flash sentinels missing. flash_beacon() and generate_beacon() never emit ">>> Flashing successful!" or ">>> Flashing failed!". Beacon flashes will finish without any visual feedback. stripGitSuffix works around the git describe --long format mismatch instead of fixing it in build_manager.py. some minor stuff, not really necessary but should probably be fixed in this pr: httpx.AsyncClient created per beacon device in get_fleet_versions(). Could lift the client outside the loop. addDeviceToFleet uses a per-category if/else for optimistic UI. removeDeviceFromFleet already uses the cleaner forEach pattern. Should match. |
…ttpx client - Set beacon post-flash status to 'service' (not 'ready') since services restart - Add '>>> Flashing Beacon failed!' sentinel on error - Reuse single httpx.AsyncClient for all beacon devices in get_fleet_versions() - Add cleanup to close beacon client after use - Add --long flag to git describe for consistent version format
|
Thanks for the review @JohnBaumb - summary of the work that was done for you to review. Changes Made:
|
|
Looks good! Good job @bassco! I'll merge it to dev |
Features: - Beacon probe discovery, registration, and flashing with remote version checking - Dynamic printer UI detection via /api/printer-ui endpoint with return link - Custom make command support per device, threaded through individual builds - Optimistic UI updates for fleet card add/remove operations Fixes: - Version comparison now uses full git version strings for accurate update detection - Helpful tooltip for devices not configured as MCUs in Klipper - Post-flash serial rescan for USB descriptor changes in batch operations - Fleet device ID preserved across reboot phase mutations - UI symlink handling in install script - Beacon hidden from serial/UART discovery Refactors: - Post-flash rescan moved to FlashManager - Beacon remote version cached at startup with TTL - Local-only gitignore entries moved to .git/info/exclude - Dead fleet_id fallback code removed PRs merged: #18 (batch flash rescan), #19 (beacon support), #21 (symlink fix), #22 (return link fix)










Summary
beacon_klipper/update_firmware.py, version reporting via Moonrakerupdate_manager, and batch exclusionmethod: "beacon"withprofile: null(no Klipper build artifact) and default toexclude_from_batch: truepathfield — parsesmoonraker.confdirectlyFiles changed:
flash_manager.py(+93),main.py(+74/-12),ui/index.html(+99/-44),tests/test_beacon.py(+286 new)Live Testing Evidence
Tested on two physical printers against a clean
mainbaseline (be1d71e).Test Machines
v24~/beacon_klipper(v2.0.0-26)/home/pi/KlipperFleetv02~/klipperfleetStep 0 — Clean Baseline (main, be1d71e)
Both machines on
main, fleet.json cleaned of any prior beacon entries, services restarted.Step 1 — Before State (main, no beacon code)
v24 Discovery (main) — no
"beacon"key, beacon USB shows as unmanaged serial device:{ "serial": [{"id": "/dev/serial/by-id/usb-Beacon_Beacon_RevH_698D6C3C...-if00", "managed": false}], "can": [{"id": "c20262880b1b", "name": "mcu", "managed": true}, {"id": "5f999a28d081", "name": "mcu ebb36", "managed": true}], "dfu": [], "linux": [{"id": "linux_process", "managed": true}] }v24 Fleet (main) — 3 devices, no beacon:
v02 Discovery (main) — no
"beacon"key:{ "serial": [{"id": "...stm32g0b1xx...", "managed": true}, {"id": "...stm32f103xe...", "managed": true}], "can": [], "dfu": [], "linux": [{"id": "linux_process", "managed": true}] }Both printers: Klipper
ready, KlipperFleetactive (running), journals clean.Step 2 — Deploy
add-beacon-flashingStep 3 — Discovery (PASS)
v24 — new
"beacon"category appears with RevH device:{ "serial": [{"id": "...Beacon_RevH_698D6C3C...", "managed": false}], "can": [{"id": "c20262880b1b", "managed": true}, {"id": "5f999a28d081", "managed": true}], "dfu": [], "linux": [{"id": "linux_process", "managed": true}], "beacon": [{"id": "/dev/serial/by-id/usb-Beacon_Beacon_RevH_698D6C3C5154354D38202020FF0A1227-if00", "name": "Beacon RevH", "revision": "RevH", "serial": "698D6C3C5154354D38202020FF0A1227", "mode": "service", "managed": false}] }v02 —
"beacon": [](empty array, no hardware):{"serial": [...], "can": [], "dfu": [], "linux": [...], "beacon": []}Step 4 — Registration (PASS)
After registration, discovery shows
"managed": truefor both serial and beacon categories.Fleet entry:
{"name": "Beacon RevH", "id": "/dev/serial/by-id/usb-Beacon_Beacon_RevH_...", "profile": null, "method": "beacon", "exclude_from_batch": true, "status": "service"}Step 5 — Versions (PASS)
{"/dev/serial/by-id/usb-Beacon_Beacon_RevH_...": {"live_version": "v2.0.0-26", "flashed_version": null}}live_versionpulled from Moonrakerupdate_manager— confirmedv2.0.0-26.Step 6 — Flash (PASS)
curl -X POST http://localhost:8321/flash -d '{"device_id": "/dev/serial/by-id/usb-Beacon_Beacon_RevH_...", "method": "beacon"}'Output:
Post-flash: Klipper
ready, beacon serial device present.Note: Initial attempt failed because Moonraker's API does not return a
pathfield forgit_repoentries. Fixed in4f933ebby adding a fallback that parsesmoonraker.confdirectly. Re-tested successfully.Step 7 — Batch Exclusion (PASS)
Batch summary:
All 3 Klipper MCUs built and flashed, beacon correctly excluded.
Step 8 — After State (PASS)
readyreadyactive (running)active (running)Unit Tests
14 new beacon-specific tests in
tests/test_beacon.pycovering discovery, registration, flash, version lookup, batch exclusion, and edge cases.Test plan
main— no beacon in fleet or discoverymethod: "beacon",exclude_from_batch: truelive_versionpulled from Moonraker update_managerupdate_firmware.pyexecuted, services restartedmoonraker.confwhen API omitspathEXCLUDEDin batch summary"beacon": []