Implement pipeline stats/analytics#572
Draft
joaoantoniocardoso wants to merge 45 commits intomavlink:masterfrom
Draft
Implement pipeline stats/analytics#572joaoantoniocardoso wants to merge 45 commits intomavlink:masterfrom
joaoantoniocardoso wants to merge 45 commits intomavlink:masterfrom
Conversation
joaoantoniocardoso
force-pushed
the
stats
branch
3 times, most recently
from
2955533 to
1db0b93
Compare
Add lower_thread_priority() (nice 10) and lower_to_background_priority() (SCHED_OTHER, nice 19) to allow non-GStreamer threads and auxiliary pipelines to yield CPU to realtime video stream threads. Adds libc as a Linux-only dependency.
… runner Refactor PipelineRunner::try_new into try_new/try_new_inner with a realtime_threads parameter. Add try_new_background() that sets GStreamer streaming threads to SCHED_OTHER with nice 19 instead of SCHED_RR realtime, for auxiliary pipelines (e.g. thumbnail generation) that must not preempt video stream threads.
Replace the Rust `image` crate-based thumbnail pipeline with a fully GStreamer-native approach using videoscale, videoconvert, and jpegenc. This eliminates the need to decode to raw RGB, copy frame data out of GStreamer, and re-encode in userspace. Key changes: - Decode → videoscale → capsfilter → videoconvert → jpegenc → appsink replaces decode → appsink → image::thumbnail → image::write_to - JPEG quality and target resolution are updated dynamically per request by modifying jpegenc and capsfilter properties at runtime - Software decoders are limited to max-threads=1 to reduce CPU usage - Pipeline runs at background priority via try_new_background() - Optional pad blocker (MCM_THUMBNAIL_PAD_BLOCKER=1) pauses data flow between snapshots to further reduce idle CPU - Removes the `image` crate dependency
…rease timeout for thumbnail capture Sending an upstream ForceKeyUnit event ensures the encoder produces a keyframe immediately rather than waiting for the next natural one (which can be many seconds away with default x264enc settings). The receive timeout is raised from 2 s to 5 s to accommodate the round-trip.
Set nice=10 via setpriority() on the main thread, all Tokio worker threads, the TURN server runtime threads, and the thumbnail generation thread. This ensures GStreamer pipeline threads (which run at SCHED_RR realtime when CAP_SYS_NICE is available) are always preferred by the OS scheduler, reducing latency spikes on resource-constrained systems. Also replaces the #[tokio::main] attribute macro with an explicit runtime builder so that on_thread_start can be set for worker threads.
…ndant depay/repay Previously the RTSP sink received RTP-encapsulated data from the rtp_tee, requiring a depay/repay cycle (e.g. rtph264depay ! rtph264pay) inside the RTSP server pipeline. This was wasteful and introduced unnecessary CPU overhead. Route the RTSP sink through the video_tee instead, feeding raw encoded video (e.g. H264) directly to the RTP payloader. Update the RTSP server to detect the stream format from video caps structure names instead of RTP encoding-name fields.
Replace the shared-memory IPC bridge (shmsink → socket → shmsrc) with an in-process appsink/appsrc pair. This eliminates filesystem socket overhead and enables direct buffer passing between the source pipeline and the RTSP server's internal pipeline. The appsink callback rebases PTS/DTS so each new RTSP client connection sees timestamps starting from 0, preventing the server's queue from holding stale buffers. A media-configure callback on the RTSP factory configures the appsrc caps and resets the PTS offset on each new connection.
During WebRTC connection setup, webrtcbin cannot consume data until the DTLS handshake completes (~200-300ms). The leaky queue accumulates stale RTP packets during this period, and because input rate equals output rate in steady state, the queue level never drains — adding permanent latency to every WebRTC session. After the connection state transitions to Connected, reduce the queue's max-size-time from the initial 1s backstop to one frame interval. The leaky=downstream policy immediately flushes the stale handshake backlog. Once the queue fully drains (underrun signal), dynamic sizing kicks in: each overrun grows max-size-time by one frame interval (capped at 10x), letting the queue self-tune to the minimum size that avoids frame drops under the current CPU load.
Even with fec-type=None on the transceiver, webrtcbin still creates rtpulpfecenc and rtpredenc elements that copy every RTP buffer for no benefit. Once the peer connection is established, surgically unlink and remove them. Also strip FEC/RED payload types from the SDP offer so the peer never negotiates them.
The wrap_fn trace and actix_web::middleware::Logger were redundant with TracingLogger and added measurable per-request overhead.
The DOT pipeline graph endpoint is a debugging tool that should not be active in production. It now returns 404 unless --enable-dot is passed.
Add a StreamStatusState enum (Running / Idle / Stopped) to the API response and a disable_lazy flag to ExtendedConfiguration so users can opt out of lazy pipeline suspension per stream.
Introduce per-stream consumer_count / idle tracking and a watcher loop that suspends pipelines to GStreamer NULL after a 5-second grace period with zero consumers, then resumes on demand. RTSP flow is managed through a new RtspFlowHandle that wraps a valve element: the valve drops buffers when no RTSP clients are connected and opens on the first client_connected callback, while updating the shared consumer_count. The old logic that set pipelines to NULL when no tee source pads were linked is removed in favour of the new watcher-driven lifecycle. The pipeline runner watchdog now skips position checks while the pipeline is not in Playing state.
Wake idle pipelines before WebRTC session setup and thumbnail capture, waiting for data flow (query_position advancing) rather than just the Playing state to ensure sinks can negotiate successfully. Track consumer_count in add_session / remove_session and only decrement on the first successful sink removal to avoid usize underflow from duplicate cleanup calls. Expose StreamStatusState (Running / Idle / Stopped) in the streams information API.
…sconnect Maintain an ActiveSessions list per WebSocket connection so that when a client disconnects without sending EndSession (e.g. browser navigated away), all its orphaned sessions are cleaned up and consumer_count is correctly decremented. Also filter stopped streams from the WebRTC producer list so idle pipelines remain discoverable.
The test stream had no extended_configuration, defaulting to lazy idle enabled. This caused the pipeline to suspend during the 5-second prepare wait and then resume on the first session cycle, creating a fragile window where WebRTC negotiation could fail for one session. Since this test focuses on thread leak detection, not idle lifecycle, set disable_lazy: true to keep the pipeline continuously running.
joaoantoniocardoso
force-pushed
the
stats
branch
from
1db0b93 to
308a3d6
Compare
joaoantoniocardoso
force-pushed
the
stats
branch
from
308a3d6 to
f24d3dd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To be merged after #564.
For an example use-case, see https://github.com/joaoantoniocardoso/mcm-test-harness
Impact: ~3% CPU at 1Hz probing at full level.
Frontend will be implemented in another PR, after enough validation.