Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 21 additions & 6 deletions docs/api/rest.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2837,8 +2837,9 @@ Requires: ``systemd_introspection`` plugin and ``libsystemd``.
Container Introspection (x-medkit-container)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Requires: ``container_introspection`` plugin. Only supports cgroup v2
(Ubuntu 22.04+, Fedora 31+).
Requires: ``container_introspection`` plugin. Supports the unified cgroup
hierarchy (v2), the legacy one (v1), and hybrid hosts, under both the
``host`` and ``private`` cgroup namespace modes.

``GET /api/v1/apps/{id}/x-medkit-container``
Get container information for the app's process.
Expand All @@ -2851,15 +2852,29 @@ Requires: ``container_introspection`` plugin. Only supports cgroup v2
"container_id": "a1b2c3d4e5f6...",
"runtime": "docker",
"memory_limit_bytes": 1073741824,
"memory_limit_state": "limited",
"cpu_quota_us": 100000,
"cpu_period_us": 100000
"cpu_period_us": 100000,
"cpu_quota_state": "limited"
}

Fields ``memory_limit_bytes``, ``cpu_quota_us``, and ``cpu_period_us`` are only present
when the container has resource limits configured.
Fields ``memory_limit_bytes`` and ``cpu_quota_us`` are present only when a limit is in
force. ``cpu_period_us`` is present whenever the CPU limit was read at all, including
when the quota is unlimited. ``memory_limit_state`` and ``cpu_quota_state`` are
always present and carry one of ``limited``, ``unlimited``, ``unreadable`` or
``unavailable``, so a client can tell an unconstrained container from one whose limit
files could not be read. ``cpu_quota_state`` covers the quota and its period together.

``cpu_quota_us`` is the CFS bandwidth limit. It does not reflect the set of CPUs the
container is pinned to (``--cpuset-cpus``), which is visible only through
``sched_getaffinity()``; the effective CPU budget needs both.

``container_id`` is empty when the cgroup namespace hides it (``--cgroupns=private``
reports the namespace root as the path). The container is still recognised from the
markers its runtime leaves behind, and the limits are still reported.

- **404:** Process not found or not running in a container
- **503:** Failed to read cgroup information
- **503:** The cgroup of the process could not be determined at all

``GET /api/v1/components/{id}/x-medkit-container``
Aggregate container info for all apps in the component. Containers are
Expand Down
86 changes: 73 additions & 13 deletions docs/tutorials/linux-introspection.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@ Apps and Components.
unit properties (ActiveState, SubState, NRestarts, WatchdogUSec) via sd-bus. Requires
``libsystemd``.
- **container** - detects containerization via cgroup path analysis. Supports Docker,
podman, and containerd. Reads cgroup v2 resource limits (``memory.max``, ``cpu.max``).
podman, and containerd. Reads resource limits from the unified hierarchy
(``memory.max``, ``cpu.max``) and from the legacy one (``memory.limit_in_bytes``,
``cpu.cfs_quota_us``, ``cpu.cfs_period_us``).

Each plugin maintains its own PID cache that maps ROS 2 node fully-qualified names to
Linux PIDs by scanning ``/proc``. The cache refreshes on each discovery cycle and on
Expand All @@ -24,8 +26,10 @@ Requirements
- **procfs**: Linux only (reads ``/proc`` filesystem). No extra dependencies.
- **systemd**: requires ``libsystemd-dev`` at build time, systemd at runtime. Skipped
automatically if ``libsystemd`` is not found during the build.
- **container**: requires cgroup v2, which is the default on modern kernels (Ubuntu 22.04+,
Fedora 31+).
- **container**: Linux only. Works on the unified cgroup hierarchy (v2, the default on
Ubuntu 22.04+ and Fedora 31+), on the legacy hierarchy (v1), and on hybrid hosts that
mount both. Both cgroup namespace modes (``--cgroupns=host`` and ``--cgroupns=private``)
are handled.

Building
--------
Expand Down Expand Up @@ -217,15 +221,27 @@ Returns container metadata for a node running inside a container:
"container_id": "a1b2c3d4e5f6...",
"runtime": "docker",
"memory_limit_bytes": 536870912,
"memory_limit_state": "limited",
"cpu_quota_us": 100000,
"cpu_period_us": 100000
"cpu_period_us": 100000,
"cpu_quota_state": "limited"
}

.. note::

The ``memory_limit_bytes``, ``cpu_quota_us``, and ``cpu_period_us`` fields are only
present when cgroup v2 resource limits are set. If no limits are configured, these
fields are omitted from the response.
present when a limit was actually read. ``memory_limit_state`` and ``cpu_quota_state``
are always present and say why a number is missing:

- ``limited`` - a limit is in force, and the numeric field carries it
- ``unlimited`` - the container may use the whole machine
- ``unreadable`` - a limit file was found but its contents could not be parsed
- ``unavailable`` - no limit file exists in any supported cgroup layout

The distinction matters: an unreadable limit file reported as "no limit" would look
exactly like an unconstrained container. The CPU limit is the quota together with its
period, so ``cpu_quota_state`` covers both and ``cpu_period_us`` has no state of its
own; the period is reported even when the quota is ``unlimited``.

**GET /components/{id}/x-medkit-container**

Expand All @@ -243,8 +259,10 @@ Returns aggregated container info for all child Apps, deduplicated by container
"container_id": "a1b2c3d4e5f6...",
"runtime": "docker",
"memory_limit_bytes": 536870912,
"memory_limit_state": "limited",
"cpu_quota_us": 100000,
"cpu_period_us": 100000,
"cpu_quota_state": "limited",
"node_ids": ["temp_sensor", "rpm_sensor"]
}
]
Expand Down Expand Up @@ -275,8 +293,11 @@ validation errors (404 for unknown entities) are handled automatically by
| 404 | ``x-medkit-not-containerized`` | Node's process is not running inside a |
| | | container (no container cgroup path detected).|
+-----+---------------------------------------+-----------------------------------------------+
| 503 | ``x-medkit-cgroup-read-failed`` | Failed to read cgroup info for the container. |
| | | Check cgroup v2 filesystem access. |
| 503 | ``x-medkit-cgroup-read-failed`` | The cgroup of the process could not be |
| | | determined at all. Check access to |
| | | ``/proc/{pid}/cgroup``. A limit that could |
| | | not be read is reported as a state on a 200, |
| | | not as this error. |
+-----+---------------------------------------+-----------------------------------------------+

.. note::
Expand Down Expand Up @@ -369,17 +390,28 @@ Without system bus access, the systemd plugin will return 503 errors for all que
Container detection
~~~~~~~~~~~~~~~~~~~

The container plugin relies on cgroup v2 path analysis. To verify your system uses
cgroup v2:
The container plugin relies on cgroup path analysis. To see which hierarchy your
system mounts:

.. code-block:: bash

mount | grep cgroup2
# Should show: cgroup2 on /sys/fs/cgroup type cgroup2 (...)
mount | grep cgroup
# unified (v2): cgroup2 on /sys/fs/cgroup type cgroup2 (...)
# legacy (v1): cgroup on /sys/fs/cgroup/memory type cgroup (...)

# Or check a process's cgroup path
cat /proc/self/cgroup
# cgroup v2 output: "0::/user.slice/..."
# unified (v2): "0::/user.slice/..."
# legacy (v1): "12:memory:/docker/<id>" - one line per hierarchy

Both are supported, as is the hybrid layout that mounts the legacy controllers at the
top level and the unified hierarchy beside them at ``/sys/fs/cgroup/unified``.

The reported path is relative to the cgroup namespace of the container, so where the
limit files sit depends on the namespace mode: ``--cgroupns=private`` mounts the
container's own cgroup at the mount point, while ``--cgroupns=host`` exposes the whole
hierarchy and the reported path leads into it. The plugin looks in both places and does
not need to be told which mode is in use.

Supported container runtimes and their cgroup path patterns:

Expand All @@ -389,3 +421,31 @@ Supported container runtimes and their cgroup path patterns:

If your runtime uses a different cgroup path format, the plugin will not detect the
container. The ``runtime`` field in the response indicates the detected runtime.

Under ``--cgroupns=private`` the reported cgroup path is the namespace root and carries
no container ID at all. The plugin then falls back to the markers a runtime leaves
behind, read through the inspected process's own root (``/proc/<pid>/root``) so that a
containerized gateway does not answer for processes outside it:

- ``/.dockerenv`` (Docker) or ``/run/.containerenv`` (Podman), which also name the runtime
- ``/run/systemd/container`` (systemd-nspawn and others)
- an overlay filesystem mounted as the process's root, but only together with a cgroup
path of ``/``. An overlay root on its own proves nothing, because whole distributions
boot that way

The limits are still reported, with ``container_id`` empty. A process that matches none of
these gets ``404 x-medkit-not-containerized``.

A runtime that leaves no marker and does not use an overlay root - containerd or CRI-O on
a btrfs or ZFS snapshotter, for example - is not detected under a private namespace, and
its apps are reported as non-containerized.

CPU limits and cpusets
~~~~~~~~~~~~~~~~~~~~~~

``cpu_quota_us`` and ``cpu_period_us`` describe the CFS bandwidth limit
(``--cpus``/``--cpu-quota``) and nothing else. A container pinned to a subset of the
machine with ``--cpuset-cpus`` has no quota at all, so it is reported as
``"cpu_quota_state": "unlimited"`` even though it cannot use every core. The cpuset is
visible only through ``sched_getaffinity()``; a caller that wants the effective CPU
budget has to combine the two.
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,29 @@ Maps ROS 2 nodes to OS processes and reports CPU, memory, systemd unit status, a
## Key Components

- **ProcReader** - Parses procfs files for process metrics with configurable proc root
- **CgroupReader** - Reads cgroup v2 hierarchy for container/service context
- **CgroupReader** - Reads the unified (v2), legacy (v1) and hybrid cgroup hierarchies for
container/service context, under both cgroup namespace modes. Each limit is reported
together with the outcome of reading it, so "unlimited" and "could not be read" stay
distinguishable. A container whose ID the cgroup namespace hides is recognised from the
markers its runtime leaves behind
- **SystemdUtils** - Queries systemd via D-Bus for service metadata
- **PidCache** - TTL-based cache mapping ROS 2 node names to Linux PIDs

## Configuration

Configure via `gateway_params.yaml` plugin parameters:

Each plugin is configured on its own. `proc_root` is the prefix that contains `proc/` and
`sys/`, so the default `/` is what a normal deployment wants; pointing it at `/proc` makes
the plugin look for `/proc/proc/<pid>`.

```yaml
plugins: ["linux_introspection"]
plugins.linux_introspection.path: "/path/to/libros2_medkit_linux_introspection.so"
plugins.linux_introspection.pid_cache_ttl_sec: 30
plugins.linux_introspection.proc_root: "/proc"
plugins: ["procfs", "systemd", "container"]
plugins.procfs.path: "/path/to/libprocfs_introspection.so"
plugins.systemd.path: "/path/to/libsystemd_introspection.so"
plugins.container.path: "/path/to/libcontainer_introspection.so"
plugins.container.pid_cache_ttl_seconds: 30
plugins.container.proc_root: "/"
```

## Documentation
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,13 @@ Plugins

- Registers ``x-medkit-container`` capability on Apps and Components
- Reads ``/proc/{pid}/cgroup`` to extract container ID and runtime (Docker, containerd, Podman)
- Handles the unified (v2), legacy (v1) and hybrid hierarchies, and looks for the limit
files both at the bare mount point (``cgroupns=private``) and at the mount point joined
with the reported path (``cgroupns=host``)
- Reports each limit with a ``LimitState`` (``limited``, ``unlimited``, ``unreadable``,
``unavailable``) so a failed read cannot be mistaken for an unconstrained container
- Recognises a container even when the cgroup namespace hides its ID, from
``/.dockerenv``, ``/run/.containerenv`` or an overlay root filesystem
- Component-level endpoint aggregates containers, deduplicating by container ID
- Returns 404 for entities not running in a container (not an error - just bare-metal)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,19 +22,116 @@

namespace ros2_medkit_linux_introspection {

/// Outcome of reading one cgroup limit.
///
/// "No limit is in force" and "the limit could not be read" are different facts
/// about a container: only the first one means the container may use the whole
/// machine, so they are reported as different states rather than both as an
/// absent value.
enum class LimitState {
kUnavailable, ///< No file for this limit exists in any supported cgroup layout
kUnreadable, ///< A file exists but its contents could not be parsed
kUnlimited, ///< The cgroup reports no limit ("max" on v2, -1 on v1)
kLimited, ///< A numeric limit is in force
};

/// Wire form of a LimitState: "unavailable", "unreadable", "unlimited" or "limited".
std::string limit_state_to_string(LimitState state);

/// One cgroup limit together with the outcome of reading it.
///
/// The value is present if and only if the state is LimitState::kLimited, an
/// invariant the named constructors are the only way to establish.
template <typename T>
class CgroupLimit {
public:
CgroupLimit() = default;

static CgroupLimit limited(T value) {
CgroupLimit limit;
limit.state_ = LimitState::kLimited;
limit.value_ = value;
return limit;
}

static CgroupLimit unlimited() {
return CgroupLimit(LimitState::kUnlimited);
}

static CgroupLimit unreadable() {
return CgroupLimit(LimitState::kUnreadable);
}

static CgroupLimit unavailable() {
return CgroupLimit(LimitState::kUnavailable);
}

LimitState state() const {
return state_;
}

const std::optional<T> & value() const {
return value_;
}

private:
explicit CgroupLimit(LimitState state) : state_(state) {
}

LimitState state_{LimitState::kUnavailable};
std::optional<T> value_;
};

struct CgroupInfo {
std::string cgroup_path;
std::string container_id; // 64-char hex from cgroup path, or empty
std::string container_runtime; // "docker", "podman", "containerd", or empty
std::optional<uint64_t> memory_limit_bytes;
std::optional<int64_t> cpu_quota_us;

/// Whether the process runs inside a container.
///
/// Not the same as having a container id: under ``cgroupns=private`` the
/// reported cgroup path is the namespace root and carries no id, so the path
/// alone cannot answer the question and a runtime marker does instead. A
/// container is therefore reported with an empty ``container_id``.
bool containerized{false};

/// Memory limit in bytes, from ``memory.max`` (v2) or ``memory.limit_in_bytes`` (v1).
CgroupLimit<uint64_t> memory_limit;

/// CPU bandwidth quota in microseconds of CPU time per period, from ``cpu.max``
/// (v2) or ``cpu.cfs_quota_us`` (v1).
///
/// This is the CFS bandwidth limit and nothing else. It does not reflect the
/// set of CPUs a container is pinned to (``--cpuset-cpus``); that is visible
/// only through sched_getaffinity(). A caller that wants the effective CPU
/// budget needs both numbers.
CgroupLimit<int64_t> cpu_quota;

/// Period the quota is measured over, in microseconds. It is read together
/// with the quota and carries no state of its own, so it is set whenever the
/// CPU limit was read at all - including when the quota itself is unlimited.
std::optional<int64_t> cpu_period_us;
};

/// Detect if PID runs inside a container (based on cgroup path)
/// Detect if PID runs inside a container.
///
/// Prefers the container id carried by the cgroup path. When the cgroup
/// namespace hides that id, falls back to the markers a runtime leaves behind:
/// ``/.dockerenv``, ``/run/.containerenv``, or an overlay filesystem mounted as
/// the process's root.
bool is_containerized(pid_t pid, const std::string & root = "/");

/// Read cgroup info for a PID
/// Read cgroup info for a PID.
///
/// Handles the unified (v2) and legacy (v1) hierarchies as well as the hybrid
/// layout that mounts both, and both cgroup namespace modes: the interface
/// files are looked for at the bare mount point, where ``cgroupns=private``
/// puts the process's own cgroup, and at the mount point joined with the
/// reported path, where ``cgroupns=host`` puts it.
///
/// Fails only when the cgroup of the process cannot be determined at all. A
/// limit that could not be read is reported through its LimitState, not as a
/// missing limit.
tl::expected<CgroupInfo, std::string> read_cgroup_info(pid_t pid, const std::string & root = "/");

/// Extract container ID from a cgroup path string
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,17 +21,27 @@
namespace ros2_medkit_linux_introspection {

/// Convert CgroupInfo to JSON for the container plugin HTTP response.
/// Optional fields (memory_limit_bytes, cpu_quota_us, cpu_period_us) are
/// omitted from the JSON when not set.
///
/// The numeric fields (memory_limit_bytes, cpu_quota_us, cpu_period_us) are
/// present only when a value was read, so a client that reads no limit still
/// has to know why. That is what memory_limit_state and cpu_quota_state carry:
/// "limited", "unlimited", "unreadable" or "unavailable", always present.
///
/// The CPU limit is the quota and its period together, so cpu_quota_state
/// covers both and cpu_period_us has no state of its own.
inline nlohmann::json cgroup_info_to_json(const CgroupInfo & info) {
nlohmann::json j;
j["container_id"] = info.container_id;
j["runtime"] = info.container_runtime;
if (info.memory_limit_bytes) {
j["memory_limit_bytes"] = *info.memory_limit_bytes;

j["memory_limit_state"] = limit_state_to_string(info.memory_limit.state());
if (info.memory_limit.value()) {
j["memory_limit_bytes"] = *info.memory_limit.value();
}
if (info.cpu_quota_us) {
j["cpu_quota_us"] = *info.cpu_quota_us;

j["cpu_quota_state"] = limit_state_to_string(info.cpu_quota.state());
if (info.cpu_quota.value()) {
j["cpu_quota_us"] = *info.cpu_quota.value();
}
if (info.cpu_period_us) {
j["cpu_period_us"] = *info.cpu_period_us;
Expand Down
Loading
Loading