Skip to content

[action] [PR:334] Ignore invalid output from docker stats command#341

Open
mssonicbld wants to merge 1 commit intosonic-net:202511from
mssonicbld:cherry/202511/334
Open

[action] [PR:334] Ignore invalid output from docker stats command#341
mssonicbld wants to merge 1 commit intosonic-net:202511from
mssonicbld:cherry/202511/334

Conversation

@mssonicbld
Copy link

Why I did this

We found some error in syslog:

2025 Dec 10 22:42:06.221428 sonic ERR memory_threshold_check: Failed to parse memory usage for "{'CPU%': '--', 'MEM%': '--', 'MEM_BYTES': '0', 'MEM_LIMIT_BYTES': '0', 'NAME': '--', 'PIDS': '--'}": could not convert string to float: '--'
2025 Dec 10 22:42:06.221527 sonic ERR memory_threshold_check: Failure occurred could not convert string to float: '--'

The error is statistical. The flow is as below:

  1. procdockerstatsd calls "docker stats" command periodically, parse the output and store to STATE DB
  2. memory_threshold_check which is called by monit service, handles the data in STATE DB and check the memory usage

After reviewing sonic code and docker code, I found that sonic would never generate a data with NAME=--; while docker might do so. Docker might generate such invalid output in a flow like this:

  1. user issues "docker stats -a --no-stream --format json" command
  2. docker CLI sends a command to docker engine for all existing containers: container a, b, c and so on
    3. other user removes container a
  3. docker engine starts to handle the command and find container a is no longer there, so it returns empty stats data
  4. docker CLI finds the stats data for docker a in empty and fill the data with "--"

For detailed docker code, please check:
https://github.com/docker/cli/blob/93fa57bbcd08f2f5be7f6cf22f4273a2b5a49e71/cli/command/container/formatter_stats.go#L25
https://github.com/docker/cli/blob/93fa57bbcd08f2f5be7f6cf22f4273a2b5a49e71/cli/command/container/formatter_stats.go#L169

This PR is to fix the issue

How I did this

procdockerstatsd should check the command output and ignore invalid value:

  1. if the name is empty of "--", ignore the output
  2. log a warning message to syslog

How I test this

Manual test

**Why I did this**

We found some error in syslog:

```
2025 Dec 10 22:42:06.221428 sonic ERR memory_threshold_check: Failed to parse memory usage for "{'CPU%': '--', 'MEM%': '--', 'MEM_BYTES': '0', 'MEM_LIMIT_BYTES': '0', 'NAME': '--', 'PIDS': '--'}": could not convert string to float: '--'
2025 Dec 10 22:42:06.221527 sonic ERR memory_threshold_check: Failure occurred could not convert string to float: '--'
```

The error is statistical. The flow is as below:

1. procdockerstatsd calls "docker stats" command periodically, parse the output and store to STATE DB
2. memory_threshold_check which is called by monit service, handles the data in STATE DB and check the memory usage

After reviewing sonic code and docker code, I found that sonic would never generate a data with `NAME`=`--`; while docker might do so. Docker might generate such invalid output in a flow like this:

1. user issues "docker stats -a --no-stream --format json" command
2. docker CLI sends a command to docker engine for all existing containers: container a, b, c and so on
**3. other user removes container a**
4. docker engine starts to handle the command and find container a is no longer there, so it returns empty stats data
5. docker CLI finds the stats data for docker a in empty and fill the data with "--"

For detailed docker code, please check:
https://github.com/docker/cli/blob/93fa57bbcd08f2f5be7f6cf22f4273a2b5a49e71/cli/command/container/formatter_stats.go#L25
https://github.com/docker/cli/blob/93fa57bbcd08f2f5be7f6cf22f4273a2b5a49e71/cli/command/container/formatter_stats.go#L169

This PR is to fix the issue

**How I did this**

procdockerstatsd should check the command output and ignore invalid value:

1. if the name is empty of "--", ignore the output
2. log a warning message to syslog

**How I test this**

Manual test
@mssonicbld
Copy link
Author

Original PR: #334

@mssonicbld
Copy link
Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@azure-pipelines
Copy link

Commenter does not have sufficient privileges for PR 341 in repo sonic-net/sonic-host-services

@prsunny
Copy link
Contributor

prsunny commented Mar 4, 2026

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants