What happened?
We're running plc4j (S7 driver) inside a long-lived industrial gateway process on constrained embedded Linux hardware (512MB total RAM, JVM heap capped at -Xmx128m). When the remote S7 PLC becomes unreachable for an extended period (in our case ~3 hours, confirmed the PLC and network path were healthy — a purely transient outage), our application repeatedly calls PlcDriverManager.getDefault().getConnectionManager().getConnection("s7://...") on a backoff schedule (5-600s) to retry the connection. Each attempt times out after 5000ms without ever returning a PlcConnection instance to the caller.
Over the ~3 hour outage (roughly 17-18 failed connect attempts), the process's resident memory (RSS) climbed from its normal baseline to ~78% of the device's total RAM, at which point the Linux kernel OOM-killer killed the JVM (confirmed via uptime that the device itself never rebooted — only our process died and was restarted by our process supervisor). No Java OutOfMemoryError was ever logged, which is consistent with the growth happening outside the JVM heap (native/off-heap), invisible to heap-based OOM detection.
A thread dump taken shortly after the process restarted (before any new failures had occurred) already showed threads named plc4x-tm-thread-*, plc4x-app-thread-*, plc4x-s7ha-thread-*, plc4x-evt-processor-*, plc4x-evt-dispatcher-*, and nioEventLoopGroup-*.
This looks like the "Orphaned Resources on Failed Connection" scenario discussed (but not fixed) in #2544: since connect() fails before ever returning a connection object, close() is never invoked, and whatever EmbeddedChannel / HashedWheelTimer / protocol thread pool got allocated during that attempt is never released. #2544 was closed as "not planned" after the original reporter retracted their thread-leak claim for the normal-disconnect path, but the failed-connect orphaned-resource scenario was confirmed by maintainers on that same thread and, as far as we can tell, was never itself addressed. We're filing separately since our field data (RSS growth over time + thread dump + confirmed OOM-kill) is different evidence than what was available on #2544, and might help re-evaluate whether this specific path needs a fix.
Steps to reproduce:
- Configure an S7 connection (
s7://<host>:102?remote-rack=0&remote-slot=1) to an address that is reachable at the TCP level but never completes (or is simply firewalled/unreachable so getConnection() always times out).
- Repeatedly call
getConnectionManager().getConnection(...) on a timer (every 10-60s) for an extended period (hours), always against the same unreachable address.
- Monitor process RSS and
/proc/<pid>/task/*/comm thread names over time.
Happy to provide our thread-dump and RSS-over-time data if useful.
Version
0.13.1
Programming Languages
Protocols
What happened?
We're running plc4j (S7 driver) inside a long-lived industrial gateway process on constrained embedded Linux hardware (512MB total RAM, JVM heap capped at
-Xmx128m). When the remote S7 PLC becomes unreachable for an extended period (in our case ~3 hours, confirmed the PLC and network path were healthy — a purely transient outage), our application repeatedly callsPlcDriverManager.getDefault().getConnectionManager().getConnection("s7://...")on a backoff schedule (5-600s) to retry the connection. Each attempt times out after 5000ms without ever returning aPlcConnectioninstance to the caller.Over the ~3 hour outage (roughly 17-18 failed connect attempts), the process's resident memory (RSS) climbed from its normal baseline to ~78% of the device's total RAM, at which point the Linux kernel OOM-killer killed the JVM (confirmed via
uptimethat the device itself never rebooted — only our process died and was restarted by our process supervisor). No JavaOutOfMemoryErrorwas ever logged, which is consistent with the growth happening outside the JVM heap (native/off-heap), invisible to heap-based OOM detection.A thread dump taken shortly after the process restarted (before any new failures had occurred) already showed threads named
plc4x-tm-thread-*,plc4x-app-thread-*,plc4x-s7ha-thread-*,plc4x-evt-processor-*,plc4x-evt-dispatcher-*, andnioEventLoopGroup-*.This looks like the "Orphaned Resources on Failed Connection" scenario discussed (but not fixed) in #2544: since
connect()fails before ever returning a connection object,close()is never invoked, and whateverEmbeddedChannel/HashedWheelTimer/ protocol thread pool got allocated during that attempt is never released. #2544 was closed as "not planned" after the original reporter retracted their thread-leak claim for the normal-disconnect path, but the failed-connect orphaned-resource scenario was confirmed by maintainers on that same thread and, as far as we can tell, was never itself addressed. We're filing separately since our field data (RSS growth over time + thread dump + confirmed OOM-kill) is different evidence than what was available on #2544, and might help re-evaluate whether this specific path needs a fix.Steps to reproduce:
s7://<host>:102?remote-rack=0&remote-slot=1) to an address that is reachable at the TCP level but never completes (or is simply firewalled/unreachable sogetConnection()always times out).getConnectionManager().getConnection(...)on a timer (every 10-60s) for an extended period (hours), always against the same unreachable address./proc/<pid>/task/*/commthread names over time.Happy to provide our thread-dump and RSS-over-time data if useful.
Version
0.13.1
Programming Languages
Protocols