Permanent worker-thread hang on node update with micronode-list field

[kaiminus123]

Hello everyone

Summary

Updating a node that contains a micronode-list field can permanently block a Vert.x worker thread when the underlying persistence throws (e.g. a MariaDB deadlock). The exception is swallowed by an RxJava 2 anti-pattern, so the blockingGet() in RestUpdaters.MICRONODE_LIST_UPDATER never returns. The thread is lost for good; repeated occurrences exhaust the worker pool and the instance stops serving requests.

We observed worker threads blocked for 10–14 hours in production (a permanent hang, not slowness). This became frequent after migrating to the v3 Hibernate/MariaDB backend, which raises real SQL exceptions where the legacy graph backend did not.

Environment

• Gentics Mesh 3.2.x (Hibernate/MariaDB backend)
• Vert.x 5.x, RxJava 2.2.21
• MariaDB 12.x

Symptoms

vertx-blocked-thread-checker reports threads blocked far beyond the limit, always on the same stack:

WARN [vertx-blocked-thread-checker] Thread vert.x-worker-thread-19 has been blocked for 36885865 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
    at java.util.concurrent.CountDownLatch.await(...)
    at io.reactivex.Single.blockingGet(Single.java:2870)
    at ...RestUpdaters.lambda$static$11(RestUpdaters.java:509)   // MICRONODE_LIST_UPDATER
    at ...HibUnmanagedFieldContainer.updateFieldsFromRest(...)
    at ...PersistingNodeDao.update(...)
    at ...NodeEndpoint.lambda$addUpdateHandler$23(NodeEndpoint.java:590)

The thread is parked on Single.blockingGet → CountDownLatch.await, i.e. it is waiting for a reactive result that will never arrive.

Root cause

HibMicronodeFieldList.update(InternalActionContext, MicronodeFieldList) uses Observable.create(...) and subscribes with a (onSuccess, onError) pair. The persistence writes (removeAll(), insertReferenced(), deleteReferenced()) run inside the onSuccess consumer:

return Observable.<Boolean>create(subscriber -> {
    ...
    ...toList().subscribe(micronodeList -> {          // onSuccess consumer
        removeAll();                                  // Hibernate write — can throw
        for (HibMicronode m : micronodeList) {
            insertReferenced(counter++, m);           // Hibernate write — can throw
        }
        existing.values().forEach(m -> deleteReferenced(m));  // Hibernate write — can throw
        subscriber.onNext(true);
        subscriber.onComplete();                      // reached ONLY if nothing above threw
    }, e -> {
        subscriber.onError(e);                        // does NOT catch throws from the onSuccess consumer
    });
}).singleOrError();

Called from RestUpdaters.MICRONODE_LIST_UPDATER:

// RestUpdaters.java (~509 in 3.2.1)
// TODO instead this method should also return an observable
micronodeGraphFieldList.update(ac, micronodeList).blockingGet();

The RxJava 2 trap: if the onSuccess consumer (micronodeList -> { … }) throws, the exception is not delivered to the sibling onError consumer. It is routed to the global RxJavaPlugins.onError handler (as an UndeliverableException) and effectively lost. Consequently:

1. removeAll() / insertReferenced() / deleteReferenced() throw a Hibernate exception.
2. subscriber.onNext() / onComplete() are never called.
3. The outer Observable.create never terminates.
4. singleOrError().blockingGet() waits forever on its CountDownLatch.
5. The worker thread is lost — without even returning the expected HTTP 500.

The existing // TODO instead this method should also return an observable shows this blocking design is already known to be problematic.

Why the v3 backend exposes it

With the Hibernate/MariaDB backend, persisting micronodes (removeAll / insert / delete) now hits a real SQL database and can throw (constraint violations, lock-wait timeout, deadlocks, optimistic locking). In our case the trigger is a MariaDB deadlock:

Error 1213 (SQLState 40001): Deadlock found when trying to get lock; try restarting transaction
SQL: INSERT INTO mesh_stringlistitem (...)   // string-list items inside the micronodes

The swallow-and-hang bug already existed; the v3 backend just makes it fire in practice.

Two failure modes (same underlying exception)

Depending on where the exception is thrown inside update(), behaviour differs — distinguishable by the blockingGet line:

Where the exception is thrown	RxJava routing	blockingGet	Result
Inside the flatMap mapper (micronode.updateFieldsFromRest(...))	correctly routed to onError	Single.java:2869	clean HTTP 500, thread released
Inside the onSuccess consumer (removeAll/insert/delete)	swallowed	Single.java:2870	thread hangs forever

The first mode is a working stack (exception propagates); the second is the permanent hang described above.

Minimal reproduction

The hang is purely an RxJava issue and reproduces without Mesh or a database, using the same RxJava version (2.2.21):

Single<Boolean> update = Observable.<Boolean>create(subscriber -> {
    Observable.fromIterable(List.of("a", "b"))
        .toList()
        .subscribe(list -> {
            throw new RuntimeException("simulated Hibernate deadlock"); // onSuccess throws
            // subscriber.onNext(true); subscriber.onComplete();  // never reached
        }, e -> subscriber.onError(e));                                 // never receives it
}).singleOrError();

update.blockingGet(); // blocks forever on CountDownLatch.await (Single.java:2870)

Impact

• Severity: high. Each occurrence permanently loses a worker thread (and holds its DB connection). Repeated occurrences exhaust the worker pool and the HikariCP pool → the instance stops serving requests.
• Not detectable by a standard livenessProbe on /health/live: that check runs on the event loop, which stays responsive while all worker threads are dead.
• Affects any node update touching a micronode-list field whenever persistence throws.