[Driver/C] Receive destination can leak when Driver closes after add acknowledgement

[JinSanPang3]

Version

• Aeron 1.52.2 (5b62f21d917af027cdf5a3241aa5f355149b04fa)
• Also inspected current master at b2487f02702a624fe1c1e5fd7cbddaac5bde02a5
• Linux, embedded C Media Driver, AERON_THREADING_MODE_DEDICATED, ASan/LSan enabled

Problem

Closing an embedded C Media Driver can leak a receive destination when the Driver Conductor has completed an ADD_RCV_DESTINATION command but the Receiver has not yet consumed the corresponding proxy command.

The public async destination operation has already reported success at this point. The client has no later Receiver-applied completion signal to wait for.

One leaked destination consists of:

• 472 bytes allocated by aeron_receive_destination_create
• 12,968 bytes allocated by aeron_udp_channel_do_initial_parse
• the destination UDP transport/socket is also not explicitly closed before process exit

Deterministic reproducer

The following reproducer uses only Aeron APIs. It creates a manual-control subscription, waits for the destination operation to report success, closes the Subscription and Aeron Client, and finally closes the embedded Driver.

#include <Aeron.h>

extern "C"
{
#include <aeron_driver.h>
}

#include <filesystem>
#include <iostream>
#include <memory>
#include <stdexcept>
#include <string>
#include <thread>

#include <unistd.h>

namespace
{
void check(int result, const char *message)
{
    if (result < 0)
    {
        throw std::runtime_error(std::string(message) + ": " + aeron_errmsg());
    }
}
}

int main()
{
    const auto directory = std::filesystem::path("/tmp") /
        ("aeron-receive-destination-leak-" + std::to_string(::getpid()));
    aeron_driver_context_t *context = nullptr;
    aeron_driver_t *driver = nullptr;

    check(aeron_driver_context_init(&context), "context init");
    check(aeron_driver_context_set_dir(context, directory.c_str()), "set directory");
    check(aeron_driver_context_set_dir_delete_on_start(context, true), "delete on start");
    check(aeron_driver_context_set_dir_delete_on_shutdown(context, true), "delete on shutdown");
    check(aeron_driver_context_set_threading_mode(context, AERON_THREADING_MODE_DEDICATED), "threading mode");
    check(aeron_driver_init(&driver, context), "driver init");
    check(aeron_driver_start(driver, false), "driver start");

    aeron::Context client_context;
    client_context.aeronDir(directory.string());
    auto client = aeron::Aeron::connect(client_context);
    const auto registration = client->addSubscription("aeron:udp?control-mode=manual", 1001);
    std::shared_ptr<aeron::Subscription> subscription;
    while (!(subscription = client->findSubscription(registration)))
    {
        std::this_thread::yield();
    }

    auto *operation = subscription->addDestinationAsync("aeron:udp?endpoint=127.0.0.1:0");
    while (true)
    {
        const auto result = aeron_subscription_async_destination_poll(operation);
        check(result, "destination poll");
        if (result > 0)
        {
            break;
        }
        std::this_thread::yield();
    }

    subscription.reset();
    client.reset();
    check(aeron_driver_close(driver), "driver close");
    check(aeron_driver_context_close(context), "context close");
    std::cout << "closed after destination add acknowledgement\n";
}

Build the reproducer against the Aeron C Driver and C++ wrapper with AddressSanitizer, then run:

ASAN_OPTIONS=detect_leaks=1:halt_on_error=1 \
AERON_RECEIVER_IDLE_STRATEGY=sleeping \
AERON_RECEIVER_IDLE_STRATEGY_INIT_ARGS=100ms \
./receive_destination_leak_repro

The 100 ms Receiver sleep makes the shutdown window deterministic. The report is:

Direct leak of 472 byte(s) in 1 object(s) allocated from:
    #1 aeron_alloc
    #2 aeron_receive_destination_create
    #3 aeron_driver_conductor_execute_add_receive_network_destination
    #4 aeron_driver_conductor_do_work
    #5 agent_main

Indirect leak of 12968 byte(s) in 1 object(s) allocated from:
    #1 aeron_alloc
    #2 aeron_udp_channel_do_initial_parse
    #3 aeron_driver_conductor_on_add_receive_network_destination
    #4 aeron_driver_conductor_on_command
    #5 aeron_mpsc_rb_controlled_read
    #6 aeron_driver_conductor_do_work
    #7 agent_main

SUMMARY: AddressSanitizer: 13440 byte(s) leaked in 2 allocation(s).

With the normal Receiver idle strategy this is intermittent, but the same two allocations and byte counts were observed in a larger lifecycle test.

Suspected shutdown race

For DEDICATED mode the relevant runner indexes are:

• Conductor: 0
• Receiver: 2

aeron_driver_close stops runners from AERON_AGENT_RUNNER_MAX - 1 down to zero, so the Receiver is stopped before the Conductor.

aeron_driver_conductor_execute_add_receive_network_destination then does the following:

1. Creates aeron_receive_destination_t.
2. Enqueues it through aeron_driver_receiver_proxy_on_add_destination.
3. Immediately calls aeron_driver_conductor_on_operation_succeeded.

The destination is not inserted into endpoint->destinations until aeron_driver_receiver_on_add_destination executes on the Receiver thread. If the Receiver is already stopped, the command remains in its proxy ring buffer.

aeron_driver_receiver_on_close frees Receiver arrays and closes its poller, but does not drain or cancel the pending Receiver command queue. Later, aeron_driver_conductor_on_close deletes receive endpoints and only frees destinations present in each endpoint's destinations array. The queued destination therefore has no remaining cleanup owner.

Expected behavior

After an async destination add reports success, closing the Subscription, Aeron Client, and embedded Driver should reclaim the destination even if the Receiver has not yet applied the internal proxy command.

Relationship to #2123

This appears separate from #2123. Commit 45cf30986d4b4a7280967592e41da6b14398ac65 drains pending native-resource FREE_LOG_BUFFER commands, but does not drain or cancel ownership-bearing Receiver commands.

Fix considerations

Simply reversing the global runner stop order may move the race to commands sent from Receiver to Conductor. The shutdown path likely needs an ownership-aware drain/cancellation rule for pending Receiver commands, especially aeron_driver_receiver_on_add_destination, plus a regression that closes the Driver after the destination operation is acknowledged but before Receiver application.

[JinSanPang3]

Follow-up validation to rule out C++ wrapper lifetime or mixed-API behavior:

• Rewrote the reproducer using only the Aeron C API (aeron_context_init, aeron_init, aeron_async_add_subscription, aeron_subscription_async_add_destination).
• Waited until aeron_subscription_async_destination_poll returned 1.
• Called aeron_subscription_close with an on_close_complete callback and waited until that callback fired.
• Only then closed the Aeron Client, client context, embedded Driver, and Driver context.

With AERON_RECEIVER_IDLE_STRATEGY_INIT_ARGS=100ms, immediate Driver close still deterministically reports the identical leak:

Direct leak:     472 bytes from aeron_receive_destination_create
Indirect leak: 12968 bytes from aeron_udp_channel_do_initial_parse
SUMMARY: AddressSanitizer: 13440 byte(s) leaked in 2 allocation(s)

As an A/B control, inserting a 300 ms wait after the Subscription close completion callback, with all other settings unchanged, exits cleanly with no LSan report. This rules out the wrapper, failure to wait for Subscription close completion, and the async operation object as the cause; the result is consistent with the destination remaining in the stopped Receiver's pending command queue during immediate Driver shutdown.

[VatsalKhosla]

This looks very interesting. Can i get this assigned?