[Bug]: DockerCleanupJob can leave orphaned running execution after worker interruption

[naumanahmed19]

Description and Error Message

On Coolify v4.3.17, a scheduled Docker cleanup can leave its docker_cleanup_executions record permanently in running with a null message and finished_at after the queue worker is interrupted. A later scheduled execution can then fail with:

Illuminate\Queue\MaxAttemptsExceededException: App\Jobs\DockerCleanupJob has been attempted too many times.

The cleanup schedule is still dispatched, but no cleanup is completed. The UI continues to show the orphaned execution as running, and there is no warning that scheduled cleanup has stopped making progress.

The current job configuration makes this failure difficult to recover from automatically:

public $timeout = 600;
public $tries = 1;

(new WithoutOverlapping('docker-cleanup-'.$this->server->uuid))
    ->expireAfter(600)
    ->dontRelease();

DockerCleanupJob::failed() normally updates the latest running execution. If the worker is terminated before Laravel invokes that handler, the record remains running indefinitely. With one allowed attempt, the job has no retry path.

This is an execution-state recovery gap that remains after the fixes in #5645 and #8618. The original worker interruption may be environmental; the Coolify-specific issue is that the cleanup execution can remain orphaned and later scheduled cleanup does not recover automatically.

Expected Behavior

1. A cleanup execution should always reach success, failed, or cancelled.
2. A hard timeout or worker interruption should not leave an execution permanently marked running.
3. A later scheduled cleanup should run normally without manual database repair or a Horizon restart.
4. Coolify should surface a warning when a scheduled cleanup has not completed within its expected interval.

Steps to Reproduce

1. Enable scheduled Docker cleanup on a self-hosted Coolify instance.
2. Allow DockerCleanupJob to start, then simulate an unexpected Horizon worker termination while cleanup is running.
3. Inspect the Docker Cleanup execution list or docker_cleanup_executions: the execution can remain running with no finished_at value.
4. Wait for or trigger the next scheduled cleanup. It can fail with DockerCleanupJob has been attempted too many times instead of recovering.
5. Restart Horizon and dispatch a new cleanup; the newly dispatched cleanup completes, while the older execution remains orphaned unless it is repaired manually.

Example Repository URL

Not applicable.

Coolify Version

v4.3.17

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Not included; the failure is in the queue/execution-state lifecycle and contains no host-specific dependency found so far.

Screenshots / Visuals

None. The error and relevant state are included above without host or application details.

Additional Information

Potential safeguards:

• Set $failOnTimeout = true.
• Retry cleanup with bounded backoff instead of tries = 1.
• Reconcile running executions older than the job timeout before dispatching another cleanup.
• Add a heartbeat or lease timestamp so abandoned executions can be detected reliably.
• Warn when the most recent scheduled cleanup is older than the expected schedule interval.