Large cleanup (70k pictures) - very slow?

[clemens-eckert]

Hi RazgrizHsu, everyone,

I used Immich to move away from Google Photos. Due to the use of Immich Go and the Immich app on my phone I now have many pictures multiple times: There are close to 70k pictures and videos in my instance. I try to cleanup now with your Deduper. It runs a in a Docker.
My question: everytime I have results from "Find Similar" I choose "No confirm" -> "keep selected (x) and delete others (x)". Then nothing happens for a while (some minutes) but when I try to do start another action the Deduper tells me a job is already runnung. If I wait longer the screen still does not change. And if I click on "Find similar" again, it starts looking again - which takes literally hours!

I think I am using it wrong. What should I do/not do?

I really appreciate your tool and your help!

Best regards,
Clemens

[RazgrizHsu]

Hi @clemens-eckert,

I need to see what's happening on the server side.

Can you share the log file?
It's at ${DEDUP_DATA}/logs/deduper.log on your host, or run:
docker exec immich-deduper cat /app/data/logs/deduper.log

Also:

• Docker resource limits (CPU/RAM)?

[clemens-eckert]

Hi @RazgrizHsu

Please find attached the result from "docker exec immich-deduper cat /app/data/logs/deduper.log". cat deduper.log

The server has a i7-9700K with 80GB RAM. It is not super fast, but the CPU has not been at its limit.
I cannot run the CUDA-version since it is a Windows-based PC. At least that is what I understood from the documentation.

I also attached the YAML part of the deduper. Deduper-yaml.txt

Thank you for your support!

Best regards,
Clemens

[RazgrizHsu]

I noticed the log only covers the search phase; it reached 32/100 groups in 2 hours. Given the data size (70k assets), the processing time seems longer than expected, which might point to a bottleneck.

I’ve pushed some database indexes to help optimize the queries. To verify this, perhaps we could test with Max Groups set to 2 or 5 first? This would let us observe the full flow (search → select → delete) more quickly.

Seeing the complete log (including [similar] and [sim:selOk]) after a successful run will be really helpful for further debugging.

[RazgrizHsu]

Hi @clemens-eckert,

I just pushed a performance update that adds database indexes to speed up the similarity search queries.
This should help with the slow response you're experiencing.

Could you try pulling the latest image and run the search again?

If the issue persists, please share the new log file so I can investigate further.

[clemens-eckert]

Hi @RazgrizHsu,

Thank you for looking into this! Please find attached a videos of me using the Deduper with max 5 groups. I also attached the logs.

I always need to change any setting in Autoselect for it to apply. After "Keep selected and delete others" it takes a while until the next group is presented.

Best regards,
Clemens
deduper log 5 groups.txt

deduper-small.mp4

[RazgrizHsu]

Hi @clemens-eckert,

Thank you for sharing the video and logs.

From what I observed, this appears to be a chain reaction triggered by a state synchronization issue in the system.
I will investigate this in detail and post updates here as I make progress.

Your detailed reports have been very helpful in narrowing this down.

[RazgrizHsu]

Hi @clemens-eckert,

I haven't been able to pinpoint the exact root cause yet, but I noticed some operations in the log taking unusually long (around 16 seconds).

I've refactored some database table structures and addressed a potential UI state synchronization issue that might be related.

When you have time, perhaps you could try again with the latest version?
No need to record a video this time.
If you still encounter issues, just sharing the new log file would be really helpful for further investigation.