gphotosdl stops responding to requests from rclone - Google photo transfers drop to zero

[SteakmanAus]

First of all great idea - appreciate the effort to get a solution to the google photos issue.

The proxy solution has a few limitations. First and foremost is the speed, with only one transfer at a time and waiting for the proxy browser to pull info it will take weeks to transfer my photos out of google (hence google takeout isnt an option). The multiple tab idea might be worth persuing to make this much faster.

There does seem to be an issue with the proxy browser stopping responding randomly to rclone download requests. To solve this I am running gphotosdl in a bat file that kills and restarts gphotosdl after 15 mins. Thankfully rclone waits patently and reconnects to the proxy once it starts.

Suggest a little more debugging is required to make this more stable. Let me know how I can assist.

---

I'm using the following command to start rclone which is configured to run gPHOTOS remote as a separate user in Google Cloud:

rclone copy -vvP --disable copy --tpslimit 5  --tpslimit-burst=5 --transfers 10 --checkers 10 --fast-list --gphotos-proxy "http://localhost:8282" gPHOTOS:media/by-month "\\truenas.local\Storage\Photos\Google Photos\media\by-month" 

Transferred:      397.833 MiB / 397.833 MiB, 100%, 0 B/s, ETA 0s
Transferred:          424 / 10444, 4%
Elapsed time:     46m36.5s
Transferring:
-*                     2003/2003-02/Scan0046.jpg: transferring
-*                     2003/2003-02/Scan0051.tif: transferring
-*                2003/2003-02/Scan0052 copy.jpg: transferring
-*                2003/2003-02/Scan0053 copy.jpg: transferring
-*                2003/2003-02/Scan0054 copy.jpg: transferring
-*                2003/2003-02/Scan0055 copy.jpg: transferring
-*                2003/2003-02/Scan0056 copy.jpg: transferring
-*                2003/2003-02/Scan0057 copy.jpg: transferring
-*                2003/2003-02/Scan0058 copy.jpg: transferring
-*                2003/2003-02/Scan0059 copy.jpg: transferring"

---

For the gphotosdl my batch file looks like this for others benefit:

:loop
start C:\rclone\gphotosdl.exe
timeout /t 900 >null
taskkill /f /im gphotosdl.exe >nul
timeout /t 10 >null
goto loop

---

Output from gphotosdl seems to be ok when its starts working, however it stops showing download photo notifications when it stops responding and rclone slowing drops transfer speed to 0 B/s as seen above:

2025/01/15 13:12:27 INFO Downloaded photo id=AB0N6ed1-7bpgTi844UACO-N7MO1WyYONI9C355CPk_m4AaG-s-qgrpseL0C3qDgQsBMMVrn1TlPSkfEXaTcNUt9GtUqGQTObA path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\8d2dbd4a-ed58-48a6-82ec-80905eacc555
2025/01/15 13:12:36 INFO got photo request id=AB0N6eet4FmD8w6BdtvJZYhqN5tf-B_kHUOVbjJ4KZXtB2TtRDYe7nl2apfuvB8_1qzpXNxqryqWsW52Zaoidll_B8Qn60vtDQ
2025/01/15 13:12:36 INFO got photo request id=AB0N6edm-Jgnr5sGelQCYeQckcrdp_hDNNMYnbhYZoBGTpxPtMg6BW1yCKR1eguPzHJh7FqMxyPh6Ufp0dn5zMAV3YTmOzMd0w
2025/01/15 13:12:37 INFO got photo request id=AB0N6eeUURDJ6MBqV5dNcTb9QV4Ww3kbp7CVxIkNAyvIYYIfFH795xowoHZL-uOZ9Eakaufgdckl_s_9bPKWy8kEgogypZhs5g
2025/01/15 13:12:37 INFO got photo request id=AB0N6ecOG0PfZgHtQgksE2Z_ImA80O8DL7nCvJBFSu3WfR2rTqCG2-JPLlk8caj4p8kTPOEC_aTYYJXzfK64NREm7uzbZT9bJA
2025/01/15 13:12:37 INFO got photo request id=AB0N6eck26gttFWCwNiAxrUDcJScjRu6IUCljLZCraN8K6aSX83dlAuVZDdbOgSjIophcF4mNt4TFEh25g6BVqk2x9ML7JLBlQ
2025/01/15 13:12:37 INFO got photo request id=AB0N6edyHXawCNlRzZobBXBJtiJvU5OVrT2LkbG0iVpYZsYeqcGJoJ4y9MLHFn9gr2gHf8mpiuaSejpGTgTfapQKNGgTn6KTNQ
2025/01/15 13:12:37 INFO got photo request id=AB0N6edSHlrgCGVQ8_GsKXrke6N34k2Fzqw9XV-LRfXYLl8GjR1oTsnIvZcvHO4w-sZ-1qL4tsxB1M7ySUaoVG1Xt7tUed6DEA
2025/01/15 13:12:37 INFO got photo request id=AB0N6ed3vXKEyn-mH8dxBAbx8X8urj5U-ycq2V5C6N5azRR_y9SPeSyVwj6k4Ro-2wOtiROJULT3lHDTHWeiKg2m3tsMkYXPzA
2025/01/15 13:12:37 INFO got photo request id=AB0N6eeHXzXiXg0u5vPdd3XxvNvfA5r3o9TpXTuZUHMQ-wXF0VZ5LRkWRpTzn43OY-o-MV3yv1l_pYQgD-UkNupQ3mo8Z8ry0Q
2025/01/15 13:12:37 INFO got photo request id=AB0N6ec2EwkKdmRE0VkJk7c0UHBWQuQwenhgsyU1lpzdnDd7vG8p3ANirokZyBSJY8hveTFVi_xoacmh3qOanNCKjmighgi7Ig
2025/01/15 13:12:45 INFO Downloaded photo id=AB0N6eet4FmD8w6BdtvJZYhqN5tf-B_kHUOVbjJ4KZXtB2TtRDYe7nl2apfuvB8_1qzpXNxqryqWsW52Zaoidll_B8Qn60vtDQ path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\63ecb457-1294-4ec8-97b6-02c37053ae43
2025/01/15 13:12:45 INFO got photo request id=AB0N6eeX8CUl7C41zzSUlpHDkxcvreElPlFegQ27HZllcpPGL7pCbq2NSurNySpwc7B47p2K4Z0trEs_LeajcihCObF8Th0d8w
2025/01/15 13:12:56 INFO Downloaded photo id=AB0N6edm-Jgnr5sGelQCYeQckcrdp_hDNNMYnbhYZoBGTpxPtMg6BW1yCKR1eguPzHJh7FqMxyPh6Ufp0dn5zMAV3YTmOzMd0w path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\960446ea-3a6c-4181-9a67-ccde44eaab5a
2025/01/15 13:12:56 INFO got photo request id=AB0N6efdkD8t0x4psQizPGZ4gjPsBwDWlASE4rT_ySff-y0jp-GtmkYSZZhGLHbQGMYf3je7lrVoWVS7hJ6kozvy0SnQTomvOw
2025/01/15 13:13:01 INFO Downloaded photo id=AB0N6eeUURDJ6MBqV5dNcTb9QV4Ww3kbp7CVxIkNAyvIYYIfFH795xowoHZL-uOZ9Eakaufgdckl_s_9bPKWy8kEgogypZhs5g path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\5cc95267-c591-4abf-b61b-afdc13d80c89
2025/01/15 13:13:01 INFO got photo request id=AB0N6ed-yw6glDK2W1Bg7IFbzPOBWFHgXxy4K73HuRUe1-yfmDuYRYudgIYEI1w2kuoisr6qwxVWgprSLkFArqJMilt3yIzBCQ
2025/01/15 13:13:11 INFO Downloaded photo id=AB0N6ecOG0PfZgHtQgksE2Z_ImA80O8DL7nCvJBFSu3WfR2rTqCG2-JPLlk8caj4p8kTPOEC_aTYYJXzfK64NREm7uzbZT9bJA path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\d457ad76-ed7c-441d-bd95-9418ac2c29dd
2025/01/15 13:13:11 INFO got photo request id=AB0N6eeBYtMQOFDFI59lZ4dPjNW15eSLKQGnIsMSDZXj_w5DorYCB3A4DvPJHNu40nDBIhO5yzQ5xzyTOhSn9mH0tuuwyeYX4w
2025/01/15 13:13:19 INFO Downloaded photo id=AB0N6eck26gttFWCwNiAxrUDcJScjRu6IUCljLZCraN8K6aSX83dlAuVZDdbOgSjIophcF4mNt4TFEh25g6BVqk2x9ML7JLBlQ path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\49cb23e9-1043-4ed9-9f64-163968dbce7a
2025/01/15 13:13:19 INFO got photo request id=AB0N6eebwYWO6bMC_a8irRRWokufSD_cuGtHd8dSHPswOciefOPClBL7eW_5DPO_IhkO2dADq-tEFlQgPKY4e51UAGnRTEMwZw
2025/01/15 13:13:27 INFO Downloaded photo id=AB0N6edyHXawCNlRzZobBXBJtiJvU5OVrT2LkbG0iVpYZsYeqcGJoJ4y9MLHFn9gr2gHf8mpiuaSejpGTgTfapQKNGgTn6KTNQ path=C:\Users\David\AppData\Local\Temp\gphotosdl2887146680\6e49817b-ba7d-4504-ba24-e3d2f5515f7e
2025/01/15 13:13:27 INFO got photo request id=AB0N6ecvoiLZZfj9KKsheM5sEPbf5fc4rdid1WvRHEuMur8oGI4t9ORc7zkD5uFgirIEgfMmdpLyTQm0AyPwrFF24AvA__3z

[ncw]

First of all great idea - appreciate the effort to get a solution to the google photos issue.

Thanks!

The proxy solution has a few limitations. First and foremost is the speed, with only one transfer at a time and waiting for the proxy browser to pull info it will take weeks to transfer my photos out of google (hence google takeout isnt an option). The multiple tab idea might be worth persuing to make this much faster.

I think we need to make it reliable first, but yes I think multiple tabs is probably the way to go.

There does seem to be an issue with the proxy browser stopping responding randomly to rclone download requests. To solve this I am running gphotosdl in a bat file that kills and restarts gphotosdl after 15 mins. Thankfully rclone waits patently and reconnects to the proxy once it starts.

I didn't see this in my testing. It is possible that something has changed at google.

Suggest a little more debugging is required to make this more stable. Let me know how I can assist.

The best thing to do would be to run gphotosdl with the -show flag and have a look at the browser to see if you see anything unusual when it stops. Look in the javascript console also.

It would be worth running gphotosdl with the -debug flag also - that might give a clue. If you could attach a gphotosdl log with -debug when it stops that will help.

[SteakmanAus]

Command prompt debug output.txt

Ok get an out of memory issue by the look of it but plenty of memory available

[ncw]

Great - we are getting somewhere :-)

I tried that locally and yes, I can see that the browser memory just keeys going up and up.

I made a small change to the code to make it download each photo in a new tab. This should help with the memory use.

Let me know what you think?

This is half way to downloading multiple photos at once with multiple tabs, but I decided not to go that far just yet.

Please try release v0.0.2 with this fix.

Thanks

[SteakmanAus]

Thanks Nick, Let me know when release 0.02 is pushed through on GitHub. Not showing at the moment

…

On Fri, 17 Jan 2025 at 8:23 pm, Nick Craig-Wood ***@***.***> wrote: Great - we are getting somewhere :-) I tried that locally and yes, I can see that the browser memory just keeys going up and up. I made a small change to the code to make it download each photo in a new tab. This should help with the memory use. Let me know what you think? This is half way to downloading multiple photos at once with multiple tabs, but I decided not to go that far just yet. Please try release v0.0.2 with this fix. Thanks — Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BOQJFWLUVGJ2XXCAA5ODQ3T2LDY35AVCNFSM6AAAAABVGMQ3TGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOJYGI2TEMBUGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ncw]

Sorry, forgot to push the tag!

The release is here: https://github.com/rclone/gphotosdl/releases/tag/v0.0.2

[SteakmanAus]

Thanks for the update but unfortunately same issue with v0.0.2 <img width="801" alt="Image" src="https://github.com/user-attachments/assets/977184d7-763b-458c-8565-42af75d6c603" /> <img width="929" alt="Image" src="https://github.com/user-attachments/assets/ebe5b802-25cb-4b00-b27a-b5b967daef69" />

[ncw]

Can you show me a task manager graph of the memory increasing with the chrome process?