`xdbg` fails to generate messages for local backend

[Maelkum]

I was experimenting with a local backend for the past few days, and I noticed a problem generating messages for the local backend.

Starting from a clean setup, I generate a number of identities, followed by a number of groups and invites.

Then, I try to generate some messages. Number of messages is not important and it fails even with a single message.

This command fails after 30 seconds:

$ time xdbg --backend local --d14n generate --entity message --amount 1
xdbg::app::generate using concurrency
	concurrency Concurrency(16)
Error: 
   0: failed to load client for b14df1798bbdd461b05aa80ea05b5ee66595555ce9c44e6c95380f8bf7fe9d24, 78 other clients succeeded
   1: tried to open sqlite database file@/Users/aleksandarcekrlic/Library/Application Support/org.xmtp.xdbg/sqlite/3/b14df1798bbdd461b05aa80ea05b5ee66595555ce9c44e6c95380f8bf7fe9d24:3.db3
   2: Pool error: timed out waiting for connection: Unable to open the database file
   3: timed out waiting for connection: Unable to open the database file

Location:
   crates/xmtp_debug/src/app/clients.rs:153

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
xdbg --backend local --d14n generate --entity message --amount 1  0.88s user 3.84s system 15% cpu 30.665 total

However, if I run the same command and include the "-v" flag, it works:

$ time xdbg --backend local --d14n -v generate --entity message --amount 1         
xdbg::app created project directories
	directory /Users/aleksandarcekrlic/Library/Application Support/org.xmtp.xdbg
	sqlite_stores /Users/aleksandarcekrlic/Library/Application Support/org.xmtp.xdbg/sqlite/3
xdbg::app setting fdlimit
	fdlimit 512
xdbg::app::generate using concurrency
	concurrency Concurrency(16)
xdbg::app::clients creating client from identity
	inbox_id "011f6a60634d7715da83a38fbd678c1008112f39e4b0ccfc717c8e860c707be1"
	db_path /Users/aleksandarcekrlic/Library/Application Support/org.xmtp.xdbg/sqlite/3/011f6a60634d7715da83a38fbd678c1008112f39e4b0ccfc717c8e860c707be1:3.db3
xdbg::app::clients creating client from identity
	inbox_id "045f0ff2e67b855bb7ce210b0006545c8f4b763ecae5108aa122a0690b05e795"
	db_path /Users/aleksandarcekrlic/Library/Application Support/org.xmtp.xdbg/sqlite/3/045f0ff2e67b855bb7ce210b0006545c8f4b763ecae5108aa122a0690b05e795:3.db3
	
// ... trimmed for brevity

xdbg::app::clients creating client from identity
	inbox_id "fafe95385a7697355daee71d9723352c98a20c03fdfbddb0e06c8707de3ce0e4"
	db_path /Users/aleksandarcekrlic/Library/Application Support/org.xmtp.xdbg/sqlite/3/fafe95385a7697355daee71d9723352c98a20c03fdfbddb0e06c8707de3ce0e4:3.db3
xdbg::app::clients took 745.79925ms to load 120 clients
xdbg::app::generate::messages generating messages
	fdlimit 512
xdbg::app::generate::messages sending message
	time Instant { tv_sec: 293133, tv_nsec: 176395958 }
	group "8db7bd2a9b444dfc68c8953a35ed7d76"
xdbg::app::generate messages generated
xdbg --backend local --d14n -v generate --entity message --amount 1  1.02s user 6.16s system 453% cpu 1.582 total

My guess it that there might be a race condition, and the act of printing additional info about existing identities slows it down enough that it works?

Also, the issue does not manifest when running against the staging environment.

The workaround is pretty simple and effective so don't think this is hugely important.