Question: realistic account scanning capacity per instance + multi-machine scaling

[roe23874-collab]

Context

I'm building a custodial XMR ecommerce store where pledge is handled entirely in PostgreSQL (no on-chain movement between deposit and withdrawal).

Architecture

monerod (single instance, NVMe SSD)
  └── monero-lws (scanning deposits via ZMQ webhook)

Questions

1. What is the realistic account scanning ceiling for a single monero-lws instance before scan lag starts falling behind new blocks? Approximate hardware context (e.g. 8-core, 12GB RAM, NVMe) would be very helpful.

2. The multi-machine remote scanning feature (msgpack over TCP/ASIO) is already merged — is it production-ready for this use case, or still experimental?

3. With remote scanning, is the architecture below correct: one monero-lws REST frontend + N scanner worker processes all connected to one monerod?

monerod (1 instance)
  ├── lws-scanner-worker-A  (users 0–500k)
  ├── lws-scanner-worker-B  (users 500k–1M)
  └── lws REST frontend (webhooks → app)

4. Any rough guidance on accounts-per-worker given your testing with dummy accounts (referenced in MAGICGrants/Monero-Fund#31)?

What I've already read

Issue #134 (REST load testing), Q1/Q2 2024 CCS completion notes, Q3 2024 CCS proposal, and the remote scanning PR discussion. The REST throughput numbers are clear — the scanning capacity question is the missing piece.

[vtnerd]

1. What is the realistic account scanning ceiling for a single monero-lws instance before scan lag starts falling behind new blocks? Approximate hardware context (e.g. 8-core, 12GB RAM, NVMe) would be very helpful.

I haven't run tests to get an estimate. And the number will be better once we finally get cbor into monerod ZMQ (currently JSON is used). If it's a problem for you now, a hacked together HTTP-BIN implementation could be done, but it would always be a patch on the side as cbor over ZMQ is cleaner.

If you need help with such a perf test, let me know. It would be great to have some rough numbers for reference in the docs.

2. The multi-machine remote scanning feature (msgpack over TCP/ASIO) is already merged — is it production-ready for this use case, or still experimental?

I would consider it production ready, but I'm not aware of anyone that has deployed this feature in practice. Edge Wallet may be the only one, but not certain about them.

The primary limitation is that sub-addresses won't work with the feature. Someone pointed out a workaround for this, but it uses a bit more CPU, so there's some tradeoffs to deploying it.

The second limitation is communication overhead (see next response).

3. With remote scanning, is the architecture below correct: one monero-lws REST frontend + N scanner worker processes all connected to one monerod?

This is correct. You could theoretical run multiple monerod instances but this will be tricky if they are slightly off on block height.

This design unfortunately creates some overhead with monerod as it has to respond to each worker individually. Fixing this is complicated because most workers will be on the "chain tip" (latest block), but any number of threads/workers could be behind chain tip attempting to catch-up accounts. Eventually some coalescing of chain-tip threads would be nice, but requires some non-trivial work (imo).