Rate limit guidance for large-scale donation migration (100k+ records)

[BrendanRomanDev]

I've seen this issue regarding bulk donations, among others, but we'd need some guidance on using the API or platform to make a smooth transition.

I'm part of an org looking to move historical donation records from another system into Planning Center to maintain accurate financial history. We're looking at over 100k donations that need to be imported. I noticed the Planning Center web portal supports CSV uploads for some entities, but not donations!

If we're limited to posting a single donation at a time, that's 100k API calls for 100k rows, not including prerequisite queries for the required person_id, or creation of the batches! Even after implementing a caching strategy, we're facing bottlenecks and failed requests on jobs as small as 100 csv rows.

What's the recommended approach for this kind of one-time historical data migration?

1. Is there a bulk import process or tool that we're missing, that would be better suited for this use case?
2. Would it be possible to request a temporary rate limit increase for a one-time data migration like this?
3. Are there any best practices you'd recommend for handling large-scale imports to minimize impact on PCO's infrastructure? The inability to POST multiple donations in a single call feels at odds with the API's rate limiting - especially for conversions 😵

Appreciate any guidance you can provide. We want to make sure we're doing this the right way🙏

[jonsextonbrick]

I ran into this recently with Giving migration myself. Not quite as many as you but roughly 30k records. I did them by batches of years and was able to rate limit it to 50 calls every 5 seconds without any issues. I did store my Person ID for repeat donors to ensure I wasn’t having to repeatedly do a Person search and based that on my export file from the previous giving platform.

[BrendanRomanDev]

I ran into this recently with Giving migration myself. Not quite as many as you but roughly 30k records. I did them by batches of years and was able to rate limit it to 50 calls every 5 seconds without any issues. I did store my Person ID for repeat donors to ensure I wasn’t having to repeatedly do a Person search and based that on my export file from the previous giving platform.

Just logged in to see if anyone responded, and this perfectly coincided with the moment you commented!

We're caching person_ids as well. Before we add any manual backoffs or manual throttling on our end, I wanted to be certain about what the PCO API natively supports. Thanks for sharing! I have a more confidence in making the decision to self-throttle our API calls proactively.

Would love to hear a developer chime in as well, as I'm sure there will be more folks like you and I in the future looking through GH issues to handle donation migrations!

Cheers!

[michaelhvisser]

I've built a SaaS product specifically for Planning Center data migrations, so I've spent a lot of time in this exact problem space. Unfortunately, there's no way around the API limitations—PCO doesn't offer a bulk donations endpoint, and CSV imports aren't available for Giving data.

Here's the approach that's worked best for us:

Cache everything locally. The biggest efficiency gain comes from minimizing redundant lookups. Build a local database that stores person IDs mapped to their identifying information (name, email, phone). Before searching the API for a donor, check your cache first. When you do find or create someone, immediately store that mapping so you never have to look them up again.

Use background workers for person resolution. Rather than processing donations linearly, separate the "find or create person" step into a dedicated worker queue. This lets you batch your lookups more efficiently and keeps your donation posting process cleaner.

Implement smart rate limit handling. Post at a pace slightly under the predicted limit, and when you do hit a 429, use time.sleep() (or equivalent) with the Retry-After header value. Don't just retry immediately—back off gracefully.

Consider a PAT if possible. If you're using a Personal Access Token rather than OAuth, you avoid token refresh overhead, which simplifies the process.

For a 100k record migration, expect this to take several days even with optimizations. It's a marathon, not a sprint.

If you'd rather not build this infrastructure yourself, feel free to reach out—this is literally what we do.

[molawson]

@BrendanRomanDev We don't offer any bulk donation creation at this point, and we don't have any immediate plans to support that.

I appreciate @michaelhvisser chiming in on this one. He's very well acquainted with all of our APIs. And all of his advice here is spot on. I'd definitely encourage writing code that takes into consideration the Retry-After header. That'll take the guess work out of pacing your requests optimally.

The other advice I'd give is about Batch size. Large batches can make for slow requests, especially when aggregate data is involved (both in the web UI and via the API). While there are no hard limits on their size currently, keeping them small can be very beneficial. Depending on the number of years your 100k donations are spread across, a batch per month of history might work well.

I'm going to close this issue, but if you have any further questions, please let us know.