Speeding up for pages with large histories

[tingofurro]

Hey,

First of all great work, this is going to be a tremendous resource and save a lot of time to fetch and organize Wikipedia revision histories.
I've noticed that for some "larger" pages, the loading time is very slow, because of what I assume is a large number of revisions to load.
For example when I run:

import wikipedia_histories
histories = wikipedia_histories.get_history("Paris")

It has been running for 5+ minutes without any end in sight.

Do you have an idea of what we can do to speed up the process? Perhaps batch retrieval? It could also be useful to do also have a progress bar (tqdm) or a "limit" function to specify for example edits in a time range.

Thanks again,
Philippe

[ndrezn]

Glad to hear you're finding the package useful!

Yes, performance is generally an issue here. To my knowledge, Wikipedia does not have an API endpoint to collect the text from multiple revisions at once. Note that this does not apply to all data collected -- the piece that takes the longest is collecting the text of each revision, as this requires a unique API call per revision. This package will collect revisions asynchronously in batches of 100 (a number I picked by trial & error).

The get_history function does have a parameter to disable collecting the text, which makes queries much faster (e.g. wikipedia_histories.get_history("Paris", include_text=False) (but of course this disables a particularly useful functionality of the package).

I used a progress bar while working with this locally, but dropped it from the PyPi release as to my mind it's just extra reliance on a third-party package. I'm not strictly opposed to adding back some functionality to show progress, I'll think on a good API to do this. A limit function of some sort is a great idea. It would be interesting to implement this in the library, potentially I could take some time to do this.