SHA Mismatch: Wikimedia Provides SHA1, But Databus Uses SHA256

[tech0priyanshu]

Body

I'm currently working on integrating Wikimedia dump files into the Databus and ran into an issue related to checksum verification.

Wikimedia provides only SHA1 hashes for its dumps while the Databus expects and stores SHA256 hashes (sha256sum).

Since SHA256 is not provided by Wikimedia, I am currently forced to hardcode the SHA256 values manually during the publish process to get around this. This is obviously not ideal and could cause integrity issues or maintenance overhead in the future.

Future Solution

Allow alternate hash formats during the Databus publishing flow?

Add a script to automatically calculate the SHA256

Or provide guidance on how best to handle this mismatch?

cc:- @kurzum, @ronitblenz, @m1ci

[RitikaSen27]

Hello,

I’m interested in working on this issue. I understand the problem is that Wikimedia only provides SHA1 checksums while Databus expects SHA256, which currently requires manually hardcoding SHA256 values during publishing.

I see a few possible approaches, such as automatically computing SHA256 during the publish/ingestion flow, supporting alternate hash formats with clear labeling, or documenting a recommended best practice for this case. My initial thought is that auto-deriving SHA256 could preserve Databus’ integrity guarantees while avoiding manual work, but I’d like to align with the maintainers first.

Please let me know which direction is preferred, and I can follow up with a small, scoped PR.

Thank you.

[tech0priyanshu]

Approach	Pros	Cons
Allow alternate hash formats during the Databus publishing flow	• Flexible: supports multiple existing hash formats (e.g., MD5, SHA1) • Easier integration with legacy or external data sources • Reduces friction for contributors who already have hashes	• Inconsistent hashing standards across datasets • Weaker hashes may reduce security and trust • Additional validation logic needed in the publishing flow
Add a script to automatically calculate the SHA256	• Enforces a single, strong, standard hash (SHA256) • Improves data integrity and reproducibility • Less burden on contributors to manually compute hashes	• Extra computation step in the pipeline • Slight increase in publishing time • Requires maintaining and documenting the script

[tech0priyanshu]

Approach Add a script to automatically calculate the SHA256 LGTM

[alizahh-7]

Hi @tech0priyanshu 👋

I’ve just pushed a fix for this issue in PR #4 by adding automatic SHA256 computation during the publish flow, following the suggested approach. Thank you for the discussion and guidance here, it really helped clarify the direction.

I’m a student contributor and I’m interested in getting more involved with DBpedia, especially in the context of GSoC 2026. This was my first contribution to the Databus/Wikimedia dumps pipeline, and I really enjoyed working on it.

If there are any other beginner-friendly issues or improvements related to Databus, Wikimedia dumps, or automation that you’d recommend, I’d be happy to take a look and contribute further.

Thanks again, and looking forward to learning more!