Adding LRGB to the HuggingFace hub

[clefourrier]

Hi!
@migalkin suggested on Twitter adding your datasets to the HuggingFace hub, which I think is a super cool idea, so I'm opening this issue to see if you need any help with that!

Here is the step by step tutorial on how to do so.
Ping me if you need anything in the process 🤗

[SauravMaheshkar]

PascalVOC-SP

• edge_wt_only_coord, slic = 10 | SauravMaheshkar/voc_superpixels_edge_wt_only_coord_10
• edge_wt_only_coord, slic = 30 | SauravMaheshkar/voc_superpixels_edge_wt_only_coord_30

[clefourrier]

Amazing!
Do you want me to create an LRBG org on the hub so all datasets can be under the same namespace?

[SauravMaheshkar]

Amazing! Do you want me to create an LRBG org on the hub so all datasets can be under the same namespace?

Yes thank you, that would be great. I also realized I uploaded .pickle files without processing. Will preprocess and update the dataset.

My HF Username is : SauravMaheshkar

[clefourrier]

I did and I added you to it! Once the datasets are correctly processed, feel free to transfer them to the org namespace!

@rampasek @vijaydwivedi75 Would one of you want to be an admin of that?
(If yes I would need your HuggingFace hub username)

[SauravMaheshkar]

I pre-processed and added all the PascalVOC datasets to the organization.

[vijaydwivedi75]

Thanks a lot @SauravMaheshkar @clefourrier!

@clefourrier, sure. My username is vijaypradwi.
[I will check the steps linked in above comments for the HF datasets, as I haven't used before :')]

[clefourrier]

@vijaydwivedi75 Added you as admin!

Feel free to ask any questions you need here :)

[SauravMaheshkar]

I pre-processed and added all the COCO-SP datasets to the organization.

[SauravMaheshkar]

I pre-processed and added the peptides-functional dataset to the organization.

[SauravMaheshkar]

I pre-processed and added the peptides-structural dataset to the organization.

[SauravMaheshkar]

I pre-processed and add the PCQM-Contact dataset to the organization.

That's all the datasets done ✅ .

@clefourrier @vijaydwivedi75 can you folks go through the datasets and make sure they look good ? Maybe then we can close this issue.

[clefourrier]

Thank you very much for your work! I think we're very close to being good, just 2 last points:

• why do the datasets have a .pt extension?
• relatedly, how do you usually load them? so that they integrate perfectly into datasets, they could benefit from having a loading script, see this doc.

[SauravMaheshkar]

1. I ran the pre-processing scripts on all the datasets and they outputted *.pt files. I assumed we wanted to upload the pre-processed datasets instead of raw files, right ?

2. How would you propose we work on the loading script ? Is there any pre-existing scripts that can be used as reference in the lrgb repository.

[clefourrier]

Regarding 1, I might be missing context, since I don't know LRGB that well: which preprocessing scripts did you use? (We do usually want the pre-processed datasets.)

Regarding 2, it depends on 1, I'd need to understand better what the preprocessing does to give you a hand :)

[SauravMaheshkar]

All the datasets have a process function in their respective classes within the dataset/ dir. I simply ran that script and uploaded the processed datasets obtained as the output.

For 2, I'll refer to @vijaydwivedi75 for more context.