Using pandas for data import

[kolibril13]

Hi there!

I recently wrote a python script that will import dataframes via pandas to Blender:
https://github.com/kolibril13/bpy-pandas-mesh/blob/main/pandas_mesh.py

https://x.com/kolibril13/status/1854860936961462783

and now I'm thinking of shipping this as it's own extension.

but as blender_spreadsheet_import is quite similar:
What do you think about including a pandas import to this package, @simonbroggi ?

[simonbroggi]

Hello
To me it does look like we could merge these tow add-ons on firs glance.
I've never heard of Pandas Data Frames, just read about it now.
Are they usually used for large datasets? I think my add-on currently doesn't work well with large datasets because my implementation is very crude: I think it currently just loads the entire file into RAM and then parses it. This should probably be optimized first, otherwise people with large files wont be happy.

[simonbroggi]

Aha, your script uses Panda package. That means to make it work in a Blender add-on the package would need to be installed in Blender Python. Probably that's possible somehow, but I'm not sure how, or if the licenses even allow it.

[simonbroggi]

This code could be used to install pandas into the users Blender Python. I guess the downside would be that doing this would install Pandas into a users Blender and leave it there even if the add-on is uninstalled afterwards. I'm not sure if this is a problem. Or if this would be rejected by the Blender Extensions platform..

import os
import subprocess
try:
  import pandas
except ModuleNotFoundError:
  # install pandas module into blender python
  python_exe = os.path.abspath(sys.executable)
  subprocess.call([python_exe, "-m", "ensurepip"])
  subprocess.call([python_exe, "-m", "pip", "install", "--upgrade", "pip"])
  subprocess.call([python_exe, "-m", "pip", "install", "pandas"])
  import pandas

[simonbroggi]

From the addons guidelines:

Be Self Contained:
Add-ons must not install Python modules, PIP packages, Python-wheels etc.
If some additional software required that cannot be bundled, this can be run by the user.

[simonbroggi]

Maybe the user can be given a choice to install pandas.
Maybe with https://docs.blender.org/api/current/bpy.types.AddonPreferences.html ?...

[simonbroggi]

Viktor Stepanov from Blender Cummunity Discord:

You would need to package your Python package dependencies via Python Wheels
https://docs.blender.org/manual/en/latest/advanced/extensions/python_wheels.html
or with https://pypi.org/project/vendorize/

install != contain
You can and should package your Python-wheels in your Blender Extension.
You just need to prepackage them into your zip archive

Example
https://extensions.blender.org/add-ons/molecularnodes/

https://github.com/BradyAJohnston/MolecularNodes/blob/main/molecularnodes/blender_manifest.toml

[kolibril13]

Thanks for the investigation!
Can you link to the Blender Cummunity Discord server conversation? Would be interesting to chat more about this.
Shipping pandas wheels with the extension sounds great.
@simonbroggi would you be interested in incorporating the pandas script into your extension? Otherwise I m considering making it its own extension.
I also talked recently we Brady from molecular nodes about shipping python wheels as Blender extensions at the Science Blender Discord server.

[simonbroggi]

Hi @kolibril13, yes, I think integrating Pandas into the spreadsheet importer would make a lot of sense! It might improve performance for big datasets. So if you have the time to create a pull request I'd be happy to merge it. Otherwise I'll see when I get the time to integrate it myself.

Here's the start of the discord conversation: https://discord.com/channels/185590609631903755/869803787785736222/1307988064248922133

[kolibril13]

Hi @simonbroggi,
thanks for your reply!
I'm just thinking that it might be easier to just start a new project for this kind of import.
You already have an infrastructure in place, and probably a new pandas backend might break some of your current infrastructure.

[kolibril13]

Also, thanks a lot for the links to the discord!

[simonbroggi]

Hi @simonbroggi, thanks for your reply! I'm just thinking that it might be easier to just start a new project for this kind of import. You already have an infrastructure in place, and probably a new pandas backend might break some of your current infrastructure.

The Pandas package could be used to import CSV also, right? I think it might make sense to eventually have one add-on that just imports data into the blender spreadsheet from various sources. Maybe Pandas is the right package to use for CSV, it might be more performant then what I am currently doing, not sure.. If you start a new project for Pandas import, maybe at some point we will see that we can merge it, or that this add-on is no longer be needed :-)

[kolibril13]

The Pandas package could be used to import CSV also, right?

yep, here's an example from my Blender talk 3 weeks ago using pandas to import https://youtu.be/umS8jFXpC-o?si=gZX1g01QtEKKKEXe&t=1346

I'm also considering Polars. Polars is like pandas, but on GPU, which makes it a lot faster.

Regarding development:
Polars provides five wheel files, each around 30MB in size (Windows, Linux ARM64, Linux x86, macOS ARM64, macOS x86). You can find them here: https://pypi.org/project/polars/#files.
I'm now investigating what's the best way to package them into a blender extension.

[kolibril13]

Hi @simonbroggi,
My csv importer is now in the review process:
https://extensions.blender.org/approval-queue/csv-importer/

I've also tried to import data with your importer, but it's not adding my data to the spreadsheet (tried using this dataset: https://github.com/kolibril13/blender_csv_import/blob/main/data_gemeran_cities.csv)

Also, other people reported that spreadsheet-import is currently not working:
https://extensions.blender.org/add-ons/spreadsheet-import/

Therefore, the question:
Does it make sense to deprecate spreadsheet-import and link to the csv-importer extension?

[simonbroggi]

hey, congrats to your CSV importer!!

Yeah, I guess it probably makes sense to deprecate mine then. I'm quite curious what the problem with my CSV Importer is (I didn't understand the problem of the 1 Star report). But I'm sure you did things better, and probably my spreadsheet import add-on is no longer useful. Only thing that I could imagine is that my add-on allows importing JSON data, but I doubt anyone uses that since it needs to be in a table structure anyway..

I'll test your add-on later today or tomorrow and see how I can deprecate mine and link to yours!

[kolibril13]

Thanks a lot! The add-on got now approved:
https://extensions.blender.org/add-ons/csv-importer/

And here's a quick screencast how to use it:

Screen.Recording.2024-12-16.at.22.53.58.mp4

I'll test your add-on later today or tomorrow and see how I can deprecate mine and link to yours!

Awesome, that's great to hear, let me know if there are some low-hanging fruits that I could add to the extension.

E.g. I was thinking of adding your feature, where the imported Mesh is automatically named after the imported file name.

my add-on allows importing JSON data

Polars supports all kind of imports to data frames:
https://docs.pola.rs/api/python/stable/reference/api/polars.read_json.html

If there's a need, I can also wrap some other import formats.