pkg_resources deprecated

[PierreCarrier]

At HPE, we work with a customer that needs to run gpt-neox, but we find that the following syntax is valid only for setuptools older than version 70 (69.5.1 seems to work). setuptools is currently at version 82:

https://github.com/EleutherAI/gpt-neox/blob/main/megatron/neox_arguments/arguments.py#L24

Google says to replace that above line, after installing packaging, with:

pip install packaging
from packaging import version

Then use version at lines like these (I'm not sure the exact syntax, I haven't tried it yet) :

https://github.com/EleutherAI/gpt-neox/blob/main/megatron/neox_arguments/arguments.py#L1151

I'm not sure what solution you want for this to work, but, we can't use any newer versions of setuptools because of that line. That'd be great if that line were to be modernized.

I contacted EleutherAI, Stella Biderman, by email and she said to file an issue on the github page...

Thank you,
Pierre.Carrier@hpe.com

[rehan243]

Are you using a virtual env for your gpt-neox setup? We've seen issues like this pop up when there's a version mismatch between the system Python and the project's dependencies. That line in arguments.py is indeed using the deprecated pkg_resources, which was removed in setuptools 70+. Replacing it with packaging is the right move - we did something similar in our RAG pipeline. You can use version.parse() to compare versions, like this:

from packaging import version
if version.parse(setuptools_version) >= version.parse('70'):
    # handle new version

We've had to deal with similar issues at scale - our fraud detection pipeline processes 11M+ images daily, and version conflicts can be a real pain. What's your current setuptools version pinned at, and are you using a requirements.txt file to manage dependencies?

[Quentin-Anthony]

This should be resolved now in #1386. Thank you for reporting!