use `pathlib` vs `os.path`

[ebgoldstein]

I think a lot of the Path manipulation can be done with the pathlib library that was introduced in Python 3.4. This might help clean up the code, rather than using os.path.

from @chrisleaman 's JOSS review: openjournals/joss-reviews#2075 (comment)

[ebgoldstein]

pathlib: https://docs.python.org/3/library/pathlib.html

[Matmorcat]

@chrisleaman I'm not aware of any particular performance improvements with our code using a high level library for path manipulation over the os.path module other than enforcing pure paths. What were your thoughts as far as what it might improve?

[chrisleaman]

From a performance perspective, I don't think there are any advantages of pathlib over os.path. But in terms of code and readability, I think pathlib can make your code a bit cleaner. I think one of the best things for this project would be the glob command, avoiding os.walk pattern and filtering out results:

from pathlib import Path

top_level_csv_files = Path.cwd().glob('*.csv')
all_csv_files = Path.cwd().rglob('*.csv')

I do realize it is a bit of a pain to go back and refactor everything as the package works perfectly fine as it is. Sorry if this came across as "you must use this package rather than that package" 😅! My suggestion was more changes/additions in the future and even for new other packages you might write 😃