New Core PR, after Rebased with upstream/master by HugoOnghai · Pull Request #28 · materialsproject/MPCite

HugoOnghai · 2025-07-30T22:45:58Z

Summary

New source code to find old DOI records, update them, and post new records.
some changes and misc. notes

core.py contains Python functions that will be used to orchestrate the constant updating and matching of OSTI's ELink Record database and our mongo DOI collection
no handling of "missing sponsor organization identifiers" is handled by the update/posting scripts

test_core.py contains the steps to ...

turn a stored json file into an ELink record
post that record to the review environment
take the resulting record response and make a doi document for our mongo doi collection
(Temporary fix due to using json files instead of prod-environment directly) updates missing datetime information in the collection by querying matching records in the production environment
from this datetime information, find the records that need to be updated with new robocrys descriptions
update those records on the ELink Review Environment

Todos

make new DOI collection in mp-core. Currently all tests are done on my local mongo database
build more tests
eventually get code operating for mp-core and ELink's production environment
use uv to package the project and get it onto TestPyPI (for Dagster use)

after upgrading... pip freeze > requirements.txt the old requirements are saved in oldrequirements.txt for reference

- deserialization works - re-serialization does not, there are additional dictionary keys that exist that don't match the provided config_file. will need to discuss the goal of this doi_builder a bit more to understand it

This reverts commit afa51f7.

…ng ELinkAPI RecordResponse

…on files) on ELink found bug with rows greater than 100 on ElinkAPI query_records (144845 dois under 10.17188, 12 are not titled Materials Data On... (edge cases), 144833 Materials have DOIs)

… steps + new doi collection

…tabase

* move old code to 'legacy' * setup project using uv * add license * testing skeleton * gh actions skeleton * remove old reqs file to prevent dependabot alerts --------- Co-authored-by: Tyler Mathis <35553152+tsmathis@users.noreply.github.com>

…accepted)

tsmathis

copied over my review comments from the previous PR (forgot to add the review there)

src/mp_cite/core.py

tests/file_to_jsonForUpload.py

tests/outputs.txt

tests/manage_backfills.py

tests/prod_to_review.py

tests/test_elink_api.py

…sting-suite PR handle any pytest testing files, after new-core-to-rebase PR is merged.

src/mp_cite/core.py

src/mp_cite/doi_builder.py

tsmathis · 2025-08-06T23:31:28Z

is oldrequirements.txt needed?

tsmathis · 2025-08-06T23:34:13Z

re: the models.py and doi_builder.py files, would it make sense to rename doi_builder.py to models.py since it only has pydantic models in it?

And to also move MinimumDARecord to models.py?

…ents on core.py

…ved DAMinimum model to models.py

HugoOnghai and others added 16 commits August 5, 2025 09:37

(after rebase) updating dependencies

eaf99a3

after upgrading... pip freeze > requirements.txt the old requirements are saved in oldrequirements.txt for reference

Trying to get test_doi_builder to work

6b96d33

- deserialization works - re-serialization does not, there are additional dictionary keys that exist that don't match the provided config_file. will need to discuss the goal of this doi_builder a bit more to understand it

Revert "Trying to get test_doi_builder to work"

46a688d

This reverts commit afa51f7.

(after rebase) preliminary version of ELinkGetResponseModel made, usi…

588c54e

…ng ELinkAPI RecordResponse

(after rebase) queried all desired data entries (stored as batched js…

06a3a7d

…on files) on ELink found bug with rows greater than 100 on ElinkAPI query_records (144845 dois under 10.17188, 12 are not titled Materials Data On... (edge cases), 144833 Materials have DOIs)

(after rebase) almost done with pipeline, need to incorporate pymongo…

4d1f471

… steps + new doi collection

(after rebase) template for core

a8172c9

(after rebase) push latest doi_builder and other thigns

1286279

find_out_of_date func

5466b4f

remaining core functions

f4325ba

testing core functionalities on review environment and local mongo da…

ce7651d

…tabase

(after rebase) Merged upstream (#1)

4905bcb

* move old code to 'legacy' * setup project using uv * add license * testing skeleton * gh actions skeleton * remove old reqs file to prevent dependabot alerts --------- Co-authored-by: Tyler Mathis <35553152+tsmathis@users.noreply.github.com>

(after rebase) updated gitignore and moved linting to new branch

c10e9b7

Remove accidentally committed files

9e89ad5

Cleaning up new-core PR, restore workflows and legacy stuff

f5a1cf4

Clean up legacy for PR

503696a

HugoOnghai force-pushed the new-core-to-rebase branch from 34a08fd to 503696a Compare August 5, 2025 19:14

HugoOnghai added 2 commits August 5, 2025 12:16

Clean Up branch for PR (after rebasing after lint + release PRs were …

a66640b

…accepted)

Cleaned up .gitignore

1cef222

tsmathis requested changes Aug 5, 2025

View reviewed changes

HugoOnghai added 4 commits August 5, 2025 16:50

Addresses comments raised in PR for core.py

a4ab306

Removed extraneous files from PR

ed89067

Removed models.py and cleaned up doi_builder.py

cf449f5

Removed extraneous files for debugging from tests/ -- will let the te…

1218576

…sting-suite PR handle any pytest testing files, after new-core-to-rebase PR is merged.

tsmathis reviewed Aug 6, 2025

View reviewed changes

src/mp_cite/core.py Outdated Show resolved Hide resolved

tsmathis reviewed Aug 6, 2025

View reviewed changes

src/mp_cite/core.py Outdated Show resolved Hide resolved

tsmathis reviewed Aug 6, 2025

View reviewed changes

src/mp_cite/doi_builder.py Outdated Show resolved Hide resolved

improved DOIModel to accept RecordResponse splat, and cleaned up comm…

8191ffc

…ents on core.py

HugoOnghai and others added 3 commits August 6, 2025 17:24

Addressed more comments, removed oldreqs, renamed doi_builder, and mo…

99d1ca9

…ved DAMinimum model to models.py

move docstring to beginning of function

3ef1405

remove empty docstring

5f21e46

tsmathis merged commit 2fbaff7 into materialsproject:master Aug 7, 2025
0 of 3 checks passed

tsmathis linked an issue Aug 8, 2025 that may be closed by this pull request

Add core functionality for using Elink 2.0 python client #17

Closed

6 tasks

tsmathis mentioned this pull request Aug 8, 2025

Add core functionality for using Elink 2.0 python client #17

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Core PR, after Rebased with upstream/master#28

New Core PR, after Rebased with upstream/master#28
tsmathis merged 26 commits intomaterialsproject:masterfrom
HugoOnghai:new-core-to-rebase

HugoOnghai commented Jul 30, 2025

Uh oh!

tsmathis left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tsmathis commented Aug 6, 2025

Uh oh!

tsmathis commented Aug 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

HugoOnghai commented Jul 30, 2025

Summary

test_core.py contains the steps to ...

Todos

Uh oh!

tsmathis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tsmathis commented Aug 6, 2025

Uh oh!

tsmathis commented Aug 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants