v0.14.3

What's Changed

Enhancements and Cleanup:

ISA MODEL

• fix to Comments on several ISA objects
• fix to Datafile.py
• fix to Material.py

ISA TAB

• fix to isatab writer / reader and serialisation of Characteristics, Parameter and FactorValue values
• roundtrip testing

ISA JSON

• upgrade the all ISA json_schemas overcoming the silent schema validation failure
• fix to json serialiser affecting declaration/referencing of units

ISA Documentation:

• addition of several new jupyter notebooks documenting ISA object usages
• creation of jupyter notebooks documenting known limitations or anti patterns (i.e., "don't" when using ISA tab representations)

Versions update:

• Dropped python 3.8 and 3.9
• Added compatibility for python 3.12 (production ready)
• Added compatibility for python 3.13 (test only)
• Bump numpy, pandas, sqlAlchemy

New Contributors

• @ptth222 made their first contribution in #549

special thanks for reporting and addressing issues in isatab writer

• @oyurekten made their first contribution in #590

special thanks for contributing the migration to uv

• @sorenwacker made their first contribution in #589

special thanks for contributing the sqlAlchemy upgrade

• @knirirr made their first contribution in #567

special thanks for code review, overall contribution and help.

Former Contributors

special thanks to @terazus for huge contribution to isa refactoring, isajson-ld among many other things.

More details

• Issue #399 addressed by #601 and #603
• Issue #409
• Issue #414 by @knirirr in https://github.com/ISA-tools/isa-api/pull/580/files
• Issue #416 in in #590
• Issue #427 by @knirirr and @proccaserra in #592
• Issue #483
• Merge from master by @terazus in #493
• merge from master by @terazus in #495
• Update requirements.txt by @terazus in #499
• Issue 497 fix by @proccaserra in #507
• Fix to determining JSON directory location. by @terazus in #546
• Testing changes to run on Windows. by @ptth222 in #549
• Update defaults.py by @ptth222 in #539
• More changes by @ptth222 in #550
• Rename i_df and investigation_df. by @ptth222 in #529
• Fix rule and check. by @terazus in #548
• Fairly significant changes to check_protocol_fields by @ptth222 in #531
• Changed messaging on study assays errors by @ptth222 in #530
• Extended 511 by @proccaserra in #551
• Added better messaging by @ptth222 in #527
• Refactor isatab loader by @terazus in #555
• Protocol types refactor by @ptth222 in #556
• Issue #511 by @proccaserra in #514
• Issue #520 by @knirirr in #592
• Issue #523 [Characteristic values are recast as string in ISA tab in https://github.com//pull/590
• Issue #540 relaxing requirement on Person Mid Initials in XML config.
• Issue #562 python312 by @proccaserra in #564
• Mock pubmed 557 by @knirirr in #567
• adjusting bh23 workbook, and model class by @proccaserra in #565
• issue #566
• Issue #570 by @oyurekten in #590
• Issue #571 by @oyurekten in #590
• Deprecation future warnings by @knirirr in #574
• Update README.md by @proccaserra in #576
• adding toml file, updates to setup.py, fixing broken shield and ... by @proccaserra in #577
• merging into master from develop to make a new release by @proccaserra in #573
• Issue #575 addressing SQLAlchemy breaking change when upgrading from 1.4 to 2.+ by @oyurekten in #590
• creating a lock file, updating toml file by @proccaserra in #578
• upping numpy and pandas version for isacookbok by @proccaserra in #579
• Issue #584 addressed in #590
• UV Package Manager Integration by @oyurekten in #590
• PR on dev by @proccaserra in #588
• Deprecation warnings prs by @proccaserra in #593
• check_isa_schemas updated to new json validator. #591 by @knirirr in #592
• Develop by @proccaserra in #580
• Removed references to deprecated materials in favour of other_material. by @knirirr in #594
• Hid some PerformanceWarnings... by @knirirr in #596
• fix to get_ols_ontology by @proccaserra in #599
• Re-enable some skipped tests. by @knirirr in #600
• Develop by @proccaserra in #595
• jupyter notebook testing and revisions/corrections by @proccaserra in #603

Full Changelog: v0.14.2...v0.14.3

Release 0.14.2

Enhancements and Cleanup:

ISA MODEL

• The model.py file has been split into the model module. Each class has been split into a dedicated file.
• Added a method from_dict() to the ISA Investigation class to let users load objects from a dictionary.
• Added a method to_dict() to each ISA model class that let users dump ISA objects to dictionaries. (also see ISA JSONLD section)
• Added a new GraphQL interface to ISA investigation (see below querying isa for more information)

ISA TAB

• The isatab.py file has been split into the isatab module.
  Functions and classes have been split into three submodules: load (to read isatab files), dump (to write isafiles) and validate (to run validation rules against an isatab file). Usage remains the same.
• The isatab validation pipeline have been reworked to allow the rules set to be extended with user custom rules. Please contact us if you would like to implement your own validation rules.

ISA JSON

• the isajson.py file has been split into the isajson module.
  Functions and classes have been split into three files: load.py, dump.py and validate.py. Usage remains the same but there is now an alternative using dicts instead of files (see below)
• load.py and dump.py are now wrappers. The code of the isa json serializer/deserializer has been moved to be part of the ISA model. Users can still use the old load() and dump() methods but are now able to do it directly from an investigation object. Example

from json import load
from isatools.model import Investigation

isa_json_file = 'path/to/isa.json'  # the json file path
with open(isa_json_file, 'r') as f:
    isa_json_input = load(f)  # load the dict from the file
investigation = Investigation()  # create en investigation
investigation.from_dict(isa_json_input)  # feed in the dictionary, will generated missing identifiers make inputs and outputs different
isa_json_output = investigation.to_dict()  # Dump it back to a dict
another_investigation = Investigation()  # create en investigation
another_investigation.from_dict(isa_json_output)  # feed in the dictionary
assert investigation == another_investigation  # assert equality between the created investigations.
assert investigation.to_dict() == another_investigation.to-dict()  # compare to_dict() methods

Note: identifiers may be missing from the json file and autogenerated by the ISA objects. In this case, output dictionnaries won't match which is why the investigation is being compared to another investigation.

ISA JSON-LD

• Using the new to_dict() function from model.py, it is possible to produce JSON-LD.
  The The method takes an optional boolean parameter ld (default to False) that serializes to json-ld dictionaries by injecting @context, @id and @type attributes. See isa json below for more information.
• Customisation of the JSON-LD serialization is achieved through the set_context() function of the model module.
• Added support for SIO, WD, OBO and SDO annotations (see https://github.com/ISA-tools/isa-api/tree/master/isatools/resources/json-context).
• Added support for generating URL identifiers (necessary for hosting triple stores).
• Check https://github.com/ISA-tools/isa-api/blob/master/isa-cookbook/content/notebooks/querying-isa-with-graphql-and-sparql.ipynb for examples.

SPARQL querying over ISA:

• The outputs of the ISA JSON LD serializer have been tested as a valid RDF representation. Using MTBLS instances with rdflib and settting the contexts with set_context(vocab='wd', all_in_one=False, local=False, include_contexts=False) we were able to produce RDF triples and store them into a virtuosoDB and a berkeleyDB with SPARQL support.
• Check https://github.com/ISA-tools/isa-api/blob/master/isa-cookbook/content/notebooks/querying-isa-with-graphql-and-sparql.ipynb for examples.

GRAPHQL querying over ISA:

• Each ISA Investigation now has a graphQL interface bound to it.
  A full documentation of the graphQL interface is available at https://github.com/ISA-tools/isa-api/tree/master/isatools/graphQL/README.md

MTBLS, the EMBL-EBI MetaboLights Client:

• Refactored the MTBLS client but kept the same functions (and signature) exposed. You can now use MTBLInvestigation instead of the old functions. Example:

from isatools.net.mtbls import MTBLSInvestigation

investigation =  MTBLSInvestigation('MTBLS1')
# This creates a temp dir for the investigation that gets deleted when the object is destroyed through the automatic garbage collector. You can pass a directory if you want to persist the investigation.
investigation =  MTBLSInvestigation(mtbls_id='MTBLS1', output_directory="path/to/output/dir", output_format='tab')

# You can then run all the old function through methods with the same name and chain methods without downloading the whole investigation again.
print(investigation.get_characteristics_summary())
print(investigation.get_factor_names())

ISA SQL:

• Created a relational SQL Alchemy model for ISA objects tested with PSQL and SQLite. All model files are available here: https://github.com/ISA-tools/isa-api/tree/master/isatools/database/models
  

Other

Versions update:

• Dropped python 3.6 and 3.7
• Added compatibility for python 3.10 (production ready)
• Added compatibility for python 3.11 (test only)
• Bump numpy, pandas and mzml.

Cookbook

• Added a cookbook for querying isa with sparql and grapql. See https://github.com/ISA-tools/isa-api/blob/master/isa-cookbook/content/notebooks/querying-isa-with-graphql-and-sparql.ipynb

Testing:

• The application has been further tested with CI, especially the model. Global coverage increased from 75% to 84%.
• Build time was reduced due to mocking some network operations.

ISA-API performance profiler:

• Added a small command line interface using cProfile to generate performance reports of different ISA features (in particular serialization and deserialization to tab and json).

Known issues:

• There is a know performance issue with the coveralls/coverage modules on python 3.11 making the build very slow (see coveragepy/coveragepy#1287). Performances for end users shouldn't be affected.

Contributors:

• Batista Dominique, University of Oxford (https://orcid.org/0000-0002-2109-489X)
• Philippe Rocca Serra, University of Oxford (https://orcid.org/0000-0001-9853-5668)

Release v0.13.0 - Release Candidate 2

Release v0.13.0 - Release Candidate v2

Enhancements and Cleanup:

• significant optimisation of isatab.dump() via reworking the study/assay graph implementation. This can improve up to 3 or 4 orders of magnitude the writing to ISA-tab for datasets with thousands of samples (PR #403). An additional tweak has been implemented to compute only once the study/assay process graph (8c2d09d)
• All the major objects of the ISA models are now Commentable (PR #395)
• isatools.create.connectors now supports both strings and Ontology Annotations for Subject Type and Sample Type (PR #405)
• CI is now migrated from Travis CI to GitHub CI (#404)

Breaking changes:

• Changes to the isatools.create.connectors module to align it with the Datascriptor study JSON config. Now the Datascriptor study config has a "design" property holding all the study design information (PR #405)

Bug Fixes:

• all names for protocols in studies generated by isatools.create.connectors from Datascriptor study configs are unique (issue #398)
• identifiers for studies and investigations are coerced to strings (commit 7e9f468)

Release v0.13.0 - Release Candidate

Release v0.13.0 - Release Candidate

Enhancements and Cleanup:

• significant optimisation of isatab.dump() via reworking the study/assay graph implementation. This can improve up to 3 or 4 orders of magnitude the writing to ISA-tab for datasets with thousands of samples (PR #403)
• All the major objects of the ISA models are now Commentable (PR #395)
• isatools.create.connectors now supports both strings and Ontology Annotations for Subject Type and Sample Type (PR #405)
• CI is now migrated from Travis CI to GitHub CI (#404)

Breaking changes:

• Changes to the isatools.create.connectors module to align it with the Datascriptor study JSON config. Now the Datascriptor study config has a "design" property holding all the study design information (PR #405)

Bug Fixes:

• all names for protocols in studies generated by isatools.create.connectors from Datascriptor study configs are unique (issue #398)
• identifiers for studies and investigations are coerced to strings (commit 7e9f468)

Release v0.12.2 - bugfix release

This Bugfix release fixes the broken networkx dependency in setup.py.

Release v0.12.1 - bugfix release

Update to fix:

• missing Factor Values from isatools.create serialisation (PR #391)

Release v0.12.0 - Stable

Release v0.12.0

Features:

• Added the isatools.create.connectors module. This module contains the generate_study_design_from_config function to generate an isatools.create.StudyDesign from a JSON configuration document containing metadata about the study design, including sampling and assay plan. These JSON documents can be generated by the Datascriptor application.(issues #355, #356, #377)
• Added support for observational variables to isatools.create.model.StudyArm (with the parameter source_characteristics) (PR #367)
• StudyDesign.generate_isa_study()now generates one Assay per assay type (via StudyDesign._generate_samples()). (issue #369, #373)
• The names of the assay nodes (extract, labelled, extract, protocol node, and data file) (issue #370, #373)
• isatools.create.model.ProductNode now has an attribute extension (e.g. a file extension fo a DataFile node) (issue #378)

Enhancements and Cleanup:

• Introduced support for (built-in, non-configurable) synonyms in protocol types. Protocol types have also been made case insensitive. (issue #382)
• Refactored constants and error messages for the create mode, introducing the modules isatools.create.constants and isatools.create.errors (issue #342, #368)
• Removed asterisk imports (*) from all the core modules (issue #368)
• Removed leftover print() statements (issue #374)
• overall test coverage increased from 67% to 75%. The increase is due both to an increase in the number of tests and the removal of some deprecated, experimental or spurious modules.

Breaking changes:

• isatools.create.models has been renamed to isatools.create.model (issue #368)
• IsaModelAttributeError has been replaced with AttributeError. We will favour standard Exceptions whenever applicable.

Bug Fixes:

• Fixed typo in NAME_PROPERTY_ASSIGNMENT_ERROR (issue #328)
• Silent expansion of Characteristic categories from str to OntologyAnnotation now supported. You can assign a string to a characteristic category and it will be automatically wrapped into an ontology annotation. (issue #332)
• Fixed duplicated in OntologyAnnotation ids (issue #381)
• Fixed raising of an error when correctly setting a comment value (issue #360)
• Assigned correct header "Data Transformation Name" for protocol type "sequence analysis data transformation" (commit b8f2f55)

Release v0.12.0 - Release Candidate 3

Release v0.12.0 - Release Candidate

Features:

• Added the isatools.create.connectors module. This module contains the generate_study_design_from_config function to generate an isatools.create.StudyDesign from a JSON configuration document containing metadata about the study design, including sampling and assay plan. These JSON documents can be generated by the Datascriptor application.(issues #355, #356, #377)
• Added support for observational variables to isatools.create.model.StudyArm (with the parameter source_characteristics) (PR #367)
• StudyDesign.generate_isa_study()now generates one Assay per assay type (via StudyDesign._generate_samples()). (issue #369, #373)
• The names of the assay nodes (extract, labelled, extract, protocol node, and data file) (issue #370, #373)
• isatools.create.model.ProductNode now has an attribute extension (e.g. a file extension fo a DataFile node) (issue #378)

Enhancements and Cleanup:

• Introduced support for (built-in, non-configurable) synonyms in protocol types. Protocol types have also been made case insensitive. (issue #382)
• Refactored constants and error messages for the create mode, introducing the modules isatools.create.constants and isatools.create.errors (issue #342, #368)
• Removed asterisk imports (*) from all the core modules (issue #368)
• Removed leftover print() statements (issue #374)
• overall test coverage increased from 67% to 75%. The increase is due both to an increase in the number of tests and the removal of some deprecated, experimental or spurious modules.

Breaking changes:

• isatools.create.models has been renamed to isatools.create.model (issue #368)
• IsaModelAttributeError has been replaced with AttributeError. We will favour standard Exceptions whenever applicable.

Bug Fixes:

• Fixed typo in NAME_PROPERTY_ASSIGNMENT_ERROR (issue #328)
• Silent expansion of Characteristic categories from str to OntologyAnnotation now supported. You can assign a string to a characteristic category and it will be automatically wrapped into an ontology annotation. (issue #332)
• Fixed duplicated in OntologyAnnotation ids (issue #381)
• Fixed raising of an error when correctly setting a comment value (issue #360)
• Assigned correct header "Data Transformation Name" for protocol type "sequence analysis data transformation" (commit b8f2f55)

Release v0.12.0 - Release Candidate 2

Release v0.12.0 - Release Candidate

Features:

• Added the isatools.create.connectors module. This module contains the generate_study_design_from_config function to generate an isatools.create.StudyDesign from a JSON configuration document containing metadata about the study design, including sampling and assay plan. These JSON documents can be generated by the Datascriptor application.(issues #355, #356, #377)
• Added support for observational variables to isatools.create.model.StudyArm (with the parameter source_characteristics) (PR #367)
• StudyDesign.generate_isa_study()now generates one Assay per assay type (via StudyDesign._generate_samples()). (issue #369, #373)
• The names of the assay nodes (extract, labelled, extract, protocol node, and data file) (issue #370, #373)
• isatools.create.model.ProductNode now has an attribute extension (e.g. a file extension fo a DataFile node) (issue #378)

Enhancements and Cleanup:

• Introduced support for (built-in, non-configurable) synonyms in protocol types. Protocol types have also been made case insensitive. (issue #382)
• Refactored constants and error messages for the create mode, introducing the modules isatools.create.constants and isatools.create.errors (issue #342, #368)
• Removed asterisk imports (*) from all the core modules (issue #368)
• Removed leftover print() statements (issue #374)
• overall test coverage increased from 67% to 75%. The increase is due both to an increase in the number of tests and the removal of some deprecated, experimental or spurious modules.

Breaking changes:

• isatools.create.models has been renamed to isatools.create.model (issue #368)
• IsaModelAttributeError has been replaced with AttributeError. We will favour standard Exceptions whenever applicable.

Bug Fixes:

• Fixed typo in NAME_PROPERTY_ASSIGNMENT_ERROR (issue #328)
• Silent expansion of Characteristic categories from str to OntologyAnnotation now supported. You can assign a string to a characteristic category and it will be automatically wrapped into an ontology annotation. (issue #332)
• Fixed duplicated in OntologyAnnotation ids (issue #381)
• Fixed raising of an error when correctly setting a comment value (issue #360)
• Assigned correct header "Data Transformation Name" for protocol type "sequence analysis data transformation"

Release v0.12.0 - Release Candidate

Release v0.12.0 - Release Candidate

Features:

• Added the isatools.create.connectors module. This module contains the generate_study_design_from_config function to generate an isatools.create.StudyDesign from a JSON configuration document containing metadata about the study design, including sampling and assay plan. These JSON documents can be generated by the Datascriptor application.(issues #355, #356, #377)
• Added support for observational variables to isatools.create.model.StudyArm (with the parameter source_characteristics) (PR #367)
• StudyDesign.generate_isa_study()now generates one Assay per assay type (via StudyDesign._generate_samples()). (issue #369, #373)
• The names of the assay nodes (extract, labelled, extract, protocol node, and data file) (issue #370, #373)
• isatools.create.model.ProductNode now has an attribute extension (e.g. a file extension fo a DataFile node) (issue #378)

Enhancements and Cleanup:

• Introduced support for (built-in, non-configurable) synonyms in protocol types. Protocol types have also been made case insensitive. (issue #382)
• Refactored constants and error messages for the create mode, introducing the modules isatools.create.constants and isatools.create.errors (issue #342, #368)
• Removed asterisk imports (*) from all the core modules (issue #368)
• Removed leftover print() statements (issue #374)
• overall test coverage increased from 67% to 76%. The increase is due both to an increase in the number of tests and the removal of some deprecated, experimental or spurious modules.

Breaking changes:

• isatools.create.models has been renamed to isatools.create.model (issue #368)
• IsaModelAttributeError has been replaced with AttributeError. We will favour standard Exceptions whenever applicable.

Bug Fixes:

• Fixed typo in NAME_PROPERTY_ASSIGNMENT_ERROR (issue #328)
• Silent expansion of Characteristic categories from str to OntologyAnnotation now supported. You can assign a string to a characteristic category and it will be automatically wrapped into an ontology annotation. (issue #332)
• Fixed duplicated in OntologyAnnotation ids (issue #381)
• Fixed raising of an error when correctly setting a comment value (issue #360)