-
Notifications
You must be signed in to change notification settings - Fork 45
Intake-ESM Integration based on #1218 #2690
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
charles-turner-1
wants to merge
56
commits into
main
Choose a base branch
from
intake-esm
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
56 commits
Select commit
Hold shift + click to select a range
c129966
Add recognised intake-esm datastores on NCI systems to config_developβ¦
charles-turner-1 b1b76fb
Skeleton
charles-turner-1 dd73d1d
Playing around
charles-turner-1 ed1676b
Almost at a working IntakeDataset.load()
charles-turner-1 fa1ea2e
Working intake-esm implementation - probably still some kinks to ironβ¦
charles-turner-1 648f119
Working with multiple catalogues per project
charles-turner-1 2b91fec
Cleanup - mypy & ruff errors
charles-turner-1 c7b8ffb
Remove WIP
charles-turner-1 31b35cb
Update depenencies & dev environment
charles-turner-1 a8532a5
Pre-commit modifications
charles-turner-1 7e56959
Merge branch 'main' into intake-esm
charles-turner-1 568cb8d
Fixed most of codacy (mypy-strict?) gripes
charles-turner-1 91fee56
Fix typo
charles-turner-1 9d894b9
Beginning to work on Bouwe's comments (WIP)
charles-turner-1 59d0d02
Updates - restructured esmvalcore/data/intake following Bouwe's suggeβ¦
charles-turner-1 2050081
Reorder imports (ruff maybe?)
charles-turner-1 59e4205
Add `_read_facets` to intake configuration: see https://github.com/inβ¦
charles-turner-1 2527059
Add `merge_intake_seach_history` function (see https://github.com/intβ¦
charles-turner-1 4641965
Merge branch 'main' into intake-esm
charles-turner-1 1b26148
Merge branch 'main' into intake-esm
valeriupredoi b77d194
readd intake
valeriupredoi e131cfc
Merge branch 'main' into intake-esm
charles-turner-1 a53e140
Add `data.io.intake_esm.py, scaffold off `data.io.intake_esgf.py`
charles-turner-1 b84cf70
WIP
charles-turner-1 ef6fdba
Scaffold tests
charles-turner-1 f1b8f55
Remove /data/intake stuff, /config/_intake
charles-turner-1 ec17bfa
Pre-commit
charles-turner-1 81fc7dc
Merge branch 'main' into intake-esm
charles-turner-1 5417f6c
Nearly there I think - all tests passing. Hopefully CI can tell me whβ¦
charles-turner-1 d988b17
Pre-commit
charles-turner-1 8533644
Remove old intake-esm file
charles-turner-1 1c41a35
Merge branch 'main' into intake-esm
charles-turner-1 15a79f5
Merge branch 'main' into intake-esm
charles-turner-1 b7ceeea
Sort keys when finding data - should guarantee order stability
charles-turner-1 6dd736f
Change path import style to match `/tests/integration/preprocessor/_iβ¦
charles-turner-1 5403ac8
Revert ugly type ignore stuff
charles-turner-1 27aa007
Un-ignore the intake-esm data ncfiles
charles-turner-1 b911086
Merge branch 'main' into intake-esm
charles-turner-1 ce4b2eb
- Pass through quiet arg correctly, remove special time range parsingβ¦
charles-turner-1 e731b2e
Merge branch 'main' into intake-esm
charles-turner-1 664044a
Remove comment
charles-turner-1 b9fccda
Merge branch 'intake-esm' of github.com:ESMValGroup/ESMValCore into iβ¦
charles-turner-1 67ea7fd
Remove line subsetting variables - we're using paths anyway, so subseβ¦
charles-turner-1 ecb5057
Add a `time_separator` field (needed for flexibility in how datastoerβ¦
charles-turner-1 a891762
Allow catalog into constructor
charles-turner-1 f7396e4
Add sample config
charles-turner-1 0592d08
Merge branch 'main' into intake-esm
charles-turner-1 758f43a
Add description to intake-esm config file
charles-turner-1 d26a52b
Merge branch 'intake-esm' of github.com:ESMValGroup/ESMValCore into iβ¦
charles-turner-1 7c71929
Merge branch 'main' into intake-esm
charles-turner-1 11e3fa7
Merge branch 'main' into intake-esm
charles-turner-1 e90ed39
- Update tests & functionality to use pangeo cloud catalog and pass tβ¦
charles-turner-1 08153dd
Update data-intake-esm config.yml & quickstart config docs
charles-turner-1 cbbc4fb
Add configs for various locations
charles-turner-1 2c74ff2
update docs for intake-esm
charles-turner-1 543226c
Remove duplicate key
charles-turner-1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| .. _intake_esm_api: | ||
|
|
||
| esmvalcore.io.intake_esm | ||
| ========================= | ||
|
|
||
| Read data from intake-esm catalogs. | ||
|
|
||
| This module provides data source classes integrating `intake-esm` catalogs | ||
| with ESMValCore. See the Quickstart `Data sources <config-data-sources>`_ | ||
| page (the "Cloud" subsection) for examples and usage. | ||
|
|
||
| .. automodule:: esmvalcore.io.intake_esm | ||
| :members: | ||
| :undoc-members: | ||
| :show-inheritance: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Documentation example: intake-esm catalog (AWS) | ||
| # Use `esmvaltool config copy data-intake-esm-aws.yml` to copy this example | ||
| # to your configuration directory, then update the `catalog` URL if needed. | ||
| projects: | ||
| CMIP6: | ||
| data: | ||
| aws-catalog: | ||
| type: esmvalcore.io.intake_esm.IntakeEsmDataSource | ||
| facets: | ||
| activity: "activity_id" | ||
| dataset: "source_id" | ||
| ensemble: "member_id" | ||
| exp: "experiment_id" | ||
| institute: "institution_id" | ||
| grid: "grid_label" | ||
| mip: "table_id" | ||
| short_name: "variable_id" | ||
| version: "version" | ||
| timerange: "time_range" | ||
| time_separator: "/" | ||
| catalog: "https://<AWS_INTAKE_ESM_CATALOG_URL>" | ||
21 changes: 21 additions & 0 deletions
21
esmvalcore/config/configurations/data-intake-esm-dkrz-cloud.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Documentation example: intake-esm catalog (DKRZ - cloud access) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The DKRZ catalogs appear to be for an older version of intake-esm, probably easiest to drop them for now and add them later |
||
| # Use `esmvaltool config copy data-intake-esm-dkrz-cloud.yml` to copy this example | ||
| # to your configuration directory, then update the `catalog` URL if needed. | ||
| projects: | ||
| CMIP6: | ||
| data: | ||
| dkrz-cloud-catalog: | ||
| type: esmvalcore.io.intake_esm.IntakeEsmDataSource | ||
| facets: | ||
| activity: "activity_id" | ||
| dataset: "source_id" | ||
| ensemble: "member_id" | ||
| exp: "experiment_id" | ||
| institute: "institution_id" | ||
| grid: "grid_label" | ||
| mip: "table_id" | ||
| short_name: "variable_id" | ||
| version: "version" | ||
| timerange: "time_range" | ||
| time_separator: "/" | ||
| catalog: "https://<DKRZ_INTAKE_ESM_CATALOG_URL>" | ||
21 changes: 21 additions & 0 deletions
21
esmvalcore/config/configurations/data-intake-esm-dkrz-disk.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Documentation example: intake-esm catalog (DKRZ - on-disk access) | ||
| # Use `esmvaltool config copy data-intake-esm-dkrz-disk.yml` to copy this example | ||
| # to your configuration directory, then update the `catalog` path. | ||
| projects: | ||
| CMIP6: | ||
| data: | ||
| dkrz-disk-catalog: | ||
| type: esmvalcore.io.intake_esm.IntakeEsmDataSource | ||
| facets: | ||
| activity: "activity_id" | ||
| dataset: "source_id" | ||
| ensemble: "member_id" | ||
| exp: "experiment_id" | ||
| institute: "institution_id" | ||
| grid: "grid_label" | ||
| mip: "table_id" | ||
| short_name: "variable_id" | ||
| version: "version" | ||
| timerange: "time_range" | ||
| time_separator: "/" | ||
| catalog: "/path/to/dkrz/intake-catalog.json" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| # Documentation example: intake-esm catalog (GCS) | ||
| # Use `esmvaltool config copy data-intake-esm-gcs.yml` to copy this example | ||
| # to your configuration directory, then update the `catalog` URL. | ||
| projects: | ||
| CMIP6: | ||
| data: | ||
| gcs-catalog: | ||
| type: esmvalcore.io.intake_esm.IntakeEsmDataSource | ||
| facets: | ||
| activity: "activity_id" | ||
| dataset: "source_id" | ||
| ensemble: "member_id" | ||
| exp: "experiment_id" | ||
| institute: "institution_id" | ||
| grid: "grid_label" | ||
| mip: "table_id" | ||
| short_name: "variable_id" | ||
| version: "version" | ||
| timerange: "time_range" | ||
| catalog: "https://storage.googleapis.com/cmip6/pangeo-cmip6.json" | ||
| CMIP5: | ||
| data: | ||
| gcs-catalog: | ||
| type: esmvalcore.io.intake_esm.IntakeEsmDataSource | ||
| facets: | ||
| activity: "activity_id" | ||
| dataset: "source_id" | ||
| ensemble: "member_id" | ||
| exp: "experiment_id" | ||
| institute: "institution_id" | ||
| grid: "grid_label" | ||
| mip: "table_id" | ||
| short_name: "variable_id" | ||
| version: "version" | ||
| timerange: "time_range" | ||
| catalog: "https://storage.googleapis.com/cmip6/pangeo-cmip5.json" | ||
| CMIP3: | ||
| data: | ||
| gcs-catalog: | ||
| type: esmvalcore.io.intake_esm.IntakeEsmDataSource | ||
| facets: | ||
| activity: "activity_id" | ||
| dataset: "source_id" | ||
| ensemble: "member_id" | ||
| exp: "experiment_id" | ||
| institute: "institution_id" | ||
| grid: "grid_label" | ||
| mip: "table_id" | ||
| short_name: "variable_id" | ||
| version: "version" | ||
| timerange: "time_range" | ||
| catalog: "https://storage.googleapis.com/cmip6/pangeo-cmip3.json" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Documentation example: intake-esm catalog (DKRZ - on-disk access) | ||
| # Use `esmvaltool config copy data-intake-esm-dkrz-disk.yml` to copy this example | ||
| # to your configuration directory, then update the `catalog` URL if needed. | ||
| projects: | ||
| CMIP6: | ||
| data: | ||
| dkrz-disk-catalog: | ||
| type: esmvalcore.io.intake_esm.IntakeEsmDataSource | ||
| facets: | ||
| activity: "activity_id" | ||
| dataset: "source_id" | ||
| ensemble: "member_id" | ||
| exp: "experiment_id" | ||
| institute: "institution_id" | ||
| grid: "grid_label" | ||
| mip: "table_id" | ||
| short_name: "variable_id" | ||
| version: "version" | ||
| timerange: "time_range" | ||
| time_separator: "/" | ||
| catalog: "/path/to/nci/intake-catalog.yml" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Replace by the actual URL?