feat: add API client, auth, and backend abstraction layer#127
Open
lewisjared wants to merge 13 commits intomainfrom
Open
feat: add API client, auth, and backend abstraction layer#127lewisjared wants to merge 13 commits intomainfrom
lewisjared wants to merge 13 commits intomainfrom
Conversation
lewisjared
added a commit
that referenced
this pull request
Feb 6, 2026
lewisjared
added a commit
that referenced
this pull request
Feb 6, 2026
Add comprehensive support for the new bookshelf-platform API: - API client module (bookshelf.api): - BookshelfAPIClient for volumes, books, and authentication - Pydantic schemas for request/response validation - Custom error types (APIError, AuthenticationError, NotFoundError, ServerError) - Authentication module (bookshelf.auth): - Credential storage using platformdirs - Token expiry management - Environment variable overrides (BOOKSHELF_API_URL, BOOKSHELF_TOKEN) - CLI tool (bookshelf-client): - auth commands: login, logout, status - volumes commands: list, show - books commands: list, show - Rich terminal output with tables and panels - 76 unit tests covering all new functionality Dependencies added: httpx, click, rich, respx (dev)
Remove authentication requirement from volumes and books CLI commands to allow unauthenticated access to public datasets. Auth is still available and will be used if credentials exist. Also fix uv deprecation warning by migrating from tool.uv.dev-dependencies to dependency-groups.dev in all pyproject.toml files.
- Set default API URL to https://api.staging.climateresource.com.au/bookshelf/v1 - Remove /api prefix from endpoint paths (base URL includes full path) - Update all tests to match new URL structure
Migrate CLI authentication from username/password (POST /auth/token) to WorkOS OAuth with PKCE and Device Code flows. The old endpoint no longer exists after the platform migrated to WorkOS. - Add oauth.py with PKCE, Device Code, and token refresh flows - Extend Credentials with refresh_token, add auto-refresh on expiry - Replace login command: --device-code flag for headless environments - Add on_token_refresh callback to API client with 401 retry - Add refresh_token to TokenResponse schema
WorkOS requires a connection selector (connection_id, organization_id, or provider) in the authorize request. Without it, the flow fails with invalid_connection_selector.
Upgrade pydantic dependency from >=1.10.17 to >=2.0 and migrate all schema code to use native v2 APIs. This eliminates the pydantic.v1 compatibility layer and prepares the codebase for the API backend abstraction layer. Changes: - Update pydantic import from pydantic.v1 to pydantic in schema.py - Add explicit `= None` defaults to optional fields (url, doi, description) that were implicitly optional in v1 but required in v2 - Replace .dict() with .model_dump() across notebook.py, 15 notebooks, and docs - Replace .json() with .model_dump_json() in actions.py - Update test imports to use pydantic v2 ValidationError
…rces Introduce BookshelfBackend protocol with semantic operations (resolve_version, list_versions, fetch_datapackage, download_resource, list_volumes) and two implementations: - S3Backend: extracts existing S3 logic from shelf.py - APIBackend: wraps BookshelfAPIClient with response mapping Also extends ResourceSummary with optional download fields (download_url, hash, filename, timeseries_name, shape, content_hash) for future API-driven resource access.
The login command now uses WorkOS OAuth (authorization_code_flow) instead of username/password POST to /auth/token. Update all three login tests to mock the OAuth flow via monkeypatch instead of mocking HTTP endpoints with respx.
…nd tests BookShelf now delegates to self._backend (defaults to S3Backend for backward compatibility). shelf.py no longer imports requests.exceptions directly - error handling lives inside each backend. LocalBook.timeseries() and get_long_format_data() prefer download_url from resource descriptor, falling back to S3 URL construction. Adds 29 unit tests for S3Backend (10) and APIBackend (19) covering resolve_version, list_versions, fetch_datapackage, download_resource, list_volumes, private derivation from EditionInfo.status, and error mapping.
…e fallback Complete Phase 2 TODOs 2.7-2.10: - BOOKSHELF_BACKEND=api env var auto-selects APIBackend - OfflineError exception for clear offline diagnostics - Graceful fallback to cached data on ConnectionError/OSError - _find_cached_versions() scans local cache for fallback candidates - 12 new tests covering env var selection and offline behavior
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds REST API support to the bookshelf consumer package as the foundation for migrating away from direct S3 access. This PR includes:
API Client & Authentication
BookshelfAPIClientHTTP client wrapping the bookshelf-platform REST APIlogin,logout,status) and API browsing (volumes list/show,books list/show)Pydantic v2 Migration (Phase 0.5)
.dict()→.model_dump(),.json()→.model_dump_json()across core library and notebooksBackend Abstraction Layer (Phase 1)
BookshelfBackendprotocol with 5 semantic operations (resolve_version,list_versions,fetch_datapackage,download_resource,list_volumes)S3Backendextracting current S3 logic fromshelf.pyAPIBackendwrappingBookshelfAPIClientwith response mappingResourceSummaryextended with optional download fields for future API-driven resource accessWire Backend into BookShelf (Phase 2)
BookShelfnow delegates all data operations toself._backend(S3Backend by default)LocalBooksupportsdownload_urlfor direct resource downloads via API backendshelf.pyrefactored: removed inline S3 logic, all access goes through backend protocolBOOKSHELF_BACKEND=apienv var auto-selects APIBackend (withBOOKSHELF_API_URLandBOOKSHELF_TOKEN)ConnectionError/OSError, withOfflineErrorwhen no cache exists_find_cached_versions()scans local cache for fallback candidatesTest Fixes
/auth/tokenendpointAll existing tests pass with zero regressions (203 passed).
Checklist
Please confirm that this pull request has done the following:
changelog/