feat(prepro): Host name validation

[maverbiest]

Adding host validation to preprocessing

Implementation

Host validation is implemented across three new processing functions:

validate_host Takes unvalidated input, calls out to the taxonomy service to validate it, and returns a taxon ID when validation succeeds. If the input casts to an int, it is assumed to be a taxon id. Otherwise, it is assumed to be a scientific name. Responses to successful validations are cached in taxon_cache (see below).

It is possible that multiple taxa have the same scientific name. In these cases,
we return the tax_id of the most generic taxon (i.e., the one that's closest to
the root of the taxonomy).

If the taxon ID or host name does not exists, we return None and add a warning (if is_insdc_ingest_group) or an error (for everyone else).

@staticmethod
    def validate_host(
        input_data: InputMetadata,
        output_field: str,
        input_fields: list[str],
        args: FunctionArgs,
    ) -> ProcessingResult:

scientific_name_from_id takes a validated taxon ID and maps it to a scientific name. Responses to successful validations are cached in taxon_cache (see below).

    @staticmethod
    def scientific_name_from_id(
        input_data: InputMetadata,
        output_field: str,
        input_fields: list[str],
        args: FunctionArgs,
    ) -> ProcessingResult:

common_name_from_id takes a validated taxon ID and maps it to a common name. If the input taxon has a common name itself, that taxon is returned. If it does not, the nearest ancestor with a common name is returned instead. Responses to successful validations are cached in common_name_cache (see below).

    @staticmethod
    def common_name_from_id(
        input_data: InputMetadata,
        output_field: str,
        input_fields: list[str],
        args: FunctionArgs,
    ) -> ProcessingResult:

Caching

Successful requests to the taxonomy service are cached. Since the Number of distinct host organisms in a dataset are expected to be relatively constrained, this reduces the amount of network requests needed at the cost of keeping a small cache in memory.

The caching approach is implemented in the new class RequestCache:

class RequestCache:    
    """Class for caching requests to external services during preprocessing.

    Keys are the fully formatted URLs that have already been used to make sucessful requests.
    Values are requests.Response as they were returned by the service.
    """

This is now used to cache requests to the taxonomy service:

taxonomy_cache = RequestCache(max_size=64)

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by appropriate, automated tests.
• Any manual testing that has been done is documented (i.e. what exactly was tested?)

🚀 Preview: https://prepro-hostname-validatio.loculus.org

[anna-parker]

We tested on staging and this is working well - using these functions in PPX should be done only after a migration of the originally submitted metadata to handle cases where the scientific name is invalid. We should wait till #6254 is merged to do this