AvroSchema implementation and schema registries

[gsson]

Hi!

I'm trying to use rsgen-avro to generate code for parsing Avro messages published on a Kafka topic.

The schema(ta) for the topic is stored in a Confluent schema registry.

When I check the latest schema for compatibility with older versions of the schema it succeeds if i

1. Fetch latest schema from registry (get-schema-by-version API)
2. Check compatibility with older versions with the returned schema verbatim in the request. (test-compatibility-against-a-particular-schema-subject-version API)

However, it fails if I:

1. Fetch latest schema from registry (as above)
2. Generate code with rsgen-avro; only non-default being implement_avro_schema(ImplementAvroSchema::CopyBuildSchema)
3. Check compatibility with older versions with the schema returned by either T::get_schema().canonical_form() or serde_json::to_string(&T::get_schema()).

As far as I can tell, all the errors I encounter is due to READER_FIELD_MISSING_DEFAULT_VALUE. The canonical form conversion strips default and such (which I think is according to spec), but it causes the generated schema not to be backwards compatible with earlier versions of the schema.

Similarly this:

    // `client` being the schema registry client
    let latest_schema = client.latest_schema().await?;
    let prev_schema = client.schema_version(latest_schema.version - 1).await?;

    let prev_parsed = apache_avro::Schema::parse_str(&prev_schema.schema)?;
    let latest_parsed = apache_avro::Schema::parse_str(&latest_schema.schema)?;

    // can_read(writer_schema, reader_schema) -- "I" only care about backwards compatibility, not full compatibility.
    eprintln!("{:?}", apache_avro::schema_compatibility::SchemaCompatibility::can_read(&prev_parsed, &T::get_schema()));
    eprintln!("{:?}", apache_avro::schema_compatibility::SchemaCompatibility::can_read(&prev_parsed, &latest_parsed));

prints

Err(Incompatible schemata! Field 'xxx' in reader schema does not match the type in the writer schema: Incompatible schemata! Field 'yyy' in reader schema does not match the type in the writer schema)
Ok(())

I fear a change to apache-avro might be required to fix this, but if versioned schema compatibility is considered a valid use case here, it might be worth addressing.

It might be worth noting that I am relatively new to the world of Avro, so I might have gotten something wrong :)

[gsson]

I'm quite willing to file upstream tickets and PRs and such if this seems like a sane thing to want :)

[martin-g]

I think you filled this issue in the wrong repository.
There is nothing related to rsgen-avro in your reproducer.
I think you have to report it at apache-avro's repo: https://github.com/apache/avro-rs

There were some improvements to the schema compatibility checks in the non-released 0.22 version, so please try with a local build of the main branch first.

[Kriskras99]

It might be partially related to rsgen-avro as ImplementAvroSchema::CopyBuildSchema used the independent_canonical_form function to get the JSON for the schema, which does strip defaults I think.

[gsson]

Yea, I think it's a little bit of both.

I don't think I can (reliably, neatly and easily) do what I want without a change to rsgen-avro, and you probably can't make that change without a change to apache-avro.

The Parsing Canonical Form for Schemas says attributes like "default" should be stripped, but for backwards compatibility I need a (non-canonical) schema that has defaults when I call eg. from_avro_datum(&writer_schema, &mut Cursor::new(payload), Some(&reader_schema))

Currently independent_canonical_form does this:

    pub fn independent_canonical_form(&self, schemata: &[Schema]) -> Result<String, Error> {
        let mut this = self.clone();
        this.denormalize(schemata)?;
        Ok(this.canonical_form())
    }

If the schema instead was generated using eg.

    pub fn foo(root: &Schema, schemata: &[Schema]) -> Result<String, Error> {
        let mut root = root.clone();
        root.denormalize(schemata)?;
        let json = serde_json::to_value(&root)
            .unwrap_or_else(|e| panic!("Cannot parse Schema from JSON: {e}"));
        Ok(json)
    }

that...might(?) work. At least it doesn't do the PCF stripping that Schema::canonical_form() does.

Unfortunately Schema::denormalize(...) is private.

[Kriskras99]

I've openeded a PR to make denormalize public: apache/avro-rs#470
We can then use that instead of independant_canonical_from in this crate.

[gsson]

And it's already merged! Thanks a lot!

Now I just have to wait for 0.22 :)