Array encode error if optional key is not presented

[JessicaHsu]

I need to put a record into a array as an optional item.
It seems the optional key must be presented in this case.
Is this an expected behavior?

Elixir 1.8
OTP 22
:avro_ex 2.1

Schema

profile_schema = %{
    "type" => "record",
    "name" => "profile",
    "fields" => [
      %{"name" => "name", "type" => "string"},
      %{"name" => "age", "type" => ["null", "int"]},            
    ]
}

array_schmea = %{
      "type" => "array",
      "items" => [
        "null",
        "string",
        profile_schema
      ]
}

final_schema = AvroEx.decode_schema!(array_schmea, strict: true)

Usage

NG

profile = %{
  name: "Alice"
}

example = ["1","2", profile]

example_bin = AvroEx.encode!(final_schema, example) |> IO.inspect(label: "encode")
AvroEx.decode!(final_schema, example_bin) |> IO.inspect(label: "decode")

** (AvroEx.EncodeError) Schema Mismatch: Expected value of Union<possibilities=null|string|Record<name=profile>>, got %{name: "Alice"}
    (avro_ex 2.1.0) lib/avro_ex.ex:163: AvroEx.encode!/3
    #cell:k6mfb4za2ebwu6sux5ix32zdwrvnacar:27: (file)

OK

profile = %{
  name: "Alice",
  age: nil
}

example = ["1","2", profile]

example_bin = AvroEx.encode!(final_schema, example) |> IO.inspect(label: "encode")
AvroEx.decode!(final_schema, example_bin) |> IO.inspect(label: "decode")

[davydog187]

Hello thanks for the report! i verified this behavior using the following iex session.

With that being said, this is the intended behavior of the API. nullable fields are not optional, so you can't omit their keys. I believe this behavior is preferable as it ensures that you have the correct data shape.

Can you please explain your usecase?

# Avro EX #90

```elixir
Mix.install([
  {:avro_ex, "~> 2.1"}
])

Schema

profile_schema = %{
  "type" => "record",
  "name" => "profile",
  "fields" => [
    %{"name" => "name", "type" => "string"},
    %{"name" => "age", "type" => ["null", "int"]}
  ]
}

array_schmea = %{
  "type" => "array",
  "items" => [
    "null",
    "string",
    profile_schema
  ]
}

final_schema = AvroEx.decode_schema!(array_schmea, strict: true)

NG

profile = %{
  name: "Alice"
}

example = ["1", "2", profile]

example_bin = AvroEx.encode!(final_schema, example) |> IO.inspect(label: "encode")
AvroEx.decode!(final_schema, example_bin) |> IO.inspect(label: "decode")

OK

profile = %{
  name: "Alice",
  age: nil
}

example = ["1", "2", profile]

example_bin = AvroEx.encode!(final_schema, example) |> IO.inspect(label: "encode")
AvroEx.decode!(final_schema, example_bin) |> IO.inspect(label: "decode")

[JessicaHsu]

Hi thanks for the reply!

Actually, my data structure have some optional fields, and I thought "type" => ["null", "int"] would do the trick.
Since nullable fields are not optional, do you know how to define my profile_schema to make the age be an optional key so that I can encode the profile and omit the age? Also, I need the profile to be one of my array item, which means the array_schmea would contains profile in the end.

My use case:
I have an array which takes a payload. The payload could be different data structure which have several optional fields.