Integration with Amazon MSK failed

[Noam Tamim (noamtamim)]

I'm trying to use the Schema Registry with Amazon MSK (managed Kafka).
Running Schema Registry as docker as follows:

docker run -e 'SCHEMA_REGISTRY_KAFKASTORE_TOPIC=_mySchemas' \
  -e 'SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS=PLAINTEXT://my.cluster.kafka.us-east-1.amazonaws.com:9092' \
  -e 'SCHEMA_REGISTRY_HOST_NAME=localhost' \
  -p 8081:8081 \
  -it confluentinc/cp-schema-registry:7.5.0
# Also tried older versions such as 5.1.2, 5.4.0, 6.1.0

The server seems to start fine, ending with:

[2023-09-13 14:27:49,015] INFO Server started, listening for requests... (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain)

However, when I try to register a test schema (the first curl call from https://github.com/confluentinc/schema-registry#quickstart-api-usage-examples) - after 60 seconds, I get the following response:

{"error_code":50003,"message":"Error while forwarding register schema request to the leader"}

The server log shows multiple levels of stack traces with causes:

[2023-09-13 14:39:49,216] ERROR Failed to send HTTP request to endpoint: http://10.202.47.204:8081/subjects/Kafka-key/versions?normalize=false (io.confluent.kafka.schemaregistry.client.rest.RestService)
java.net.SocketTimeoutException: connect timed out
...
[2023-09-13 14:39:49,242] ERROR Request Failed with exception  (io.confluent.rest.exceptions.DebuggableExceptionMapper)
io.confluent.kafka.schemaregistry.rest.exceptions.RestRequestForwardingException: Error while forwarding register schema request to the leader
...
Caused by: io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryRequestForwardingException: Unexpected error while forwarding the registering schema request to http://10.202.47.204:8081
	at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.forwardRegisterRequestToLeader(KafkaSchemaRegistry.java:1183)
	at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.registerOrForward(KafkaSchemaRegistry.java:846)
	at io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource.register(SubjectVersionsResource.java:412)
	... 61 more
Caused by: java.net.SocketTimeoutException: connect timed out

The full stack trace is attached.

However, it's obvious Schema Registry does successfully communicate with Kafka: it creates the _mySchemas topic (which I set just to make it create a new non-default one) and it writes a NOOP message to it ({"keytype":"NOOP","magic":0}).

There's no networking error, so what can be the issue?