How to make Custom Connector to Ingest Data into Cloudera Platform through Impala

[toyotaknighd]

Hi, I currently trying to find and explore other ETL tools to replace our old Talend, I successfully installed Airbyte into my WSL2 environment to try around, but I notice that in Airbyte both on official connector or marketplace there isn't a connector to Cloudera.

The problem is that, In our current Talend setup, we rely on a Cloudera/Impala component, which allows us to directly interact with Impala on our Cloudera platform. This is an important part of our ETL workflows, but in Airbyte this connector doesn't exist

Since Airbyte does not appear to have a similar connector available, I would like to understand:

1. Is there any official or community-supported connector for Cloudera / Impala in Airbyte that I miss of?
2. If not, what would be the recommended approach to integrate Airbyte with Cloudera (e.g., via JDBC, custom connector, or intermediate storage)?
3. Has anyone successfully implemented Airbyte with Cloudera Impala in production?

Any guidance would be greatly appreciated.

[Aaron ("AJ") Steers (aaronsteers)]

Hi toyotaknighd, thank you for opening this discussion!

You are correct — there is currently no official or community-supported connector for Cloudera / Impala in the Airbyte connector catalog or marketplace. I searched through the full connector list and confirmed this.

Here are some recommended approaches to integrate Airbyte with your Cloudera platform:

Option 1: Build a Custom Destination Connector (Recommended)

Since Impala supports JDBC via the Cloudera JDBC Driver for Apache Impala, you can build a custom destination connector using the Airbyte Connector Development Kit (CDK). This would allow you to write data directly to Impala tables.

Steps:

1. Review the Connector Development documentation
2. Use the Python CDK to build a destination connector that connects to Impala via its JDBC or HiveServer2 Thrift interface
3. Publish it as a custom connector in your workspace

Option 2: Use Intermediate Storage (S3, GCS, HDFS)

A common pattern is to use Airbyte to land data in an intermediate storage layer that Cloudera can access:

1. Use an existing Airbyte destination connector to write data to S3, GCS, or HDFS (in Parquet, CSV, or JSON format)
2. Create Impala external tables pointing to that storage location
3. Query the data through Impala as usual

This avoids building a custom connector and leverages Impala's native support for external tables over cloud/distributed storage.

Option 3: Use the Connector Builder (for HTTP API sources)

If your use case also involves pulling data from Cloudera REST APIs, the Connector Builder provides a no-code/low-code UI for creating source connectors for HTTP-based APIs.

For community support and to connect with others who may have similar setups, I'd recommend joining the Airbyte Community Slack — the #connector-development channel is a great place to ask questions about building custom connectors.

---

Need more help? Join Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this URL.

This message is generated from Devin.ai (internal session link).

[toyotaknighd]

Thank you Aaron ("AJ") Steers (@aaronsteers) for the answer, I don't think the option 2 and option 3 will work in our case. Because

1. For Option 2, If I understand it, you want me to use the existing Airbyte destination connector to write data to S3, GCS, or HDFS, well doesn't this a separate problem? What I have is a cloudera complete with its Hadoop Server
2. For Option 3, its not also correct. What we need from our team is a connector that can write to our parquet and kudus file using Impala connector, read tables, load tables, your basic component functionalities that we used on Talend

I think the option No.1 is the most closest solution I can think of, however I'm not quite understand the flow to create this custom connector in the Airbyte. I previously tried making them in your Connection Builder menu on the Airbyte on Start from Scratch, but what I got when create new custom connector is

This stream relies on custom component code to execute, which is not currently supported in this deployment of Airbyte

Am I doing it wrong? Because what I understand from your answer on Option 1, I somehow need to rebuild the connector beside the Airbyte?