adb-kafka

ADB - Kafka Single VM Demo environment

This template provisions a single VM and Azure Databricks workspace, installation of Kafka service is a manual step. Major components to deploy include:

• 1 Vnet with 3 subnets (2 for Databricks, 1 for Kafka VM)
• 1 Azure VM (to host Kafka and Zookeeper services), with port 9092 exposed to other devices in same VNet (allowed by default NSG rules).
• 1 VNet injected Azure Databricks Workspace
• NSGs for Databricks and Kafka subnets

Folder Structure

.
├── main.tf
├── outputs.tf
├── data.tf
├── providers.tf
├── variables.tf
├── vnet.tf
├── workspace.tf
├── terraform.tfvars
├── images
├── modules
    ├── general_vm
        ├── main.tf
        ├── outputs.tf      
        ├── providers.tf
        ├── variables.tf

terraform.tfvars is provided as reference variable values, you should change it based on your need.

Getting Started

Step 1: Preparation

Clone this repo to your local, and run az login to interactively login thus get authenticated with azurerm provider.

Step 2: Deploy resources

Change the terraform.tfvars to your need (you can also leave as default values as a random string will be generated in prefix), then run:

terraform init
terraform apply

This will deploy all resources wrapped in a new resource group to your the default subscription of your az login profile; you will see the public ip address of the VM after the deployment is done. After deployment, you will get below resources:

Step 3: Configure your VM to run Kafka and Zookeeper services

At this moment, you have a vanilla VM without any bootstraping performed. We are to manually log into the VM and install Kafka and Zookeeper services.

The VM's private key has been generated for you in local folder; replace the public ip accordingly. SSH into VM by (azureuser is the hardcoded username for VMs in this template):

ssh -i <private_key_local_path> azureuser@<public_ip>

Now you should follow this guide from DigitalOcean to install Kafka on the VM. Note that a few commands need to be updated:

1. When downloading the kafka binary, go to https://kafka.apache.org/downloads.html and copy the latest binary link and replace it here:

curl "https://downloads.apache.org/kafka/3.3.2/kafka_2.12-3.3.2.tgz" -o ~/Downloads/kafka.tgz

1. When testing your Kafka installation, --zookeeper is deprecated, use --bootstrap-server instead:

~/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic TutorialTopic

At the end of the guide, you should have a running Kafka service on your VM. You can test it by running the following command:

sudo systemctl status kafka

Step 4: Integration with Azure Databricks

Now your Kafka Broker is running; let's connect to it in Databricks. We first create a topic TutorialTopic2 in Kafka via your VM's Command Line:

~/kafka/bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic TutorialTopic2

Then we can write from Spark DataFrame to this topic; you can also test the connection by telnet vm-private-ip 9092 first.

Read from this topic in another stream job:

Requirements

Name	Version
terraform	>= 1.9.0
azurerm	>=4.0.0

Providers

Name	Version
azurerm	>=4.0.0
external	n/a
http	n/a
random	n/a

Modules

Name	Source	Version
kafka_broker	./modules/general_vm	n/a

Resources

Name	Type
azurerm_databricks_workspace.this	resource
azurerm_network_security_group.this	resource
azurerm_network_security_group.vmnsg	resource
azurerm_resource_group.this	resource
azurerm_subnet.private	resource
azurerm_subnet.public	resource
azurerm_subnet.vm_subnet	resource
azurerm_subnet_network_security_group_association.example	resource
azurerm_subnet_network_security_group_association.private	resource
azurerm_subnet_network_security_group_association.public	resource
azurerm_virtual_network.this	resource
random_string.naming	resource
azurerm_client_config.current	data source
azurerm_resource_group.this	data source
external_external.me	data source
http_http.ipinfo	data source

Inputs

Name	Description	Type	Default	Required
dbfs_prefix	n/a	string	n/a	yes
rglocation	n/a	string	n/a	yes
spokecidr	n/a	string	n/a	yes
subscription_id	Azure Subscription ID to deploy the workspace into	string	n/a	yes
workspace_prefix	n/a	string	n/a	yes
cidr	n/a	string	"10.179.0.0/20"	no
create_resource_group	Set to true to create a new Azure Resource Group. Set to false to use an existing Resource Group specified in existing_resource_group_name	bool	true	no
existing_resource_group_name	Specify the name of an existing Resource Group only if you do not want Terraform to create a new one	string	""	no

Outputs

Name	Description
azure_resource_group_id	The Azure resource group ID
pip	Public IP of the Kafka broker
workspace_id	The Databricks workspace ID
workspace_url	The Databricks workspace URL