Skip to main content

AWS MSK Kafka

Continuously stream data directly from AWS Managed Apache Kafka (MSK) into Onehouse tables.

Click Sources > Add New Source > MSK Kafka. Then, follow the setup guide within the Onehouse console to configure your source.

Cloud Provider Support​

  • AWS: ✅ Supported
  • GCP: Not supported

Prerequisites​

  • Ensure that you have granted permission for MSK in the Terraform or CloudFormation configurations when you connected your cloud account.

Reading Kafka Messages​

Onehouse supports the following serialization types for Kafka message values:

Serialization Type (for message value)Schema RegistryDescription
AvroRequiredDeserializes message value in the Avro format. Send messages using Kafka-Avro specific libraries; vanilla AVRO libraries will not work.
JSONOptionalDeserializes message value in the JSON format.
JSON_SR (JSON Schema)RequiredDeserializes message value in the Confluent JSON Schema format.
ProtobufRequiredDeserializes message value in the Protocol Buffer format.
Byte ArrayN/APasses the raw message value as a Byte Array without performing deserialization. Also adds the message key as a string field.

Onehouse currently does not support reading Kafka message keys for Avro, JSON, JSON_SR, and Protobuf serialized messages.

Usage Notes​

  • If a message is compacted or deleted within the Kafka topic, it can no longer be ingested since the payload will be a tombstone/null value.
  • Ensure your MSK Cluster is in the same region as the Onehouse project. This will reduce costs for moving data across regions.

Networking and permissions checklist​

When source creation fails with errors like Failed to get topics ... Timed out waiting for a node assignment, or topics you expect to see are missing from the "Create Flow" topic picker, walk this checklist before opening a support ticket:

  • MSK security group inbound rules must permit the full Onehouse data-plane CIDR (e.g. 10.x.0.0/18) on the entire Kafka port range 9094-9098, not just one port. A common mistake is allowing only 9094 (TLS) and missing 9098 (IAM auth).
  • Kafka ACLs must grant the Onehouse principal DescribeTopic and Read on every topic you intend to ingest. Missing ACLs cause topics to silently disappear from the picker rather than producing a clear permission error.
  • AWS Glue Schema Registry rate limits can surface as schema validation timeouts when many Flows share a single registry. If you see 400 Rate exceeded from Glue, request a rate-limit increase from AWS support.

Topic-partition sizing​

Very high partition counts (thousands of partitions per topic) can push the broker's listOffsets call past Onehouse's 60-second default timeout — producing errors like Failed to get offsets by times in 60001ms — especially when broker CPU is saturated.

Recommendations:

  • Keep per-topic partition counts moderate. Thousands of partitions on a single topic is a code smell; consider sharding into multiple topics or right-sizing partitions for actual throughput.
  • If you genuinely need a high partition count, contact Onehouse support to tune the consumer timeout for the affected source.

Migrating or re-creating a topic​

The same caveat as Apache Kafka applies: deleting and re-creating an MSK topic in place resets broker offsets while leaving the Flow's checkpoint ahead, silently halting ingestion. Use a new versioned topic name and request a source switch via support.

Guide: Configure AWS VPC Peering and Test Connectivity​

  1. Create a VPC Peering entry between the requesting VPC (Onehouse) and the accepter VPC (your MSK cluster).

  1. Modify the MSK security group to allow "All Traffic" from the Onehouse EKS security group.

  1. Modify the Route Table for all the EKS VPCs and add a route to the MSK VPC CIDR.

  1. Modify the Route Table for the MSK VPC and add a route to the EKS VPC CIDR.

  1. Create one EC2 instance in each VPC. EC2-1 mimics the connection from Onehouse and is in one of the private subnets in the EKS VPC. The security group(s) should be the same as the ones attached to EKS cluster. EC2-2 mimics where the Kafka instance is deployed and is in one of the private subnets (or public subnet) where you deployed Kafka. The security group(s) should be the same as the ones attached to Kafka.

  2. Create a test path in AWS's Reachability Analyzer between those instance

  1. Analyze Path

Secrets Management (BYOS)​

If you are using Bring Your Own Secrets (BYOS), store your credentials in AWS Secrets Manager using the JSON formats below. See the Secrets Management documentation for setup instructions and tag requirements.

SASL Protocol​

Use your API key and API secret as the username and password.

{
  "username": "<value>",
  "password": "<value>"
}

TLS Protocol​

{
  "keystore_password": "<value>",
  "key_password": "<value>"
}