Skip to main content

Schema Registry

Overview​

A Schema Registry can be added to a Source in Onehouse. When you add a Schema Registry to a Source, schemas from that Schema Registry will be available for all Flows using the Source.

How Schema Registries are used​

Schemas from your Schema Registry are used while creating Flows:

  • Some Sources (e.g. Apache Kafka) require a schema for reading the data, and some Sources (e.g. Amazon S3) allow you to optionally specify a schema for the incoming data.
  • Schemas can be used for Schema Validation in Data Quality Validations for any Source.

See a specific Source's docs to understand if a schema is required.

Schema Registry Options​

File-Based Schema Registry​

With a File-Based Schema Registry, Onehouse reads your schema files from a cloud storage bucket. Schema files should be in the AVRO schema format (see examples) stored as .avsc.

Important notes:

  • You must grant Onehouse access to the bucket containing your schema files.
  • Only schema files directly under schema path can be used in Flows. For example, your schema should be under <base_path>/schema.avsc, not <base_path>/folder/schema.avsc.

Schema file types supported:

  • Avro: Use the file extension .avsc and match the format defined defined here in the Apache Avro spec.
  • XSD: If you are ingesting XML data would like to use XSD schemas, our team can convert these schemas to AVRO format. Please contact Onehouse support.

Proto JAR Schema Registry​

You can use a Proto JAR file-based schema registry for Kafka sources with messages in the Proto (Protocol Buffer) format. Follow these steps to set it up:

  1. Protocol Buffer schemas should use the file extension .proto and match the format defined in the proto3 spec.
  2. Create a JAR from the compiled .proto files. This JAR will contain the Java classes generated by your schemas.
  3. In the Onehouse console, navigate to Settings > Integrations > Manage JARs, then upload the JAR.

AWS Glue Schema Registry​

Onehouse can read, write to and manage an existing Glue Schema Registry within your AWS account.

Note: You can create a Glue Schema Registry through AWS Console UI or AWS CLI or SDK.

Confluent Schema Registry​

Onehouse can read and manage an existing Confluent Schema Registry. You can also use schemas with Schema Contexts.

JSON_SR limitation: open object types

JSON Schema fields declared as "type": "object" with no properties cannot be converted to Avro and will fail with Cannot parse <null> schema. Either define explicit properties for the object, or change the field to "type": "string" and parse the payload downstream using the Parse JSON transformation. Note that adding properties is a backward-incompatible schema change in Confluent.

Domain whitelist for schema registry endpoints

If schema registry creation returns domain not allowed, the registry's hostname is not on the Onehouse allowlist. Contact support with the registry endpoint so it can be added.

Google Managed Schema Registry​

Onehouse can read, write to, and manage an existing Google Managed Schema Registry (part of Google Cloud Managed Service for Apache Kafka) within your GCP project.

You will provide the schema registry URL in the form:

https://managedkafka.googleapis.com/v1/projects/<project>/locations/<region>/schemaRegistries/<registry>

Note: You can create a Google Managed Schema Registry through the GCP Console or the gcloud CLI.

Secrets Management (BYOS)​

If you are using Bring Your Own Secrets (BYOS) with Confluent Schema Registry, store your credentials in AWS Secrets Manager or Google Cloud Secret Manager using the JSON format below. See the Secrets Management documentation for setup instructions and tag requirements.

{
  "api_key": "<value>",
  "api_secret": "<value>"
}