Schema Evolution

Onehouse supports the following forms of backward-compatible schema evolution for Flows writing to a destination table.

Flow failures

If a Flow fails due to invalid schema evolution, you can usually resolve it by fixing the schema in your data source or schema registry. After fixing the schema, you may need to Pause the Flow, then Resume to pick up the changes.

Supported Operations

The following schema evolution operations are supported on your Onehouse tables. These operations are supported for both top-level and nested fields.

Adding Columns

Behavior

New columns will be added in your destination table.

Usage notes

• New columns must be nullable.
• New columns must be added at the end of the schema (not in the middle).
• If a column with the same name and position was previously removed from the table, new records will automatically populate that existing column.
• New columns cannot start with "_" (e.g. "_row_key"). These are supported during initial creation of the table, but not in schema evolution.

Deleting Columns

Behavior

Deleted columns will remain in your destination table and their existing rows will not be deleted. The deleted columns will have nulls for all new records.

Usage notes

• Only nullable columns can be deleted.

Widening Column Types

Behavior

Column types can be evolved to a “wider” type that preserves the existing data in that column. Onehouse will attempt to write incoming records to the table using the super-type (i.e. the wider type) between the column in the incoming record and the corresponding column in the table.

For example, SuperType(Int, Long) = Long because Int can be converted to Long without losing any data, but not the other way around.

The matrix below shows the expected behavior for each type combination. Matrix rows are the type in the incoming record and matrix columns are the type in the Onehouse table. The value in the cell is the new type in the Onehouse table (the super-type). Cells marked "X" indicate that the incoming record is incompatible with the Onehouse table.

Incoming Type ↓ / Table Type →	Int	Long	Float	Double	String	Bytes
Int	Int	Long	Float	Double	String	X
Long	Long	Long	Float	Double	String	X
Float	Float	Float	Float	Double	String	X
Double	Double	Double	Double	Double	String	X
String	String	String	String	String	String	Bytes
Bytes	X	X	X	X	String	Bytes

Usage notes

• Booleans, Complex Types (Array, Map, etc.), and Logical Types (Decimal, Date, Timestamp, etc.) as defined by the Avro spec are not supported for Schema Evolution in either direction. However, they can be changed from a non-nullable to a nullable field of the same type.
• Schema Evolution is supported for changing the type within a Map or Array. For example, you can evolve an Array column from Int[] to Long[]. However, you cannot perform operations that modify the overall structure such as converting an Array to a Map or changing the structure of nested Complex Types.
• In the case of Bytes and String, both types can be converted to each other, so the table will accept either type without evolving (as shown in the matrix above).
• Any type can be evolved from non-nullable to nullable. Types cannot be evolved from nullable to non-nullable.

Examples

• Example 1: Incoming records of multiple compatible types
  1. Onehouse table has col1 of type Int
  2. Incoming record r1 has col1 of type Long → Onehouse table col1 type is widened to Long
  3. Incoming record r2 has col1 of type Int → Onehouse writes r2 as Long in the table
• Example 2: Incompatible types
  1. Onehouse table has col1 of type Int
  2. Incoming record r1 has col1 of type String → Onehouse fails the Flow or quarantines r1 (based on the Flow configs)

info

We recommend that you do not directly modify the schema of your destination table through DDL operations while a Flow is writing data to the table, as it may cause errors. Please reach out to Onehouse support if you need to do this.

Unsupported Operations

• Renaming tables or databases
• Renaming columns (except for the last column in the schema)
• Adding non-nullable columns
• Adding columns that are not at the end of the schema
• Adding columns that start with "_"
• Deleting non-nullable columns
• Non-backward-compatible type changes (see above)
• Type changes to or from Boolean, Complex Types (Array, Map, etc.), and Logical Types (Decimal, Date, Timestamp, etc.)
  • Changing from non-nullable to nullable of the same type is supported

Troubleshooting common failures

The errors below cover the most common reasons a Flow stops ingesting after a schema change. After fixing the root cause, Pause and Resume the Flow to pick up the corrected schema.

"Missing required field" or ArrayIndexOutOfBoundsException after adding a new field

Onehouse-supported additions require the new field to be (a) nullable, (b) appended at the end of the schema, and (c) not prefixed with _. Three patterns trip up Flows that look like they followed the rules:

• The new field was inserted in the middle of the schema rather than appended to the end. Move the field to the last position in the Avro/source schema.
• The producer still emits the field as required even though the registered schema says it is nullable. Update the producer to write the field as nullable, or stop writing it for records that have no value.
• Backfilling old offsets that pre-date the schema change. Old records lack the new field entirely and may trip an ArrayIndexOutOfBoundsException in the Avro decoder. Contact Onehouse support — replaying offsets older than the schema change typically needs a support-side adjustment.

"Cannot read TIME logical type" or ParquetDecodingException

Spark's vectorized Parquet reader does not support the Parquet TIME logical type (int64 representing time-of-day). Mixed-nullability schemas across files written by different engines or Hudi versions can also produce ParquetDecodingException with required != optional for the same column.

To resolve:

• Change the source column to a supported Spark type — typically Timestamp or Date.
• For mixed-nullability across writers, make the upstream column nullable at the writer that still emits it as required.
• If the type cannot change, contact Onehouse support — the vectorized reader can be disabled per Flow as a workaround.

Confluent JSON_SR: "Cannot parse <null> schema"

Open object types ("type": "object" with no properties) cannot be converted from JSON Schema to Avro and produce a null parsed schema. Either:

• Define explicit properties for the object in your JSON Schema, or
• Change the field to "type": "string" and parse the payload downstream using the Parse JSON transformation.

Note that adding properties is a backward-incompatible schema change in Confluent Schema Registry.

Schema inference fails on S3 sources: "Unable to find S3 schema"

Onehouse converts inferred schemas to Avro, and Avro forbids spaces, colons, hyphens, and leading digits in field names. Common offenders:

• Parquet columns with names like Agent Name (space).
• XML elements with namespace prefixes like ns2:Foo (colon).

Rename the offending columns or strip namespace prefixes at the source.

Further Reading

Read about practical schema evolution cases in our blog.