Skip to main content

Data Plane

Recent Data Plane releases — each version covers the Hudi engine and the OneHouse Dataplane. Showing the 10 most recent versions.

HUDI-2.22.2-spark35.DATAPLANE-1.199.2 — 2026-08-20​

OneHouse Dataplane​

Bug Fixes​

  • BigQuery meta sync with the COMPACTION policy now publishes the manifest as of the last completed compaction, eliminating duplicate records caused by premature manifest updates.
  • The last completed compaction instant is now correctly identified by instant time, fixing a regression that could cause compaction state to be misreported.
  • Clustering operations now read only the metadata they requested, fixing a correctness issue that could affect clustering on Hudi 1.x tables.
  • The columnar rewriter now correctly falls back when it encounters local-timestamp columns, preventing failures during data processing for affected table configurations.
  • The data-pending-to-sync metric no longer reports negative values when the source backlog is transiently zero.

General​

  • A new metadata table rebootstrap job is available to safely recover corrupted or inconsistent metadata tables, with durable pre-state backup, post-validation, and support for batch operation across multiple tables.
  • Oracle CDC ingestion via Debezium is now supported, including partial-update payloads, schema post-processing, and correct handling of temporal and variable-scale decimal types.
  • Kinesis data-stream ingestion is now available for tables running on Hudi 1.x.
  • Cross-account Kinesis stream ingestion is now supported via role assumption, enabling flows that read from Kinesis streams in a different AWS account.
  • Table services now correctly honor the configured meta sync trigger policy, so syncs fire only on the intended write operations (e.g., all writes or compaction only).
  • Incremental clustering plans now cap the number of input file groups, preventing oversized plans that could stall or overload clustering jobs.
  • Table sync emptiness checks now scan the timeline newest-first in parallel batches, significantly reducing latency for large tables.
  • Table sync no longer loads the full archived timeline to determine whether a table is empty, reducing overhead for tables with large archives.

HUDI-2.9.3-spark35.DATAPLANE-1.185.4 — 2026-08-10​

Hudi​

General​

  • Glue/Hive metadata sync no longer stalls on idle tables by correctly advancing the last-synced commit timestamp when it trails the active timeline.
  • The streamer now writes a non-partitioned sample table when estimating record sizes, fixing inaccurate estimates for non-partitioned datasets.
  • SQS message receive and delete operations in the S3 events source are now parallelized, reducing ingestion cycle latency.

HUDI-1.276.0.DATAPLANE-1.194.0 — 2026-08-04​

Hudi​

General​

  • Write operations now correctly fail when marker files cannot be durably flushed to storage, preventing silent data consistency issues.
  • The RECORD_LEVEL_INDEX index type is now accepted as a valid alias for PARTITIONED_RECORD_INDEX on Hudi 0.x, easing configuration compatibility across versions.
  • Ingestion flows no longer fail due to transient resource-release errors during source cleanup, improving pipeline reliability.
  • Metadata table validation no longer crashes on operation cancellation, improving stability during concurrent table operations.
  • Record index integrity validation against the files partition is now enabled by default on writes, catching index inconsistencies earlier.

OneHouse Dataplane​

General​

  • XTable format sync now correctly coerces Hudi column statistics values to the target table's schema types, preventing type mismatch errors during sync.
  • The "Data Pending to Sync" metric no longer reports negative values, ensuring accurate backlog visibility in monitoring dashboards.
  • Clustering operations now load only the metadata they require, reducing unnecessary I/O and improving clustering performance.
  • Oracle databases can now be used as a CDC source via Debezium, with full support for Oracle-specific data types including temporal values and variable-scale decimals.

HUDI-1.267.0.DATAPLANE-1.187.0 — 2026-07-20​

Hudi​

General​

  • BigQuery sync now supports configuring a separate billing project ID, enabling cost attribution to a dedicated project when syncing table metadata.
  • Fixed a serialization error that caused Protobuf UINT64 fields with default values to fail validation when using Avro 1.12.
  • Table size analysis now reports richer skew metrics and produces more accurate output by fixing a metadata-listing issue.

OneHouse Dataplane​

Security​

  • Resolved critical container image vulnerabilities identified in a recent security scan.

General​

  • Kinesis data-stream is now supported as an ingestion source.
  • Fixed an incorrect fallback in the columnar rewriter that caused failures when processing tables with local-timestamp columns.
  • Fixed incorrect column statistics being written during XTable format conversion by coercing Hudi source stat values to match the target table schema type.
  • Fixed negative values appearing in the data-pending-to-sync monitoring metric by correcting the backlog re-scan calculation.
  • The autoscaler no longer scales executors to zero while a shuffle or active commit is in progress, preventing job failures caused by lost shuffle data.

HUDI-1.252.0.DATAPLANE-1.181.0 — 2026-07-09​

OneHouse Dataplane​

New Features​

  • Ad-hoc SQL queries can now be executed directly against tables, enabling on-demand data exploration without a dedicated pipeline.

Bug Fixes​

  • Stale Kafka topic offsets are now correctly cleared when a topic is deleted, preventing inaccurate source lag estimates after topic removal.
  • A recurring error that caused table synchronization to fail when no prior writes existed on the timeline has been fixed, improving table service reliability.