Convert CDC Data

Description

Updates records in a table using CDC data. Reads a Spark struct of "before" and "after" values and updates records based on insert or update logic.

Supported CDC formats:

• PostgresSQL (Debezium)
• MySQL (Debezium)
• Oracle (Debezium)
• MongoDB (Atlas)

Parameters

• CDC Format: Specify the format of the CDC data.
• Destination Table Schema (only for Atlas format): Specify the final schema of the destination table after all transformations have been applied.

Input Requirements

Input dataset should follow exact same schema which Debezium or Atlas connector generates.

Expected Output

• Debezium Formats: Transformation will read the CDC data and extract the "before" values for deletes or "after" values for inserts/updates. Metadata from the transaction will also be added as fields in the row. Rows will be deleted in the target table when row has op = d.
• Atlas Formats: Transformation will read the CDC data and produce a struct with fields as the "before" values for deletes or "after" values for updates. Rows will be deleted in the target table when row has operationType = delete.

Example

// Input
// CDC Format: PostgresSQL
{
  "schema": {...},
  "payload": {
    "before": {
    "id": 123,
    "name": "bill",
    "rides": 5
  },
    "after": {
      "id": 123,
      "name": "bill",
      "rides": 7
    },
    "source": {
      "version": 123,
      "connector": "my_connector"
      ...
    }
  },
  "op": "u",
  "ts_ms": 1589362330904
}

// Output
{
  "id": 123,
  "name": "bill",
  "rides": 7
  // Metafields specific to the CDC source (PostgresSQL, MySQL, etc.)
  "_change_operation_type" // Type: String
  "_upstream_event_processed_ts_ms" // Type: String
  "db_shard_source_partition" // Type: String
  "_event_origin_ts_ms" // Type: Long
  "_event_tx_id" // Type: Long
  "_event_lsn" // Type: Long
  "_event_xmin" // Type: Long
}

Tips

• Ensure the Write Mode is set to Mutable if you want to update records in the table using CDC logs. If you prefer to land the raw CDC logs in the table without updating records, you can use Append-only Write Mode to improve write performance

Generated fields available downstream

The Convert CDC Data transformation injects metadata columns into the post-transformation schema. These fields are then selectable in downstream UI pickers (record key, partition key, etc.) once the schema is registered:

Field	Type	Available for
_change_operation_type	String	Debezium (Postgres, MySQL, Oracle), Atlas
_upstream_event_processed_ts_ms	String	Debezium
db_shard_source_partition	String	Debezium
_event_origin_ts_ms	Long	Debezium
_event_tx_id	Long	Debezium
_event_lsn	Long	Debezium (Postgres)
_event_xmin	Long	Debezium (Postgres)
_debezium_metadata (struct)	Struct	All Debezium formats

If a field you expect is missing from the record-key or partition-key dropdown, verify that:

1. The Convert CDC Data transformation is positioned before the configuration step where you're picking the key.
2. The Flow has produced at least one commit so the post-transformation schema is registered. New Flows may need to start ingesting before all generated fields become selectable.

Partial updates and unavailable values

Some CDC sources emit partial updates — records that include only the changed columns rather than the full row. The most common cases:

• Hudi's PartialUpdateAvroPayload treats null in a field as "no change" and merges only non-null columns into the existing record.
• Postgres with replica identity set to default (rather than full) emits __debezium_unavailable_value as a sentinel for columns that did not change, instead of emitting null.

If your downstream table is showing the literal string __debezium_unavailable_value in update events, your source's replica identity does not match what PartialUpdateAvroPayload expects. Either:

• Switch the Postgres table's replica identity to FULL so updates carry every column, or
• Add a custom transformation before Convert CDC Data that replaces __debezium_unavailable_value with actual null so the partial-update logic treats those fields as "no change".