Column Filtering

Column Filtering selects or excludes specific columns from the source data, allowing you to reduce the schema to only the fields needed in the destination table.

When to Use

Use Column Filtering when:

• Your source contains many fields but your downstream consumers need only a subset.
• You want to exclude sensitive or irrelevant columns before writing to the destination table.
• You need to reduce storage footprint by omitting unused fields.

Configuration

When adding a Column Filtering transformation to a Flow:

1. Select Column Filtering from the list of available transformations.
2. Choose the columns to include or exclude from the source schema.
3. The resulting schema reflects only the selected columns.

Usage Notes

• Column Filtering operates on the source schema. If the source schema evolves (e.g., new fields are added), the filter configuration determines whether new fields are included or excluded by default.
• This transformation is applied before other transformations in the pipeline, so downstream transformations operate on the filtered schema.