Skip to main content

Create a Flow

Create a Flow to ingest data into a new Onehouse table. To begin, open the Flows page in the Onehouse console and click Add New Flow.

Open Table Formats

Flows create Onehouse tables in the Apache Hudi table format. To add read-compatibility in Apache Iceberg and Delta Lake, simply add a OneTable catalog in your Flow.

Prequisites​

  • Ensure you have a running Managed Cluster in your project.
  • Add a data source in your project to use in the Flow.
  • Optionally, add any catalogs where you'd like to sync the new table.

Step 1: Select a Data Source​

Select the data source to ingest into a Onehouse table.

Step 2: Configure your Flows​

Set up the Flow's transformations, data quality validations, table keys, and more. Review all Flow configurations here.

Step 3: Select Destination​

  1. Select a Lake and Database where your Flow will create the new table.
  2. Optionally, select catalogs where Onehouse will regularly sync the new table's metadata.
Flows require a Managed Lake

Flows cannot be created on Observed Lakes — only on Managed Lakes. Selecting an Observed Lake will produce the error Stream capture can only be created on managed lakes. If you need to ingest into the same logical location as an existing Observed Lake, create a Managed Lake in a sibling folder; the same folder cannot be both Managed and Observed simultaneously.

Step 4: View the Table​

  1. After you've created the Flow, open it in the Onehouse console.
  2. Next to 'Destination', you can click to open the table created by the Flow.