Skip to main content

Flows

Flows ingest data from a Source into a new Onehouse table. Flows are fully-managed with built-in orchestration, scaling, and alerting.

Common Use Cases​

Key Capabilities​

  • Ingest: Incrementally ingest data from a wide variety of sources.
  • Transform: Perform low-code transformations on data in-flight.
  • Validate: Define data validations and quarantine bad data without breaking the ingestion pipeline.
  • Monitor: Easily view and search logs, monitor comprehensive dashboards and metrics, and receive alerts when Flows are broken or delayed.

Statuses​

Flows can have the following statuses:

  • Provisioning: The Flow is being set up. This status applies to Flows that require infrastructure provisioning before data ingestion can begin (e.g., CDC sources such as Postgres, MySQL, and Confluent CDC).
  • Running: The Flow is active. This state does not specify whether an active sync is in progress.
  • Delayed: The Flow's in-progress or most recent sync took longer than the specified delay threshold set with the flow.delayThreshold.numSyncIntervals advanced configuration.
  • Paused: The Flow is paused and will not perform syncs.
  • Failed: The Flow encountered errors after three attempts to sync data. While in the failed state, the Flow continues to retry on an interval that increases with each failure. If a sync succeeds during a retry, the Flow moves to the Running status.

Logs​

You can view logs for Flows directly in the Onehouse console. Open the Flow, then click the "Logs" tab.

info

Flow logs are retained for 7 days.

Usage Guidelines​

  • Flows run on Managed Clusters.
  • Flows will fail if the data volume of shuffle operations exceeds the available storage (disk space) for the project.
    • If you encounter situations that require additional storage (e.g. exploding an array with many elements), you can increase the Cluster's OCU Limit.

Troubleshooting​

Records are being quarantined and I can't tell why​

When a Flow quarantines records, the rejected rows are written to a versioned sibling location alongside the destination table:

<table_base_path>_quarantine/v<n>/

Query the quarantine table directly (via SQL, Athena, or any engine that can read Hudi) to inspect the error reason column attached to each rejected record. The most common reasons:

  • Missing required (non-nullable) field — the source record dropped a column the table requires.
  • Type mismatch — the source emitted a value that doesn't widen into the table column type. See Schema Evolution.
  • Transformer produced an invalid row — a SqlQueryBasedTransformer returned rows that don't conform to the target schema.

Once you identify the failing column and fix the source (or the transformer), Pause and Resume the Flow. The Flow will resume from the next checkpoint; quarantined records are not automatically re-ingested. Replay them by writing them back to the source if needed.

Console metadata vs. ground truth​

The Onehouse console caches metadata about Flows, Sources, and Tables. Immediately after creating, deleting, or re-creating a Flow — especially with the same name as a previous Flow — the console may briefly show stale state: a duplicated source table, a "Table not found" error on a freshly re-created Flow, a missing Flow on the list, or an empty commit history. The underlying ingestion is healthy in these cases; only the UI view is lagging.

When the console and the API disagree, treat the following as authoritative:

  • SHOW FLOWS and DESCRIBE FLOW <name> via the SQL Command API — these read the source-of-truth Flow registry, not the UI cache.
  • The commit timeline in your table's storage (<table_base_path>/.hoodie/) — every successful commit lands here, regardless of UI display.

If the UI continues to show stale state after a few minutes, contact Onehouse support and provide the Flow name and project ID so the console metadata can be re-synced.

Expected UI lag

Newly created Flows, new commits, and renamed sources can take a few minutes to appear in the console. Empty commit history on a brand-new Flow is normal until the first sync completes.