Table Services

Overview

Onehouse Table Services automatically optimize your table for reading and writing performance. This document will describe how to configure and monitor services that run on your tables.

When you create a flow, cleaning and compaction table services are automatically created for the table Onehouse is managing. You cannot turn off cleaning and compaction for Onehouse managed tables, but you can modify their settings.

If you register External Tables and follow the prerequisites documented, you can also configure Onehouse table services to run on the tables that you are writing with your own external pipelines.

Most table services described in these pages operate on Apache Hudi tables. Clustering additionally runs on native Apache Iceberg tables; the trigger, frequency, and monitoring surfaces are the same, and what differs is how the result is recorded on storage. See Table Service Offerings for per-service format support and for Iceberg-specific behavior.

If you are also using Iceberg or Delta Lake, the optimizations applied to the tables benefit all formats. Optimizing file sizes, partitions, implementing sort keys, cleaning up versioned files, etc. translate into benefits for all formats.

Table Services Overview Page

When you open the Onehouse Console you can navigate to the 'Table Services' page which will show you the status of all table services running across all of your tables in your project. Use the search bar or the filters to quickly find the services you are interested in monitoring.

On this page you can see the current status of the table service:

• Active — the table service is ready to run.
• Enabled — the table service is configured and enabled.
• Running — the table service is currently executing.
• Failed — the table service failed the most recent execution attempt.
• Paused — the table service is paused and not actively running.

Click on any table service to load the table services Details Page

Table Services Details Page

On the table services details page you can see fine-grained detail about the service. The Status tab shows basic statistics and a chart showing the table service performance over time.

The Setup tab shows you the configurations set for this table service. You can modify the settings by clicking the "Edit" button on the far right.

The History tab shows you the history of all actions performed by this table service. You can drill into more details with the Details link on the far right.

Trigger Mode

For all the table services, we offer Automatic and On Demand trigger modes.

• Automatic mode will run the service (Clustering, Compaction and Cleaning) automatically based on the configured commit frequency. For MetaSync, the service will run automatically for each commit in the table.
• On Demand mode will allow you to manually trigger the service using the RUN SERVICE IN TABLE API or Actions -> Trigger button on the table service page.

note

Apache Iceberg tables run on the Automatic trigger only. On Demand mode for Iceberg tables is coming soon.

Frequency Configuration

Table services support two frequency modes for scheduling execution:

Commit-Based Frequency

Run the service after every N commits to the table. This is the default mode. For example, setting the frequency to 4 commits means the service runs after every 4th commit.

note

For Apache Iceberg tables, this frequency is expressed in snapshots rather than commits — the console field is labelled Number of snapshots. Setting it to 1 plans a run after every new snapshot on the table.

Time-Based Schedule

Run the service on a recurring time schedule. Choose from the following intervals:

• Hourly — specify how often (e.g., every 2 hours).
• Daily — specify a start time (hour and minute).
• Weekly — specify days of the week and a start time.
• Monthly — specify a day of the month and a start time.

info

Time-based scheduling may not be available for all projects. The Metadata Sync service does not use frequency settings — it runs automatically on each commit.

Table Service Offerings

Details about each table service, its settings, and its behavior on each table format are documented in Table Service Offerings:

Service	Purpose
Clustering	Optimize file sizes and sort data to improve query performance.
Compaction	Merge incoming data into the table for MERGE-ON-READ tables.
Cleaning	Reclaim storage by removing old committed data.
Metadata Sync	Sync the table to connected catalogs.

Sizing and contention

Table services share the Services Cluster with the Flows writing to the same tables. When a heavy table service runs alongside high-throughput ingestion, the two can compete for driver and executor resources.

Symptoms of contention

• Flow commit times grow from minutes to 1–3 hours after enabling or scaling up a table service.
• The cluster driver pod restarts under sustained CPU pressure.
• Clustering or compaction enters a Pending state and stays there while writes pile up.
• DROP PARTITION or other DDL fails with errors about a pending clustering or compaction instant.

Mitigations

• Isolate clustering on a separate Services Cluster for tables with high write volume. Clustering is the most resource-intensive table service and benefits most from dedicated capacity.
• Lower the clustering frequency, or switch from Automatic to On Demand trigger mode and run clustering during quieter ingestion windows.
• Scale the Services Cluster up by increasing its OCU Limit — but verify whether the cluster needs a restart for the new limit to take effect.
• Pause the conflicting service while you investigate. Pausing clustering releases its pending instants after the next commit advances the timeline, which also unblocks DDL like DROP PARTITION.

Cluster capacity limits

Each Services Cluster has a combined limit on the number of concurrent Flows and table services it can host, based on the driver instance type. If you approach the limit, table services may queue or fail to start. Distribute workloads across multiple Services Clusters as you grow. Contact Onehouse support for the exact limit on your driver class.

Diagnosing a Pending service that never runs

If a table service shows Pending indefinitely via DESCRIBE TABLE_SERVICE, the most common cause is that the lock provider (DynamoDB or Zookeeper) is unreachable from the Services Cluster — usually a security group or VPC misconfiguration. Verify lock-provider connectivity from the cluster's network before scaling resources.

Record-level index on large tables

The Hudi record-level index (RLI) improves point-lookup and delete performance, but for very large tables (multi-million rows and above) the metadata commit at the end of an RLI-enabled service can exceed Spark's default RPC message size (spark.rpc.message.maxSize, 128MB).

• Best practice: enable RLI at table creation time, not after the fact. Adding RLI to a large existing table requires sizing the metadata layer up front; bolting it on later can hit the RPC limit.
• If you must enable RLI on a large existing table, contact support — the underlying hoodie.properties may need tuning to accommodate the metadata commit size.
• If you see TBLPROPERTIES advertising hoodie.metadata.record.index.enable=true despite explicitly disabling RLI, check whether downstream queries are re-setting the property; Onehouse does not auto-enable RLI silently.