Skip to main content

Troubleshooting

Helpful tips for troubleshooting common issues you may encounter while working with Onehouse Jobs.

General Issues​

Database creation error​

# Example
Database creation is not allowed through Onehouse SQL. Please use the Onehouse Console or API to create databases.

Jobs currently do not create databases in your catalog.

  • If using the Onehouse catalog, you should use the CREATE DATABASE API command or create the database in the Onehouse console.
  • If using an external catalog, you should create the database directly in the external catalog.

Apache Iceberg Issues​

Apache Iceberg not properly installed​

# Data source issue example
Caused by: org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find the data source: iceberg.

# Class access issue example
java.lang.IllegalAccessError: class org.apache.iceberg.SparkDistributedDataScan cannot access its abstract superclass

These errors indicate Apache Iceberg is not installed and set up properly on your Cluster.

When you set your Cluster to use an external IRC, such as Glue IRC or Snowflake Open Catalog, Apache Iceberg will be pre-installed and pre-configured.

If you don't set the Cluster to use an external IRC, you should install Apache Iceberg as a dependency and add in your Job configuration:

spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

In your code, you must also specify the catalog name (i.e. run USE catalog.db instead of USE db) before accessing data.

Incorrect catalog setup​

# Warehouse path issue example
Cannot initialize HadoopCatalog because warehousePath must not be null or empty

The Iceberg REST Catalog (IRC) is not properly configured.

When you set your Cluster to use an external IRC, such as Glue IRC or Snowflake Open Catalog, the IRC configurations will be set up automatically.

If you don't set the Cluster to use an external IRC, you must explicitly provide the spark.sql.catalog.hadoop_catalog.warehouse Spark configuration when creating database.

Glue naming requirements​

# Naming issue example
org.apache.iceberg.exceptions.ValidationException: Cannot convert namespace performance-benchmark-iceberg-10tb to Glue database name, because it must be 1-252 chars of lowercase letters, numbers, underscore

When using Glue IRC as your catalog, you cannot include special characters or dashes in the database name.

Resource and Performance Issues​

Driver or executor OOM​

Out-of-memory failures on large Jobs typically indicate one of three problems: undersized executors, under-parallelized shuffles, or unsupported function fallbacks on the Quanton engine.

Sizing checklist:

  • For multi-terabyte ingests with heavy shuffle operations (e.g. posexplode, MERGE INTO), start with at least 64 GB driver memory and 64 GB executor memory and adjust upward based on the first run's Spark UI shuffle metrics.
  • Monitor disk pressure on executor nodes — if the Spark UI shows 70%+ disk usage during shuffle, increase the cluster's OCU Limit and consider using NVME-backed instance types.

Memory layout for the Quanton engine:

  • Quanton uses off-heap memory for columnar (Velox) execution, while the JVM heap holds Spark scheduling state. The recommended split is roughly 50/50 between heap and off-heap for the executor's total memory budget.
  • A user-set spark.executor.memoryOverheadFactor overrides the platform's auto-tuned split — only set this if you understand the trade-off.

Shuffle parallelism:

  • Default spark.sql.shuffle.partitions is often too low for large transforms. For shuffles producing hundreds of GB of intermediate data, increase to 2000–5000+ partitions, or add an explicit repartition() before the wide transform.
  • Heavy posexplode / unnest operations multiply input row counts; account for this when sizing shuffle partitions.

Disk space exhaustion ("Cannot create directory")​

Long-running Jobs with large shuffles can exhaust executor-node disk during shuffle spill. Onehouse rotates nodes based on disk usage thresholds, but if your Job pattern produces sustained heavy spill, consider:

  • Increasing the cluster's storage allocation by raising the OCU Limit.
  • Reducing shuffle volume by increasing partition counts or restructuring the query to avoid wide transforms.
  • Splitting a single long Job into smaller staged Jobs.

INSERT OVERWRITE OOM with posexplode / row_number​

INSERT OVERWRITE queries that explode arrays or apply window functions can OOM even with high executor memory because the explosion multiplies shuffle data volume. Mitigations:

  • Add an explicit repartition() before the explode.
  • Increase spark.sql.shuffle.partitions significantly (5000+ for billion-row inputs).
  • For very large transforms, consider moving the logic from a SQL query into a Spark Job where you have more control over memory and parallelism.

Quanton function fallback to Spark​

When a query uses a SQL function that Quanton does not yet support (for example, certain JSON or timestamp helpers), Quanton falls back to standard Spark execution for the affected operator. Fallbacks work correctly but are slower and consume more memory. If a previously fast query suddenly slows down after adding a function, check the query plan for fallback operators.

Connection and Cluster Issues​

TSocket read 0 bytes or Unexpected end of file from HS2​

Transport errors on JDBC connections to SQL clusters typically point to one of:

  • Missing Spark configs on a newly created cluster. Newly provisioned clusters can lack the production cluster's connection-stability tuning. Compare configs against a known-good cluster.
  • Exceeding the Thrift worker-thread limit. The default hive.server2.thrift.max.worker.threads can be saturated by many concurrent JDBC clients. Contact support to raise the limit if you legitimately need more concurrency.
  • Long-running query exceeding the load balancer idle timeout. See SQL endpoint troubleshooting for details.

Data Consistency Issues​

DELTA_FILE_NOT_FOUND_DETAILED on reads​

A Delta transaction log out-of-sync with S3 — typically caused by manual edits to _delta_log/ or a partial restore — produces this error. Recovery:

  1. Pause the Flow writing to the table.
  2. Open a support ticket; resolution requires a snapshot-sync rebuild of the Delta log.
  3. Resume the Flow after the rebuild.

See Open-Format Sync Troubleshooting for the full procedure.

INT32 != bigint column type mismatch on parquet reads​

This indicates schema evolution where the Hudi table's schema was widened (e.g. INT32 → INT64) but older base files were not rewritten. The Spark vectorized parquet reader is stricter than the row-based reader and fails on the mismatch.

Temporary workaround at the session level:

SET spark.sql.parquet.enableVectorizedReader=false;

Permanent fix: rewrite the affected base files via a clustering run or contact support for a one-time data rewrite.