Skip to main content

Lakes, Databases, and Tables

Overview​

Tables in Onehouse are organized under the following hierarchy: Lake > Database > Table.

Lakes​

Lakes are a logical entity for organizing your Onehouse tables. You can create a lake in an existing cloud storage bucket and directory (lakes themselves do not create a new directory). Make sure that you've granted Onehouse access to the bucket in which you are creating the lake.

Prerequisites​

  • Ensure that you have granted permission to the bucket in the Terraform or CloudFormation configurations (by adding it to s3BucketArns for AWS or gcsBucketNames for GCP) when you connected your cloud account. For a new bucket, add it to that list and re-apply; otherwise lake creation will fail.

Lakes may be used for the following purposes:

  • User Permissions: Permissions for Onehouse resources (eg. Flows and Tables) can be managed at lake-level.
  • Storage Isolation: Lakes can be created in separate buckets, enabling you to write tables to different buckets. This can be useful for access control on the read side when paired with a governance solution such as AWS Lake Formation.

When creating a new Data Lake, by default the type will be created as a "Managed Lake". This means it is a lake that you intend to write new tables to with Onehouse. If you want to simply add other existing tables that are managed external to Onehouse, you can choose the type of "Observed Lake".

Working with Catalogs

Catalogs such as Hive Metastore or Glue Data Catalog do not have the concept of lakes. When Onehouse syncs tables to an external catalog, only the database and table will be synced.

Data Privacy

Each Project creates a bucket named onehouse-customer-bucket-<link-id>. Onehouse support can access this bucket for debugging. Do not store sensitive data here.

Lakes in your own buckets require your explicit approval before Onehouse support can access them (break-glass policy).

Databases​

Databases are a physical entity for organizing your Onehouse tables. When you create a new database, Onehouse creates a sub-directory for the database under the parent lake's directory.

Database names must be globally unique across all lakes (ie. you cannot have lake1 > databaseA and lake2 > databaseA). This is important for enforcing uniqueness in cloud storage and catalogs.

Tables​

Tables are created as a new directory in cloud storage under the parent database's directory. You can create tables with Flows.

Deleting tables vs deleting Flows

Deleting a Flow (Flow → Delete) marks the destination table as Stopped in the Onehouse console but leaves the table itself in place. The table continues to occupy its BASE_PATH, so trying to create a new Flow on the same path will be rejected with A table with base path '...' already exists. To free the path for reuse, delete the table from the Data tab as well.

Deleting a table in the Onehouse console will remove it from the console. The underlying data will be deleted from object storage only if the table was created by a Flow.

Other tables (eg. those created with SQL or Jobs) will retain the underlying data in object storage. You can can delete those manually by using a command like DROP TABLE ... PURGE with a SQL Cluster.

Learn more about tables in Onehouse: