Skip to main content

Amazon S3

Ingest data from any Amazon S3 bucket into Onehouse tables.

Click Sources > Add New Source > S3. Then, follow the setup guide within the Onehouse console to configure your source.

Cloud Provider Support​

  • AWS: ✅ Supported
  • GCP: Not supported

Prerequisites​

  1. Ensure that you have granted permission to the bucket in the Terraform or CloudFormation configurations (by adding its ARN to s3DataLoad.s3DataSourceBucketArns) when you connected your cloud account. For a new bucket, add its ARN to that list and re-apply; otherwise source creation will fail.
  2. Your S3 bucket should not have any notification rule configured to it

Schema​

By default, Onehouse will infer the source schema by reading a sample of the files to be ingested. Alternatively, you can provide a schema for the incoming records with a schema registry.

Supported File Formats​

Onehouse supports the following file formats for ingestion from S3:

  • Parquet
  • Avro
  • JSON
    • Single-object JSON files
    • JSON file with a key and list as a value
    • NDJSON files
    • JSON files with lists of objects (use the Explode Array transformation in your Flow)
  • JSONL
  • CSV
  • ORC
  • XML

Usage Notes​

  • Each file is processed exactly once.
  • If the content of an object key is modified by overwriting it, Onehouse may or may not process the updated content, depending on when the object is consumed.
    • To ensure data correctness and completeness, it is recommended to create a new file instead of modifying the content of an existing object.
  • Empty parquet files (zero records) are skipped by design and will not produce a commit. If you expect data but see no rows, verify the source files actually contain records.

Field name requirements​

Onehouse converts inferred schemas to Avro internally, and Avro forbids spaces, colons, hyphens, and leading digits in field names. If schema inference returns Unable to find S3 schema with no other error, the most common causes are:

  • Parquet columns with names like Agent Name (contains a space).
  • XML elements with namespace prefixes like ns2:Foo (contains a colon).
  • Field names starting with a digit.

Rename the offending columns at the source, or strip namespace prefixes during XML extraction, before re-running schema inference.

Bootstrapping large datasets​

When a Flow is first created against an S3 source with billions of existing rows, the default configuration is intentionally conservative — the early commits may be metadata-only (no data rows) for several hours while Onehouse plans the bootstrap. Empty initial commits are normal and not a sign of failure.

Recommendations for large S3 sources:

  • For tables with more than ~1 billion rows, request custom bootstrap configurations from Onehouse support before creating the Flow.
  • Verify the file_path_pattern regex (if set) matches actual files in the bucket. A pattern that matches zero files will silently prevent any data from being ingested, even though the Flow appears healthy.

S3 event notification latency​

When using S3 event notifications (SNS/SQS) to drive incremental ingestion, expect end-to-end latency of 30+ minutes from S3 PUT to the data appearing in the Onehouse table during heavy load. This is a property of S3 event delivery, not Onehouse processing.

If ingestion appears stalled, check the S3EventsHoodieIncrSource metadata table to see numInstantsLeftToIngest — this indicates whether events are still pending or whether the source has actually caught up.