S3 Log Forwarding
Overview
S3 Log Forwarding archives your Apache Spark driver and executor logs to an Amazon S3 bucket in your own AWS account, giving you a durable, long-term record of every Job run.
By default, Spark logs are available in the Onehouse console for 7 days after a Job run completes (see Monitor Job Runs). S3 Log Forwarding extends this to 30 days and delivers the logs into your own bucket where you can query them directly — useful for troubleshooting incidents discovered weeks after a run. For retention beyond 30 days, see Retention and cost.
Key characteristics:
- Logs only. This integration forwards Spark logs. It does not forward metrics.
- Your account, no extra setup. Logs are written to a bucket in your own data plane using the cluster's existing IAM node role — there is no cross-account access, no API key, and no secret to manage.
- Driver and executor logs, separated in the object layout so you can query each independently.
Enable S3 Log Forwarding
S3 Log Forwarding is enabled per project by the Onehouse team. Contact Onehouse (or your account team) to turn it on for your project. Once enabled, you can configure it in the console using the steps below.
- In the Onehouse console, go to Settings ▸ Integrations ▸ Observability.
- Add or configure an exporter and select AWS S3 as the vendor.
- Set the configuration options described below.
- Turn on the Enabled toggle to start forwarding.
Configuration options
| Option | Values | Default | Notes |
|---|---|---|---|
| Signal | Logs | Logs | Fixed. S3 Log Forwarding handles logs only. |
| Min log level | ERROR, WARN, INFO | WARN | Minimum level for driver logs (executor logs are always forwarded at WARN and above — see the note below). Selecting INFO greatly increases driver log volume — and therefore S3 storage cost — so the console shows a warning. |
| Log sources | Driver, Executor, All | All | Which Spark sources to forward. Driver and executor logs are always stored under separate prefixes regardless of this selection. |
| Bucket | (read-only) | Your Onehouse customer bucket | Logs are written to your project's onehouse-customer-bucket-<id>. This is fixed in the current version and cannot be changed in the console. |
| Compression | gzip, zstd, none | gzip | Compression applied to log objects. |
The Min log level setting applies to driver logs. Executor logs are always forwarded at WARN and above, regardless of this setting. So selecting INFO adds INFO-level driver logs (and increases driver log volume and S3 cost), but executor log volume is unchanged.
Custom region and key prefix are not configurable in the console in this version; Onehouse applies sensible defaults. If you need a custom destination, contact Onehouse.
Where your logs land
Logs are written to your customer bucket under a partitioned prefix:
s3://onehouse-customer-bucket-<id>/spark-logs/job=<job>/run=<run>/source=<driver|executor>/date=<YYYY-MM-DD>/
Each part of the path identifies the logs it contains:
| Partition | What it is | Example |
|---|---|---|
job | The name of the Spark job (its scheduled-application name) — human-readable, not a UUID. It stays the same across every run of that job. | daily-ingest |
run | The unique identifier of a single run (the submission ID, a UUID). A new value is generated for each run. | 0f25bc4f-8a1c-4e2b-9f3d-1a2b3c4d5e6f |
source | Which Spark process produced the logs: driver, executor, or unknown. | driver |
date | The UTC date the log object was written (YYYY-MM-DD). | 2026-07-20 |
Example
Driver logs from a run of the job daily-ingest on 20 Jul 2026 land under:
s3://onehouse-customer-bucket-ab12cd34/spark-logs/job=daily-ingest/run=0f25bc4f-8a1c-4e2b-9f3d-1a2b3c4d5e6f/source=driver/date=2026-07-20/
- Objects within each prefix are gzip-compressed, newline-delimited JSON (ndjson) — a new object per batch.
- The
job,run, andsourcepartitions keep each job's runs — and each run's driver vs. executor logs — isolated. - This layout is directly queryable with engines such as Amazon Athena, AWS Glue, or Apache Spark.
Delivery timing
Logs reach S3 in near-real-time — expect a short lag, not a live stream. Delivery is batch-based: an object is written to S3 when a batch reaches a size threshold or a set time window elapses, whichever comes first — that batching is what introduces the lag. The window is at most a few minutes, but in practice logs usually land sooner than that, and the final logs of a run appear shortly after it finishes.
Retention and cost
Onehouse automatically deletes forwarded logs after 30 days — objects under spark-logs/ are expired 30 days after they are written — so storage cost stays bounded with no action on your part.
The lifecycle policy on your Onehouse customer bucket is managed by Onehouse, so editing the bucket's lifecycle rules directly is not recommended: Onehouse's configuration takes precedence and overwrites manual changes on the next update. If you need to keep Spark logs longer than 30 days — for example, for audit or compliance — copy them to a bucket you manage, or contact Onehouse to discuss a longer retention window.
Limitations
- Logs only — metrics are not forwarded through this integration.
- Single destination — logs are written to your Onehouse customer bucket; a custom bucket, region, and key prefix are not configurable in the console in this version.
- Fixed 30-day retention — Onehouse manages the bucket lifecycle and deletes
spark-logs/objects after 30 days; retention is not configurable in the console (contact Onehouse for a longer window). - Log lines that cannot be attributed to a driver or executor are stored under a
source=unknownprefix.