Self-Managed Kubernetes

A Self-Managed Kubernetes project runs the Onehouse data plane on a Kubernetes cluster you already operate (on-prem, bare metal, colocation, or a provider Onehouse has no native integration with) and stores data in an S3-compatible object store.

Unlike AWS, Google Cloud and Azure, there is no Terraform customer stack and no cloud identity. Onehouse is given exactly two things:

An admin credential to your cluster, created when you install the Onehouse connect Helm chart.
An S3 endpoint with a static access key, which you enter in the onboarding wizard.

You own the cluster, its nodes and its storage. Onehouse never creates, scales, labels or tags a node, and installs no storage provisioner. Prepare everything on this page before you start the wizard.

Kubernetes Cluster

Requirement	Detail
Access	A kubeconfig with `cluster-admin` rights for the person running `helm install`. `kubectl auth can-i '' '' -A` must return `yes`.
Version	A Kubernetes version your distribution still supports.
Default StorageClass	A StorageClass with a working CSI provisioner that supports volume expansion. You enter its name in the wizard. See Persistent Storage.
`/mnt` on every node	The directory `/mnt` must exist on each node's OS. An Onehouse maintenance pod mounts it as a `hostPath` of type `Directory`.
Autoscaling	Fixed node counts are recommended. Onehouse schedules Spark pods through YuniKorn, and these pods do not reliably trigger a cluster autoscaler. New nodes you add appear in Onehouse within 5 minutes.
Node auto-upgrade	If your platform replaces nodes during upgrades, apply labels at the node-pool level (below) so replacement nodes keep them.

You do not need a LoadBalancer, a public API server endpoint, cloud IAM or VPC peering. The control plane reaches your API server through an outbound tunnel that the data plane opens.

Node Labels

Onehouse schedules its pods onto the nodes you label. Nodes without Onehouse labels are left alone, so the cluster can be shared with your own workloads.

Label node pools, not nodes

Where your platform supports it, set the labels on the node pool (or the node template, kubelet --node-labels, or your provisioning tool) rather than with kubectl label. Labels applied with kubectl label are lost when a node is replaced.

Node type label: `oh-instance-type`

Every node that carries any Onehouse role label below must also carry oh-instance-type=<node type name>.

Use one value per hardware size. Nodes that share a value must have the same vCPU count and memory within 2%.
The value must match ^[a-z0-9]([a-z0-9.-]{0,61}[a-z0-9])?$, for example cpu-16c-64g or r6.4xlarge.
The value becomes the node type name you pick when creating a compute cluster. Onehouse reads each type's vCPU and memory from the nodes carrying it, and Spark pods select oh-instance-type=<type>, so a pod sized for a type only lands on a node of that type.

What happens when the rule is broken:

Situation	Result
A node carries a role label but no `oh-instance-type`	Onboarding fails and lists the nodes with the `kubectl label` command to fix them.
A node has no Onehouse labels at all	Ignored. It is not counted anywhere.
Nodes sharing one `oh-instance-type` value differ in size	The node type is frozen at its last consistent size and flagged in Project Settings → Node Types until you relabel the nodes.

Role labels

Label	Pool	Used for	Checked
`oh-essential-pods=true`	Essential	Onehouse agent, tunnel, ingress and other platform pods	At onboarding
`oh-essential-pods-vpc-cni=true`	Essential	Platform pods that hard-select this label	Required for platform pods to schedule
`monitoring-pods=true`	Essential	Prometheus, Alertmanager and Grafana	At onboarding
`pool=job-workers-repair`	Repair	Table service monitor and Hudi timeline inspection jobs	At onboarding
`driver-pods=true`	Workers	Spark driver pods for Onehouse-managed jobs	When you create a compute cluster
`job-workers=true`	Workers	Spark executors for ingestion and table services	When you create a compute cluster
`sql-workers=true`	Workers	SQL, Spark, PySpark and Lakecache cluster pods	When you create a compute cluster
`openengine-trino=true`, `openengine-flink=true`, `openengine-ray-cpu=true`	Open engines	Trino, Flink and Ray	When you create an open engine cluster

Example, for nodes not managed by a node-pool API:

kubectl label nodes <system-node> \
  oh-essential-pods=true oh-essential-pods-vpc-cni=true monitoring-pods=true \
  oh-instance-type=<node type name>

kubectl label nodes <repair-node> \
  pool=job-workers-repair oh-instance-type=<node type name>

kubectl label nodes <worker-node> \
  driver-pods=true job-workers=true sql-workers=true oh-instance-type=<node type name>

Pool sizing

Pool	Minimum	Notes
Essential	2 nodes, at least 4 vCPU each, at least 20 GiB allocatable memory in total	The platform and monitoring stack request about 11.5 GiB, plus your own DaemonSets and the system metrics job. Undersized essential nodes leave platform pods `Pending`.
Repair	1 node with 4 vCPU and at least 12 GiB allocatable (16 GiB recommended)	Must carry `pool=job-workers-repair` and no other role label. Use 2 nodes if your Onehouse team enables metadata-table validation jobs.
Workers	1 node with at least 4 vCPU and 10 GiB allocatable	Size for your workload. A compute cluster's capacity is bounded by the Ready nodes of its node type carrying its role label.

Onehouse never caps a compute cluster's Max OCU by your node capacity. If you set Max OCU above what your labelled nodes can hold, the UI warns you and jobs wait for capacity.

Optional: local scratch disk

If worker nodes have NVMe or another fast local disk, Spark shuffle and the Lakecache cache can use it instead of the root disk. Format and mount the disk yourself (for example at /mnt/instance-store, mode 1777), then annotate each node:

kubectl annotate node <node> \
  onehouse.ai/local-scratch-path=/mnt/instance-store \
  onehouse.ai/local-scratch-capacity=<capacity in bytes> --overwrite

Prepare all nodes of one node type the same way. Onehouse prefills the node type's scratch settings only when every node of that type agrees. Onehouse never runs scripts on your nodes.

Reference script: prepare-local-scratch.sh

Run as root on each node, from your own provisioning (cloud-init, Ansible, a golden image or by hand). It refuses devices that are mounted or already carry a filesystem unless FORCE=1, does nothing if the mount point is already mounted, and mounts by filesystem label so the fstab entry survives device renumbering. On EC2-style instance stores use DEVICE_GLOB='/dev/disk/by-id/nvme-*Instance_Storage*'; on bare metal, name the devices. If hostname -s differs from the Kubernetes node name, use the name from kubectl get nodes in the printed command.

#!/usr/bin/env bash
# prepare-local-scratch.sh — format and mount local disk(s) as Onehouse scratch space.
# Run as root on the node. Usage:
#   prepare-local-scratch.sh /dev/nvme1n1 [/dev/nvme2n1 ...]
#   DEVICE_GLOB='/dev/disk/by-id/nvme-*Instance_Storage*' prepare-local-scratch.sh
# Env: MOUNT_POINT (default /mnt/instance-store), RAID_NAME (default onehouse-scratch), FORCE=1 to wipe signatures.
set -euo pipefail

MOUNT_POINT="${MOUNT_POINT:-/mnt/instance-store}"
RAID_NAME="${RAID_NAME:-onehouse-scratch}"
FS_LABEL="oh-scratch"

if [[ $# -gt 0 ]]; then
  devices=("$@")
elif [[ -n "${DEVICE_GLOB:-}" ]]; then
  mapfile -t devices < <(for p in ${DEVICE_GLOB}; do [[ -e "$p" ]] && readlink -f "$p"; done | sort -u)
else
  echo "usage: $0 <block-device>... | DEVICE_GLOB=<glob> $0" >&2
  exit 2
fi

[[ ${EUID} -eq 0 ]] || { echo "must run as root" >&2; exit 1; }
[[ ${#devices[@]} -gt 0 ]] || { echo "no devices matched" >&2; exit 1; }

if findmnt -rn "${MOUNT_POINT}" >/dev/null; then
  echo "${MOUNT_POINT} is already mounted; nothing to do"
  exit 0
fi

for d in "${devices[@]}"; do
  [[ -b "$d" ]] || { echo "not a block device: $d" >&2; exit 1; }
  if findmnt -rn -S "$d" >/dev/null; then
    echo "$d is mounted; refusing to touch it" >&2
    exit 1
  fi
  if blkid -p "$d" >/dev/null 2>&1 && [[ "${FORCE:-0}" != "1" ]]; then
    echo "$d already carries a filesystem or RAID signature; re-run with FORCE=1 to wipe it" >&2
    exit 1
  fi
done

mkdir -p "${MOUNT_POINT}"

if [[ ${#devices[@]} -eq 1 ]]; then
  target="${devices[0]}"
else
  command -v mdadm >/dev/null || { echo "mdadm is required to stripe ${#devices[@]} devices" >&2; exit 1; }
  target="/dev/md/${RAID_NAME}"
  mdadm --create "${target}" --run --level=0 --name="${RAID_NAME}" \
    --raid-devices="${#devices[@]}" "${devices[@]}"
  conf=/etc/mdadm.conf; [[ -d /etc/mdadm ]] && conf=/etc/mdadm/mdadm.conf
  mdadm --detail --scan | grep -F "name=${RAID_NAME}" >> "${conf}" || true
fi

mkfs.ext4 -F -L "${FS_LABEL}" "${target}"
mount "${target}" "${MOUNT_POINT}"
grep -qF "LABEL=${FS_LABEL} " /etc/fstab || \
  echo "LABEL=${FS_LABEL} ${MOUNT_POINT} ext4 defaults,nofail,noatime 0 2" >> /etc/fstab
chmod 1777 "${MOUNT_POINT}"

capacity_bytes=$(df -B1 --output=size "${MOUNT_POINT}" | tail -1 | tr -d ' ')
node_name=$(hostname -s)

echo "mounted ${target} at ${MOUNT_POINT} (${capacity_bytes} bytes)"
echo
echo "Annotate the node so Onehouse discovers the scratch disk (run from a machine with kubectl):"
echo "  kubectl annotate node ${node_name} \\"
echo "    onehouse.ai/local-scratch-path=${MOUNT_POINT} \\"
echo "    onehouse.ai/local-scratch-capacity=${capacity_bytes} --overwrite"

Persistent Storage

Prometheus (50 GiB) and Alertmanager (4 GiB) need persistent volumes. You enter a StorageClass name in the wizard. It must:

Use a working CSI provisioner with dynamic provisioning.
Support volume expansion (allowVolumeExpansion: true).
Provide ReadWriteOnce volumes. Every Onehouse volume is RWO.

Provisioner	Status
Longhorn	Validated
TopoLVM, OpenEBS LVM	Compatible
Your cloud provider's block-storage CSI driver	Compatible
`local-path`	Not supported. It is not CSI and cannot expand volumes.

Onboarding checks that the StorageClass exists. It does not create a test volume, so a provisioner that cannot bind volumes shows up as the monitoring stack's volumes staying Pending.

Object Storage

The object store must speak the S3 API. NetApp StorageGRID is the validated store. Other S3-compatible stores that meet the requirements below are supported on a best-effort basis.

Requirement	Detail
API	S3 with Signature V4 and multipart upload. Conditional writes are not required.
Endpoint	`https://<dns-name>[:port]`. HTTP, IP addresses and `localhost` are rejected.
TLS	A publicly trusted certificate for the endpoint's DNS name. Private or custom certificate authorities are not supported.
Addressing	Path-style (`https://<host>/<bucket>/<key>`). Onehouse forces path-style everywhere, so no wildcard DNS is needed.
Signing region	A SigV4 region your store accepts. Defaults to `us-east-1`. Most S3-compatible stores accept any value; change it only if your store rejects requests with `AuthorizationHeaderMalformed`.
Bucket	Named exactly as the wizard shows it: `onehouse-customer-bucket-<prefix>`. Create it before onboarding. Keep it private.
Access key	A static access key and secret with read, write, list and delete permission on that bucket. Scope it to the bucket if your store supports it.
Reachability	Reachable from the cluster's nodes. Onehouse reaches it through the data plane's tunnel, so it does not need to be on the public internet.

Enable bucket versioning

Onehouse keeps its Terraform state for your data plane in this bucket. Versioning is not required, but enabling it lets you recover from an accidental overwrite or deletion.

The access key is stored in the Onehouse control-plane secret manager and delivered to the data plane at runtime. It is never written into the Helm values file. To rotate it, resubmit the storage step of the wizard with the new key. Running jobs pick up the new key when they restart.

ZooKeeper Lock Provider

Onehouse coordinates concurrent writers (multi-writer Hudi tables, SQL, Spark and Lakecache clusters) through a ZooKeeper ensemble that you run. Self-managed object stores generally cannot provide the conditional-write locking Onehouse uses on AWS, Google Cloud and Azure, and there is no DynamoDB.

Requirement	Detail
Version	ZooKeeper 3.5 or newer
Topology	Three nodes recommended
Authentication	None. SASL and TLS client authentication are not supported.
Reachability	Every server reachable on its client port from the Onehouse namespaces in the cluster. If you use `NetworkPolicy`, allow it.

You enter the connection string (host:port,host:port,host:port) in the wizard. The connect job checks every server before onboarding continues, and Onehouse registers the ensemble as this data plane's lock provider automatically. Sizing, high availability, upgrades and backups of the ensemble are yours. Onehouse alerts when a server becomes unreachable.

Networking

The data plane dials out to Onehouse. No inbound connection to your network is required, you do not need to expose the Kubernetes API server, and there are no control plane NAT IPs to allow.

From the cluster's nodes, allow outbound access to:

The Onehouse control plane on TCP 443, plus TCP 7202 to default-frp-connect.onehouse.ai for the tunnel.
Your S3 endpoint and your ZooKeeper servers.
The domains below, if you run an egress firewall.

The connect job tests each required host, your S3 endpoint and your ZooKeeper servers from inside the cluster, and onboarding does not continue until they are reachable. See Verify Egress Connectivity for the port details.

Required: allowlist for every Self-Managed Kubernetes deployment (TCP 443)

.onehouse.ai (control plane APIs, gateway, telemetry, image distribution)
.docker.io (Docker Hub registry)
production.cloudflare.docker.com
production.cloudfront.docker.com
docker-images-prod.6aa30f8b08e16409b46e0173d6de2f56.r2.cloudflarestorage.com
docker-registry-production.d24a988e385e0074d717b6bdaea58f0d.r2.cloudflarestorage.com
auth.docker.io
.cloudfront.net
.gcr.io
.pkg.dev
.googleapis.com
.k8s.io (registry.k8s.io)
ghcr.io
.quay.io
registry.terraform.io
releases.hashicorp.com
get.helm.sh
github.com, .github.com, .githubusercontent.com, .github.io
.strimzi.io
.pagerduty.com (Onehouse operational alerting)

Conditional: only if the corresponding feature or integration is enabled

.confluent.cloud (Confluent sources)
.jupyter.org (notebooks)
.debezium.io
datadoghq.com
docker.elastic.co, docker-auth.elastic.co
.snowflakecomputing.com (Snowflake catalog)
.databricks.com (Unity / Databricks)
.splunkcloud.com
.slack.com (Slack notifications)
api.honeycomb.io
.openai.com (AI features)
apachesuperset.docker.scarf.sh

Kafka and other source hosts

Onehouse only accepts source and catalog endpoints whose hostnames match the project's allowlist. Before creating a Kafka source, add your broker hostnames (and schema registry, if any) under Project Settings → Host Pattern Allowlist.

Security Model

A Self-Managed Kubernetes data plane has no cloud identity to lean on.

Cluster access. The connect chart creates an onehouse-admin ServiceAccount bound to cluster-admin and hands Onehouse a bound, time-limited token. No long-lived token Secret is created. The control plane rotates the token before it expires, and the Onehouse agent in your cluster can renew it. If your API server caps token lifetime (--service-account-max-token-expiration), rotation follows the granted lifetime.
Object storage access. A static access key with the permissions you grant it. Onehouse recommends a key scoped to the Onehouse bucket.
Nothing privileged. Onehouse installs no DaemonSet that touches the host, no node-start script and no storage provisioner. The onboarding checks are read-only against the Kubernetes API.
Secrets. Source and catalog credentials are stored in the Onehouse-managed secret manager. Your own secrets manager is not supported for this project type.

Tooling

Tool	Version	Purpose
kubectl	Matching your cluster	Labelling nodes, checks
Helm	≥ 3.8 (OCI registry support)	Installing the connect chart

No Terraform is needed.

Next Step

Review the Prerequisites Checklist, then continue to Deploy: Self-Managed Kubernetes.

Kubernetes Cluster​

Node Labels​

Node type label: oh-instance-type​

Role labels​

Pool sizing​

Optional: local scratch disk​

Persistent Storage​

Object Storage​

ZooKeeper Lock Provider​

Networking​

Kafka and other source hosts​

Security Model​

Tooling​

Next Step​