Skip to main content

oh validation

Manage data quality validations applied to records during ingestion. Records that fail a validation are sent to a quarantine table.

Validation types: SCHEMA TIMESTAMP

oh validation show​

Lists all validations.

oh validation show

oh validation describe NAME​

Describes a validation.

oh validation describe schema-check

oh validation create NAME​

Creates a validation.

Flags​

FlagRequiredDescription
--typeYesSCHEMA or TIMESTAMP
--options KEY=VALUENoWITH clause parameters; repeatable (see Special parameters)

Special parameters​

All WITH clause parameters from the CREATE VALIDATION API are passed via --options KEY=VALUE.

Examples​

Schema validation​

Validates records against a schema from your schema registry.

oh validation create schema-check \
  --type SCHEMA \
  --options schema.target_schema_name=my-kafka-schema

Timestamp validation — no range constraint​

Validate that a timestamp field is parseable in the expected format, with no range restriction.

oh validation create ts-format-check \
  --type TIMESTAMP \
  --options timestamp.field=event_time \
  --options timestamp.format=yyyy-MM-dd HH:mm:ss \
  --options timestamp.expected_range.type=no_range

Timestamp validation — not in the future​

Reject records where the timestamp is in the future.

oh validation create ts-not-future \
  --type TIMESTAMP \
  --options timestamp.field=event_time \
  --options timestamp.format=epoch_millis \
  --options timestamp.expected_range.type=relative \
  --options timestamp.expected_range.relative.filter=is_not_future

Timestamp validation — within a rolling window​

Reject records older than a rolling window (e.g., last 7 days).

oh validation create ts-recent \
  --type TIMESTAMP \
  --options timestamp.field=event_time \
  --options timestamp.format=yyyy-MM-dd HH:mm:ss \
  --options timestamp.expected_range.type=relative \
  --options timestamp.expected_range.relative.filter=is_in_last \
  --options timestamp.expected_range.relative.is_in_last.quantity=7 \
  --options timestamp.expected_range.relative.is_in_last.unit=days

Timestamp validation — after a specific date​

Reject records with a timestamp before a given date. Valid operators: LT, LEQ, GT, GEQ.

oh validation create ts-after-launch \
  --type TIMESTAMP \
  --options timestamp.field=event_time \
  --options timestamp.format=yyyy-MM-dd \
  --options timestamp.expected_range.type=specific \
  --options timestamp.expected_range.specific.specific.filter=GEQ \
  --options timestamp.expected_range.specific.specific.date=2024-01-01

oh validation delete NAME​

Deletes a validation.

oh validation delete schema-check