Skip to main content

Credential Management

Onehouse Credential Management enables users to securely store, update and delete credentials required to access for the Flow operations. Onehouse supports the following credential management options:

  • Onehouse Managed Secrets (OMS)
  • Bring Your Own Secrets (BYOS)

Onehouse Managed Secrets (OMS)​

Credentials are securely stored by Onehouse. Users input relevant credential information like username/password, API keys etc. in the Onehouse UI for adding a source or a catalog.

Onehouse project uses Onehouse Managed Secrets by default, this is determined by credentialsManagementType=OMS config in the Terraform and CloudFormation scripts.

Refer to the account linking docs (AWS, GCP) for details on Terraform and CloudFormation configurations.

Credential lifecycle management​

Pause Flows

Currently, users are expected to pause the Flows before making changes to the secret used in the Source configured for the Flow. The Flows should be resumed by the user after the change is done.

To update the credentials like username/password or key rotation use the "Edit" button.

Credential Type and relevant fields are enabled for editing on clicking the Edit button.

Deleting a Source also deletes all credentials used in that Source.

Usage​

Use the "Credentials" option under the "Credential Type" in the "Add" or "Edit" Source.

Bring Your Own Secrets (BYOS)​

Onehouse users can also store credentials as Secrets in the Secret Manager of the linked cloud account, i.e., AWS Secret Manager or Google Cloud Secret Manager. Use AWS Secret Manager Console or Google Cloud Secret Manager console to create a new secret. Then add the Secret Resource Name in the Onehouse UI when creating a Onehouse Source.

The secrets are expected to have tag accessibleTo set to onehouse and secrets for different resources are expected to be stored in a pre-defined JSON format as described here.

To use BYOS, users can configure credentialsManagementType=BYOS config in the Terraform or CloudFormation scripts.

Onehouse gke_node_role (GCP) and eks_node_role (AWS) will be additionally granted read permissions to secrets stored in the Secret Manager of the linked cloud account during the onboarding process via Terraform or CloudFormation scripts.

Credential lifecycle management (BYOS)​

Modifying and rotating secrets

Currently, users are expected to pause the Flows before making changes to the Secret used in the Source configured for the Flow. The Flows should be resumed by the user after the change is done. However, in case of Secret rotation, Onehouse will automatically pick up the new Secret in the connector, no pause/resume is required.

Editing the secret (BYOS)​

You can modify or rotate the BYOS secrets by following the steps below:

  1. First pause the Flows that will be affected by the Secret change.
  2. Navigate to the Onehouse Source where the Secret was configured, click "Edit" and update the "Secret Resource Name".
  3. Click "Save" to update the Source with the new Secret.
  4. Resume any Flows that were previously paused.

Secret Deletion​

You can delete the BYOS Secrets by following these steps:

  1. Before deleting a Secret, please ensure that Sources using the Secret are updated (with a new Secret)/deleted otherwise corresponding Flows will be impacted.
  2. To remove the Secret, delete the Secret using the Secret Manager console of the linked cloud account.

Usage​

Use "Secret Manager" option under "Credential Type" in the "Add" or "Edit" Source.

Additional Details​

  • By default, Onehouse Managed Secrets (OMS) is enabled, and users will not be able to use BYOS.
  • If Bring Your Own Secret (BYOS) is configured for your Onehouse project via Terraform or CloudFormation scripts, users will be able to use OMS or BYOS.
  • After Source addition, credentials such as username/password, API keys etc. are shown as masked values in Source details and when editing the Source.

Limitation​

  • Datahub Metastore Auth token cannot be updated from Onehouse UI. Please contact Onehouse support if auth token update is needed.

Secrets JSON format for Bring Your Own Secret (BYOS)​

Apache Kafka with SASL protocol​

Depending on the distribution of Kafka, use api_key/api_secret or username/password as username and password as shown below in your secret. Below is an example of the Secret JSON format for Apache Kafka with SASL. Here username and password are the username and password respectively.

{
"username":"<value>",
"password":"<value>"
}

Apache Kafka with TLS protocol​

Below is an example of the Secret JSON format for Kafka with TLS. Here keystore_password and key_password are the TLS key store password and TLS key password respectively.

{
"keystore_password":"<value>",
"key_password":"<value>"
}

Confluent Kafka or MSK Kafka with SASL protocol​

Below is an example of the Secret JSON format for Confluent Kafka or Amazon Managed Streaming for Apache Kafka (MSK) with SASL. Here username and password are the API key and API secret respectively.

{
"username":"<value>",
"password":"<value>"
}

Confluent Kafka or MSK Kafka with TLS protocol​

Below is an example of the Secret JSON format for Confluent Kafka or Amazon Managed Streaming for Apache Kafka (MSK) with TLS. Here keystore_password and key_password are the TLS key store password and TLS key password respectively.

{
"keystore_password":"<value>",
"key_password":"<value>"
}

Postgres or MySQL Database Credentials​

Below is an example of the Secret JSON format for Postgres or MySQL Database Credentials. Here username and password are the username and password respectively.

{
"username":"<value>",
"password":"<value>"
}

Confluent Cloud CDC Source​

Below is an example of the Secret JSON format for Confluent Cloud CDC source. Here key and secret are the API key and API secret respectively.

{
"key":"<value>",
"secret":"<value>"
}

Confluent Schema Registry for Confluent Kafka or MSK Kafka or Confluent Cloud CDC Source​

Below is an example of the Secret JSON format for Confluent Schema Registry for Confluent Kafka or MSK Kafka or Confluent Cloud CDC Source. Here api_key and api_secret are the API key and API secret respectively.

{
"api_key":"<value>",
"api_secret":"<value>"
}

Datahub Metastore​

Below is an example of the Secret JSON format for Datahub Metastore. Here auth_token is the auth token for the Datahub Metastore.

{
"auth_token":"<value>"
}