Skip to main content

Accessing AWS Secrets Manager

Overview​

This guide demonstrates how to securely access credentials stored in AWS Secrets Manager from your Python Spark jobs running on Onehouse. This pattern is useful when your jobs need to authenticate with external APIs or services.

BYOS Only

Jobs currently only support Bring Your Own Secrets (BYOS). Onehouse Managed Secrets (OMS) are not yet supported for jobs.

For general information about secrets management in Onehouse, see the Secrets Management documentation.

Use Case​

We want to:

  • Store sensitive credentials (API keys, passwords) securely in AWS Secrets Manager
  • Access those credentials from Python code running in Onehouse Jobs
  • Support local development with environment variable fallbacks

Pre-requisites​

  • An AWS Secrets Manager secret with the accessibleTo tag set to onehouse. IAM permissions for your Onehouse job role are automatically configured by Terraform during account setup. See the Secrets Management documentation for details.
  • A requirements.txt file with the following dependencies: 
    boto3>=1.26.0
  • Create venv.tar.gz with the required dependencies and upload it to S3. Follow the steps to create the file

Step-by-Step Instructions​

Step 1: Create a Secret in AWS Secrets Manager​

Using AWS CLI​

aws secretsmanager create-secret \
  --name "your-project/credentials" \
  --region us-west-1 \
  --secret-string '{"API_KEY": "your-api-key", "API_SECRET": "your-api-secret"}' \
  --tags Key=accessibleTo,Value=onehouse

Using AWS Console​

  1. Go to Secrets Manager > Store a new secret
  2. Choose Other type of secret
  3. Add key/value pairs
  4. Name it (e.g., your-project/credentials)
  5. Important: Add a tag with key accessibleTo and value onehouse
Required Tag

Secrets must have the tag accessibleTo set to onehouse for Onehouse to access them. Secrets without this tag will not be accessible.

Step 2: Create a Python file to access secrets​

Use the boto3 library to retrieve secrets from AWS Secrets Manager. The following helper function fetches a secret by name and returns the parsed JSON contents as a dictionary.

import json
import boto3


def get_secret(secret_name, region_name="us-west-1"):
    """
    Retrieve secret from AWS Secrets Manager.

    Args:
        secret_name: Name of the secret (e.g., "your-project/credentials")
        region_name: AWS region where secret is stored

    Returns:
        dict: Secret key-value pairs
    """
    client = boto3.client("secretsmanager", region_name=region_name)
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response["SecretString"])


# Usage in your Spark job
secrets = get_secret("your-project/credentials")
api_key = secrets["API_KEY"]
api_secret = secrets["API_SECRET"]

Step 3: (Optional) Add Environment Variable Fallback for Local Development​

When developing locally, you may not have access to AWS Secrets Manager or may want to avoid making AWS API calls. This pattern checks for environment variables first, allowing you to set credentials locally without modifying your code.

import json
import os
import boto3


def get_credentials(secret_name="your-project/credentials", region_name="us-west-1"):
    """
    Get credentials from environment variables (local) or Secrets Manager (production).
    """
    # Try environment variables first (local development)
    api_key = os.getenv("API_KEY")
    if api_key:
        return api_key, os.getenv("API_SECRET")

    # Fall back to Secrets Manager (production on Onehouse)
    client = boto3.client("secretsmanager", region_name=region_name)
    response = client.get_secret_value(SecretId=secret_name)
    secrets = json.loads(response["SecretString"])

    return secrets["API_KEY"], secrets["API_SECRET"]