Implementation Guide

Airbyte Self-Managed is in an early access stage for select priority users. Once you are qualified for an Airbyte Self Managed license key, you can deploy Airbyte with the following instructions.

Airbyte Self Managed must be deployed using Kubernetes. This is to enable Airbyte's best performance and scale. The core components (api server, scheduler, etc) run as deployments while the scheduler launches connector-related pods on different nodes.

Prerequisites

There are three prerequisites to deploying Self-Managed: installing helm, a Kubernetes cluster, and having configured kubectl to connect to the cluster.

For production, we recommend deploying to EKS, GKE or AKS. If you are doing some local testing, follow the cluster setup instructions outlined here.

To install kubectl, please follow these instructions. To configure kubectl to connect to your cluster by using kubectl use-context my-cluster-name, see the following:

Configure kubectl to connect to your cluster

GKE

1. Configure gcloud with gcloud auth login.
2. On the Google Cloud Console, the cluster page will have a "Connect" button, with a command to run locally: gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE_NAME --project $PROJECT_NAME
3. Use kubectl config get-contexts to show the contexts available.
4. Run kubectl config use-context $GKE_CONTEXT to access the cluster from kubectl.

EKS

1. Configure your AWS CLI to connect to your project.
2. Install eksctl.
3. Run eksctl utils write-kubeconfig --cluster=$CLUSTER_NAME to make the context available to kubectl.
4. Use kubectl config get-contexts to show the contexts available.
5. Run kubectl config use-context $EKS_CONTEXT to access the cluster with kubectl.

Deploy Airbyte Self-Managed

Add Airbyte Helm Repository

Follow these instructions to add the Airbyte helm repository:

1. Run helm repo add airbyte https://airbytehq.github.io/helm-charts, where airbyte is the name of the repository that will be indexed locally.
2. Perform the repo indexing process, and ensure your helm repository is up-to-date by running helm repo update.
3. You can then browse all charts uploaded to your repository by running helm search repo airbyte.

Clone & Configure Airbyte

1. git clone the latest revision of the airbyte-platform repository

2. Create a new airbyte.yml file in the configs directory of the airbyte-platform folder. You may also copy airbyte.sample.yml to use as a template:

cp configs/airbyte.sample.yml configs/airbyte.yml

3. Add your Airbyte Enterprise license key to your airbyte.yml.

4. Add your auth details to your airbyte.yml. Auth configurations aren't easy to modify after Airbyte is installed, so please double check them to make sure they're accurate before proceeding.

Configuring auth in your airbyte.yml file

To configure SSO with Okta, add the following at the end of your airbyte.yml file:

auth:   
    identity-providers:
        -   type: okta
            domain: $OKTA_DOMAIN
            app-name: $OKTA_APP_INTEGRATION_NAME
            client-id: $OKTA_CLIENT_ID
            client-secret: $OKTA_CLIENT_SECRET

To configure basic auth (deploy without SSO), remove the entire auth: section from your airbyte.yml config file. You will authenticate with the instance admin user and password included in the your airbyte.yml.

Install Airbyte Self Managed

Install Airbyte Enterprise on helm using the following command:

./tools/bin/install_airbyte_pro_on_helm.sh

The default release name is airbyte-pro. You can change this via the RELEASE_NAME environment variable.

Customizing your Airbyte Self Managed Deployment

In order to customize your deployment, you need to create values.yaml file in a local folder and populate it with default configuration override values. A values.yaml example can be located in charts/airbyte folder of the Airbyte repository.

After specifying your own configuration, run the following command:

./tools/bin/install_airbyte_pro_on_helm.sh --values path/to/values.yaml $RELEASE_NAME airbyte/airbyte