Deploying to your own Kubernetes Service
Note: The commands referenced in this document will be kubernetes cli specific but the same can be used by changing the kubectl command to oc using the openshift cli
This document is split into 2 sections:
New deployment will take you through the steps to deploy K for the first time.
Upgrades will take you through the steps to update your K installation.
Before you start
You should have
Received a configuration package from KADA.
Be familiar or be knowledgable about the network and load balancer settings for exposing services on your Kubernetes instance.
Request from your network team a DNS alias and certificate for the KADA Platform.
Access to an environment a Kubernetes cluster
In your local environment
Install kubectl
Install your cloud provider cli: eg
azure cli, aws cli
For windows environments install gitbash.
1. New Deployments
A new deployment will take approximately 1 hour in duration to complete.
The commands in the steps below assume you are running in a unix bash env.
We strongly recommend using a Linux based environment over windows for compatibility of the deployment scripts. However windows use may run gitbash if you are deploying from a windows environment.
Step 1) Create a new Kubernetes cluster.
Kubernetes Service Providers supported: Amazon’s Elastic Kubernetes Service (EKS) & Microsoft Azure’s Kubernetes Service (AKS). Our customers have deployed onto Openshift Kubernetes Service. Reach out for assistance with other Kubernetes options that are not listed.
For cluster requirements see https://kadaai.atlassian.net/wiki/spaces/KSL/pages/1899692038/How+to+deploy+on+your+cloud#Minimum-infrastructure-requirements
Step 2) Label Kuberenetes cluster nodes
The nodes in the Kubernetes cluster need to be named.
KADA services use the following labels for deployment:
core → location of zk/solr/scheduler/redis
worker → location of the workers
db → location of postgres
auth → location of keycloak
gateway → location of the API and App related services
Use the following command to label the nodes in the cluster
kubectl get nodes # Return you the node ids within the cluster
kubectl label nodes <node id> <label name>=true
# Example
kubectl label nodes aks-agentpool-27844483-vmss000000 core=true
kubectl label nodes aks-agentpool-27844483-vmss000000 auth=true
kubectl label nodes aks-agentpool-27844483-vmss000001 core=true
kubectl label nodes aks-agentpool-27844483-vmss000001 gateway=true
kubectl label nodes aks-agentpool-27844483-vmss000003 db=true
kubectl label nodes aks-agentpool-27844483-vmss000002 worker=true
kubectl label nodes aks-agentpool-27844483-vmss000002 core=true
Example of a KADA deployment into a 4 Node cluster.
Node 1 | Node 2 | Node 3 | Node 4 |
---|---|---|---|
zk | zk | zk | postgres |
solr | solr | cerebrum batch worker |
|
keycloak | solr gatekeeper | cerebrum worker |
|
keycloak postgres | cerebrum api |
|
|
cerebrum scheduler | cortex |
|
|
redis |
|
|
|
Step 3 ) Setting up access to KADA Image repository
KADA will provide a KADA_CLIENT_ID
and KADA_CLIENT_SECRET
to access the KADA Image repository. The following setups your Kubernetes service to access the repository
Create a secrete
kubectl create secret docker-registry kada-image-credentials \
--docker-server=kadaexternal.azurecr.io \
--docker-username=$KADA_CLIENT_ID \
--docker-password=$KADA_CLIENT_SECRET
Patch the service account with the above secret
kubectl patch serviceaccount <the service_account or "default"> \
-p "{\"imagePullSecrets\": [{\"name\": \"kada-image-credentials\"}]}"
Step 4 ) Creating a certificate
Create a certificate and key
Raise a cerificate request for the domain hosting the K Platform.
Load the cert / key into Kubernetes
Step 5 ) Kubernetes ingress
Your organisation will most likely have a standard pattern for routing network traffic to a Kubernetes cluster via a Load Balancer / HA Proxy / Ingress routes.
Using your organisations Load Balancer and ingress service
KADA Deployment can make use of your organisations pattern with a few additional configuration steps.
Note the domain of the Load Balancer URL. We will refer to this as DOMAIN_URL from here on.
[OPENSHIFT ONLY] Openshift Load Balancer definition
Add the following mappings to the config of your organisation’s ingress
Deploy KADA’s generic Load Balancer and ingress service
If you are not using your own load balancer / ingress service you can use the one packaged with the K Platform.
Step 6 ) Deploying KADA Services into Kubernetes
Download the distribution package and unzip it
Navigate to the kube_setup directory
Populate the k8s_env.sh with the correct values according to your desired values
NOTES
HOST is in the format of the alias name or canonical host name. It must be lowercase e.g. if I intend to access K via
https://prod.kada.ai
, then the host value isprod.kada.ai
FERNET_KEYS should be generated using one of these methods
Python
Unix shell
SSL_SECRET_NAME is the Kubernetes secret name you installed the SSL Certificate as
Avoid special characters in the values above if possible. Any value that contains the following special characters need to be escaped with a
\
backslash:\
→\\\\
`
→\`
$
→\$
Save k8s_env.sh in a secure location so that it can be used when upgrading to the K Platform.
Run to create a generated-k8s-common folder containing 2 yaml files.
Make sure
kubectl
is configured and pointing to a Kubernetes cluster.Deploy the generated config
Deploy the K platform. Note the
y
arg will deploy an ingress which terminates SSL.[OPENSHIFT ONLY] Update the users that k8s runs the contains
Modify PV Policy
For these 2 claims, set to Retain
default/postgres-storage-postgres-statefulset-0
default/keycloak-postgres-storage-keycloak-postgres-statefulset-0
Then run to validate Reclaim policy has updated to Retain
Step 7) Post deployment verification
Check all Kubernetes services are running and not in error
Example of expected output
Check the status API. It should return 200 if successful
Log into Keycloak and create a new user covered in this article.
https://kadaai.atlassian.net/wiki/spaces/KSL/pages/1888813129
The Keycloak portal is accessible at the following link
Log into KADA with the created user using the following link.
Step 8) Setup Landing Storage
KADA uses object store as a landing zone for metadata and log file transfer.
We currently support AWS s3 or Azure Blob.
AWS s3 setup
Configure the following in Admin > Platform Settings > Settings
storage_type = s3
storage_root_folder = <s3 bucket name>
storage_aws_region = <Your AWS region >
storage_aws_access_key_id = <Your AWS IAM user access key>
storage_aws_secret_access_key = <Your AWS IAM user secret>
Azure Blob setup
Configure the following in Admin > Platform Settings > Settings
storage_type = azure
storage_root_folder = <Azure container name>
storage_azure_storage_account = <Your azure storage account>
storage_azure_access_key = <Your azure storage account access key>
Performing an upgrade to your K installation
Step 1 ) Pre checks
Check that no jobs are currently running. Admin > Monitor.
Then scale down the following services.
Step 2) Deploying KADA updates
Download the distribution package and unzip it
Diff
cerebrum-extract-scripts.yaml
andcerebrum-oic.yaml
against your existing version if you have made changes to the extract queries and merge any new changes.Navigate to the kube_setup directory
Populate the k8s_env.sh or overwrite with the k8s_env.sh from a prior deployment
Deploy the K platform.
Step 3) Post deployment verification
Follow the same verification as per New Deployments (Step 5 in New Deployments)