What is GitOps ?
It's a new concept introduced by Weave and seems to be the new trend for managing Kubernetes applications.
GitOps is a way to do Kubernetes cluster management and application delivery. It works by using Git as a single source of truth for declarative infrastructure and applications.
It is very similar to infrastructure as code, the main goal is to avoid developers or Ops to manage Kubernetes resources from their own kubectl. It could be resumed by Operations by Pull Request
. No new stuff for now.
Some nice features from this concept :
- Fast error recovery: Indeed, it is only a Git revert to restore your environment.
- Deploy Faster: No need to wait for an OPS to be in the mood to prod.
- Security: Developers have not access to clusters (and CI tools neither, we'll see why later).
- Documentation: Because Git is the single source of truth, no need to connect to clusters to see which resources are present.
Two ways to implement this pattern
Push based
Classic way to deploy your application. Once committed, a trigger starts a job on your CI tool.
Pull based
In this case, an agent is running on the cluster and pull the modification. It avoids giving cluster access to CI tool.
Tools
There are many tools which implement this pattern. But in fact, a simple script could do the job :
git clone https://awesome-repo
cd awesome-repo
while true {
git pull
kubectl apply -f .
sleep 30
}
Let's see some tools ready for production.
Flux CD
Flux CD seems to be the most famous product of the GitOps concept. It has been developed by Weaveworks. Probably the simplest GitOps tool to use though it only has a CLI and you need one agent per managed application.
Argo CD
Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.
Argo CD tracks all the changes made in a repository to deploy the desired application state.
Some nice features :
- Automated or manual deployments of applications.
- Could manage multi-cluster.
- Rollback to any Git commit.
- Canary deployment.
- Execute operations before or after sync.
- Health status of existing applications resources.
- Support multiple templating tools like Kustomize or Helm.
Focus on Argo CD
We will focus on Argo CD for this demo, it has more features and its Web UI is very powerful to easily see our managed applications.
Demo
Part 1 - Deployment
Requirements needed before starting :
- GCP account
- kubectl
- kustomize
- SOPS
- argocd cli (optional: it's used to update admin password)
Secret Management with SOPS and Google KMS
I have discovered SOPS some months ago and I like to use it now to manage secrets in git. We are going to use it with GCP kms. For more information about SOPS, don't hesitate to read our previous article https://dev.to/stack-labs/manage-your-secrets-in-git-with-sops-g0a
Step 1: Create a KMS Key
gcloud kms keyrings create "gke-argocd-demo" \
--location=global
gcloud kms keys create "gke" \
--location "global" \
--keyring "gke-argocd-demo" \
--purpose "encryption"
gcloud kms keys list \
--location "global" \
--keyring "gke-argocd-demo"
KMS_ID=$(gcloud kms keys list --location "global" --keyring "gke-argocd-demo" --format 'get(name)')
Step 2: Launch on GKE with workload Identity
Workload Identity is the recommended way to access Google Cloud services from within GKE due to its improved security properties and manageability. https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity?hl=en
# Create the GKE cluster with workload identity enabled
PROJECT_ID=$(gcloud config list --format 'value(core.project)')
gcloud beta container clusters create gke-argocd-demo \
--release-channel regular \
--workload-pool=$PROJECT_ID.svc.id.goog
Step 3: Prepare Workload Identity
# Configure kubectl
gcloud container clusters get-credentials gke-argocd-demo
# Create iam service account
gcloud iam service-accounts create gke-argocd-demo
# Create kubectl namespace (we will deploy argo resources inside later)
kubectl create ns argocd
kubectl create serviceaccount --namespace argocd gke-argocd-demo
# Link Kubernetes service account to GCP service account
gcloud iam service-accounts add-iam-policy-binding \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$PROJECT_ID.svc.id.goog[argocd/gke-argocd-demo]" \
gke-argocd-demo@$PROJECT_ID.iam.gserviceaccount.com
# Add permissions to interact with KMS from pods
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member serviceAccount:"gke-argocd-demo@$PROJECT_ID.iam.gserviceaccount.com" \
--role "roles/cloudkms.cryptoKeyEncrypterDecrypter"
kubectl annotate serviceaccount \
--namespace argocd gke-argocd-demo \
iam.gke.io/gcp-service-account=gke-argocd-demo@$PROJECT_ID.iam.gserviceaccount.com
Kustomize with SOPS
In this demo, argo CD uses Kustomize to deploy k8s resources. There is a SOPS plugin for Kustomize ( https://github.com/viaduct-ai/kustomize-sops ). It is not mandatory to install this plugin locally though it is recommended to check your manifests before pushing to Git and avoid potential errors during deployment.
Step 4: Deploy Argo CD
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# Mandatory with GKE
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin \
--user="$(gcloud config get-value account)"
# Optional: if you want to access to the Web UI with an external IP
kubectl patch svc argocd-server \
-n argocd \
-p '{"spec": {"type": "LoadBalancer"}}'
Step 5: Get default password
Default Argo CD password is the name of the server pod. Save it somewhere, or change it with next step.
# get current password
kubectl get pods \
-n argocd \
-l app.kubernetes.io/name=argocd-server \
-o "jsonpath={.items[0].metadata.name}"
Step 6: Change default password
# forward port is needed (Argo server service is a clusterIP) unless you have patched the service with a LoadBalancer
kubectl port-forward svc/argocd-server -n argocd 8080:443
### Optional steps if you want to change the default password
# Install Argo CD CLI https://argoproj.github.io/argo-cd/cli_installation
# login
argocd login 127.0.0.1:8080
argocd account update-password
Step 7: Patch Argo CD to use SOPS
Edit config map to enable Kustomize with alpha plugins. For this, you need to add data
part (see below).
kubectl -n argocd edit cm argocd-cm
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
labels:
app.kubernetes.io/name: argocd-cm
app.kubernetes.io/part-of: argocd
# add this :
data:
kustomize.buildOptions: "--enable_alpha_plugins"
To integrate kustomize-sops with Argo CD, we need to patch the deployment server with an init-container
and the new service account
created. You can look the patch here.
# Patch command
kubectl patch \
-n argocd deployment/argocd-repo-server \
-p "$(curl https://raw.githubusercontent.com/pabardina/gitops-argocd-sops/master/argo-cd-repo-server-ksops-patch.yaml)"
# Watch for the new pod to be ready
kubectl -n argocd get pod \
-l app.kubernetes.io/name=argocd-repo-server \
--watch=true
Step 8: Launch UI
Open http://127.0.0.1:8080, connect with admin
and password from last step and register an application.
You can fork my demo application: https://github.com/pabardina/gitops-argocd-sops.git and use your fork repository url in Argo CD.
The application deployed:
Part 2 - Demo with secrets
Step 1: Clone locally your fork repository
git clone https://github.com/<your-username>/gitops-argocd-sops.git
cd gitops-argocd-sops
Step 2: Add a secret
cat <<EOF > base/secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: super-secret
data:
pass: c3VwZXItcGFzc3cwcmQ=
EOF
Step 3: Encrypt the new secret with SOPS
# encrypt the file with sops
sops -e -i --gcp-kms $KMS_ID base/secret.yaml
# Create Kustomize generator
cat <<EOF > base/secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
# Specify a name
name: example-secret-generator
files:
- ./secret.yaml
EOF
# Update base/kustomization.yaml to add generator part
vi base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
# add this :
generators:
- ./secret-generator.yaml
Step 4: Valid your manifest and push to git
If you haven't installed ksops locally, you won't be able to do next command :
# Validate your manifest locally
kustomize build --enable_alpha_plugins base
Push your secret to Git
git add . \
&& git commit -m "Improved architecture with secrets" \
&& git push
In a minute, argo CD will detect the new commit and apply the changes. You will be able to see your new secret added to the architecture.
Conclusion
Argo CD is a powerful product, the SOPS integration is not that simple though. Even if some steps are required, it's cool to use this kind of tool.
Final step: Destroy resources
# kubernetes
gcloud container clusters delete gke-argocd-demo
# iam
gcloud projects remove-iam-policy-binding $PROJECT_ID \
--member serviceAccount:"gke-argocd-demo@$PROJECT_ID.iam.gserviceaccount.com" \
--role "roles/cloudkms.cryptoKeyEncrypterDecrypter"
gcloud iam service-accounts delete gke-argocd-demo@$PROJECT_ID.iam.gserviceaccount.com