GitOps deployment of a sample microservice on a Kubernetes cluster: Argo CD as GitOps tool, Loki/Alloy/Grafana for logging, Bitnami Sealed Secrets for secrets. Cloud Operations course project, team of 2.
#kubernetes #gitops #argocd
I was responsible for deploying ArgoCD as the GitOps tool and owning the overall structure: the bootstrap process, the Kustomize layout, and the repo organisation. My teammate built the logging stack (Loki/Alloy/Grafana) and the secrets management with Bitnami Sealed Secrets.
My part in short:
As part of the presentation, I drew out the architecture in my usual, classically overkill fashion. Doing so made it clear how Kustomize and ArgoCD intersect and work together. It also refreshed and solidified my Kubernetes knowledge.
# create a deploy token (read_repository scope) in GitLab repo settings first,
# then copy bootstrap/argocd/repo-credentials.env.example -> repo-credentials.env (gitignored)
# and fill in username/password from the deploy token
# install Argo CD + Kustomize Helm rendering
kubectl apply --server-side --force-conflicts -k bootstrap/argocd
# register the root app (this is the only thing applied by hand, everything else flows from here)
kubectl apply -f bootstrap/root-app.yaml
kubectl port-forward --namespace argocd svc/argocd-server 8080:443
kubectl get --namespace argocd secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
bootstrap/argocd/kustomization.yaml uses Kustomize’s secretGenerator to turn the .env into the kubermeister-repo Secret. Needed since Argo CD can’t read its own repo credentials before it exists:
secretGenerator:
- name: kubermeister-repo
namespace: argocd
envs:
- repo-credentials.env
options:
disableNameSuffixHash: true
labels:
argocd.argoproj.io/secret-type: repository
type: Opaque
Everything is configured with Kustomize, and everything is deployed as its own Argo CD Application.
.
├── bootstrap/
│ ├── root-app.yaml # only thing applied by hand
│ └── argocd/ # installs Argo CD itself + repo secret
├── argocd/
│ ├── applications/ # one Application per deployed thing
│ │ ├── cert-manager.yaml
│ │ ├── cert-manager-issuers.yaml
│ │ ├── sealed-secrets.yaml
│ │ ├── logging-loki.yaml
│ │ ├── logging-alloy.yaml
│ │ ├── logging-grafana.yaml
│ │ ├── moviedb-app-dev.yaml
│ │ ├── moviedb-app-staging.yaml
│ │ └── moviedb-app-prod.yaml
│ └── projects/
│ └── cldop-platform.yaml # AppProject all Applications are scoped to
├── cluster/ # kustomize configs for cluster-wide stuff
│ ├── certmanager-kustomize/
│ ├── logging-kustomize/
│ └── secrets/
└── apps/
└── moviedb-app-kustomize/
├── base/
└── overlays/{dev,staging,prod}/
bootstrap/root-app.yaml is the only Application applied by hand. It points at the argocd/ folder, which is a Kustomization listing every Application in argocd/applications/:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root
namespace: argocd
spec:
project: default
source:
repoURL: https://gitlab.ost.ch/ins-stud/cldop/fs2026/g12/kubermeister-gitops.git
path: argocd
targetRevision: main
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: false
selfHeal: false
Each Application targets one Kustomize overlay and gets a sync-wave for ordering, e.g.:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: moviedb-app-prod
namespace: argocd
annotations:
argocd.argoproj.io/sync-wave: "3"
spec:
project: cldop-platform
source:
repoURL: https://gitlab.ost.ch/ins-stud/cldop/fs2026/g12/kubermeister-gitops.git
path: apps/moviedb-app-kustomize/overlays/prod
targetRevision: main
destination:
server: https://kubernetes.default.svc
namespace: moviedb-app-prod
syncPolicy:
automated:
prune: false
selfHeal: true
syncOptions:
- CreateNamespace=true
Per-environment overlay bumps the image tags and patches the Ingress host/cert-issuer. apps/moviedb-app-kustomize/overlays/prod/kustomization.yaml:
namespace: moviedb-app-prod
resources:
- ../../base
images:
- name: .../frontend
newTag: v1.3.0
- name: .../movie
newTag: v1.3.0
- name: .../description
newTag: v1.3.0
- name: .../comment
newTag: v1.3.0
patches:
- patch: |- # rename namespace (frontend calls are hardcoded to e.g. http://movie:8080, so no nameSuffix allowed)
- op: replace
path: /metadata/name
value: moviedb-app-prod
target: {kind: Namespace, name: moviedb-app}
- patch: |- # per-env host + cert-manager issuer
- op: add
path: /metadata/annotations
value: {cert-manager.io/cluster-issuer: letsencrypt-prod}
- op: add
path: /spec/rules/0/host
value: cldop-stud-g12.network.garden
- op: add
path: /spec/tls
value: [{hosts: [cldop-stud-g12.network.garden], secretName: moviedb-app-tls}]
target: {kind: Ingress, name: moviedb-app, version: v1}
Cutting a release = bump newTag, commit, push:
# edit images.*.newTag in the overlay, then
git add apps/moviedb-app-kustomize/overlays/prod/kustomization.yaml
git commit -m "Update images"
git push
# Argo CD auto-syncs, or force it:
argocd login localhost:8080
argocd app sync moviedb-app-prod
kubectl rollout status deployment/frontend-v1 -n moviedb-app-prod
kubectl get deployment frontend-v1 -n moviedb-app-prod -o jsonpath='{.spec.template.spec.containers[0].image}'
Alloy collects logs via the Kubernetes API and ships them to Loki.
kubectl get pods -n logging
# test Loki directly
kubectl port-forward --namespace logging svc/loki 3100:3100
curl -H "Content-Type: application/json" -XPOST "http://127.0.0.1:3100/loki/api/v1/push" \
--data-raw '{"streams": [{"stream": {"job": "test"}, "values": [["'$(date +%s)'000000000", "fizzbuzz"]]}]}'
curl "http://127.0.0.1:3100/loki/api/v1/query_range" --data-urlencode 'query={job="test"}' | jq .data.result
# Grafana, behind Traefik at grafana.cldop-stud-g12.network.garden, admin pw:
kubectl get secret --namespace logging grafana-admin -o jsonpath="{.data.admin-password}" | base64 --decode; echo
User-Agent breakdown dashboard runs on this LogQL query:
sum by (ua) (
count_over_time(
{app="frontend"}
| json
| __error__=""
| regexp "User-Agent=(?P<ua>[^;]+)"
[$__range]
)
)
Generate test traffic with different UAs:
for ua in Firefox Chrome Safari; do curl -A "$ua" https://cldop-stud-g12.network.garden > /dev/null; done
# or continuous:
while true; do curl -sS https://cldop-stud-g12.network.garden > /dev/null; sleep 0.2; done
Filtering a specific request in Grafana Explore: Loki datasource, {app="frontend"} | json, then add | regexp "User-Agent=(?P<ua>[^;]+)" and pick the fields you want (method, path, ua).
# get the controller's public cert (needed to seal anything)
kubeseal --fetch-cert --controller-name=sealed-secrets-controller --controller-namespace=kube-system
Add a secret:
kubectl create secret generic db-credentials \
--from-literal=username=admin --from-literal=password='super-secret-password' \
--dry-run=client -o yaml > secret.yaml
kubeseal --format yaml \
--controller-name=sealed-secrets-controller --controller-namespace=kube-system \
< secret.yaml > sealed-secret.yaml
rm secret.yaml # never commit the plaintext one
git add sealed-secret.yaml && git commit -m "Add sealed secret" && git push
argocd app sync logging-grafana --prune --force --replace
Revoke = delete the file and push:
echo "" > cluster/logging-kustomize/grafana/grafana-admin-sealed-secret.yaml
git add -u && git commit -m "Revoke Grafana admin credentials" && git push
argocd app sync logging-grafana --prune --force --replace
kubectl get secret grafana-admin -n logging # -> NotFound