One of the first things I do after deploying a Kubernetes cluster is to install a CSI driver to provide persistent storage to my workloads; coupled with a GitOps workflow; it takes only seconds to be able to run stateful workloads.

The GitOps process is nothing more than a few principles:

Git as a single source of truth
Resource explicitly declarative
Pull based

Nonetheless, to ensure that the process runs smoothly, you must make certain that the application you will manage with GitOps complies with these principles.

This article describes how to use the Microsoft Azure Arc GitOps solution to deploy the Dell CSI driver for Dell PowerMax and affiliated Container Storage Modules (CSMs).

The platform we will use to implement the GitOps workflow is Azure Arc with GitHub. Still, other solutions are possible using Kubernetes agents such as Argo CD, Flux CD, and GitLab.

Azure GitOps itself is built on top of Flux CD.

Install Azure Arc

The first step is to onboard your existing Kubernetes cluster within the Azure portal.

Obviously, the Azure agent will connect to the Internet. In my case, the installation of the Arc agent fails from the Dell network with the error described here: https://docs.microsoft.com/en-us/answers/questions/734383/connect-openshift-cluster-to-azure-arc-secret-34ku.html

Certain URLs (even when bypassing the corporate proxy) don't play well when communicating with Azure. I have seen some services get a self-signed certificate, causing the issue.

The solution for me was to put an intermediate transparent proxy between the Kubernetes cluster and the corporate cluster. That way, we can have better control over the responses given by the proxy.

In this example, we install Squid on a dedicated box with the help of Docker. To make it work, I used the Squid image by Ubuntu and made sure that Kubernetes requests were direct with the help of always_direct:

docker run -d --name squid-container ubuntu/squid:5.2-22.04_beta ; docker cp squid-container:/etc/squid/squid.conf ./ ; egrep -v '^#' squid.conf > my_squid.conf
docker rm -f squid-container

Then add the following section:

acl k8s        port 6443        # k8s https
always_direct allow k8s

You can now install the agent per the following instructions: https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/quickstart-connect-cluster?tabs=azure-cli#connect-using-an-outbound-proxy-server.

export HTTP_PROXY=http://mysquid-proxy.dell.com:3128
export HTTPS_PROXY=http://mysquid-proxy.dell.com:3128
export NO_PROXY=https://kubernetes.local:6443
 
az connectedk8s connect --name AzureArcCorkDevCluster \
                        --resource-group AzureArcTestFlorian \
                        --proxy-https http://mysquid-proxy.dell.com:3128 \
                        --proxy-http http://mysquid-proxy.dell.com:3128 \
                        --proxy-skip-range 10.0.0.0/8,kubernetes.default.svc,.svc.cluster.local,.svc \
                        --proxy-cert /etc/ssl/certs/ca-bundle.crt

If everything worked well, you should see the cluster with detailed info from the Azure portal:

Add a service account for more visibility in Azure portal

To benefit from all the features that Azure Arc offers, give the agent the privileges to access the cluster.

The first step is to create a service account:

kubectl create serviceaccount azure-user kubectl create clusterrolebinding demo-user-binding --clusterrole cluster-admin --serviceaccount default:azure-user kubectl apply -f - <

Then, from the Azure UI, when you are prompted to give a token, you can obtain it as follows:

kubectl get secret azure-user-secret -o jsonpath='{$.data.token}' | base64 -d | sed $'s/$/\\\n/g'

Then paste the token in the Azure UI.

Install the GitOps agent

The GitOps agent installation can be done with a CLI or in the Azure portal.

As of now, the Microsoft documentation presents in detail the deployment that uses the CLI; so let's see how it works with the Azure portal:

Organize the repository

The Git repository organization is a crucial part of the GitOps architecture. It hugely depends on how internal teams are organized, the level of information you want to expose and share, the location of the different clusters, and so on.

In our case, the requirement is to connect multiple Kubernetes clusters owned by different teams to a couple of PowerMax systems using only the latest and greatest CSI driver and affiliated CSM for PowerMax.

Therefore, the monorepo approach is suited.

The organization follows this structure:

.

├── apps

│ ├── base

│ └── overlays

│ ├── cork-development

│ │ ├── dev-ns

│ │ └── prod-ns

│ └── cork-production

│ └── prod-ns

├── clusters

│ ├── cork-development

│ └── cork-production

└── infrastructure

├── cert-manager

├── csm-replication

├── external-snapshotter

└── powermax

apps: Contains the applications to be deployed on the clusters.
We have different overlays per cluster.
cluster: Usually contains the cluster-specific Flux CD main configuration; using Azure Arc, none is needed.
Infrastructure: Contains the deployments that are used to run the infrastructure services; they are common to every cluster.
- cert-manager: Is a dependency of powermax reverse-proxy
- csm-replication: Is a dependency of powermax to support SRDF replication
- external-snapshotter: Is a dependency of powermax to snapshot
- powermax: Contains the driver installation

You can see all files in https://github.com/coulof/fluxcd-csm-powermax.

Note: The GitOps agent comes with multi-tenancy support; therefore, we cannot cross-reference objects between namespaces. The Kustomization and HelmRelease must be created in the same namespace as the agent (here, flux-system) and have a corresponding targetNamespace to the resource to be installed.

Conclusion

This article is the first of a series exploring the GitOps workflow. Next, we will see how to manage application and persistent storage with the GitOps workflow, how to upgrade the modules, and so on.

Resources

Browse By Board

Developer Blog

Devops Forum

Developers

Register Now! for Secure Secrets Management with Kubernetes & DevSecOps

The Godfather of DevOps offers his advice for those getting started

cdCon1 : Firefly's cloud asset management solution

CSI/CSM for PowerMax with Azure Arc agent - Blog by Florian Coulombel

Ansible - create dellemc.powerstore.filesystem with config type VMware

Unity CSI driver version VendorVersion=2.3.0 support OCP 4.11

KubeCon 2022 - Who is going ? Join our DevOps Deb8s

KubeCon 2022 - Who is going ? Have fun with us!

PowerScale Ansible Module v1.6 dellemc.powerscale.smb returns ******** for character "a"?

Dell’s Developer Relations team welcomes Tim Banks

Getting started with DevOps, GitOps and an intro to OpenShift

Authorization URL - TechDirect API in Power Automate

VRO plugin workflow intermittent network connection error

Initiator alias fails as invalid in host module

csi unity v2.2.0 unable to attach pv to the node where the pod is scheduled

Dell Support Resources

Latest Solutions

Data Refresh Latency in Open Vulnerability API: Is there a 30-day lag?

test color post

Unable to fetch version of array error when running Ansible module

Asset API Authentication

Unable to import PowerScale python library. Please install isilon-sdk. Please install the.