This article is part of the EKS Anywhere series EKS Anywhere., extending the Hybrid cloud momentum

Not for the faint-hearted… This is a deep dive in architectural composition of EKS Anywhere and quite a long-read detailing functional components and their interactions.

Don't get jittery about it! Remember, installation of EKS Anywhere with your first production grade cluster (if prerequisites are met) is a mere 45-to-60-minute task. The subsequent clusters are merely 8–10 minutes each (for a 2 control + 2 worker + 3 etcd node) configuration.

Let's first understand EKS Anywhere from a solution standpoint

Question: What is EKS Anywhere., and I have also come across the term EKS Distro. Are these different., or the same…. I am confused!!!

Let’s attempt to answer this through the below visual, where you can see that EKS Distro is the actual Kubernetes distribution that runs in the clusters, whereas EKS Anywhere is an orchestrated tooling packaged as a bundled software package created by AWS that is used to create EKS Distro based Kubernetes clusters

For the technology scuba and someone who likes to uncover the actual implementations, the below visual is really oversimplified and places EKS Distro as a separate method., which in my view is misleading!!!

In my own perspective and to best of my knowledge, I would re-paint the visual like this, placing EKS-Distro right in the center, where it’s agnostic of the methods.

Most importantly, what needs to be highlighted is that EKS Distro can be self-deployed using community tools (KOPS, Kubeadm, Kubeinit) on a variety of underlying infrastructure., but the key consideration in treading this path would parallel an enlightened Kubernetes soul!!!

OR

using EKS Anywhere, the preferred and purpose-built toolchain built by AWS for on-premises installations of EKS Distro on vSphere environments (bare-metal support coming soon)

The EKS Anywhere project wraps Cluster API, various CLI and plugins (eksctl cli, anywhere plugin, kubectl, aws-iam-authenticator) and other methods, routines of downloading required artefacts and release bundles in a single package to ease the administrative burden of deploying EKS Distro. In summary., a rich and evolving software package that is a preferred way of deploying EKS Distro based clusters on-premises.

Wow quite a few heavy terms used: EKS Anywhere, EKS Distro, EKS Anywhere control plane, Bootstrap cluster, CAPI/CAPV-Cluster API, Standalone clusters, Management cluster, Workload cluster. DON’T WORRY., we will dig deeper into that later on.

For now, understand that AWS has open-sourced EKS Distro and EKS Anywhere projects and both of these are available via GitHub. The beauty of EKS Distro is that it is the same Kubernetes distribution that runs on AWS EKS cloud service. Being open sourced, one can deploy EKS Distro in a variety of ways for a consistent Kubernetes distribution experience.

Question: Ok, now I understand a bit about EKS Anywhere and EKS Distro Can you tell me a bit more of how it is deployed?

A high-level understanding of the two topologies/models is explained in this video

EKS Anywhere deployments can occur in two patterns and AWS has nicely documented the differences in this URL

https://anywhere.eks.amazonaws.com/docs/concepts/cluster-topologies/#when-should-i-deploy-a-management-cluster

First pattern: Standalone clusters. Since this is a standalone cluster, the EKS Anywhere control plane would be a combination of the components running on EKS Administrative machine plus the Cluster API components running on the standalone clusters.

Second pattern: Dedicated management cluster model. An interesting callout for this model would be that the term “management cluster” is more notional in nature. It is actually a standalone cluster referenced as management cluster that further acts to manage other workload clusters. The difference between the management (or standalone cluster) and the workload cluster is the presence of the cluster API components.

EKS Anywhere, Under the Hood!

The first and primary thought: Kubernetes clusters are a set of nodes or virtual machines stacked up with all the software required to run Kubernetes. So how are these clusters defined, configured, etc. in the context of EKS Anywhere. In that stride it’s quintessential to at-least understand the overview of Cluster API framework.

The video provides more than peep into the overall interactions of various build tooling in EKS Anywhere framework

Welcome Cluster API, a comprehensive method to building Kubernetes clusters using the declarative Kubernetes approach itself. In this framework, the cluster API software leverages a set of machine templates written in the same declarative style used by Kubernetes. And then., it communicates with the infrastructure provider APIs to create these machines intended to be the Kubernetes nodes based on these machine templates.

The cluster and machine templates themselves are a resource definition of sorts including compute, storage, network, operating system, etc. configurations that define the cluster nodes. In addition, the templates also consist of methods to talk to the infrastructure provider.

Note: Cluster API is a broader framework supporting major infrastructure providers, however in the context of EKS Anywhere this list may be limited to the specific integrations (e.g., vSphere, Apache Cloudstack) for virtualized IaaS

The below figure is straight out of Cluster API handbook and basically gives a high level overview of a parent task for creating clusters

Once the machine templates are validated and rendered by the underlying infrastructure provider, a series of ongoing reconciliations and health checks are performed. In addition to that, changes in the form of resource requests are monitored and deployed for existing clusters. These actions could be scale, upgrade or delete these machines, nodes via a change in the template. Now let’s do a quick peep into the Cluster API itself. Let’s reason this as a software package itself that constitutes of all the magic of doing the above.

Question: How and where do I run this Cluster API software? Simple answer is that Cluster API can run on any compliant Kubernetes cluster. However, to avoid a chicken and egg situation, the cluster API can be bootstrapped using a KinD (Kubernetes in Docker) cluster!

Most importantly you do not have to manually install Cluster API in EKS Anywhere, it's embedded as a part of the build tooling.

If you haven’t read or are not familiar with Cluster API framework., then the below URL can be a good starting point. However, for now I would suggest that you skip this read and come back when you have some practical hands on with deploying EKS Anywhere., else the tendency to procrastinate will be higher.

Introduction — The Cluster API Book (k8s.io)

Now back to our topic and some more obvious questions!!!

When the command to deploy a cluster is executed for a standalone workload or a dedicated management cluster, a temporary KinD cluster known as a bootstrap cluster is automatically created on the EKSA administrative machine. This bootstrap KinD cluster then uses the inbuilt Cluster API routines that instructs the vSphere to create the cluster nodes.

You can see from the process of cluster creation for Standalone workload cluster or a management cluster and observe what’s going on

If you are a serious deep-diver and want to understand more

EKS Anywhere release manifest: This release bundle gets downloaded everytime one creates a new standalone workload cluster or management cluster. The manifest is the entry point to everything else that happens in creating the EKS Anywhere cluster/s
Specific artifacts used by EKS Anywhere: Visit the page eks-anywhere/artifacts.go at main · aws/eks-anywhere (github.com) and one can view the v1alpha1 version of artifacts bundled within EKS Anywhere
EKS Anywhere Build tooling: go to EKS-Anywhere-build-tooling. The URL should lead to the README section of Base image track, and you can select “Click to view/hide table”. It should give you an indicative view of what tools are used to create the artefacts for EKS Anywhere.
EKS-Distro release bundles: If one is interested in gaining further knowledge, you can check here EKS Distro (amazonaws.com). Click on the individual releases to see the release manifest and the components included.

OK., now on to the cluster constructs, what runs inside our EKS Anywhere clusters

Depending on the role of the node (control/worker/etcd) and the type of cluster (standalone/management/managed workload), the functional components will be rendered as PODS in different namespaces.

Please increase the browser zoom to see which PODS are running in the respective nodes.

Let's unravel some additional points

Operating system used by EKS Anywhere cluster nodes: Bottle-rocket (AWS developed OS) or a standard Ubuntu 20.04 OS image or a custom Ubuntu 20.04 OVA image. There are good number of reasons why one would prefer one versus other. However, the primary reason of using ubuntu would be to customize the OS via additional software packages or bring in self-signed certificates since bottle-rocket does not allow for any of those.

Bottle-rocket is automatically downloaded by EKS Anywhere from the AWS repositories while creating the first cluster itself. The ubuntu OS template needs to be built with a specific procedure and if required one can perform customizations.

Infrastructure provider: EKS Anywhere can run on vSphere 7+ and vSphere 6.7 u3 and above, bare metal or apache cloudstack. There are ongoing developments and will continue and I expect that in future this list will only expand. Although my PoC environment works perfectly fine on vSphere 6.7 u3+, it is not recommended as AWS will not support it contractually.

Other dimensions: Lot of information available on EKS anywhere dedicated website Amazon EKS Anywhere | EKS Anywhere (amazonaws.com) in terms of preferred CNI, CSI, load balancers, ingress, Authentication mechanisms, etc. But the key point to note is that while the website states some specific tools, the EKS Anywhere partner ecosystem and the community driven use-cases are on the rise.

As an Example, the default CNI used by EKS Anywhere is Cilum., well does it mean I cannot use Calico. Of course, you can use Calico., but it could mean a complex arrangement of sorts from a support standpoint… see the blog from Calico team. Calico on EKS Anywhere (tigera.io)

Likewise, the preferred Ingress as listed on the EKS Anywhere website is Gloo Edge or Emissary-ingress., but that does not mean one cannot deploy NGINX ingress on EKS Anywhere clusters.

Deployment & Support model: Well, the only reference I could see is that AWS supports EKS Anywhere with a pre-requisite that customers should have AWS Enterprise support subscription. I am quite unsure of the partner plays here as the entire thing is open-sourced., so I guess this is an interesting playground for many services partners as the deployments increase. An important point to note is that when subscribed to AWS support, one can avail of add-ons in the form of curated packages for certain open-source

This brings us to a logical conclusion to the architectural decoding of EKS Anywhere. I know there is a lot to digest., but if you fancy the inner workings then hopefully this article along with the links included gives you a good headstart!

To be continued., if I find more useful information that can be added to this article

cheers,

Ambar Hassani

#iwork4dell

Browse By Board

Developer Blog

Devops Forum

Developers

EKS Anywhere, ubuntu OS images for cluster nodes

EKS Anywhere, extending the Hybrid cloud momentum

Cloud native app development using Dell XPS 13 Plus Developer Edition and MicroK8s

Backup Assessment Tool for PowerProtect Data Manager

EKS Anywhere, Building the Administrative machine

REST API and Dell Technologies Storage - PowerStore

REST API and Dell Technologies Storage - Unity

EKS Anywhere, Environmental substrate & prerequisites

EKS Anywhere, decoding the architecture.

REST API and Dell Technologies Storage - PowerMax

Snyk - Making Cloud Security Accessible to Developers

An introduction to the 4C’s of Cloud Native Security

CockroachDB, giving back and building community

The Foundry: a quick-launch solution for getting into containers

cdCon2: -- A quick overview of Spinnaker: the open source CD project

Dell Support Resources

Latest Solutions

Data Refresh Latency in Open Vulnerability API: Is there a 30-day lag?

test color post

Unable to fetch version of array error when running Ansible module

Asset API Authentication

Unable to import PowerScale python library. Please install isilon-sdk. Please install the.