Deploying Prometheus in EKS with CSI Driver: A Comprehensive Guide
Introduction:
Elastic Kubernetes Service (EKS) is one of the easiest ways to use Kubernetes managed by AWS. This guide assumes you have a basic understanding of EKS, including deploying pods. Here, we address a common issue: Prometheus pods (or other pods requiring volumes) are stuck in a Pending status due to improper volume configuration.
Prerequisites:
- EKS Cluster:
- Create an EKS cluster using the following command: eksctl create cluster –name <name> –region <region> –node-type t2.large –managed
This provisions an EKS cluster with t2.large EC2 instances. Note: This incurs costs.
- Create an EKS cluster using the following command: eksctl create cluster –name <name> –region <region> –node-type t2.large –managed
- kubectl Installed:
- Install the kubectl command for configuring the EKS cluster. Official Documentation
- Helm Package Manager:
- Install Helm for Kubernetes. Installation Guide
- Knowledge of OIDC:
- Understand OIDC (Open ID Connect) and its integration with AWS. AWS OIDC Documentation
Principles of the CSI Driver in the EKS Cluster:
In a Kubernetes cluster, the Service Account is the final entity responsible for essential tasks, such as provisioning pods and accessing resources outside the cluster (e.g., provisioning EBS volumes in AWS, GCP, etc.). Initially, Kubernetes core included a volume plugin to manage all volume-related actions, such as:
- Deleting volumes
- Provisioning new volumes
- Increasing volume sizes
However, managing these tasks directly within the Kubernetes core created significant administrative overhead. To address this, Kubernetes introduced the CSI (Container Storage Interface) driver, which offloads these actions from the core. The CSI driver operates as a DaemonSet in the kube-system namespace, where most core functionalities reside (e.g., etcd server, controller, kube-config, etc.).
In a Kubernetes cluster, the ServiceAccount serves as the primary entity responsible for executing key tasks such as provisioning pods and accessing external resources required by the cluster, such as EBS volumes in cloud environments like AWS or GCP. Previously, Kubernetes managed storage operations through in-tree volume plugins that handled tasks like provisioning, deleting, and resizing volumes. However, this approach became an administrative burden for the Kubernetes core. To address this, the Container Storage Interface (CSI) driver was introduced to offload these responsibilities, removing the need for volume management to be tightly coupled with the Kubernetes core.
The CSI driver is deployed as DaemonSets in the kube-system namespace, where core Kubernetes services like the API server, controller manager, etcd operate. These DaemonSets implement storage functionality and interact with external storage systems to handle tasks such as provisioning, attaching, and mounting volumes. This architecture allows Kubernetes to remain modular while relying on CSI drivers to manage storage operations efficiently.
When a Kubernetes workload requests persistent storage, the process begins with Kubernetes checking for a suitable volume in the cluster. If none exists, Kubernetes uses the CSI driver to dynamically provision the required volume, ensuring seamless integration with external storage systems. The sequence of events during this process is outlined below.
Sequence of Events: