AWS EKS Advanced Architecture and Optimization
VPC CNI Custom Networking
Solving EKS IP Exhaustion
In large-scale Amazon EKS deployments, a common and critical bottleneck is the exhaustion of RFC1918 IP address space. The default VPC CNI behavior, where pods draw IPs from the same subnet as their host nodes, can rapidly deplete available addresses, especially in shared VPC environments. Custom networking with the Amazon VPC CNI plugin directly addresses this by decoupling pod networking from the node's primary network interface.
This approach enables pods to acquire IP addresses from secondary VPC CIDR blocks, which can reside in entirely different subnets from the worker nodes. The mechanism for this is the ENIConfig Custom Resource Definition (CRD). By defining an ENIConfig for each Availability Zone, you instruct the CNI's IPAM daemon where to provision Elastic Network Interfaces (ENIs) for pod networking. This effectively creates a separate, logical network for your data plane, preserving the primary CIDR for node and control plane communication.
The ENIConfig Custom Resource
The ENIConfig object is the cornerstone of custom networking. It's a simple yet powerful resource that contains two key pieces of information: the subnet ID for pod ENIs and the security groups to be attached to those ENIs. You create one ENIConfig per AZ in your cluster.
apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
name: us-east-1a-pod-net
spec:
securityGroups:
- sg-0123456789abcdef0
subnet: subnet-0fedcba9876543210
To activate this configuration, you must patch the aws-node DaemonSet to set AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG to true. You also annotate each worker node with k8s.amazonaws.com/eniConfig, specifying the name of the ENIConfig for its AZ. For example, a node in us-east-1a would be annotated to use the us-east-1a-pod-net configuration. When a new node joins, the CNI reads this annotation and uses the specified ENIConfig to attach ENIs from the designated pod-only subnet.
Boosting Pod Density and Security
Beyond solving IP exhaustion, custom networking unlocks powerful features for optimizing pod density and enforcing granular security policies. These are not mutually exclusive; you can combine them to build highly efficient and secure clusters.
Prefix Delegation
noun
A feature where the VPC CNI plugin requests a /28 IPv4 prefix (16 IPs) from the EC2 control plane for an ENI, rather than requesting individual secondary IP addresses. This significantly reduces API calls and increases the number of available IPs per ENI.
When you enable prefix delegation (ENABLE_PREFIX_DELEGATION=true), the IPAM daemon requests entire /28 IPv4 prefixes instead of individual /32 IPs. Each prefix provides 16 addresses. This dramatically increases the IP density per ENI, allowing for a much higher pod count per node. The maximum number of pods is no longer limited by the number of secondary IPs an instance type supports but rather by the number of prefixes available.
Another major advantage is the ability to use effectively. When pods get their IPs from a dedicated ENI, that ENI can have its own security groups, defined in the ENIConfig. This allows you to apply fine-grained, pod-level network security rules directly through AWS security groups, a familiar and powerful tool. This isolates workloads far more effectively than relying solely on Kubernetes Network Policies.
Calculating Max Pods
With custom networking and prefix delegation, the formula for calculating the maximum number of pods per node changes. It's no longer a simple function of instance memory. The primary constraint becomes the number of available IP addresses, which is determined by the number of ENIs and the prefixes assigned to them.
Always consult the AWS documentation for the specific ENI, IPv4, and prefix limits for your chosen instance types. These values are the key inputs to accurately calculate your cluster's pod density potential.
Here is a transition into a quiz.
What is the primary problem that custom networking with the Amazon VPC CNI plugin is designed to solve in large-scale EKS deployments?
Which Kubernetes resource is used to define the specific subnet and security groups for pods in each Availability Zone when implementing custom networking?
Configuring custom networking correctly is essential for building scalable and secure EKS clusters, preventing common issues like IP exhaustion before they can impact production workloads.