When you run Kubernetes on AWS with EKS, your Pods get a special status. Instead of living in a separate, virtual overlay network, they get their own IP addresses directly from your Virtual Private Cloud (VPC). This is handled by the Amazon VPC Container Network Interface (CNI) plugin, the default networking layer for EKS.

This approach makes Pods first-class citizens within your AWS environment. They can communicate directly with other AWS services like RDS databases or S3 buckets using their native VPC IP, just like an EC2 instance would. There's no need for Network Address Translation (NAT), which simplifies network paths, improves performance, and makes security rules more straightforward.

The magic behind this native IP addressing involves a few key components. A Kubernetes DaemonSet called aws-node runs on every worker node in your cluster. This Pod contains two main processes: the CNI plugin itself and a service called IPAMD, or the IP Address Manager Daemon.

The CNI plugin is the binary that Kubernetes's kubelet process calls whenever a Pod is created or destroyed. It's responsible for setting up the network namespace for the Pod and assigning it an IP address.

But where does it get the IP? That's IPAMD's job. IPAMD proactively manages a warm pool of available IP addresses on each node. It does this by requesting secondary private IPv4 addresses from the VPC and attaching them to the node's Elastic Network Interfaces (ENIs). When the CNI plugin needs an IP for a new Pod, it simply grabs one from the local cache managed by IPAMD. This makes Pod startup incredibly fast because there's no waiting for an API call to the EC2 control plane.

This direct-addressing model has a significant constraint: the number of IP addresses an EC2 instance can support. Each instance type has a hard limit on how many ENIs it can have and how many secondary IP addresses can be assigned to each ENI. This directly translates into a maximum number of Pods you can run on a single node.

For clusters with high pod density, this can lead to IP address exhaustion, where you run out of available IPs in your VPC subnet long before you exhaust the node's CPU or memory. To solve this, AWS introduced Prefix Delegation.

Instead of assigning individual IP addresses to a node's ENIs, Prefix Delegation mode assigns entire /28 CIDR blocks (prefixes). Each /28 prefix contains 16 IP addresses. The VPC CNI then manages this block, assigning individual IPs from the prefix to Pods as they launch. This dramatically increases the number of Pods a node can support without needing more ENIs, making much more efficient use of your VPC address space.

For even more control, the VPC CNI supports advanced features like Custom Networking and Security Groups for Pods.

Custom Networking allows you to launch Pods into different subnets than their host nodes. You can create ENIConfig custom resources that specify a subnet and security group. By annotating a worker node to use a specific ENIConfig, you can ensure that all Pods scheduled on that node get their IPs from the designated secondary subnet. This is useful for logically separating traffic or using a VPC address space that is separate from your node's primary network.

Security Groups for Pods takes network policy to the next level. It integrates Kubernetes Pods with EC2 Security Groups. You can assign specific security groups to individual Pods or deployments, allowing you to define fine-grained network rules that control traffic flow between Pods and to other AWS resources. This leverages the familiar, robust security model of EC2 and applies it directly to your containerized workloads, all without needing a separate network policy engine.

Time to test your knowledge on EKS networking.

Mastering the VPC CNI is key to running efficient, secure, and scalable Kubernetes clusters on AWS. By understanding how it assigns native VPC IPs, you can better design your network architecture and troubleshoot connectivity issues.