Data and Type Internals

The Secret Life of Python Objects

In Python, every piece of data, from the number 5 to a complex data structure, is an object. This might sound simple, but it has profound implications for how your code behaves, especially regarding memory and performance. An object is more than just its value; it's a specific entity living at a unique address in your computer's memory.

You can peek behind the curtain using the built-in id() function, which gives you the memory address of an object. This address is the object's 'identity.' When you compare two objects, you can check if they have the same value or if they are the exact same object.

a = [1, 2, 3]
b = [1, 2, 3]
c = a

# Checking for value equality
print(a == b)  # True, their contents are the same

# Checking for identity
print(a is b)  # False, they are two separate objects in memory
print(a is c)  # True, c is just another name for the object a refers to

Understanding this distinction between value (==) and identity (is) is the first step toward writing more efficient, professional Python code. It's not just about what a variable holds, but where it is and what it is.

Interning and Identity

Things get interesting with certain common, immutable objects. To save memory and speed up comparisons, Python pre-allocates and reuses objects for small integers and some strings. This optimization is called object internings.

For integers, Python keeps a global list of all integers from -5 to 256. Any time you create an integer in this range, Python simply points your variable to the existing object instead of creating a new one.

x = 256
y = 256
print(x is y)  # True, because 256 is in the interned range

a = 257
b = 257
print(a is b)  # False, these are two separate objects

A similar process happens with short strings that look like identifiers. This is an implementation detail of and might behave differently in other Python interpreters, but it highlights a key principle: Python works hard behind the scenes to manage memory efficiently.

The True Cost of Data

Every object in Python carries some overhead. An integer isn't just the few bytes needed to store its value; it's a C structure that includes its value, its type, and a reference counts. We can use the sys module to inspect this.

import sys

my_list = []
my_tuple = ()

# The base size of a list object is 64 bytes, even when empty
print(f"Empty list size: {sys.getsizeof(my_list)} bytes")

# A tuple is smaller, with a base size of 48 bytes
print(f"Empty tuple size: {sys.getsizeof(my_tuple)} bytes")

A list has more overhead than a tuple because it's mutable. It needs extra space to accommodate future appends without having to resize on every single addition. This is a fundamental trade-off: mutability offers flexibility at the cost of memory. The sys.getrefcount() function tells you how many variables or data structures are currently pointing to a specific object. The count is always at least one higher than you expect because calling the function itself creates a temporary reference.

This object overhead is why a list of a million integers takes up significantly more memory than a raw array of integers in a language like C.

Copies Deep and Shallow

When you have complex, nested data structures, simply copying a list doesn't always do what you expect. Python distinguishes between shallow and deep copies.

A shallow copy creates a new object, but inserts references to the objects found in the original. If you change a nested object in one, it changes in both.

A deep copy creates a new object and recursively copies all objects found inside the original. This creates a fully independent clone.

import copy

original = [1, [2, 3]]

# Shallow copy
shallow = copy.copy(original)
shallow[1][0] = 'X'
print(f"Original after shallow modify: {original}") # Prints [1, ['X', 3]]

# Reset original
original = [1, [2, 3]]

# Deep copy
deep = copy.deepcopy(original)
deep[1][0] = 'Y'
print(f"Original after deep modify: {original}") # Prints [1, [2, 3]]

Choosing the right copy method is crucial. Deep copies prevent unintended side effects but use more memory and are slower to create. Shallow copies are fast but require careful management of shared mutable state.

Optimizing with Specialized Tools

The Python standard library offers specialized data structures in the collections module that provide performance and memory benefits over general-purpose lists and dictionaries.

Collection	Use Case	Benefit
`NamedTuple`	Creating simple, immutable classes without boilerplate	Lightweight, self-documenting, memory-efficient
`Deque`	Fast appends and pops from both ends (a double-ended queue)	O(1) time complexity for append/pop from either side
`Counter`	Counting hashable objects	Efficiently tallies items in a sequence or iterable

For custom classes, you can significantly reduce memory usage by using __slots__. When you define __slots__ in a class, you tell Python not to use a dynamic dictionary to store instance attributes. Instead, it allocates a fixed amount of space for just the specified attributes, much like a C struct. This can lead to massive memory savings when creating thousands or millions of small objects.

import sys

class PointDict:
    def __init__(self, x, y):
        self.x = x
        self.y = y

class PointSlots:
    __slots__ = ['x', 'y']
    def __init__(self, x, y):
        self.x = x
        self.y = y

p_dict = PointDict(1, 2)
p_slots = PointSlots(1, 2)

# The __slots__ version is significantly smaller
print(f"Dict version size: {sys.getsizeof(p_dict)} bytes") # Varies, but larger
print(f"Slots version size: {sys.getsizeof(p_slots)} bytes") # Varies, but smaller

The trade-off for __slots__ is that you can't add new attributes to instances on the fly. You gain memory efficiency by sacrificing some of Python's dynamic nature.

Ready to test your understanding of Python's data internals?

Quiz Questions 1/6

Consider the following Python code:

a = [1, 2, [3, 4]]
b = a.copy() # This is a shallow copy
b[0] = 99
b[2][0] = 77

What will be the value of a after this code executes?

Quiz Questions 2/6

In CPython, why does x = 256; y = 256; x is y typically evaluate to True, while a = 257; b = 257; a is b often evaluates to False?

Understanding these internals—how Python treats identity, manages memory, and provides specialized tools—separates a hobbyist from a professional. It's the foundation for writing code that is not only correct but also scalable and efficient.

Python Mastery from Core to Advanced Architecture