Messaging Layer Security (MLS) implements a Continuous Group Key Agreement (CGKA) protocol, representing a significant architectural departure from the pairwise session model used in legacy secure messaging. Protocols like Signal establish security through a series of independent Double Ratchet sessions between all pairs of members in a group. While effective for small groups, this approach scales poorly, as the number of sessions grows quadratically with the number of members ($O(n^2)$).

MLS, as defined in RFC 9420, replaces this pairwise architecture with a shared, tree-based group state. The core of this model is the Ratchet Tree, a binary tree where each leaf node corresponds to a group member's public key. This structure allows the entire group to efficiently compute a shared secret key by performing Diffie-Hellman calculations only along the direct path from a leaf to the root. The result is a system that scales logarithmically ($O(\log n)$), making it viable for groups with thousands of members.

The cryptographic state of an MLS group is not static; it evolves through a sequence of discrete periods called epochs. An epoch represents a fixed group membership and a corresponding set of cryptographic keys. Any change to the group's composition or a member's key material triggers a transition to a new epoch.

These transitions are driven by three fundamental operations, delivered via a Commit message:

• Add: A new member is added to the group. The committer adds a new leaf to the Ratchet Tree and sends a Welcome message containing the group's state to the new member.
• Update: An existing member refreshes their key material. They generate a new key pair, update their leaf node, and this change propagates up their direct path to the root of the tree.
• Remove: A member is removed from the group. The committer effectively blanks the removed member's leaf node and updates the direct path to ensure the removed member cannot compute future group secrets.

This state evolution mechanism is central to providing forward secrecy (FS) and post-compromise security (PCS). FS ensures that a compromise of the current group state does not compromise past messages, as each epoch uses distinct keys. PCS allows the group to regain security after a compromise. If a member's device is compromised, they can issue an Update from an uncompromised device, or an admin can Remove them. Both actions trigger a new epoch with new keys that the adversary, who possessed the old state, cannot compute.

The security of each epoch hinges on a shared group_secret. This secret is derived from the root node of the Ratchet Tree. To transition to a new epoch, the group_secret is updated using a Key Derivation Function (KDF). The MLS Key Schedule, specified in RFC 9420, defines this process.

The group_secret for a new epoch is derived from two inputs: the group_secret of the preceding epoch and an update_secret. The update_secret is generated by the member committing the change and is included in their Commit message. This chaining mechanism ensures that each epoch's secret is cryptographically bound to the previous one.

From this epoch-specific group_secret, the Key Schedule derives a full suite of working keys, including:

• An application_secret for encrypting application data.
• A sender_data_secret for encrypting sender information and metadata.
• A handshake_secret for encrypting subsequent Commit messages within the same epoch.
• An exporter_secret for use by higher-level application protocols.

This separation of keys adheres to the principle of cryptographic hygiene, ensuring that keys are used for a single, specific purpose.

MLS distinguishes between two primary message types that interact with the group state:

1. Handshake Messages: These messages (Add, Update, Remove packaged in a Commit) modify the group state. They are processed in a well-defined order by all group members to ensure that everyone converges on the same cryptographic state for the next epoch.

2. Application Messages: These are the actual data messages (e.g., text, files) exchanged by members. They are encrypted using keys derived from the current epoch's application_secret. Critically, they do not alter the group state.

This design is shaped by several core constraints. Scalability is achieved through the Ratchet Tree, which minimizes the computation and communication overhead for updates. Asynchronicity is a fundamental requirement; members do not need to be online simultaneously to participate. A member coming online can receive a sequence of Commit messages and process them to catch up to the current group state. Finally, Security (FS and PCS) is baked into the state transition model, ensuring the group can maintain confidentiality and recover from compromise efficiently, even at scale.