Protein Language Models: Learning from the Sequence

A protein is, at one level, just a string of characters drawn from a 20-letter alphabet. The letters are amino acids. The string is the sequence.

The claim behind protein language models (PLMs) is that if you train a large transformer on enough of these strings, the model will learn representations that capture something real about protein biology — structure, function, evolutionary relationships. This turns out to be true, and it's one of the more surprising results in recent computational biology.

Why sequences contain information

Evolution is a compression algorithm. Two proteins with similar sequences are usually similar because they share a common ancestor. Mutations that break protein function are selected against, so the sequence space of functional proteins is not random. It's highly constrained.

This means that a multiple sequence alignment (MSA) of homologous proteins — proteins that evolved from the same ancestor — contains a record of which positions can vary and which cannot. Positions under strong evolutionary constraint are structurally or functionally important. Correlated mutations (if position $i$ changes, position $j$ tends to change too) reveal spatial contacts in the folded structure.

Extracting this signal manually via co-evolutionary analysis was the basis of methods like EVcouplings and Gremlin. The PLM approach is different: learn it end-to-end.

The transformer architecture

A transformer maps a sequence of tokens $x_1, x_2, \ldots, x_n$ to a sequence of representations $h_1, h_2, \ldots, h_n$ through layers of self-attention and feedforward networks.

The self-attention operation at each layer computes:

$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$

where $Q = XW_Q$ , $K = XW_K$ , $V = XW_V$ are the query, key, and value projections of the input $X$ .

For a protein sequence, each token is an amino acid. After enough layers, the representation $h_i$ of residue $i$ aggregates context from every other residue in the sequence — including the long-range correlations that encode contact information.

Masked language modeling

PLMs are trained with masked language modeling (MLM), borrowed directly from BERT in NLP. During training, a fraction $p$ of tokens are replaced with a special [MASK] token, and the model is trained to predict the original amino acid:

$\mathcal{L} = -\sum_{i \in \mathcal{M}} \log P(x_i \mid x_{\setminus \mathcal{M}})$

where $\mathcal{M}$ is the set of masked positions and $x_{\setminus \mathcal{M}}$ is the unmasked context.

This is a self-supervised objective — no labels required. You need only sequences, of which there are hundreds of millions in databases like UniRef and UniProt.

The remarkable thing is that optimizing this objective forces the model to learn something about protein biochemistry. To predict a masked residue accurately, the model must understand what residues are compatible with the surrounding context — which is determined by structural and functional constraints.

ESM-2: scale matters

The ESM family of models (Evolutionary Scale Modeling, from Meta AI) are the most widely used PLMs. ESM-2 was trained on 250 million protein sequences and comes in sizes ranging from 8M to 15B parameters.

import esm
import torch

model, alphabet = esm.pretrained.esm2_t33_650M_UR50D()
batch_converter = alphabet.get_batch_converter()
model.eval()

data = [("protein_1", "MKTIIALSYIFCLVFA")]
batch_labels, batch_strs, batch_tokens = batch_converter(data)

with torch.no_grad():
    results = model(batch_tokens, repr_layers=[33], return_contacts=True)

# Per-residue embeddings from the last layer
embeddings = results["representations"][33]  # shape: (batch, seq_len, 1280)

The embeddings from the last layer encode rich information. Probing experiments show that:

Linear classifiers trained on ESM-2 embeddings predict secondary structure with high accuracy
The attention maps predict residue-residue contacts without any structural supervision
The representations cluster by protein family in embedding space

Where this is useful in drug design

In my research, we use PLM embeddings in two ways:

1. Antibody affinity maturation. Given an antibody sequence and a target protein sequence, we want to predict how tightly the antibody binds. We extract PLM embeddings for both sequences and pass them through a cross-attention layer that learns to model the interface. The embeddings carry enough structural information that the model can generalize across protein families.

2. Conditioning generative models. In conditional molecular generation, we want to design small molecules that bind a specific protein. The protein can be represented as its PLM embedding, which the generator uses as a conditioning signal. This avoids the need for explicit 3D structure (though structure-based conditioning is also possible and often better when structure is available).

Limitations

PLMs are powerful but not magic. A few things to keep in mind:

They don't model structure directly. ESM-2 is sequence-only. AlphaFold2's embeddings come from a structure-conditioned model, which is different.
They generalize within sequence space, not beyond it. For highly novel sequences with no homologs, the representations are less reliable.
Embeddings can be large. A 1280-dimensional vector per residue is expensive if you're processing millions of sequences.

ESMFold combines ESM-2 embeddings with a structure module to predict 3D structure — essentially using PLM representations as the starting point for structure prediction. This has made fast, single-sequence structure prediction practical, which matters a lot for large-scale virtual screening.