Modern language models are typically trained on subword sequences, but define probabilities over character strings. A subword tokenizer, most commonly byte-pair encoding, maps between character strings and subword strings, with the concrete mapping affecting the assigned probabilities. Morphologically informed extensions to byte-pair encoding aim to optimize this mapping by introducing linguistically motivated boundaries to the process. Unfortunately, this usually increases tokenizer fertility, confounding compression with boundary alignment.

We introduce a novel extension to the byte-pair encoding training process that trades frequency-based merges against morphologically motivated ones, controlled through a single parameter. This effectively allows for tokenizers to be trained at matched fertility. Matching fertility bounds how far the segmentation can change, and at that scale neither perplexity nor accuracy on downstream tasks separates the aligned tokenizers.

We find that estimates of tokenization bias (Lesci et al., 2025) do separate them. At higher vocabulary sizes, tokenizers of varying degrees of morphological alignment separate clearly from standard byte-pair encoding. However, we observe no stable trend across alignment levels. In a morphologically simple language at low fertility, the signal-to-noise ratio is too weak to establish such a trend, and an effect of alignment may fall below what our setup can resolve. Separately, we estimate tokenization bias across training checkpoints. For every model we train it declines, where previous work reports the opposite.