There are many models to choose from when training a musical corpus. We’ll run through the main candidates briefly and explain why we settled on the transformer model with attention when we turn to AI training in the future.
Artificial Intelligence Models
AI models were designed for various purposes, such as pictures, words, and classifying objects (like cats and dogs). Music requires a model that can handle a sequence of events that are in a particular order, and only a handful of those are applicable. The primary ones are shown here:
- RNN: The Recurrent Neural Network architecture is designed to process sequential data one element at a time while maintaining memory from previous elements. However, it is not very good at capturing long-range dependencies as they exist in the Skiptune melodies.
- LSTM/GRU: The Long-Short Term Memory with Gated Recurrent Unit is better than RNNs, but still fundamentally sequential and less scalable to truly long relationships.
- CNN: The Convolutional Neural Network is good at local musical patterns but weaker at modeling relationships between distant phrases.
- Diffusion models: Extremely successful for continuous domains such as images and audio, but less natural for discrete symbolic token prediction.
- State-space models (e.g., Mamba): Promising for very long sequences and efficient inference, but they have less empirical validation than transformers for metadata-conditioned symbolic music generation and offer fewer established tools for interpreting token-level interactions.
Transformers
The transformer with self-attention in 2017 replaced RNN, enabling more effective modeling of long-range dependencies and highly parallel training. For symbolic music tasks like Skiptune, this has become the dominant architecture. It is the architecture of choice for all the mainstream Large Language Models (LLMs). That ability for music to be converted into “words” make it a perfect fit for transformers as an AI model.
Attention is the mechanism that allows the model to decide which earlier tokens are most relevant when predicting the next token. When generating a new tune, for instance, the next note is predicted based on the previous note, but with attention to a host of other factors. These include the key, the meter, the motif that appeared 30 notes earlier, the first notes of the tune, and so on. We can’t list them all because that’s up to the model to figure out in training, and it varies from tune to tune. Every previous token gets a weight. Important factors increase that weight and less important ones decrease that weight.
We often say “transformers with self-attention” because the model is attending to itself. It’s not as though someone, human or otherwise, supplies the information. The input sequence itself provides both the information being examined and the information doing the examining. For every token, the model asks, “Which other tokens in this ssame melody should influence me?”
Transformers with attention allow “conditional generation,” defined as the ability to specify the melody you want in terms of labels. These might include key, meter (time signature), the decade the tune is associated with, the composer, a musical era, a genre, a sung tune vs a dance, a country or ethnic group, etc. The transformer’s self-attention mechanism enables the model to dynamically focus on the conditioning labels that are most relevant when generating each subsequent token. That is, they affect the weights in a way that’s consistent with the labels specified by the user.
Unlike LSTMs, transformers generally continue improving as you increase, provided there is sufficient training data and appropriate regularization. That is, you can increase the number of attention heads, the embedding size, the number of layers and other factors as the training data grows in size. When we begin AI model training, we will need to start small and keep expanding the capacity of the model until it no longer improves its ability to generate good melodies.
An additional advantage is that attention weights can be inspected. Although attention is not a complete explanation of model behavior, it provides a window into which earlier tokens influenced a prediction. That is valuable for both research and supporting the technical rationale of melodic generation.
Finally, transformers have become the dominant architecture for symbolic sequence modeling across human languages, programming languages, DNA sequences, protein folding (in medicine), and, of course, symbolic music.
The duration and duration ratio analysis, pitch and pitch differential analysis, and the Heaps Curves all point to treating symbolic music as having a vocabulary, grammar and syntax all its own. Thus, the weights generated by the Large Language Models are useless for symbolic music, but the architecture they were trained on can be used on symbolic music to create a new set of weights peculiar to musical melodies.
We don’t know whether treating music notes as words in this way will result in the same success that Large Language Models have had, but it will be an exciting avenue to explore in the future.