We wrap up the discussion on Heaps Law this week by summarizing what we’ve learned so far and making a final decision for what definition of “word” we use when we get around to training the AI model. This table summarizes all the Heaps Curve analyses performed:
| Corpus | Representation | Ordering | N (No. of ‘words’) | V (Vocabulary) | Beta | R-Squared |
|---|---|---|---|---|---|---|
| Human Languages | Words | Natural | Varies | Varies | 0.4 – 0.6 | 0.95 – 0.99+ |
| Moby Dick | Words | Original text | 209,000 | 95,000 | 0.60 | 0.99+ |
| Skiptune | Note | As Entered | 4.9 million | 3,940 | 0.32 | 0.97 |
| Skiptune | Two-Notes | As Entered | 4.9 million | 6,609 | 0.38 | 0.98 |
| Skiptune | Note | Sorted by Year | 4.9 million | 3,940 | 0.52 | 0.94 |
| Skiptune | Two-Notes | Sorted by Year | 4.9 million | 6,609 | 0.53 | 0.98 |
| Skiptune | Note | Shuffled | 4.9 million | 3,940 | 0.41 | 0.99 |
| Skiptune | Two-Notes | Shuffled | 4.9 million | 6,609 | 0.45 | 0.99 |
The first two rows refer to human natural languages generally and the English language in the form of Moby Dick specifically. The rest of the rows all involve the Skiptune corpus.
The second and third columns display the representation and ordering of the corpus. The representation for human languages and Moby Dick are, of course, words. For Skiptune, the representation is either a single note (a pitch and its duration) or two-notes (a pitch differential and its duration ratio). Ordering for Skiptune corpuses are either As Entered, Shuffled, or Sorted by Year.
The final four columns are the Heaps Law data. N is the number of total words (for human languages) or tokens (for the Skiptune corpuses). V is the vocabulary size. Beta is the key metric that falls out of the Heaps Curve analyses. R-squared is how much of the variation is explained by the Heaps Curve fit to the data.
The most important column is the Beta column. Scanning down that column, we see that in every case the beta for the two-note “word” is higher and closer to human languages than the single note “word”. For As Entered, beta rises from 0.32 to 0.38 going from single note to two-note vocabularies. For Sorted by Year, beta rises from 0.52 to 0.53, a small change. And for the 1,000 random shuffles, beta rises from 0.41 to 0.45.
The R-squared statistic is high and quite explanatory no matter what the definition of a “word” we use, and therefore does not contribute to our decision as to which definition to use for AI model training.
Our logic for choosing two-note “words” over single note “words” goes something like this:
- The architecture of large language models was designed around human languages
- A two-note definition of word in melodic music produces a Heaps Law calculation that is closer to human languages than a single note definition
- Therefore, using that same AI model to train music will have better success if it uses the two-note definition
Having decided that we will use two-note “words” in our AI model training, we observe that the difference in Heaps Law calculations isn’t so great that it eliminates the single note definition of a “word” from consideration. At some point, it would be useful to test out the above logic by trying to train an AI model using a single note as a “word” and comparing the results to the two-note model.