Before we can finish up our discussion of rests, notably their duration ratios, we need to introduce and explain entropy, as it’s a metric we will use when deciding how (or not) to use rests in our AI training.
“Entropy” puts off a lot of people, but it turns out to be highly useful in not only physics (especially in the second law of thermodynamics), but also in music. And while the meaning of entropy in physics is not exactly the same as it is in music, they are identical mathematically. We won’t get into entropy in physics except to say that entropy in physics measures how much order there is in something. That’s also true of entropy in music.
Remember that music is a form of communication. In communication theory researchers studied the ability to learn of the next part of an incoming message given the section of the message that had already arrived. We don’t know for sure, but given the way English (or any human language) works, we can eliminate some possibilities and give other possibilities greater weight. If the message so far is “The best way t”, we can expect the letter “o” to have a high probability of being the next letter.
Mathematicians found a way of measuring this uncertainty. I’m going to show the equation here in full, but then I’ll say it again in simple English. Don’t worry. If this looks like gibberish to you, you’ll still be able to understand it in the next paragraph.
In simple English this equation says that entropy (H) is equal to the sum of the probability of the next letter multiplied by the logarithm of that same probability. It’s best to give an example using our own database notation. Let’s say our database only had four tokens:
| Token | Frequency |
|---|---|
| A | 50% |
| B | 25% |
| C | 15% |
| D | 10% |
This table says that the A token appears half the time, that B appears a quarter of the time, and so on. We calculate the entropy of our tiny example database using the above formula:
The probabilities are
0.50, 0.25, 0.15, 0.10
The entropy is
which becomes
Notice that the scary equation is really all just multiplications and additions. The result is
We measure entropy H in bits because that will become useful later when we train our AI model.
That number by itself is meaningless to us because we have nothing to compare it to. Let’s say that a different corpus had this set of tokens:
| Token | Frequency |
|---|---|
| A | 25% |
| B | 25% |
| C | 25% |
| D | 25% |
This is the same table as before except that all pairs of notes appear in the database an equal number of times. Each pair constitutes a quarter of the database. Sparing you the math, this database has an entropy of
Note that 2 bits is more than 1.74 bits, so that the entropy for the first database is lower than the entropy of the second database with all equal frequencies. Higher entropy means the prediction for the next pair of notes is more uncertain in the second database. You should be able to see that intuitively. In the first database, we would expect the first token A to appear more often because half the database is made up of them. In the second example with all equal probabilities, it’s a coin flip as to which is next. The lower the entropy, the less random something is. The higher the entropy, the more random something is.
Randomness is bad for predictive models because true randomness means we have no information on which to base the next letter (in language) or note (in music). We therefore need to make sure our database is ordered rather than random.
Suppose one token dominates. Take this example database:
| Token | Frequency |
|---|---|
| A | 98% |
| B | 0.01% |
| C | 0.01% |
| D | 0.005% |
In this example, the A token appears 98 times out of 100. That makes it a very ordered database. Its entropy is
An entropy of 0.2 is not much higher than zero, and quite lower than either of the other examples. We can say that a database with this extremely low level of entropy is highly ordered (almost all the notes are the same note), and that it is highly predictive, but also that it’s boring. Music produced from this database would be mostly the same note!
The value of the entropy measurement is that it tells us where on a scale any particular database of music falls: Highly ordered but boring, moderately ordered and interesting, or highly disordered and random. We, of course, want our music to be moderately ordered and interesting.
Entropy will become important when we get around to training our AI model because such models are all about “what comes next”. They are prediction machines. With each iteration, the model tries to predict the thing that comes next based not only on how well the model was trained, but on how ordered and rich the database that it was trained on.
The entropy for “duration ratios” in the Skiptune database is calculated the same way: Adding up all the multiplications of the frequency of each duration ratio by their logarithms. There are over 400 such duration ratios. When we perform that calculation we get
A duration ratio entropy of 2.35 bits means that if you knew nothing else about the next duration ratio, roughly 2 to 3 bits of information would describe it. For comparison purposes, in the English language, it is 4 to 5 bits; for English words it is 10+ bits.
We conclude that rhythm in the database is, overall, highly predictable and well ordered. Here’s how the most frequent duration ratios in the Skiptune database look if presented in table form for the first 5 most common duration ratios:
|
Skiptune Duration Ratio |
Probability |
|
1 |
59% |
|
2 |
11% |
|
1/2 |
9% |
|
1/3 |
6% |
|
3 |
4% |
The sum of those probabilities is around 89 percent. In other words, almost 9 out of 10 duration ratios that appear in tunes are one of those listed in the above table. The duration ratio “1” is almost 60 percent itself and dominates all others. That suggest composers find it pleasing to keep a constant rhythm for the most part, as a duration ratio of 1 means the next note lasts the same duration as the note before it. Several of those in a row and you have a “music with a beat”.
That concludes the section on entropy. Next week we put all this together and come to a decision on how, or even whether, to handle rests in the database.