Our workflow includes a step to make sure the tune we’re entering into the database is not already there. For that we devised a search engine that, as a by-product, tells us how many tunes already in the database have all the search terms in them. In doing so over the years, we noticed something curious: When entering a folk tune, we usually find a few or maybe a dozen tunes with all the search terms of the tune we’re entering. In those cases, the search is over in a blink of an eye, telling us whether it found a match or not.
Once in a while, however, our search engine will let us know there are, say, 4000 other tunes with these same tuples we’re searching for, and the search can take a couple minutes to complete.
These are generally folk tunes presumably made up of common tuples (two-note patterns), and the vast majority were not the written with the sophistication of professional composers, but are rather simple tunes. One would assume a lot of tunes would have the tuples of a new folk tune, especially as there are around 29,000 folk tunes in the database. The puzzle or paradox is that despite the common nature of the tuples in folk tunes, the vast majority of our searches for folk tune selections result in quick searches because other tunes don’t have all those search terms.
To make sure this paradox is clear, an example might be best. Let’s say we’ve just entered this line of a new tune:
The tune is entered via LilyPond and our workflow has us copying the first line of the LilyPond file for insertion into the search engine. The number of notes copies varies because LilyPond notation varies depending on accidentals, but in this case we copied the first 17 notes, namely the dotted eighth note A to start the tune all the way through the first G sixteenth note in the last measure.
When we paste that into the search engine, the algorithm first converts the LilyPond code into two-note tuples and prints those out thusly:
[2, 0.333], [2, 4.0], [-4, 0.75], [-5, 0.333], [5, 4.0], [0, 0.75], [2, 0.333], [2, 4.0], [-4, 0.75], [-3, 0.333], [3, 3.0], [0, 0.333], [0, 2.0], [-1, 1.0], [-2, 1.5], [2, 0.333]
For efficiency, we only want the search engine to search tunes that have all the search terms in them. The logic is that if a tune doesn’t have all these tuples, it can’t possibly have this exact search term and we can therefore ignore such tunes from our search. The algorithm removes duplicate search terms (note that [2, 0.333] occurs three times) and comes up with the following 12 search tuples:
[2, 0.333], [2, 4.0], [-4, 0.75], [-5, 0.333], [5, 4.0], [0, 0.75], [-3, 0.333], [3, 3.0], [0, 0.333], [0, 2.0], [-1, 1.0], [-2, 1.5]
The algorithm then creates a set of tunes for each search term such that the set consists of only those tunes with that tuple. For this example, there are 12 such sets. The last step is to find the intersection of all 12 sets. In other words, the algorithm creates a new set that is made up only of tunes with all 12 search terms.
For this particular example, when the database already had over 83,000 tunes, here’s what the search engine printed:
0 tunes with all required search terms.
Total of 0 tune(s) with this pattern.
Now take another look at that tune. There is just nothing special about any of those notes or even pairs of them. The intervals are all small (nothing larger than a perfect fourth) and the duration ratios are all common. There are no tied notes that might create an unusual duration ratio. There aren’t even any rests that might interrupt or complicate the search. It’s a simple tune, yet there is no other tune in the database that has all these search terms in any order.
That’s the paradox.
The other end of the extreme is not so surprising, at least not in many cases. Take an Irish jig, such as this one:
Here are the de-duplicated search terms for the first 15 notes:
[5, 1.0], [0, 1.0], [-5, 1.0], [2, 1.0], [1, 1.0], [-4, 1.0], [-1, 1.0], [-7, 1.0]
And the output from the search engine:
2943 tunes with all required search terms.
Total of 1 tune(s) with this pattern.
The search engine found there were nearly 3,000 tunes that each had all these search tuples in them. Only one tune had the tuples in the correct order, but the point is that this folk tune had a lot in common with another 3,000 tunes that had the same tuples.
We can understand how this tune differs from the first one. A jig like this is mostly made up of eighth notes, one after another. The duration ratio is always or mostly always 1. Even though we had an interval as high as a perfect fifth, suggesting a broader range than the first tune, there were still thousands of tunes containing all the search terms.
Most other folk tunes are not so easily explained, and that is the subject of this inquiry. Next week we’ll discuss how to organize a proper analysis of this interesting search engine paradox.