This week we continue our examination of why some tunes have search patterns that contain terms for which no other tune contains the same set of terms in any order, and why other tunes have search terms found in thousands of other tunes. We’ll expand upon our 10-term tuples to see how using other search term lengths affect our tentative conclusion.
What Changes with Length
We re-run the same algorithm we used for the 10-tuple search length using the following lengths: 4, 6, 8, 12, 14, 16, 18, and 20. Here are the results with 10-tuples included for comparison:
|
Length |
Zero results |
Bottleneck |
Combinatorial |
Mixed |
|
4 |
100 |
97.0% |
1.0% |
2.0% |
|
6 |
479 |
89.4% |
6.7% |
4.0% |
|
8 |
1,259 |
83.6% |
9.3% |
7.1% |
|
10 |
2,396 |
74.9% |
15.9% |
9.1% |
|
12 |
3,632 |
65.9% |
22.2% |
11.9% |
|
14 |
4,824 |
58.9% |
28.2% |
13.0% |
|
16 |
5,969 |
53.7% |
32.4% |
13.9% |
|
18 |
6,894 |
48.7% |
37.4% |
14.0% |
|
20 |
7,750 |
44.2% |
42.4% |
13.5% |
Let’s walk through the table. The first column contains the length of the search term. The second column shows the number of search terms, for that length, that result in no other tunes having all of the search tuples in the search term. For example, at four-tuple search terms, there are only 100 tunes that are missing at least one of the search terms. The vast majority of tunes have all four search terms, though not necessarily in the same order. We see that this column increases in number as we approach our max tuple length of 20. That makes sense, as we would expect longer search terms to have more tunes that do not contain all the tuples in the search term.
The third column, labeled “Bottleneck”, refers to the rarest-tuple bottleneck, which means that the complete intersection set has no matches, but removing a tuple tied for the lowest frequency restores at least one match. One individually unusual pitch differential and duration ratio is therefore pivotal. Note that this column drops as we move to longer and longer search terms. That also makes sense because the longer the search term, the less likely it is that just one tuple causes a lack of intersection with other tunes.
The fourth column is labeled “Combinatorial,” and refers to robustly combinatorial, which means the zero intersecting set remains zero no matter which tuple you remove. No individual tuple explains the lack of an intersection in other tunes. The selection is unusual because of the collective combination of its tuples, not because of any one tuple. As one would expect, this column of numbers increases as we increase the length of the search term.
The final column is “Mixed,” meaning that removal of the rarest tuple does not restore a match, but removing some other tuple does. This suggests an incompatibility involving a more common tuple and the rest of the set, rather than a simple rarest-tuple bottleneck. “Mixed” is what’s left over if we can’t explain the intersection set with “bottleneck” or “combinatorial” reasons. It also generally increases as the search term lengthens, but notice that at the 20-tuple length the percentage decreases a little.
Resolving the Search Engine Paradox
There is a clear story here. At short lengths, a zero result is overwhelmingly caused by one exceptionally rare tuple. At increasing lengths, the importance of that single bottleneck steadily declines, while combinatorial incompatibility steadily increases. By length 20 they’re nearly equal: 44.2% versus 42.4%.
Mixed behaves differently because it occupies the boundary between those two mechanisms. It rises while the population is transitioning from bottleneck-dominated to combinatorial-dominated zeroes, increasing from 2 percent to 14 percent before slipping to 13.5 percent.
The drop in “mixed” suggests it is an intermediate mechanism rather than a third independent mechanism. As search terms become longer, initially almost all failures are bottlenecks. Adding tuples makes interactions among otherwise individually supportable tuples increasingly capable of destroying the intersection. During the transition, you get more cases exhibiting characteristics of both mechanisms; hence, “Mixed” rises.
But eventually sufficiently long passages become so combinatorially restrictive that the classification starts resolving toward pure combinatorial failure. At that point “Mixed” should plateau and ultimately decline. The drop in in the final “Mixed” percentage is evidence that this turnaround is already be beginning. We even see evidence as early as length 14 where the rise in the “Mixed” percentage is starting to slow. .
Another important point is that the denominator is getting quite large. At length 20 there are 7,750 zero-result tunes. So 13.5 percent represents roughly 1,046 mixed cases. At length 18, 14.0 percent of 6,894 is about 965 cases. Thus the number of “Mixed” cases actually increased from 965 to 1,046. What decreased is their share of zeroes, because the combinatorial category grew even faster.
That’s an important distinction. There is no evidence here that mixed failures are becoming less common in absolute terms. Rather, among the rapidly growing population of zero-result searches, new zeroes at length 20 are disproportionately combinatorial. We can estimate that directly.
For length 18:
- Bottleneck ≈ 3,357
- Combinatorial ≈ 2,578
- Mixed ≈ 965
For length 20:
- Bottleneck ≈ 3,426
- Combinatorial ≈ 3,286
- Mixed ≈ 1,046
So the additional ~856 zero-result cases between 18 and 20 break approximately into:
- +69 bottleneck
- +708 combinatorial
- +81 mixed
Allowing for rounding, roughly 83 percent of the newly appearing zeroes are combinatorial. That is much more informative than the 0.5-point decline in Mixed because it suggests that we’re approaching a mechanism crossover at length 20:
- Bottleneck 44.2%
- Combinatorial 42.4%
Extrapolating the trend, we would expect combinatorial to overtake bottleneck around length 21–22. Since we’re measuring even lengths, we’d predict that at length 22, combinatorial will probably be the largest category.
And if the mechanism interpretation is correct, we would make another testable prediction:
Mixed should now begin declining gradually rather than resume its earlier growth.
This is potentially an interesting result about the music corpus itself. At short search lengths, uniqueness is mostly lexical, in the sense that a melody contains an unusual individual interval-duration relationship. As passages become longer, uniqueness becomes increasingly syntactic/combinatorial: None of the individual musical gestures has to be particularly unusual; what becomes unique is their particular conjunction.
Cliffhanger
That’s as far as our analysis has taken us, and unfortunately, it is unsatisfying:
We accidentally landed on a cliffhanger: What happens if we extend the search term length beyond 20? Will “Mixed” begin declining gradually rather than resume its earlier growth? Are we at an inflection point, or is there a significant turn of events at length 20? This is the risk of doing a live blog on real-time analysis. We know the results when you do. We will finish the Search Engine Paradox next week (for sure, this time).