We have our 25 metrics for the cluster analysis developed last week. The rest (18 metrics) have been archived for now, though we expect to return to them later for purposes other than cluster analysis.
Three Prep Steps
We have three tasks to perform with the 25 metrics before we proceed to the cluster analysis itself:
- Scale. Metric 35, for instance, is measured on a completely different numerical scale from Metric 15 or Metric 41. Because cluster analysis uses distance, putting every metric on the same scale is essential we don’t distort the analysis by whatever scale each metric happens to be in. The first step is standardizing all the metrics so they have the same scale.
- Distribution Shape. The diagnostics showed severe skew for metric 7, 9, 13, 21, 22, and 41, while metric 5 has 26 percent zeros and metric 9 has 44 percent zeros. Metrics 21 and 22 are also missing for roughly 36 percent of the tunes because they only exist when a certain transition to or from a rest occurs, and they have extreme positive skew.
- Conceptual Weighting. The 25 metrics are distributed unevenly among the musical families, so equal weighting of the 25 metrics would result in unequal weighting of musical ideas. Rhythm and duration alone would contribute seven dimensions of distance in the cluster analysis, while rests would contribute one.
With respect to the third bullet, we would use the following weights for the seven metric families:
| Family | Metrics | Initial weight per metric |
| Pattern vocabulary / rarity | 6 | 0.167 |
| Pitch movement / contour | 5 | 0.200 |
| Rhythm / duration | 7 | 0.143 |
| Tonality | 3 | 0.333 |
| Pitch vocabulary / range | 2 | 0.500 |
| Repeated pitch | 1 | 1.000 |
| Rests | 1 | 1.000 |
However, this set of weights gives each conceptual family exactly the same total weight. But doing so makes the singleton families (repeated pitch and rests) each as influential as the entire seven-metric rhythm family. Instead, we’ll grow the total influence of a family with the square root of the number of metrics, not directly with the number of metrics. Within a family, each metric starts with a weight proportional to 1/√n where n is the number of metrics in each family. We then normalize to sum to 1.00.
When we do that, we end up with family totals of about 18.9 percent for pattern vocabulary, 18.7 percent for rhythm, 18.5 percent for pitch contour, 15.7 percent for tonality, 10 percent for pitch vocabulary and range, and 9.1 percent each for rests and repeated pitch. Those sum to 100 percent.
While this is a bit overkill, we’re including here our weighting table for each of the 25 metrics, largely for completeness:
| # | Metric | Transformation before scaling | Family base factor | Redundancy factor | Final clustering weight | |
|---|---|---|---|---|---|---|
| 1 | Patterns Within Tune % Notes | Identity | 0.4082 | 0.5472 | 0.02031 | |
| 3 | Single-occurrence Patterns % Tune Patterns | Identity | 0.4082 | 1.0000 | 0.03712 | |
| 5 | Rests per Note | log1p(x) | 1.0000 | 1.0000 | 0.09092 | |
| 7 | Relative Pitch Change Average | log1p(x) | 0.4472 | 1.0000 | 0.04066 | |
| 8 | Use of Common Patterns | Identity | 0.4082 | 0.5497 | 0.02041 | |
| 9 | Use of Rare Patterns | log1p(x); preserve zero | 0.4082 | 1.0000 | 0.03712 | |
| 10 | Range of Pitches | log1p(x) | 0.7071 | 1.0000 | 0.06429 | |
| 13 | Average Run Length | log1p(x) | 0.4472 | 1.0000 | 0.04066 | |
| 15 | Repetitive Note Durations | Identity | 0.3780 | 0.3655 | 0.01256 | |
| 16 | Repetitive Note Pitches | Identity | 1.0000 | 1.0000 | 0.09092 | |
| 17 | Pattern Frequency Spread, Weighted | Identity | 0.4082 | 1.0000 | 0.03712 | |
| 21 | Duration Ratio Going to Rests | log1p(x) | 0.3780 | 1.0000 | 0.03437 | |
| 22 | Duration Ratio Coming from Rests | log1p(x) | 0.3780 | 1.0000 | 0.03437 | |
| 27 | Normalized Different Pitches | Identity | 0.7071 | 0.5508 | 0.03541 | |
| 28 | Normalized Pitch Differentials | Identity | 0.4472 | 0.5508 | 0.02239 | |
| 30 | Normalized Duration Ratios | Identity | 0.3780 | 0.5472 | 0.01880 | |
| 33 | 3-Note Palindromes % | log1p(x) | 0.4472 | 1.0000 | 0.04066 | |
| 34 | Tunes with Same Individual Patterns % | Identity | 0.4082 | 1.0000 | 0.03712 | |
| 35 | Absolute Average Duration Change | Identity | 0.3780 | 0.5217 | 0.01793 | |
| 37 | Average Duration Run Length | Identity | 0.3780 | 1.0000 | 0.03437 | |
| 39 | Ascending Pitch Changes % | Identity | 0.4472 | 1.0000 | 0.04066 | |
| 40 | Increasing Duration Changes % | Identity | 0.3780 | 1.0000 | 0.03437 | |
| 41 | Diatonic Pitch Percentage | -log1p(100-x) | 0.5774 | 1.0000 | 0.05249 | |
| 42 | Tonic Emphasis | Identity | 0.5774 | 1.0000 | 0.05249 | |
| 43 | Pitch-Class Entropy | Identity | 0.5774 | 1.0000 | 0.05249 |
We put the musical measurements on comparable scales. Each tune receives a score showing how far above or below the typical tune it falls. We use a robust method so a few unusual tunes don’t distort the scale for everyone else. We call this “Robust z” and it applies to each metric the same way except for Metrics 21 and 22, for which we use it on just the observed values (not the zero values).
How Many Clusters?
We then performed an experiment to determine k, the number of clusters we expect to see. After re-analyzing each metric’s features, we determine that two clusters do not appear robust. We also learn that we’re not going to get “clean” clusters, by which we mean clusters with space around each one before running into the next cluster. That’s because music is continuous, not discrete.
The result of that experiment favors a k equals 3; that is, 3 clusters for the Skiptune database. Aside from the statistical tests, which also favor 3 clusters, the family balance that results makes sense:
Cluster 1: Narrow, repetitive melodies, about 22 percent. These are tunes comprised of a relatively restricted pitch vocabulary with considerable immediate note reptition and less melodic dispersion. A reasonable cluster name would be “compact/repetitive melody”.
- Very high repetitive-pitch use (Metric 16)
- Very low pitch-class entropy (Metric 43)
- Small average pitch movement (Metric 7)
- Relatively narrow pitch range (Metric 10)
- Slightly increasing duration movement (Metric 40)
Cluster 2: High-vocabulary, highly differentiated tunes, about 33 percent. This is essentially the opposite of the first cluster. Its tunes explore a larger pitch, interval, rhythmic, and pattern vocabulary. A reasonable cluster name would be “vocabulary-rich/differentiated”.
- Many normalized different pitches (Metric 27)
- Many different duration ratios (Metric 30)
- Many different pitch intervals (Metric 28)
- Many distinct patterns relative to tune size (Metric 1)
- relatively low pattern-frequency spread (Metric 17)
- Relatively few 3-note palindromes (Metric 33)
Cluster 3: Broad-range but structurally conventional tunes, about 44 percent. These tunes cover substantial melodic rane without constantly introducing new intervals, duration relationships, and patterns. Their vocabulray is less densely differentiated than is Cluster 2 and more conventional or repetitive in construction, but not so much that they woulc be confused with the note-repetitive tunes of Cluster 1. A reasonable cluster name would be “broad-range/conventional”.
- Wider pitch range (Metric 10)
- More repeated note durations (Metric 15)
- Fewer patterns per available transition (Metric 1)
- Fewer single-occurence patterns (Metric 3)
- Fewer distinct duration ratios (Metric 30)
- Greater use of common patterns (Metric 8)
Why Not Four Clusters?
When we look at whether the database is better analyzed as four clusters, we don’t get new clusters with different metrics. Instead, it leaves two of the three clusters largely intact and splits the vocabulary-rich cluster (#2 above) into two clusters. Furthermore, 88 percent of Cluster 1 stays together and 90 percent of Cluster 3 stays together. Cluster 2 splits approximately 43 percent and 56 percent into two new groups.
Both the two new clusters make musical sense. One becomes a high-vocabulary cluster while the other has a very different signature, consisting of extremely low pattern-frequency spread (Metric 17) and high absolute duration change (Metric 35), as well as low use of common patterns (Metric 8). The fourth cluster could be characterized as “rhythmically irregular/pattern-atypical”.
However, three is still better for the number of clusters. First, every weighting system gives “k equals 3” as the higher silhouette. The silhouette measures how well each tune fits its own cluster compared with the other clusters. Second, three clusters is at least as robust as four clusters (and in fact is slightly better). Third, four clusters deteriorates noticeably in that its robustness depends too much on how we weight the metrics.
Conclusion
The Skiptune database appears to have three relatively robust large-scale regions or musical characteristics: 1) Compact, repetitive tunes, 2) vocabulary-rich and differentiated tunes, and 3) broad-range but more conventional tunes.
That concludes the selection of clusters. Next week we validate thse three proposed musical interpretations against information we deliberately did not use to form the clusters (genre, composer, decade, etc.). If those three selected clusters are genuine musical structures, the external variables should show systematic differences among them.