Polyphony: Multi-Stream Concurrency

• Polyphony is the simultaneous occurrence of multiple streams, such as overlapping notes in music or concurrent events in audio analysis.
• It serves as a target property and structural prior in tasks like MIDI generation, sound event detection, multi-pitch estimation, and Japanese TTS.
• Operational definitions vary by domain, influencing metrics such as Polyphony Rate, BLEU scores, and error rates in SELD and SED tasks.

Polyphony denotes the coexistence of multiple simultaneous streams within a signal, sequence, or symbolic structure. In the literature considered here, the term is operationalized in several distinct but related ways: in symbolic music and MIDI generation, it refers to multiple notes active at the same time; in sound event analysis, it denotes the number of overlapping sources or events; in multi-instrument pitch estimation, it is the number of simultaneously active pitches in a frame; and in Japanese text-to-speech, it denotes the context-dependent multiplicity of kanji readings (Kundu et al., 2024, Nguyen et al., 2021, Taenzer, 31 May 2026, Liu et al., 24 Jun 2026). Across these settings, polyphony is not merely a descriptive label: it functions as a target property, a difficulty variable, a regularizer, or a structural prior.

1. Domain-specific meanings and operational definitions

The cited work does not use a single universal definition of polyphony. Instead, the meaning is task-dependent and tied to the representation under study.

Domain	Operational meaning	Unit of analysis
Symbolic music / MIDI	Multiple notes or pitches played simultaneously	Time step or score event
Sound event analysis	Number of overlapping sound events or sources	Frame or segment
MI-MPE	Number of simultaneously active pitches in a frame	Mixture frame or slot
Japanese TTS	Multiple possible readings of one kanji	Character in context

In “Emotion-Guided Image to Music Generation,” polyphony is evaluated through Polyphony Rate, intended to measure how often generated music plays multiple notes at the same time:

$$\text{Polyphony Rate} = \frac{\text{MPA}_{\text{time steps}}}{\text{time steps}}$$

where $\text{MPA}_{\text{time steps}}$ is the number of time steps during which multiple pitches are played simultaneously (Kundu et al., 2024).

In SELD and SED, polyphony is the number of overlapping sound events active at the same time. In “What Makes Sound Event Localization and Detection Difficult?,” the central conclusion is that polyphony is the main challenge in SELD because the system must both detect all active events and assign each detected event the correct direction-of-arrival (Nguyen et al., 2021). In “Polyphonic sound event detection for highly dense birdsong scenes,” polyphony is explicitly the number of overlapping bird vocalizations, with subsets synthesized at maximum polyphony 3, 6, and 10 (Parrilla et al., 2022).

In “A Lightweight Slot-Attention Framework for Multi-Instrument Multi-Pitch Estimation,” polyphony is the number of simultaneously active pitches in a frame, represented both at the mixture level and at the slot level; global polyphony is supervised with frame-wise cross-entropy over clipped polyphony classes, with counts capped at seven, while slot-level polyphony is supervised by Smooth L1 on summed pitch probability mass (Taenzer, 31 May 2026).

In “Sarashina2.2-TTS,” polyphony is linguistic rather than acoustic: kanji are polyphonic because the same character can have many readings, with the correct reading depending heavily on context. The paper states that the Joyo Kanji list contains 2,136 regular-use kanji and 4,378 recognized readings, and gives the example that 生 has 12 readings (Liu et al., 24 Jun 2026).

2. Symbolic music, MIDI generation, and structural richness

In symbolic music generation, polyphony is treated both as a representational problem and as a measurable aspect of musical richness. “Emotion-Guided Image to Music Generation” uses Polyphony Rate as one of the key music-quality metrics, alongside pitch entropy and groove consistency. The metric is computed by generating MIDI tokens, converting them to MIDI sequences, counting time steps with more than one pitch, and dividing by the total number of time steps. Reported values show large differences across architectures: the LSTM_enc2 baseline has Polyphony Rate $=0.0317$, Trans_enc3_dec3_VGG has $0.6975$, and Trans_enc3_dec3_VA_Loss has $0.7818$. The same study reports that increasing polyphony did not noticeably damage groove consistency, which remained around $0.9995$–$0.9999$ across almost all models (Kundu et al., 2024).

“Polyphonic Music Generation with Sequence Generative Adversarial Networks” addresses polyphony by converting a polyphonic MIDI file into a discrete sequence suitable for SeqGAN. The representation combines duration, the octave of the note, the pitch of the note, the octave of the chord, and the pitch of the chord into a single integer. Chords are assigned separate indices for distinct pitch sets, with about 30 pitch sets retained; tokens appearing fewer than 10 times are removed, velocity is omitted, and the final vocabulary size is 3,216. This design allows the model to generate melody and chords together rather than a monophonic melody line. Quantitatively, adversarial training improves BLEU-4, and LS-SeqGAN, unconditional, achieves 0.6852, the best reported value in the paper (Lee et al., 2017).

“Coupled Recurrent Models for Polyphonic Music Composition” proposes a stronger structural claim: polyphonic music should not be modeled as one flat sequence of notes, but as a collection of concurrent, coupled sequences, namely voices. The score representation is

$x_t \in \{0,1\}^{V \times 2P},$

with one bit indicating that pitch $p$ is on in voice $v$ at time $\text{MPA}_{\text{time steps}}$0, and a second bit indicating that the pitch begins at that time. The paper’s sequential voice factorization keeps voices in approximate lock-step and reduces computational cost: for the KernScores multi-voice corpus, sequential factorization requires roughly 5 predictions per beat, versus about 50 for raster prediction at process resolution (Thickstun et al., 2018).

“Analyzing Byte-Pair Encoding on Monophonic and Polyphonic Symbolic Music” shows that polyphony changes the merge dynamics of BPE itself. Monophonic supertokens are generally longer than polyphonic ones; on a piano corpus, it takes over 10 times more BPE steps to reach an average supertoken length comparable to that of the monophonic corpus. In a qualitative analysis using a Structured tokenization with interval pitches, only 4.2% of supertokens overlap phrase boundaries, whereas randomly splitting each piece into the same number of chunks yields 71% overlap. In the downstream phrase-segmentation task, BPE steadily improves performance in the polyphonic setting, while monophonic performance improves only within a specific range of merges (Le et al., 2024).

3. Polyphony in score-to-audio, instrument synthesis, and singing voice conversion

In score-to-audio synthesis, polyphony is presented as a primary reason music differs from speech. “Deep Performer” states that unlike speech, music often contains polyphony and long notes, so the conditioning sequence cannot be treated as a single stream of sequential symbols. The paper introduces a polyphonic mixer, which sums note embeddings that are active at the same frame according to their onsets and durations, and a note-wise positional encoding: $\text{MPA}_{\text{time steps}}$1 where $\text{MPA}_{\text{time steps}}$2 is the relative position within a note. The authors describe the system as the first to allow unaligned, polyphonic scores as inputs. In listening tests, the model is competitive with the baseline on violin and significantly outperforms it on piano in overall quality, with overall MOS 2.17 versus 1.49 on the piano dataset (Dong et al., 2022).

For acoustic guitar, polyphony is further complicated by string-dependent structure. “Expressive Acoustic Guitar Sound Synthesis with an Instrument-Specific Input Representation and Diffusion Outpainting” emphasizes that guitar polyphony includes multiple simultaneous notes across six strings, strumming order, microtiming differences between strings, open strings versus fretted notes, pick noise, and expression variability. Its proposed guitarroll is “a matrix consisting of size-6 vectors, each containing the note pitch number for each time frame,” embedded into an input of size $\text{MPA}_{\text{time steps}}$3. The paper reports that the guitarroll-based outpainting model improves transcription F1 from 0.484 to 0.531 overall, with comp F1 improving from 0.330 to 0.375 and solo F1 from 0.638 to 0.688 (Kim et al., 2024).

In singing voice conversion, the relevant phenomenon is residual vocal harmony rather than instrumental polyphony. “Poly-SVC” argues that standard SVC pipelines built around a single F0 contour are mismatched to real-world vocals because imperfect vocal separation leaves residual harmonies behind. The system replaces scalar F0 with a CQT-based pitch representation, computed over 32 Hz to 1 kHz using 12 bins per octave and 84 total bins, and regularizes it with a Random Sampler trained by

$\text{MPA}_{\text{time steps}}$4

Subjective evaluation shows the best MOS and SIM-MOS among compared systems in both single-melody and harmony settings; in harmony-rich recordings, Poly-SVC reaches MOS 3.75 and SIM-MOS 3.42 (Geng et al., 12 May 2026).

4. Overlapping events in SED, SELD, and few-shot audio

In acoustic scene analysis, polyphony is frequently the dominant source of error. “What Makes Sound Event Localization and Detection Difficult?” analyzes strong SELD systems from NTU’20 and NTU’21 and concludes that polyphony is the main challenge because systems struggle to detect all present events in polyphonic cases. On the 2021 dataset, as overlap increases from 1 to 2 to 3 sources, $\text{MPA}_{\text{time steps}}$5 drops from 0.784 to 0.763 to 0.704, $\text{MPA}_{\text{time steps}}$6 rises from $\text{MPA}_{\text{time steps}}$7 to $\text{MPA}_{\text{time steps}}$8 to $\text{MPA}_{\text{time steps}}$9, and $=0.0317$0 falls from 0.816 to 0.764 to 0.701. The paper also reports that the systems achieve lower error rates at the polyphony level dominant in the training set: single-source for 2020 and two-source for 2021 (Nguyen et al., 2021).

“Polyphonic sound event detection for highly dense birdsong scenes” extends this logic to very dense bioacoustic mixtures. Using a CRNN, it studies synthetic dawn-chorus subsets with maximum polyphony 3, 6, and 10. The O6 model performs best overall, while O10 is the most consistent across all test polyphonies. The O3 model degrades sharply as test polyphony increases, whereas the model trained with the densest samples maintains a relatively stable score even at polyphony 10. This suggests that training on denser scenes improves robustness to high overlap, although O10 can be worse than O6 on simpler scenes (Parrilla et al., 2022).

“Polyphonic Sound Event and Sound Activity Detection: A Multi-task approach” addresses dynamic polyphony by combining a class-specific SED branch with a binary SAD branch. The SAD posteriogram is repeated across classes and used to re-weight the SED output by Hadamard product: $=0.0317$1 The best joint configuration uses shared first two convolutional layers and equal loss weights $=0.0317$2. Relative to the standalone SED baseline, segment F1 improves from 35.48% to 41.03% and event F1 from 7.34% to 8.76%, while segment ER decreases from 1.54 to 0.97 (Pankajakshan et al., 2019).

For SELD with unknown or same-class overlap, “AD-YOLO” abandons fixed class-wise tracks and assigns responsibility by angular distance on a spherical grid. The model uses angular thresholds $=0.0317$3, and on DCASE 2022 same-class overlap evaluation, AD-YOLO achieves $=0.0317$4 with degradation only $=0.0317$5 at $=0.0317$6, smaller than competing methods. The paper’s central claim is that location-oriented responsibility is more robust than class-slot allocation under class-homogeneous polyphony (Kim et al., 2023).

Polyphony also affects inference-time support-set design in few-shot audio. “Who calls the shots?” defines clip-level polyphony as the number of active labels in a 1-second window and reports that monophonic support examples work best for monophonic test data, polyphonic support examples work best for highly polyphonic test data, and polyphonic support is more robust when test conditions are unknown. The paper also notes that LR is more sensitive to support-test polyphony mismatch than DFSL (Wang et al., 2021).

5. Separation, counting, pitch-density control, and plagiarism analysis

Several studies treat polyphony as a cue that should be separated, counted, or regularized rather than merely detected. “DeFT-Mamba” addresses realistic polyphonic mixtures with reverberation, moving sources, unknown source count, and in-class polyphony. Its framework first separates sound objects and then classifies each separated track, using a Dense Frequency-Time Mamba object extraction network with GCB, MHSA, and Mamba-FFN. In classification, DeFT-Mamba reports ER 31.8 and F1 60.3, outperforming CRNN at ER 74.3 and F1 20.9, AST at ER 70.6 and F1 25.8, and AudioMamba at ER 68.4 and F1 27.0 (Lee et al., 2024).

In source-aware multi-instrument multi-pitch estimation, “A Lightweight Slot-Attention Framework for Multi-Instrument MPE” uses polyphony to regularize pitch density. Mixture-level polyphony is supervised with frame-wise cross-entropy over clipped polyphony classes capped at seven, while slot-level polyphony compares the summed pitch probability mass of a matched slot to the source polyphony target using Smooth L1. The strongest stem-level result appears in Exp 3c, which combines mixture- and slot-level supervision and reaches URMP stem AP 35.43 and stem F1 42.60; the paper states that explicit polyphony supervision “recovers and even improves performance” and that Exp 3c reaches the best URMP stem AP and F1 among all stem-level experiments (Taenzer, 31 May 2026).

“SoundCount” treats polyphony as the core determinant of counting difficulty and introduces three polyphony-aware metrics: $=0.0317$7 The paper reports that MAE and MSE worsen as $=0.0317$8-polyp, $=0.0317$9-polyp, and $0.6975$0-polyp increase, and that DyDecNet, which combines dyadic decomposition with energy gain normalization, degrades less than the baselines as difficulty rises (He et al., 2023).

Polyphony also appears as a challenge in content-based comparison. “Plagiarism Detection in Polyphonic Music using Monaural Signal Separation” argues that existing plagiarism systems often ignore the issue of polyphony in recordings, simply using a monophonic approximation instead. The proposed NMF-based representation models each magnitude spectrum as

$0.6975$1

using a 1024-dimensional magnitude spectrum and 64 basis vectors. In the reported results, the baseline achieves 45.1% accuracy, NMF-only reaches 72.6%, and the enhanced system combining NMF with traditional features reaches 78.4% (De et al., 2015).

6. Formal, linguistic, and transferred uses of the concept

Some recent work treats polyphony as a formal model of concurrent structure rather than only as an observable property of music or sound. “Density Matrix RNN (DM-RNN)” replaces the classical recurrent hidden vector with a density matrix

$0.6975$2

allowing the hidden state to encode a mixed state

$0.6975$3

The paper interprets diagonal entries as populations or classical probabilities of latent musical basis states and off-diagonal entries as coherences. Polyphonic voice interaction is formalized by a tensor-product decomposition $0.6975$4 and quantified with quantum mutual information

$0.6975$5

Within this framework, polyphony is a problem of ambiguity, coupling, and inter-voice dependence rather than merely simultaneous onset density (Seo et al., 8 Jan 2026).

In Japanese TTS, the term designates reading ambiguity rather than overlapping sound. “Sarashina2.2-TTS” addresses kanji polyphony through large-scale balanced training and targeted synthesis covering all 2,136 Joyo kanji and their 4,378 readings. The augmentation pipeline yields about 280k synthetic training samples totaling about 320 hours before filtering, with 95.1% retention after quality checks. On the Joyo Kanji Yomi Benchmark, Stage 2 achieves Kana-CER$0.6975$6 and Kana-CER$0.6975$7, improving over Stage 1 and supporting the claim that targeted synthetic data helps polyphony disambiguation (Liu et al., 24 Jun 2026).

The term is also used as a system name in domains centered on concurrent interacting streams. “Prediction-Guided Control in Data Center Networks” names its controller Polyphony and studies multi-class network control for tail-latency SLOs. The system converges within 3.7, 6.0, and 9.5 minutes for low, medium, and high SLO tightness, respectively, and re-stabilizes within fifteen minutes after workload shifts (Zhao et al., 7 Jan 2026). “Polyphony: Diffusion-based Dual-Hand Action Segmentation” uses the name for a framework that predicts two frame-wise action streams, one for each hand, with alternating training and cross-hand fusion; on HA-ViD with MAS features it reaches LH accuracy 57.1 and RH accuracy 60.6, and on Breakfast it reaches 82.5% accuracy with a ViT-Base backbone (Zheng et al., 29 May 2026). This suggests an analogical extension of polyphony from simultaneous notes or events to coupled concurrent streams that must be modeled both independently and jointly.