2027 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2027) —

Melody-Guided Generation

Cristóbal Andrade 1 Sebastian J. Schlecht1

1Multimedia Communications and Signal Processing
Friedrich-Alexander-Universität Erlangen-Nürnberg
Erlangen, Germany

Abstract

This page hosts audio demonstrations for our melody-guided generation method. Placeholder text — replace with the paper abstract.

Melody-Conditioned Generation

Each melody guide is rendered in three target timbres.

Singing Voice

These examples demonstrate melody adherence. Each example shows different phonemes, which are resolved by the unconditional model, since our method provides no control over them.

Example Melody Guide Singing Voice
Ach Gott und Herr
Die Nacht
Christ, der du bist Tag und Licht

Guitar

These examples demonstrate melody adherence. The onsets sometimes sound as if played with a pick or bent, which is resolved by the unconditional model, as our method provides no control over them.

Example Melody Guide Guitar
Ach Gott und Herr
Die Nacht
Christ, der du bist Tag und Licht

Piano

These examples demonstrate melody adherence, but with degraded perceptual qualit. Decays lack piano-like characteristics and onsets are underarticulated. We attribute this to distribution mismatch, as the unconditional prior was trained on polyphonic material, making a monophonic melody line out of distribution.

Example Melody Guide Piano
Ach Gott und Herr
Die Nacht
Christ, der du bist Tag und Licht

Effect of $t_\text{init}$

Generated audio for a fixed melody guide as the initialization noise level $t_\text{init}$ is swept. $\lambda$ fixed at 0

Melody Guide $t_\text{init}$ = 0.3 $t_\text{init}$ = 0.6 $t_\text{init}$ = 0.7 $t_\text{init}$ = 0.8 $t_\text{init}$ = 1.0

Effect of $\lambda$

Generated audio for a fixed melody guide as the guidance scale $\lambda$ is swept.

Without Warm Initialization

Reverse diffusion starts from pure Gaussian noise. Increases in $\lambda$ lead to more faithful reproduction of the melody guide, but degrades quality until the point of sound out of distribution.

Melody Guide $\lambda$ = 0 $\lambda$ = 0.5 $\lambda$ = 1.0 $\lambda$ = 2.0 $\lambda$ = 3.0

With Warm Initialization

Reverse diffusion is warm-started from the target melody, with $t_\text{init}$ fixed at 1. Increases in $\lambda$ lead to more faithful reproduction of the melody guide, but degrades quality until the point of sound has a very accurate melody, but sounds robotic.

Melody Guide $\lambda$ = 0.5 $\lambda$ = 1.0 $\lambda$ = 1.5

Baseline Comparison

Side-by-side comparison against prior melody-conditioned models on the same melody guides.

Singing Voice

These results should be taken with a grain of salt.

Example Melody Guide DiffSinger [1] Diff-SVC [2] WORLD [3] Ours
Ach Gott und Herr
Christ, der du bist Tag und Licht
Die Nacht

Guitar

These results should be taken with a grain of salt.

Example Melody Guide DDSP-Guitar [4] MSD [5] RAVE [6] Ours
Ach Gott und Herr
Christ, der du bist Tag und Licht
Die Nacht

Piano

Example Melody Guide MSD [5] Piano Kontext [7] Ours – CQTDiff Ours – MR-CQTDiff
Piano

Future Work

Long Gap Inpainting

A 0.7 s gap (3.00–3.70 s) is removed from the signal and reconstructed. The blind variant inpaints with no melody guidance; the melody-guided variant conditions the fill on the $f_0$ contour of the missing region.

Clean signal Gapped input Inpainted blind Inpainted melody-guided

References

  1. J. Liu, C. Li, Y. Ren, F. Chen, and Z. Zhao, “DiffSinger: Singing voice synthesis via shallow diffusion mechanism,” in Proc. AAAI Conf. Artificial Intelligence, 2022. arXiv:2105.02446.
  2. S. Liu, Y. Cao, D. Su, and H. Meng, “DiffSVC: A diffusion probabilistic model for singing voice conversion,” in Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021, pp. 741–748.
  3. M. Morise, F. Yokomori, and K. Ozawa, “WORLD: A vocoder-based high-quality speech synthesis system for real-time applications,” IEICE Trans. Information and Systems, vol. E99-D, no. 7, pp. 1877–1884, 2016.
  4. J. Engel, L. Hantrakul, C. Gu, and A. Roberts, “DDSP: Differentiable digital signal processing,” in Proc. Int. Conf. Learning Representations (ICLR), 2020. arXiv:2001.04643.
  5. C. Hawthorne, I. Simon, A. Roberts, N. Zeghidour, J. Gardner, E. Manilow, and J. Engel, “Multi-instrument music synthesis with spectrogram diffusion,” in Proc. Int. Soc. Music Information Retrieval Conf. (ISMIR), 2022. arXiv:2206.05408.
  6. A. Caillon and P. Esling, “RAVE: A variational autoencoder for fast and high-quality neural audio synthesis,” arXiv preprint arXiv:2111.05011, 2021.
  7. D. Gavrilev, “PianoKontext: Expressive performance rendering from deadpan context,” ICML Workshop on Machine Learning for Audio, 2026. arXiv:2606.12282.