@audio/tune-curve

Retune a voice along a pitch curve, its own glottal cycles re-spaced (TD-PSOLA): formants and consonants kept (Melodyne class)

import tuneCurve from '@audio/tune-curve'

tuneCurve(voice, { semitones: 3, fs: 44100 })                               // a constant shift
tuneCurve(voice, { semitones: { t: [0, 1, 1.5], v: [0, 2, 0] }, fs })      // a drawn curve: seconds → semitones
tuneCurve(voice, { semitones: t => 0.5 * Math.sin(2 * Math.PI * 5 * t), fs }) // vibrato
tuneCurve(voice, { semitones: 2, f0: { t: times, v: hz }, fs })            // cycles found on a pitch track given

let write = tuneCurve({ semitones: curve, fs })   // streaming: write(chunk) → output `write.latency` samples behind
let out = write(chunk), tail = write()

A cycle at a time is cut around its mark by a two-lobed window and laid at the spacing the curve asks for (Moulines & Charpentier, Speech Communication 9, 1990), each lobe the source's period that side and no longer than the spacing, as Praat's PSOLA has it: two lobes meet across each gap and sum to one, so a raised voice keeps its level (lobes of a whole period overlapped three at a time, and dividing by their sum lost 1 to 5 dB from +5 semitones). The pulse shape, the vocal tract's ringing on it, the breath and the jitter stay the voice's own, so the formants stay where they were and nothing is resynthesized. Unvoiced audio (consonants, breath, silence) passes as it is. Where the ratio is 1 and the cycles are laid where they were, the output is the input.

Re-spaced, the cycles run ahead of the input by ∫(ratio − 1), and when the ratio comes back to 1 a part of a period is left over: what follows (the input, a crossfade back into it) would meet them that far off, half a period at worst, where the two cancel. Over the 100 ms before the ratio returns to 1 they drift back onto their marks, on them from 10 ms before it (about the time a voice takes to move its pitch a few semitones at its fastest: Xu & Sun, JASA 111, 2002), and from there the input comes back sample for sample. On eight recordings (VoiceBank, Spoken Wikipedia), a curve 0.05 to 5 semitones up or 3 down and back, its ends mid-vowel: over the 10 ms after it returns to 0, the difference from the input stood 2.2 dB above the input's own level (median of 32; at worst 5.2), and now none in any of them.

The cycles are found as the audio streams: f0 by pYIN (Mauch & Dixon, ICASSP 2014; @audio/pitch-pyin) on the channels' mean, a frame every 5.8 ms, its Viterbi path decided within 0.15 s of the last frame read (within 0.1 s a lecture's frames already took every octave offline Viterbi did); a mark per cycle, the first at the loudest sample of a voiced run's first period and each next a period on, moved to where a period of waveform matches the one before best (normalized correlation, refined between samples); from a run's first mark, back up to 40 ms a cycle at a time while each matches the one after with a correlation over .7 (Praat's test for a pulse past a voiced interval's edge), never into a voiced mark: pYIN finds a voice once its frame is mostly voiced, and the first 20 ms or so of each run stayed at its pitch while the rest moved (6.5% of the frames Praat finds voiced on eight recordings, now 0.9%). YIN, at the threshold that keeps running speech voiced (.35: its own .15 holds only a voice that does not glide within the frame), took a strong second harmonic's period for the voice's (/i/, /u/, /ʊ/ with their first formant on it, in every frame): the pulse and the ripple between pulses were cut as two cycles, and each cycle skipped or repeated put two of a kind side by side, the voice's phase jumping there. A lecture lowered a semitone had 2.4 such seams a voiced second, and now none; over eight solo singers of Vocadito (158 s), pYIN's path (offline) in place of YIN took the 30-ms frames whose periodicity (1 less YIN's least normalized difference) fell by more than 0.2 from over 0.8 from 103 to 36 at +2 semitones, 183 to 48 at −2 and 278 to 76 at +5. A lowered pitch lays its cycles further apart than they are long and leaves the gaps between them: lobes widened to fill them took in the cycles either side, and an octave down, two periods apart, gave the input's own period back. Cycles are read at fractional places through a Kaiser-windowed sinc, so the output is periodic at the new pitch, not jittered to whole samples.

Channels are retuned on one set of cycles, found on their mean, so a stereo voice keeps its image. Streaming output equals the batch output sample for sample, however the input is chunked: how far a run is followed back is its own audio's to say (bounded by how far the cycles had been laid instead, a sung take chunked otherwise came out up to .76 of full scale apart). The stream runs write.latency samples behind its input: 0.36 s at 44.1 kHz for a voice down to 60 Hz lowered a semitone, most of it the 0.15 s pYIN's decisions wait and the 40 ms a run is looked for back.

Measured

Against a synthetic voice re-synthesized at the target pitch (Rosenberg pulses through Peterson & Barney's /a/ formants, the same at both pitches), from −7 to +12 semitones at 120 and 220 Hz:

On a sung recording (audio-lena) and a speech sample, the long-term spectrum over voiced frames stays within ±0.5 to ±1.7 dB of the input from −4 to +7 semitones (the vocoder: ±3.7 to ±12.6 dB), the pitch within a median 0.7 to 2.2 cents of the input's times the ratio.

Use when: one voice, speech or singing: pitch correction, intonation, drawn pitch curves.
Not for: chords, mixes, drums (no single cycle to re-space: they pass unshifted where unvoiced), or a pure tone (its "formant" is the tone itself, which a kept envelope holds in place).

Part of @audio/tune, the tune family umbrella.

MIT © audiojs