Speechify's pitch tag swaps the speaker

A real, expressive line from Speechify’s simba-3.2, voice harper_32: “Oh my goodness, you remembered my birthday. I really wasn’t expecting this. Thank you so much!” Read it plain first:

Now the same line, same voice ID, with the SSML you’d write for it — a pause after “goodness,” emphasis on “remembered,” a slowed middle, and <prosody pitch="+15%" volume="+3dB">Thank you so much!</prosody> on the closing:

Listen to the last three words. The break, the emphasis, and the slow-down all keep harper’s voice. Then the pitch lifts on “Thank you so much,” and halfway through you’re hearing someone else. Same voice ID, same generation, different person by the end.

To isolate the cause, we cut the line to the closing — “Thank you so much!” — and moved only the pitch. Plain:

<prosody pitch="+15%">:

And +40%:

Same one-second line, different speaker. It isn’t longer, it didn’t “wander,” nothing accumulated over time. The pitch transform lands and the person changes.

This is not drift

We published a piece on Cartesia drift — a voice that decays away from itself over a two-minute take. That’s a temporal failure: the longer the generation runs, the further the autoregressive state slides from its anchor. Length causes it; a shorter take or a steadier voice fixes it.

This is a different problem. The audio is one second long and the identity is already gone. Nothing drifted. The <prosody pitch> transform replaced the speaker outright. Two different axes:

	Cartesia drift	The pitch trap
Trigger	elapsed time in one long take	a single SSML attribute
Audio length	~2 minutes	~1 second
Cause	autoregressive state decay	the pitch transform
Fix	shorter takes / stabler voice	don’t pitch a branded voice

Drift is about duration. This is about the transform you applied.

The number

We generated one fixed utterance under six conditions, changing only the SSML wrapper. Each was embedded with ECAPA-TDNN — the same speaker-verification model our drift probe uses — and compared by cosine similarity to the plain, un-tagged voice. Duration is identical across conditions, so length can’t be the variable. Only the tag changes.

SSML applied	ECAPA cosine to plain voice	Reading
(second plain take — baseline)	0.63	same voice, different generation
rate="x-slow"	0.60	same speaker
volume="+6dB"	0.69	same speaker
pitch="+15%"	0.28	halfway to a stranger
pitch="+40%"	0.02	a different speaker
pitch="-40%"	−0.04	a different speaker

Two plain takes of the same line land at 0.63 — the “same voice, different roll of the dice” baseline (short clips embed noisier than long ones, which is why we compare within the same length rather than against an absolute threshold). rate and volume sit right on it, at 0.60 and 0.69. They change delivery without changing who’s talking.

pitch doesn’t. A modest +15% drops similarity to 0.28, less than half the baseline and already an audibly different person. At ±40% the cosine falls to ~0 — in embedding space, orthogonal to the original speaker. The pitch conditions don’t degrade the voice; they replace it. And it scales cleanly: more pitch, more distance. A mechanism, not noise.

Hear the control

The controls make the case. Same line at volume="+6dB" — louder, still the same person:

And at rate="x-slow" — slower, same person:

Play those next to the pitch clips above. Volume and rate are knobs on one voice. Pitch changes the voice.

The mechanism, briefly

A pitch shift can be done two ways. The correct way for TTS is formant-preserving — move the fundamental frequency (F0) but hold the vocal-tract resonances (the formants) that carry a speaker’s identity. The cheap way is a naive resample of the whole signal, which drags the formants along with F0 and makes the speaker sound smaller or larger, younger or older: a different person.

Speechify’s behaviour is the signature of the cheap path. We tried to confirm it by tracking formants directly, but on a one-second exclamation the estimates were too noisy to trust, so we lead with the ECAPA result instead — unambiguous about the outcome even where the mechanism measurement isn’t. Either way the practical point holds: on simba-3.2, <prosody pitch> is a different-voice dial, not a louder-voice dial.

Caveats

• One voice, one model, one line. Measured on simba-3.2 / harper_32, English, “Thank you so much!”. A clean scope, not a survey.
• Short-clip embeddings are noisy — so we compare within length against a same-length baseline and controls, not an absolute cosine threshold. The pitch conditions fall far below both. That’s the load-bearing fact.
• Measured through the Speko gateway, on the audio the gateway actually returns.
• This is a Speechify result. Other providers may implement <prosody pitch> with proper formant preservation — we haven’t run the same probe on them yet. That’s the obvious next piece.

What it means for production

The takeaway is practical, and it runs against the usual SSML advice:

• Don’t use <prosody pitch> on a branded or consistent agent voice. It doesn’t nudge the pitch up a little. It changes who the customer thinks they’re talking to, on the first sentence, with no drift required.
• rate, volume, and break are safe. They control pace, level, and pauses without touching identity. That’s your toolkit for expressive control on a fixed voice.
• If you want a different-sounding voice, pitch is a fine lever — just know that’s what it’s doing.

There’s a benchmark gap here too. Leaderboards score clarity and naturalness; our drift probe scores steadiness over time. Nobody scores identity-preservation under prosody — whether a provider’s own control tags keep the speaker intact. Speechify’s pitch tag fails that test; its rate and volume tags pass. Worth measuring across the field.