WordVoice: Explicit and Decoupled Multi-Dimensional
Word-Level Control for LLM-Based Text-to-Speech WordVoice: 基于大语言模型的高精度、多维度字级解耦控制语音合成

Anonymous submission

匿名投稿

Code & Dataset (Coming Soon) 代码与数据集 (即将开源)

Abstract 摘要

While recent LLM-based TTS systems have achieved remarkable naturalness, they predominantly rely on implicit end-to-end generation paradigms, resulting in coarse-grained control. To address this, we propose WordVoice, a comprehensive solution for highly precise word-level control. First, we construct WordVoice-5A, a massive 4.7k-hour bilingual dataset featuring five-dimensional word-level annotations (duration, boundary, energy, pitch, and tone). Second, we pioneer a bound-token mechanism within the LLM to enable adaptive multi-task prosodic planning and flexible manual intervention. Furthermore, we augment the token-to-waveform stage with a fine-grained acoustic modulation module. Experiments demonstrate that WordVoice achieves superior, decoupled control over multiple acoustic dimensions while maintaining competitive zero-shot synthesis stability.

尽管近期基于大语言模型（LLM）的语音合成系统在自然度上取得了显著进展，但它们主要依赖隐式的端到端生成范式，导致控制粒度较粗。为了解决这一问题，我们提出了 WordVoice，一个实现高精度字级控制的综合解决方案。首先，我们构建了 WordVoice-5A，这是一个包含 4700 小时双语数据的大规模数据集，具有五个维度的字级标注（时长、边界、能量、音高和音调）。其次，我们在 LLM 中首创了边界符（bound-token）机制，以实现自适应的多任务韵律规划和灵活的人工干预。此外，我们在声学解码阶段引入了细粒度的声学调制模块。实验表明，WordVoice 在实现多声学维度解耦控制的同时，保持了极具竞争力的零样本合成稳定性。

1. Model Architecture & Pipeline

1. 模型架构与数据处理流程

Motivation

Figure 1: Dual-Mode Synthesis Paradigm. Users can either rely on the model's autonomous prosodic planning (Free Mode) or explicitly manipulate 5D acoustic attributes for specific words (Control Mode).

图 1：双模式合成范式。 用户可以依赖模型的自主韵律规划（自由模式），也可以显式地操纵特定单词的五维声学属性（控制模式）。

Annotation Pipeline

Figure 2: Linguistically-Guided Annotation Pipeline. Extracting 5D attributes (Duration, Boundary, Energy, Pitch, Tone) with morphological modeling to mitigate coarticulation.

图 2：语言学指导的标注流程。 提取 5D 属性（时长、边界、能量、音高、音调），并结合形态学建模以消除协同发音的干扰。

WordVoice Framework

Figure 3: WordVoice Framework. (a) WordVoice-LLM via Bound-Token guided control. (b) WordVoice-FM via fine-grained style modulation.

图 3：WordVoice 框架。 (a) 基于边界符引导控制的 WordVoice-LLM。(b) 基于细粒度风格调制的 WordVoice-FM。

2. Oracle Prosody Reconstruction (GT Attribute Control)

2. 真实韵律重构 (基于 Ground Truth 属性控制)

To demonstrate the upper bound of WordVoice's control precision, we extract the 5D word-level attributes from the Ground Truth (GT) audio and feed them into WordVoice-Control. We compare this with CosyVoice3 (Free Generation) to show how explicit attributes help reconstruct the intricate prosody and emotional fluctuations of real human speech. 为了展示 WordVoice 控制精度的上限，我们从真实音频（Ground Truth）中提取 5D 字级属性，并将其输入到 WordVoice-Control 中。我们将其与 CosyVoice3（自由合成）进行对比，以展示显式属性如何帮助重构真实人类语音中复杂的韵律和情感波动。

Sample 1 (Chinese)示例 1 (中文)

📝 Text:📝 文本: "确实因为哎呦，我就现在想吐槽这个北京地铁就设计这这个人。"

Ground Truth Audio真实参考音频	CosyVoice3 (Free Gen)CosyVoice3 (自由合成)	WordVoice (with GT Attrs)WordVoice (使用真实属性)

Sample 2 (Chinese)示例 2 (中文)

📝 Text:📝 文本: "这是我当时幻想的，我希望我的未来的另一半是浓眉大眼，高鼻梁。第四个呢，第四个。"

Ground Truth Audio真实参考音频	CosyVoice3 (Free Gen)CosyVoice3 (自由合成)	WordVoice (with GT Attrs)WordVoice (使用真实属性)

Sample 3 (English)示例 3 (英文)

📝 Text:📝 文本: "Spent my twenties going like back and forth between I'm drinking, I'm not drinking, I'm drinking, I'm not drinking. But the not drinking was always."

Ground Truth Audio真实参考音频	CosyVoice3 (Free Gen)CosyVoice3 (自由合成)	WordVoice (with GT Attrs)WordVoice (使用真实属性)

Sample 4 (English)示例 4 (英文)

📝 Text:📝 文本: "Your beer doesn't taste like mushrooms, but they can help keep your mind and body in good form. So the trend that we're talking about here is adaptogenic or functional mushrooms."

Ground Truth Audio真实参考音频	CosyVoice3 (Free Gen)CosyVoice3 (自由合成)	WordVoice (with GT Attrs)WordVoice (使用真实属性)

3. Zero-Shot Quality & Stability (Free Mode vs. Baseline)

3. 零样本生成质量与稳定性 (自由模式 vs. 基线模型)

The results demonstrate that our model achieves zero-shot long-utterance stability that is fully on par with, if not superior to, the baseline, proving that explicit word-level modeling does not compromise general TTS robustness. 以下演示数据来源于 cv3-eval 数据集。我们将 WordVoice-Free 与基线 CosyVoice3 进行了对比。结果表明，我们的模型在 zero-shot 长句生成上，具备完全不输于甚至优于基线的稳定性，证明了显式字级建模不会削弱通用 TTS 的鲁棒性。

Sample 1 (Chinese Long Utterance)示例 1 (中文长句)

📝 Target Text:📝 目标文本: "板凳宽，扁担长，板凳比扁担宽，扁担比板凳长，扁担要绑在板凳上，板凳不让扁担绑在板凳上，扁担偏要板凳让扁担绑在板凳上。"

Prompt Audio提示音频	CosyVoice3 (Baseline)CosyVoice3 (基线模型)	WordVoice-Free (Ours)WordVoice-Free (我们的)

Sample 2 (Chinese Long Utterance)示例 2 (中文长句)

📝 Target Text:📝 目标文本: "吕小绿家养了红鲤鱼绿鲤鱼和驴。李小莉家养了红驴绿驴和鲤鱼。吕小绿家的红鲤鱼绿鲤鱼和驴要跟李小莉家的红驴绿驴和鲤鱼比一比谁更红谁更绿。吕小绿说他家的绿鲤鱼比李小莉家的绿驴绿，李小莉说她家的绿驴比吕小绿家的绿鲤鱼绿。"

Prompt Audio提示音频	CosyVoice3 (Baseline)CosyVoice3 (基线模型)	WordVoice-Free (Ours)WordVoice-Free (我们的)

Sample 3 (English Long Utterance)示例 3 (英文长句)

📝 Target Text:📝 目标文本: "Peter Piper picked a peck of pickled peppers. A peck of pickled peppers Peter Piper picked. If Peter Piper picked a peck of pickled peppers, where’s the peck of pickled peppers Peter Piper picked?"

Prompt Audio提示音频	CosyVoice3 (Baseline)CosyVoice3 (基线模型)	WordVoice-Free (Ours)WordVoice-Free (我们的)

Sample 4 (English Long Utterance)示例 4 (英文长句)

📝 Target Text:📝 目标文本: "Whether the weather be fine or whether the weather be not. Whether the weather be cold or whether the weather be hot. We'll weather the weather whether we like it or not."

Prompt Audio提示音频	CosyVoice3 (Baseline)CosyVoice3 (基线模型)	WordVoice-Free (Ours)WordVoice-Free (我们的)

4. Single-Word Control Scenario

4. 单字控制场景

This section demonstrates WordVoice's precise manipulation of specific acoustic attributes for a single word. Benefiting from the LLM's context-awareness, altering a target word not only executes the precise control but also prompts the model to autonomously adapt the subsequent prosody, ensuring a natural and coherent stylistic flow. 本节展示了 WordVoice 对单个字/词特定声学属性的精准操纵能力。得益于大语言模型的上下文感知能力，当我们改变目标字的属性（如重读或拉长）时，模型不仅能精准执行控制指令，还会自发地调整后文的韵律走向，使其与当前字的风格保持自然连贯。