fixie-ai/ultravox-v0_7-glm-4_6

Model Card for Ultravox

Ultravox is a multimodal Speech LLM built around a pretrained LLM (GLM 4.6 in this case) and a speech encoder (whisper-large-v3-turbo) backbone.

See https://ultravox.ai for the GitHub repo and more information.

Model Details

Model Description

Ultravox is a multimodal model that can consume both speech and text as input (e.g., a text system prompt and voice user message). The input to the model is given as a text prompt with a special <|audio|> pseudo-token, and the model processor will replace this magic token with embeddings derived from the input audio. Using the merged embeddings as input, the model will then generate output text as usual.

In the v0.7 series, the Ultravox model is trained on GLM 4.6, taking the lead on audio reasoning tasks over closed source models like gpt4o-audio, while retaining advantages in speech understanding from previous versions.

• Developed by: Ultravox.ai
• License: MIT

Model Sources

• Repository: https://ultravox.ai
• Demo: See repo

Usage

Coming soon.

Training Details

The model uses a pre-trained LLM (GLM, Llama, Gemma, Qwen, etc) backbone as well as the encoder part of whisper-large-v3-turbo.

The multi-modal adapter is trained, the Whisper encoder is fine-tuned, and the LLM is kept frozen.

We use a knowledge-distillation loss where Ultravox is trying to match the logits of the text-based LLM backbone.

Training Data

The training dataset is a mix of ASR datasets, extended with continuations generated by Llama 3.1 8B, speech translation datasets, and noise datasets.

Training Procedure

Supervised speech instruction finetuning via knowledge-distillation. For more info, see training code in Ultravox repo.

Training Hyperparameters

• Training regime: BF16 mixed precision training
• Hardware used: 8x B200 GPUs

Evaluation

Evaluations are conducted big bench audio (audio reasoning measured in accuracy), VoiceBench (overall score averaged across multiple evaluations), as well as on covost2 (speech translation measured in BLEU), and LibriSpeech (speech recognition measured in WER).

Audio Reasoning & General Understanding

	v0_7-glm-4_6 w/ reasoning	v0_7-glm-4_6 w/o reasoning	v0_6-llama-3_3-70b	v0_6-gemma-3-27b	v0_6-qwen-3-32b	gpt4o-audio
big bench audio	97.00	91.80	85.48	83.84	84.22	82.80
voicebench overall	90.75	87.05	81.81	–	–	86.75

Speech Translation & Recognition

	v0_7-glm-4_6	v0_6-llama-3_3-70b	v0_6-gemma-3-27b	v0_6-qwen-3-32b
covost2 en_ar	22.89	18.92	22.68	16.91
covost2 en_ca	41.48	38.73	39.67	33.63
covost2 en_de	35.44	33.69	34.76	31.09
covost2 es_en	42.88	41.39	41.11	41.20
covost2 ru_en	50.30	43.73	49.29	47.08
covost2 zh_en	23.85	17.81	20.88	22.24
librispeech	2.28	2.55	2.73	2.88