AVCC
Dataset · Interspeech 2026

AVCCAudio-Visual Conversation Corpus

31 hours of real, unscripted conversation between two or three people. Each segment is one continuous take from a single static camera, and every word is transcribed, timed and tied to the face of the person who said it.

Test video diiJlowDHbA, segment 29. The boxes, captions, timeline and turn events are all drawn from the released annotation files.

31.0 hof conversation
29conversations: 17 train, 5 val, 7 test
408unbroken segments, 20 s to 72 min
356,588words, checked by hand
2 or 3people on screen: 217 and 191 segments
1,730hold/shift test events, 4,290 human guesses
Why AVCC

Built for turn-taking in the wild

A turn-taking model predicts who speaks next, and when. To learn that from video it needs conversations whose timing is untouched, where everyone stays in view and every voice belongs to a known face. AVCC was collected for exactly that.

One take, no edits

Each segment is a stretch of video with no cuts, jump cuts or zooms. Pauses, overlaps and the moment before someone starts to speak keep their real timing.

Up to 72 minin one unbroken take; the median segment runs 2 min

One camera, everyone in view

A single static camera shows all participants, as an outside observer would see them. A model sees the listeners as well as the speaker.

96.9%of the time, every participant's face is on screen

Speakers, not face tracks

Speakers 0, 1 and 2 keep the same label in the words, utterances, RTTM and per-frame face boxes for the whole segment, so every voice is matched to a face.

1 labelper person, across audio and video

Words and turn events

Word-level transcripts were force-aligned, then corrected by hand. The test set adds balanced hold/shift events and next-speaker guesses from 154 people.

59.8%of human guesses picked the right next speaker
The annotations

Every word, every speaker, every turn

Test video sL3jHOX826s, segment 4. Turn the sound on.

  • A face box for each person, every frameDetected with RetinaFace, matched to speakers with ArcFace, then checked by hand. The box lights up while that person is talking.
  • Words appear as they are spokenEach word has its own start time and duration, and the speaker who said it. Overlapping speech shows on two lines at once.
  • A timeline of who speaks whenSeven seconds back and three ahead. Bars are utterances: a speaker's words joined across pauses of 0.1 s or less.
  • Hold or shift at every pauseAt each pause the badge asks who speaks next, then shows the answer: the same person (hold) or someone else (shift). This is what turn-taking models predict.
  • Explore in the interactive viewerPlay both test segments, zoom the timeline down to one second and hover any word, utterance or turn event.
A hard case

Three people talking over each other

Real conversation is not tidy. Here two speakers talk at the same time for several seconds, back-channels like "yeah" and "mhm" land in the middle of other people's sentences, and the floor still changes hands. Every word keeps its own speaker and timing, so overlaps are annotated, not smoothed away.

Test video yGGsg7_xxsI, segment 10. Turn the sound on.

Real timing

A third of speaker changes start in overlap

Because nothing is cut, the timing between speakers is exactly as it happened. The next speaker often starts before the previous one has finished, and gaps are short.

Time from one speaker's end to the next speaker's start

Each speaker change between two utterances, in 0.1 s bins. Negative values are overlaps: the next speaker started early. Two- and three-person segments are shown separately, as share of all speaker changes in that group.

Show the numbers as a table
Comparison

How AVCC compares

AVA-ActiveSpeaker and MSDWild are the standard audio-visual benchmarks for active speaker detection and diarization. Both are valuable, but neither lets you study multiparty turn-taking from video, for two different reasons.

AVA-ActiveSpeakerRoth et al., ICASSP 2020 MSDWildLiu et al., Interspeech 2022 AVCCQi & Skantze, Interspeech 2026
Source15-minute clips of feature filmsPublic videos, mostly vlogsPodcasts and livestreams from YouTube and Twitch
SpeechScripted film dialogueSpontaneous, many languagesSpontaneous, English
Size38.5 h of face tracks80 h, 3,143 clips31.0 h, 408 segments from 29 conversations
Camera Edited film with frequent shot changes Edited: jump cuts and zooms inside clips One static camera, no edits
Unit of labelling Face tracks, speaking or not. Not linked to people Speakers (who spoke when), plus face boxes Speakers, with one label shared by words, RTTM and face boxes
People in viewMostly one face at a time2 to 10 speakers per clip2 or 3, all on screen 96.9% of the time
Transcripts None None Word level, hand-checked
Turn-taking None None 1,730 hold/shift events and a human baseline

Swipe the table sideways to see MSDWild and AVCC.

AVA and MSDWild sizes are from their papers. The AVA figures below are measured on its labels for the 153 train and val movies, the MSDWild speaker counts on its RTTM files, and the MSDWild frames come from its released videos.

AVA labels face tracks, not people

AVA marks each face track as speaking or not. A track ends at every shot change and is never linked to the person, so the same actor gets a new ID each time the film cuts back to them. You can tell whether a face is talking, but not who holds the turn or who takes it next.

2.5 smedian length of an AVA face track
240separate IDs in a 15-minute clip (median)
69%of labelled frames show one face
On screenSpeaking

MSDWild clips are edited

MSDWild is spontaneous conversation, but its clips come from edited videos. Jump cuts skip time and zooms re-frame the scene inside a single clip, so a pause on screen is not the pause that happened. Each clip below is four seconds around one edit; the badge appears on the frame where it happens.

Clip 00178: jump cut. Both people change pose from one frame to the next.
Clip 00543: zoom and jump cut. The shot zooms out and time jumps at the same moment.
Clip 00364: zoom. The picture jumps closer, with subtitles burned in.
Download

Get AVCC

AVCC is on Hugging Face. Request access with your name, institution and what you will use it for. It is free for non-commercial research.

hf auth login
hf download Haotian-Qi/AVCC --repo-type dataset \
    --local-dir AVCC
python3 AVCC/script/unzip_videos.py --delete-zips
python3 AVCC/script/viewer.py     # browse any segment

The release includes tools to rebuild utterances and RTTM from the words, extract face crops, audio and model windows the way MuVAP does, browse any segment and correct the annotations. Formats are described on the dataset page.

SplitVideosSegmentsHoursWords
Train1725820.6230,071
Val5542.730,820
Test7967.795,697
Total2940831.0356,588

Splits are by video, so no conversation appears in two splits. Videos are 25 fps, mostly 1920×1080, with 44.1 kHz audio.

Citation

Cite AVCC

If you use AVCC, please cite the AVCC paper. AVCC was first introduced with MuVAP, a multimodal model that predicts multiparty turn-taking from one camera and one microphone; if you also use MuVAP or compare against it, please cite the MuVAP paper as well.

@misc{qi2026avcc,
  title         = {{AVCC: Audio-Visual Conversation Corpus for Turn-Taking Prediction with a Human Benchmark}},
  author        = {Haotian Qi and Gabriel Skantze},
  year          = {2026},
  eprint        = {2611.13957},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2611.13957},
}

@inproceedings{qi26_interspeech,
  title     = {{MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the wild}},
  author    = {Haotian Qi and Gabriel Skantze},
  year      = {2026},
  booktitle = {{Interspeech 2026 [Long Track]}},
  pages     = {5952--5961},
  doi       = {10.21437/Interspeech.2026-1381},
  issn      = {2958-1796},
}