Competition

Extracting subtitles in visual modality with audio annotations

To extract subtitles from video frames, a large number of keyframes should be annotated with bounding boxes and contents, which is extremely costive. However, speech transcripts are much easier to obtain, and they contain almost all the content of the subtitles. In this subtask, we present a challenge that explores learning visual subtitles with the supervision of speech scripts. We expect that the annotation from the audio modality can improve the subtitle extraction from the visual modality. In this subtask, we will present 75h of video content, divided into sets of 50, 5, and 20 as training, validation, and testing sets, respectively. For the training set, only audio annotations will be provided. The participants are required to design subtitles OCR systems with these annotations. To pretrain an OCR system, participants can also use a limited number of open datasets and fine-tune their model with audio supervision. Under these conditions, will be asked to produce subtitle text for each video in our testing set, and the submitted results will be ranked using the CER metric.

Submission

• submission should be a .zip file containing all the results in the format of xml
• results for all the test videos must be provided
• we suggest using our code of splitting video frames, ensuring that the frame id is consistent
• please find the detailed evaluation method from our website

The submitted xml files should be as the following format.

<annotation>

<video_id> test_video1.mp4 </video_id>

<subtle>

<start_id> s1 </start_id>

<end_id> e1 </end_id>

<content> content1 </content>

</subtile>

<subtile>

<start_id> s2 </start_id>

<end_id> e2 </end_id>

<content> content2 </content>

</subtile>

.....

<subtile> *** </subtile>

</annotation>