Subtitling and Translation Process Research

Within CompAsS scientific support is provided by a team from the Translation & Cognition Center (TraCo) at the Johannes Gutenberg University Mainz. This research group has as its central aim the cognitive modelling of the translation process with the help of empircal research methods such as corpora, keylogging, eytracking and EEG.

Before processes can be optimised, they first have to be analysed and understood which is the core of translation process research: understanding translation processes and analysing translator behavior with regard to efficiency and cognitive load. While translation processes have been observed in numerous studies, research of subtitling processes as a special type of translation is still in its infancy. Though the reception of subtitles has been the subject of various eyetracking studies, the process of subtitling, has yet to be investigated with established methods from translation process research such as keylogging and eyetracking. So far subtitling processes have – if at all – mainly been observed in survey or case studies.

As the academic partner in this project, TraCo supports development with data from basic and applied empirical research studies carried out both at ZDF Digital Mainz and the Johannes Gutenberg-University of Mainz. Applying innovative methods we peform both controlled basic research studies looking into the cognitive load during multimodal and multilingual text processing as well as usability studies where results directly impact user interface design and development of the Compass Subtitling platform. Studies carried out so far and currently being analyzed include:

Where Do Subtitlers look? Split-Attention in the Intralingual Subtitling Process

This usabilty study of a commercial subtitling tool in an intralingual subtitling task lays the basis for the Compass subtitling tool. In a mixed-method approach of eyetracking and keylogging, this study investigates the cognitive load and split attention of subtitlers using FAB Subtitler, a market leader in commercial subtitling tools. The tool includes common subtitle features such as spotting editor, video player with subtitle overlay, a subtitle length and reading speed monitor, as well as an audio track. Four experienced subtitlers from ZDF Digital and four student subtitlers were asked to create intralingual subtitles of three 5-minute snippets from the German ZDF documentary series "Terra X" following the Timed Text Style Guide by one of the leading video-on-demand platforms (Netflix). Recording sessions took place in participants' usual work setup, lasted about 1 hour, including recalibrations to ensure data quality, and participants completed a questionnaire regarding their use of the subtitling tool. For quality annotation all final subtitle files went through quality assurance.

The aim of this study is to find bottlenecks in the subtitling process and to analyse where participants work particularly efficient by using shortcuts, or where they lose time and produce errors due to split attention when working on too many tasks and processing audio-visual input simultaneously (listening to audio, typing, adapting and spotting subtitles, keeping track of shot changes and reading times etc.). Eyetracking can help understanding part of the process as measures such as fixation count and duration in particular areas of interest provide information on attention and cognitive load and efficiency in the process. Together with data on revisions and pauses tracked with a keylogging software and questionnaires, we test hypotheses regarding split attention similar to that of online revision compared to final revision during the translation process (cf. Hansen-Schirra et al., in press). Initial results of this study were presented at the Scandinavian Workshop for Applied Eye Tracking in June 2018 at Copenhagen Business School.

Cognitive Load in Multimodal and Multilingual Text Processing: The impact of ASR and Post-Editing in Transcription Processes based on Experience

In a second study the focus is more on basic research regarding the impact of assisting technologies on the process. As the subtitling process consists of several integrated processes, it is hard to isolate which aspects impact cognitive load. Therefore this study is limited to investigating the processes of transcription and language transfer, i.e. translation of video clips from TV series as the two main subprocesses involved in the complex subtitling process. Substantial gains in quality driven by NMT for literary texts (Toral et al. 2018) and advances in Automatic Speech Recognition (ASR) suggest that post-editing could boost productivity while maintaining quality standards also in subtitling. With established gaze and typing measures indicating cognitive effort (de Sousa et al 2011), we investigate the impact of ASR and NMT on students’ and professional subtitlers’ transcription and translation behavior and triangulate them with data from questionnaires and quality evaluations. The experiment where 13 professional translators and 13 translation students performed 8 transcription tasks of 2-minute video clips always working into German is motived by the planned pipeline of the Compass tool featuring support via ASR, NMT and translation with English as relay language. The tasks include intralingual transcription with and without ASR, translation from English into German directly from the video, with an additional English ASR script and with a correct English transcript. Finally, participants performed post-editing on German NMT output of a Swedish TV series previously transcribed and translated into English. To assist their post-editing, they first had both the Swedish video and English transcript, and then either the Swedish video or English transcript only. We expected ASR to help in intralingual and interlingual transcription but find that only a reliable transcript and the NMT brought about the expected efficiency gains. Regarding cognitive load, we observed different strategies in the two groups and found access to the images in the video to be essential.


Hansen-Schirra, S., Hoffmann, S., Schaeffer, M. & Tardel, A. (in press). Cognitive Effort and Efficiency in Translation Revision. In: E. Huertas Barros, S. Vandepitte & E. I. Fernández (Eds). Quality Assurance and Assessment Practices in Translation and Interpreting.

de Sousa, S., Aziz, W. & Specia, L. (2011) ‘Assessing the Post-Editing Effort for Automatic and Semi-Automatic Translations of DVD Subtitles’, Proceedings of the International Conference Recent Advances in Natural Language Processing 2011, (September), pp. 97–103. Available at:

Toral, A., Wieling, M., & Way, A. (2018). Post-editing Effort of a Novel with Statistical and Neural Machine Translation. Frontiers in Digital Humanities, 5, 9

CompAsS Team