Portal de Periódicos da CAPES

Video Scene Detection Using Transformer Encoding Linker Network (TELNet)

2023; Multidisciplinary Digital Publishing Institute; Volume: 23; Issue: 16 Linguagem: Inglês

10.3390/s23167050

1424-8220

Shu‐Ming Tseng, Zhi-Ting Yeh, Chia-Yang Wu, J. Chang, Mehdi Norouzi,

Advanced Vision and Imaging

This paper introduces a transformer encoding linker network (TELNet) for automatically identifying scene boundaries in videos without prior knowledge of their structure. Videos consist of sequences of semantically related shots or chapters, and recognizing scene boundaries is crucial for various video processing tasks, including video summarization. TELNet utilizes a rolling window to scan through video shots, encoding their features extracted from a fine-tuned 3D CNN model (transformer encoder). By establishing links between video shots based on these encoded features (linker), TELNet efficiently identifies scene boundaries where consecutive shots lack links. TELNet was trained on multiple video scene detection datasets and demonstrated results comparable to other state-of-the-art models in standard settings. Notably, in cross-dataset evaluations, TELNet demonstrated significantly improved results (F-score). Furthermore, TELNet's computational complexity grows linearly with the number of shots, making it highly efficient in processing long videos.