Portal de Periódicos da CAPES

Change Captioning: A New Paradigm for Multitemporal Remote Sensing Image Analysis

2022; Institute of Electrical and Electronics Engineers; Volume: 60; Linguagem: Inglês

10.1109/tgrs.2022.3195692

1558-0644

Genc Hoxha, Seloua Chouaf, Farid Melgani, Youcef Smara,

Multimodal Machine Learning Applications

Change detection (CD) is among the most important applications in remote sensing that allows identifying the changes that occurred in a given geographical area across different times. Even though CD systems have seen a lot of progress in RS, their output is either a binary map highlighting the changing area or a semantic change map that indicates the type of change for each pixel. The change maps are often difficult to interpret by end-users and they omit important information such are relationships and attributes of the changed areas. Motivated by the recent advancement of image captioning in the RS community, in this article we propose to describe the changes over bi-temporal images through change sentence descriptions. The aim of this article is to provide a user-friendly interpretation of the occurred changes. To this end, we propose two change captioning (CC) systems that take as input bi-temporal images and generate coherent sentence descriptions of the occurred changes. Convolutional neural networks (CNNs) are used to extract discriminative features from the bi-temporal images and recurrent neural networks (RNNs) or support vector machines (SVMs) are exploited to generate coherent change descriptions. Furthermore, in absence of a CC dataset to test our systems, we propose two new datasets. One is based on very high-resolution RGB images and the other one is based on multispectral RS images. The obtained experimental results show promising capabilities of the proposed systems to generate coherent change descriptions from the bi-temporal images. The datasets are available at the following link: https://disi.unitn.it/~melgani/datasets.html.