RGB-IR Paired Dataset with Terrestrial-view for Maritime Object Detection

Abstract

Research into unmanned vehicles is expanding across diverse fields, with a particular focus on national defense, necessitating ongoing development efforts. Among the research, there is research about unmanned robots designed for monitoring regions that cannot be observed by conventional maritime and ground surveillance equipment. For the effective operation of unmanned robots in amphibious environments, researching algorithms mounted to the robot's platform is essential. Deep learning stands out as a main area for developing these algorithms. To applicate deep learning in practical scenarios, it is necessary to use datasets that contain the information of the real-world environment in which the models will be deployed. As a result, we generated a dataset for the inaugural application of an algorithm deployed on amphibious surveillance robots. Subsequently, we conducted a baseline experiment employing transformer-based detection algorithms with the MTRIP dataset. In Chapter 1, we introduced the purpose of this study and related studies. In Chapter 2, we introduced MTRIP (Maritime Terrestrial-Viewed RGB-IR Paired) dataset, the first RGB IR paired dataset acquired from a terrestrial view in the maritime environment. This dataset is used to train algorithms for day and night detection on unmanned vehicles that can maneuver in an amphibious environment. The contribution of the dataset is that proposed dataset is applicable while existing datasets are difficult to apply to unmanned vehicles that can maneuver in an amphibious environment. In this case, it is also advantageous that by using the paired RGB-IR images, so it can be applied to various tasks. In Chapter 3, we introduced the baseline performance for the MTRIP dataset. We studied the effective information fusion method for transformer-based RGB-IR multimodal detection model. To develop a detection model that is robust to illumination, we used information from IR (Infra-Red) images that are robust to illumination and information from RGB images. In conclusion, the expectation of this study is to improve the performance of object detection algorithms for unmanned maritime vehicles.