Real-Time Lightweight Semantic Segmentation using Fourier Neural Operator Network

Abstract

Semantic segmentation of image data is necessary for understanding images without presence of human. However, deploying AI models to mobile device has difficulty of computational resource and memory constraints issues. To perform semantic segmentation in real-time environment, model should have small parameters and low inference latency. To achieve the goal, we propose a framework of one-stage semantic segmentation model named Dense prediction Fourier Neural Operator(DFNO) that has small parameters and low inference latency using Fourier Neural Operator Network.

In the segmentation encoder framework, we propose a fourier layer instead of self-attention layer for better performance and lower latency. Also, we are the first work to use a fourier structure in segmentation decoder. Our Fourier Decoder showed lower inference latency with better accuracy compared to previous lightweight decoder structures.

We conduct extensive empirical evaluations on 3 datasets including Pascal Context, COCO Stuff10k and ADE20K, which are the commonly used benchmark in the literature. Our empirical studies show that our proposed method shows comparable results compared to prior arts.