Scene classification is a highly useful task in Remote Sensing (RS) applications. Many efforts have been made to improve the accuracy of RS scene classification. Scene classification is a challenging problem, especially for large datasets with tens of thousands of images with a large number of classes and taken under different circumstances. One problem that is observed in scene classification is the fact that for a given scene, only one part of it indicates which class it belongs to, whereas the other parts are either irrelevant or they actually tend to belong to another class. To address this issue, this paper proposes a deep attention Convolutional Neural Network (CNN) for scene classification in remote sensing. CNN models use successive convolutional layers to learn feature maps from larger and larger regions (or receptive fields) of the scene. The attention mechanism computes a new feature map as a weighted average of these original feature maps. In particular, we propose a solution, named EfficientNet-B3-Attn-2, based on the pre-trained EfficientNet-B3 CNN enhanced with an attention mechanism. A dedicated branch is added to layer 262 of the network, to compute the required weights. These weights are learned automatically by training the whole CNN model end-to-end using the backpropagation algorithm. In this way, the network learns to emphasize important regions of the scene and suppress the regions that are irrelevant to the classification. We tested the proposed EfficientNet-B3-Attn-2 on six popular remote sensing datasets, namely UC Merced, KSA, OPTIMAL-31, RSSCN7, WHU-RS19, and AID datasets, showing its strong capabilities in classifying RS scenes.

/pdf/classification-of-remote-sensing-images-using-efficientnet-1avltznlhl.pdf

Classification of Remote Sensing Images Using EfficientNet-B3 CNN Model With Attention

The rapid development of deep learning brings effective solutions for remote sensing image interpretation. Training deep neural network models usually require a large number of manually labeled samples. However, there is a limitation to obtain sufficient labeled samples in remote sensing field to satisfy the data requirement. Therefore, it is of great significance to conduct the research on few-shot learning for remote sensing image interpretation. First, this article provides a bibliometric analysis of the existing works for remote sensing interpretation related to few-shot learning. Second, two categories of few-shot learning methods, i.e., the data-augmentation-based and the prior-knowledge-based, are introduced for the interpretation of remote sensing images. Then, three typical remote sensing interpretation applications are listed, including scene classification, semantic segmentation, and object detection, together with the corresponding public datasets and the evaluation criteria. Finally, the research status is summarized, and some possible research directions are provided. This article gives a reference for scholars working on few-shot learning research in the remote sensing field.

/pdf/research-progress-on-few-shot-learning-for-remote-sensing-579ow3nd5w.pdf

Research Progress on Few-Shot Learning for Remote Sensing Image Interpretation

Artificial intelligence (AI) plays a growing role in remote sensing (RS). Applications of AI, particularly machine learning algorithms, range from initial image processing to high-level data understanding and knowledge discovery. AI techniques have emerged as a powerful strategy for analyzing RS data and led to remarkable breakthroughs in all RS fields. Given this period of breathtaking evolution, this work aims to provide a comprehensive review of the recent achievements of AI algorithms and applications in RS data analysis. The review includes more than 270 research papers, covering the following major aspects of AI innovation for RS: machine learning, computational intelligence, AI explicability, data mining, natural language processing (NLP), and AI security. We conclude this review by identifying promising directions for future research.

Artificial Intelligence for Remote Sensing Data Analysis: A review of challenges and opportunities

The scene classification approaches using deep learning have been the subject of much attention for remote sensing imagery. However, most deep learning networks have been constructed with a fixed architecture for natural image processing, and they are difficult to apply directly to remote sensing images, due to the more complex geometric structural features. Thus, there is an urgent need for automatic search for the most suitable neural network architecture from the image data in scene classification, in which a powerful search mechanism is required, and the computational complexity and performance error of the searched network should be balanced for a practical choice. In this article, a framework for scene classification network architecture search based on multi-objective neural evolution (SceneNet) is proposed. In SceneNet, the network architecture coding and searching are achieved using an evolutionary algorithm, which can implement a more flexible hierarchical extraction of the remote sensing image scene information. Moreover, the computational complexity and the performance error of the searched network are balanced by employing the multi-objective optimization method, and the competitive neural architectures are obtained in a Pareto solution set. The effectiveness of SceneNet is demonstrated by experimental comparisons with several deep neural networks designed by human experts.

SceneNet: Remote sensing scene classification deep learning network using multi-objective neural evolution architecture search

The past years have witnessed great progress on remote sensing (RS) image interpretation and its wide applications. With RS images becoming more accessible than ever before, there is an increasing demand for the automatic interpretation of these images. In this context, the benchmark datasets serve as an essential prerequisites for developing and testing intelligent interpretation algorithms. After reviewing existing benchmark datasets in the research community of RS image interpretation, this article discusses the problem of how to efficiently prepare a suitable benchmark dataset for RS image interpretation. Specifically, we first analyze the current challenges of developing intelligent algorithms for RS image interpretation with bibliometric investigations. We then present the general guidances on creating benchmark datasets in efficient manners. Following the presented guidances, we also provide an example on building RS image dataset, i.e., Million Aerial Image Dataset (Online. Available: https://captain-whu.github.io/DiRS/0 ), a new large-scale benchmark dataset containing a million instances for RS image scene classification. Several challenges and perspectives in RS image annotation are finally discussed to facilitate the research in benchmark dataset construction. We do hope this article will provide the RS community an overall perspective on constructing large-scale and practical image datasets for further research, especially data-driven ones.

/pdf/on-creating-benchmark-dataset-for-aerial-image-snt0rfbwlw.pdf

On Creating Benchmark Dataset for Aerial Image Interpretation: Reviews, Guidances, and Million-AID

Remote sensing image scene classification, which aims at labeling remote sensing images with a set of semantic categories based on their contents, has broad applications in a range of fields. Propelled by the powerful feature learning capabilities of deep neural networks, remote sensing image scene classification driven by deep learning has drawn remarkable attention and achieved significant breakthroughs. However, to the best of our knowledge, a comprehensive review of recent achievements regarding deep learning for scene classification of remote sensing images is still lacking. Considering the rapid evolution of this field, this article provides a systematic survey of deep learning methods for remote sensing image scene classification by covering more than 160 papers. To be specific, we discuss the main challenges of remote sensing image scene classification and survey: first, autoencoder-based remote sensing image scene classification methods; second, convolutional neural network-based remote sensing image scene classification methods; and third, generative adversarial network-based remote sensing image scene classification methods. In addition, we introduce the benchmarks used for remote sensing image scene classification and summarize the performance of more than two dozen of representative algorithms on three commonly used benchmark datasets. Finally, we discuss the promising opportunities for further research.

Remote Sensing Image Scene Classification Meets Deep Learning: Challenges, Methods, Benchmarks, and Opportunities

Automatic and robust object detection in remote sensing images is of vital significance in real-world applications such as land resource management and disaster rescue. However, poor performance arises when the state-of-the-art natural image detection algorithms are directly applied to remote sensing images, which largely results from the variations in object scale, aspect ratio, indistinguishable object appearances, and complex background scenario. In this paper, we propose a novel Feature Enhancement Network (FENet) for object detection in optical remote sensing images, which consists of a Dual Attention Feature Enhancement (DAFE) module and a Context Feature Enhancement (CFE) module. Specifically, the DAFE module is introduced to highlight the network to focus on the distinctive features of the objects of interest and suppress useless ones by jointly recalibrating the spatial and channel feature responses. The CFE module is designed to capture global context cues and selectively strengthen class-aware features by leveraging image-level contextual information that indicates the presence or absence of the object classes. To this end, we employ a context encoding loss to regularize the model training which promotes the object detector to understand the scene better and narrows the probable object categories in prediction. We achieve our proposed FENet by unifying DAFE and CFE into the framework of Faster R-CNN. In the experiments, we evaluate our proposed method on two large-scale remote sensing image object detection datasets including DIOR and DOTA and demonstrate its effectiveness compared with the baseline methods.

/pdf/feature-enhancement-network-for-object-detection-in-optical-510fvzm1yy.pdf

Feature Enhancement Network for Object Detection in Optical Remote Sensing Images

Oriented object detection is a practical and challenging task in remote sensing image interpretation. Nowadays, oriented detectors mostly use horizontal boxes as intermedium to derive oriented boxes from them. However, the horizontal boxes are inclined to get a small Intersection-over-Unions (IoUs) with ground truths, which may have some undesirable effects, such as introducing redundant noise, mismatching with ground truths, detracting from the robustness of detectors, etc. In this paper, we propose a novel Anchor-free Oriented Proposal Generator (AOPG) that abandons the horizontal boxes-related operations from the network architecture. AOPG first produces coarse oriented boxes by Coarse Location Module (CLM) in an anchor-free manner and then refines them into high-quality oriented proposals. After AOPG, we apply a Fast R-CNN head to produce the final detection results. Furthermore, the shortage of large-scale datasets is also a hindrance to the development of oriented object detection. To alleviate the data insufficiency, we release a new dataset on the basis of our DIOR dataset and name it DIOR-R. Massive experiments demonstrate the effectiveness of AOPG. Particularly, without bells and whistles, we achieve the highest accuracy of 64.41$\%$, 75.24$\%$ and 96.22$\%$ mAP on the DIOR-R, DOTA and HRSC2016 datasets respectively. Code and models are available at https://github.com/jbwang1997/AOPG.

Anchor-free Oriented Proposal Generator for Object Detection

In the past few years, object detection in remote sensing images has achieved remarkable progress. However, the detection of oriented and densely packed objects are still unsatisfactory due to the following spatial and feature misalignments. i) Most two-stage oriented detectors only introduce an orientation regression branch in the detection head, while still leverage horizontal proposals for classification and regression. This inevitably results in the spatial misalignment problem between horizontal proposals and oriented objects. ii) The features used for classification are in fact extracted from the region proposals which have shifted to the final predictions via the regression branch. This leads to the feature misalignment problem between the classification and the localization tasks. In this paper, we present a two-stage oriented object detection method, termed Dual-aligned Oriented Detector (DODet), towards evading the aforementioned problems of spatial and feature misalignments. In DODet, the first stage is an Oriented Proposal Network (OPN), which generates high-quality oriented proposals via a novel representation scheme of oriented objects. The second stage is a Localization-guided Detection Head (LDH) that aims at alleviating the feature misalignment between classification and localization. Comprehensive and extensive evaluations on three benchmarks, including DIOR-R, DOTA, and HRSC2016, indicate that our method could obtain consistent and substantial gains compared with the baseline method. The source code will be publicly available at https://github.com/yanqingyao1994/DODet.

Dual-aligned Oriented Detector

Detecting objects in remote sensing images (RSIs) using oriented bounding boxes (OBBs) is flourishing but challenging, wherein the design of OBB representations is the key to achieving accurate detection. In this article, we focus on two issues that hinder the performance of the two-stage oriented detectors: 1) the notorious boundary discontinuity problem, which would result in significant loss increases in boundary conditions, and 2) the inconsistency in regression schemes between the two stages. We propose a simple and effective bounding box representation by drawing inspiration from the polar coordinate system and integrate it into two detection stages to circumvent the two issues. The first stage specifically initializes four quadrant points as the starting points of the regression for producing high-quality oriented candidates without any postprocessing. In the second stage, the final localization results are refined using the proposed novel bounding box representation, which can fully release the capabilities of the oriented detectors. Such consistency brings a good trade-off between accuracy and speed. With only flipping augmentation and single-scale training and testing, our approach with ResNet-50-FPN harvests 76.25% mAP on the DOTA dataset with a speed of up to 16.5 frames/s, achieving the best accuracy and the fastest speed among the mainstream two-stage oriented detectors. Additional results on the DIOR-R and HRSC2016 datasets also demonstrate the effectiveness and robustness of our method. The source code is publicly available at https://github.com/yanqingyao1994/QPDet.

Xingxing Xie

Papers

Remote Sensing Image Scene Classification Meets Deep Learning: Challenges, Methods, Benchmarks, and Opportunities

Feature Enhancement Network for Object Detection in Optical Remote Sensing Images

Anchor-free Oriented Proposal Generator for Object Detection

Dual-aligned Oriented Detector

On Improving Bounding Box Representations for Oriented Object Detection