Deep Learning for Remote Sensing Data_A technical tutorial on the state of the art

• 一，Abstract
• 二，ADVANTAGES OF REMOTE SENSING METHODS遙感方法的優點
• • 注釋2：Handcrafted features
  • 注釋1：discriminability and robustness
  • 注釋3：unitary transformation
• 三，THE GENERAL FRAMEWORK 總體框架
• • 注釋1：pan sharpening，全色銳化/全色波段融合panchromatic (PAN) images全色(PAN)圖像
  • 注釋2：object proposals
  • 注釋3：shallow module such as an autoencoder (AE) or a sparse coding algorithm 淺模塊，例如：自動編碼器(AE)或稀疏編碼算法
• 四，BASIC ALGORITHMS IN DEEP LEARNING深度學習的基本算法
• • 注釋1：restricted Boltzmann machines (RBMs)
  • 注釋2：機器學習中DL算法的更多詳細信息可以在[14]和[44]中找到
  • 1，Convolutional neural networks
  • • 1.1CONVOLUTIONAL LAYER卷積層
    • 1.2NONLINEARITY LAYER非線性層
    • 1.3POOLING LAYER池化層/采樣層
    • 1.4 auto encoders（AE)自動編碼器
    • 1.5 restricted boltzmann machines受限玻爾茲曼機
    • 1.6 sparse coding稀疏編碼
  • 2，DEEP LEARNING FOR REMOTE SENSING DATA
  • • 2.1 REMOTE SENSING IMAGE PREPROCESSING
    • • 2.1.1 RESTORATION AND DENOISING 修復和去噪
      • 2.1.2 PAN SHARPENING 全色波段融合/全色銳化
      • • 解釋 PAN SHARPENING 全色銳化
    • 2.2 PIXEL-BASED CLASSIFICATION
    • • 2.2.1 SPECTRAL FEATURE CLASSIFICATION光譜特征分類
      • 2.2.2 CLASSIFICATION WITH SPATIAL INFORMATION空間信息分類
    • 2.3 TARGET RECOGNITION
    • • 2.3.1 GENERAL DEEP-LEARNING FRAMEWORK OF REMOTE SENSING TARGET RECOGNITION 遙感目標識別的一般深度學習框架
      • 2.3.2 SAMPLE SELECTION PROPOSALS樣本選擇的建議
      • 2.3.3 LOW-TO MIDDLE-LEVEL FEATURE LEARNING低級到中級的特征學習
      • 2.3.4 TRAINING THE DEEP-LEARNING NETWORKS
      • 2.3.5 SUPERVISED METHODS
      • 2.3.6 UNSUPERVISED METHODS
    • 2.4 SCENE UNDERSTANDING
    • 2.4.1 UNSUPERVISED HIERARCHICAL FEATURE-LEARNING-BASED METHODS
    • • 2.4.2 PATCH EXTRACTION
      • 2.4.3 FEATURE EXTRACTION
      • 2.4.4 FEATURE REPRESENTATION
      • 2.4.5 CLASSIFICATION
      • 2.4.6 SUPERVISED HIERARCHICAL FEARTURE-LEARNING-BASED METHODS
• 五，EXPERIMENTS AND ANALYSIS
• 六，CONCLUSIONS AND FUTURE WORK

文章來源：https://ieeexplore.ieee.org/abstract/document/7486259Advances in Machine Learning for Remote Sensing and Geosciences

一，Abstract

Deep-learning (DL) algorithms, which learn the representative and discriminative features in a hierarchical manner from the data, have recently become a hotspot in the machine-learning area and have been introduced into the geoscience and remote sensing (RS) community for RS big data analysis.深度學習(deep learning, DL)算法從數據中分層學習代表性特征和區分性特征，是近年來機器學習領域的一個研究熱點，已被引入地球科學和遙感領域，用于遙感大數據分析。Considering the low-level features (e.g., spectral and texture) as the bottom level, the output feature representation from the top level of the network can be directly fed into a subsequent classifier for pixel-based classification。將底層特征(如光譜和紋理)作為底層，網絡頂層的輸出特征表示可以直接被送入后續的基于像素的分類器中進行分類。As a matter of fact, by carefully addressing the practical demands in RS applications and designing the input–output levels of the whole network, we have found that DL is actually everywhere in RS data analysis: from the traditional topics of image preprocessing, pixel-based classification, and target recognition, to the recent challenging tasks of high-level semantic feature extraction and RS scene understanding. 事實上，通過仔細處理RS應用的實際需求和設計整個網絡的輸入—輸出水平,我們發現DL實際上在RS數據分析中無處不在:從圖像預處理的傳統主題，基于像素的分類、目標識別，到最近的有挑戰性的任務：高水平語義特征提取和RS場景理解。

In this technical tutorial, a general framework of DL for RS data is provided, and the state-of-the-art DL methods in RS are regarded as special cases of input–output data combined with various deep networks and tuning tricks. Although extensive experimental results confirm the excellent performance of the DL-based algorithms in RS big data analysis, even more exciting prospects can be expected for DL in RS. Key bottlenecks and potential directions are also indicated in this article, guiding further research into DL for RS data.在本技術教程中，給出了用于RS數據的DL通用框架，并且RS中最先進的DL方法被認為是輸入—輸出數據與各種深度網絡和調優技巧相結合的特殊情況。盡管大量的實驗結果證實了基于dl的算法在RS大數據分析中的優異性能，但是還可以期待DL在RS領域會有更令人興奮的前景。本文還指出了關鍵瓶頸和潛在發展方向，為進一步開展DL在RS數據中的研究提供指導。

二，ADVANTAGES OF REMOTE SENSING METHODS遙感方法的優點

RS techniques have opened a door to helping people widen their ability to understand the earth. In fact, RS techniques are becoming more and more important in data-collection tasks. RS技術在數據采集中發揮著越來越重要的作用. Information technology companies depend on RS to update their location-based services. 信息技術公司依靠RS來更新他們的基于位置的服務. Google Earth employs high-resolution (HR) RS images to provide vivid pictures of the earth’s surface. 谷歌地球采用高分辨率(HR) RS圖像，提供生動的地球表面圖片。 Governments have also utilized RS for a variety of public services, from weather reporting to traffic monitoring. 政府還將RS用于各種公共服務，從天氣報告到交通監控。 Nowadays, one cannot imagine a life without RS. Recent years have even witnessed a boom in RS satellites,
providing for the first time an extremely large number of geographical images of nearly every corner of the earth’s surface.近年來，RS衛星發展迅猛，
首次提供了大量地球表面幾乎每個角落的地理圖像. Data warehouses of RS images are increasing daily, including images with different spectral and spatial resolutions. 遙感影像數據倉庫每天都在增加，包括不同光譜分辨率和空間分辨率的影像。

How can we extract valuable information from the various kinds of RS data? How should we deal with the ever-increasing data types and volume? 如何從各種遙感數據中提取有價值的信息?我們應該如何處理不斷增長的數據類型和容量? The traditional approaches exploit features from RS images with which information-extraction models can be constructed. 傳統的方法利用遙感圖像的特征,利用這些特征可以構建信息提取模型. Handcrafted features have proved effective and can represent a variety of spectral, textural, and geometrical attributes of the images. However, since these features cannot easily consider the details of real data, it is impossible for them to achieve an optimal balance between discriminability and robustness.手工制作的特征已被證明是有效的，可以代表圖像的各種光譜、紋理和幾何屬性. 然而，由于這些特征不能很容易地考慮真實數據的細節，它們無法在可區分性和魯棒性之間達到最優平衡。When facing the big data of RS images, the situation is even worse, since the imaging circumstances vary so greatly that images can change a lot in a short interval. Thanks to DL theory , which provides an alternative way to automatically learn fruitful features from the training set, unsupervised feature learning from very large raw-image data sets has become possible. Actually, DL has proven to be a new and exciting tool that could be the next trend in the development of RS image processing. 當面對遙感圖像大數據時，情況更是如此，因為成像環境變化很大，圖像在短時間內可以發生很大的變化。DL理論提供了一種從訓練集自動學習豐富特性的替代方法，DL理論使得從非常大的原始圖像數據集進行無監督特征學習成為可能。Actually, DL has proven to be a new and exciting tool that could be the next trend in the development of RS image processing. 事實上，DL已經被證明是一種新的和令人興奮的工具，它可能是RS圖像處理的下一個發展趨勢。

注釋2：Handcrafted features

注釋1：discriminability and robustness

可區分性和魯棒性
robustness——魯棒性；穩健性；健壯性

RS images, despite the spectral and spatial resolution, are reflections of the land surface , with an important property being their ability to record multiple-scale information within an area. 盡管遙感圖像具有光譜和空間分辨率，但它是地表的反射，其重要的特性是能夠記錄一個區域內的多尺度信息. According to the type of information that is desired, pixel-based, object-based, or structure-based features can be extracted. 根據所需的信息類型，可以提取基于像素的、基于對象的或基于結構的特征. However, an effective and universal approach has not yet been reported to optimally fuse these features, due to the subtle relationships between the data. In contrast, DL can represent and organize multiple levels of information to express complex relationships between data. 然而，由于數據之間的微妙關系，目前還沒有一種有效的、通用的方法來最佳地融合這些特征. 相反，DL可以表示和組織多層次的信息來表達數據之間的復雜關系. In fact, DL techniques can map different levels of abstractions from the images and combine them from low level to high level. 事實上，DL技術可以從圖像中映射不同層次的抽象，并將它們從低層次組合到高層次。 Consider scene recognition as an example, where, with the help of DL, the scenes can be represented as a unitary transformation by exploiting the variations in the local spatial arrangements and structural patterns captured by the low-level features, where no segmentation stage or individual object extraction stage is needed. 以場景識別為例，在DL的幫助下，利用底層特征捕捉到的局部空間布局和結構模式的變化，將場景表示為一個酉變換，在這里不需要分割階段或單獨的對象提取階段。

注釋3：unitary transformation

Despite its great potential, DL cannot be directly used in many RS tasks, with one obstacle being the large numbers of bands. 盡管DL具有巨大的潛力，但它不能直接用于許多RS任務，一個障礙是波段的數量太多。Some RS images, especially hyperspectral ones, contain hundreds of bands that can cause a small patch to be a really large data cube, which corresponds to a large number of neurons in a pretrained network .一些遙感圖像，尤其是高光譜圖像，包含數百個波段，可以使一個小斑塊變成一個非常大的數據立方體，這對應著一個預先訓練好的網絡中的大量神經元。 In addition to the visual geometrical patterns within each band, the spectral curve vectors across bands are also important information. 除了每個波段內的視覺幾何模式外，跨波段的光譜曲線向量也是重要的信息。 However, how to utilize this information still requires
further research. Problems still exist in the high-spatial-resolution RS images, which have only green, red, and blue channels, the same as the benchmark data sets for DL. In practice, very few labeled samples are available, which may make a pretrained network difficult to construct. Furthermore, images acquired by different sensors present large differences. How to transfer the pretrained network to other images is still unknown. 然而，如何利用這些信息仍然需要進一步的研究。與DL的基準數據集一樣，高空間分辨率遙感圖像只有綠、紅、藍三通道，但是高空間分辨率遙感圖像中仍然存在問題。在實踐中，可用的標簽樣本很少，這可能使預先訓練好的網絡難以構建。另外，不同傳感器獲取的圖像差異較大。如何將預先訓練好的網絡遷移到其他圖像上仍然是未知的。

In this article, we survey the recent developments in DL for the RS field and provide a technique tutorial on the design of DL-based methods for optical RS data. Although there are also several advanced techniques for DL for synthetic aperture radar images and light detection and ranging (LiDAR) point clouds data , they share the
similar basic DL ideas of the data analysis model. 在這篇文章中，我們調查了遙感領域DL的最新發展，并提供了一個關于設計處理光學遙感數據的DL方法的技術教程。盡管也有一些用于合成孔徑雷達圖像和光探測和測距(LiDAR)點云數據的DL先進技術，但它們的數據分析模型的基本DL思想類似。

三，THE GENERAL FRAMEWORK 總體框架

Despite the complex hierarchical structures, all of the DL-based methods can be fused into a general framework. 盡管層次結構復雜，但所有基于DL的方法都可以融合為一個通用框架.Figure 1 illustrates a general framework of DL for RS data analysis. 圖1展示了用于RS數據分析的DL的一般框架. The flowchart includes three main components, the prepared input data, the core deep networks, and the expected output data. 流程圖包括三個主要部分，準備好的輸入數據、核心深度網絡和預期的輸出數據. In practice, the input output data pairs are dependent on the particular application. 在實踐中，輸入輸出數據對 依賴于特定的應用.For example, for RS image pan sharpening, they are the HR and low-resolution (LR) image patches from the panchromatic (PAN) images; for pixel-based classification, they are the spectral–spatial features and their feature representations (unsupervised version) or label information (supervised version) ; while, for tasks of target recognition and scene understanding , the inputs are the features extracted from the object proposals, as well as the raw pixel digital numbers from the HR images and RS image databases respectively, and the output data are always the same as in the application of pixel-based classification, as described previously. 例如，對于RS圖像的pan銳化，它們是來自全色(pan)圖像的高分辨率(HR)和低分辨率(LR)圖像斑塊;對于基于像素的分類，它們是光譜—空間特征及其特征表示(無監督版本)或標簽信息(監督版本);而對于目標識別和場景理解，輸入分別為從目標建議中提取的特征，以及從HR圖像和RS圖像數據庫中提取的原始像素數字，并且如前所述，輸出數據始終是與在應用基于像素的分類時是相同的。

注釋1：pan sharpening，全色銳化/全色波段融合panchromatic (PAN) images全色(PAN)圖像

注釋2：object proposals

When the input–output data pairs have been properly defined, the intrinsic and natural relationship between the input and output data is then constructed by a deep architecture composed of multiple levels of nonlinear operations, where each level is modeled by a shallow module such as an autoencoder (AE) or a sparse coding algorithm. 當輸入—輸出數據對已被正確定義,輸入和輸出數據之間的內在和自然關系由多層非線性操作組成的深層體系結構構成，其中每一層都是由一個淺層模塊(如自動編碼器(AE)或稀疏編碼算法)建模的。由深架構組成的多級非線性操作,在每個級別由淺建模模塊如autoencoder (AE)或稀疏編碼算法。It should be noted that, if a sufficient training sample set is available, such a deep network turns out to be a supervised approach. 需要指出的是，如果有足夠的訓練樣本集，那么這種深度網絡是一種有監督的方法。 It can be further fine-tuned by the use of the label information, and the top-layer output of the network is the label information rather than the abstract feature representation learned by an unsupervised deep network. 它可以利用標簽信息進行進一步的微調，網絡的頂層輸出是標簽信息，而不是無監督深度網絡學習的抽象特征表示。When the core deep network has been well trained, it can be employed to predict the expected output data of a given test sample. 當核心深度網絡經過良好的訓練后，就可以用來預測給定測試樣本的預期輸出數據。Along with the general framework in Figure 1, we describe a few basic algorithms in the deep networkconstruction tutorial in the following section, and we then review the representative techniques in DL for RS data analysis from four perspectives: 1) RS image preprocessing, 2) pixel-based classification, 3) target recognition, and 4) scene understanding. 除了圖1中的一般框架外，我們將在下一節的深度網絡構建教程中描述一些基本算法，然后從四個角度回顧RS數據分析中的DL代表性技術：1）RS圖像預處理，2）基于像素分類，3）目標識別，4）場景理解

注釋3：shallow module such as an autoencoder (AE) or a sparse coding algorithm 淺模塊，例如：自動編碼器(AE)或稀疏編碼算法

四，BASIC ALGORITHMS IN DEEP LEARNING深度學習的基本算法

In recent years, the various DL architectures have thrived and have been applied in fields such as audio recognition, natural language processing, and many classification tasks, where they have usually outperformed the traditional methods. M近年來，各種DL體系結構蓬勃發展，并已應用于音頻識別、自然語言處理和許多分類任務等領域，在這些領域它們的性能通常優于傳統的方法. The motivation for such an idea is inspired by the fact that the mammal brain is organized in a deep architecture, with a given input percept represented at multiple levels of abstraction, for the primate visual system in particular. 產生這種想法(DL體系)的動機是由于哺乳動物的大腦是在一個深層結構中組織起來的，給定的輸入知覺在多個抽象層次上表示，尤其是靈長類動物的視覺系統。Inspired by the architectural depth of the human brain, DL researchers have developed novel deep architectures as an alternative to shallow architectures. 受人類大腦架構深度的啟發，DL研究人員開發了新穎的深層架構，作為淺層架構的替代方案。Deep belief networks (DBNs) are a major breakthrough in DL research and train one layer at a time in an unsupervised manner by restricted Boltzmann machines (RBMs) . 深度信念網絡(DBNs)是DL研究的一個重大突破，它采用限制玻爾茲曼機器(RBMs)進行無監督的逐層訓練。A short while later, a number of AE-based algorithms
were proposed that also train the intermediate levels of representation locally at each level (i.e., the AE and its variants, such as the sparse AE and the denoising AE). 不久之后，提出了一些基于自動編碼器(AE)的算法，可以在每一層(即聲發射及其變體，如稀疏聲發射和去噪聲發射)局部地訓練中間層(自動編碼器及其變體，如稀疏自動編碼器和去噪自動編碼器)。Unlike AEs, the sparse coding algorithms generate sparse representations from the data themselves from a different perspective by learning an overcomplete dictionary via self-decomposition. 與自動編碼器不同的是，稀疏編碼算法通過自分解來學習一個超完整的字典，借此從不同的角度從數據本身生成稀疏表示。In addition, as the most representative supervised DL model, convolutional neural networks (CNNs) have outperformed most algorithms in visual recognition. 并且，作為最具代表性的監督DL模型，卷積神經網絡(CNNs)在視覺識別中的表現優于大多數算法。The deep structure of CNNs allows the model to learn highly abstract feature detectors and to map the input features into representations that can clearly boost the performance of the subsequent classifiers. CNNs的深層結構使得模型可以學習高度抽象的特征檢測器，并將輸入的特征映射成表征，可以明顯提升后續分類器的性能。Furthermore, there are many optional techniques that can be used to train the DL architecture shown in Figure 1.此外，還有許多可選的技術可以用來訓練圖1所示的DL架構。In this review, we only provide a brief introduction to the following four typical models that have already been used in the RS community and can be embedded into the general framework to achieve the particular application. 在這篇綜述中，我們只對以下四種典型模型進行簡單的介紹，這些模型已經在RS界使用并且可以嵌入到通用框架中實現特定的應用。More detailed information regarding the DL algorithms in the machine-learning community can be found in [14] and [44]. 關于機器學習社區中DL算法的更多詳細信息可以在[14]和[44]中找到。

注釋1：restricted Boltzmann machines (RBMs)

注釋2：機器學習中DL算法的更多詳細信息可以在[14]和[44]中找到

論文： [14] Y. Bengio, A. Courville, and P. Vincent, “Representation learn- ing: A review and new perspectives,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1798–1828, 2013. [44] Y. Bengio, “Learning deep architectures for AI,” Found. Trends Mach. Learning, vol. 2, no. 1, pp. 1–127, 2009.

1，Convolutional neural networks

The CNN is a trainable multilayer architecture composed of multiple feature-extraction stages. Each stage consists of three layers: 1) a convolutional layer, 2) a nonlinearity layer, and 3) a pooling layer. CNN是一個可訓練的多層架構，由多個特征提取階段組成。The architecture of a CNN is designed to take advantage of the two-dimensional structure of the input image.CNN的架構被設計可以利用輸入圖像的二維結構。A typical CNN is composed of one, two, or three such feature-extraction stages, followed by one or more traditional, fully connected layers and a final classifier layer. Each layer type is described in the following sections.一個典型的CNN是由一個、兩個或三個這樣的特征提取階段組成，然后是一個或多個傳統的、全連接層和一個最后的分類層。每種層類型在下面的章節中進行描述。

1.1CONVOLUTIONAL LAYER卷積層

The input to the convolutional layer is a three-dimensional array with r two-dimensional feature maps of size m # n. 卷積層的輸入是一個三維數組(r個尺寸為m#n的二維特征圖)。

每個分量表示為每個特征圖表示為xi。輸出也是一個三維數組mi×ni×k，由k個尺寸為mi×ni的特征圖組成。卷積層有k個尺寸為L×L×q的可訓練的濾波器，也被稱為濾波器組
W（濾波器組將輸入特征圖連接到輸出特征圖）。卷積層計算輸出特征圖，在這里 * 是一種二維離散卷積算子并且b是可訓練的偏差參數。

1.2NONLINEARITY LAYER非線性層

In the traditional CNN, this layer simply consists of a pointwise nonlinearity function applied to each component in a feature map. The nonlinearity layer computes the output feature map , as is commonly chosen to be a rectified linear unit (ReLU) .
在傳統的CNN中，這一層僅僅由一個點態(逐點的)非線性函數組成，點態(逐點的)非線性函數應用于特征圖中的每個組件。非線性層計算輸出特征圖,通常被選擇為修正線性單元(ReLU)。

1.3POOLING LAYER池化層/采樣層

The pooling layer involves executing a max operation over the activations within a small spatial region G of each feature map:池化層涉及在每個特征圖的一個小空間區域G內對激活執行一個最大化操作:。To be more precise, the pooling layer can be thought of as consisting of a grid of pooling units spaced s pixels apart, each summarizing a small spatial region of size p * p centered at the location of the pooling unit. 更準確地說，池化層可以被認為是由間隔s個像素的池化單元網格組成，每個網格概括了以池化單元位置為中心的尺寸為p*p的小空間區域。After the multiple feature-extraction stages, the entire network is trained with back propagation of a supervised loss function such as the classic least-squares output, and the target output y is represented as a 1–of–K vector, where K is the number of output and L is the number of layers:。在經過多個特征提取階段后，對整個網絡進行監督損失函數的反向傳播訓練（如經典的最小二乘輸出），并且目標輸出y表示為1-of-K向量，其中K為輸出數，L為層數：在這里，I指層數。Our goal is to minimize
as a function of .To train the CNN, we can apply stochastic gradient descent with backpropagation to optimize the function.我們的目標是最小化。為了訓練CNN，我們可以使用帶有反向傳播的隨機梯度下降來優化函數。CNNs have recently become a popular DL method and have achieved great success in large-scale visual recognition, which has become possible due to the large public
image repositories, such as ImageNet. CNN最近成為一種流行的DL方法，并在大規模的視覺識別中取得了巨大的成功，（這得益于大量的公共圖像存儲庫，如ImageNet）。In the RS community, there are also some recent works on CNN-based RS image pixel classification, target recognition, and scene understanding.在RS界，最近也有一些基于CNN的RS圖像像素分類、目標識別和場景理解的工作。

1.4 auto encoders（AE)自動編碼器

An AE is a symmetrical neural network that is used to learn the features from a data set in an unsupervised manner by minimizing the reconstruction error between the input data at the encoding layer and its reconstruction at the decoding layer. AE是一種對稱的神經網絡，它通過最小化編碼層輸入數據和其在解碼層重構之間的重構誤差，以無監督的方式從數據集中學習特征。During the encoding step, an input vector is processed by applying a linear mapping and a nonlinear activation function to the network: where is a weight matrix with K features, is the encoding bias, and g(x) is the logistic sigmoid function We decode a vector using a separate linear decoding matrix:where is a weight matrix and is the decoding bias. Feature extractors in the data set are learned by minimizing the cost function, and the first term in the reconstruction is the error term. The second term is a regularization term (also called a weight decay term in a neural network):where X and Z are the training and reconstructed data, respectively. 在編碼步驟中，通過對網絡
應用線性映射和非線性激活函數來處理輸入向量。在這里，是一個具有K個特征的權值矩陣，是解碼偏差。數據集中的特征提取器是通過最小化代價函數來學習的，重構中的第一項是誤差項。第二項是正則化項（在神經網絡中也稱為權重衰減項)：。其中，X和Z分別為訓練數據和重構數據。

We recall that a denotes the activation of hidden units in the AE. 我們回顧一下，a表示AE中隱藏單元的激活。Thus, when the network is provided with a specific input be the average activation of averaged over the training set. We want to approximately enforce the constraint where is the
sparsity parameter, which is typically a small value close to zero. In other words,we want the average activation of each hidden neuron to be close to zero. To satisfy this constraint, the hidden units activations must be mostly inactive and close to zero so that most of the neurons are inactive.因此，當網絡被提供給一個特定的輸入時,假設為 在訓練集上的平均的平均激活量。我們想要近似地執行約束條件，在這里是稀疏性參數，通常是一個接近于零的小值。換句話說，我們要每個隱藏神經元的平均激活量近似為0。為了滿足這個約束條件，隱藏單元的激活必須大部分是不活躍的，并且接近于零，所以大部分神經元是不活躍的. To achieve this, the objective in the sparse AE learning is to minimize the reconstruction error with a sparsity constraint, i.e., a sparse AE: where β is the weight of the sparsity penalty, K is the number of features in the weight matrix, and KL(.) is the Kull-back-Leibler divergence given by .為了實現這一目標，稀疏AE學習中的目標是在稀疏性約束下最小化重構誤差，即稀疏AE。在這里，β 是稀疏性懲罰的權重，K是權重矩陣中的特征數。KL(.) 是Kull-back-Leibler分叉點。

This penalty function has the property that .Otherwise, it increases monotonically as diverges from ρ，which acts as the sparsity constraint. An AE can be directly employed as a feature extractor for RS data analysis [51], and it has been more frequently stacked into the AEs for DL from RS data [52]–[54]. 這個懲罰函數具有以下性質 。否則，它單調地遞增，因為偏離ρ（其作為稀疏性約束）。AE可以直接作為RS數據分析的特征提取器，并且它更多地被堆積到了RS數據的DL的AEs中。

1.5 restricted boltzmann machines受限玻爾茲曼機

An RBM is commonly used as a layer-wise training model in the construction of a DBN.在構建DBN的過程中，通常使用RBM作為層間訓練模型。It is a two-layer network, presenting a particular type of Markov random field with visible units and hidden units .它是一個兩層網絡，呈現的是一種特殊類型的馬爾科夫隨機場（可見單元v和隱藏單元h）。A joint configuration of the units has an energy given by where is the weight between visible unit i and hidden unit j, and are bias terms of the visible and hidden unit, respectively. 單元的聯合配置具有以下能量 ，在這里是可見單元i和隱藏單元j之間的權重，而分別為可見單元和隱藏單元的偏置項。

The joint distribution over the units is defined by where Z(θ) is the normalizing constant. 這里,Z(θ)是歸一化常數。The network assigns a probability to every input vector via the energy function. 網絡通過能量函數給每個輸入向量分配一個概率。The probability of the training vector can be raised by adjustment to lower the energy, as given in (7).通過調整降低能量，可以提高訓練向量的概率，如（7）所示。 The conditional distributions of hidden unit h and input vector v are given by the logistic function(如下). 隱藏單元h和輸入向量v的條件分布由對數函數給出。.
Once the states of hidden units are chosen, the input data can be reconstructed by setting each vi to 1 with the probability in (11). 一旦選擇了隱藏單元的狀態，就可以按(11)中的概率將每個vi設為1來重構輸入數據。The hidden units’ states are then updated to represent the features of the reconstruction. 然后更新隱藏單元的狀態來代表重構的特征。The learning of W is done through a method called contrastive divergence (CD). W的學習是通過一種叫做對比散度（CD）的方法來完成的。The DBN has been applied to the RS image spatial–spectral classification and shows superior performance compared to the conventional feature dimensionality-reduction methods, such as principal component analysis (PCA), and classifiers, such as support vector machines (SVMs) [55], [29]. DBN已應用于RS圖像空間光譜分類，與傳統的特征維度還原方法相比，表現出優越的性能。In recent years, it has also been successfully proposed for object recognition [56] and scene classification [57].近年來，還成功提出目標識別和場景分類 。

1.6 sparse coding稀疏編碼

Sparse coding is a type of unsupervised method for learning sets of overcomplete bases to represent data efficiently to find a set of basis vectors such that we can represent an input vector x as a linear combination of these basis vectors: 稀疏編碼是一種無監督的方法，用于學習過完整的基數集來有效地表示數據，以找到一組基向量 以致于我們可以表示一個輸入向量x是這些基向量的線性組合。While techniques such as PCA allow us to learn a complete set of basis vectors efficiently, we wish to learn an overcomplete set of basis vectors to represent the input vectors x. 雖然PCA等技術允許我們高效地學習一組完整的基向量，但我們希望學習一組超完整的基向量來表示輸入向量x。 The advantage of having an overcomplete basis set is that our basis vectors are better able to capture structures and patterns inherent in the input data. 擁有一個過度完整的基礎集的好處是，我們的基礎向量能夠更好地捕捉輸入數據中固有的結構和模式。However, with an overcomplete basis set, the coefficients are no longer uniquely determined by the input vector x. 然而，在一個過完整的基礎集上，系數不再由輸入向量x唯一決定。 Therefore, in sparse coding, we introduce the additional criterion of sparsity to resolve the degeneracy introduced by the overcompleteness. 因此，在稀疏編碼中，我們引入了額外的稀疏性標準，以解決過度完整所帶來的退化問題。

We define the sparse coding cost function on a set of m input vectors as 我們將一組m個輸入向量的稀疏編碼代價函數定義為 where S(.) is a sparsity cost function that penalizes
for being far from zero. 其中S(.)是一個稀疏性代價函數，對遠離零進行懲罰。 We can interpret the first term of the sparse coding objective as a reconstruction term that tries to force the algorithm to provide a good representation of x, and the second term can be defined as a sparsity penalty that forces our representation of x to be sparse. 我們可以將稀疏編碼目標的第一項解釋為重構項，其試圖迫使算法提供x的良好表示，第二項可以定義為稀疏性懲罰，迫使我們對x的表示是稀疏的。

A large number of sparse coding methods have been proposed. 已有大量的稀疏編碼方法被提出。Notably, for RS scene classification, Cheriyadat [58] introduces a variant of sparse coding that combines local scale-invariant feature transform (SIFT)-based feature descriptors to generate a new sparse representation, while, in [59], the sparse coding is used to reduce the potential redundant information in the feature representation. 值得注意的是，對于RS場景分類，Cheriyadat引入了一種稀疏編碼的變體，其結合基于局部尺度不變特征變換（SIFT）的特征描述符來生成新的稀疏表示，而在[59]中，稀疏編碼是用來減少特征表示中潛在的冗余信息。In addition, as a computationally efficient unsupervised feature-learning technique, k-means clustering has also been played as a single-layer feature extractor for RS scene classification [60]–[62] and achieves state-of-the-art performance. 此外，作為一種計算效率很高的無監督特征學習技術，k-means聚類作為RS場景分類的單層特征提取器也得到了利用[60]-[62]，并達到了最先進的性能。

2，DEEP LEARNING FOR REMOTE SENSING DATA

The “Basic Algorithms in Deep Learning” section discussed some of the basic elements used in constructing a DL architecture as well as the general framework. "深度學習中的基本算法 "部分討論了構建DL架構時使用的一些基本元素以及總體框架。 In practice, the mathematical problems of the various RS data analysis techniques can be regarded as special cases of input–output data combined with a particular DL network based on the aforementioned algorithms. 在實踐中，各種RS數據分析技術的數學問題可以看作是輸入輸出數據與基于上述算法的特定DL網絡相結合的特殊情況。In this section, we provide a tutorial on DL for RS data from four perspectives: 1) image preprocessing, 2) pixel-based classification, 3) target recognition, and 4) scene understanding. 在本節中，我們從四個方面對RS數據的DL提供了指導。1）圖像預處理，2）基于像素的分類，3）目標識別，4）場景理解。

2.1 REMOTE SENSING IMAGE PREPROCESSING

In practice, the observed RS images are not always as satisfactory as we demand due to many factors, including the limitations of the sensors and the influence of the atmosphere. 在實際工作中，由于傳感器的局限性和大氣層的影響等諸多因素，觀測到的RS圖像并不總是像我們要求的那樣令人滿意。 Therefore, there is a need for RS image preprocessing to enhance the image quality before the subsequent classification and recognition tasks. 因此，在后續的分類和識別任務之前，需要對RS圖像進行預處理，以提高圖像質量。 According to the related RS literature, most of the existing methods in RS image denoising, deblurring, superresolution, and pan sharpening are based on the standard image-processing techniques in the signal processing society, while there are very few machine-learning-based techniques. 根據相關的RS文獻，現有的RS圖像去噪、去模糊、超解像、平移銳化等方法大多是基于信號處理領域的標準圖像處理技術，而基于機器學習的技術非常少。 In fact, if we can effectively model the intrinsic correlation between the input (observed data) and output (ideal data) by a set of training samples, then the observed RS image could be enhanced by the same model. 事實上，如果我們能夠通過一組訓練樣本有效地模擬輸入（觀察數據）和輸出（理想數據）之間的內在相關性，那么觀察到的RS圖像就可以通過相同的模型來增強。 According to the basic techniques in the previous section, such an intrinsic correlation can be effectively explored by DL. 根據上一節的基本技術，這樣的內在關聯性可以通過DL進行有效的探索。In this tutorial, we consider two typical applications as the case studies, i.e., RS image restoration and pan sharpening, to show the state-of-the-art DL achievements in RS image preprocessing. 在本教程中，我們以兩個典型的應用為案例，即RS圖像修復和平移銳化，來展示DL在RS圖像預處理方面的最新成果。

Followed by the general framework of DL-based RS data preprocessing that we introduced in the “General Framework” section, the input data of the framework are usually the whole original image or the local image patches. 按照我們在 "總體框架 "部分介紹的基于DL的RS數據預處理的總體框架，該框架的輸入數據通常是整個原始圖像或局部圖像補丁。A specific deep network is then constructed, such as a deconvolution network [63] or a sparse denoising AE [28]. After that, the observed RS image is recovered by the learned DL model per spectral channel or per patch. 然后構建一個特定的深度網絡，如解卷積網絡[63]或稀疏去噪AE[28]。之后，觀察到的RS圖像由學習到的DL模型每一個頻譜通道或每一個補丁進行恢復。

2.1.1 RESTORATION AND DENOISING 修復和去噪

For RS image restoration and denoising, the original image is the input to a certain network that is trained with the clean image to obtain the restored and denoised image. 對于RS圖像的恢復和去噪，原始圖像是輸入（到一定的網絡中），用干凈的圖像進行訓練，得到恢復和去噪后的圖像。For instance, Zhang et al. utilized the deconvolution network for the restoration and denoising of RS images [63], which is an improved version of the L1-regularized deconvolution network. 例如，Zhang等利用解卷網絡用于RS圖像的恢復和去噪[63]，它是L1-正則化解卷網絡的改進版。The classical deconvolution network model is based on the convolutional decomposition of images under an L1 regularization, which is a sparse constraint term. 經典的解卷積網絡模型是基于L1正則化下圖像的卷積分解，這是一個稀疏的約束項。In the experiments undertaken in this study, adopting the L1∕2 regularization in the deep network gave sparser solutions than their L1 counterpart and has achieved satisfactory results. 在本研究進行的實驗中，在深度網絡中采用L1∕2正則化，得到的解比其對應的L1更稀疏，已經取得了令人滿意的結果。

2.1.2 PAN SHARPENING 全色波段融合/全色銳化

解釋 PAN SHARPENING 全色銳化

全色銳化使用分辨率較高的全色圖像（或柵格波段）與分辨率較低的多波段柵格數據集進行融合。最終生成一個具有全色柵格的高分辨率的多波段柵格數據集，該數據集中的兩個柵格完全重疊。
一些圖像公司可提供相同場景下的低分辨率多波段圖像和較高分辨率全色圖像， 此過程用于提高空間分辨率，并使用高分辨率單波段圖像提供視覺效果更佳的多波段圖像。
panchromatic image 全色影像，全色圖像

以上為全色銳化的解釋

By introducing deep neural networks, Huang et al. proposed a new pan-sharpening method for RS image preprocessing [28] that used a stacked modified sparse denoising AE (S-MSDA) to train the relationship between HR and LR image patches. 通過引入深度神經網絡，Huang等人提出了一種新的用于RS圖像預處理的全色銳化方法[28]，該方法采用堆疊修改的稀疏去噪自動編碼器(S-MSDA)來訓練HR（hign-resolution）和LR（low-resolution）圖像斑塊之間的關系。Similar to the structure of the sparse AE, S-MSDA is constructed by stacking a series of MSDAs. 與稀疏自動編碼器的結構類似，S-MSDA是通過堆疊一系列MSDAs（修正的稀疏去噪自動編碼器）來構建的。The MSDA is a modified version of the sparse denoising AE (SDA), which is obtained by combining sparsity and a denoising AE together. MSDA是稀疏去噪自動編碼器(SDA)的改進版，它是將稀疏性和去噪自動編碼器結合在一起得到的。 The SDA is trained to reconstruct a clean, repaired input from the corresponding corrupted version [64]. SDA被訓練為從相應的損壞版本中重建一個干凈的、修復的輸入。 Meanwhile, the modified version (i.e., the MSDA) takes the HR image patches and the corresponding LR image patches as clean data and corrupted data, respectively, and represents the relationship between them. 同時，修改版(即MSDA)將HR圖像斑塊和對應的LR圖像斑塊分別作為干凈數據和損壞數據，并表示它們之間的關系。 There is a key hypothesis that the HR and LR multispectral (MS) image patches have the same relationship as that between the HR and LR PAN image patches; thus, it is a learning-based method that requires a set of HR–LR image pairs for training. 有一個關鍵的假設，即HR和LR多光譜（MS）圖像斑塊與HR和LR PAN（全色）圖像斑塊之間具有相同的關系；因此，它是一種基于學習的方法，需要一組HR-LR圖像對進行訓練。 Since the HR PAN is already available, we have designed an approach to obtain its corresponding LR PAN. 由于HR PAN（全色）已經存在，因而我們設計了一種方法來獲取其對應的LR PAN（全色）。Therefore, we can use the fully trained DL network to reconstruct the HR MS image from the observed LR MS image.因此，我們可以使用完全訓練的DL網絡來從觀察到的LR MS圖像重建HR MS圖像。The experimental results demonstrated that the DL-based pan sharpening method outperforms the other traditional and state-of-the-art methods.實驗結果表明，基于DL的全色銳化方法優于其他傳統和最先進的方法。 The aforementioned methods are just two aspects of DL-based RS image preprocessing. 上述方法只是基于DL的RS圖像預處理的兩個方面。In fact, we can use the general framework to generate more DL algorithms for RS image-quality improvement for different applications. 事實上，我們可以利用該通用框架生成更多的DL算法，用于不同應用的RS圖像質量改進。

2.2 PIXEL-BASED CLASSIFICATION

Pixel-based classification is one of the most popular topics in the geoscience and RS community. 基于像素的分類是地球科學和RS界最熱門的話題之一。 Significant progress has been achieved in recent years, e.g., in the aspects of handcrafted feature description [65]–[68], discriminative feature learning [13], [69], [70], and powerful classifier designing [71], [72]. 近年來，在手工制作的特性描述[65]-[68]、判別性特征學習[13]、[69]、[70]和強大的分類器設計[71]、[72]等方面都取得了重大進展。 However, from the DL point of view, most of the existing methods can extract only shallow features of the original data (the classification step can also be treated as the top level of the network), which is not robust enough for the classification task. DL-based pixel classification for RS images involves constructing a DL architecture for the pixel-wise data representation and classification. 但從DL的角度來看，現有的方法大多只能提取原始數據的淺層特征(分類步驟也可視為網絡的頂層)，對分類任務的魯棒性不夠。基于DL的RS圖像像素分類涉及到構建一個DL架構來實現像素級數據的表示和分類。 By adopting DL techniques, it is possible to extract more robust and abstract feature representations and thus improve the classification accuracy. 通過采用DL技術，它可以提取出更健壯、更抽象的特征表示，從而提高分類精度。

The scheme of DL for RS image pixel-based classification consists of three main steps: 1) data input, 2) hierarchical DL model training, and 3) classification. 用于RS圖像像素分類的DL的方案主要包括三個步驟。1）數據輸入，2）層次化DL模型訓練，3）分類。 A general flow chart of this scheme is shown in Figure 2. 本方案的總體流程圖如圖2所示。In the first steps, the input vector could be the spectral feature, the spatial feature, or the spectral–spatial feature, as we will discuss
later. 在第一步中，輸入向量可以是光譜特征、空間特征或光譜-空間特征，我們將討論的是后者。 Then, for the hidden layers, a deep network structure is designed to learn the expected feature representation of the input data. 然后，針對隱藏層，設計深度網絡結構，學習輸入數據的預期特征表示。 In the related literature, both the supervised DL structures (e.g., the CNN [45]) and the unsupervised DL structures (e.g., the AEs [73]–[75], DBNs [29], [76], and other self-defined neurons in each layer [77]) are employed. 在相關文獻中，既采用了有監督的DL結構(如CNN[45])，也采用了無監督的DL結構(如AEs[73]-[75]、DBNs[29]、[76]和其他各層自定義神經元[77])。 The third step is the classification, which involves classification by utilizing the learned feature in the second step (the top layer of the DL network). 第三步是分類，即利用第二步（DL網絡的頂層）學習的特征進行分類。In general, there are two main styles of classifiers: 1) the hard classifiers, such as SVMs, which directly output an integer number as the class label of each sample [76], and 2) the soft classifiers, such as logistic regression, which can simultaneously fine-tune the whole pretrained network and predict the class label in a probability distribution manner [29], [73], [74], [78]. 一般來說，分類器主要有兩種風格：1）硬分類器，如SVM，直接輸出一個整數作為每個樣本的類標簽[76]；2）軟分類器，如邏輯回歸，可以同時對整個預訓練網絡進行微調，并以概率分布的方式預測類標簽[29]，[73]，[74]，[78]。

figure 2. 利用DL方法對RS圖像進行像素分類的一般框架。DL網絡的輸入可以分為
三類：光譜特征、空間特征、光譜空間特征。

2.2.1 SPECTRAL FEATURE CLASSIFICATION光譜特征分類

The spectral information usually contains abundant discriminative information.光譜信息通常包含豐富的判別信息。A frequently used and direct approach for RS image classification is spectral feature-based classification, i.e., image classification with only the spectral feature. RS圖像分類中經常使用的、直接的方法是基于光譜特征的分類，即只有光譜特征的圖像分類。Most of the existing common approaches for RS image classification are shallow in their architecture, such as SVMs and k-nearest neighbor (KNN). Instead, DL adopts a deep architecture to deal with the complicated relationships between the original data and the specific class label.現有常見的RS圖像分類方法大多是淺層架構，如SVMs和k-最近鄰（KNN）。相反，DL采用深層架構來處理原始數據和特定類標簽之間的復雜關系。

For spectral feature classification, the spectral feature of the original image data is directly deployed as the input vector. 對于光譜特征分類，直接部署原始圖像數據的光譜特征作為輸入向量。The input pixel vector is trained in the network part to obtain the robust deep feature representation, which is used as the input for the subsequent classification step. 在網絡部分對輸入像素向量進行訓練，得到魯棒的深度特征表示，作為后續分類步驟的輸入。The selected deep networks could be the deep CNN [45] and a stack of the AE [73], [75], [79], [80]. 選用的深度網絡可以是深度CNN[45]和自動編碼器 [73], [75], [79], [80]的疊加。In particular, Lin et al. adopted an AE plus SVMs and a stacked AE plus logistic regression as the network structure and classification layer to perform the classification task with shallow and deep representation, respectively. 其中，Lin等采用了自動編碼器加SVMs和疊加自動編碼器加邏輯回歸作為網絡結構和分類層，分別完成淺層和深層表示的分類任務。It is worth noting that, due to the deeper network structure and the fine-tuning step, the deep spectral representation achieved a better performance than the shallow spectral representation [73].值得注意的是，由于網絡結構較深、微調步驟較多，深光譜表示法取得了比淺光譜表示法更好的性能。

2.2.2 CLASSIFICATION WITH SPATIAL INFORMATION空間信息分類

Land covers are known to be continuous in the spatial domain, and adjacent pixels in an RS image are likely to belong to the same class.眾所周知，土地覆蓋物在空間域中是連續的，RS圖像中相鄰的像素可能屬于同一類別。 As indicated in many spectral–spatial classification studies, the use of the spatial feature can significantly improve the classification accuracy [81]–[83]. 在許多光譜-空間分類研究中表明，使用空間特征可以顯著提高分類精度[81]-[83]。However, traditional methods cannot extract robust deep feature representations due to their shallow properties. 然而，傳統方法由于其淺層特性，無法提取出穩健的深層特征表征。To address this problem, a number of DL-based feature-learning methods have been proposed to find a new way of extracting the deep spectral–spatial representation for classification [84].為了解決這個問題，人們提出了許多基于DL的特征學習方法，找到了一種用于分類的提取深度光譜—空間表示的新方法。

For a certain pixel in the original RS image, it is natural to consider its neighboring pixels to extract the spatial feature representation.對于原始RS圖像中的某一像素，自然要考慮它的相鄰像素，以提取空間特征表征。 However, due to the hundreds of channels along the spectral dimension of a hyperspectral image, the region-stacked feature vector will result in too large an input dimension. 然而，由于高光譜圖像的光譜維度有數百個通道，區域疊加的特征向量將導致輸入維度過大。As a result, it is necessary to reduce the spectral feature dimensionality before the spatial feature representation. 因此，在空間特征表示之前，有必要降低光譜特征維度。 PCA is commonly executed in the first step to map the data to an acceptable scale with a low information loss. 第一步通常執行PCA，在信息損失小的情況下，將數據映射到一個可接受的尺度。Then, in the second step, the spatial information
is collected by the use of a w#w (w is the size of window) neighboring region of every certain pixel in the original image [85]. 然后，在第二步中，空間信息是通過使用原始圖像中每一個像素的 w#w（w是窗口尺寸）鄰近區域（鄰域）來收集的。 After that, the spatial data is straightened into a one-dimensional vector to be fed into a DL network. Lin et al.[73] and Chen at al. [74] adopted the stacked AE as the deep network structure. 之后，將空間數據拉直為一維向量，送入DL網絡。Lin等[73]和Chen等[74]采用堆疊式自編碼器作為深度網絡結構。 When the abstract feature has been learned, the final classification step is carried out, which is similar to the spectral classification scheme. 當抽象特征學習完畢后，進行最后的分類步驟，這與光譜分類方案類似。

When considering a joint spectral and spatial feature-extraction and classification scheme, there are two mainstrategies to achieve this goal under the framework summarized in Figure 2. 當考慮光譜和空間特征結合的提取和分類方案時，在圖2總結的框架下，有兩個主要策略來實現這一目標。Straightforwardly, differing from the spectral–spatial classification scheme, the spectral and initial spatial features are combined together into a vector as the input of the DL network in a joint framework, as presented in the works [29], [53]–[55], [73], and [74].直觀地說，與光譜-空間分類方案不同的是，在聯合框架中，將光譜和初始空間特征一起組合成一個向量作為DL網絡的輸入，如[29]、[53]-[55]、[73]和[74]等著作中提出的.The preferred deep networks in these papers are SAEs and DBNs, respectively. Then, by the learned deep network, the joint spectral–spatial feature representation of each test sample is obtained for the subsequent classification task, which is the same as the spectral–spatial classification scheme described previously. 這些論文中首選的深度網絡分別是SAEs和DBNs。然后，通過學習的深度網絡，得到每個測試樣本的聯合光譜-空間特征表示，用于后續的分類任務，這與前面介紹的光譜-空間分類方案相同。The other approach is to address the spectral and spatial information of a certain pixel by a convolutional deep network, such as the CNNs [46], [47], [76], the convolutional AEs [78], and a particular defined deep network [78]. 另一種方法是通過卷積深層網絡來解決某個像素的光譜和空間信息，如CNNs[46]、[47]、[76]、卷積AEs[78]和特定定義的深層網絡[78]等。Moreover, there are a few hierarchical learning frameworks that take each step of operation (e.g., feature extraction, classification, and postprocessing) as a single layer of the deep network [86] ,[90]. 此外，有一些分層學習框架將每一步操作（如特征提取、分類和后處理）都作為深度網絡的單層[86] ，[90] 。We also regard them as the spectral–spatial DL techniques in this tutorial article.我們也把它們看作是本教程文章中的光譜-空間DL技術。

2.3 TARGET RECOGNITION

Target recognition in large HR RS images, such as ship, aircraft, and vehicle detection, is a challenging task due to the small size and large numbers of targets and the complex neighboring environments, which can cause the recognition algorithms to mistake irrelevant ground objects for target objects. 在大型HR RS（高分遙感）圖像中的目標識別，如艦船、飛機、車輛檢測等，由于目標體積小、數量多，且相鄰環境復雜，會導致識別算法將不相關的地面物體誤認為目標物體，是一項具有挑戰性的任務。However, objects in natural images are relatively large, and the environments in the local fields are not that complex compared to RS images, making the targets easier to recognize. 但是，自然圖像中的物體相對較大，與RS圖像相比，局部場中的環境沒有那么復雜，因此目標更容易識別。This is one of the main differences between detecting RS targets and natural targets. Although many studies have been undertaken, we are still lacking an efficient location method and robust classifier for target recognition in complex environments.這也是檢測RS目標與自然目標的主要區別之一。雖然我們已經進行了很多研究，但是對于復雜環境下的目標識別，我們仍然缺乏一種高效的定位方法和魯棒分類器。In the literature, Cai et al. [91] showed how difficult it is to segment aircraft from the background, and Chen et al. [30], [92] made great efforts in vehicle detection in HR RS images.在文獻中，Cai等[91]表明了從背景中分割飛機的難度，Chen等[30]、[92]在HR RS圖像中的車輛檢測方面做了很大努力。

The performance of target recognition in such a complex context relies on the features extracted from the objects. 在如此復雜的背景下，目標識別的性能依賴于從對象中提取的特征。DL methods are well suited for this task, as this type of algorithm can extract low-level features with a high frequency, such as edges, contours, and outlines of objects, whatever the shape, size, color, or rotation angle of the targets. DL方法很適合這項任務，因為這種類型的算法可以提取出頻率很高的低級特征，如物體的邊緣、輪廓和輪廓，無論目標的形狀、大小、顏色或旋轉角度如何。This type of algorithm can also learn hierarchical representations from the input images or patches, such as the parts of the objects that are compounded by the lower-level features, making recognition of RS targets discriminative and robust. 這類算法還可以從輸入的圖像或斑塊中學習分層表示，如物體的部分是由低層特征復合而成的，使得對RS目標的識別具有分辨性和魯棒性。 A number of these approaches achieved state-of-the-art performance in target recognition by use of a DL method [30], [48], [49], [52], [56], [93]–[96]. 其中一些方法通過使用DL方法，在目標識別方面取得了最先進的性能。

2.3.1 GENERAL DEEP-LEARNING FRAMEWORK OF REMOTE SENSING TARGET RECOGNITION 遙感目標識別的一般深度學習框架

The DL methods used in target recognition can be divided into two maincategories: unsupervised methods and supervised methods. 目標識別中使用的DL方法可以分為兩大類：無監督方法和有監督方法。 The unsupervised methods learn features from the input data without knowing the correlated labels or other supervisory information, while the supervised methods use the input data as well as the supervisory information attached to the input to discriminatively learn the feature representations. 無監督方法在不知道相關標簽或其他監督信息的情況下，從輸入數據中學習特征，而監督方法則利用輸入數據以及附加在輸入上的監督信息來辨別學習特征表征。However, both of these DL methods are utilized to learn features from the object images, and the learning processes can be unified into the same framework, as depicted in Figure 3.然而，這兩種DL方法都是利用從對象圖像中學習特征，學習過程可以統一到同一個框架中，如圖3所示。

The RS images are first preprocessed to subtract the mean and divide the variance, or to simply convert the images to gray images with only one channel. Other preprocessing techniques compute the gradient images [97] of the original image with a certain threshold [30]. 首先對RS圖像進行預處理，減去均值并劃分方差，或者干脆將圖像轉換為只有一個通道的灰色圖像。其他的預處理技術是計算原始圖像的梯度圖像[97]，有一定的閾值[30]。The second term of this general pipeline is extracting the object proposals, which is a bounding box locating the probable targets. 這個一般流程的第二項是提取對象建議，這是一個定位可能目標的邊界框。Following the process of selecting the proposals from the whole image, a simple feature extraction is conducted for each proposal or the whole image to extract the low-level descriptors that are invariant to shift, rotation, and scaling,
to some extent, such as SIFT [98], Gabor [99], and the histogram of oriented gradients (HOG) [97]. 接下來，從整幅圖像中選取提案后，對每幅提案或整幅圖像進行簡單的特征提取，提取出一定程度上對移位、旋轉和縮放不變的低級描述符，如SIFT[98]、Gabor[99]、定向梯度直方圖（HOG）[97]。Next, the middle-level feature representations can be generated by performing codebook learning on the learned descriptors. 接下來，可以通過對學習到的描述符進行代碼本學習來生成中間層次的特征表示。 This step is not essential, but using these low- or middle-level features usually outperforms merely using the raw pixels when learning hierarchical feature representations by the following deep neural networks. 這一步并不是必不可少的，但在通過以下深度神經網絡學習分層特征表示時，使用這些低級或中級特征的表現通常優于僅僅使用原始像素。

The deep neural networks such as the CNNs, sparse AEs, and DBNs are hierarchical models that can learn high-level feature representations in the deep layers automatically generated by the features learned in the shallow layers. CNNs、稀疏AEs和DBNs等深度神經網絡是分層模型，可以在深層學習由淺層中學習到的特征自動生成的高級特征表示。Having learned the discriminative and robust representations of the proposals, a classifier such as an SVM is trained with training samples composed of the representations of some data and the corresponding supervisory information. 在學習了提案的判別性和魯棒性表示后，用一些數據的表示和相應的監督信息組成的訓練樣本來訓練一個分類器，如SVM。When a new proposal is generated from a new image, this framework can automatically learn the high-level features from the raw image, and then classification is undertaken by the well-trained classifier to tell whether the proposal is the target or not. 當從一張新的圖像中生成一個新的提案時，這個框架可以從原始圖像中自動學習高級特征，然后由訓練好的分類器進行分類，判斷提案是否是目標。

FFIGURE 3. 利用DL方法進行目標識別的一般框架。深度網絡學習到的高級特征被發送到要分類的分類器（或直接由深度網絡對監督網絡進行分類）。

2.3.2 SAMPLE SELECTION PROPOSALS樣本選擇的建議

To choose the most accurate area that exactly contains the target, a number of proposals should be extracted from the input image. 為了選擇準確包含目標的最精確區域，應從輸入圖像中提取一些建議。Each proposal is usually a bounding box covering an object that probably contains the target. 每個提案通常是一個邊界框，覆蓋一個可能包含目標的對象。 The most satisfactory case is that the target is in the center of the bounding box, and the bounding box can just cover the edge of the object. 最滿意的情況是，目標在邊界框的中心，邊界框可以剛好覆蓋物體的邊緣。

There are different ways of selecting the proposals. The baseline technique is the sliding window method [100], which slides the bounding box over the whole image with a small stride to generate a number of proposals. 有不同的選擇提案的方法。基線技術是滑動窗口法[100]，它將邊界框在整個圖像上以很小的步幅滑動，以產生若干提案。The sliding window technique is accurate and will not miss any possible proposals that may exactly contain the target, yet it is slow and burdens the subsequent feature-learning algorithms and classifiers, especially when there are quite a lot of objects in an image (e.g., in RS images). 滑動窗口技術是準確的，不會漏掉任何可能完全包含目標的提案，然而它的速度很慢，給后續的特征學習算法和分類器帶來了負擔，特別是當圖像中存在相當多的物體時（如RS圖像中）。 Other methods have been proposed to solve this problem, e.g., Chen et al. [30] proposed an object-location technique that can discover the coarse locations of the targets, and hence can greatly reduce the number of proposals. 還有一些方法被提出來解決這個問題，例如，Chen等[30]提出了一種對象定位技術，可以發現目標的粗位置，因此可以大大減少提案的數量。 The search efficiency of this method is more than 20 times the baseline sliding window method. Tang et al. [101] proposed a coarse ship location technique that can extract the candidate locations of the targets with little decrease in accuracy. 該方法的搜索效率是基線滑動窗口法的20倍以上。 Tang等人[101]提出了一種粗船定位技術，可以在精度下降不大的情況下提取目標的候選位置。

2.3.3 LOW-TO MIDDLE-LEVEL FEATURE LEARNING低級到中級的特征學習

Low-level features, which have the ability to handle variations in terms of intensity, rotation, scale, and affine projection, are utilized to characterize the local region of the key points in each image or patch. 低級特征，具有處理強度、旋轉、尺度和仿射投影等方面變化的能力，被用來描述每個圖像或斑塊中關鍵點的局部區域。Han et al. [94] utilized SIFT descriptors to represent the image patches, which made the subsequent training process of the DBMs easier. Dalal and Triggs [97] proposed a method for object detection using the HOG.Han等人[94]利用SIFT描述符來表示圖像斑塊，這使得DBMs的后續訓練過程更加容易。Dalal和Triggs[97]提出了一種利用HOG進行目標檢測的方法。

The low-level descriptors can boost the feature-learning performance of the deep networks. 低級描述符可以提升深度網絡的特征學習性能。However, they catch only limited local spatial geometric characteristics, which can lead to poor classification or detection performance when they are directly used to describe the structural contents of image patches. 然而，它們只能捕捉到有限的局部空間幾何特征，當它們直接用于描述圖像斑塊的結構內容時，會導致分類或檢測性能不佳。 To tackle this problem, some work has been done to learn codebooks that are used to encode the local descriptors and generate middle-level feature representations and to alleviate the unrecoverable loss of discriminative information. 為了解決這個問題，已經做了一些工作，學習代碼本，用于編碼局部描述符和生成中間層特征表示，并緩解不可恢復的判別信息的損失。For instance, Han et al. [94] applied the locality-constrained linear coding model [102] to encode the SIFT descriptors into the image patch representation. 例如，Han等[94]應用局域性約束的線性編碼模型[102]將SIFT描述符編碼到圖像斑塊表示中。

2.3.4 TRAINING THE DEEP-LEARNING NETWORKS

Although the middle-level features are extracted based on the low-level local descriptors to obtain the structural information and preserve the local relevance of elements in the local region, they cannot provide enough strong description and generalization abilities for object detection when confronted with objects and backgrounds with a large variance.雖然基于低級局部描述符提取中級特征，獲取結構信息，保留局部區域內元素的局部相關性，但在面對差異較大的物體和背景時，不能為物體檢測提供足夠強的描述和泛化能力。To better understand the complexity of the environments in an image, better descriptors should be utilized. The DL methods can handle complex ground objects with large variance, as the features learned by the deep neural networks can be highly abstract, which makes them invariant to relatively large deformations, including different shapes, sizes, and rotations, and discriminative to some objects that belong to different categories but resemble each other in some other aspect, such as white targets on a white background. 為了更好地理解圖像中環境的復雜性，應該利用更好的描述器。由于深度神經網絡學習的特征可以是高度抽象的，這使得DL方法可以處理復雜的、具有較大差異的地面物體，這就使得DL方法對比較大的變形，包括不同的形狀、大小和旋轉都是不變的，對一些屬于不同類別的物體，但在其他方面又很相似的物體，如白色背景上的白色目標，也有一定的分辨能力。Generally speaking, the DL methods used in target recognition in RS images can be divided into two categories: 1) the supervised DL algorithms and 2) the unsupervised DL algorithms.一般來說，RS圖像中目標識別中使用的DL方法可以分為兩類。1）有監督的DL算法；2）無監督的DL算法。

2.3.5 SUPERVISED METHODS

There are two typical supervised DL methods for target recognition: the CNN and the multilayer perceptron (MLP) [103]. 目標識別有兩種典型的監督DL方法：CNN和多層感知器（MLP）[103]。 The CNNs are hierarchical models that transform the input image or image patch into layers of feature maps, which are high-level discriminative features representing the original input data. CNN是一種分層模型，它將輸入圖像或圖像斑塊轉化為層層特征圖，這些特征圖是代表原始輸入數據的高級判別特征。For the MLP model, the input image or patch should be reshaped into a vector. Then, after the transformation of each fully connected layer, the final feature representation can be generated. 對于MLP模型，應將輸入的圖像或斑塊重塑為一個矢量。然后，經過每個全連接層的變換，就可以生成最終的特征表示。The final features are then sent to the classification layer to generate the label of the input image. Both types of supervised networks transform the input image into a two-dimensional vector for a one-class object detection. 最后的特征被送到分類層，生成輸入圖像的標簽。 這兩種類型的監督網絡都是將輸入圖像轉化為二維向量，進行一類對象檢測。This vector indicates the predicted label (whether the input candidate is the target or not, or the probability of the proposal being the target). 這個向量表示預測的標簽（投入的候選人是否是目標，或者說提案是目標的概率）。In the training stage, to learn the weights or kernels, the supervised networks are trained with the training samples composed of positive samples that contain the target and negative samples that do not contain the target. In the testing stage, the proposals extracted from a new RS image are processed by the models and attached with a probability y. 在訓練階段，為了學習權重或內核，監督網絡的訓練樣本由包含目標的正樣本和不包含目標的負樣本組成。在測試階段，從新的RS圖像中提取的建議被模型處理并以概率y附加。The candidates then considered to contain the target are selected by a given empirical threshold or other criteria.然后，通過給定的經驗閾值或其他標準選擇被認為包含目標的候選者。

Although the CNN has shown robustness to distortion, it only extracts features of the same scale and, therefore, cannot tolerate a large-scale variance of objects. 雖然CNN表現出了對失真的魯棒性，但它只能提取相同尺度的特征，因此，不能容忍對象的大規模差異。 When it comes to RS images that have a large variance in the backgrounds and objects, training a CNN that extracts multiscale feature representations is necessary for a better detection accuracy. 當涉及到背景和物體差異較大的RS圖像時，為了提高檢測精度，訓練一個能提取多尺度特征表示的CNN是必要的。Chen et al.[30] proposed a hybrid deep neural network (HDNN) by dividing the maps of the final convolutional layer and the max-pooling layer of the deep neural network into multiple blocks of variable receptive field sizes or max-pooling field sizes to enable the HDNN to extract variable-scale features for detecting the RS objects. Chen等[30]提出了一種混合深度神經網絡(HDNN)，將深度神經網絡的最后卷積層和最大池化層的地圖劃分為多個可變接收場大小或最大池化場大小的區塊，使HDNN能夠提取可變尺度的特征來檢測RS對象。The input of the HDNN with L convolutional layers is a gray image. 具有L個卷積層的HDNN的輸入是一幅灰色圖像。The image is filtered by the filters in the first convolutional layers
to get the feature maps
, which are then subsampled by the first max-pooling layer to select the representative features as well as reduce the number of parameters to be processed. 圖像經過第一卷積層中的濾波器過濾后，得到特征圖 然后由第一個最大池化層對其進行子采樣，以選擇具有代表性的特征，并減少需要處理的參數數量。After transferring the L layers’ activations or feature maps,the final convolutional feature maps of the Lth layer
are generated. 轉移L層的激活或特征圖后，最終得到第L層的卷積特征圖。 In the architecture of the conventional CNNs, the final layer is followed by some fully connected layers and finally the classification layer. 在傳統CNN的架構中，最后一層是一些完全連接的層，最后是分類層。
However, this kind of feature-processing method does not make full use of the features and the filters. 但是，這種特征處理方法并沒有充分利用特征和過濾器。 The receptive field size of each convolutional layer is fixed, and thus it cannot extract multiscale features. 每個卷積層的接受場尺寸是固定的，因此它無法提取多尺度特征。 However, there are still rich features in the final convolutional layer that can be learned and transformed into more discriminative representations. 然而，在最后的卷積層中仍有豐富的特征，可以學習并轉化為更有辨別力的表征。One way to better utilize the rich features is to increase the depth of the convolutional layers, which may, however, introduce a huge amount of computational burden when training the model. 為了更好地利用豐富的特征，一種方法是增加卷積層的深度，然而，這可能會在訓練模型時帶來巨大的計算負擔。Another way is to use a multiscale receptive field size that can train filters with different sizes and generate multiscale feature maps.另一種方法是使用多尺度的接受場尺寸，可以訓練不同尺寸的濾波器，并生成多尺度的特征圖。

In the HDNN, the last layer’s feature maps are divided into T blocks with filter sizes of ， respectively. 在HDNN中，最后一層的特征圖被劃分為T塊，濾波器大小分別為。The ith block covers
feature maps of the final convolutional layer. Then the activation propagation between the last two convolutional layers can be formulated as where
denotes the block of the last feature maps, denotes the filters of the corresponding block, and
denotes the activation function. 第i個塊覆蓋了最后卷積層的ni個特征圖。那么最后兩個卷積層之間的激活傳播可以表述為，在這里Bt表示最后特征圖的第t個塊，ft表示對應塊的濾波器，表示激活函數。

Having learned the multiscale feature representations to form the final convolutional layer, an MLP network is used to classify the features. 在學習了多尺度特征表示以形成最后的卷積層后，使用MLP網絡對特征進行分類。The output of the HDNN is a two-node layer, which indicates the probability of whether the input image patch contains the target.HDNN的輸出是一個雙節點層，它表示輸入圖像斑塊是否包含目標的概率。 Some of the vehicle-detection results are referred to in [30], from which it can be concluded that, although there are a number of vehicles in the scene, the modified CNN model can successfully recognize the precise location of most of the targets, indicating that the HDNN has learned fairly discriminative feature representations to recognize the objects.在[30]中提到了一些車輛檢測的結果，從中可以得出結論，雖然場景中存在一些車輛，但修改后的CNN模型可以成功識別大部分目標的精確位置，說明HDNN已經學會了相當有分辨力的特征表示來識別物體。

2.3.6 UNSUPERVISED METHODS

Although the supervised DL methods like the CNN and its modified models can achieve acceptable performances in target recognition tasks, there are limitations to such methods since their performance relies on large amounts of labeled data, while, in RS image data sets, high-quality images with labels are limited.雖然像CNN及其修改后的模型這樣的監督DL方法可以在目標識別任務中獲得可接受的性能，但由于其性能依賴于大量的標簽數據，而在RS圖像數據集中，帶有標簽的高質量圖像是有限的，因此這類方法存在局限性。It is therefore necessary to recognize the targets with a few labeled image patches while learning the features with the unlabeled images. 因此，在用未標記的圖像學習特征的同時，有必要用一些標記的圖像斑塊來識別目標。

Unsupervised feature-learning methods are models that can learn feature representations from the patches with no supervision. 無監督的特征學習方法是指可以在沒有監督的情況下從斑塊中學習特征表示的模型。 Typical unsupervised feature-learning methods are RBMs, sparse coding, AEs, k-means clustering, and the Gaussian Mixture Model [104]. 典型的無監督特征學習方法有RBMs、稀疏編碼、AEs、k-means聚類和高斯混合物模型[104]。 All of these shallow feature-learning models can be stacked to form deep unsupervised models, some of which have been successfully applied to recognizing RS scenes and targets. 所有這些淺層的特征學習模型都可以疊加起來形成深層的無監督模型，其中一些模型已經成功應用于識別RS場景和目標。For instance, the DBN generated by stacking RBMs has shown its superiority over conventional models in the task of recognizing aircraft in RS scenes [105].例如，在RS場景中識別飛機的任務中，通過堆疊RBMs生成的DBN已經顯示出比傳統模型的優越性[105]。

The DBN is a deep probabilistic generative model that can learn the joint distribution of the input data and its ground truth. DBN是一個深度概率生成模型，它可以學習輸入數據及其真值的聯合分布。 The general framework of the DBN model is illustrated in Figure 4. DBN模型的總體框架如圖4所示。The weights of each layer are updated through layer-wise training using the CD algorithm, i.e., training each layer separately. 通過使用CD算法進行逐層訓練，更新各層的權重，即分別對各層進行訓練。 The joint distribution between the observed vector x and the L hidden layers is ， where is a conditional distribution for the visible units conditioned on the hidden units of the RBM at level k, and is the visible–hidden joint distribution in the top-level RBM. 觀察到的向量x與L個隱藏層之間的聯合分布為， 這里是可見單元的條件分布，條件是RBM的隱藏單元在水平k上的條件分布，并且是頂層RBM中可見-隱藏的聯合分布。Some aircraft detection results from large airport scenes can be seen in [105], from which we can see that most aircrafts with different shapes and rotation angles have been detected. 在[105]中可以看到一些大型機場場景下的飛機檢測結果，從中我們可以看到大部分形狀和旋轉角度不同的飛機都被檢測到。

2.4 SCENE UNDERSTANDING

Satellite imaging sensors can now acquire images with a spatial resolution of up to 0.41 m. 衛星成像傳感器現在可以獲得空間分辨率高達0.41米的圖像。These images, which are usually called very high-resolution (VHR) images, have abundant spatial and structural patterns. 這些圖像通常被稱為甚高分辨率（VHR）圖像，具有豐富的空間和結構模式。 However, due to the huge volume of the image data, it is difficult to directly access the VHR data containing the scenes of interest. 但由于圖像數據量巨大，很難直接獲取包含感興趣場景的VHR數據。 Due to the complex composition and large number of land-cover types, efficient representation and understanding of the scenes from VHR data have become a challenging problem, which has drawn great interest in the RS field. 由于土地覆蓋類型構成復雜、數量眾多，從VHR數據中高效地表示和理解場景成為一個具有挑戰性的問題，這引起了RS領域的極大興趣。

Recently, a lot of work in RS scene understanding has been proposed that focuses on learning hierarchical internal feature representations from image data sets [50], [106]. 最近，在RS場景理解方面提出了很多工作，主要是從圖像數據集中學習層次化的內部特征表征。Good internal feature representations are hierarchical. In an image, pixels are assembled into edgelets, edgelets into motifs, motifs into parts, and parts into objects.好的內部特征表示是分層的。在圖像中，像素被組合成邊緣小點，邊緣小點被組合成圖案，圖案被組合成部件，部件被組合成對象。 Finally, objects are assembled into scenes [107], [108]. This suggests that recognizing and analyzing scenes from VHR images should have multiple trainable feature-extraction stages stacked on top of each other, and we should learn the hierarchical internal feature representations from the image. 最后，將對象組裝成場景[107]，[108]。這說明從VHR圖像中識別和分析場景應該有多個可訓練的特征提取階段疊加在一起，我們應該從圖像中學習層次化的內部特征表示。

2.4.1 UNSUPERVISED HIERARCHICAL FEATURE-LEARNING-BASED METHODS

As indicated in the “General Framework” section, there is some work that focuses on unsupervised feature-learning techniques for RS images scene classification, such as sparse coding [58], k-means clustering [60], [109], and topic model [110], [111]. 如“一般框架”部分所示，有一些工作是針對RS圖像場景分類的非監督特征學習技術，如稀疏編碼[58]，k-means聚類[60]，[109]，主題模型[110]，[111]。These shallow models could be considered to stack into deep versions in a hierarchical manner [31], [106]. Here, we summarize an overall architecture of the unsupervised feature-learning framework for RS scene classification.這些淺層模型可以考慮以分層的方式堆疊成深層版本[31]，[106]。在這里，我們總結了RS場景分類的無監督特征學習框架的整體架構。 As depicted in Figure 5, the framework consists of four parts: 1) patch extraction, 2) feature extraction, 3) feature representation, and 4) classification. 如圖5所示，該框架由四個部分組成：1）斑塊提取，2）特征提取，3）特征表示，4）分類。

2.4.2 PATCH EXTRACTION

In this step, the patches are extracted from the image by using random sampling or another method. Each patch has a dimension of w×w and has three bands (R, G, and B), with w referred to as the receptive field size.在這一步驟中，利用隨機采樣或其他方法從圖像中提取出斑塊。每個斑塊的維度為w×w，有三個波段（R、G、B），w稱為接受場尺寸。Each w×w patch can be represented as a vector in of the pixel intensity values, with N=w×w×3. 每個w×w斑塊可以用表示像素強度值的向量，N=w×w×3。A data set of m sampled patches can thus be constructed. Then various features can be extracted from the image patch to construct the training and test data, such as raw pixel or other low-level features (e.g., color histogram, local binary pattern, and SIFT). 因此，可以構建一個由m個采樣斑塊組成的數據集。然后可以從圖像斑塊中提取各種特征來構建訓練和測試數據，如原始像素或其他低級特征(如顏色直方圖、局部二元模式和SIFT)。The patch feature composed of the training feature and test feature is then fed into an unsupervised feature-learning method that is used for the unsupervised learning of the feature extractor W. 然后將訓練特征和測試特征組成的斑塊特征輸入到無監督特征學習方法中，用于特征提取器W的無監督學習。

2.4.3 FEATURE EXTRACTION

After the unsupervised feature learning, the features can be extracted from the training and test images using the learned feature extractor W, as illustrated in Figure 6.在無監督的特征學習之后，可以使用學習到的特征提取器W從訓練和測試圖像中提取特征，如圖6所示。 Given a w-×-w image patch, we can now extract a representative for that patch by using the learned feature extractor 。給定一個w-×-w的圖像斑塊，我們現在可以通過使用學習到的特征提取器來提取該斑塊的代表 。We then define a new representation of the entire image using the feature extractor function with each image. 然后，我們使用每個圖像的特征提取函數 對整個圖像定義一個新的表示。Specifically, given an image of n-×-n pixels (with three channels: R, G, and B), we can define an ( n-w+1）-×-（n-w+1） representation (with K channels) by computing the representative for each w-×-w subpatch of the input image. 具體來說，給定一幅n-×-n像素的圖像（有三個通道：R，G，B），我們可以通過計算輸入圖像的每個w-×-w子塊的代表來定義一個( n-w+1）-×-（n-w+1）的代表（有K個通道）。 More formally, we denote as the K-dimensional feature extracted from location i, j of the input image. 更正式地，我們表示是從輸入圖像的位置i，j中提取的K維特征。For computational efficiency, we can also convolute our n-×-n image with a step size (or stride) greater than 1 across the image. 為了提高計算效率，我們也可以對n-×-n圖像進行卷積，整個圖像的步長（或步幅）大于1。

FIGURE 6.特征提取采用w-×-w特征提取器，步幅為s，我們首先提取w-×-w的斑塊，每個斑塊之間用s個像素隔開。然后將它們映射到K維的特征向量上，形成新的圖像表示。然后將這些向量匯集到圖像的16個象限，形成分類的特征向量。

2.4.4 FEATURE REPRESENTATION

After the feature extraction, the new feature representation for an image scene will usually have a very high dimensionality. For computational efficiency and storage volume, it is standard practice to use max-pooling or another strategy to reduce the dimensionality of the image representation [112], [36]. 在特征提取后，圖像場景的新特征表示通常會有很高的維度。為了計算效率和存儲量，標準的做法是使用最大池化或其他策略來降低圖像表示的維度[112]，[36]。
For a stride of s = 1, the feature mapping produces an ( n-w+1）-×-( n-w+1）-×-K representation. 對于s=1的步長，特征映射產生( n-w+1）-×-( n-w+1）-×-K表示。We can reduce this by finding the maximum over local regions of ，as done previously. This procedure is commonly used in computer vision,
with many variations, as well as in deep feature learning. 我們可以通過尋找的局部區域的最大值來減少這個問題，就像之前做的那樣。這個過程在計算機視覺中常用，有很多變化，在深度特征學習中也是如此。

2.4.5 CLASSIFICATION

Finally, the extracted feature is combined with the SVM or another classifier to predict the scene label. 最后，將提取的特征與SVM或其他分類器相結合，預測場景標簽。 However, most methods for unsupervised feature learning produce filters that operate either on intensity or color information.然而，大多數無監督特征學習的方法都會產生對強度或顏色信息進行操作的濾波器。Vladimir [113] proposed a quaternion PCA and k-means combined approach for unsupervised feature learning that makes joint encoding of the intensity and color information possible. Vladimir[113]提出了一種四元PCA和k-means相結合的無監督特征學習方法，使得強度和顏色信息的聯合編碼成為可能。In addition, Cheriyadat [58] introduced a variant of sparse coding that combines local SIFT-based feature descriptors to generate a new sparse representation, producing an excellent classification accuracy. 此外，Cheriyadat[58]引入了一種稀疏編碼的變體，將基于SIFT的局部特征描述符結合起來，生成新的稀疏表示，產生了優秀的分類精度。The sparse AE-based method also produces excellent performance.基于稀疏AE的方法也會產生優異的性能。 In [31], Zhang et al. proposed a saliency-guided sparse AE method to learn a set of feature extractors that are robust and efficient, proposing a saliency-guided sampling strategy to extract a representative set of patches from a VHR image so that the salient parts of the image that contain the representative information in the VHR image can be explored, which differs from the traditional random sampling strategy. 在[31]中，Zhang等人提出了一種顯著性引導的稀疏AE方法，以學習一組穩健高效的特征提取器，提出了一種顯著性引導的采樣策略，從VHR圖像中提取一組有代表性的斑塊，從而可以探索出VHR圖像中包含代表性信息的突出部分，這與傳統的隨機采樣策略不同。They also explored the new dropout technique in the feature-learning procedure to reduce data overfitting [114]. The extracted feature generated from the learned feature extractors can characterize a complex scene very well and can produce an excellent classification accuracy.他們還在特征學習過程中探索了新的dropout技術，以減少數據過擬合[114]。通過學習的特征提取器產生的提取特征可以很好地描述復雜場景的特征，可以產生很好的分類精度。

2.4.6 SUPERVISED HIERARCHICAL FEARTURE-LEARNING-BASED METHODS

Before 2006, it was believed that training deep supervised neural networks was too difficult to perform (and indeed did not work). The first breakthrough in training happened in Geoff Hinton’s lab with an unsupervised pretraining by RBMs, as discussed in the previous subsection. 在2006年之前，人們認為訓練深度監督神經網絡太難了，無法進行（事實上也沒有用）。訓練方面的第一次突破發生在Geoff Hinton的實驗室，由RBMs進行無監督的預訓練，如前一小節所述。 However, more recently, it was discovered that one could train deep supervised networks by proper initialization, just large enough for the gradients to flow well and the activations to convey useful information.然而，最近發現，人們可以通過適當的初始化來訓練深度監督網絡，只要足夠大，梯度就能很好地流動，激活就能傳遞有用的信息。 These good results with the pure supervised training of deep networks seem to be especially apparent when large quantities of labeled data are available.當有大量的標簽數據時，深度網絡的純監督訓練的這些良好效果似乎特別明顯。

In the early years after 2010, based on the latent Dirichlet allocation (LDA) model [115], various supervised hierarchical feature-learning methods have been proposed in the RS community [116]–[120]. 2010年以后的初期，基于潛伏的Dirichlet分配（LDA）模型[115]，RS界提出了各種監督的分層特征學習方法[116]-[120]。LDA is a generative probabilistic graphical model for independent collections of discrete data and is a three-level hierarchical model, in which the documents inside a corpus are represented as random mixtures over a set of latent variables called topics. LDA是一種針對離散數據獨立集合的生成性概率圖形模型，是一種三層層次模型，其中，語料庫內的文檔被表示為一組稱為主題的潛變量上的隨機混合物。Each topic is in turn characterized by a distribution over words. The LDA model captures all of the important information contained in a corpus by considering only the statistics of the words. 每一個主題又由一個詞的分布來描述。LDA模型只考慮詞的統計量，就能捕捉到語料庫中的所有重要信息。The contextual relationships are neglected due to the Bayesian assumption. For this reason, LDA iscategorized as a bag of words model. Its main characteristic is based on the words’ exchangeability. 由于貝葉斯假設，上下文關系被忽略了。為此，LDA被歸類為詞袋模型。其主要特點是基于詞的可交換性。The LDA-based supervised hierarchical feature-learning approaches have been shown to generate excellent hierarchical feature representations for RS scene classification. 基于LDA的監督分層特征學習方法已經被證明可以為RS場景分類生成優秀的分層特征表示。

In fact, the LDA-based models are still not deep enough compared to the other techniques in the DL family. More recently, a few pure DL methods have been proposed for RS image scene understanding based on CNNs [121]. 事實上，與DL家族的其他技術相比，基于LDA的模型仍然不夠深入。最近，有人提出了一些基于CNN的RS圖像場景理解的純DL方法[121]。Zhang et al. proposed in detail a gradient-boosting random convolutional network framework for RS scene classification that can effectively combine many deep neural networks [50]. Zhang等人詳細地提出了一種用于RS場景分類的梯度提升隨機卷積網絡框架，可以有效地結合多種深度神經網絡[50]。Marmanis et al. considered a pretrained CNN by the ImageNet challenge and exploited it to extract an initial set of representations for earth observation classification [122]. Marmanis等人通過ImageNet挑戰賽考慮了一個預訓練的CNN，并利用它提取了一個初始表征集，用于地球觀測分類[122]。Hu et al. investigated how to transfer features from the ex-isting successfully pretrained CNNs for RS scene classification [123]. Luus et al. suggested a multiscale input strategy for multiview DL with the aid of convolutional layers to shift the burden of feature determination from hand-engineering to a deep CNN [124]. Hu等人研究了如何將現有成功預訓練的CNN的特征轉移到RS場景分類中[123]。Luus等人提出了一種借助卷積層的多尺度輸入策略，用于多視角DL，將特征確定的負擔從手工工程轉移到深度CNN上[124]。These advanced supervised DL methods all outperform the state-of-the-art methods with the various RS scene classification data sets.這些先進的監督DL方法在各種RS場景分類數據集上的表現都優于最先進的方法。

五，EXPERIMENTS AND ANALYSIS

In this section, we present some experimental results on the DL algorithms for RS data scene understanding that we believe can bring the most significant improvement compared to existing methods. Due to page limitations, experimental results relating to RS image preprocessing, pixel-based classification, and target recognition can be found within the papers in the reference list. 在本節中，我們將介紹一些關于RS數據場景理解的DL算法的實驗結果，我們認為這些算法與現有的方法相比能夠帶來最顯著的改進。由于篇幅限制，有關RS圖像預處理、基于像素的分類和目標識別的實驗結果可以在參考文獻列表中的論文中找到。

FIGURE 7, 與加州大學梅西分校數據集中的21個土地利用類別相關的圖像示例。1）農業，2）飛機，3）棒球。鉆石、4）海灘、5）建筑物、6）小樹叢、7）密集住宅、8）森林、9）高速公路、10）高爾夫球場、11）港口、12）十字路口。 13）中型住宅、14）移動房屋公園、15）立交橋、16）停車場、17）河道、18）跑道、19）稀疏住宅、20）儲罐 和21）網球場。

The first data set chosen for investigation is the well-known University of California (UC) Merced data set [125]. Figure 7 shows a few example images representing various RS scenes that are included in this data set, which contains 21 challenging scene categories with 100 image samples per class. 第一個選擇的調查數據集是著名的加州大學（UC）梅西數據集[125]。圖7顯示了代表各種RS場景的幾個示例圖像，該數據集包含21個具有挑戰性的場景類別，每類100個圖像樣本。 Following the experimental setup in [58], we randomly selected 80% of the samples from each class for training and set the remaining images for testing. For this data set, we compared our proposed supervised DL method, i.e., the random convolutional network (RCNet) [50], with the spatial pyramid matching kernel (SPMK) method [112], the SIFT + sparse coding (SSC) approach described in [58], and our unsupervised feature-learning method, i.e., the saliency-guided sparse AE (SSAE) method that was previously proposed in [31]. 按照[58]中的實驗設置，我們從每個類中隨機選取80%的樣本進行訓練，并設置剩余的圖像進行測試。 對于這個數據集，我們將我們提出的監督DL方法，即隨機卷積網絡（RCNet）[50]，與空間金字塔匹配內核（SPMK）方法[112]、[58]中描述的SIFT+稀疏編碼（SSC）方法，以及我們的無監督特征學習方法，即之前在[31]中提出的顯著性引導的稀疏 AE（SSAE）方法進行了比較。 For the RCNet algorithm, we trained the RCNet function using stochastic gradient descent with a batch size of 64, a momentum of 0.9, a weight decay of 0.0005, and a learning rate of 0.01. In addition, we trained each RCNet for roughly 500 cycles with the whole training set. All of these experiments were run on a personal computer (PC) with a single Intel core i7 central processing unit, an NVIDIA Titan graphics processing unit, and 6-GB memory. 對于RCNet算法，我們使用隨機梯度下降訓練RCNet函數，批次大小為64，動量為0.9，權重衰減為0.0005，學習率為0.01。 此外，我們用整個訓練集對每個RCNet進行了大約500次循環訓練。所有這些實驗都是在一臺個人計算機（PC）上運行的，該計算機擁有一個英特爾核心i7中央處理單元、一個NVIDIA Titan圖形處理單元和6-GB內存。The operating system was Windows 7, and the implementation environment was under MATLAB 2014a with a CUDA kernel. We compared the reported classification performances with the challenging UC Merced data set, and, among the four strategies we compared, the supervised DL method RCNet produced the best performance, as shown in Table 1.操作系統為Windows 7，實現環境為MATLAB 2014a下，帶有CUDA內核。我們將報告的分類性能與具有挑戰性的UC Merced數據集進行比較，在我們比較的四種策略中，監督DL方法RCNet產生的性能最好，如表1所示。

The other image data set was constructed from a large satellite image that was acquired from Google Earth of Sydney, Australia. The spatial resolution of the image was approximately 1.0 m.另一個圖像數據集是從谷歌地球上獲得的一張澳大利亞悉尼的大型衛星圖像構建的。圖像的空間分辨率約為1.0米。 The large image to be annotated was of 7,849 × 9,073 pixels, as shown in Figure 8. There were eight classes of training images: residential, airplane, meadow, rivers, ocean, industrial, bare soil, and runway. 需要注釋的大圖像為7 849×9 073像素，如圖8所示。訓練圖像有8類：住宅、飛機、草地、河流、海洋、工業、裸土和跑道。Figure 8 shows some examples of such images. This data set consisted of not only the eight defined classes, but also some other classes that had not been learned such as the bridges and the main roads. 圖8顯示了這類圖像的一些例子。 這個數據集不僅包括8個定義的類，還包括一些其他尚未學習的類，如橋梁和主要道路。We manually labeled part of the image to obtain a subregion image data set, in which each subregion was of the size of 128 × 128,whereby each subimage was supposed to contain a certain scene. 我們對部分圖像進行人工標注，得到一個子區域圖像數據集，其中每個子區域的大小為128×128，每個子圖像應該包含一定的場景。The training set for each class contained 25 samples of the labeled images for each class, while the remaining images were used for testing, as shown in Table 2.每個類的訓練集包含了每個類的25個標簽圖像樣本，而剩余的圖像用于測試，如表2所示。

For the Sydney data set, we trained the RCNet function using stochastic gradient descent with a batch size of 32, a momentum of 0.9, a weight decay of 0.0005, and a learning rate of 0.01. 對于悉尼數據集，我們使用隨機梯度下降法訓練RCNet函數，批次大小為32，動量為0.9，權重衰減為0.0005，學習率為0.01。We trained each RCNet for roughly 800 cycles with the whole training set. The PC environment was the same as previously mentioned. We also compared the final classification accuracies for RCNet and the traditional methods. 我們用整個訓練集對每個RCNet進行了大概800個周期的訓練。 PC環境和前面提到的一樣。我們還比較了RCNet和傳統方法的最終分類精度。Table 3 shows the average overall accuracies for the four methods. The results confirm that using the supervised DL method is an efficient way to increase the RS scene classification accuracy.表3顯示了四種方法的平均總體精度。結果證實，使用監督DL方法是提高RS場景分類精度的有效途徑。

FIGURE 8. (a)用于圖像標注的整幅圖像 (b)圖像地面真值 ? 與八種土地利用相關聯的實例圖像,從圖像中分類。1）跑道，2）飛機，3）住宅，4）河流，5）海洋，6）草地，7）工業，8）裸土。

六，CONCLUSIONS AND FUTURE WORK

In this technical tutorial, we have systematically reviewed the state-of-the-art DL techniques in RS data analysis. The DL techniques were originally rooted in machine-learning fields for classification and recognition tasks, and they have only recently appeared in the geoscience and RS community.在本技術教程中，我們系統地回顧了RS數據分析中最先進的DL技術。DL技術最初植根于分類和識別任務的機器學習領域，最近才出現在地球科學和RS界。From the four perspectives of image preprocessing, pixel-based classification, target recognition, and scene understanding, we have found that DL techniques have had significant successes in the areas of target recognition and scene understanding, i.e., areas that have been widely accepted as challenges in recent decades in the RS community because such applications require us to abstract the high-level semantic information from the bottom-level features (usually the raw pixel representation), while the traditional RS methods of feature describing feature extraction classification are shallow models, with which it is extremely difficult or impossible to uncover the high-level representation. 從圖像預處理、基于像素的分類、目標識別和場景理解四個角度，我們發現DL技術在目標識別和場景理解領域取得了顯著的成功，即近幾十年來RS界普遍認為是挑戰的領域，因為這類應用需要我們從底層特征（通常是原始像素表示）中抽象出高層的語義信息，而傳統RS的特征描述特征提取分類方法都是淺層模型，用它來發掘高層表示是非常困難或不可能的。

On the other hand, the achievements of DL techniques in image preprocessing and pixel-based classification (especially considering the cost of the large training set) have not been as dramatic, which is partly because the image-quality improvement is more likely to relate to the image-degradation model (as with the traditional approaches), and the task of predicting the label of pixels in the RS image is relative shallow for most conditions, even when only addressing the spectral feature. Despite this, we strongly believe that DL techniques are crucial and important in RS data analysis, particularly for the age of RS big data.另一方面，DL技術在圖像預處理和基于像素的分類方面取得的成就并不那么引人注目（尤其是考慮到大型訓練集的成本），這部分是因為圖像質量的改善更可能與圖像退化模型有關（與傳統方法一樣），而且對于大多數條件下，即使只解決光譜特征，預測RS圖像中像素的標簽的任務也相對較淺。盡管如此，我們堅信，DL技術在RS數據分析中是至關重要的，特別是對于RS大數據時代，DL技術是非常重要的。

However, the research in DL is still young and many questions remain unanswered [44]. The following are some potentially interesting topics in RS data analysis.然而，DL的研究還很年輕，很多問題還沒有得到解答[44]。以下是RS數據分析中一些潛在的有趣課題。

The number of training samples: Although DL methods can learn highly abstract feature representations from raw RS images, the detection and recognition performance relies on large numbers of training samples. 訓練樣本的數量：雖然DL方法可以從原始RS圖像中學習高度抽象的特征表示，但檢測和識別性能依賴于大量的訓練樣本。However, there is usually a lack of high-quality training images because the collection of labeled HR images is difficult. Under these circumstances, how to retain the representation learning performance of the DL methods with fewer adequate training samples remains a big challenge.然而，由于標簽化的HR圖像收集困難，通常缺乏高質量的訓練圖像。在這種情況下，如何在沒有足夠訓練樣本的情況下保留DL方法的表征學習性能仍然是一個很大的挑戰。

The complexity of RS images: Unlike natural scene images, HR RS images include various types of objects with different sizes, colors, rotations, and locations in a single scene, while distinct scenes belonging to different categories may resemble each other in many respects. The complexity of RS images contributes a lot to the difficulty of learning robust and discriminative representations from scenes and objects with DL. RS圖像的復雜性：與自然場景圖像不同的是，HR RS圖像在一個場景中包含了各種類型的物體，它們的大小、顏色、旋轉和位置都不相同，而屬于不同類別的不同場景可能在許多方面彼此相似。RS圖像的復雜性在很大程度上造成了用DL從場景和物體中學習魯棒性和鑒別性表示的難度。

Transfer between data sets: An interesting direction is the transfer of the feature detectors learned by deep networks from one data set to another, since there is often a lack of training images in some fields of RS. Especially when facing the large variations of RS data sets, the problem may be even worse, which needs further and systematic research. 數據集之間的轉移： 一個有趣的方向是將深度網絡學習到的特征檢測器從一個數據集轉移到另一個數據集，因為在RS的某些領域往往缺乏訓練圖像。特別是面對RS數據集的大變化時，問題可能會更加嚴重，這需要進一步的系統研究。

Depth of the DL model: The deeper the deep networks are, the better the performance of the models. For supervised networks such as CNNs, deeper layers can learn more complex distributions, but they may result in many more parameters to learn, and hence can lead to the problem of overfitting, especially when the training samples are inadequate. The computation time is also a vital factor that should be considered. Exploring the proper depth of a DL model for a given data set is still an open topic to be researched. DL模型的深度：深度網絡越深，模型的性能越好。對于CNN等監督網絡來說，更深的層數可以學習更復雜的分布，但可能會導致更多的參數需要學習，因此可能會導致過擬合的問題，特別是當訓練樣本不足時。 計算時間也是一個應該考慮的重要因素。對于一個給定的數據集，探索DL模型的適當深度仍然是一個有待研究的開放性課題。

總結

以上是生活随笔為你收集整理的Deep Learning for Remote Sensing Data的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。