介紹了一種基于低階矩自適應估計的隨機目標函數一階梯度優化算法Adam。該方法易于實現，計算效率高，對內存的要求少，對梯度的對角重新縮放是不變的，并且非常適合于數據和/或參數很大的問題。該方法也適用于非平穩目標和具有非常嘈雜和/或稀疏梯度的問題。超參數有直觀的解釋，通常需要很少的調整。本文討論了一些與相關算法的聯系，Adam 正是在這些算法上受到啟發。我們還分析了算法的理論收斂性，并給出了與在線凸優化框架下的最優結果相當的收斂遺憾界。實驗結果表明，該方法在實際應用中效果良好，與其它隨機優化方法相比具有一定的優越性。最后，我們討論了AdaMax，一個基于無窮范數的Adam變體。

Stochastic gradient-based optimization is of core practical importance in many fields of science and engineering. Many problems in these fields can be cast as the optimization of some scalar parameterized objective function requiring maximization or minimization with respect to its parameters. If the function is differentiable w.r.t. its parameters, gradient descent is a relatively efficient optimization method, since the computation of first-order partial derivatives w.r.t. all the parameters is of the same computational complexity as just evaluating the function. Often, objective functions are stochastic. For example, many objective functions are composed of a sum of subfunctions evaluated at different subsamples of data; in this case optimization can be made more efficient by taking gradient steps w.r.t. individual subfunctions, i.e. stochastic gradient descent (SGD) or ascent. SGD proved itself as an efficient and effective optimization method that was central in many machine learning success stories, such as recent advances in deep learning (Deng et al., 2013; Krizhevsky et al., 2012; Hinton & Salakhutdinov, 2006; Hinton et al., 2012a; Graves et al., 2013). Objectives may also have other sources of noise than data subsampling, such as dropout (Hinton et al., 2012b) regularization. For all such noisy objectives, efficient stochastic optimization techniques are required. The focus of this paper is on the optimization of stochastic objectives with high-dimensional parameters spaces. In these cases, higher-order optimization methods are ill-suited, and discussion in this paper will be restricted to first-order methods.

基于隨機梯度的優化方法在許多科學和工程領域具有核心的實際意義。這些領域中的許多問題都可以轉化為對某個標量參數化目標函數的優化，該函數的參數需要最大化或最小化。如果函數的參數是可微的，梯度下降法是一種比較有效的優化方法，因為一階偏導數的計算與函數的求值具有相同的計算復雜度。通常，目標函數是隨機的。例如，許多目標函數由在不同數據子樣本上求值的子函數和組成;在這種情況下，優化可以通過采取梯度步驟w.r.t.單獨的子函數，即隨機梯度下降(SGD)或上升來提高效率。SGD證明了自己是一種高效和有效的優化方法，這在許多機器學習成功的故事中都是核心，比如最近在深度學習方面的進展(Deng et al.， 2013;Krizhevsky等，2012;Hinton & Salakhutdinov, 2006;Hinton等，2012a;Graves 等人，2013)。目標也可能有數據子采樣之外的其他噪音源，如dropout (Hinton et al.， 2012b)正則化。對于所有這些有噪聲的目標，都需要有效的隨機優化技術。本文主要研究具有高維參數空間的隨機目標的優化問題。在這種情況下，高階優化方法是不合適的，本文的討論將局限于一階方法。