AdaDelta

What is AdaDelta? AdaDelta Explained.

AdaDelta is an optimization algorithm that is commonly used for training deep neural networks. It is an extension of the Adagrad algorithm, which adapts the learning rate for each parameter in a neural network based on the historical gradients. However, Adagrad suffers from the drawback of continually decreasing the learning rate, which can eventually lead to very small updates and slow convergence.

The AdaDelta algorithm addresses this issue by introducing a modification that allows the learning rate to adapt without decreasing monotonically. Instead of accumulating all historical gradients like Adagrad, AdaDelta maintains a rolling window of the past gradients. It calculates the root mean square (RMS) of these gradients and uses it to update the parameters.

Here’s a brief overview of how AdaDelta works:

Initialize variables: Initialize two variables, E[g^2] and E[∆x^2], to accumulate the squared gradients and squared parameter updates, respectively.

Compute gradients: Compute the gradients of the loss function with respect to the parameters.

Accumulate squared gradients: Update E[g^2] by taking a decayed average of the squared gradients.

Compute parameter update: Calculate the root mean square (RMS) of the past parameter updates from E[∆x^2].

Compute the adaptive learning rate: Compute the learning rate adjustment factor based on the ratio of the RMS of the parameter updates to the RMS of the gradients.

Update parameters: Update the parameters using the learning rate adjustment factor.

Accumulate squared parameter updates: Update E[∆x^2] by taking a decayed average of the squared parameter updates.

Repeat: Steps 2 to 7 are repeated for each iteration of the training process.

AdaDelta’s adaptive learning rate scheme allows it to converge faster and avoid the need for manual tuning of the learning rate. It has been shown to be effective in training deep neural networks, especially in scenarios where the gradients have high variance or when dealing with sparse data.