com.aliasi.stats
Class AnnealingSchedule

java.lang.Object
  com.aliasi.stats.AnnealingSchedule

public abstract class AnnealingSchedule
extends Object

An AnnealingSchedule instance implements a method to return the learning rate for a specified epoch. It also has a method to receive callback reports on the resulting error rate for an epoch so that implementations may adapt the learning rate to received error.

This class provides an abstract base class implementation along with static factory methods to create the two most popular annealing schedules, constant, exponential decay and inverse scaling.

Constant Learning Rate

The contant learning rate schedule always returns the same learning rate, which is fixed at construction time.

Exponential Decay

The exponential decay annealing schedule sets the learning rate as follows based on a specfieid exponential base:

 learningRate(epoch) = initialLearningRate * Math.pow(base,epoch)

Under this schedule, the learning rate undergoes exponential decay starting at the initial learning rate and decaying exponentially at a rate determined by the base of the exponent.

The exponential learning rate can find solutions quickly in fairly well-behaved spaces, but may stop short of the minimum error solution due to too much decay in later epochs.

Inverse Scaling

The inverse learning rate scaling sets the learning rate as:

 learningRate(epoch) = initialLearningRate / (1 + epoch/annealingRate)

The inverse scaling annealing schedule lowers the rate more quickly than the exponential rates initially and then more slowly for later epochs.

This is a popular learning rate because it is guaranteed to converge in the limit. It can be slower to converge once it gets near a solution than exponential decay.

Rejecting Updates

The return value of the receivedError() method is a boolean flag indicating whether to accept the updates to the underlying vectors or not. This allows sampling-based annealing schedules to be implemented that evaluate several learning rates and accept just the one with the most error reduction.

Convergence Guarantees in the Limit

The inverse scale metric is popular because it is theoretically guaranteed to converge in the limit.

An annealing rate will converge in the limit within arbitrary precision of a solution if the learning rate satisfies:

 Σepoch learningRate(epoch) = ∞

and:

 Σepoch learningRate(epoch)2 < ∞

References

• Wikipedia: Simulated Annealing

Since:

LingPipe3.5

Version:

3.5

Author:

Bob Carpenter

Constructor Summary

AnnealingSchedule()
Do-nothing constructor to be used by concrete implementations.

Method Summary

static AnnealingSchedule	constant(double learningRate) Return the annealing schedule for the specified constant learning rate.
static AnnealingSchedule	exponential(double initialLearningRate, double base) Return the exponential annealing schedule with the specified initial learning rate and exponent.
static AnnealingSchedule	inverse(double initialLearningRate, double annealingRate) Return the inverse annealing schedule with the specified initial learning rate and annealing rate.
abstract double	learningRate(int epoch) Return the learning rate for the specified epoch.
boolean	receivedError(int epoch, double rate, double error) Receive a report from an optimizer about the effect of the specified learning rate in the specified epoch and return true if the update producing the error should be accepted or rejected.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AnnealingSchedule

public AnnealingSchedule()

Do-nothing constructor to be used by concrete implementations.

Method Detail

learningRate

public abstract double learningRate(int epoch)

Return the learning rate for the specified epoch.

Parameters:

epoch - Epoch being evaluated.

Returns:

Learning rate for that epoch.

receivedError

public boolean receivedError(int epoch,
                             double rate,
                             double error)

Receive a report from an optimizer about the effect of the specified learning rate in the specified epoch and return true if the update producing the error should be accepted or rejected.

This abstract class's implementation of this method is to do nothing. Concrete subclasses which adapt learning rates based on empirical error reports from the optimizer must override this method.

inverse

public static AnnealingSchedule inverse(double initialLearningRate,
                                        double annealingRate)

Return the inverse annealing schedule with the specified initial learning rate and annealing rate. The inverse annealing schedule is not adaptive. adaptation.

Parameters:

initialLearningRate - Initial learning rate for epoch zero.

annealingRate - Rate at which initial learning rate anneals.

Returns:

The inverse annealing schedule for the specified initial learning rate and annealing rate.

Throws:

IllegalArgumentException - If the initial learning rate or the annealing rates are not finite and positive.

exponential

public static AnnealingSchedule exponential(double initialLearningRate,
                                            double base)

Return the exponential annealing schedule with the specified initial learning rate and exponent. The exponential annealing schedule is not adaptive.

Parameters:

initialLearningRate - Initial learning rate for epoch 0.

base - Base of the exponential decay.

Returns:

The exponential annealing schedule for the specified initial learning rate and exponent.

Throws:

IllegalArgumentException - If the initial learning rate is not finite and positive, or if the exponent is not between 0.0 (exclusive) and 1.0 (inclusive).

constant

public static AnnealingSchedule constant(double learningRate)

Return the annealing schedule for the specified constant learning rate. The constant annealing schedule is not adaptive.

Parameters:

learningRate - The constant rate returned by this annealing schedule.

Returns:

The annealing schedule returning the specified constant learning rate.

Throws:

IllegalArgumentException - If the learning rate is not finite and positive.