Struct Adamax

pub struct Adamax {
    pub lr: Array,
    pub betas: (Array, Array),
    pub eps: Array,
    pub state: State<(Array, Array)>,
}

The Adamax optimizer, a variant of Adam based on the infinity norm [1].

Our Adam implementation follows the original paper and omits the bias correction in the first and second moment estimates. In detail,

[1]: Kingma, D.P. and Ba, J., 2015. Adam: A method for stochastic optimization. ICLR 2015.

Fields

lr: Array

The learning rate.

betas: (Array, Array)

The beta coefficients

eps: Array

The epsilon added to the denominator to improve numerical stability.

state: State<(Array, Array)>

Inner state.

Implementations

impl Adamax

pub fn new(lr: impl Into<f32>) -> Self

Creates a new instance of Adamax.

impl Adamax

pub const DEFAULT_BETAS: (f32, f32) = _

Default value for betas.

pub const DEFAULT_EPS: f32 = 9.99999993E-9f32

Default value for eps.

Trait Implementations

impl Buildable for Adamax

type Builder = AdamaxBuilder

The builder type for this buildable type

impl Builder<Adamax> for AdamaxBuilder

type Error = Infallible

Error with building

fn build(self) -> Result<Adamax, Self::Error>

Build the type

impl Clone for Adamax

fn clone(&self) -> Adamax

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Adamax

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Optimizer for Adamax

type State = HashMap<Rc<str>, (Array, Array)>

State of the optimizer.

fn state(&self) -> &Self::State

Get the state of the optimizer.

fn state_mut(&mut self) -> &mut Self::State

Get the mutable state of the optimizer.

fn update_single( &mut self, key: &Rc<str>, gradient: &Array, parameter: &mut Array, ) -> Result<()>

Update a single parameter with the given gradient.

fn update<M>( &mut self, model: &mut M, gradients: impl Borrow<FlattenedModuleParam>, ) -> Result<()>

where M: ModuleParameters,

Apply the gradients to the parameters of the model and update the model with the new parameters.

impl Updatable for &mut Adamax

fn updatable_states(&self) -> impl IntoIterator<Item = &Array>

Returns a list of references to the updatable states.

fn updatable_states_mut(&mut self) -> impl IntoIterator<Item = &mut Array>

Returns a list of mutable references to the updatable states.

impl Updatable for Adamax

fn updatable_states(&self) -> impl IntoIterator<Item = &Array>

Returns a list of references to the updatable states.

fn updatable_states_mut(&mut self) -> impl IntoIterator<Item = &mut Array>

Returns a list of mutable references to the updatable states.