Struct AdamW

pub struct AdamW {
    pub lr: Array,
    pub betas: (Array, Array),
    pub eps: Array,
    pub weight_decay: Array,
    pub state: State<(Array, Array)>,
}

The AdamW optimizer [1].

Following the above convention, in contrast with [1], we do not use bias correction in the first and second moments for AdamW. We update the weights with a weightDecay lambda value:

[1]: Loshchilov, I. and Hutter, F., 2019. Decoupled weight decay regularization. ICLR 2019.

Fields

lr: Array

The learning rate.

betas: (Array, Array)

The coefficients used for computing running averages of the gradient and its square.

Default to AdamW::DEFAULT_BETAS.

eps: Array

The epsilon added to the denominator to improve numerical stability.

Default to AdamW::DEFAULT_EPS.

weight_decay: Array

The weight decay

Default to AdamW::DEFAULT_WEIGHT_DECAY.

state: State<(Array, Array)>

Inner state.

Implementations

impl AdamW

pub fn new(lr: impl Into<f32>) -> Self

Creates a new instance of AdamW.

impl AdamW

pub const DEFAULT_BETAS: (f32, f32) = super::Adam::DEFAULT_BETAS

Default value for betas.

pub const DEFAULT_EPS: f32 = 9.99999993E-9f32

Default value for eps.

pub const DEFAULT_WEIGHT_DECAY: f32 = 0.00999999977f32

Default value for weight_decay.

Trait Implementations

impl Buildable for AdamW

type Builder = AdamWBuilder

The builder type for this buildable type

impl Builder<AdamW> for AdamWBuilder

type Error = Infallible

Error with building

fn build(self) -> Result<AdamW, Self::Error>

Build the type

impl Clone for AdamW

fn clone(&self) -> AdamW

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AdamW

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Optimizer for AdamW

type State = HashMap<Rc<str>, (Array, Array)>

State of the optimizer.

fn state(&self) -> &Self::State

Get the state of the optimizer.

fn state_mut(&mut self) -> &mut Self::State

Get the mutable state of the optimizer.

fn update_single( &mut self, key: &Rc<str>, gradient: &Array, parameter: &mut Array, ) -> Result<(), Exception>

Update a single parameter with the given gradient.

fn update<M>( &mut self, model: &mut M, gradients: impl Borrow<FlattenedModuleParam>, ) -> Result<()>

where M: ModuleParameters,

Apply the gradients to the parameters of the model and update the model with the new parameters.

impl Updatable for &mut AdamW

fn updatable_states(&self) -> impl IntoIterator<Item = &Array>

Returns a list of references to the updatable states.

fn updatable_states_mut(&mut self) -> impl IntoIterator<Item = &mut Array>

Returns a list of mutable references to the updatable states.

impl Updatable for AdamW

fn updatable_states(&self) -> impl IntoIterator<Item = &Array>

Returns a list of references to the updatable states.

fn updatable_states_mut(&mut self) -> impl IntoIterator<Item = &mut Array>

Returns a list of mutable references to the updatable states.