nntoolbox.optim.coord_desc module

Coordinate Descent Optimizer

class nntoolbox.optim.coord_desc.CoordDescOptimizer(optimizers: List[torch.optim.optimizer.Optimizer])

Bases: torch.optim.optimizer.Optimizer

step(closure=None)

Performs a single optimization step (parameter update).

Arguments:

closure (callable): A closure that reevaluates the model and

returns the loss. Optional for most optimizers.

Note

Unless otherwise specified, this function should not modify the .grad field of the parameters.

zero_grad()

Sets the gradients of all optimized torch.Tensor s to zero.

Arguments:

set_to_none (bool): instead of setting to zero, set the grads to None.

This is will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example: 1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently. 2. If the user requests zero_grad(set_to_none=True) followed by a backward pass, .grads are guaranteed to be None for params that did not receive a gradient. 3. torch.optim optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether).