On the Variance of the Adaptive Learning Rate and Beyond

Deep learning library in plain Numpy.

This repository contains the results for the paper: "Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers"

A book on the mathematical foundations of AI from an engineering perspective.

CS F425 Deep Learning course at BITS Pilani (Goa Campus)

ADAS is short for Adaptive Step Size, it's an optimizer that unlike other optimizers that just normalize the derivative, it fine-tunes the step size, truly making step size scheduling obsolete, achieving state-of-the-art training performance

Lion and Adam optimization comparison

Google Street View House Number(SVHN) Dataset, and classifying them through CNN

Reproducing the paper "PADAM: Closing The Generalization Gap of Adaptive Gradient Methods In Training Deep Neural Networks" for the ICLR 2019 Reproducibility Challenge

PyTorch/Tensorflow solutions for Stanford's CS231n: "CNNs for Visual Recognition"

Implemented Adam optimizer in python

Toy implementations of some popular ML optimizers using Python/JAX

A collection of various gradient descent algorithms implemented in Python from scratch

This library provides a set of functionalities for different type of deep learning (and ML) algorithms in C

From linear regression towards neural networks...

Modified XGBoost implementation from scratch with Numpy using Adam and RSMProp optimizers.

The project aimed to implement Deep NN / RNN based solution in order to develop flexible methods that are able to adaptively fillin, backfill, and predict time-series using a large number of heterogeneous training datasets.

A compressed adaptive optimizer for training large-scale deep learning models using PyTorch

Lookahead optimizer ("Lookahead Optimizer: k steps forward, 1 step back") for tensorflow

[Python] [arXiv/cs] Paper "An Overview of Gradient Descent Optimization Algorithms" by Sebastian Ruder

📈Implementing the ADAM optimizer from the ground up with PyTorch and comparing its performance on six 3-D objective functions (each progressively more difficult to optimize) against SGD, AdaGrad, and RMSProp.

Implementation of Adam Optimization algorithm using Numpy

A fast, interactive tool to visualize how different gradient descent algorithms (like vanilla gradient Descent, Momentum, RMSprop, Adam, etc.) navigate complex loss surfaces in real time.

A family of highly efficient, lightweight yet powerful optimizers.

Learning about Haskell with Variational Autoencoders

A Ray Tracing-Inspired Approach to Neural Network Optimization

Short description for quick search

Grams: Gradient Descent with Adaptive Momentum Scaling (ICLR 2025 Workshop)

Improved Hypergradient optimizers for ML, providing better generalization and faster convergence.

MetaPerceptron: A Standardized Framework For Metaheuristic-Driven Multi-layer Perceptron Optimization