Statistics and Computational Methods Seminar Series - Fall 2026

Speaker: Gian Paolo Leonardi

Title: Rate-convergence and regularization of quantized networks

Abstract:
Quantization of deep neural networks is nowadays a crucial task for the deployment of AI tools on edge computing hardware. One of the most popular techniques that are currently adopted for quantization requires training a quantized DNN using an update backward pass in which classical gradients are replaced by proxies (like the Straight-Through Estimator, STE). According to STE protocol, the gradient of the Heaviside function (distributionally, a Dirac delta at 0) is replaced by the gradient of the hard sigmoid. The STE idea is basically heuristic and not supported by a robust theory. In this talk, we present a different perspective, based on the idea of pointwise approximating a DNN with Heaviside activations by means of smooth networks. This idea leads to the notion of "rate-convergence", and in turn to a pointwise compositional approximation theorem together with training methods in which gradient proxies are chosen according to suitable graded annealing protocols (with potential reduction of the training time). In conclusion, we provide a partial theoretical support to various methods that are used for training discontinuous networks, by enlightening the key principle behind their compositional approximation.

Link Teams: https://teams.microsoft.com/meet/3412317001696116?p=RfIqHI3locbQQxtXqn