Fig. 5
From: Reversible designs for extreme memory cost reduction of CNN training
Illustration of the numerical errors arising from batch normalization layers. Comparison of the theoretical and empirical evolution of the \(\alpha\) ratio for different \(\rho\) values in our toy example. Empirical values were computed for a Gaussian input signal with zero mean and standard deviation 1 and a white Gaussian noise of standard deviation \(10^{-5}\)