Neural Networks: Decoding Spectral Bias

What Fourier Analysis Does Not Cover

Neural networks have transformed fields such as image classification, system autonomy, and language modeling due to their ability to fit high-dimensional nonlinear functions. Their efficiency is such that they can achieve a zero error rate even on datasets where labels are assigned randomly, demonstrating their remarkable flexibility. Despite this flexibility, these models often provide a useful inductive bias, as they generally generalize better to unseen test data than other types of models.

However, regression with neural networks has a notable drawback, known as spectral bias in the literature. Popularized in 2019, spectral bias states that neural networks fit regression targets from low to high frequencies. As illustrated in Figure 1, the network first learns the low-frequency content of a function before refining the fit to capture higher frequencies. In this literature, the "frequency content" of the regression target is often analyzed using its Fourier transform.

Issues with High-Frequency Fitting

Fitting high-frequency functions with neural networks is notoriously slow, requiring many training epochs. Studies have shown that commonly used activation functions, such as ReLU, hyperbolic tangent, and sigmoid, have spectra that rapidly decrease at high frequencies, which partly explains this bias.

An influential approach, the Neural Tangent Kernel (NTK), proposes that in a network with infinite width, the network's output evolves according to a linear dynamical system. By using linear dynamical systems theory to decompose the network's output into orthogonal modes, the authors show that the rate of convergence is inversely proportional to the frequency content of the mode, explaining why high frequencies are harder to learn.

Strategies to Mitigate Spectral Bias

To counter spectral bias, several strategies have been proposed. Second-order optimization is one of them, but architectural modifications are also being explored. For example, replacing standard activation functions with sinusoids, as in SIREN networks, or using random periodic embeddings in Fourier feature networks are promising approaches.

The success of standard neural network architectures, such as multilayer perceptrons and convolutional networks, in traditional machine learning suggests that fitting high frequencies is not a bottleneck for many application domains. However, an inability to robustly or efficiently fit high-frequency functions can pose problems in scientific applications, where multi-scale issues and wave propagation heavily depend on oscillatory solution fields.

Intuitive Understanding of Spectral Bias

While the Fourier spectrum of the activation function provides some insight into the origin of spectral bias for general neural network training problems, and the NTK offers an explanation in the case of networks with infinite width, we believe that a more intuitive understanding of spectral bias is possible. In this article, we argue that, in many cases, the spectral bias of multilayer perceptron (MLP) networks with hyperbolic tangent activations can be understood from the perspective of what we call "sequential fitting."

We define sequential fitting as the phenomenon where neural networks adjust their target function by starting at the boundary and then progressing into the domain, building one oscillation of the target function at a time. We show that this behavior holds across several example problems in one and two spatial dimensions, and we also find evidence of a "boundary effect," where the training process is influenced not only by the frequency content of the target function but also by its behavior near the boundaries.

One-Dimensional Regression

In the following examples, we work with two-hidden-layer MLP networks with hyperbolic tangent activation functions. We can explicitly write the network as follows:

[ u(\mathbf x; \boldsymbol \theta ) = \mathbf w^3 \cdot \tanh( \mathbf w^2 ( \tanh(\mathbf w^1 \mathbf x + \mathbf b^1)) + \mathbf b^2 ), \quad \boldsymbol \theta=[ \mathbf w^3 , \mathbf w^2 , \mathbf b^2 , \mathbf w^1 , \mathbf b^1],]

where (\boldsymbol \theta) is the collection of all trainable parameters (weights and biases) of the network, (\mathbf x \in \Omega) is the spatial coordinate (either one-dimensional or two-dimensional), and (\Omega) is the computational domain. The widths of the two hidden layers are taken to be equal, and we denote this width by ( H ). We call the target function (v(\mathbf x)), and we define the training objective as follows:

[ \underset{\boldsymbol \theta}{\text{argmin }} \frac{1}{2} \int \Big( u(\mathbf x ; \boldsymbol \theta) – v(\mathbf x) \Big)^2 d\Omega. ]

To demonstrate the phenomenon we call sequential fitting, we start with a one-dimensional regression problem over the unit domain, for example, ( \Omega =[0,1] ), with a target function given by ( v(x) = \sin(26 \pi x) ). The width of the network is ( H=100 ), and the regression problem is solved using ADAM optimization with a learning rate of ( 5 \times 10^{-3}). The integral in the objective is approximated using midpoint quadrature on a uniform grid with 500 points. Unless otherwise stated, all subsequent one-dimensional examples will be solved with this network architecture and this integration rule, along with these optimization parameters. The number of training epochs will be displayed on the graphs showing the progress of the fit, and thus specified on a case-by-case basis.

Figure 2 shows the results of this first example. The network initializes the fitting process near the boundaries and then iteratively works towards the center of the domain, fitting one high-frequency oscillation at a time. This is the phenomenon we call sequential fitting.

Influence of the Oscillatory Function Envelope

A second example demonstrates how the envelope of the oscillatory function influences the training process. If sequential fitting starts at the boundaries, we hypothesize that the behavior of the function near the boundaries of the domain may have an effect on the training. In particular, we test the case where an envelope function reduces the amplitude of oscillations to zero at one end of the domain. Our target function is (v(x)=\sqrt{x} \sin(26 \pi x)), where the envelope (\sqrt{x}) suppresses the oscillations at the left end of the domain. Figure 3 shows the results. The sequential fitting process starts from the right side of the domain, where the oscillations have a greater amplitude. As before, the network fits one oscillation at a time, except that the process is now unilateral due to the suppressed oscillations on the left.