Univariate Radial Basis Function Layers: Brain-inspired Deep Neural Layers for Low-Dimensional Inputs

by Daniel Jost, Basavasagar Patil, Xavier Alameda-Pineda, and Chris Reinke

[preprint] | [code]

Abstract: Deep Neural Networks (DNNs) became the standard tool for function approximation with most of the introduced architectures being developed for high-dimensional input data. However, many real-world problems have low-dimensional inputs for which the standard Multi-Layer Perceptron (MLP) are a common choice. An investigation into specialized architectures is missing. We propose a novel DNN input layer called the Univariate Radial Basis Function (U-RBF) Layer as an alternative. Similar to sensory neurons in the brain, the U-RBF Layer processes each individual input dimension with a population of neurons whose activations depend on different preferred input values. We verify its effectiveness compared to MLPs and other state-of-the-art methods in low-dimensional function regression tasks. The results show that the U-RBF Layer is especially advantageous when the target function is low-dimensional and high-frequent.

Problem: Low Dimensional Inputs

Existing deep learning approaches were mainly developed for high-dimensional inputs such as Convolutional Neural Networks (CNNs) for images. However, many real-world problems have low-dimensional inputs for which the standard MLP are a common choice. As an example, the Penn Machine Learning Benchmarks (PMLB) (Romano et al., 2021) introduce a diverse set of real-world classification and regression problems with up to 15 features.

Inspiration: Sensor Processing in the Brain

Our approach is inspired by the population coding and the tuning curve stimulus encoding theory from neuroscience (Dayan & Abbott, 2001). According to these theories, some neurons encode low-dimensional continuous stimuli by having a bell-shaped tuning curve where their activity peaks at a preferred stimuli value. Several neurons (or groups of neurons) encode a single stimulus dimension by having their preferred values span over the stimulus value range. One such example is the processing of movement directions in the medial temporal visual area (Maunsell & Van Essen,1983) where different populations of neurons are responsive to different directions of movement. Similar to their neuroscientific inspiration a unit in a U-RBF layer encodes a single continuous input with several neurons each having a Gaussian activation function peaking at a certain input value. The preferred value is different for each of the U-RBF neurons.

Method: The U-RBF Layer

The U-RBF layer maps each input independently to multiple output dimensions with different mean c and spread σ values for each output dimension. The higher dimensional output is then further processed by a Multilayer Perceptron (MLP). The whole network can be trained end-to-end including the parameters of the U-RFB layer. The mapping of the U-RBF layer is defined by:

with

where the high dimensional representation is denoted by z_d,k for each of the D input dimensions (RBF units) and K_d Gaussian kernels. The Gaussian function is denoted as G and outputs the activation for an input value x_d according to the Gaussian parameters c_d,k and σ_d,k.

Experiments

We evaluated the U-RBF layer on a synthetic regression task and a collection of real-world datasets. In the synthetic case, we use low-pass filtered white noise to show the ability to regress random functions of different complexities. We increase the complexity by increasing the low-pass filter cut-off frequency.

We investigated the impact of using tasks with up to 3 input dimensions and different numbers of RBF neurons per input dimension. We can show that the U-RBF produces superior results in most of the scenarios and is therefore able to approximate the given function better than other methods such as a standard MLPs. 

In the case of real world datasets, we evaluated the methods on a collection of datasets from the Penn Machine Learning Benchmark with up to 5 input features. The results show competitive or superior performance compared to other approaches.

Hyperparameters

The usage of the U-RBF layer requires the uniform initialization of the Gaussians according to the ranges of the input data for each dimension. Therefore, these ranges need to be known in advance and also limit the usage of the U-RBF layer to the first layer in the network since we can easily determine the required parameters in that case. Compared to other state-of-the-art methods, such as the Fourier Feature Mapping (FFM) (Tancik et al., 2020), the required hyperparameters are easy to determine beforehand without requiring a computational intensive hyperparameter sweep.

Conclusion

The introduction of the U-RBF layer in the context of regression tasks has demonstrated a significant improvement in performance over commonly used methods. The U-RBF layer’s inherent bias towards axial changes does not hinder its effectiveness, even in scenarios where this characteristic is not specifically advantageous. Particularly in cases with fewer input neurons and lower target function frequencies, the U-RBF layer outperforms competing approaches considerably. In the context of real-world datasets with low dimensional inputs the U-RBF layer improves the results compared to other methods such as FFM or standard MLP. Overall, the U-RBF layer’s introduction is a considerable advancement in the field of regression analysis, offering robustness, adaptability, and often superior performance compared to both traditional and other deep learning methods.