Representation learning for heterogeneous databases

We develop a deep-learning methodology on heterogeneous databases, comprising relational data and tabular datasets at large scale.
The stakes are (1) to build machine-learning models that apply readily to the raw, uncurated data so as to avoid manual cleaning, data formatting and integration, (2) to extract reusable representations that reduce sample complexity on new databases by transforming the data in well-distributed vectors.

The challenges are:

• Tokens (symbolic entries) often with high cardinality, such ICD10 classification of disease
• Numerical entries with different, non-Gaussian marginals, such as salary (long-tailed distribution) or age
• Missing values, often missing not at random and thus not ignorable without causing biases
• Shoft text (possibly corresponding to non-normalized entities)
• Operation across tables: aggregation (learning on sequences, joins (optimal transport)

These challenges call for dedicated architectures, that borrow from neural architectures used in natural language
processing and add new tricks. The fundamental ingredients are:

– Entity embeddings in tabular and relational data
– Neural-network formulations of classical statistical techniques such General Additive Models or EM for missing values (Le Morvan et al., 2020);
– Simple attention and self-attention mechanisms, as those we developed for short texts (Chen et al., 2021);

Current research focus

• Learning despite database normalization errors
• Tabular deep learning

Publications