Background and Recent Results:
Deep learning and specifically neural networks (NNs) have emerged as some of the most powerful algorithms applicable to a wide range of problems. Recent works have investigated the convexity of high-dimensional datasets, both in the input domain, as well as in the latent space of feature extractors. It is found that image datasets such as CIFAR-10 and MNIST form convex hulls in the input domain and admit (partially) convex decision boundaries in their respective latent representation. Convexity is, furthermore, associated with few-shot learning and generalization. These findings motivate us to study the emergence of convex decision boundaries in modern deep learning architectures and datasets through nearest convex hull classification (NCHC).Recent theoretical and empiric work found that there are convex separable problems on which multi-layer perceptrons (MLPs) struggle, whereas NCHC excels with significantly reduced error floors at lower number of training samples. This effect, however, is only observed on synthetic data which is asymptotically convex separable in the noise-free regime.
Overall Project Aim:
The project has two main aims.
Firstly, we want to analyze if NCHC can translate the significantly reduced generalization errors on convexly separable synthetic data to latent space representations of real-world data. This includes the study of NCHC on latent representations of NN-based feature extractors, end-2-end training/fine-tuning of a NN followed by NCHC read-out layer, and the study of further datasets of different modalities, i.e. video, audio or text.
Secondly, we need to refine our understanding of how convex separability relates to convex decision boundaries. To this end the synthetic dataset shall be analyzed for convex decision boundaries using a Euclidean convexity measure. Additionally, we aim to investigate if modern deep-learning architectures handle convex separable tasks better than standard MLPs.
Project Objectives:
The main objectives of this thesis can be summarized as follows:
- Identification of state-of-the-art (SOTA) feature extractors on image datasets.
- Evaluation of NCHC on latent space representation of feature extractors on 1-3 datasets.
- Implementation of end-2-end fine-tuning of the feature extractor and NCHC.
- Comparison of NCHC generalization error and Euclidean convexity measures.
- Evaluation of modern deep learning architectures (transformers, convex neural networks, modern Hopfield models, …) on the synthetic, convex separable dataset.
The objectives may be adapted based on the project type (Bachelor Thesis, Research Project, Master Thesis).
Prerequisites:
The following qualifications are required in order to pursue this project:
- Good knowledge of linear algebra.
- Basic knowledge of (convex) optimization.
- Good programming skills in Python (JAX, PyTorch or Haiku, numpy).
- Beneficial: experience with Git.
- Beneficial: experience with convex solvers (e.g. cvxpy or jaxopt).
- Beneficial: experience with bash and slurm.
If you are interested in pursuing this project, please contact Levi Bohnacker (levi.bohnacker@fau.de). The full project description can be found here.