Konstantin Kirchheim

I am currently a PhD student at the Otto-von-Guericke University Magdeburg (Germany) in the Department of Computer Science.

My research focuses on anomaly detection in high-dimensional data with deep learning models, and more recently with neuro-symbolic methods, mostly on image, video and text data. I also did some work on reproducibility of experiments involving deep learning models, as well as mining of academic literature.

I maintain pytorch-ood, a Python library for out-of-distribution detection with deep neural networks.

In my spare time, I like to scrape data from the web and mine it. This hobby spawned some interesting projects with dedicated websites, such as sworm and extra-mining.

On the right, you can find a rather dated picture of me in the style of an image from Picasso’s blue period. It was generated with Neural Style Transfer.

Latest Updates

Training a German LLM from Scratch 🦜, 14 Nov. 2024 (posts)
This article is not finished and will be updated. The research group I work with has access to a small GPU cluster, which occasionally sits idle. To avoid wasting valuable compute resources (IDLE GPUs essentially burn money through opportunity costs), I decided to train a German GPT-2-style model from scratch, using only German text. Existing German models available on Hugging Face have 137M parameters and a context length of 1024 tokens1, which is quite limited compared to recently released …
Categories: Deep Learning
1806 Words, Tagged with: Deep Learning · Generative Models · LLM
Thumbnail for Training a German LLM from Scratch 🦜
Language Models as Reasoners for Out-of-Distribution Detection, 17 Sep. 2024 (papers)
Our paper Language Models as Reasoners for Out-of-Distribution Detection has been presented at the Workshop on AI Safety Engineering (WAISE) 2024 and recieved the best paper award by popular vote. It constitutes a basic extension of our idea of Out-of-Distribution Detection with Logical Reasoning, where we replaced the prolog-based reasoning component with a LLM. Abstract Deep neural networks (DNNs) are prone to making wrong predictions with high confidence for data that does not stem from their …
Categories: Anomaly Detection · Neuro-Symbolic
195 Words, Tagged with: WAISE · Anomaly Detection · Large Language Models · Neuro-Symbolic
Thumbnail for Language Models as Reasoners for Out-of-Distribution Detection
Deep learning-based harmonization and super-resolution of Landsat-8 and Sentinel-2 images, 17 May. 2024 (papers)
Our paper Deep learning-based harmonization and super-resolution of Landsat-8 and Sentinel-2 images, which is based on the masters thesis of my colleague Venkatesh Thirugnana Sambandham, has been published in the ISPRS Journal of Photogrammetry and Remote Sensing. This work is an extension of our previous workshop paper on transformers for satellite homogenization. In summary, we find that a simple UNet model is able to provide surprisingly good performance for the satellite homogenization task. …
Categories: Deep Learning
345 Words, Tagged with: Deep Learning · Superresolution
Thumbnail for Deep learning-based harmonization and super-resolution of Landsat-8 and Sentinel-2 images
Out-of-Distribution Detection with Logical Reasoning, 04 Jan. 2024 (papers)
Our paper Out-of-Distribution Detction with Logical Reasoning has been accepted on the WACV 2024. Abstract Machine Learning models often only generalize reliably to samples from the training distribution. Consequentially, detecting when input data is out-of-distribution (OOD) is crucial, especially in safety-critical applications. Current OOD detection methods, however, tend to be domain agnostic and often fail to incorporate valuable prior knowledge about the structure of the training …
Categories: Anomaly Detection · Neuro-Symbolic
224 Words, Tagged with: WACV · Anomaly Detection · Neuro-Symbolic
Thumbnail for Out-of-Distribution Detection with Logical Reasoning
Towards Deep Anomaly Detection with Structured Knowledge Representations, 15 Jun. 2023 (papers)
My paper Towards Deep Anomaly Detection with Structured Knowledge Representations has been accepted on the Workshop on AI Safety Engineering at SafeComp. Abstract Machine learning models tend to only make reliable predictions for inputs that are similar to the training data. Consequentially, anomaly detection, which can be used to detect unusual inputs, is critical for ensuring the safety of machine learning agents operating in open environments. In this work, we identify and discuss several …
Categories: Anomaly Detection · Neuro-Symbolic
180 Words, Tagged with: WAISE · Anomaly Detection · Neuro-Symbolic
Thumbnail for Towards Deep Anomaly Detection with Structured Knowledge Representations
Mining the Bundestag, 22 Jan. 2023 (posts)
Did you know the German parliament publishes protocols for all of its proceedings in PDF format? It is relatively straightforward to download and parse them, so we can easily collect a dataset of transcripts of what seems to be every speech in the Bundestag since the Second World War. My original idea was to mine the speeches for word associations. Some words will be associated with other words based on the intended connotation, and this association might change over time as the connotations …
Categories: Data Mining
1024 Words, Tagged with: Bundestag · Data Mining · Generative Models
Thumbnail for Mining the Bundestag
Mining tagesschau.de, 26 Nov. 2022 (posts)
I like to read tagesschau.de, so I wrote a script to scrape it in regular intervals. My original goal was to determine which articles stay on the front page the longest, which ones allow commenting (a feature that seems to have been disabled almost entirely since March 2020), and if articles are modified after the initial release (without mentioning this), because I sometimes feel that headlines change. Dataset Creation Tagesschau provides a JSON API, so fetching all of the articles is …
Categories: Data Mining
1040 Words, Tagged with: Tagesschau · Generative Models · Data Mining
Thumbnail for Mining tagesschau.de
On Outlier Exposure with Generative Models, 23 Nov. 2022 (papers)
Our paper On Outlier Exposure with Generative Models has been accepted on the NeurIPS Machine Learning Safety Workshop. Abstract While Outlier Exposure reliably increases the performance of Out-of-Distribution detectors, it requires a set of available outliers during training. In this paper, we propose Generative Outlier Exposure (GOE), which alleviates the need for available outliers by using generative models to sample synthetic outliers from low-density regions of the data distribution. The …
Categories: Anomaly Detection
110 Words, Tagged with: MLSW · Generative Models · Anomaly Detection
Thumbnail for On Outlier Exposure with Generative Models
Social Work Research Map, 11 Nov. 2022 (papers)
During the last weeks, I worked with some colleagues on a website that aims to improve access to social work literature. We described the results in out paper Social Work Research Map – ein niederschwelliger Zugang zu internationalen Publikationen der Sozialen Arbeit, which has been published in the journal Soziale Passagen. While the paper is written in german, there is also a technical report in english. Abstract Internationalization is a central topic in higher education policy in Germany. An …
Categories: Data Mining
280 Words, Tagged with: Soziale Passagen · SWORM · Data Mining
Thumbnail for Social Work Research Map
Towards Transformer-based Homogenization of Satellite Imagery for Landsat-8 and Sentinel-2, 13 Aug. 2022 (papers)
Our abstract Towards Transformer-based Homogenization of Satellite Imagery for Landsat-8 and Sentinel-2 was accepted for presentation on the Transformers Workshop for Environmental Science. In summary, we somewhat surprisingly found that transformers, a neural network architecture that achieves state-of-the-art results on most tasks it is applied to, does not outperform a vanilla U-Net model on our particular superresolution task.
Categories: Deep Learning
58 Words, Tagged with: ESST · Superresolution
Thumbnail for Towards Transformer-based Homogenization of Satellite Imagery for Landsat-8 and Sentinel-2
Filter Visualization, 27 Jul. 2022 (posts)
What is Filter Visualization? Deep Neural Networks are often seen as black-boxes: they map some input to some output, and we can make them do this surprisingly well. However, we usually have no idea how this mapping works. Particularly Convolutional Neural Networks (CNNs), which employ “convolutions” as filters, achieved some impressive results (before Vision Transformers came along). Filter Visualization can help us understand what kind of patterns the convolutional filters in CNNs …
Categories: Deep Learning
419 Words, Tagged with: Deep Learning · Explainability · CNN
Thumbnail for Filter Visualization
Multi-Class Hypersphere Anomaly Detection (MCHAD), 13 Jul. 2022 (papers)
Our Paper Multi-Class Hypersphere Anomaly Detection (MCHAD) has been accepted for presentation at the ICPR 2022. In summary, we propose a new loss function for learning neural networks that are able to detect anomalies in their inputs. Poster for MCHAD (PDF). How does it work? Omitting some details, the loss we propose has three different components, each of which we will explain in the following. Intra-Class Variance We want the $f(x)$ of one class to cluster as tightly around a class center …
Categories: Anomaly Detection
366 Words, Tagged with: ICPR · Anomaly Detection
Thumbnail for Multi-Class Hypersphere Anomaly Detection (MCHAD)
PyTorch-OOD: A library for Out-of-Distribution Detection based on PyTorch, 13 Jul. 2022 (papers)
Our paper PyTorch-OOD: A library for Out-of-Distribution Detection based on PyTorch has been presented at the CVPR 2022 Workshops. You can find the most recent version of the python source code on GitHub. Abstract Machine Learning models based on Deep Neural Networks behave unpredictably when presented with inputs that do not stem from the training distribution and sometimes make egregiously wrong predictions with high confidence. This property undermines the trustworthiness of systems depending …
Categories: Anomaly Detection
214 Words, Tagged with: CVPR Workshops · Anomaly Detection
Thumbnail for PyTorch-OOD: A library for Out-of-Distribution Detection based on PyTorch
On Challenging Aspects of Reproducibility in Deep Anomaly Detection, 13 Jul. 2022 (papers)
Our companion paper On Challenging Aspects of Reproducibility in Deep Anomaly Detection has been accepted for presentation on the Fourth Workshop on Reproducible Research in Pattern Recognition (satellite event of ICPR 2022). In it, we discuss aspects of reproducibility for our anomaly detection algorithm MCHAD, as well as anomaly detection with deep neural networks in general. In particular, we discussed the following challenges for the reproducibility: Nondeterminism: conducting the same …
Categories: Anomaly Detection · Reproducibility
212 Words, Tagged with: RRPR · Anomaly Detection · Reproducibility
Thumbnail for On Challenging Aspects of Reproducibility in Deep Anomaly Detection
Addressing Randomness in Evaluation Protocols for Out-of-Distribution Detection, 13 Jul. 2021 (papers)
Our Paper Addressing Randomness in Evaluation Protocols for Out-of-Distribution Detection has been accepted at the ICJAI 2021 Workshop for Artificial Intelligence for Anomalies and Novelties. In summary, we investigated the following phenomenon: when you train neural networks several times, and then measure their performance on some task, there is a certain variance in the performance measurements, since the results of experiments may vary based on several factors (that are effectively …
Categories: Anomaly Detection · Reproducibility
252 Words, Tagged with: AI4AN · Anomaly Detection · Reproducibility
Thumbnail for Addressing Randomness in Evaluation Protocols for Out-of-Distribution Detection
Data-Mining als Werkzeug empirischer Sozialforschung, 13 Jul. 2020 (papers)
Inspired by the “Spiegel-Mining” talk from David Kriesel, a friend of mine and a Prof. from the Hochschule Magdeburg scraped a german website that regularly publishes reviews of social work literature, and mined the resulting 18.000 articles, hoping for interesting insights. In an attempt to visualize the discourse, we created several topic maps, like the one below, which you can find on the accompanying (german) website. The colours represent the gender of the authors of the review. …
Categories: Data Mining
150 Words, Tagged with: Sozial Extra · Data Mining
Thumbnail for Data-Mining als Werkzeug empirischer Sozialforschung
Explanation-based Anomaly Detection in Deep Neural Networks, 01 Feb. 2020 (posts)
Masters Thesis (PDF). If an AI gives you a weird explanation for its prediction, you should remain septical about the accuracy of the prediction. Sounds reasonable? This was the general idea of my masters thesis, which was originally titled Self-Assessment of Visual Recognition Systems based on Attribution. Today, I would call it Explanation-based Anomaly Detection in Deep Neural Networks. The general idea was to use attribution-based explanation methods to detect anomalies (such as unusual …
Categories: Anomaly Detection
340 Words, Tagged with: Deep Learning · Anomaly Detection · CNN · Explainability
Thumbnail for Explanation-based Anomaly Detection in Deep Neural Networks