Posts

These posts describe everything I put out here that is not yet published in any other form. Usually, the content is about projects that I do in my spare time or just implementations of techniques that I find interesting.

Some of the articles are unfinished and will be updated later.

Training a German LLM from Scratch 🦜, 14 Nov. 2024 (posts)
This article is not finished and will be updated. The research group I work with has access to a small GPU cluster, which occasionally sits idle. To avoid wasting valuable compute resources (IDLE GPUs essentially burn money through opportunity costs), I decided to train a German GPT-2-style model from scratch, using only German text. Existing German models available on Hugging Face have 137M parameters and a context length of 1024 tokens1, which is quite limited compared to recently released …
Categories: Deep Learning
1806 Words, Tagged with: Deep Learning · Generative Models · LLM
Thumbnail for Training a German LLM from Scratch 🦜
Mining the Bundestag, 22 Jan. 2023 (posts)
Did you know the German parliament publishes protocols for all of its proceedings in PDF format? It is relatively straightforward to download and parse them, so we can easily collect a dataset of transcripts of what seems to be every speech in the Bundestag since the Second World War. My original idea was to mine the speeches for word associations. Some words will be associated with other words based on the intended connotation, and this association might change over time as the connotations …
Categories: Data Mining
1024 Words, Tagged with: Bundestag · Data Mining · Generative Models
Thumbnail for Mining the Bundestag
Mining tagesschau.de, 26 Nov. 2022 (posts)
I like to read tagesschau.de, so I wrote a script to scrape it in regular intervals. My original goal was to determine which articles stay on the front page the longest, which ones allow commenting (a feature that seems to have been disabled almost entirely since March 2020), and if articles are modified after the initial release (without mentioning this), because I sometimes feel that headlines change. Dataset Creation Tagesschau provides a JSON API, so fetching all of the articles is …
Categories: Data Mining
1040 Words, Tagged with: Tagesschau · Generative Models · Data Mining
Thumbnail for Mining tagesschau.de
Filter Visualization, 27 Jul. 2022 (posts)
What is Filter Visualization? Deep Neural Networks are often seen as black-boxes: they map some input to some output, and we can make them do this surprisingly well. However, we usually have no idea how this mapping works. Particularly Convolutional Neural Networks (CNNs), which employ “convolutions” as filters, achieved some impressive results (before Vision Transformers came along). Filter Visualization can help us understand what kind of patterns the convolutional filters in CNNs …
Categories: Deep Learning
419 Words, Tagged with: Deep Learning · Explainability · CNN
Thumbnail for Filter Visualization
Explanation-based Anomaly Detection in Deep Neural Networks, 01 Feb. 2020 (posts)
Masters Thesis (PDF). If an AI gives you a weird explanation for its prediction, you should remain septical about the accuracy of the prediction. Sounds reasonable? This was the general idea of my masters thesis, which was originally titled Self-Assessment of Visual Recognition Systems based on Attribution. Today, I would call it Explanation-based Anomaly Detection in Deep Neural Networks. The general idea was to use attribution-based explanation methods to detect anomalies (such as unusual …
Categories: Anomaly Detection
340 Words, Tagged with: Deep Learning · Anomaly Detection · CNN · Explainability
Thumbnail for Explanation-based Anomaly Detection in Deep Neural Networks