Mining tagesschau.de

I like to read tagesschau.de, so I wrote a script to scrape it in regular intervals.

My original goal was to determine which articles stay on the front page the longest, which ones allow commenting (a feature that seems to have been disabled almost entirely since March 2020), and if articles are modified after the initial release (without mentioning this), because I sometimes feel that headlines change.

Dataset Creation

Tagesschau.de provides a JSON API, so the “scraping” is relatively straight forward and can be done with just a few lines of code.

now = datetime.now()
date_time = now.strftime("%Y-%m-%d_%H_%M_%S")

url = f"https://www.tagesschau.de/api2/"
r = requests.get(url)
path = join(root, f"{date_time}.json")

if r.status_code == 200:
    data = r.content

    with open(path, "w") as f:
        f.write(data.decode())

I automatically ran this script once per hour for more then two years, which gave me $\approx$ 15,000 unique news articles.

Statistics

Now that we have a dataset, we can do some exploratory data analysis. For example, we can investigate when articles are published. Let’s plot the number of articles per weekday:

Apparently, more articles are published on Wednesday and Friday, while, during the weekend, the least articles are published. This sounds reasonable: less people work on the weekend, so there are fewer articles. But what is the reason for the spike on Fridays? Since the articles contain the exact publication date, we can plot the distribution of articles for each day, over each hour. The plot looks like this:

Here, we notice something interesting: Quite a lot of articles are published on Friday around 17:00 and 20:00. My hypothesis is that these are articles that the editorial staff pushed out so that people have stuff to read during the weekend.

Lets have a look at the length of the articles:

If the hypothesis is true, we could expect that the articles published on a Friday evening are longer then average. The histogram of the number of articles, plotted against the hour and the number of words in the articles looks like this:

This seems to support the hypothesis: Friday evening after the tagesschau has aired, an unusual amount of lengthy articles is published. This does not seem too far fetched.

Masked Language Modeling

Masked language modeling can be seen as a special kind of classification task. Given the previous and the next word, what is the probability of the masked word.

For example

The [mask] jumps over the lazy dog.  

we are trying to find the most probably word for [mask].

Or, in mathematical terms $$ p(\mathcal{D} \vert \theta) = \prod_{x \in \mathcal{D}} p(x_i | x_{j \neq i}, \theta) $$ where $\mathcal{D}$ is a dataset with a set of documents and $\theta$ are the parameters of our model.

Below is a picture.

Clustering

Below, you can find a clustering of the articles based on their content. Articles are vectorized by a german version of BERT, the visualization uses PCA and T-SNE. The color represents the category articles were assigned to. Using the categories as a sanity check, the clustering seem to work reasonably well. In fact, we can even find some articles that apparently have been categorized wrong.

Generative Language Modeling

We can use this dataset to create a fake news generator.

GPT-2

GPTagesschau

I fine-tuned a german GPT-2-based language model on the dataset to generate news articles in the style of tagesschau.de. The model is not that good yet, which is probably due to the facts that

• german pre-trained language models are not as good as their english counterparts and
• the dataset is to small (15k unique articles at the time of writing).

Still, the generated (fake) news articles are somewhat coherent, even it they tend to contradict themselves. The model is also able to generate titles and headlines.

In the following, I give an, in my opinion, rather funny example. Note that I did not select this article for its realism, but because I though the apparent mixture of two topics makes it an interesting read.