Data Mining

Did you know that the german parliament publishes protocols of all of its proceedings in PDF format? It is relatively straight-forward to download and parse them, so we can easily collect a dataset of transcripts of what seems to be every speech in the Bundestag since the second world war. My original idea was to mine the speeches for word associations: some words will be associated with other words, based on the intended connotation, and this association might change over time as the connotations change.

I like to read tagesschau.de, so I wrote a script to scrape it in regular intervals. My original goal was to determine which articles stay on the front page the longest, which ones allow commenting (a feature that seems to have been disabled almost entirely since March 2020), and if articles are modified after the initial release (without mentioning this), because I sometimes feel that headlines change. Dataset Creation Tagesschau.de provides a JSON API, so the “scraping” is relatively straight forward and can be done with just a few lines of code.

Our paper Social Work Research Map – ein niederschwelliger Zugang zu internationalen Publikationen der Sozialen Arbeit has been published in the journal Soziale Passagen. There should have been a video here but your browser does not seem to support it.

During the last weeks, I worked with some colleagues on a website that aims to improve access to social work literature. There should have been a video here but your browser does not seem to support it.

Inspired by the “Spiegel-Mining” talk from David Kriesel, a friend of mine and a Prof. from the Hochschule Magdeburg scraped a german website that regularly publishes reviews of social work literature, and mined the resulting 18.000 articles, hoping for interesting insights. In an attempt to visualize the discourse, we created several topic maps, like the one below, which you can find on the accompanying (german) website. The colours represent the gender of the authors of the review.