Reproducibility | kkirchheim.de

When we draw conclusions from an experiment, we would like the conclusions to also hold in a different experimental setting; that is: slight changes in the initial conditions should not falsify our hypotheses about the underlying mechanism.

In experiments involving deep learning models, it has been observed that results can vary drastically, depending on different sources of non-determinism that are often not accounted for, which can ultimately lead to findings that are not reproducible. My research in this area focuses on investigating these effects and to improve and incentivize the reproducibility of experiments.

Our companion paper On Challenging Aspects of Reproducibility in Deep Anomaly Detection has been accepted for presentation on the Fourth Workshop on Reproducible Research in Pattern Recognition (satellite event of ICPR 2022). In it, we discuss aspects of reproducibility for our anomaly detection algorithm MCHAD, as well as anomaly detection with deep neural networks in general. In particular, we discussed the following challenges for the reproducibility: Nondeterminism: conducting the same experiment with different random seeds might lead to significantly different outcomes.

Our Paper Addressing Randomness in Evaluation Protocols for Out-of-Distribution Detection has been accepted at the ICJAI 2021 Workshop for Artificial Intelligence for Anomalies and Novelties. In summary, we investigated the following phenomenon: when you train neural networks several times, and then measure their performance on some task, there is a certain variance in the performance measurements, since the results of experiments may vary based on several factors (that are effectively controlled by the random seed).