Our companion paper On Challenging Aspects of Reproducibility in Deep Anomaly Detection has been accepted for presentation on the Fourth Workshop on Reproducible Research in Pattern Recognition (satellite event of ICPR 2022).
In it, we discuss aspects of reproducibility for our anomaly detection algorithm MCHAD, as well as anomaly detection with deep neural networks in general. In particular, we discussed the following challenges for the reproducibility:
- Nondeterminism: conducting the same experiment with different random seeds might lead to significantly different outcomes.
- Sensitivity to hyper-parameters: slight changes in hyper-parameters can drastically alter the outcomes.
- Complexity: the more complex an algorithm, the more likely an implementation is to contain errors.
- Dataset Selection: the performance of a method is going to depend on the dataset you evaluate it on.
- Resource Limitations: resource requirements can limit the number of individuals or institutions that are able to reproduce the training.
- Dependencies: dependencies, in the form data, pre-trained weights, or software libraries, might get taken down at some point.
We found that the the large number of dependencies in out experiments may harm the reproducibility of our exact numerical results. However, we argue that the reproducibility of conclusions should be prioritized over the reproducibility of exact numerical results, since the former contributes to the advancement of scientific knowledge.
