RRRR 2022

 |  Programme  |  Submissions  |  Timeline  |  Organisers

The First Workshop on Reproducibility and Replication of Research Results (RRRR 2022) is part of ETAPS 2022 in Munich, Germany. It provides a forum to present novel approaches to foster reproducibility of research results, and replication studies of existing work, in the broad area of formal methods research. Its goal is to spread the word on best practices, and reward the work invested in replicating results.

Where reproducible research can be independently confirmed by third parties using artifacts provided by the original authors, replicating a result means to independently obtain it using new measurements, data, or implementations. RRRR will improve the knowledge transfer between the many separate initiatives – like artifact evaluations and tool competitions – that today support reproducibility, and provide a venue to formally publish replication studies, recognising their immense benefit to the scientific community and the hard work involved.


RRRR 2022 takes place on Saturday, April 2, 2022, as part of the ETAPS workshops at the computer science building at TU Munich in Garching. The programme for the day is as follows:

10:00-10:30 Coffee break
10:30-10:40 Opening
(Arnd Hartmanns)
10:40-12:30 Session 1: Reproducibility (chair: Matthias Volk)
System Configurations Matter – An Experience Report on Reproducibility
(Clemens Dubslaff)
Experience Report: Research Result Reproduction in a Grad Student Course [PDF]
(Dirk Beyer and Stefan Winter)
Reproducible Cluster(less) Speedup Analysis [PDF]
(Maximilian Heisinger and Martina Seidl)
The Artifact Evaluation Recommended Practice Compendium (AERPC) [PDF]
(Dirk Beyer and Stefan Winter)
Discussion: The Future of Artifacts (moderator: Dirk Beyer)
12:30-14:00 Lunch
14:00-16:00 Session 2: Replication (chair: Benjamin Lucien Kaminski)
Deterministic State Space Exploration [PDF]
(Martin Nowack)
Reimplementing and Evaluating Static Fault Tree Analysis using BDDs [PDF]
(Daniel Basgöze)
What is the Best Algorithm for Probabilistic Model Checking? [PDF]
(Tim Quatmann)
Backwards Reachability for Probabilistic Timed Automata: A Replication Report [PDF]
(Arnd Hartmanns and Bram Kohlen)
Replication Study: Defining Cyber Risk [PDF]
(Justin Hogg, Nawaris Almerza and Dawn Childs)
16:00-16:30 Coffee Break
16:30-17:30 Session 3: R & R for Theory (chair: Erika Ábrahám)
Reliably Reproducing Machine-Checked Proofs with the Coq Platform [PDF]
(Karl Palmskog, Enrico Tassi and Théo Zimmermann)
Replicable Theory [PDF]
(Benjamin Lucien Kaminski)
Discussion: R & R for Theory (moderator: Arnd Hartmanns)
17:30-17:40 Closing
(Dirk Beyer)


RRRR invited submissions in two topics and two formats. All accepted submissions are collected informally as PDFs in the programme above. Submissions were via EasyChair.


Submissions were encouraged in, but not limited to, two main topics:

Submissions should be related to research in the broad area of formal methods that would be in scope for one of the ETAPS conferences in principle. It is not required that the earlier work that a replication study is based on has been published at ETAPS.

Submission Types

RRRR accepted two types of submissions:

The page limits are to be interpreted as guidelines, not as absolute limits. Short papers and presentation abstracts were selected by the organisers. We encouraged, but did not require, short papers to be accompanied by an artifact for reproduction. Authors of all accepted submissions presented their work at the workshop. We encouraged demonstrations to be integrated in presentations, in particular for replication studies.

Special Issue in STTT

Authors of accepted papers and abstracts will be invited to submit an extended/full version of their work to a special issue in the International Journal on Software Tools for Technology Transfer (STTT). Where applicable, we expect these journal submissions to be accompanied by an artifact for reproduction that will be permanently archived with a DOI upon publication. All journal submissions will undergo a full reviewing process.


All dates are in 2022 and all deadlines are "anywhere on Earth" (UTC-12).


Dirk Beyer (Ludwig-Maximilians-Universität München, Germany),
Arnd Hartmanns (University of Twente, The Netherlands)