RRRR 2024

The Third Workshop on Reproducibility and Replication of Research Results (RRRR 2024) is part of ETAPS 2024 in Luxembourg. It will take place on April 6, 2024, as part of the ETAPS workshops.

The RRRR workshop provides a forum to present novel approaches to foster reproducibility of research results, and replication studies of existing work, in the broad area of formal methods research. Its goal is to spread the word on best practices, and reward the work invested in replicating results.

Where reproducible research can be independently confirmed by third parties using artifacts provided by the original authors, replicating a result means to independently obtain it using new measurements, data, or implementations. RRRR will improve the knowledge transfer between the many separate initiatives – like artifact evaluations and tool competitions – that today support reproducibility, and provide a venue to formally publish replication studies, recognising their immense benefit to the scientific community and the hard work involved.

Programme

RRRR 2024 takes place on Saturday, April 6, 2024, as part of the ETAPS workshops at Parc Hotel Alvisse in Luxembourg. The programme for the day is as follows:

Submissions

RRRR invited submissions in two topics and two formats. Accepted submissions are collected informally as PDFs on this website, linked in the programme above. Submissions were handled via EasyChair.

Topics

Submissions were encouraged in, but not limited to, two main topics:

• Reproducibility initiatives: Novel developments in, or retrospective studies on, initiatives supporting and strengthening the reproducibility of research, such as artifact evaluations. These should go beyond the pure presentation of an initiative, by e.g. scientifically discussing the context, comparing with other approaches, and/or justifying the novelty/expected impact of the contribution.
• Reproduction reports and replication studies: Reports on successful or failed attempts at reproducing or replicating earlier research results. These reports should precisely describe what could and could not be reproduced/replicated, fill in gaps in the original publication if necessary, and where applicable come with a reproducibility package (artifact).

Submissions should be related to research in the broad area of formal methods that would be in scope for one of the ETAPS conferences in principle. It is not required that the earlier work that a reproduction report or replication study is based on has been published at ETAPS.

Submission Types

RRRR accepted two types of submissions:

• Presentation abstracts (1-2 pages in LNCS style including references). These may cover work in progress, or work related to RRRR that has recently been published at a conference or in a journal.
• Short papers (around 6 pages in LNCS style plus references). These should describe finished, not-yet-published work.

The page limits are to be interpreted as guidelines, not as absolute limits. Short papers and presentation abstracts were reviewed by the organisers. We encouraged short papers to be accompanied by an artifact for reproduction. Authors of all accepted submissions presented their work at the workshop. We encouraged demonstrations to be integrated in presentations, in particular for replication studies.

Special Issue in STTT

We plan to invite authors of accepted papers and abstracts to submit an extended/full version of their work to a special issue in the International Journal on Software Tools for Technology Transfer (STTT). Where applicable, we expect these journal submissions to be accompanied by an artifact for reproduction that will be permanently archived with a DOI upon publication. All journal submissions will undergo a full reviewing process.

Timeline

All dates are in 2024 and all deadlines are "anywhere on Earth" (UTC-12). The organisers appreciate an early informal indication, via , of the intention to submit to RRRR 2024.

• February 26: Submission deadline
• March 4: Acceptance notification
• March 18: Final versions for informal publication on website
• April 6: Workshop day

Organisers

Dirk Beyer (Ludwig-Maximilians-Universität München, Germany),
Arnd Hartmanns (University of Twente, The Netherlands)