Reproducibility Checklist for Software Engineering Experiments

Reproducibility has become one of the most important quality indicators in modern software engineering research. Whether a study focuses on artificial intelligence, cloud systems, cybersecurity, IoT, distributed computing, or empirical software engineering, researchers are increasingly expected to provide sufficient experimental detail so that independent researchers can verify, replicate, and extend the reported findings.

For journals such as Ubiquitous Technology Journal (UTJ) under Crosslink Studies(CLS0 reproducibility aligns directly with the journal’s emphasis on methodological rigor, scientific transparency, and high-quality computer science research. UTJ particularly encourages technically sound and well-documented studies across areas such as artificial intelligence, pervasive computing, IoT, edge computing, cybersecurity, distributed systems, and software engineering.

Why Reproducibility Matters in Software Engineering Research

In software engineering experiments, results are heavily influenced by software environments, dataset versions, hardware configurations, programming frameworks, experimental procedures, hyper parameter settings, random initialization and evaluation metrics. Without clear documentation, even strong research contributions may become difficult to validate or reuse.

Reproducibility strengthens scientific credibility, peer-review confidence, experimental reliability, long-term research impact and collaboration opportunities.

What Is Reproducibility in Software Engineering?

Reproducibility refers to the ability of independent researchers to recreate the experimental workflow and obtain comparable outcomes using the same methodology and research artifacts.

A reproducible study should allow others to access datasets, rebuild the environment, execute the experimental pipeline, reproduce evaluation procedures and validate reported findings.

Reproducibility Checklist for Software Engineering Experiments

1. Clearly Define the Research Objective

Every experiment should begin with a precise research goal. It includes problem statement, research questions, hypotheses, experimental objectives and scope and limitations. Clear objectives improve experimental structure and evaluation consistency.

Example

“This study evaluates the impact of automated defect prediction models on software maintenance efficiency.”

2. Describe the Experimental Environment

One of the most common reproducibility failures is incomplete environment documentation.

Authors should report:

Component	Required Details
Operating System	Windows, Linux, macOS
Hardware	CPU, GPU, RAM
Frameworks	TensorFlow, PyTorch, Scikit-learn
Programming Language	Python, Java, C++
Library Versions	Exact package versions
Database Systems	MySQL, PostgreSQL, MongoDB
Containerization	Docker or virtualization tools

Even minor version differences may alter experimental outcomes.

3. Provide Dataset Information

Datasets should be thoroughly documented.

Include dataset source, access links, dataset version, preprocessing methods, cleaning procedures, feature engineering techniques and train-test-validation split ratios.

Best Practice

If permitted, provide public dataset access or repository links. Transparent datasets significantly improve replicability.

4. Document Algorithms and Model Configurations

Research manuscripts should clearly explain algorithms used, system architecture, model structure, feature selection methods and optimization strategies.

For AI-based software engineering studies, include learning rate, batch size, epochs, optimizer, loss functions and random seed values. Insufficient algorithmic detail is one of the major causes of irreproducible AI experiments.

5. Explain the Experimental Procedure

A reproducible paper should describe the experiment step-by-step. It includes data preparation, system setup, execution workflow, validation procedure, statistical analysis and performance evaluation. Readers should understand exactly how the experiment was conducted from beginning to end.

6. Report Evaluation Metrics Clearly

Metrics must be explicitly defined and justified. Common software engineering metrics include accuracy, precision, recall, F1-score, Mean Absolute Error (MAE), Root Mean Square Error (RMSE), execution time and scalability measurements. Also explain why specific metrics were selected.

7. Include Statistical Validation

Experimental claims should be statistically supported. Recommended practices include confidence intervals, significance testing, effect size analysis and cross-validation.

8. Share Source Code and Artifacts

Open science practices are increasingly encouraged in software engineering research. Authors should provide source code repositories, configuration files, scripts, documentation, installation instructions. Public repositories improve transparency and accelerate future research.

9. Report Threats to Validity

Professional software engineering manuscripts usually include a dedicated “Threats to Validity” section.

This should discuss:

Validity Type	Description
Internal Validity	Experimental bias
External Validity	Generalizability
Construct Validity	Measurement accuracy
Conclusion Validity	Statistical correctness

10. Ensure Long-Term Accessibility

Research artifacts should remain accessible beyond publication. Long-term accessibility supports sustainable scientific progress.

Common Reproducibility Mistakes

Many manuscripts face reviewer criticism because they omit software versions, use undocumented pre-processing steps, fail to share datasets, exclude hyper parameter details, provide incomplete evaluation methods, ignore statistical analysis and lack experimental workflow explanations.

Reproducibility and Peer Review Success

Well-documented experiments help reviewers validate findings efficiently, assess methodological quality, compare results with prior work, evaluate technical rigor and trust reported conclusions.  Studies with stronger reproducibility practices often receive fewer revision requests and improved reviewer confidence.

Alignment with Modern Software Engineering Research Standards

Top software engineering journals increasingly promote:

• Open science
• Transparent experimentation
• Artifact sharing
• Reproducible workflows
• Structured reporting standards

Empirical software engineering literature consistently highlights the importance of reporting guidelines and reproducibility frameworks for improving research quality.

For authors submitting to Ubiquitous Technology Journal (UTJ), applying reproducibility principles can substantially strengthen the scientific reliability and professional presentation of their manuscripts.