Case study is a suitable research methodology for software engineering research since it studies contemporary phenomena in its natural context. However, the understanding of what constitutes a case study varies, and hence the quality of the resulting studies. This paper aims at providing an introduction to case study methodology and guidelines for researchers conducting case studies and readers studying reports of such studies. The content is based on the authors' own experience from conducting and reading case studies. The terminology and guidelines are compiled from different methodology handbooks in other research domains, in particular social science and information systems, and adapted to the needs in software engineering. We present recommended practices for software engineering case studies as well as empirically derived and evaluated checklists for researchers and readers of case study research.

/pdf/guidelines-for-conducting-and-reporting-case-study-research-q10xdzyou5.pdf

Guidelines for conducting and reporting case study research in software engineering

BACKGROUND: A software engineering systematic map is a defined method to build a classification scheme and structure a software engineering field of interest. The analysis of results focuses on frequencies of publications for categories within the scheme. Thereby, the coverage of the research field can be determined. Different facets of the scheme can also be combined to answer more specific research questions.

OBJECTIVE: We describe how to conduct a systematic mapping study in software engineering and provide guidelines. We also compare systematic maps and systematic reviews to clarify how to chose between them. This comparison leads to a set of guidelines for systematic maps.

METHOD: We have defined a systematic mapping process and applied it to complete a systematic mapping study. Furthermore, we compare systematic maps with systematic reviews by systematically analyzing existing systematic reviews.

RESULTS: We describe a process for software engineering systematic mapping studies and compare it to systematic reviews. Based on this, guidelines for conducting systematic maps are defined.

CONCLUSIONS: Systematic maps and reviews are different in terms of goals, breadth, validity issues and implications. Thus, they should be used complementarily and require different methods (e.g., for analysis).

/pdf/systematic-mapping-studies-in-software-engineering-4h4l7vgsw3.pdf

Systematic mapping studies in software engineering

Agile software development represents a major departure from traditional, plan-based approaches to software engineering. A systematic review of empirical studies of agile software development up to and including 2005 was conducted. The search strategy identified 1996 studies, of which 36 were identified as empirical studies. The studies were grouped into four themes: introduction and adoption, human and social factors, perceptions on agile methods, and comparative studies. The review investigates what is currently known about the benefits and limitations of, and the strength of evidence for, agile methods. Implications for research and practice are presented. The main implication for research is a need for more and better empirical studies of agile software development within a common research agenda. For the industrial readership, the review provides a map of findings, according to topic, that can be compared for relevance to their own settings and situations.

/pdf/empirical-studies-of-agile-software-development-a-systematic-4kky3l7vnd.pdf

Empirical studies of agile software development: A systematic review

Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.Objective: This paper presents guidelines for conducting literature reviews using a snowballing approach, and they are illustrated and evaluated by replicating a published systematic literature review.Method: The guidelines are based on the experience from conducting several systematic literature reviews and experimenting with different approaches.Results: The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.Conclusions: It is concluded that using snowballing, as a first search strategy, may very well be a good alternative to the use of database searches.

/pdf/guidelines-for-snowballing-in-systematic-literature-studies-106rgo5oni.pdf

Guidelines for snowballing in systematic literature studies and a replication in software engineering

Like other sciences and engineering disciplines, software engineering requires a cycle of model building, experimentation, and learning. Experiments are valuable tools for all software engineers who are involved in evaluating and choosing between different methods, techniques, languages and tools. The purpose of Experimentation in Software Engineering is to introduce students, teachers, researchers, and practitioners to empirical studies in software engineering, using controlled experiments. The introduction to experimentation is provided through a process perspective, and the focus is on the steps that we have to go through to perform an experiment. The book is divided into three parts. The first part provides a background of theories and methods used in experimentation. Part II then devotes one chapter to each of the five experiment steps: scoping, planning, execution, analysis, and result presentation. Part III completes the presentation with two examples. Assignments and statistical material are provided in appendixes. Overall the book provides indispensable information regarding empirical studies in particular for experiments, but also for case studies, systematic literature reviews, and surveys. It is a revision of the authors book, which was published in 2000. In addition, substantial new material, e.g. concerning systematic literature reviews and case study research, is introduced. The book is self-contained and it is suitable as a course book in undergraduate or graduate studies where the need for empirical studies in software engineering is stressed. Exercises and assignments are included to combine the more theoretical material with practical aspects. Researchers will also benefit from the book, learning more about how to conduct empirical studies, and likewise practitioners may use it as a cookbook when evaluating new methods or techniques before implementing them in their organization.

Experimentation in Software Engineering

Background: In 2004 the concept of evidence-based software engineering (EBSE) was introduced at the ICSE04 conference. Aims: This study assesses the impact of systematic literature reviews (SLRs) which are the recommended EBSE method for aggregating evidence. Method: We used the standard systematic literature review method employing a manual search of 10 journals and 4 conference proceedings. Results: Of 20 relevant studies, eight addressed research trends rather than technique evaluation. Seven SLRs addressed cost estimation. The quality of SLRs was fair with only three scoring less than 2 out of 4. Conclusions: Currently, the topic areas covered by SLRs are limited. European researchers, particularly those at the Simula Laboratory appear to be the leading exponents of systematic literature reviews. The series of cost estimation SLRs demonstrate the potential value of EBSE for synthesising evidence and making it available to practitioners.

/pdf/systematic-literature-reviews-in-software-engineering-a-5ez032qyiz.pdf

Systematic literature reviews in software engineering - A systematic literature review

This paper describes the results of a process whereby the detection capability of 5 code replication detection tools for large software application are evaluated. Specifically this work focuses on the benefits of identification for preventative maintenance that is with the aim to remove some of the identified clones from the source code. A number of requirements are therefore identified upon which the tools are evaluated. The results of the analysis processes show that each tool has its own strengths and weaknesses and no single tool is able to identify all clones within the code. The paper proposes that it may be possible to use a combination of tools to perform the analysis process providing that adequate means of efficiently identifying false matches is found.

/pdf/evaluating-clone-detection-tools-for-use-during-preventative-4qs9jgj386.pdf

Evaluating clone detection tools for use during preventative maintenance

Following these papers, staff at the Keele University School of Computing and Mathematics proposed a research project to investigate the feasibility of EBSE This proposal was funded by the UK Economics and Physical Science Research Council (EPSRC) The proposal was amended to include the Department of Computer Science, University of Durham when Professor David Budgen moved to Durham The EPSRC have now funded a joint Keele and Durham follow-on project (EPIC)

Protocol for a Tertiary study of Systematic Literature Reviews and Evidence-based Guidelines in IT and Software Engineering

There is little empirical knowledge of the effectiveness of the object-oriented paradigm. To conduct a systematic review of the literature describing empirical studies of this paradigm. We undertook a Mapping Study of the literature. 138 papers have been identified and classified by topic, form of study involved, and source. The majority of empirical studies of OO concentrate on metrics, relatively few consider effectiveness.

/pdf/evidence-relating-to-object-oriented-software-design-a-kgij4bemkd.pdf

Evidence relating to Object-Oriented software design: A survey

Context: Secondary studies, such as systematic literature reviews and mapping studies, are an essential element of the evidence-based paradigm. A critical part of the review process is the identification of all relevant research. As such, any researcher intending to conduct a secondary review should be aware of the strengths and weakness of the search engines available. Objectives: Analyse the overlap between search engine results for software engineering studies. Methods: Three independent studies were conducted to evaluate the overlap between multiple search engines for different search areas. Results: The findings indicate that very little overlap was found between the search engines. Conclusions of the study: To complete a systematic review, researchers must use multiple search terms and search engines. The lack of overlap might also be caused by inconsistent keyword selection amongst authors.

John W. Bailey

Papers

Systematic literature reviews in software engineering - A systematic literature review

Evaluating clone detection tools for use during preventative maintenance

Protocol for a Tertiary study of Systematic Literature Reviews and Evidence-based Guidelines in IT and Software Engineering

Evidence relating to Object-Oriented software design: A survey

Search Engine Overlaps: Do they agree or disagree?