Portal de Periódicos da CAPES

Artigo Produção Nacional Revisado por pares

BIBTEX RIS

Requirement patterns: a tertiary study and a research agenda

2019; Institution of Engineering and Technology; Volume: 14; Issue: 1 Linguagem: Inglês

10.1049/iet-sen.2019.0016

1751-8814

Taciana Novo Kudo, Renato F. Bulcão-Neto, Auri Marcelo Rizzo Vincenzi,

Advanced Software Engineering Methodologies

IET SoftwareVolume 14, Issue 1 p. 18-26 Review ArticleOpen Access Requirement patterns: a tertiary study and a research agenda Taciana N. Kudo, Corresponding Author Taciana N. Kudo taciana@dc.ufscar.br orcid.org/0000-0002-7238-0562 Instituto de Informática, Universidade Federal de Goiás, Goiânia-GO, Brazil Departamento de Computação, Universidade Federal de São Carlos, São Carlos, SP, BrazilSearch for more papers by this authorRenato F. Bulcão-Neto, Renato F. Bulcão-Neto Instituto de Informática, Universidade Federal de Goiás, Goiânia-GO, BrazilSearch for more papers by this authorAuri M.R. Vincenzi, Auri M.R. Vincenzi Departamento de Computação, Universidade Federal de São Carlos, São Carlos, SP, BrazilSearch for more papers by this author Taciana N. Kudo, Corresponding Author Taciana N. Kudo taciana@dc.ufscar.br orcid.org/0000-0002-7238-0562 Instituto de Informática, Universidade Federal de Goiás, Goiânia-GO, Brazil Departamento de Computação, Universidade Federal de São Carlos, São Carlos, SP, BrazilSearch for more papers by this authorRenato F. Bulcão-Neto, Renato F. Bulcão-Neto Instituto de Informática, Universidade Federal de Goiás, Goiânia-GO, BrazilSearch for more papers by this authorAuri M.R. Vincenzi, Auri M.R. Vincenzi Departamento de Computação, Universidade Federal de São Carlos, São Carlos, SP, BrazilSearch for more papers by this author First published: 01 February 2020 https://doi.org/10.1049/iet-sen.2019.0016Citations: 4AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The low performance of software projects generally arises from erroneous, omitted, misinterpreted, or conflicting requirements. To produce better quality specifications, the practise of requirements reuse through requirement patterns has been widely debated in the secondary literature. However, a tertiary study that provides an overview of secondary studies on the state of the art and the practise of requirement patterns does not exist. This study describes a study of secondary literature on requirement patterns under a perspective on research and practise. The identification and selection methods of secondary studies include automatic search on five sources, inclusion, and exclusion criteria, the snowballing technique, and the quality assessment of those studies. Four secondary studies are considered relevant according to the purpose of this research from a 26-distinct-study group. The authors' contribution is two-fold: the tertiary study itself and a preliminary research agenda dealing with state of the art and practise on requirement patterns. 1 Introduction While recognising the importance of requirements engineering in the software development process, recent studies [1, 2] demonstrate that a large percentage of software projects still extrapolate budget or deadlines or need to be cancelled. The primary cause of this poor performance is mainly due to the dependency on the customer and the interdisciplinary nature and inherent uncertainty of the requirements engineering process. This results in the bulk of incorrect, omitted, misinterpreted, or conflicting requirements. This kind of problem can be addressed through the requirements reuse practise [3], in which a new software project makes use of requirements knowledge obtained from previous projects. As stated by Wiegers and Beatty [4], effectively reusing trusted requirements can promote consistency both within and across applications, fewer defects, reduced rework, higher team productivity, lower development costs, and faster delivery. Among several proposals dealing with requirements reuse, the adoption of requirement patterns [5] has been much discussed in primary and secondary studies [6-9]. Instead of writing a requirement in natural language, a requirement pattern offers a systematic way to specify a particular type of requirement in the form of a template with categories of information. The structure and content of a requirement written as a pattern facilitate reuse. Increasingly, secondary studies in software engineering have been used as a valuable research tool [10-12] for providing knowledge about a subject as well as allowing the identification of gaps and themes for future research. However, before starting work on a secondary study, it is essential to know if such type of study on a particular topic already exists. If this occurs, a tertiary review may be conducted, in which the subject of interest includes secondary studies on that topic [13]. This paper describes a tertiary study aiming at investigating the comprehensiveness of research on requirement patterns, from the selection and synthesis of relevant secondary studies on this subject. The methodology used relies on both classic tertiary studies in software engineering [14, 15] and the tool support [16] on the planning and conduction of the study protocol. A gap between the state of the art and practise concerning requirement patterns is suggested by the results of our in-depth analysis of four relevant secondary studies. Such evidence allows us to propose a research agenda as a prelude to drive current and future research toward addressing this gap. As a tertiary study on requirement patterns does not exist, we elaborate and report the study protocol in such a way that one can easily replicate or augment it. The organisation of this paper is as follows. Section 2 details the study protocol and Section 3 reports the data extraction of the secondary studies selected. Section 4 presents the synthesis of such secondary studies considering the research questions of this tertiary study and Section 5 discusses the threats to the validity of this research. Section 6 introduces a research agenda to gear future works on requirement patterns and Section 7 outlines the contributions of this work. Finally, the Appendix shows the list of studies identified by our search strategy. 2 Tertiary study protocol As a tertiary review can adopt the same procedures used in secondary studies [17], we defined a process composed of three phases: planning, conducting, and publishing the results (see Fig. 1). Fig. 1Open in figure viewerPowerPoint Phases and activities of this tertiary study, adapted from [17] The planning phase of this tertiary study starts with the statement of the objective, followed by the definition of keywords. To find an adequate set of keywords, a pilot search has to be executed as many times as necessary. Next, the study protocol is created that, in turn, is evaluated to compose possible modifications, if necessary. The conduction phase encompasses the activities of identification and selection of secondary studies, data extraction, and synthesis. The search strategy as part of the study protocol allows the identification of the studies, whereas the selection of these relies on inclusion and exclusion criteria (EC) and assessment quality criteria for secondary studies, both previously defined in the study protocol. The data extraction activity starts as soon as the relevant secondary studies are selected. Next, a synthesis of these studies is performed to answer the research questions of the tertiary study. Then, synthesis results are reported as technical reports or scientific papers combining multiple formats such as textual, tabular, and graphical descriptions to name a few. In general, these documents are later disseminated and evaluated by expert readers. The iterative aspect of the whole process in Fig. 1 allows the refinement of phases and activities in such a way that the study protocol contains every information. The management of systematic studies generally counts on tool support; in our case, we have been using the StArt tool [16]. 2.1 Research questions and pilot search The primary goal of this tertiary study is to investigate the scope of research on requirement patterns, which is compiled in the following research questions (RQs): RQ1: What is state of the art in requirement patterns? RQ2: What are the most searched topics on requirement patterns? RQ3: What are the current gaps in requirement patterns research? A proper search strategy is required to answer these research questions based on the maximum of relevant secondary studies. The first step is to perform a pilot search to find a search string that balances search comprehensiveness and accuracy in the relevance of the retrieved studies [17]. During the pilot search, a good practise is the search string analysis to find the gold-standard set of papers as well as to aid in the definition of a more consistent protocol. Sensitivity and precision are two critical metrics [18] in the search string analysis: a highly sensitive search strategy retrieves most of the relevant studies, but it can also retrieve many irrelevant ones; a highly accurate search strategy retrieves a small number of irrelevant papers, but it can miss important ones. The ideal strategy is one that can be both sensitive and precise, capturing precisely the gold standard without unnecessary items. For that reason, our pilot search exploits the following synonyms related to the central research theme: requirement pattern, requirement reuse, requirement template, systematic review, systematic mapping, systematic literature review, systematic literature mapping, review of studies, structured review, literature analysis, in-depth survey, literature survey, meta-analysis, past studies, subject matter expert, analysis of research, empirical body of knowledge, overview of existing research, and body of published research. After the analysis of the pilot search results, we re-examine the set of keywords, as suggested in [17], from which we made the following observations: The term 'requirement reuse' often returns studies whose focus is not requirement patterns, but is requirements reuse. For instance, to support requirements reuse, some studies retrieved propose the reuse of use cases, requirements traceability, and the extraction of terms from requirements. Requirement patterns focused studies always have the term 'requirement pattern' in their content, but seldom have the term 'requirement reuse'. The term 'requirement template' shows up as keyword during the pilot search from studies in requirement patterns. The pilot search returns papers that describe literature surveys on requirement patterns. Dozens of papers whose knowledge area is not computer science arise because of some variants of the terms 'systematic literature', 'systematic mapping', and 'survey'. Besides, some information sources do not provide users with a filtering mechanism by knowledge area. Despite using a long list of synonyms for systematic literature study, as suggested in [13], many of these synonyms do not change search results – an iterative refinement was necessary to obtain a more concise search string. The analysis of the original set of keywords of the pilot search produces a 6.25% precision, i.e. three relevant papers from a group of 48 studies returned. After the test of keywords possibilities, a more balanced search string regarding sensitivity and precision is achieved: 4 relevant papers from a group of 26 papers retrieved, which means 15.4% of precision. Comparatively, one more relevant paper is retrieved, and almost half of unnecessary papers are rejected. On the basis of this analysis, we reach the final set of keywords arranged as the following search string: (('requirement pattern' OR 'requirement template') AND (survey OR 'systematic review' OR 'systematic literature review' OR 'systematic mapping' OR 'systematic literature mapping')) 2.2 Search strategy Once defined the string search, the next step is the choice of the ideal set of study sources applicable to the requirement patterns theme. Here is the list of sources selection criteria defined in this protocol: Sources considered relevant for the software engineering area. Sources with a search mechanism available on the web and logical expressions support. Sources that allow search over study abstract, at least. As a result, the sources chosen for this tertiary study include the following search engines and digital libraries: ACM Digital Library, Engineering Village, IEEE Xplorer, Science Direct, and Scopus. When necessary, the search string is tailored in response to the search capabilities of each source. In some sources, for example, variations of the terms 'requirement pattern' and 'requirement template' in the plural form are necessary. Owing to indexing a broader collection of papers, the search scope of the ACM Digital Library must be configured to The ACM Guide to Computing Literature. We also decide not to use temporal and idiom filtering on search results. The Appendix presents a list of 26 studies, which are referenced from now on as [Sn], resulting from this search strategy. In general, including duplicates, 40 studies are found and distributed as follows: ACM Digital Library (4), Engineering Village (13), IEEE Xplorer (2), Science Direct (11), and Scopus (10). 2.3 Studies selection This section describes the method used to identify which of the 26 non-duplicate studies remaining are relevant to answer the research questions of this tertiary study. The selection method is composed of inclusion criteria (IC) and EC [13], the snowballing technique [19], and quality assessment criteria (QC) [20]. 2.3.1 Selection criteria A set of six ECs were defined and applied to those 26 studies. The ECs are as follows: EC1: Study not published in journals or conference proceedings. EC2: The study does not cover computer science-related subjects. EC3: Requirement patterns are not the main subject of the study. EC4: It is not a secondary study. EC5: It is an informal study since this does not define research questions, the search strategy, and the processes of data extraction and synthesis. EC6: Full text not accessible. The exclusion of a study occurs when it falls into at least one of such ECs. If not excluded, the study should meet one of the following inclusion criteria (IC): IC1: The study features a systematic literature review. IC2: The study features a systematic literature mapping. IC3: The study features a literature survey. After reading the title, abstract, and keywords of every study, the IC and EC drive which study should be rejected or not. In cases when the reading of papers' metadata is not sufficient for decision making, a further reading of their introduction and conclusion sections is needed. As a result of the application of the IC and EC of the 40 studies identified in the automatic search, 14 are duplicates, 21 removed by EC, and 5 included. The number of papers identified, duplicated, excluded, and evaluated before data extraction is found in Fig. 2, which depicts the activities of identification and selection of secondary studies on requirements patterns of this tertiary study. Fig. 2Open in figure viewerPowerPoint Detailed view of the identification and selection processes of secondary studies Regarding duplicate studies, Table 1 lists the 14 duplicate papers per information source. The symbol represents each study instance excluded (14) because of its copies in more than one bibliographic database. The symbol, in turn, represents the instance of a duplicate study kept for the studies selection phase. Therefore, of the 26 studies identified through automatic search, 9 of them () have duplicates, and 17 do not. Observe also that automatic searches on Engineering Village and Scopus returned two instances of the study [S22] each, but only one of them went to the selection phase. This is because there are two versions of the same study [S22], both indexed by these two databases. Table 1. Duplicate studies per information source Study ACM Digital Library Engineering Village IEEE Xplorer Scopus [S1] — [S2] — — [S4] — — [S15] — — [S20] — — [S21] — — [S22] — — [S23] — [S24] — Concerning the 21 excluded studies Table 2 presents them grouped by EC. By analysing this table, the exclusion of five secondary studies is due to the fact they do not focus on requirement patterns (EC3). By also analysing their abstracts, we observe that these studies are about requirements reuse or specification, or software patterns, in general. None of these five secondary studies excluded by the EC3 criterion, therefore, deepens their research on requirement patterns, but only cite this term in their abstracts. Table 2. Studies excluded and the respective criterion Criterion Studies excluded EC1 [S1–S4] EC2 [S5–S7] EC3 [S8–S12] EC4 [S13–S19] EC5 [S20, S21] After the application of the EC, five studies remain, among them, a study [S23], which after being read in detail is also considered as duplicate, since the research questions and experiment are very similar to a same authors' another study [S24]. Finally, after applying the inclusion and EC, the following four secondary studies remain: [S22, S24–S26], which refer to [8, 9, 21, 22]. 2.3.2 Snowballing The snowballing technique [19] allows the identification of relevant studies by scanning the list of bibliographic references or citations of a paper. When you examine the former, this is called backward snowballing; in the case of the latter, it is called forward snowballing. The backward snowballing technique applied to the four secondary studies selected [S22, S24–S26] complements the search strategy of this tertiary study, besides automatic searching. The list of bibliographic references of these studies is then analysed to check whether other relevant secondary studies on requirement patterns exist or not. From this analysis, one identifies that [S25] see the [S22] and [S24] works, which demonstrates consistency among these three studies. Fig. 3 illustrates a new secondary study [3] found after backward snowballing over [S24] work. This new study is missing in the initial selection phase because of the search string used (only the term 'survey' is found). Even after reading the title, abstract, and keywords of this study, the decision for its inclusion or not happens only after reading the introduction and conclusion sections. Chernak's work [3] reports research on requirements reuse in practise, but it does not cite requirement patterns at any point in its analysis. Therefore, this study was excluded by the EC3 criterion because it did not have requirement patterns as the central theme. Fig. 3Open in figure viewerPowerPoint Backward snowballing result: a new secondary study [3] is identified from [S24] 2.3.3 Quality assessment of secondary studies An important decision when performing systematic literature studies is to check the quality of primary or secondary studies. The Centre for Reviews and Dissemination (CDR) [20] maintains a database of systematic reviews in medicine selected according to pre-established quality criteria. These same quality criteria can also be used to assess systematic studies regardless of knowledge area, as some works reuse them in software engineering [13, 15]. In this tertiary study, we analyse each secondary study from the light of four questions such as QC based on the CDR criteria. The acceptable responses for each question are Y (Yes), P (Partially), and N (Not). The following is the list of quality assessment-related questions. QC1: Are the IC and EC appropriately described? Y: Criteria are explicit. P: Criteria are implicit. N: Criteria are not defined or not easily identified. QC2: Does the search cover all relevant studies? Y: It uses four or more sources relevant and one additional search strategy. P: It uses three sources relevant, but no extra search strategy. N: It uses at most two sources relevant to the area of interest. QC3: Is the quality or validity of the included primary studies assessed? Y: Quality criteria are explicit and associated with each primary study. P: Research questions from the secondary study address the quality of primary studies. N: No quality assessment. QC4: Are primary studies adequately described? Y: Details of each primary study are explicit. P: There is only a summary of each primary study. N: There are no results of the primary studies analysed. Considering the response to each question, we assign the following score: 1 (one) for each 'Yes' response; 0.5 (medium) to 'Partly'; and 0 (zero) for 'No'. This benchmark evaluates the scores of the secondary studies selected for data extraction [S22, S24–S26]. The final quality score of each secondary study is the arithmetic mean of the scores for its quality questions. The importance of a final quality score is two-fold: to reject studies that do not meet quality criteria defined in the study protocol and to guide the synthesis activity of the secondary studies. In this tertiary study, we reject those studies that scored 0 (zero) on all quality criteria described (QC1–QC4). Table 3 presents the final quality score for each secondary study selected. Note that there is no study excluded since all studies score higher than zero on at least one of the quality questions. It is worth mentioning that the highest quality score study [S24] includes a literature review that explicitly defines the search strategy, search string, information sources, studies selection criteria, analysis, and synthesis. Table 3. Quality score of secondary studies for extraction, in descending order Secondary study QC1 QC2 QC3 QC4 Score Palomares et al. [S24] 1 1 1 1 1 Barros-Justo et al. [S25] 1 1 0.5 1 0.87 Irshad et al. [S26] 1 0 1 0.5 0.62 Silva and Benitti [S22] 1 0.5 0 0 0.37 3 Data extraction This section describes information extracted from the full-text reading of the secondary studies selected, which includes: The main objective and respective research questions. The selection methods of primary studies. The evidence collected from the synthesis of these studies. The four secondary studies selected for data extraction and synthesis dates from 2011, 2017, and 2018: three of them published in international journals and one in an international workshop. Following the order of quality score of the secondary studies, Table 4 presents the respective objectives, search strategies (A for automatic, M for manual, and S for snowballing), and the number of sources and primary studies analysed. Table 4. Data extracted from secondary studies: number of sources, search strategy, and number of primary studies analysed Secondary study Sources Search strategy Primary studies Palomares et al. [S24] 4 A M 11 venue: empirical software engineering: an international journal objective: map state of the practise of requirements reuse; identify why proposals of requirements reuse are not used in practise string: reuse AND requirements Barros-Justo et al. [S25] 4 A M S 22 venue: computer standards and interfaces journal objective: detect patterns reported in scientific research; identify requirements engineering activities impacted by the use of requirement patterns; point out properties of the software development process or the final product affected by the use of requirement patterns string: pattern AND engineering AND (software OR requirement) Irshad et al. [S26] 2 A S 14 venue: information and software technology journal objective: identify different approaches of requirements reuse; discover evaluation methods and the respective results for requirements reuse approaches; and the relevance and rigour of the empirical studies of requirements reuse approaches string: requirements AND reuse Silva and Benitti [S22] 14 A 3 venue: workshop on requirements engineering objective: map current standards for writing software requirements string: (pattern OR catalogue OR category) AND ('software requirement' OR 'requirements engineering' OR 'requirements elicitation') Palomares et al. [S24] map the state of practise of requirements reuse and patterns through a literature review that presents search strategy, search string, studies selection criteria, synthesis, and a conclusion of the analysis made. The sources of studies used are ACM Digital Library, IEEE Xplorer, Science Direct, and Springer Link. The review results indicate an unusual practise of requirements reuse and patterns in industry. Authors also look for surveys and interviews on the state of the art of requirements reuse in companies and identify the need for work that gives more evidence on the positive and negative points of the different abstractions and artefacts in the requirements reuse proposals. For this reason, authors carry out an exploratory survey with 71 requirements engineers that answer an online questionnaire on both requirements reuse and the use of requirement patterns in industry practise. As a result, the respondents identify problems that the use of requirement patterns can solve such as specification completeness and requirements uniformity, inconsistency, and ambiguity. In other secondary study relevant for this tertiary review, Barros-Justo et al. [S25] present a systematic mapping to find the software patterns generally reported in scientific research, the requirements engineering activities impacted by the use of these patterns, and which software development or end product properties are affected by the use of software patterns. The authors do not specify the type of software pattern (e.g. requirement pattern or design pattern); the point is whether the industry makes use of software patterns or not. The search strategy for the selection of primary studies includes manual and automatic search, the snowballing technique, and the sources listed next: ACM Digital Library, IEEE Xplorer, Scopus, and Web of Science. The next secondary study analysed is Irshad et al. [S26] that presents a systematic review of research on requirements reuse. The main result is the creation of 11 groups with different reuse approaches, of which three of these focus on requirement patterns: (i) a structure group, with studies on requirements organisation to facilitate reuse; (ii) a template group, with studies that provide a guide for writing requirements in a specific format; and (iii) a hybrid group, with studies that use a combination of different approaches to support requirements reuse. In this recent study, the main source of primary studies is the Google Scholar search engine, and the search strategy also includes snowball sampling. However, these authors carry out a sanity check searching on the Scopus database. The point is to check the existence of new primary studies not found through Google Scholar. These authors also recognise that metamodels can be used to structure and facilitate the reuse of requirement patterns. They conclude that most of the studies proposing the use of templates also suggest the use of natural language. Additionally, there are studies on ontologies and templates of use cases intended for requirement patterns reuse. As for the hybrid approach, one primary study uses a catalogue of requirements to facilitate requirements traceability, whereas other study makes use of families of applications to identify requirement patterns and, in turn, for easier requirements reuse. Finally, Silva and Benitti [S22] identify and compare the available approaches to requirements writing. Regarding the set of evidence found as a result of this systematic mapping, requirement patterns support the specification of both non-functional and functional requirements: non-functional requirements writing based on categories; functional requirements writing in the form of information systems requirements; and the occurrence of composite requirement patterns. The search strategy of this work includes automatic search on 14 sources including ACM Digital Library, IEEE Xplorer, Science Direct, Springer Link, and Brazilian thesis and dissertations digital libraries, to name a few. 4 Data synthesis This section presents a synthesis of the data extracted concerning the research questions of this tertiary study. 4.1 About the research question 1 To answer the question 'What is the state of the art in requirement patterns?', we made the following observations. 4.1.1 Research scope Barros-Justo et al. [S25] define a comprehensive search string and, even using only four sources, they identify a large number of non-relevant studies. In this case, the ratio of relevant and identified primary studies is 0.7% (22 out of 3070). On the one hand, Silva and Benitti [S22] define a broader string concerning the keywords used, and even using 14 sources, they identify few studies (56) in comparison with Barros-Justo et al. [S25]. However, only three primary studies among the 56 identified (∼5%) are relevant. Palomares et al. [S24], in turn, analyse 22 papers on requirements reuse, of which 11 are specific to requirement patterns. Irshad et al. [S26] make use of solely the Google Scholar search engine, but with an extensive search string ('requirements' AND 'reuse') to retrieve studies on requirements reuse, in general. Even so, 14 primary studies for data synthesis are selected. After analysing the set of primary research reported on those fou