Sunday, May 29, 2016

Insights for practitioners into empirical software architecture research


By: Matthias Galster, Danny Weyns
Associate Editor: Mehdi Mirakhorli (@MehdiMirakhorli)

Software architecture is an applied discipline. Therefore, research in that area should eventually offer practitioners useful and trustworthy results.

What we did: To find out more about the state of empirical software architecture research and its relevance for practitioners, we looked at research papers published between 1999 and 2015 at the most popular architecture conferences:
·       CBSE (International ACM SIGSOFT Symposium on Component-based Software Engineering)
·       ECSA (European Conference on Software Architecture)
·       QoSA (International ACM SIGSOFT Conference on the Quality of Software Architectures)
·       WICSA (Working IEEE/IFIP Conference on Software Architecture)
We went through a total of 667 “full” papers out of which 115 (i.e., around 17%) turned out to be “empirical”. In other words, 17% presented research based on systematically collected evidence from observation or experience [1], using methods such as case studies, experiments or surveys (the remaining 83% of papers present solutions or new methods without evaluation, experience reports, tools, etc.). The full study has been published as a research paper at WICSA 2016 [2] and a preprint is available [5]. In this post, we briefly highlight some of the insights.

Insight 1 – Amount of empirical work is increasing: Since around 2010, there is an increasing trend to publish empirical research in software architecture. The figure below shows the relative contribution of empirical papers compared to all papers published in a year.
Empirical methods are used equally to (a) to study phenomena (e.g., to understand how architecture decisions are made in practice) and (b) to evaluate approaches (e.g., to evaluate a new approach for capturing architectural knowledge). Therefore, current empirical research helps us understand software architecture practice (for example, to identify the real problems that research should tackle), but also contributes to increasing the confidence in the validity of newly proposed approaches and methods.



Insight 2 – Involvement of practitioners as researchers/co-authors is limited: On average, every two papers have one industry author but five authors from academia. Around 70% of papers have no industry author at all. Overall, empirical papers have roughly the same number of authors from industry than non-empirical papers. Little active involvement of practitioners (assuming that co-authors are actively involved in the research presented in a paper) might be due to challenges related successful industry-academia collaborations (e.g., champions in industry organizations, buy-in support commitment to contribute to industry needs) [3].

Insight 3 – Involvement of practitioners as study participants increases: Half of the empirical studies involve humans as subjects or study participants (e.g., in as participants in case studies or respondents of surveys). Most of these studies (68%) involve practitioners from industry (with the exception of experiments where students are the dominating participants). This involvement of practitioners is one way to increase the applicability and relevance of research findings for practice.

Insight 4 – Trustworthiness and rigor vary: Most studies (60%) discuss validity threats and acknowledge limitations. On the other hand, we should be careful when interpreting research findings: Many so-called “evaluations” presented in papers are often only illustrations using toy examples. Also, replications to confirm or refute previous findings and to build our confidence in research outcomes are almost non-existent in software architecture research. There might be a misconception in the research community that replications have little scientific value.

Insight 5 – There is a potential mismatch between research topics and trends in industry: Component selection and composition, architecture reasoning and decisions are dominating themes in empirical papers in the period 1999-2015. Michael Keeling in his reflection on emerging trends at SATURN [4] found architecting for DevOps, flexible design, lightweight architecture design methods and a renewed interest in architecture fundamentals as trends at SATURN. We couldn’t find these topics as trends in empirical papers.

Summary: Applying empirical methods puts an increasing demand on planning, conducting and evaluating research and its results, and presenting it in a way that is accessible and meaningful for practitioners. On the other hand, empirical research allows practitioners to get actively involved in collaborations with academics, both to explore emerging phenomena of interest (and to find out about the real problems in software architecture practice) and to evaluate new approaches that address practically relevant problems.

References:
  1. D. Sjoberg, T. Dyba, and M. Jorgensen, "The Future of Empirical Methods in Software Engineering Research," in Future of Software Engineering (FOSE) Minneapolis, MN: IEEE Computer Society, 2007, pp. 358-378.
  2. M. Galster and D. Weyns, "Empirical Research in Software Architecture - How far have we come?," in 13th Working IEEE/IFIP Conference on Software Architecture (WICSA) Venice, Italy: IEEE Computer Society, 2016.
  3. C. Wohlin, A. Aurum, L. Angelis, L. Philips, Y. Dietrich, T. Gorschek, H. Grahn, K. Henningsson, S. Kagstrom, G. Low, P. Rovegard, P. Tomaszewski, C. Van Toorn, and J. Winter, "The Success Factors Powering Industry-Academia Collaboration," IEEE Software, vol. 29, pp. 67-73, 2012.
  4. M. Keeling, "Lightweight and Flexible - Emerging Trends in Software Architecture from the SATURN Conferences," IEEE Software, vol. 32, pp. 7-11, 2015.
  5. https://people.cs.kuleuven.be/danny.weyns/papers/2016WICSA.pdf

You may also like:



Posted by at
Labels: , ,

1 comment:

Subscribe to: Post Comments (Atom)