Monday, October 2, 2017

The shape of a cryptographic keyring: How is the Debian project social structure like?

By: Gunnar Wolf 
Associate editor: Stefano Zacchiroli (@zacchiro)

Quite often, we can be quite familiar with a set of data; But looking at it with a different viewpoint reveals a completely unexpected reality.

I am part of the keyring-maint group in Debian: The developers that manages the cryptographic keyring through which developers identify themselves for most of our actions — Most important, voting in project decisions and uploading software.

There are two main models for establishing trust via cryptographic signatures: The centralized, hierarchical model based on Certification Authorities (CAs) from where all trust and authority stems and flows only downwards, and a decentralized one, the Web of Trust where each participant in the network can sign other participants' public keys; the first one is most often used in HTTPS (encrypted Web), and our project uses a specific mode of the second one, which we have termed the Curated Web of Trust [1].

Cryptographic signatures are way more secure as identifications to the omnipresent but very weak username/password scheme. However, technological advances must be factored in. Being already 24 years old, Debian is a very long-lived free software project, and it sports a great retention: Many of its developers have been active for over ten years. In the late 1990s, the recommended key size was 1024 bits — Public key cryptography is very expensive computationally, and said key size was perceived secure enough for the foreseeable future. However, according to a study on algorithms and keysizes in 2012, [2] this key size is today good only for Short-term protection against medium organizations, medium-term protection against small organizations — Clearly not below our required standards.

Our group had started pushing for migration to stronger keys back in 2009. By 2014, as the Figure 1 shows, we had achieved the migration of half of the developers to stronger keys; But the pace of migration was really insufficient. At the Debian Developers' Conference that year (DebConf14), we announced that by January 1st, 2015, we would remove all keys shorter than 2048 bits.

Figure 1: Number of keys by length in the Debian Developers keyring, between mid 2008 and late 2015

The migration process was hard and intensive. Given the curated Web of Trust model followed, our policy is that, for a key replacement, a new key must be signed by two currently trusted keys in our keyrings; being Debian a globally distributed project, many people are simply unable to meet other developers. This migration process resulted in us losing close to a fourth of all keys (that is, a fourth of all Debian Developers could no longer perform their work without asking somebody to "sponsor" their actions), we felt it to be quite successful. This migration prompted a deeper analysis into what the keyrings were able to tell about the developers themselves.

Trying to find any emergent properties, we graphed the signatures in the keyring at different points in time. Although most times we got just a useless huge blob, we observed a very odd division starting around 2011; Figure 2 shows the graph close to the point where this split was maximum: January 2012.

Figure 2: Trust relationships in the Debian keyring near the maximum /split/, January 2012

Then, trying to find the meaning of this split, we colored the edges according to their relative age — How long before each of the snapshots was each of the signatures made. This is shown for the above presented keyring in Figure 3.

Figure 3: Trust relationships in the Debian keyring near the maximum /split/, January 2012, graphed by signature age: (Blue: less than one year old; green: one to two years old; yellow: two to three years old; orange: three to four years old; red: over four years old)

Our hypothesis was that the red blob mostly represents an earlier generation of Debian Developers, who have mostly faded from project activity. We presented our findings last May at the 13th International Conference on Open Source Systems [3], together with a first foray into a statistical analysis on key aging and survival.

This rich data set can still yield much more information on the Debian project's social interactions. It's basically just a matter of finding other angles from which to read it!

[1] Wolf, G. E., & Gallegos-García, G. (2016). Strengthening a curated web of trust in a geographically distributed project. Cryptologia, 1-16.
[2] ECRYPT, I. (2012). Yearly Report on Algorithms and Keysizes (2011-2012), Rev. 1.0. ECRYPT II Network of Excellence (NoE), funded within the Information Societies Technology (IST) Programme of the European Commission’s Seventh Framework Programme (FP7).
[3] Wolf, G., & Quiroga, V. G. (2017, May). Progression and Forecast of a Curated Web-of-Trust: A Study on the Debian Project’s Cryptographic Keyring. In IFIP International Conference on Open Source Systems (pp. 117-127). Springer, Cham.

Monday, September 25, 2017

On the Effectiveness of Source Code Metrics for Fault and Change Proneness Prediction in Industrial Automation Domain

by Lov Kumar (NIT Rourkela, India) and Ashish Sureka (ABB, India) 
Associate Editor: Sridhar Chimalakonda (@ChimalakondaSri)

Domain Specific Programmable Logic Controller (PLC) Languages

Several control systems used for factory and industrial automation of electromechanical processes are developed using Programmable Logic Controllers (PLC). PLCs are specifically adapted for the control of a manufacturing process and programmed using domain specific languages. The International Electro-technical Commission (IEC) (a non-profit international standards organization) developed a standard called as IEC 61131-3 which defines the basic programming elements, syntactic and semantic rules for text-based and graphical or visual programming languages for programming PLCs [IEC]. Structured Text (ST) is one the text-based PLC programming languages defined by the IEC 61131-3 standard and is widely used in industrial automation engineering application development. Similarly, Ladder Diagram (LD) programming language is a visual programming language and is one of the five languages defined in the IEC 61131-3 standard. ST is a domain specific language and several of its language features and programming constructs are different than that of general purpose programming languages such as Java, C++, C# and Python. ST has several characteristics which are different than that of general purpose programming languages because the primary purpose of a PLC is to control an industrial process and the ST language is defined for solving problems in a specific domain of factory and industry process automation [Roos2008].

Correlation between Source Code Metrics and Software Quality for PLC Systems

There has been a lot of work done in the area of predicting change proneness and faulty components or modules of software using source code metrics for general purpose programming languages. However, defining software metrics and investigating their impact on important software engineering prediction problems such as change proneness and faulty module identification is relatively unexplored in the domain of applications developed using IEC 61131-3 languages [Kumar2016] [Kumar2017]. We believe that one of the primary reasons of lack of empirical studies in the area of source code analysis for PLC programming languages and their impact on software maintainability and quality is the lack of publicly available data for researchers and scientists in academia. Through this blog, our objective is to share our work in this relatively unexplored but a promising research direction [Kumar2016] [Kumar2017]. In [Kumar2016], we proposed source code level metrics to measure size, vocabulary, cognitive complexity and testing complexity of a visual Programmable Logic Controller (PLC) programming language. We apply Weykur’s property to validate the metrics and evaluate the number of properties satisfied by the proposed metric [Kumar2016]. In [Kumar2017], we study the correlation between the 10 ST source code metrics and their relationship with change proneness. We built predictive models using Artificial Neural Network (ANN) based techniques to predict change proneness of the software [Kumar2017]. Similarly, we are working towards examining whether source code metrics can be used to identify defective and faulty components.

Bringing Both Relevance and Rigor through Industry Academia Collaboration

Our experimental results are encouraging and provides evidences that there is a correlation between source code metrics and software quality and maintainability in the domain of industrial automation and PLC applications. However, more research studies from industry through industry and academia collaboration is needed to further add to the body of knowledge in this topic. It is hard for companies working in the industrial automation engineering domain to make their source code available in public domain for researchers in academia which creates a natural barrier for entry for academics to conduct research in this unexplored area. Industrial research labs does not have all the competencies, resources and time within a single organization to address all the research and technical challenges in this area and we believe that university and academia collaboration is the only possible solution to advance and disseminate knowledge on this topic. Our work is also a collaboration between industry and academia and gave positive results due to the successful partnership. One possible solution to build bridges is to conduct focused workshops on software engineering issues and challenges for industrial and factory automation. Some progress has been made in this direction which is encouraging and more needs to be done.


[Kumar2016] Lov Kumar, Raoul Jetley, and Ashish Sureka. 2016. Source code metrics for programmable logic controller (PLC) ladder diagram (LD) visual programming language. In Proceedings of the 7th International Workshop on Emerging Trends in Software Metrics (WETSoM '16). ACM, New York, NY, USA, 15-21. DOI:
[Kumar2017] Lov Kumar and Ashish Sureka. 2017. Using Structured Text Source Code Metrics and Artificial Neural Networks to Predict Change Proneness at Code Tab and Program Organization Level. In Proceedings of the 10th Innovations in Software Engineering Conference (ISEC '17). ACM, New York, NY, USA, 172-180. DOI:
[Roos2008] Nieke Roos. 2008. Programming PLCs using Structured Text." International Multiconference on Computer Science and Information Technology (IMCSIT’08)

[IEC] International Electrotechnical Commission, International Standards and Conformity Assessment for all electrical, electronic and related technologies

Monday, September 18, 2017

Understanding the Impressions, Motivations, and Barriers of One Time Code Contributors to FLOSS Projects: A Survey

By: Amanda S. LeeUniversity of Alabama. USA (@amandaslee15)
Jeffrey C. Carver, University of Alabama. USA (@JeffCarver32)
Amiangshu Bosu, Southern Illinois University. USA (@abosu)
Associate Editor: Bogdan Vasilescu, Carnegie Mellon University. USA (@b_vasilescu)

FLOSS, or Free/Libre Open Source Software, is becoming an increasingly important, some may even say dominant, factor in the modern software economy [2]. As opposed to the traditional methods of software development, FLOSS projects function and receive high-quality code submissions often despite the lack of financial compensation and the lack of any formalized management or governance structure [1, 2, 5]. While FLOSS projects are typically guided or managed by a small number of core developers, they survive and, indeed, thrive by attracting contributions from new, talented software developers who join the project. There are many types of contributors, ranging from those who have contributed a number of patches and are on their way to becoming part of the core, to those that lack adequate coding skills and contribute primarily by reporting bugs and editing documentation [3, 4, 6]. To be successful, FLOSS projects must constantly recruit new code contributors to replace those that leave (many do so within a year of joining) [7].

In our work [8], we define a special type of FLOSS contributor called “One-Time code Contributors” (OTCs) as those contributors who have successfully contributed, that is had merged by the project, one, and only one, code patch to a given FLOSS project. This successful contribution indicates that the OTC (1) has an appropriate level of coding skills and knowledge to make a valuable contribution to the project and (2) has the determination necessary to write the code, submit the patch, and participate in the review project to get the patch accepted. Because FLOSS projects could greatly benefit by attracting these technically competent participants to submit more code, two questions arise: (1) why do OTCs not contribute additional patches? and (2) is there any way the FLOSS projects can attract and retain these valuable contributors?

To answer these questions, and understand how OTCs could better contribute to FLOSS projects in the future, we conducted a survey of 184 OTCs from 23 popular FLOSS projects. The remainder of this post summarizes some of our key findings.

Initial Impression of Project Members
When asked about their initial impression of other project members, a large percentage of the respondents indicated they had a positive or very positive impressions of their fellow project members (Figure 1). This result is surprising because previous research indicated that peripheral developers may be neglected, so we expected them to have more negative impressions [6]. Some of the most common positive responses included: project members are skilled, helpful, or responsive. These responses indicate that OTCs appreciate the assistance that other project members can provide. However, some OTCs did report negative impressions of fellow project members. These negative impressions were just as strong as the positive impressions. The most common negative impressions included: project members are busy, unresponsive, or otherwise unhelpful.


Figure 1 - Overall Impressions

Tradeoff Between Skill and Busyness  
There was an interesting interaction between the most common positive impression, skilled, and the most common negative impression, busy. OTCs expected those project members who were more skilled to be less approachable and less obtainable, in other words, busier. Conversely, they expected less skilled project members to be less busy and more approachable. OTCs did not seem displeased with this trade-off—in fact, they seemed to expect it. Even so, the OTCs whose experience was that skilled members were not too busy to pay attention to them reported more positive impressions, while others who experienced the busier skilled members expressed more neutral or negative impressions.

OTCs’ Motivations for Contributing
Previous research found that peripheral contributors (of which OTCs are a subgroup) tend to be motivated more extrinsically (that is by external factors like fixing a bug) than intrinsically (that is, by internal factors such as enjoying coding as a hobby) [4, 7]. This observation would suggest that a large portion of the OTC contributions would be in the form of “drive-by commits,” that is, fixing a flaw without any real desire to join the project. When asked about their motivation to contribute their patch, the respondents listed a variety of motivations, as Figure 2 illuminates. While respondents indicated the desire to fix a bug as the most common motivation, they gave a more intrinsic motivation, share with the community, as the second most common motivation. Third, respondents indicated they were motivated by an employer’s need. These respondents were not hired to contribute to the FLOSS, but rather added or fixed the FLOSS project in support of other employer goals. Fourth, another intrinsic motivation, respondents wanted to add a new feature and have it maintained by the project. Perhaps unsurprisingly, this motivation often overlapped with the intrinsic motivation, share with the community. Other, less common motivations include ‘scratching an itch,’ personal reputation, or curiosity about the project. This list of varied motivations suggest that while, some OTCs may not be interested in long-term project participation, others have deeper motivations and could ultimately be attracted to join the project and contribute more than a single patch.


Figure 2 - OTC’s Motivations

Barriers Faced by OTCs
When asked whether there were any barriers that prevented them from continuing to contribute to the project, only half of the respondents indicated that the faced barriers, as seen in figure 3. The most commonly reported barrier was time. Either it took the respondent too long to make the contribution, or the respondent was too busy with other work. The next most common barriers were patch submission difficulties and entry difficulties. While FLOSS projects and tools oriented to help project newcomers cannot address the time barrier, they can encourage OTCs to continue contributing by reducing the difficulties associated with project entry and with patch submission.


Figure 3 - OTC Barriers

OTCs Who Stopped Contributing Despite Facing Barriers
Of the respondents who did not face barriers, nothing else to contribute was the most common reason to stop contributing.. This response is interesting because it suggests that if these OTCs did find something else to contribute, they might return to the project. In other words, these OTCs could be motivated to continue contributing to the project, because they exhibited no particularly strong desire to leave it. Other reasons for leaving a project included their employer no longer used the project and they never had any intention of becoming a project member. These results suggest that while not all OTCs can be motivated to make additional contributions, some very likely could be, under the right circumstances.

Though OTCs have only successfully contributed one code patch, FLOSS projects may be able to attract some of them to make further contributions. While some OTCs truly are “drive by committers,” whose only interest in the project was to have their patch included, others are more community-minded and more invested in the project. This second group of OTCs may have contributed additional code patches had they not encountered barriers or had they identified another interesting patch to contribute. By lowering the entry barriers and making patch submission easier, FLOSS projects can likely retain more OTCs, thereby increasing the size and skill of the contributor base, which leads to more successful FLOSS projects.

  1. Lakhani KR, Wolf RG (2005). “Why Hackers Do What They Do: Understanding Motivation and Effort in Free/Open Source Software Projects."  In J Feller, B Fitzgerald, S Hissam, KR Lakhani (eds.),Perspectives on Free and Open Source Software, MIT Press, Cambridge.
  2. M. Milinkovich. Keynote, Topic: "Open Collaboration, the Eclipse Way." International Conference on Software Engineering, Buenos Aires, Argentina, May 24 2017.
  3. G. Pinto, I. Steinmacher and M. A. Gerosa, "More Common Than You Think: An In-depth Study of Casual Contributors," 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Suita, 2016, pp. 112-123.
  4. Setia R. et al., “How Peripheral Developers Contribute to Open-Source Software Development,” Information Systems Research, vol. 23 issue 1 pp. 144-163, 2012.
  5. Shah, S. K., “Motivation, Governance, and the Viability of Hybrid Forms in Open Source Software Development,” Management Science, vol. 52 issue 7 pp.1000-1014, 2006.
  6. Steinmacher, I. et al., “Social Barriers Faced by Newcomers Placing their First Contribution in Open Source Software Projects.” 18th ACM Conference on Computer-Supported Cooperative Work and Social Computing, Vancouver, BC, Canada, 2015, pp. 1381-1392.
  7. Zhou, M. and Mockus, A., “What Make Long Term Contributors: Willingness and Opportunity in OSS Community.” 34th International Conference on Software Engineering (ICSE), Zurich, 2012, pp. 519-528.
  8. Lee, A., Carver, J. C. and Bosu, A., "Understanding the Impressions, Motivations, and Barriers of One Time Code Contributors to FLOSS Projects: A Survey." 39th International Conference on Software Engineering (ICSE), 2017, pp. 187-197.

Monday, September 11, 2017

IEEE July/August Issue, Blog, and SE Radio Summary

Associate Editor: Brittany Johnson (@drbrittjaydlf)

The July/August Issue of IEEE Software again provides articles that cover a range of software engineering-related topics. The themes in this issue include reliability and requirements engineering, technical debt, and agile development.

Special in this issue is a look at the history of IEEE Software in the article "Insights from the Past: The IEEE Software History Experiment" by Zeljko Obrenovic. By looking at the history website, this article proposes the practical value in using historical data. There is also discussion of the future of IEEE Software based on historical data.

The focus topic in this issue of IEEE Software was Reliability Engineering. This issue featured the following articles on reliability engineering:

One of the main concerns around reliability engineering is the growing ubiquity of software in safety critical systems. As pointed out by Diomidis Spinellis in "Software Reliability Redux", the more software is integrated into our daily lives the more we need adequate reliability engineering. Going more in depth on the topic, in "Requirements Engineering for Safety-Critical Systems: Overview and Challenges",  Martins and colleagues talk about some of the challenges that comes with engineering safety critical systems. In particular, they focus on dealing with safety requirements, discussing topics from reducing the gap between academia and industry and communicating requirements throughout the development process.

In "Safety Analysis of Safety-Critical Systems Using State-Space Models", Kumar and colleagues made one suggestion for dealing with safety analysis of safety critical systems; using state-space models for safety issue prognosis. They created an approach that takes UML state diagrams and maps them to state-space models, using Petri nets for dynamic behavioral analysis. They were able to validate their approach using a nuclear power plant's emergency core cooling system. Despite their findings, there are still strides that can be made to improve the state-of-the-art in regards to processes and tools in place for building and maintaining reliable safety critical systems.

IEEE Software Blog

The IEEE Software Blogs posts in July and August focused on vulnerabilities. Mehdi Mirakhorli wrote a blog post on tactical vulnerabilities, or vulnerabilities that result from incorrect implementation or deterioration of security tactics during coding and maintenance. Cor-Paul Bezemer and Zhen Ming (Jack) Jiang explored if and how developers use performance testing to identify vulnerabilities. They studied Java-based open source projects on GitHub and found that there may be missing tool support for effective performance testing. Maleknaz Nayebi and Federica Sarro used crowdsourcing in a case study of the Fort McMurray wildfire, along with their method called MAPFEAT, to determine useful mobile app features.

SE Radio

SE Radio welcomed two new hosts in July and August: Kishore Bhatia of BlockApps and Bryan Reinero of MongoDB.  For Kishore's first broadcast, he spoke with Kieren James-Lubin about security topics such as Blockchains, Crytocurrency, Bitcoins, and Distributed Ledger. For Bryan's first broadcast, he and Jason Hand discuss handling outages and responding to program failures. 
Other topics discussed on SE Radio in July and August include type driven development, rules engines, and P vs. NP.

Monday, September 4, 2017

IEEE May/June Issue, Blog, and SE Radio Summary

Associate Editor: Brittany Johnson (@drbrittjaydlf)

The May/June issue of IEEE Software provided a wide range articles on software-related issues; from energy-aware systems to deep learning, there's something for just about everyone. Although the focus of this issue was automotive software, there are features on various other software topics.

The focus topic in this issue of IEEE Software was Automotive Software. This issue featured the following articles on Automotive Software.

As suggested by the Editor, Diomidis Spinellis, in his article "How Abundance Changes Software Engineering", increase in software processing power has lead to changes in how we use computing technologies. This includes the relatively recent innovation of automotive software. In "Future Automotive Architecture and the Impact of IT Trends", Traub and colleagues discuss the opportunities that advances in IT and consumer-electronics afford the automotive industry. Much of the discussion focuses on the importance of seamless and scalable architecture for automotive software.

With the creation of things like Google's self driving car, and the potential for this to be a commercially available product, the importance of security in the software behind our vehicles is increasing. The authors of "Secure Automotive Software: The Next Steps" discuss this issue in depth, starting with the challenges faced by developers that work on automotive software.  Based on these challenges, the authors discuss some recommendations they have for improving the security of software used in the auto industry. Some of their recommendations include using static analysis for compile-time assurance and cryptography for runtime protection.

This issue also included articles on the feature topics of mobile app development, agile development, continuous deployment, and the business of software engineering  encompassed in the following articles:

IEEE Software Blog

The topics in the blog posts for May and June show a little more diversity than last month's...figuratively and literally. Anna Filippova of Carnegie Mellon contributed a blog post on how brainstorming can be used to support inclusiveness in diverse teams. Other topics discussed in these blog posts include cross-stack configuration errorslibrary adoption, and the effect of casual contributions on software quality.

SE Radio 

In this issue, SE Radio welcomed another new host, Matthew Farwell of Nexthink. In his debut, he spoke with Yakov Fain about Angular, including who should use it and why. Many of the episodes in this issue center around technologies and how developers are using them, including ElasticsearchDocker, and LLVM.  SE Radio also deployed a listener survey to get feedback from their listeners -- don't forget to provide your feedback!

Monday, July 24, 2017

Tactical Vulnerabilities in Chromium, PHP and Thunderbird

By:  Mehdi Mirakhorli (@MehdiMirakhorli), Associate Editor

Software engineers face a constantly growing pressure to build secure software by design, where systems have to be designed from the ground up to be secure and resistant to attacks. To achieve this goal, security architects work with various stakeholders to identify security requirements and adopt appropriate architectural solutions to address these requirements. These architectural solutions are often based on security tactics. Security tactics as reusable solutions to satisfy security quality attributes regarding resisting attacks (e.g., tactic “Authenticate Actors”), detecting attacks (e.g., tactic “Detect Intrusion”), reacting to attacks (e.g., tactic “Revoke Access”), and recovering from attacks (e.g., tactic “Audit”). Despite the significant efforts that go into designing secure systems, security can slowly erode because of ongoing maintenance activities. Incorrect implementation of security tactics or the deterioration of security tactics during coding and maintenance activities can result in vulnerabilities in the security architecture of the system, thus compromising key security requirements. We refer to these vulnerabilities as tactical vulnerabilities.

The code snippet in Listing 1 from a J2EE web application shows an example of such tactical vulnerabilities which is the incorrect implementation of the “Manage User Sessions” tactic. The correct implementation of this tactic in a web application would allow the system to keep track of users that are currently authenticated (including permissions they hold). However, in the given code snippet, the application authenticates users with LoginContext.login() without first calling HttpSession.invalidate() to invalidate any existing session. This enables attackers to fixate (i.e., find or set) another user’s session identifier (e.g., by inducing a user to initiate a session using the session identifier provided by the attacker). Once the user authenticates him/herself with this forged session identifier, the attacker would be able to hijack or steal his/her authenticated session. Although architects have used the “Manage User Sessions” tactic in the architecture design of the web application, the developers have failed to implement it correctly, resulting in a tactical vulnerability that can be exploited for session fixation attacks.

Recent empirical studies of security vulnerabilities have neglected the architectural context, including design decisions such as tactics and patterns. They mostly focus on studying and understanding coding issues related to the management of data structures and variables (e.g., buffer overflow/over-read). 

Goal of This Study

Here, I’d like to report an in-depth case study of software vulnerabilities associated with architectural security tactics across three large-scale open-source systems (Chromium, PHP, and Thunderbird). In this blog post, I only present the results, the scientific process and systematic approach used to make the conclusions can be found in our research article here.

Common Tactical Vulnerabilities

Table I lists the root causes (i.e., vulnerability types) of tactical vulnerabilities in each of the three studied systems, the related architecture tactics, as well as the total number of CVEs caused by the given vulnerability type.

Key findings

While Chromium, PHP, and Thunderbird have adopted a wide range of architectural tactics to secure the systems by design, a remarkable number of vulnerabilities discovered in these systems are due to incorrect implementations of these tactics. 

  • While Chromium, PHP, and Thunderbird have adopted a wide range of architectural tactics to secure the systems by design, a remarkable number of vulnerabilities discovered in these systems are due to incorrect implementations of these tactics
  • Improper Input Validation (CWE-20) and Improper 
Access Control (CWE-284) are the most occurring root causes for security vulnerabilities in Chromium, PHP, and Thunderbird. 

  • Vulnerabilities in the three studied systems are mostly related to tactics “Validate Inputs” and “Authorize Actors” for resisting attacks. 

  • Security of studied projects was compromised by reusing or importing vulnerable versions of third-party libraries. In the case of Chromium, such vulnerabilities occurred 106 times, while in Thunderbird and PHP, 7 and 8 times, respectively.
  • Tactical and non-tactical vulnerabilities have a similar distribution over time and releases, even though the absolute numbers of tactical and non-tactical vulnerabilities differ.
  • When fixing tactical vulnerabilities, there is no statistically higher or lower code churn compared to fixing non-tactical vulnerabilities.
  • When fixing tactical vulnerabilities, the number of affected files is not statistically significantly higher or lower compared to fixing non-tactical vulnerabilities.

Read more