Coverity Scan Report on 450m lines of open source coding shows it is still competitive V proprietary code

10 May 2013

The latest Coverity Scan Open Source Report, which the development testing firm has produced since 2006, has identified Linux as an on-going benchmark of quality. But the 2012 report which looked at 450m lines of software code has also highlighted persistent high-risk defects, such as memory corruption and illegal memory access, while still landing the quality of the open source approach versus expensive proprietary coding.

The report details the analysis of more than 450m lines of software code through the Coverity Scan service, the single largest sample size that the report has studied to date, since its launch in partnership with the US Department of Homeland Security back in 2006.

The newly standalone report is a good way to assess the quality or otherwise of open source software and Coverity concludes that overall code quality in the open source arena continues to rival that of proprietary software coding.

Linux in particular is unsurprisingly cited as a standout performer and an on-going benchmark of quality. Versions scanned in 2011 and 2012 demonstrated a defect density below 0.7, which continues the trend of consistently achieved defect densities of less than 1.0, since the launch of the report.

For its 2010 report, Coverity scanned more than 6.8 million lines of Linux code and found a defect density of 0.62. The next year Coverity scanned more than 7.4 million lines of Linux code and found a defect density of 0.66. for this 2012 report, the development testing firm, scanned 7.6 million lines of code in Linux 3.8 and found a defect density of 0.59.

The report also looks at PHP, Apache and other open source code, using an anonymous sample of nearly 300 of Coverity’s customers. Java is also at last being added this year for the next report.

Other key findings from the 2012 Coverity Scan Open Source Report include:

• As projects surpass one million lines of code, there’s a direct correlation between size and quality for proprietary projects, and an inverse correlation for open source projects. Proprietary code analysed had an average defect density of .98 for projects between 500,000 - 1,000,000 lines of code. For projects with more than one million lines of code, defect density decreased to 0.66, which suggests that proprietary projects generally experience an increase in software quality as they exceed that size. Open source projects with between 500,000 - 1,000,000 lines of code, however, had an average defect density of 0.44, while that same figure increased to 0.75 for open source projects with more than one million lines of code, marking a decline in software quality as projects get larger. This discrepancy can be attributed to differing dynamics within open source and proprietary development teams, as well as the point at which these teams implement formalised development testing processes.

• High-risk defects persist: 36% of the defects fixed by the 2012 Scan report were classified as “high-risk,” meaning that they could pose a considerable threat to overall software quality and security if undetected. Resource leaks, memory corruption and illegal memory access, all of which are considered difficult to detect without automated code analysis, were the most common high-risk defects identified in the report.

“This year’s report had one overarching conclusion that transcended all others: development testing is no longer a nice-to-have, it’s a must-have,” said Jennifer Johnson, chief marketing officer for Coverity, while banging the drum somewhat for her company. “The increasing number of open source and commercial projects that have embraced static analysis have raised the bar for the entire industry. As we see year-in and year-out, high-risk defects continue to plague organisations; simply put, if you are not doing development testing, you’re at a competitive disadvantage.”

While static analysis has long been cited for its potential to improve code quality, there have been two significant barriers to its adoption by development organisations: high false positive rates and a lack of actionable guidance to help developers easily fix defects. Coverity believes that the industry has eliminated both of these obstacles. The 2012 Scan Report demonstrated a false positive rate for Coverity static analysis of just 9.7% in open source projects. Additionally, the 2012 report noted more than 21,000 defects were fixed in open source code – more than the combined total of defects fixed from 2008-2011.

“We started the Coverity Scan project seven years ago with the US Department of Homeland Security, as a resource for the open source development community to improve the quality of their software,” said Andy Chou, co-founder and chief technology officer (CTO) at Coverity. “Each year, driven in part by advances in static analysis technology, the size and scope of the report increases - as do the number of defects identified and fixed. We look forward to continuing this work in the years to come.”

Of course to truly get an overview of the entire open source community more participants, including non-Coverity customers, would have to take part but the size and scope of the report is certainly large enough to give the technology community a very good indication of the on-going quality of open source coding versus expensive proprietary systems.

Become a bobsguide member to access the following

1. Unrestricted access to bobsguide
2. Send a proposal request
3. Insights delivered daily to your inbox
4. Career development