Open Source Acid Test Revisted

Kragen Sitaker has written a brutal reply to the story Sengan posted yesterday on the The Open Source Acid Test. It goes down and point by point shows the factual errors and fud in the story. It pleases me greatly to post this feature. Check it out.

The following was written by Slashdot Reader Kragen Sitaker

I read Ted Lewis's article, _The Open Source Acid Test_, on your web pages.

I was appalled that an organ of a prestigious international society like the IEEE would publish such error-riddled, poorly-researched, deliberately deceptive nonsense. It's as if the _New England Journal of Medicine_ had published a case study of a zombie animated by voodoo!

The author did not cite sources for any of his dubious statistics, and they are therefore hard to disprove. Given the remarkable lack of factual accuracy in the article, I doubt that they have any basis in fact.

To begin with the most obvious errors:

1. Linus Torvalds's name is not Linus Torvold.
2. Applix, Tower Technology, and NewMonics do not sell open-source software.
3. There is no such company as "Walnut Creek Stackware". www.cdrom.com belongs to Walnut Creek CDROM. There is no such company as "Tower Tech JVM". www.twr.com belongs to Tower Technology, which sells a (non-open-source) JVM. There is no such web site as www.debian.com.
4. www.python.org is operated by the Python Software Association, not CNRI, although it is currently hosted on CNRI's network.
5. Several of the "commercial enterprises" listed in Table 1 are not commercial enterprises at all. www.hungry.com, www.python.org, and www.debian.org are all operated by nonprofit organizations. The Corporation for National Research Initiatives, which was incorrectly listed as operating www.python.org, is actually a not-for-profit research organization.
6. It is absurd to say that Unix was the foundation for Hewlett-Packard and IBM, as Lewis does in his introductory paragraph. Both companies had been established for more than thirty years when the first line of Unix was written.
7. On page 126, Lewis claims that the open-source community admits that its organizational structure is weak. The evidence he adduces is a quote from a document published on www.opensource.org. What he doesn't tell you is that the document is *a leaked internal Microsoft memo*. Unless Lewis missed the 115 references to Microsoft in this document and also failed to read the introductory paragraphs, the only reasonable conclusion is that he is being deliberately deceptive.
8. On page 125, Lewis claims that "Currently, Linux's installed base numbers 7.5 million". As usual, he cites no source. However, the most widely-cited source for such figures is Robert Young's paper, Sizing the Linux Market eight different data sources to obtain an estimate of between five and ten million Linux users. However, this paper has a date of March 1998. If Linux's growth had continued to double yearly in 1998, as it did from roughly 1993 to 1998, the number of Linux users would be between ten and twenty million.
9. On page 128, Lewis says, "Windows NT market share smothers all Unix dialects combined". According to International Data Corporation's Server Operating Environment report, Unix and Linux together had 34.6% of the server market in 1998, while Windows NT had 36%. See more information. The actual number of server Linux shipments IDC tallied in 1998 was only three-quarters of a million; that suggests that if you include people installing multiple servers from the same CD and installing from Internet downloads, you would find that Linux's server market share is much greater than Windows NT's.
10. Lewis remarks, "With few exceptions, open source software has never crossed the chasm into the mainstream without first becoming a commercial product sold by a commercial enterprise." Does he think that Linux is not a commercial product sold by commercial enterprises? If not, there are literally dozens of "exceptions" to this statement -- Perl, Apache, sendmail, BIND, Linux, Tcl/Tk, Berkeley DB, Samba, the X Window system, FORTH, GNU Emacs, and trn, for example. Many of these became popular before they were commercially sold at all.
11. Lewis misstates the business case for Linux and "its open source software cousins". According to Eric Raymond -- whom Lewis quotes extensively elsewhere in this article -- a much more compelling business case is founded on the better quality of the software, choice of suppliers, choice of support and maintenance, freedom from legal exposure and license tracking. More details are available at opensource.org/for-buyers.html.

These minor factual errors, so far, merely indicate that the author knows very little about the topic he writes about and is deliberately trying to mislead his readers; they do not directly undermine his conclusions. However, as I shall show, each of his supporting arguments consist of incorrect facts and lead to faulty conclusions.

One of the author's major contentions is that as Open Source software adds more features and becomes more comparable to proprietary software, it will lose many of its advantages. He cites as examples Linux's supposed lack of video card support, wireless LAN support, and "a good selection of productivity software."; he claims that Unix contains 10 million lines of code, while Linux contains only 1.5 million. On page 126, he says, "Maintenance and support grow more complex, and costs increase due to a scarcity of talented programmers. Success leads to features, and feature creep leads to bloated software."

With regard to video card support, it is true that the Linux kernel does not have video card support in it. That facility is provided by video drivers in other software; nearly all graphical software available for Linux uses X11 for access to those video drivers. Open-source X11 drivers for most video cards are available from www.xfree86.org; the list of supported cards there currently lists 555 different kinds of video cards, many of which include numerous individual models.

For those few cards for which XFree86 support is not available, proprietary X11 drivers are available from Xi Graphics and Metro-Link.

With XFree86, Linux's video card support is better than either Windows 98 or Windows NT, and considerably more extensive than any Unix that does not use XFree86.

To claim that Linux lacks video card support is merely laughable.

With regard to wireless LAN support, it is true that many of the recent wireless LAN products do not currently have support in Linux. However, Linux has had support for packet-radio wireless networking and several kinds of LANs for years, and has supported several wireless LAN products since at least late 1997, including most of the most popular ones:

Lucent Wavelan
DEC RoamAbout DS
Lucent Wavelan IEEE
Netwave Airsurfer
Xircom Netwave
Proxim RangeLan2
Proxim Symphony
DEC RoamAbout FH
Aironet ARLAN
Raytheon Raylink
BreezeCom BreezeNet

This information is readily available on the Web in the Linux Wireless LAN Howto.

With regard to productivity software, there are several office suites available for Linux, and there have been for several years. ApplixWare and StarOffice are the two most common.

With regard to the size of Linux: first, among the utilities tested in the failure-rate study (the latest report on which is entitled "Fuzz Revisited: A Re-examination of the Reliability of Unix Utilities and Services". the quote used on page 125 appears to be from the original paper, which I cannot find on the Web) are the standard set of Unix utilities, awk, grep, wc, and so forth. These utilities have a standard set of functionality common across all Unix systems, except that the GNU utilities tend to have a great deal of extra functionality included. If the GNU utilities really are only one-sixth the size of the corresponding utilities on a Unix system, yet provide much more functionality, and still have one-third to one-sixth of the failure rate, that is not an indictment of the defect rate of free software, but rather a vindication of it -- which is why this study is linked to from the Free Software Foundation's Web pages. The study is unfairly biased in favor of less-featureful proprietary software, and that software still came out way behind.

(From my own experience, I know that frequently, the best workaround for a bug in a Unix utility is to install the GNU version.)

Lewis's claim that this represents "a single-point estimate of defect rate" is incorrect. The paper includes detailed results of the tests on 82 different utilities, along with aggregate statistics by operating system. 63 of these utilities were available either from GNU or from Linux, and were tested in this study.

With regard to the lines-of-code figure: it is not easy to measure the number of lines of code that constitute "Linux", because it is not easy to define what constitutes "Linux" -- or, for that matter, "Unix" either.

If we mean just the kernel, this site has some figures for the sizes of several OS kernels in 1994. SunOS 5.2's kernel is listed as containing 680,000 lines of code, while SunOS 5.0's kernel is listed as containing 560,000 lines of code. If the rate of increase per version remained constant (doubtful, because 5.0 and 5.1 weren't really finished products) then the latest SunOS (the one that's the kernel of just-released Solaris 7) would contain 1,280,000 lines of code.

By comparison, the source code of the 2.2.1 Linux kernel totals 1,676,155 lines of code, including comments and blank lines, counting only .c, .h, and .S (assembly) files.

The Linux project's source code has already reached a level where we would "expect Linux defect densities to get worse". They haven't.

On page 125, Lewis cites Apache as an example of support diminishing when "the hype wears off", saying "it is currently supported by fewer than 20 core members" -- implying that the "cast of thousands" is a thing of the past. The truth is that the core Apache team has never been larger than 20 people, and they *still* receive contributions from many people outside the group. He also says that "Apache is losing the performance battle against Microsoft's IIS." But Apache has never been intended to be the fastest HTTP server around -- it's already more than fast enough to saturate a T1 when running on a puny machine, so its developers have been concentrating on things like adding more features and making it more reliable.

On page 128, Lewis says, "The concept of free software is a frequently practiced strategy of the weak". While free-as-in-price giveaways are common -- Microsoft's Internet Explorer strategy is a perfect example -- they are not related to open-source software, and their patterns of success and failure have little relevance for us here.