Free and Commercial Software (feature)

This is the first of a series of articles investigating the interplay between Open Source Software and commercial vendors. The common thread will be to find out how authors have earned a living from writing and servicing GPL/Open Source Software.

The series kicks off with Michael Tiemann, one of the founders of Cygnus Solutions, and author of the C++ component of GCC. Cygnus is one of the pioneers in commercial support for open-source software. It sells support and customization services for the gnu programming tools (gcc, gdb, ld, etc). Initially, the Free Software Foundation applauded this, but more recently, RMS feels that by also selling proprietary software Cygnus has betrayed the principle of free software.

In this interview, Michael discusses what it takes to set up a business based on open source software, what Cygnus hopes to add to GCC, and why they took the decision to sell closed source software.

What is written in red, is what Michael wrote to me in an email
What is written in green, is what I transcribed from my notes of our telephone interview. Any mistakes therein are mine.

• How difficult was it for you to set up Cygnus Solutions?
• • How did the banks cope with Cygnus' non orthodox business model?
  • How difficult was it to find the first customers?
  • What advice could you give someone who has written a GPL'd (or similar) program, and wants to live off it by providing services to commercial users?
  • Can one survive purely on the provision of services?
  • What types of services have you found to be the most important for commercial customers? What sort of guaranties do they want? What should one pay special attention to?

• • The greatest challenge in setting up Cygnus was in finding a name under which we could do business. This is not a joke...for three months we went through a process of finding a name, submitting it to the dept. of corporations, only to learn that the name had already been taken. We were about to give up hope when such an ugly name as "Software Support, Inc." had been registered the same year we tried to get it. When we learned that Cygnus Support (note the embedded GNU in Cygnus) was ours to use, we were overjoyed.

    Cygnus was started with virtually no money. At the time, I wanted John Gilmore (whose userid was [e-mail]) to be one of the founders (he was employee #5 at Sun, and was the guy who ported BSD Unix to the 68020), not for his money, but for hus programming skills. I put _my_ money where my mouth was, proposing that all three founders put in the same amount of money at the start. At that time, I had less than $3000 in the bank, so I proposed we each put in $2000 at the start, and raise the ante to $5000 when I could afford it. As you can see, we did _nothing_ to try to get banks interested, so there was nothing to explain to them.

    As for finding the first customers...it was just a matter of knocking enough times on the correct doors. As the author of GNU C++, it was pretty easy for me to talk to high-level technical people, who would then introduce me to mid-level managers. We sold GNU support primarily as a cost-saving measure, so we did not need VP level approval. Nowadays we have a much more strategic sell, but we're also much better established, so we can call on and get VP level approvals.

    I've always advised everybody that if they want to start a free software business, they should not let anything stop them. Of those who did, most managed to break even or make a small profit, leading me to believe that there's more to Cygnus's success than merely addressing a market opportunity. We've always been willing to make tough decisions, and we've always listened to customers, even when they were yelling at the top of their voices. Those who do free software for love, rather than money, are not going to survive what we've been through.

    As far as what customers really want...they want somebody who will listen to their needs, formulate a long-term plan, and then deliver without a lot of hassle or risk. Some customers like to see you work; others like to see the work you do; still others think you are doing a great job if they never see you. In the end, it's not the guarantees you offer that make customers come back..it's what you can deliver. The free software model is unique in that it actually gives an incentive for customers to work with you, not independently from you. In the fast-moving high tech world, that advantage has put us in the #1 spot in our market...no mean feat.

• Cygnus and the FSC (Free Software Community)
• • What exactly are you providing to the FSC?

• • We pour about $10M/year into the development of new free software. We run the servers from which 500,000 people have downloaded our stuff. We promote the concept of free software in articles, conferences, and advertising. We sponsor free software projects and pay free software contractors who do work that's related to our business requirements (though not something we'd directly commercialize). We provide a generous matching fund so that employees and customers who want to donate to the FSF can see their donation multiplied.

• How long does it take for your new code to be released in Egcs, or other freely downloadable source code?

• • The std deviation of this number makes an average number meaningless. If you download EGCS, I'll bet that the ChangeLog entries are no more than 45 days old. Compare that to other FSF-controlled packages where they can be months or years old. It is important to realize that we don't control when the FSF releases code.

• How much do you feel you are getting from the FSC? Would it have been simpler to write you own compiler?

• • Simpler, perhaps. But we would not have been able to distinguish ourselves from the other N proprietary companies. I mentioned earlier that people are now chosing Cygnus not because we are cheaper, but because we are _strategic_. We could not have achieved that strategic positioning in the compiler space in any other way.

• What overhead is there with working with the FSC?

• • It's whatever you want. Some people like to spend hours arguing about indentation styles. Others just decide they are going to do somthing and they do it. I wrote the first native-code C++ compiler in 6 month in 1987. It was hard work, and it was rough, but once I demonstrated the momentum I could sustains, dozens joined to the GCC/G++ team.

    In 1989, when GCC 1.x was getting fairly mature, I wrote an instruction scheduler, re-wrote a branch scheduler that had been abandoned, and wrote several cool new optimizations that could not easily fit with GCC 1.x. I declared that we should start GCC 2, with the goal of integrating this new work, new C++ language features, and an eye towards optimal code generation for RISC machines.

    GCC 2.x is now reaching maturity, and it's just not possible to preserve that kind of stability with the while working on advanced/experimental/cool new stuff we want to do. Hence we created EGCS. I've been amazed at the number and quality of volunteers who are now working on that branch. Thus, for GCC 2, the overhead is high, whereas for EGCS, the overhead is negative.

• What sort of friction is there between immediate customer concerns and the interests of the FSC?

• • No difference to any other scarce-resource problem.

• • The FSF's objective is a free software world. The customer's objective is a solution that satisfies his needs. The problem is that commercial work always pays more, so more people do that, which frustrates the FSF that not more people are on its bandwagon.

• Which strengths do you see in the Bazaar and Cathedral models of free software development?

• • Although Eric Raymond is very good at coming up with beautiful theories, I dispute whether they are a true reflection of reality. I see GNU and Linux as two cathedrals. Both Linux and RMS are benevolent dictators. What sets them apart is that Linus is willing to take more risks, to rewrite more of his software, while RMS is a perfectionist. Overall, I believe that risks win, and the risk of reimplementing things lead to a better result. Eric and I were both at the Foresight institute to discuss Netscape releasing their source code. Eric said that Cygnus's model was to provide service to commercial customers for free software products, while Netscape was going to lead the standards with its Free software strategy. I disagree with this since Cygnus too is leading the standards in the embedded market with free software.

• Why did you decide to sell software without source-code, given that this soured your relation with the FSF?

• • We measure success in $$, not FSF relationship. Free software works great for support, not for products.

    Cygnus has been very successful at its challenge. In 8 years we've become the number one tool supplier for the embedded market. Our growth is limited only by the number of sales and technical people we can find to employ. I expect our business will be eventually worth 100 of millions dollars per year. But this has taken lots of effort, and is only achieved by making life easy for our customers. For instance, we support 17 different host-target combinations for Cisco alone. Their routers' CPUs vary from the very simple basic RISC chip to very complex CISC chips with fancy features, yet they want the same code to work on any host-target combination. Contrast this to Sun or Microsoft where the same code will not work on SunOS 4/Solaris or Windows 95/NT. We support SGI, HP, NT (etc) so that our customers do not have to decide on their platform because of us. In total we support 170 different host-target combinations.

    The proprietary model and the free software model provide two different kinds of simplicity. The free software model provides simplicity of APIs: there is no need to waste time reverse-engineering things if you have source. The proprietary model provides simplicity of control: your software does not go off and have a life of its own, you can control your investment, your price points and focus on command performance rather than spend time on building a consensus. For instance, say we have two customers, one of whom has a bad bug he needs fixed, the other of whom has sent us an enhancement to our product. Say furthermore that we only have enough time to service one of their needs. If we fix the bug, then the customer who sent us the enhancement will be disappointed -- they did what the GPL said to do --, and they will wonder why they need support from us, since the guys on the net included their enhancement while we did not. If we add the enhancement, then one of our customers still has a bug, and if he cares nothing for having the source-code, he will look somewhere else for business. So adopting the proprietary model allows us to control our strategy. If the need for strategic control disappears then the software can become free, but this might never happen. For instance, dejagnu was not planned to be commercial software, so we put on the net to get help from the people out there.

    In my opinion, Sun has done a bad job at being a software leader, as has Microsoft. What GNU and Linux are doing is pushing the envelope back up.

• You provide compilers for the embedded market

• • Yes. The embedded market is very different from the traditional desktop market. For instance, the object files that became popular in the embedded market differ from those used in the desktop/workstation markets. Originally embedded people worked on Sun 3's and cross-compiled 68000 code into a.out executables. But a.out has limitations such as not allowing multiple data sections. New formats were devised to address this and other issues.

    Another difference is the embedded market is extremely cost-conscious. For instance, the money made on cellular telephones is made on the subscription. Thus the bill of materials for a cellular telephone must be extremely low. Similarly, end-product functionality is more of a concern that compatibility. Because RISC processors provide approximately twice the compute-power for half the electric power of a CISC processor, they are being adopted more rapidly than in the PC market. MIPS is now dethroning the previous king of the embedded world, the 68000 family, having sold 44 million units last year versus 68000's 42 million.

    Providing one same set of compilers for a wide variety of chips has made us into the standard for the embedded market. We have close ties with CPU companies, such as Toshiba, so that each new chip is released with a new version of our compiler. This makes it easy for our customers to choose which ever chip will give the best cost/performance ratio, without worrying about high porting costs.

• What features did you have to add for this market?

• • We added many features that improve code performance and code size. We provide a very lean and mean implementation of C++ to allow embedded programmers to use the C++ features they need. Indeed, we were the first to ship C++ for the embedded world, for Nortel in 1990. We've added the ability to control via #pragmas the type of return code the compiler should use, so that interrupt routines can be written in C. Another embedded feature is the ability to specify into which segment each function should go. And to help our customers migrate rapidly from chip architecture to chip architecture, we have a whole series of #pragmas to control the way in which data structures are laid out in memory. This allows them to have binary compatibility of data structures even if the code was originally written to access words at odd boundaries, but the target processor does not allow this.

• Are there any new #pragma like commands to control code size at a lower granularity (e.g.: for single functions)

• Are there any new #pragma like commands to control speed as a lower granularity (e.g.: for single functions)

• • We find generally that programmers want the same level of optimization for a particular module. However some optimizations of that kind are described in the processor specific optimization documentation: these are processor dependent trade-offs. For instance, on an R10000 (MIPS), bigger code is faster in that keeps all 4 pipes full. On the other hand, we had to deal with very high space constraints when we worked with Sega. They used the Gnu compiler for the Sega Saturn. And although the SH (Super Hitachi processor) encodes 32 bit instructions into 16 bits, we had to change the compiler significantly to cope with the fact that loading immediate values (such as 57) is very convoluted: you read the value then jump over it, making immediate value loads very expensive when the compiler assumed they would be cheap.

    Generally, we think code size will become less and less of an issue: at the rate at which chips are shrinking and processes are improving (such as the extreme Ultra Violet process), the average DRAM chip will contain 1.7 Gbits in 2001 and 237 Gbits in 2012. Bandwidth problems will be alleviated by developments such as Mitsubishi's M32R chip, which has a core on a 16 Mbit ram, and has a huge bandwidth.

• Do you have any benchmarks to compare the current GCC levels of compilation (code size, speed) with compilers from Metaware, IBM, Microsoft, Borland, Symantec, Intel's VTUNE, etc?

• • We use the Bench++ benchmark. In contrast to SPEC benchmarks, this uses Stepanov's abstraction benchmark to determine the cost of various C++ features. For instance at the 0th level of abstraction, the code uses an integer, at the 1st it uses a double, then a class with a double member, then a class with a double pointer member... including template iterators until the 12th level is reached. GNU beats Sun and HP's compilers by a factor of 10. Similarly, GNU exception handling is 10 times faster than HP's.

    In terms of the SPEC base benchmarks EGCS attains the same level as SGI's compiler (the best in the business) with -O and -O2 options. Beyond that, SGI allows one to specify tons of options, such as cache size, which generates better code. However no one actually uses these options, so it's not much of an issue.

    As a company, other than the CygWin guys, we don't compare ourselves with the PC guys, so I can't give you a comparison. The DJGPP guys may have some info. Metaware is a non factor for us, since none of our customers complain about our performance.

• Future trends
  • What new features would you like to see in future revisions of GCC?

• • We'd like to see a good implementation bridging the gap between C/C++ and Java. There are good ideas in Java. It provides open standards that are very important to the downloadable world I believe will arise in the future: it provides a simple way to connect to the web, which essential to achieving an information aware infrastructure.

    However Sun's current behaviour is very worrying. Anything that Sun writes becomes a Java API. This is not unlike Microsoft, who redefined C APIs, even though a standard existed (POSIX), to such an extent that even "hello world" will not work (stdout and stderr are not available from winMain()). Sun is playing a similar game with Java, and we fear it will lead to incompatibilities, balkanizing Java, and letting the competition win. Although Sun snipes free versions of Java, all it is doing is replaying the Unix wars. We hope that GCC will be the free standard wedge that keeps the door open.

• What do you think of the idea that programs are JIT compiled to optimize the features being used, to maximize memory efficiency (as Intel's VTUNE does) while minimizing total footprint by leaving most of the code in a compressed bytecode format (as www.tao.co.uk is suggesting).

• • We view the problem rather differently: what do you want to use your CPU power for? Do you want to use it for JIT compilation, or for running the application? We think the latter. So the JIT should be done on the server. There is no reason why your processor could not identify itself to the server and let the server produce whatever native implementation is best for it. Profile information could also be sent to the server in this way. We will be showing in an upcoming Dr Dobbs Journal, that the performance one can achieve in native-compiled Java is comparable to that of C. Speedwise, this is a better solution than a Java chip, because Lisp Machines have already shown us that register based machines present many more opportunities for optimizations than stack machines. Powerwise it is also advantageous as compilation is rather expensive on a battery. Even in terms of code size, gzip compressed 32 bit RISC code is comparable in size to the equivalent byte-code.

• What do you think of IBM's idea of simplifying processor state-machines to increase frequency by moving more of the complexity to software?

• • Like EPIC, the proof will be in the proverbial pudding. Different RISC implementations vary greatly in power, although in principle they should provide similar levels of performance. For instance, a 200 Mhz R10000 provides a similar level of performance to a 600 Mhz Alpha.

• Do you anticipate commercial proprietary source-code will at the end of the day dominate the market-place, or do you feel that free software backed up by a service industry will take over?

• • I think the distinction will be like the difference between public and private life. While we may let all the people on the net look at our source-code we don't let them play with our money. I believe that free software will determine the standards in the world. Proprietary software will exist in niches, where customers don't care for the source code. Instead of spending time building a concensus, proprietary software allows a company to focus on producing a piece of software as quickly as possible to catch a market opportunity. For instance, we use ORACLE's database software. We don't have the source-code, and don't want it because its internal complexity is irrelevant to us: we just want to use it.