David Korn Tells All

David Korn answers! You asked the questions a little while ago, and now David Korn has kindly responded with answers (some lengthy, some pithy, all appreciated) to queries about the eponymous shell, the famous Microsoft/Korn story (in short -- yep, it's like you've heard), and proper Unix behavior. He even made his son Adam write one of the answers.

Question 1) Comparison
by Shadowlion ([e-mail])

Background: the only shell I've ever really used is bash. Bash has always seemed to be the standard UNIX shell (or, at least, the standard default UNIX shell), and for the most part I've always been able to do what I wanted in it.

Question: can you engage in a little unadulterated advocacy for a moment to offer some reasons why an informed user might consider using ksh over bash or other popular UNIX shells? What does ksh provide that other shells don't? Similarly, can you give a realistic appraisal of ksh's drawbacks as compared to bash or other shells?

Thanks.

David Korn: First of all, when I talk about ksh here, I am referring to ksh93 which was first released over six years ago. Most UNIX systems ship with ksh88 and most Linux systems ship with pdksh, neither of which has the functionality of ksh93.

There are two different areas of functionality in shells. First is interactive use and the second is scripting. Much of the debate about shells has focused on interactive use only. For example, tcsh is an acceptable shell for interactive use but practically unusable for scripting.

In many cases the argument over which shell is best for interactive use is based upon which key to press for completion. This is a little like arguing that Solaris is better than Windows because of location of the Control and Shift keys or that vi is better than emacs because you you can save a keystroke or two. Most popular shells have similar functionality with respect to interactive use.

It is hard to argue that ksh is any better for interaction, given all the features in tcsh and zsh. But the scripting features in ksh93 are far more advanced than any other shell that I am aware of. For scripting, I feel that ksh is more in the category of perl/tcl/python and I would like to see debates/comparisons for those languages rather than the antiquated bash/csh/etc.

I have not looked at bash for several years and some of the features I describe here might now be implemented by bash. I sent Chet Ramey, author of bash, the list of new features in ksh93 years ago so that if these features get implemented in bash, they would be compatible. Here is a partial list of ksh93 features:

1. Associative arrays - ksh88 already supports indexed arrays.
2. Floating point arithmetic + math library functions.
3. Arithmetic for command similar to C and awk.
4. Complete ANSI C printf formatting with extensions.
5. Run time linking of libraries and builtins.
6. A number of additional substring operations such as offset length.
7. Full extended regular expression matching capabilities
8. Compound variables which can be used to represent data aggregates.
9. Name references for passing aggregate variables by name.
10. Active variables. Users can trap variable assignment and references by associating intercept functions of the form name.get and name.set.
11. Ability to make socket connections to servers by name.
12. read with timeouts.
13. Conformance to POSIX 1003.2.
14. Command completion and variable completion, ksh93 only had file name completion.
15. A key binding mechanism that allows users to bind keys to new editing functions.

Note that only the last two features relate to interactive use. The primary focus of ksh93 is scripting and in this arena it certainly outshines bash. ksh93 runs builtin shell commands much faster than bash.

There is actually a third area of shell functionality which is related to extensibility. In this area ksh93 should be compared to tcl. ksh93 is implemented as a reusable library with a C language API for adding builtins and accessing shell internals. It can be embedded in other programs. For example, dtksh, which is part of the Common Desktop Environment (CDE), uses ksh93 as a library. Similarly, tksh, (written by Jeff Korn) which uses the tk library for graphics, uses ksh93 as a library.

The primary drawback to ksh has been that it was proprietary. This has recently changed however. The new AT&T open source license allows ksh source and binaries to be shipped as part of the system and is now just beginning to start showing up in Linux systems; for example the latest slackware. The source or binary for over a dozen architectures can be downloaded from (http://www.research.att.com/sw/download). Hopefully other systems will start shipping ksh93 and start using this for /bin/sh as well.

Question 2) What about enhancing ksh syntax?
by mirko ([e-mail])

Ksh is quite cool as it is much more compact than bash ; here are their respective sizes on a Solaris system:

-> /usr/local/bin/bash -version
GNU bash, version 2.02.0(1)-release (sparc-sun-solaris2.6)
Copyright 1998 Free Software Foundation, Inc.

-> ls -la /usr/local/bin/bash
-rwxr-xr-x 1 bin bin 3157516 Jul 14 1998 /usr/local/bin/bash
# ksh -o emacs
# Version M-11/16/88i

# ls -la `which ksh`
-r-xr-xr-x 2 bin bin 186356 Jul 16 1997 /usr/bin/ksh

On a Linux system, these are approximately 300k for bash and 160k for (pd)ksh.

In which direction do you plan to improve it?

Will you rather keep it compact or extend its functionalities regardless the volume increase ?

This issue is quite important for me as, as of yet I am working upon some System-on-a-floppy distribution and the size appears to be critical in this context.

Korn: First of all pdksh is a ksh88 clone; and I might add a better clone than the MKS Korn Shell.

A lot of effort was made to keep ksh88 small. In fact the size you report on Solaris is without stripping the symbol table. The size that I am getting for ksh88i on Solaris is 160K and the size on NetBSD on intel is 135K.

ksh88 was able to compile on machines that only allowed 64K text. There were many compromises to this approach. I gave up on size minimization with ksh93. ksh93 is larger than bash. However, because of the number of commands that can be builtin to ksh93, it is possible to put ksh93 and approximately 30 additional commands on a floppy in approximately the same space as with ksh88 and those commands.

Some of the reasons for the large size of ksh93 are:

1. Floating point arithmetic.
2. The use of Sfio for Stdio. Use of the native stdio would not have counted towards the size on most systems because they are part of the shared C library. However, Stdio has too many weaknesses to be used portably and safely by ksh.
3. Self-documenting code. All the built-ins to ksh93 can generate their own manual page in several formats and locales. This takes up space for the documentation text as well the the processing libraries.

Question 3) pdksh...
by mirko ([e-mail])

Do you collaborate (or plan to) with the pdksh development team ?

Korn: I don't know the pdksh development team but I would like to thank them for the service they have done in making a version of ksh available while ksh was proprietary. I have noticed remarkable improvements in pdksh in its ability to mimic ksh88 functionality. I don't know what plans the pdksh development team has now that ksh93 is available in open source form, but I certainly would help them try to maintain compatibility if they do continue pdksh distribution. Otherwise, I would hope that they would pick up the ksh93 source and help support and enhance it.

Question 4) ksh today
by Y-Leen

During the design of ksh, were you limited/influenced by computer hardware and consumer market?

Given the chance to completely redesign ksh for today's higher spec machines and the current consumer base, what new features would you include?

Korn: In many ways yes. In early days memory size was a big consideration. [That] sixteen bit machines were the norm influenced the design. The fork() call on most systems caused the data area to be copied in the child process and therefore keeping the data region as small as possible was needed for performance reasons.

Another design principle that was influenced by the hardware and the OS was the desire to only have features that would work on all systems so that korn shell scripts would be portable.

The decision to keep ksh compatible with the Bourne shell was influenced by the consumer base. There are very few Bourne shell scripts that will not run unchanged with ksh.

In a complete redesign, all shell functions would be thread safe so that multiple shell interpreters could run in separate threads.

There are two features that are missing the current ksh93 release that are present in perl and should be added to a future release. One is the ability to handle and process binary data. The second is related to name spaces.

Question 5) True Story?
by travisd ([e-mail])

Was the story about you embarrassing a Microsoftie at a conference true? Specifically, that he was insisting that their implementation of ksh in their unix compatibility kit was true to the "real" thing and trying to argue the point with you. The argument ended when someone else finally stood up and informed the speaker who he was arguing with.

Just curious ...

Korn: This story is true. It was at a USENIX Windows NT conference and Microsoft was presenting their future directions for NT. One of their speakers said that they would release a UNIX integration package for NT that would contain the Korn Shell.

I knew that Microsoft had licensed a number of tools from MKS so I came to the microphone to tell the speaker that this was not the "real" Korn Shell and that MKS was not even compatible with ksh88. I had no intention of embarrassing him and thought that he would explain the compromises that Microsoft had to make in choosing MKS Korn Shell. Instead, he insisted that I was wrong and that Microsoft had indeed chosen a "real" Korn Shell. After a couple of exchanges, I shut up and let him dig himself in deeper. Finally someone in the audience stood up and told him what almost everyone in the audience knew, that I had written the 'real' Korn Shell. I think that this is symbolic about the way the company works.

Question 6) Kind of a shell question...
by update()

There's a lot of squabbling in the Linux world about how the Unix mentality of small apps communicating through standard input/output to form a pipeline should be maintained in the new whiz-bang, GUI environments. Do you think that it can/should be done? What should be the most important considerations for such a messaging system and how should a standard be established?

So Korn (the band) drinks Coors Light? I might have suspected...

Korn: As far as the Korn band is concerned, as you probably already know, it was formed in 1993 to promote the Korn Shell. On the kornshell.com website (click on fun), you can see them endorsing the KornShell Command and Programming Language book, the most complete reference book on ksh93.

There are many people who use UNIX or Linux who IMHO do not understand UNIX. UNIX is not just an operating system, it is a way of doing things, and the shell plays a key role by providing the glue that makes it work. The UNIX methodology relies heavily on reuse of a set of tools rather than on building monolithic applications. Even perl programmers often miss the point, writing the heart and soul of the application as perl script without making use of the UNIX toolkit.

Our aim with ksh, and with about 150 additional tools that are part of the AST Toolkit, (http://www.research.att.com/sw/download), has been to enhance the UNIX tool kit by providing new and improved tools.

Clearly most users prefer GUI interfaces, and GUI interfaces are often hostile to the UNIX methodology. This presents a challenge for the GUI design and unfortunately many GUI applications are built as monoliths with little hope of reuse and make little reuse of the UNIX tool set. I think that the most important consideration when building a system is to separate the GUI from the rest of the system. Anything that can be done with the GUI should be able to be done without the GUI via scripting. This is important if you ever want to automate tasks in the future. Scripting can then be used for automation and for testing.

This tight coupling between GUI and application has other drawbacks as well. It makes it hard to distribute functionality from the client running the GUI to a server that could be somewhere else on the network.

Question 7) What functionality/code in ksh are you least proud of?
by segmond (segmond[at]hotmail)

It is very hard to find a programmer who is completely satisfied with his code. No matter how happy she is with it, there is always that part which she wishes to improve. As far as ksh is concerned what is it that you wish you didn't do, can improve?

Korn: The things that I most regret are not the code itself but mistakes that are in the shell language that I inherited from the Bourne shell code. A lot of effort was made to maintain compatibility while trying to improve ksh at the same time. Compatibility was an extremely important consideration for the adoption of ksh by may organizations. At one point I backed out an improvement because it broke one script in a published UNIX book. If I had my choice, the backquote would disappear as a special character and $() would need to be used instead. Double quoted strings would interpret all of the ANSI-C backslash conventions as well as expand $ expansions. The echo command would not perform any interpretation on its argument. Word splitting would be disabled by default. Note that because I did not make these changes, ksh can safely replace /bin/sh on most systems today unlike zsh which would cause many script to fail in mysterious ways.

There are also a number of features that I introduced into the shell that I wish that I had done differently or not at all. Some obvious examples are the poor design of the fc command, the let command which was superceded with ((...)), and the exporting of attributes for shell variables.

The code has changed a lot over the years which I suspect is an indication that I am never completely satisfied with my code. In fact whenever I look at code I wrote a few years ago, I can't believe how bad it is and I always believe that my new code is finally ok.

What bothers me most about the current code, is that, in spite of its modularity, there are too many places where code in one module only works because of the way another module works. The effect of this is that many changes that I make are not orthogonal and require changes to several parts of the code.

Another weakness of the current code is that it is not completely thread safe.

Question 8) UWIN and etc
by rabtech ([e-mail])

You once said that you had to learn Windows NT because you couldn't criticize what you didn't know. What I'd like to know (as a primarily Windows programmer) is what do you consider to be the best and worst parts of both the Windows NT/2000 model and the UNIX model. What advice can you give? Also, has working on the UWIN project given you any insights that you can share with the rest of the community?

Korn: I have spent the last five years writing a UNIX for windows which is called UWIN (http://www.research.att.com/sw/tools/uwin). My experience is with the WIN32 API for the core OS and not with the WIN32 interface to the GUI. Clearly I had to learn more about the WIN32 API than I think that anyone should need to know. The more I know about the WIN32 API the more I dislike it. It is complex and for the most part poorly designed, inconsistent, and poorly documented.

One of the biggest myths about Windows is that the MS operating systems are compatible, unlike UNIX which has so many variants. In reality, porting from one version of Windows to another is often more difficult than porting from one UNIX or Linux system to another. WIN32 calls on one system are not necessarily implemented on other systems, or even worse, they behave differently, requiring completely different code strategies.

That is not to say that there aren't things that I like about Windows NT. The handle model is good, but would have been better if Microsoft had done it right. The separation of dynamic libraries into interface and implementation also makes sense. Putting the libraries in the same directory as the executables is an improvement over UNIX since the allows executables and libraries to be searched for simultaneously.

The best advice I can give is to not use Windows unless you have to, or if you have to, use a UNIX operating system on top of Windows like UWIN or Cygwin.

One insight that I would give about UWIN is that it proved to be far more difficult that I had anticipated.

Question 9) Ksh Programming For the Web
by Dom2 ([e-mail])

How do you feel that ksh holds up for web programming? I have always enjoyed programming shell scripts more than anything else, but I have always been unhappy with the shell idioms for parsing securely and correctly (the myriad of substitution operators is a nightmare to control). This is one area in which Perl has really taken the lead. How do you think shell programming could be better adapted for the web?

Also, how do you feel about most commercial Unix vendors (and projects like pdksh) that are still shipping ksh88 as the default, 13 years later? ksh93 has many more useful features, but the take up has been slow.

Thanks,
-Dom

Korn: One of my biggest disappointments is the slowness of adoption of ksh93. At the time ksh88 was released, AT&T was a primary UNIX vendor and was able to put it into the standard product. This led to relatively quick adoption for most vendors.

Shell features like here-documents are very well suited for CGI scripts. However, ksh93 is far better as a web scripting language than ksh88. There is not an efficient way to handle the myriad of substitution operators that get passed into a CGI script with ksh88 or any other shell. ksh93 does this rather efficiently with a 'set' discipline which converts the CGI command line into a single compound variable for manipulation by the script. A 'get' discipline converts it back.

The kornshell.com page has the code for a script that can be included in a CGI script in which all the arguments are mapped into shell variables.

Also, recent version of ksh93 have added %H to printf so that strings can be expanded using HTML/XML %xx; form whenever necessary.

The http://www.research.att.com/sw/download website is maintained by ksh scripts in combination with AT&T nmake, and all CGI scripting is done by ksh93.

Question 10) Graphical dtksh, zend, standardizing on ksh93, rpm
by emil

a) Since the code to Motif is now free and open, is there any possibility that dtksh, the ksh93-compliant CDE shell with Motif extensions will be open-sourced? Novel wrote it; will they give it away?

b) Any possibility that you could leverage Zend from php? Or what would it take to implement a dbi-like SQL layer for ksh?

c) Under Solaris and HP-UX, ksh88 is installed in /usr/bin/ksh, ksh93 is installed in /usr/dt/bin/dtksh, but the default shell is the "Posix" shell, a superset of ksh. Is there any hope of getting this mess straightened out?

d) Will I ever expect Red Hat to include an RPM for ksh93? Or would you consider merging with Bash?

Korn: a) I can't say whether Novell will give dtksh away or not. They own the ksh93 extensions and its up to them what they do. I don't think that Motif is either free or open since my understanding is that it cannot be included in commercial products. On the other hand, tksh, which combines ksh93 with tk graphics is freely available. In addition to interpreting shell syntax, tksh can interpret tcl syntax as well.

b) I am not familiar with Zend from php. ksh93 is very extensible. Builtins can be added to ksh and use the same calling convention as tcl uses. There is an API interface to shell variables and other shell functions that can be used with built-in commands. If Zend from php can be added to tcl, then it can be added to ksh93.

c) Since ksh88 is not fully POSIX compliant, some system vendors have modified ksh88 to make it compliant and used that for their POSIX shell. One way to clean up this mess is to get all the vendors to move to ksh93. ksh93 has a single source that compiles on all systems from pc,'s mac's, unix systems, and mainframes. I have no say over what vendors do, but users on these systems certainly can state their preferences.

d) I would hope that Red Hat and other Linux and UNIX vendors would pick up ksh93 and distribute it at least as /bin/ksh. There is no reason why it could not be used as /bin/sh as well and I suspect that it could be merged with bash. We will soon be offering RPMs for the entire AST library from the AT&T website.

Question 11) UWIN and Cygwin
by Lumpish Scholar ([e-mail])

How would you compare the UWIN and Cygwin projects?

Korn: I believe that UWIN and Cygwin started at about the same time although I don't believe that either knew about the other for the first year or two. The focus for Cygwin seemed to be primarily GNU tools rather than a more complete integration. The primary focus on UWIN was to make the AST tools described in now the out of print book "Practical Reusable UNIX Software", run without change on Windows NT and Windows 95.

Cygwin has the advantage of being an OpenSource project whereas UWIN is not, although hopefully this will change in the not too distant future. However, UWIN binaries have been freely available for educational and research use, and available for 90 day evaluation for commercial use. As a result there are more than 100,000 UWIN users and this has helped in discovering problems. The source for nearly all of the UWIN commands are available in open source.

Cygwin relies on the gcc compilation system. UWIN tries to be compiler neutral and works with gcc and Visual C/C++. Since it works with the Visual C/C++ compiler, it is easier to integrate UWIN code with native Visual C/C++ modules.

The website http://www.research.att.com/sw/tools/uwin/survey.html allows users to compare UWIN with other alternatives such as Cygwin in for different categories. The feedback that I have received shows that in most instances UWIN receives higher marks than Cygwin, especially in performance.

Question 12) ksh93 as a programming language?
by Lumpish Scholar ([e-mail])

What are some of the differences between ksh88 (which I think of as comparable to bash or the Posix shell) and ksh93, that makes the latest KornShell as good or better a language than Perl?

Korn: In question 1 I gave a quick summary of ksh93 features that distinguish it from ksh88 and from other shells. For the most part ksh93 has the functionality of perl 5 and arguably a more readable syntax. In most instances the performance is similar. Scripts that interact heavily with the file system or use pipelines tend to be better suited for shell scripting.

Question 13) So, Dave ...
by An Unnamed Correspondent

Quick question: Vi or Emacs?

Korn: This is an old battle that was raging in the early '80s. It will never be settled. This is why ksh has both edit modes. It partly depends on whether you use the editor as a command or as an environment. I think of an editor as a command that I can quickly start and leave and I think of the shell as an environment which holds state. vi fits this model better and that is what I use.

I use several versions of vi depending on which system I am on. UWIN uses nvi from BSD.

Question 14) Public Apology
by watanabe (petervNOMEATivcfne.org)

Dear Dr. Korn, I feel I owe you an apology, and this seemed like a great chance! I went to school with Adam at Brown. And, in 1993, my roommate, Matt Smith, woke me up at about midnight, excitedly telling me "Do you know the Korn Shell? This guy's dad wrote the Korn shell!!!" He was coming in with Adam from somewhere.

I sort of turned over in my sleep, (I was extremely groggy), and said "I hate the Korn shell." and went back to bed. Adam never really talked to me after that, although he was polite enough to me at parties.

So, Dr. Korn, I feel I owe you an apology. I didn't mean to disrespect your lifework in front of your son! I still can't use the korn shell, but call it fear of the unknown, please, rather than lucid comments on your code.

Peter Vessenes, Brown '97

Korn: Since I could not recall the incident, I asked by son Adam to respond to this. Here is his response:

Peter,

Thanks for clearing things up! In my memory of the incident, we had woken you up that evening and I thought you said to me, "I hate the Korn smell." I was mortified! That was the first time anyone ever (publicly) accused me of having b.o., and your comment gave me quite a complex for the many months that followed. I avoided you at all social gatherings because I was scared to death that you might mention this in front of others. I am relieved to learn that this was all the result of a simple miscommunication.

To be honest, I have no recollection of this incident. I do appreciate your apology, but it is not necessary -- just as people are entitled to their choice of music, art, religion, etc, they too should have the right to pick their favorite shell. I'm just thankful I'm not the son of Bill Gates!

Best,
Adam

---

Our thanks to David for his responses, and likewise to Adam for his.