Beginning Portable Shell Scripting 186
Joe MacDonald writes "The earliest UNIX shell I encountered was the Bourne shell on a SPARCStation
2 at my university. As with many students of my generation, prior to that
nearly all of my exposure to command line interfaces was some variant of DOS.
I was quite proficient with the primitive scripting language that was available
on such platforms but I immediately felt far out of my depth in this new
environment. The commands seemed arcane, possibly dangerous, and almost
immediately I regretted stepping into this unfamiliar wilderness without some
sort of guide." Read below for the rest of Joe's thoughts.
| Beginning Portable Shell Scripting: From Novice to Professional | |
| author | Peter Seebach |
| pages | 376 |
| publisher | Apress |
| rating | 4/5 |
| reviewer | Joe MacDonald |
| ISBN | 1-4302-1043-5 |
| summary | A guide on how to write portable shell scripts. |
It was probably a few weeks after that first, rough introduction that I returned for another round with this strange but somehow seductive tool, armed with a book I'd found and a determination to learn it's secrets. I had no idea then that seventeen years later I'd still be learning new tricks, discovering new features and taking so much pleasure from sharing what I've learned with others. In fact, in those early forays into the realm of shells and scripting, I didn't even really have a strong concept of the separation between the shell and the operating system, so at the time I couldn't have conceived of how much fun I would have in later years discussing and debating the relative strengths and weakness of shells with friends and colleagues, but it is probably my favorite touchstone of computer geek conversation. Discussion of shell features, scripting tricks and semantics almost always result in my learning something new and interesting and having a new tool to add to my collection.
Peter's book, Beginning Portable Shell Scripting, therefore may sound like something intended as a gentle introduction, aimed at the initiate — the sort of text I'd been seeking to carry with me when I first attempted to write what I thought of as "batch files" on that now-ancient UNIX machine — but there's more truth in the subtitle, From Novice to Professional, than one might expect. He writes in an accessible, at times conversational, style and presents detailed technical information alongside a mixture of anecdotes and historical detail that does more than simply serve as a technical reference, it helps the reader understand a great deal about why things are the way they are. It was such an entertaining read that I frequently found myself skipping ahead, reading a section I knew was coming up, then resisting the urge to just keep going from that point. The first of these I encountered on page 18 in which he discusses the relative portability of printf in shell scripts. I knew what he knew, it's clearly non-portable and should be avoided, and thoroughly enjoyed the explanation of how he determined his (and by extension my) assumption was in error. Another on page 108 is the sort of good advice all UNIX users, not just those aiming to write good scripts, should take to heart. Many times, though, I've related precisely the same advice to colleagues to be met with confused stares, so it certainly bears repeating.
This book is a desktop reference in the truest sense of the term for me, it is an interesting, at times laugh-out-loud amusing, discussion of how to write shell scripts that will work on the widest possible range of Bourne-derived and POSIXly correct shells and why this is a desirable goal. In true UNIX tradition, the author doesn't provide simply a set of rules, but guidelines that will help you find your own way through the task of creating portable, maintainable shell scripts.
The real meat of the book begins in Chapter 3 (more on Chapter 2 in a moment) with a discussion of control structures and redirection, the latter being perhaps the defining characteristic of UNIX command line interfaces. I struggled somewhat with trying to decide if redirection would be better discussed after the material on how the shell parses tokens, presented in the first part of Chapter 4, but it does seem that the correct logical grouping is the one presented. It would be easy to get lost, for example, in the semantics of why the same streams of redirection tokens behave differently on different shells, but the key concept in the early chapters is that of many tools, each doing a specific task, working in concert. That objective is achieved quite effectively.
Chapters 5 and 6 go into detail (possibly too much for some, just right in my opinion) on how UNIX executes shells and how shells can spawn other shells, the costs and the benefits and the available alternatives for one to make an informed decision. Frequently there isn't one right answer whether some activity is better done in a script, in a shell function or in a subshell, but the material here will certainly aid in making those determinations. My personal bias being almost always toward writing a shell function — perhaps an indication I've had too much exposure to C programming, perhaps more due to a frugal upbringing and my own sense that spawning a whole new shell to do something is overkill — had me wishing for a larger section on the value of such constructs, but there should be enough there for me to win some converts to my cause.
By far the sections I learned the most from, however, would be Chapter 7: Shell Language Portability and Chapter 8: Utility Portability since I actively avoid exposure to other shells. I have my two preferred options and a third that I will use when presented with no alternative. While this does mean I know "my own" shells very well, it also means that I often bump into the furniture, so to speak, when I find myself using a new shell. These chapters haven't been immediately useful to me, but I know they're the ones that I'll be turning to in the future, I've needed something like them in the not-too-distant past, after all.
The final three chapters assemble the information presented in the earlier sections and suggest a sort of "best practices" approach to writing scripts. Concepts like "degrade gracefully" seem like pretty fundamental ideas when you hear them but I frequently find myself writing functions or scripts that don't do that at all when intended for a limited, usually singular, audience. It may seem like an okay idea when you're doing something for your own use, but when you write a complex function that works then discover a bug in it two or three years late and you have to return to fix it, it can be just as helpful for it to simply fail in an informative way as it would be to have detailed comments explaining the intent and the mechanics.
Truly, there's something here for everyone. In my office I'm considered something of an expert when it comes to complex regular expressions and the subtleties of using them in different editors and tools, but Chapter 2 and Appendix C both had enough new material in them that I found myself frequently making notes in the margins.
I have many, many books in my bookshelf in my office but nearly none on my desk. Beginning Portable Shell Scripting is going to be one of the very few that will be spending a great deal of time lying flat on my desk, in easy arm-reach.
You can purchase Beginning Portable Shell Scripting from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
more than this? (Score:3, Interesting)
Re:If you have a choice... (Score:2, Interesting)
I can't agree more. I switched over to python for all non-trivial scripting a couple of years ago, and I find it much more pleasant. I even sometimes use iPython instead of bash when I know I'll need to do something complex interactively.
By the way - if you like using python to control systems, you might also enjoy the func project [fedorahosted.org].
Re:portable shell scripting is an oxymoron (Score:3, Interesting)
As usual, it comes down to use cases. Describe the useful things that are done. One might choose to write a shell script to perform some pure mathematical utility function, but this certainly isn't the usual role of such scripts. Rather, one uses a shell script when accessing files (logs, etc.) on the disk, or when opening sockets, or when spawning host level commands. Other device level access is often required, for instance, a local or UTC clock might be consulted, requiring knowledge of timezones. All of these are very dependent on whether the script is running under a BSD or Sys-V flavored OS (assuming Unix, of course), or even on micro-versioning of specific OSes.
The question of complexity is also not some challenge to find a lengthy script, but rather a statement of the inherent design-level complexity of the application.
Re:portable shell scripting is an oxymoron (Score:5, Interesting)
Well, there is some truth to the GPP's comment. Linux and Mac OS X don't even agree on how to tell echo not to print a newline or how to enable extended regular expression mode in sed. May heaven help you if you want to do something as esoteric as creating or mounting a filesystem, creating or mounting a disk image/ramdisk, talk to a USB device in any way, get a list of processes in any useful way, etc. There's a very big lack of standardization in a lot of things you might like to do with scripts, in other words. The Single UNIX Spec and POSIX are not quite sufficient, but more annoyingly, most OSes (Linux, *BSD) out there don't even come close to conforming to it, so you end up with this dichotomy between BSD behavior and AT&T behavior.
That said, a lot of things are standardized, and many others can be worked around with clever use of variables (or possibly eval in a few extreme cases). I've written chapters on the subject myself. The big things you need to remember are that $(( $FOO + 3 )) is not portable, nor for ((...)), nor >&, nor anything involving extended regexp except using Perl, that even "the one true awk" is not quite SUS-compliant, GNU awk doubly so, bash triply so, that you should use printf instead of echo for output if you don't want newlines, that signal numbers are not portable (for trap), that proper quoting of arguments is crucial, and that you need to work with the bare minimum base behavior of utilities (using few or no flags) if you expect any hope of portability without needing to make platform-specific changes.
For some quick examples of some interesting portability issues, read some of my comments in the games at shellscriptgames.com or search for the word "compatibility" in Apple's "Shell Scripting Primer". It's a real eye opener to see how many portability problems exist even for fairly simple shell scripts.
Re:Shell scripts are a glue language (Score:5, Interesting)
I accept your challenge. :-D
But seriously, yeah, you're absolutely right. Ooh, but a basic web server written as a Bourne shell script called by inetd would be so freaking cool....
Oh, no. Somebody actually did that [debian-adm...ration.org].... Yikes! Now I'm scared.
Re:If you have a choice... (Score:3, Interesting)
No it isn't. Not even remotely close.
And in most cases the response to a request to install an entire programming language would be flat rejection, turning to raucous laughter when they realise you only want it because you don't like any of the several scripting environments that are already available.
Re:Nobody's seriously comparing C and shell (Score:3, Interesting)
I compare C and shell all the time. Sometimes the answer surprises me. e.g., until I knew about printf(1), I sometimes went to C if I needed to pretty-print output. Now sometimes I don't.
I will happily mix and match multiple languages; one of my first shipping products was written in shell, perl, and C. Each did some things well that the others didn't...
I tend not to use Ruby for things that I want to be portable, because not everything has Ruby around yet. I tend to avoid Python because it just never "clicked" for me. I use perl for self-contained programs, but I don't like it very much for code that has to run other commands or manipulate files.
To this day, I've never found any of the competing languages to be as expressive as shell for things that are essentially sequences of Unix commands. If a task is something I would normally perform sitting at a prompt typing, I tend to find shell to be the language closest to the problem space.
Re:If you have a choice... (Score:1, Interesting)
Why learn an arcane language like sh
Because sometimes it is exactly the right tool for the job. There are many tasks for which python or perl are overkill. Specifically, things which could be accomplished interactively on the command line (like repetitive tasks) but are better automated. Writing shell scripts also makes you more proficient on the command line, which is a valuable skill for any programmer or sysadmin. Your knowledge of command line tools and where/when to use them will surely increase.
#!/bin/sh
cat musicbrainz.xml | sed 's/>/>\n/g' | grep 'title>$' | sed 's/musicbrainz.txt
This is a quick and dirty script I use to convert a musicbrainz [musicbrainz.org] XML file (found on the details page of each album) to straight text. It's part of a collection of scripts which make up my custom tagging and music management system. It works great. But imagine what the equivalent script would look like in python or perl. A lot longer, for starters.
Re:Shell scripts are a glue language (Score:5, Interesting)
One of the points might be that there's a fairly specialized task which takes a person six hours to do, but which is NEARLY all done automatically -- just a bit of hand twiddling.
Lemme give an example that isn't portable. One of the things I do fairly frequently is take about six large toolchains distributed to me as binaries and source tarballs, and turn them into patches against upstream versions, reorganize them, delete some unused files, create configuration files that refer to the binaries, generate md5sums, and so on.
This is a task which, if I sit down at 10AM and start typing, is usually done by about 4PM. Testing takes a bit longer, and usually uncovers SOME kind of typo or thing I forgot to do.
Enter the shell script.
I tell the script where the files are, and I walk away. An hour later I have the results. Testing is also automated (another script). But testing is also uneventful, because the script never forgets a step, makes a typo, or otherwise screws up.
By the second time I did this, the script had saved me time. By the third, it had saved me close to a full working day. By now it's closer to a week of my time that wasn't spent messing with this stuff.
Portability isn't entirely crucial here, you might think? Well, not ENTIRELY crucial, except that when they had me start doing this on a new box running a different variety of Linux, the total time I spent revising the script was 0 minutes.
Re:If you have a choice... (Score:2, Interesting)
One reason may be that Python's startup time is an order of magnitude slower that a shell's, and sometimes it does matter, even for scripts of much more than 10 lines.
Re:If you have a choice... (Score:4, Interesting)
Re:If you have a choice... (Score:1, Interesting)
It describes most IT environments in large corporations. I've never seen Python installed on Solaris boxes unless they were running Cognos. And I've never seen Python on HP-UX, AIX or IRIX boxes either. That's why the "portable" part in the article title is important. Too many people on slashdot without "enterprise" experience assume the software in their Linux distros is installed on all UNIX systems. I've had to deal with "network appliances" running a specialized, proprietary UNIX-like OS that have Bourne shell, vi, and not a whole lot else. Installing Python is not even an option.