Beginning Portable Shell Scripting 186
Joe MacDonald writes "The earliest UNIX shell I encountered was the Bourne shell on a SPARCStation
2 at my university. As with many students of my generation, prior to that
nearly all of my exposure to command line interfaces was some variant of DOS.
I was quite proficient with the primitive scripting language that was available
on such platforms but I immediately felt far out of my depth in this new
environment. The commands seemed arcane, possibly dangerous, and almost
immediately I regretted stepping into this unfamiliar wilderness without some
sort of guide." Read below for the rest of Joe's thoughts.
Beginning Portable Shell Scripting: From Novice to Professional | |
author | Peter Seebach |
pages | 376 |
publisher | Apress |
rating | 4/5 |
reviewer | Joe MacDonald |
ISBN | 1-4302-1043-5 |
summary | A guide on how to write portable shell scripts. |
It was probably a few weeks after that first, rough introduction that I returned for another round with this strange but somehow seductive tool, armed with a book I'd found and a determination to learn it's secrets. I had no idea then that seventeen years later I'd still be learning new tricks, discovering new features and taking so much pleasure from sharing what I've learned with others. In fact, in those early forays into the realm of shells and scripting, I didn't even really have a strong concept of the separation between the shell and the operating system, so at the time I couldn't have conceived of how much fun I would have in later years discussing and debating the relative strengths and weakness of shells with friends and colleagues, but it is probably my favorite touchstone of computer geek conversation. Discussion of shell features, scripting tricks and semantics almost always result in my learning something new and interesting and having a new tool to add to my collection.
Peter's book, Beginning Portable Shell Scripting, therefore may sound like something intended as a gentle introduction, aimed at the initiate — the sort of text I'd been seeking to carry with me when I first attempted to write what I thought of as "batch files" on that now-ancient UNIX machine — but there's more truth in the subtitle, From Novice to Professional, than one might expect. He writes in an accessible, at times conversational, style and presents detailed technical information alongside a mixture of anecdotes and historical detail that does more than simply serve as a technical reference, it helps the reader understand a great deal about why things are the way they are. It was such an entertaining read that I frequently found myself skipping ahead, reading a section I knew was coming up, then resisting the urge to just keep going from that point. The first of these I encountered on page 18 in which he discusses the relative portability of printf in shell scripts. I knew what he knew, it's clearly non-portable and should be avoided, and thoroughly enjoyed the explanation of how he determined his (and by extension my) assumption was in error. Another on page 108 is the sort of good advice all UNIX users, not just those aiming to write good scripts, should take to heart. Many times, though, I've related precisely the same advice to colleagues to be met with confused stares, so it certainly bears repeating.
This book is a desktop reference in the truest sense of the term for me, it is an interesting, at times laugh-out-loud amusing, discussion of how to write shell scripts that will work on the widest possible range of Bourne-derived and POSIXly correct shells and why this is a desirable goal. In true UNIX tradition, the author doesn't provide simply a set of rules, but guidelines that will help you find your own way through the task of creating portable, maintainable shell scripts.
The real meat of the book begins in Chapter 3 (more on Chapter 2 in a moment) with a discussion of control structures and redirection, the latter being perhaps the defining characteristic of UNIX command line interfaces. I struggled somewhat with trying to decide if redirection would be better discussed after the material on how the shell parses tokens, presented in the first part of Chapter 4, but it does seem that the correct logical grouping is the one presented. It would be easy to get lost, for example, in the semantics of why the same streams of redirection tokens behave differently on different shells, but the key concept in the early chapters is that of many tools, each doing a specific task, working in concert. That objective is achieved quite effectively.
Chapters 5 and 6 go into detail (possibly too much for some, just right in my opinion) on how UNIX executes shells and how shells can spawn other shells, the costs and the benefits and the available alternatives for one to make an informed decision. Frequently there isn't one right answer whether some activity is better done in a script, in a shell function or in a subshell, but the material here will certainly aid in making those determinations. My personal bias being almost always toward writing a shell function — perhaps an indication I've had too much exposure to C programming, perhaps more due to a frugal upbringing and my own sense that spawning a whole new shell to do something is overkill — had me wishing for a larger section on the value of such constructs, but there should be enough there for me to win some converts to my cause.
By far the sections I learned the most from, however, would be Chapter 7: Shell Language Portability and Chapter 8: Utility Portability since I actively avoid exposure to other shells. I have my two preferred options and a third that I will use when presented with no alternative. While this does mean I know "my own" shells very well, it also means that I often bump into the furniture, so to speak, when I find myself using a new shell. These chapters haven't been immediately useful to me, but I know they're the ones that I'll be turning to in the future, I've needed something like them in the not-too-distant past, after all.
The final three chapters assemble the information presented in the earlier sections and suggest a sort of "best practices" approach to writing scripts. Concepts like "degrade gracefully" seem like pretty fundamental ideas when you hear them but I frequently find myself writing functions or scripts that don't do that at all when intended for a limited, usually singular, audience. It may seem like an okay idea when you're doing something for your own use, but when you write a complex function that works then discover a bug in it two or three years late and you have to return to fix it, it can be just as helpful for it to simply fail in an informative way as it would be to have detailed comments explaining the intent and the mechanics.
Truly, there's something here for everyone. In my office I'm considered something of an expert when it comes to complex regular expressions and the subtleties of using them in different editors and tools, but Chapter 2 and Appendix C both had enough new material in them that I found myself frequently making notes in the margins.
I have many, many books in my bookshelf in my office but nearly none on my desk. Beginning Portable Shell Scripting is going to be one of the very few that will be spending a great deal of time lying flat on my desk, in easy arm-reach.
You can purchase Beginning Portable Shell Scripting from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Re:Python still can't replace quick scripting (Score:4, Informative)
Why isn't there (or is there?) a simple python cheat guide, or library, that do the same things as grep, awk, find, mv and xargs?
re.findall, s.split(), os.walk, shutil.move,
" ".join
Re:Hardly Need a Whole Book (Score:5, Informative)
Re:If you have a choice... (Score:5, Informative)
Python is nice, but hardly installed everywhere. It's available on Linux certainly, but not always on AIX or Solaris. Yes, it is just an installation away, but many of the systems I maintain require change management procedures to even chmod a file.
Shell scripts do have decent error handling for what they need to do. With traps and proper usage of error codes, they are not much different from lower level languages.
I'd agree that I now *prefer* to write longer scripts in Python. However, few of the people I work with know Python, or even Perl. They can get around with korn and bourne as these are the default scripting languages on more traditional Unix systems.
Which comes down to the gist of the issue. Do you write code in a language you prefer or one that can be maintained by the admins? I'd argue that it doesn't matter what language you use. If you write poor code in shell you will likely write poor code in Python too.
Just some general comments... (Score:5, Informative)
First off, in the interests of full disclosure, Joe MacDonald is one of my coworkers.
Anyway... The big surprise to me was the word "Beginning", which somehow showed up in the publisher's cover pages, but which I didn't know about during the writing process. My tech reviewer was Gary V. Vaughan (yes, the autoconf/libtool guy). I bounced material off a number of seasoned expert scripters during the process. Basically, my goal was to write a book that I could use as a reference, and which would teach me something.
I succeeded beyond my wildest dreams. The discovery that printf(1) is essentially universal these days was a complete shock to me; I had no idea it was portable. During my first pass on the regular expressions section, I started by writing down what I believed I knew about sed, awk, etcetera. Then I tested it... and had to revise most of it. A number of things I was used to were GNU or BSD extensions. When Gary sent the chapter back for tech review, he'd flagged most of these things, because he "knew" the same things I did.
So everything there should be pretty thoroughly checked out now -- I realized very early on that this field was full of things "everyone knows". Many of them wrong. We tested things on a field of around 30 different versions of Unix and Linux. We tested them on unusual installs, we tested them on special cases.
Why?
Because portable shell is an incredibly portable language, and sometimes that matters. Because shell is a very powerful language, too. Because sometimes shell is all you have -- and because sometimes shell is more expressive for a task than your other choices. I love me some C, I program in C by preference much of the time -- but there are a lot of tasks I'll do in shell rather than in C. There are similarly many tasks I'd rather write in shell than in perl. Shell is what make uses to run commands, and sometimes you need to write something clever in shell because make doesn't have quite the right feature.
In short, it's something I have found consistently useful, day in and day out, for about twenty years now. I just wish I'd realized how much more there was to learn years ago, I coulda saved a lot of time... :)
And, to answer a question hinted at earlier: Yes, now that this book exists, I keep a copy on my desk. I look stuff up in it about once a week.
Re:portable shell scripting is an oxymoron (Score:4, Informative)
As usual, it comes down to use cases. Describe the useful things that are done.
Take a naked box and boot it (/etc/init.d/*) ?
I know it's a bit trivial but it still qualifies as useful in my book.
Actually /etc is just pretty much a collection a collection of fairly useful shell scripts. I've always found it interesting that Unix was mostly held together by /bin/sh (aka /bin/bash on a lot of systems nowadays) and spit. And that it worked.
To take one of the posts above where the poster had been exposed to DOS. The DOS system (although it wasn't really a system, merely a program loader) was configured by the autoexec script. All the Unix do the same with a number of chained scripts (and their order can even dynamically change nowadays) all running sh (or an extended version of it).
I still wonder at it sometimes. It's simple and accessible on one side. And it can degenerate into an awful mess on the other :) (less so nowadays thankfully)
More ?
Anyway, wanted something useful the shell could do ? How about run the whole operating system (find a service that isn't actually handled by a !#/bin/sh script...).
Re:portable shell scripting is an oxymoron (Score:3, Informative)
Check out GNU autoconf. That's a good example of how a script works on *nix box. And, yes, they are useful!
Autoconf is also a horrible peace of crap. One of the better reason to hate the concept of shell scripts, actually.
Check out SCons for comparison:
Program('hello.c')
or
SharedLibrary('foo', ['f1.c', 'f2.c', 'f3.c'])
And that's pretty much it. I'm not sure all the horrors required by autoconf would fit into a slashdot posting.
Re:If you have a choice... (Score:2, Informative)
> These days, e.g. Python is installed everywhere you need to go.
Sorry, but no, it isn't.
Re:Oh I hate those [ "X$var" == "X" ] (Score:5, Informative)
Why bother with portable shell scripts, seriously? Everybody has bash installed, and/or zsh that is mostly compatible, and even then you have bash anyway. I understand retro-nostalgia and all that, but necrophilia is overrated
False.
The majority of systems I work on these days and the majority of systems I have worked on since the mid 90's have not had bash installed. That includes systems running FreeBSD, NetBSD, OpenBSD, AIX, Tru64, Solaris, MacOS, and even Linux. Current versions of some of those will usually have bash in a default installation, but some still do not. Companies running stable systems as important parts of their business do not generally upgrade their OS's just for the sake of novelty. Running older systems isn't usually about nostalgia or necrophilia, it's more often about not having any compelling reason to upgrade. There is also a system hygiene practice common on the BSD's of keeping the base system minimal and only adding on what is needed, a practice that helps in keeping systems secure and stable because they are easier to fully understand. This is also common in many virtualization environments, where a running OS instance is likely to exist for a very narrow purpose and intentionally have a stripped-down set of utilities fit to that narrow purpose.
Re:portable shell scripting is an oxymoron (Score:4, Informative)
Portability isn't boolean.
I wrote a wrapper around cdrecord to clean up the UI, automatically handle things like creating an isofs from directories, and so on.
It's not 100% portable; every new system, I change the path to cdrecord, the device spec for the CD drive, and the command used to eject a CD.
Everything ELSE stays the same, and I don't need to remember how to use mkisofs, or anything like it. Directories, bzipped images, whatever; it gets burned correctly. I win.
If the script were not written in otherwise-portable shell, it might not work on the broad variety of boxes I've wanted to use it on.
I've done scripts to handle tasks like "open this file" (not as flexible or smart as the OS X one, but quite good about various compressed tarballs and archives). Surprisingly portable.
I have a script for the idiom of "for every file named or provided as standard input, run it through this filter in place". Repeating commands at intervals, for a given number of times, until they fail... Tons of little utilities like this that save me time.
If you want complete applications with no dependencies, that's harder to find. That said... Have you ever used autoconf to configure something? That's a fair bit of portable shell right there...
Re:If you have a choice... (Score:4, Informative)
Shell scripts have horrible error handling, and quickly become a maintenance nightmare. These days, e.g. Python is installed everywhere you need to go.
Python doesn't help much over shell scripts without extra libraries, which may or may not be present on any given system.
Python has changed incompatibly several times already.
Python has a large startup overhead:
20 seconds: 1000x python -c 'print("test")'
2 seconds: 1000x sh -c 'echo test'
Python is clumsy to use for gluing several programs together.
Python is not the same syntax as the shell. If you don't learn the shell then your day-to-day command lines are gimped.
So Ruby or Python or anything else is better for writing actual programs that do anything complicated, but there are plenty of appropriate uses for shell scripting. Ruby is actually much better... since it has a sensible syntax you could make a rubysh that wouldn't suck.
Re:If you have a choice... (Score:4, Informative)
sed -n -e '/<title>/ {
s/.*<title>\(.*\)<\/title>.*/\1/p
q
}' < musicbrainz.xml
Forgot that "plain old text" isn't. (To verify I had it right, of course, I pasted it into a file with the original and then ran
!sort | uniq)