Beginning Portable Shell Scripting 186
Joe MacDonald writes "The earliest UNIX shell I encountered was the Bourne shell on a SPARCStation
2 at my university. As with many students of my generation, prior to that
nearly all of my exposure to command line interfaces was some variant of DOS.
I was quite proficient with the primitive scripting language that was available
on such platforms but I immediately felt far out of my depth in this new
environment. The commands seemed arcane, possibly dangerous, and almost
immediately I regretted stepping into this unfamiliar wilderness without some
sort of guide." Read below for the rest of Joe's thoughts.
Beginning Portable Shell Scripting: From Novice to Professional | |
author | Peter Seebach |
pages | 376 |
publisher | Apress |
rating | 4/5 |
reviewer | Joe MacDonald |
ISBN | 1-4302-1043-5 |
summary | A guide on how to write portable shell scripts. |
It was probably a few weeks after that first, rough introduction that I returned for another round with this strange but somehow seductive tool, armed with a book I'd found and a determination to learn it's secrets. I had no idea then that seventeen years later I'd still be learning new tricks, discovering new features and taking so much pleasure from sharing what I've learned with others. In fact, in those early forays into the realm of shells and scripting, I didn't even really have a strong concept of the separation between the shell and the operating system, so at the time I couldn't have conceived of how much fun I would have in later years discussing and debating the relative strengths and weakness of shells with friends and colleagues, but it is probably my favorite touchstone of computer geek conversation. Discussion of shell features, scripting tricks and semantics almost always result in my learning something new and interesting and having a new tool to add to my collection.
Peter's book, Beginning Portable Shell Scripting, therefore may sound like something intended as a gentle introduction, aimed at the initiate — the sort of text I'd been seeking to carry with me when I first attempted to write what I thought of as "batch files" on that now-ancient UNIX machine — but there's more truth in the subtitle, From Novice to Professional, than one might expect. He writes in an accessible, at times conversational, style and presents detailed technical information alongside a mixture of anecdotes and historical detail that does more than simply serve as a technical reference, it helps the reader understand a great deal about why things are the way they are. It was such an entertaining read that I frequently found myself skipping ahead, reading a section I knew was coming up, then resisting the urge to just keep going from that point. The first of these I encountered on page 18 in which he discusses the relative portability of printf in shell scripts. I knew what he knew, it's clearly non-portable and should be avoided, and thoroughly enjoyed the explanation of how he determined his (and by extension my) assumption was in error. Another on page 108 is the sort of good advice all UNIX users, not just those aiming to write good scripts, should take to heart. Many times, though, I've related precisely the same advice to colleagues to be met with confused stares, so it certainly bears repeating.
This book is a desktop reference in the truest sense of the term for me, it is an interesting, at times laugh-out-loud amusing, discussion of how to write shell scripts that will work on the widest possible range of Bourne-derived and POSIXly correct shells and why this is a desirable goal. In true UNIX tradition, the author doesn't provide simply a set of rules, but guidelines that will help you find your own way through the task of creating portable, maintainable shell scripts.
The real meat of the book begins in Chapter 3 (more on Chapter 2 in a moment) with a discussion of control structures and redirection, the latter being perhaps the defining characteristic of UNIX command line interfaces. I struggled somewhat with trying to decide if redirection would be better discussed after the material on how the shell parses tokens, presented in the first part of Chapter 4, but it does seem that the correct logical grouping is the one presented. It would be easy to get lost, for example, in the semantics of why the same streams of redirection tokens behave differently on different shells, but the key concept in the early chapters is that of many tools, each doing a specific task, working in concert. That objective is achieved quite effectively.
Chapters 5 and 6 go into detail (possibly too much for some, just right in my opinion) on how UNIX executes shells and how shells can spawn other shells, the costs and the benefits and the available alternatives for one to make an informed decision. Frequently there isn't one right answer whether some activity is better done in a script, in a shell function or in a subshell, but the material here will certainly aid in making those determinations. My personal bias being almost always toward writing a shell function — perhaps an indication I've had too much exposure to C programming, perhaps more due to a frugal upbringing and my own sense that spawning a whole new shell to do something is overkill — had me wishing for a larger section on the value of such constructs, but there should be enough there for me to win some converts to my cause.
By far the sections I learned the most from, however, would be Chapter 7: Shell Language Portability and Chapter 8: Utility Portability since I actively avoid exposure to other shells. I have my two preferred options and a third that I will use when presented with no alternative. While this does mean I know "my own" shells very well, it also means that I often bump into the furniture, so to speak, when I find myself using a new shell. These chapters haven't been immediately useful to me, but I know they're the ones that I'll be turning to in the future, I've needed something like them in the not-too-distant past, after all.
The final three chapters assemble the information presented in the earlier sections and suggest a sort of "best practices" approach to writing scripts. Concepts like "degrade gracefully" seem like pretty fundamental ideas when you hear them but I frequently find myself writing functions or scripts that don't do that at all when intended for a limited, usually singular, audience. It may seem like an okay idea when you're doing something for your own use, but when you write a complex function that works then discover a bug in it two or three years late and you have to return to fix it, it can be just as helpful for it to simply fail in an informative way as it would be to have detailed comments explaining the intent and the mechanics.
Truly, there's something here for everyone. In my office I'm considered something of an expert when it comes to complex regular expressions and the subtleties of using them in different editors and tools, but Chapter 2 and Appendix C both had enough new material in them that I found myself frequently making notes in the margins.
I have many, many books in my bookshelf in my office but nearly none on my desk. Beginning Portable Shell Scripting is going to be one of the very few that will be spending a great deal of time lying flat on my desk, in easy arm-reach.
You can purchase Beginning Portable Shell Scripting from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
If you have a choice... (Score:5, Insightful)
Don't do it.
Shell scripts have horrible error handling, and quickly become a maintenance nightmare. These days, e.g. Python is installed everywhere you need to go.
Just do this:
def c(s): os.system(c)
and you have mostly covered the area where shell scripts excel. You can still write minimal "shell scripts" inside c().
Unluckily, you still *need* to grok shell scripting to some extent, or at least be able to read them. Just don't write them if you can help it.
Python still can't replace quick scripting (Score:4, Insightful)
I find shell scripting have a nasty habit of not working quiet right when moved between Linux, the BSDs and Mac to be safe, and it's always a pain to write scripts that work correctly with spaces in file names.
Why isn't there (or is there?) a simple python cheat guide, or library, that do the same things as grep, awk, find, mv and xargs?
Shell scripts are a glue language (Score:5, Insightful)
One does not write a web server in Bash, one wraps a webserver in it, pipes its output to a log analyzer, restarts it automatically if it crashes, and so on.
The most important part of any UNIX-derived shell langauge is not its syntax or power but the fact it lets you construct large ad-hoc applications out of a toolbox of tens of thousands of pieces.
This is where all other operating systems (that I've ever used, and that's 30-40) have failed.
Any serious developer should know several glue languages, Unix shells being the most flexible and accessible.
Re:If you have a choice... (Score:5, Insightful)
I agree!
My personal limit is 10 lines for a shell script. If is longer than that I convert it to Python.
Python scripts have the advantage that they work on Windows too, and they have lots of os independent abstractions for file names, processes etc.
Why learn an arcane language like sh when you can learn a nice well structured language like Python and write better scripts?
A few years ago I would have used Perl rather than Python, but I'm converted now ;-)
PSS can be a recurring problem (Score:3, Insightful)
I'm sure all of us at one time or another have had a shell script we've relied on for years fail miserably when bringing it to a new environment. The sad fact is, shell scripts were never meant to be programming languages in and of themselves, and I wonder if, knowing what we know now, it isn't overly ego-driven and masochistic to try to take this feature -- tied to a shell which is tied to an operating system -- and promote it beyond its competency.
So, let's say we take the PSS principles seriously, and abstract away any non-platform-agnostic features you can think of. A few years down the road, you've got PSS all over the shop and you want to upgrade to a different platform nominally supporting your shell of choice. Even if you shake off PSS features you thought could create incompatibilities, you discover the new system buffers differently. Or added a parameter somewhere. Point is, if you went with something like Perl which is designed for cross-compatibility you would have been fine, but now you're all wet.
Shell scripting is good for what it's meant for, but at the risk of oversimplifying with a tool analogy, I'm concerned that this falls into the trap of "If all you have is a hammer, all your problems look like nails".
Re:If you have a choice... (Score:4, Insightful)
That's exactly what the Perl people said years ago, and we all know how well that worked out (for low maintenance sysadmin-type tasks). I know the sinking feeling I get every time I find a crontab entry pointing to a Perl script.
Re:more than this? (Score:3, Insightful)
Yes. ... Well, come on, what do you expect me to say?
Re:Oh I hate those [ "X$var" == "X" ] (Score:2, Insightful)
Re:Hardly Need a Whole Book (Score:3, Insightful)
Sounds great.
Now, off the top of your head, what happens to variables set in the last component of a pipeline in ksh? Do you know whether it's the same on systems where "/bin/ksh" is actually pdksh? ... Oh, and just for reference, about half the Linux systems our IT department installs don't have ksh. No, I don't know why. (I only know because I can't log into them because my default login shell is /bin/ksh...)
Re:Why Buy? (Score:3, Insightful)
Well, I suppose because the contents are different. We're answering different questions.
I would never dream of discouraging people from looking at and using the various free guides out there. The autoconf manual is full of useful information about portability, too.
There's more than one article or book because there's more than one topic. "Shell Scripting" is not a single topic; "portable shell" is very different from "advanced bash". It solves different problems, and is useful for different circumstances.
Re:If you have a choice... (Score:2, Insightful)
Re:If you have a choice... (Score:3, Insightful)
Why learn an arcane language like sh when you can learn a nice well structured language like Python and write better scripts?
Where I work pretty much everything has bash already (I install cygwin on all the Windows boxes. Of course, Python is usually there too :) ).
If you already have a bash script (or find one via the Google), changing it is usually simpler than porting it to Python.
If you work with people that already know bash scripting but don't know Python using the lowest common denominator can be easier than training.
There is less memory overhead for a simple shell script than there is for a simple Python script.
This being said, I try to keep my bash scripts under 40 lines and port them to python if they get more complex and I know the machine I'm going to be using them on has Python.
Re:If you have a choice... (Score:4, Insightful)
That's not what they said years ago. People arguing to use Perl instead of Bash did so because Perl was just far more functional. Perl and Bash both have pretty terrible maintainability, but Perl is a million times more functional.
Python and Ruby have the functionality of Perl without the maintenance issues inherent in a language which is really a hodge-podge of ancient unix idioms.
Re:portable shell scripting is an oxymoron (Score:3, Insightful)
Yes, you're quite right that this says a lot about poor user interface choices in some utilities.
Thinking about it more: One of the key applications of portability involves scripts which are never ported.
It's that I don't use only one machine. I use a Mac desktop, a BSD server, and some Linux servers. Even if I'm just typing "for i in..." on the command line, I don't want to try to remember three or four different sets of commands to work across these environments. I want to write things that work the same everywhere -- not just so I can run them multiple places, but so it doesn't matter which machine I'm on when I'm typing.
Re:If you have a choice... (Score:5, Insightful)
maintenance issues inherent in a language which is really a hodge-podge of ancient unix idioms.
What a ridiculous claim, there are no "maintenance issues" with ancient idioms... The very fact that those techniques are ancient shows how incredibly flexible and useful they are. I'd much rather use conventions which are widely accepted and in many cases are required by Posix/SUS/XPG4 than find myself having to hack up my stuff to accommodate broad and pervasive changes such those experienced when moving from python 2.x to python 3.x...
People who are constantly advocating against shell scripts tend to be those who see system administration as something it isn't; namely a low level development job. When in reality a sys admin uses shell scripts to glue together existing products of developers in order to manage administrative tasks. If I were an auto mechanic no one would propose that I learn to master a casting foundry and a milling machine in order to work on cars, those are clearly manufacturing/development tools AND certainly no good mechanic would suggest that using a wrench to fasten a nut to a bolt is "a hodge podge of ancient idioms" which should be replaced with whatever flavour of the week fastening system and power tool happens to be popular at the moment.
Sure there are some arcane aspects to shell scripting, but when I learned Unix in college they taught a thing called "the unix philosophy" which basically said that you should always use the smallest tools for the job, leverage the pipes/redirection, and build to a usable script which doesn't replicate existing functionality of ubiquitous tools. Seems like these days every python/perl wizard around fancies themselves an administrator and yet they waste a large portion of their time rewriting tried and true unixisms; sort, wc, cut, paste, tee, etc...
Also, get off my lawn!