Beginning Portable Shell Scripting 186
Joe MacDonald writes "The earliest UNIX shell I encountered was the Bourne shell on a SPARCStation
2 at my university. As with many students of my generation, prior to that
nearly all of my exposure to command line interfaces was some variant of DOS.
I was quite proficient with the primitive scripting language that was available
on such platforms but I immediately felt far out of my depth in this new
environment. The commands seemed arcane, possibly dangerous, and almost
immediately I regretted stepping into this unfamiliar wilderness without some
sort of guide." Read below for the rest of Joe's thoughts.
| Beginning Portable Shell Scripting: From Novice to Professional | |
| author | Peter Seebach |
| pages | 376 |
| publisher | Apress |
| rating | 4/5 |
| reviewer | Joe MacDonald |
| ISBN | 1-4302-1043-5 |
| summary | A guide on how to write portable shell scripts. |
It was probably a few weeks after that first, rough introduction that I returned for another round with this strange but somehow seductive tool, armed with a book I'd found and a determination to learn it's secrets. I had no idea then that seventeen years later I'd still be learning new tricks, discovering new features and taking so much pleasure from sharing what I've learned with others. In fact, in those early forays into the realm of shells and scripting, I didn't even really have a strong concept of the separation between the shell and the operating system, so at the time I couldn't have conceived of how much fun I would have in later years discussing and debating the relative strengths and weakness of shells with friends and colleagues, but it is probably my favorite touchstone of computer geek conversation. Discussion of shell features, scripting tricks and semantics almost always result in my learning something new and interesting and having a new tool to add to my collection.
Peter's book, Beginning Portable Shell Scripting, therefore may sound like something intended as a gentle introduction, aimed at the initiate — the sort of text I'd been seeking to carry with me when I first attempted to write what I thought of as "batch files" on that now-ancient UNIX machine — but there's more truth in the subtitle, From Novice to Professional, than one might expect. He writes in an accessible, at times conversational, style and presents detailed technical information alongside a mixture of anecdotes and historical detail that does more than simply serve as a technical reference, it helps the reader understand a great deal about why things are the way they are. It was such an entertaining read that I frequently found myself skipping ahead, reading a section I knew was coming up, then resisting the urge to just keep going from that point. The first of these I encountered on page 18 in which he discusses the relative portability of printf in shell scripts. I knew what he knew, it's clearly non-portable and should be avoided, and thoroughly enjoyed the explanation of how he determined his (and by extension my) assumption was in error. Another on page 108 is the sort of good advice all UNIX users, not just those aiming to write good scripts, should take to heart. Many times, though, I've related precisely the same advice to colleagues to be met with confused stares, so it certainly bears repeating.
This book is a desktop reference in the truest sense of the term for me, it is an interesting, at times laugh-out-loud amusing, discussion of how to write shell scripts that will work on the widest possible range of Bourne-derived and POSIXly correct shells and why this is a desirable goal. In true UNIX tradition, the author doesn't provide simply a set of rules, but guidelines that will help you find your own way through the task of creating portable, maintainable shell scripts.
The real meat of the book begins in Chapter 3 (more on Chapter 2 in a moment) with a discussion of control structures and redirection, the latter being perhaps the defining characteristic of UNIX command line interfaces. I struggled somewhat with trying to decide if redirection would be better discussed after the material on how the shell parses tokens, presented in the first part of Chapter 4, but it does seem that the correct logical grouping is the one presented. It would be easy to get lost, for example, in the semantics of why the same streams of redirection tokens behave differently on different shells, but the key concept in the early chapters is that of many tools, each doing a specific task, working in concert. That objective is achieved quite effectively.
Chapters 5 and 6 go into detail (possibly too much for some, just right in my opinion) on how UNIX executes shells and how shells can spawn other shells, the costs and the benefits and the available alternatives for one to make an informed decision. Frequently there isn't one right answer whether some activity is better done in a script, in a shell function or in a subshell, but the material here will certainly aid in making those determinations. My personal bias being almost always toward writing a shell function — perhaps an indication I've had too much exposure to C programming, perhaps more due to a frugal upbringing and my own sense that spawning a whole new shell to do something is overkill — had me wishing for a larger section on the value of such constructs, but there should be enough there for me to win some converts to my cause.
By far the sections I learned the most from, however, would be Chapter 7: Shell Language Portability and Chapter 8: Utility Portability since I actively avoid exposure to other shells. I have my two preferred options and a third that I will use when presented with no alternative. While this does mean I know "my own" shells very well, it also means that I often bump into the furniture, so to speak, when I find myself using a new shell. These chapters haven't been immediately useful to me, but I know they're the ones that I'll be turning to in the future, I've needed something like them in the not-too-distant past, after all.
The final three chapters assemble the information presented in the earlier sections and suggest a sort of "best practices" approach to writing scripts. Concepts like "degrade gracefully" seem like pretty fundamental ideas when you hear them but I frequently find myself writing functions or scripts that don't do that at all when intended for a limited, usually singular, audience. It may seem like an okay idea when you're doing something for your own use, but when you write a complex function that works then discover a bug in it two or three years late and you have to return to fix it, it can be just as helpful for it to simply fail in an informative way as it would be to have detailed comments explaining the intent and the mechanics.
Truly, there's something here for everyone. In my office I'm considered something of an expert when it comes to complex regular expressions and the subtleties of using them in different editors and tools, but Chapter 2 and Appendix C both had enough new material in them that I found myself frequently making notes in the margins.
I have many, many books in my bookshelf in my office but nearly none on my desk. Beginning Portable Shell Scripting is going to be one of the very few that will be spending a great deal of time lying flat on my desk, in easy arm-reach.
You can purchase Beginning Portable Shell Scripting from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
more than this? (Score:3, Interesting)
Re: (Score:3, Insightful)
Yes. ... Well, come on, what do you expect me to say?
If you have a choice... (Score:5, Insightful)
Don't do it.
Shell scripts have horrible error handling, and quickly become a maintenance nightmare. These days, e.g. Python is installed everywhere you need to go.
Just do this:
def c(s): os.system(c)
and you have mostly covered the area where shell scripts excel. You can still write minimal "shell scripts" inside c().
Unluckily, you still *need* to grok shell scripting to some extent, or at least be able to read them. Just don't write them if you can help it.
Re:If you have a choice... (Score:5, Insightful)
I agree!
My personal limit is 10 lines for a shell script. If is longer than that I convert it to Python.
Python scripts have the advantage that they work on Windows too, and they have lots of os independent abstractions for file names, processes etc.
Why learn an arcane language like sh when you can learn a nice well structured language like Python and write better scripts?
A few years ago I would have used Perl rather than Python, but I'm converted now ;-)
Re: (Score:2, Insightful)
Re: (Score:2)
Oh yes?
How about counting each time a shell script has to spawn a separate process and python can do the job inside it?
And what when you stack together 5 such scripts?
Beyond maintainability, readability, sanity, one can also have some gains in speed and memory usage by using python;
Re: (Score:3, Insightful)
Why learn an arcane language like sh when you can learn a nice well structured language like Python and write better scripts?
Where I work pretty much everything has bash already (I install cygwin on all the Windows boxes. Of course, Python is usually there too :) ).
If you already have a bash script (or find one via the Google), changing it is usually simpler than porting it to Python.
If you work with people that already know bash scripting but don't know Python using the lowest common denominator can be easier than training.
There is less memory overhead for a simple shell script than there is for a simple Python script.
This
Re:If you have a choice... (Score:4, Interesting)
Re: (Score:2, Interesting)
One reason may be that Python's startup time is an order of magnitude slower that a shell's, and sometimes it does matter, even for scripts of much more than 10 lines.
Re: (Score:2)
sed -n -e '// {
s/.*\(.*\).*/\1/p
}' musicbrainz.txt
Of course, I think yours is skipping the first one; maybe it's an album title? I leave fixing that as an exercise to for the reader. :)
Re:If you have a choice... (Score:4, Informative)
sed -n -e '/<title>/ {
s/.*<title>\(.*\)<\/title>.*/\1/p
q
}' < musicbrainz.xml
Forgot that "plain old text" isn't. (To verify I had it right, of course, I pasted it into a file with the original and then ran
!sort | uniq)
Re:If you have a choice... (Score:4, Insightful)
That's exactly what the Perl people said years ago, and we all know how well that worked out (for low maintenance sysadmin-type tasks). I know the sinking feeling I get every time I find a crontab entry pointing to a Perl script.
Re: (Score:2)
That's exactly what the Perl people said years ago, and we all know how well that worked out (for low maintenance sysadmin-type tasks). I know the sinking feeling I get every time I find a crontab entry pointing to a Perl script.
There is a big difference between Perl and Python. While you can write readable Perl, or write horribly opaque Python, there's a reason that Perl has a reputation as a write-only language and Python has a reputation for being quite readable.
Then there's the guy I worked with years ago who wrote (writes, I'm sure) all of his scripts in elisp...
Re:If you have a choice... (Score:4, Insightful)
That's not what they said years ago. People arguing to use Perl instead of Bash did so because Perl was just far more functional. Perl and Bash both have pretty terrible maintainability, but Perl is a million times more functional.
Python and Ruby have the functionality of Perl without the maintenance issues inherent in a language which is really a hodge-podge of ancient unix idioms.
Re:If you have a choice... (Score:5, Insightful)
maintenance issues inherent in a language which is really a hodge-podge of ancient unix idioms.
What a ridiculous claim, there are no "maintenance issues" with ancient idioms... The very fact that those techniques are ancient shows how incredibly flexible and useful they are. I'd much rather use conventions which are widely accepted and in many cases are required by Posix/SUS/XPG4 than find myself having to hack up my stuff to accommodate broad and pervasive changes such those experienced when moving from python 2.x to python 3.x...
People who are constantly advocating against shell scripts tend to be those who see system administration as something it isn't; namely a low level development job. When in reality a sys admin uses shell scripts to glue together existing products of developers in order to manage administrative tasks. If I were an auto mechanic no one would propose that I learn to master a casting foundry and a milling machine in order to work on cars, those are clearly manufacturing/development tools AND certainly no good mechanic would suggest that using a wrench to fasten a nut to a bolt is "a hodge podge of ancient idioms" which should be replaced with whatever flavour of the week fastening system and power tool happens to be popular at the moment.
Sure there are some arcane aspects to shell scripting, but when I learned Unix in college they taught a thing called "the unix philosophy" which basically said that you should always use the smallest tools for the job, leverage the pipes/redirection, and build to a usable script which doesn't replicate existing functionality of ubiquitous tools. Seems like these days every python/perl wizard around fancies themselves an administrator and yet they waste a large portion of their time rewriting tried and true unixisms; sort, wc, cut, paste, tee, etc...
Also, get off my lawn!
Re: (Score:2)
I know the sinking feeling I get every time I find a crontab entry pointing to a Perl script.
For the younger folks out there, the feeling is similar too: /script/all_your_code_belong_to_us.exe
mono
Re: (Score:2, Funny)
I dont necesarily disagree with what you say here.
Ill just point out that your carpet is UGLY AS HELL.
And I Bash Script quite a lot (and even have fun at that).
Re: (Score:2, Interesting)
I can't agree more. I switched over to python for all non-trivial scripting a couple of years ago, and I find it much more pleasant. I even sometimes use iPython instead of bash when I know I'll need to do something complex interactively.
By the way - if you like using python to control systems, you might also enjoy the func project [fedorahosted.org].
Re: (Score:2)
To me it's mixed feelings there
Even though python is very easy, you can't beat the ease of use of find / sed / others
And in python you would need to go to popen/pclose usually, to get the output and return values
Re: (Score:2)
from commands import getoutput as cmd
foo = cmd("echo bar")
I often find myself writing small, special-purpose wrappers for some commands, like ps or find. I liked the MS PowerShell's concept of piping objects instead of text. Imagine... "ps | grep name=spam | kill". Yeah, sh sucks.
Re:If you have a choice... (Score:5, Informative)
Python is nice, but hardly installed everywhere. It's available on Linux certainly, but not always on AIX or Solaris. Yes, it is just an installation away, but many of the systems I maintain require change management procedures to even chmod a file.
Shell scripts do have decent error handling for what they need to do. With traps and proper usage of error codes, they are not much different from lower level languages.
I'd agree that I now *prefer* to write longer scripts in Python. However, few of the people I work with know Python, or even Perl. They can get around with korn and bourne as these are the default scripting languages on more traditional Unix systems.
Which comes down to the gist of the issue. Do you write code in a language you prefer or one that can be maintained by the admins? I'd argue that it doesn't matter what language you use. If you write poor code in shell you will likely write poor code in Python too.
Re: (Score:3, Interesting)
No it isn't. Not even remotely close.
And in most cases the response to a request to install an entire programming language would be flat rejection, turning to raucous laughter when they realise you only want it because you don't like any of the several scripting environments that are already available.
Re: (Score:2)
No it isn't. Not even remotely close.
And in most cases the response to a request to install an entire programming language would be flat rejection, turning to raucous laughter when they realise you only want it because you don't like any of the several scripting environments that are already available.
Sounds like a rather dysfunctional working environment.
I had something like this in mind when I said "everywhere you need to go" (as opposed to just saying "everywhere").
Re: (Score:2, Informative)
> These days, e.g. Python is installed everywhere you need to go.
Sorry, but no, it isn't.
Re:If you have a choice... (Score:4, Informative)
Shell scripts have horrible error handling, and quickly become a maintenance nightmare. These days, e.g. Python is installed everywhere you need to go.
Python doesn't help much over shell scripts without extra libraries, which may or may not be present on any given system.
Python has changed incompatibly several times already.
Python has a large startup overhead:
20 seconds: 1000x python -c 'print("test")'
2 seconds: 1000x sh -c 'echo test'
Python is clumsy to use for gluing several programs together.
Python is not the same syntax as the shell. If you don't learn the shell then your day-to-day command lines are gimped.
So Ruby or Python or anything else is better for writing actual programs that do anything complicated, but there are plenty of appropriate uses for shell scripting. Ruby is actually much better... since it has a sensible syntax you could make a rubysh that wouldn't suck.
Re: (Score:2)
Python doesn't help much over shell scripts without extra libraries, which may or may not be present on any given system.
What are those extra libraries without which you can't work? Python standard library is absolutely enough for shell script like tasks, both trivial and non-trivial.
Python has changed incompatibly several times already.
It's actually quite rare. Python 3.0 was created explicitly to allow breaking of the compatibility. If you write scripts that use the new stuff, of course they won't work on older scripts. But old scripts still work on new python 2.x interpreters.
Or do you have a concrete example in mind?
Python has a large startup overhead:
So a 0.02 second startup time is a problem for you? If you
Re: (Score:2)
.
The advantages are rapid prototyping, and many places to stop putting work into the project without losing functionality. If you have a really well defined large project, big design up front ma
Re: (Score:2)
c = os.system
Re: (Score:2)
Or better, just...
c = os.system
Ok, I cheated. I usually have it like this:
def c(s):
print ">",s
os.system(s)
Re: (Score:2)
"if [ $? != 0 ]; then" goes a long way and you get bonus points for wrapping it in a function.
What I like about the python way is that you don't have to write any of that - if something fails, the whole function fails (unless you specifically catch the exception). No need for explicit error handling to clutter your scripts.
Re: (Score:2)
Does python can really replace shell scripting ?
Yes, for everything apart from script you run with 'source' that manipulate the environment directly.
I'd really like to know. When I write scripts, I use intensively pipes and awk.
Would python allow me to do the same kind of things that awk do ?
Yes, you can call out for a shell by using os.system, os.popen, subprocess module.
I do think awk is pretty redundant though.
Python still can't replace quick scripting (Score:4, Insightful)
I find shell scripting have a nasty habit of not working quiet right when moved between Linux, the BSDs and Mac to be safe, and it's always a pain to write scripts that work correctly with spaces in file names.
Why isn't there (or is there?) a simple python cheat guide, or library, that do the same things as grep, awk, find, mv and xargs?
Re:Python still can't replace quick scripting (Score:4, Informative)
Why isn't there (or is there?) a simple python cheat guide, or library, that do the same things as grep, awk, find, mv and xargs?
re.findall, s.split(), os.walk, shutil.move,
" ".join
Re: (Score:2)
I've run into one big problem replacing find/xargs in Python: There's no good equivalent of the find '-print0' and xargs '-0' options.
This seems to work:
fs = os.popen('find -print0').read().split('\0')
Re: (Score:2)
Your idea works until you have a huge number of files (not uncommon for a backup script) and run out of memory on the .read() call. Not pretty. To avoid that, you have to use .readline() or the file iterator, both of which have the newline limitation I mentioned.
So, it's back to actual coding then :-). I don't have a oneliner for that, but shouldn't be too hard:
b = f.read(blocksize) ... do stuff with lines, and reloop
spl = b.split('\0')
Always add the last line from the split to the beginning of next block.
This way, you can make a memory-efficient generator that only allocates data when it needs it.
Re: (Score:2)
Why isn't there (or is there?) a simple python cheat guide, or library, that do the same things as grep, awk, find, mv and xargs?
Forgot to mention - no need to do it all in raw python. Just call out to shell when you feel like it, if you are sure it can stay unix-only. You can do wonders with some shell invocation + os.popen(...).read()
Shell scripts are a glue language (Score:5, Insightful)
One does not write a web server in Bash, one wraps a webserver in it, pipes its output to a log analyzer, restarts it automatically if it crashes, and so on.
The most important part of any UNIX-derived shell langauge is not its syntax or power but the fact it lets you construct large ad-hoc applications out of a toolbox of tens of thousands of pieces.
This is where all other operating systems (that I've ever used, and that's 30-40) have failed.
Any serious developer should know several glue languages, Unix shells being the most flexible and accessible.
Re:Shell scripts are a glue language (Score:5, Interesting)
I accept your challenge. :-D
But seriously, yeah, you're absolutely right. Ooh, but a basic web server written as a Bourne shell script called by inetd would be so freaking cool....
Oh, no. Somebody actually did that [debian-adm...ration.org].... Yikes! Now I'm scared.
Re: (Score:2)
"The most important part of any UNIX-derived shell langauge is not its syntax or power but the fact it lets you construct large ad-hoc applications out of a toolbox of tens of thousands of pieces."
What's the advantage supposed to be? Constructing a "large ad-hoc application" doesn't sound like a great idea to me.
Re:Shell scripts are a glue language (Score:5, Interesting)
One of the points might be that there's a fairly specialized task which takes a person six hours to do, but which is NEARLY all done automatically -- just a bit of hand twiddling.
Lemme give an example that isn't portable. One of the things I do fairly frequently is take about six large toolchains distributed to me as binaries and source tarballs, and turn them into patches against upstream versions, reorganize them, delete some unused files, create configuration files that refer to the binaries, generate md5sums, and so on.
This is a task which, if I sit down at 10AM and start typing, is usually done by about 4PM. Testing takes a bit longer, and usually uncovers SOME kind of typo or thing I forgot to do.
Enter the shell script.
I tell the script where the files are, and I walk away. An hour later I have the results. Testing is also automated (another script). But testing is also uneventful, because the script never forgets a step, makes a typo, or otherwise screws up.
By the second time I did this, the script had saved me time. By the third, it had saved me close to a full working day. By now it's closer to a week of my time that wasn't spent messing with this stuff.
Portability isn't entirely crucial here, you might think? Well, not ENTIRELY crucial, except that when they had me start doing this on a new box running a different variety of Linux, the total time I spent revising the script was 0 minutes.
Re: (Score:2)
I can see the value of shell scripts for small or possibly medium size ad-hoc applications but not for large ones.
Re:Shell scripts are a glue language (Score:5, Funny)
I think "One does not write a web server in Bash" is like "One does not simply walk into Mordor." You're practically daring short people with hairy feet to attempt it.
But your point is basically good. :)
Re: (Score:2)
Yeah but Linus is already working on the kernel...
Re: (Score:2)
I'm mentioning this just to point out that these distinctions of granularity are a bit artificial. Yes, glue languages give you composability at the coarse end of the scale. But they're often quite acceptable as programming languages as well.
Re: (Score:2)
Uh oh. Someone should have warned this guy [wikidot.com].
Re: (Score:2)
The most important part of any UNIX-derived shell langauge is not its syntax or power but the fact it lets you construct large ad-hoc applications out of a toolbox of tens of thousands of pieces.
Like constructing a Windows app out of ActiveX components?
No, I'm not saying that ActiveX is as good a glue as the Unix equivalents, or that Windows is as good as Unix. I'm just saying that there's more to Unix than its ability to glue stuff together.
This is where all other operating systems (that I've ever used, and that's 30-40) have failed.
I guess your definition of "failure" is "Pieter hates using it". By any other measure, there are are fair number of non-failed OSs that aren't Unix or Unix-like.
Re:Shell scripts are a glue language (Score:5, Funny)
Wow. That's a really brilliant question.
Wouldn't it have been cool if someone had written a book on the common ground of the major shells covering how to use them as a single highly-accessible and universally-available language? :)
PSS can be a recurring problem (Score:3, Insightful)
I'm sure all of us at one time or another have had a shell script we've relied on for years fail miserably when bringing it to a new environment. The sad fact is, shell scripts were never meant to be programming languages in and of themselves, and I wonder if, knowing what we know now, it isn't overly ego-driven and masochistic to try to take this feature -- tied to a shell which is tied to an operating system -- and promote it beyond its competency.
So, let's say we take the PSS principles seriously, and abstract away any non-platform-agnostic features you can think of. A few years down the road, you've got PSS all over the shop and you want to upgrade to a different platform nominally supporting your shell of choice. Even if you shake off PSS features you thought could create incompatibilities, you discover the new system buffers differently. Or added a parameter somewhere. Point is, if you went with something like Perl which is designed for cross-compatibility you would have been fine, but now you're all wet.
Shell scripting is good for what it's meant for, but at the risk of oversimplifying with a tool analogy, I'm concerned that this falls into the trap of "If all you have is a hammer, all your problems look like nails".
Re: (Score:2)
You know, that's exactly why a book on it is useful -- because it turns out that you can, in fact, write scripts which port quite nicely.
Seriously, "buffers differently"? What's THAT supposed to mean?
That said, I won't deny for a moment that there are things shell isn't a good choice for. In fact, one of the first sections in the book is under the heading "What Shell Scripting Isn't". Because sometimes the best way to write a good program is to pick the right language to begin with. Often, even.
But some
Re: (Score:2)
Seriously, "buffers differently"? What's THAT supposed to mean?
One example is how and when data gets written to disk in the absence of a flush().
Re: (Score:2)
And what "flush()" do you expect to see in a shell script?
You're thinking at a level that generally doesn't apply well to shell scripts. Scripts rarely deal with questions such as "has this been actually written to disk" -- because that's at the wrong level. If you need that information, you shouldn't be writing in shell anyway.
But normally you don't...
Re: (Score:2)
I agree with your overall message, but...
Just some general comments... (Score:5, Informative)
First off, in the interests of full disclosure, Joe MacDonald is one of my coworkers.
Anyway... The big surprise to me was the word "Beginning", which somehow showed up in the publisher's cover pages, but which I didn't know about during the writing process. My tech reviewer was Gary V. Vaughan (yes, the autoconf/libtool guy). I bounced material off a number of seasoned expert scripters during the process. Basically, my goal was to write a book that I could use as a reference, and which would teach me something.
I succeeded beyond my wildest dreams. The discovery that printf(1) is essentially universal these days was a complete shock to me; I had no idea it was portable. During my first pass on the regular expressions section, I started by writing down what I believed I knew about sed, awk, etcetera. Then I tested it... and had to revise most of it. A number of things I was used to were GNU or BSD extensions. When Gary sent the chapter back for tech review, he'd flagged most of these things, because he "knew" the same things I did.
So everything there should be pretty thoroughly checked out now -- I realized very early on that this field was full of things "everyone knows". Many of them wrong. We tested things on a field of around 30 different versions of Unix and Linux. We tested them on unusual installs, we tested them on special cases.
Why?
Because portable shell is an incredibly portable language, and sometimes that matters. Because shell is a very powerful language, too. Because sometimes shell is all you have -- and because sometimes shell is more expressive for a task than your other choices. I love me some C, I program in C by preference much of the time -- but there are a lot of tasks I'll do in shell rather than in C. There are similarly many tasks I'd rather write in shell than in perl. Shell is what make uses to run commands, and sometimes you need to write something clever in shell because make doesn't have quite the right feature.
In short, it's something I have found consistently useful, day in and day out, for about twenty years now. I just wish I'd realized how much more there was to learn years ago, I coulda saved a lot of time... :)
And, to answer a question hinted at earlier: Yes, now that this book exists, I keep a copy on my desk. I look stuff up in it about once a week.
Nobody's seriously comparing C and shell (Score:2)
Saying "I love C but there are things that are better in shell" is completely anachronistic.
Seriously. The question's been settled for over 20 years.
And there are other languages, you know. The question is more whether to use Python (for example) instead of shell in some cases, and when.
Re: (Score:3, Interesting)
I compare C and shell all the time. Sometimes the answer surprises me. e.g., until I knew about printf(1), I sometimes went to C if I needed to pretty-print output. Now sometimes I don't.
I will happily mix and match multiple languages; one of my first shipping products was written in shell, perl, and C. Each did some things well that the others didn't...
I tend not to use Ruby for things that I want to be portable, because not everything has Ruby around yet. I tend to avoid Python because it just never
Re: (Score:3)
Care to elaborate?
I have seen some rather complex scripts that are portable that do some useful things.
Re: (Score:3, Interesting)
As usual, it comes down to use cases. Describe the useful things that are done. One might choose to write a shell script to perform some pure mathematical utility function, but this certainly isn't the usual role of such scripts. Rather, one uses a shell script when accessing files (logs, etc.) on the disk, or when opening sockets, or when spawning host level commands. Other device level access is often required, for instance, a local or UTC clock might be consulted, requiring knowledge of timezones. A
Re:portable shell scripting is an oxymoron (Score:4, Informative)
As usual, it comes down to use cases. Describe the useful things that are done.
Take a naked box and boot it (/etc/init.d/*) ?
I know it's a bit trivial but it still qualifies as useful in my book.
Actually /etc is just pretty much a collection a collection of fairly useful shell scripts. I've always found it interesting that Unix was mostly held together by /bin/sh (aka /bin/bash on a lot of systems nowadays) and spit. And that it worked.
To take one of the posts above where the poster had been exposed to DOS. The DOS system (although it wasn't really a system, merely a program loader) was configured by the autoexec script. All the Unix do the same with a number of chained scripts (and their order can even dynamically change nowadays) all running sh (or an extended version of it).
I still wonder at it sometimes. It's simple and accessible on one side. And it can degenerate into an awful mess on the other :) (less so nowadays thankfully)
More ?
Anyway, wanted something useful the shell could do ? How about run the whole operating system (find a service that isn't actually handled by a !#/bin/sh script...).
Re:portable shell scripting is an oxymoron (Score:4, Informative)
Portability isn't boolean.
I wrote a wrapper around cdrecord to clean up the UI, automatically handle things like creating an isofs from directories, and so on.
It's not 100% portable; every new system, I change the path to cdrecord, the device spec for the CD drive, and the command used to eject a CD.
Everything ELSE stays the same, and I don't need to remember how to use mkisofs, or anything like it. Directories, bzipped images, whatever; it gets burned correctly. I win.
If the script were not written in otherwise-portable shell, it might not work on the broad variety of boxes I've wanted to use it on.
I've done scripts to handle tasks like "open this file" (not as flexible or smart as the OS X one, but quite good about various compressed tarballs and archives). Surprisingly portable.
I have a script for the idiom of "for every file named or provided as standard input, run it through this filter in place". Repeating commands at intervals, for a given number of times, until they fail... Tons of little utilities like this that save me time.
If you want complete applications with no dependencies, that's harder to find. That said... Have you ever used autoconf to configure something? That's a fair bit of portable shell right there...
Re: (Score:3, Insightful)
Yes, you're quite right that this says a lot about poor user interface choices in some utilities.
Thinking about it more: One of the key applications of portability involves scripts which are never ported.
It's that I don't use only one machine. I use a Mac desktop, a BSD server, and some Linux servers. Even if I'm just typing "for i in..." on the command line, I don't want to try to remember three or four different sets of commands to work across these environments. I want to write things that work the s
Re:portable shell scripting is an oxymoron (Score:5, Interesting)
Well, there is some truth to the GPP's comment. Linux and Mac OS X don't even agree on how to tell echo not to print a newline or how to enable extended regular expression mode in sed. May heaven help you if you want to do something as esoteric as creating or mounting a filesystem, creating or mounting a disk image/ramdisk, talk to a USB device in any way, get a list of processes in any useful way, etc. There's a very big lack of standardization in a lot of things you might like to do with scripts, in other words. The Single UNIX Spec and POSIX are not quite sufficient, but more annoyingly, most OSes (Linux, *BSD) out there don't even come close to conforming to it, so you end up with this dichotomy between BSD behavior and AT&T behavior.
That said, a lot of things are standardized, and many others can be worked around with clever use of variables (or possibly eval in a few extreme cases). I've written chapters on the subject myself. The big things you need to remember are that $(( $FOO + 3 )) is not portable, nor for ((...)), nor >&, nor anything involving extended regexp except using Perl, that even "the one true awk" is not quite SUS-compliant, GNU awk doubly so, bash triply so, that you should use printf instead of echo for output if you don't want newlines, that signal numbers are not portable (for trap), that proper quoting of arguments is crucial, and that you need to work with the bare minimum base behavior of utilities (using few or no flags) if you expect any hope of portability without needing to make platform-specific changes.
For some quick examples of some interesting portability issues, read some of my comments in the games at shellscriptgames.com or search for the word "compatibility" in Apple's "Shell Scripting Primer". It's a real eye opener to see how many portability problems exist even for fairly simple shell scripts.
Re: (Score:2)
'sed', 'perl', 'awk' and 'mount' are not part of the shell.
for and trap are part of a posix standard shell.
It's subtle, but the differences are easy to understand. The best example I can think of is most linux systems I've used use bash, but Ubuntu make /bin/sh a symbolic link to /bin/dash because it's faster. Many scripts are broken because they expect /bin/sh to be a symbolic link to /bin/bash. This is not an assumption you can make.
This causes a problem for example when you use bash functions like popd w
Re: (Score:2)
Fair enough about those things being extensions, though many are supported so broadly (the $(()) math syntax, for example) that it surprises people when they don't work. Unless you code on the Almquist or Debian Almquist shell regularly, chances are very good that you'll be in for a rude awakening if you ever switch to it. Things that nearly every /bin/sh implementation on the planet has provided for a decade or more (not just bash) simply don't work in ash or dash because they depend on behavior that isn
Re: (Score:2)
Good ones, but you forgot the biggie that I see people get tripped-up on the most, test.
Sometimes test -a mean test -e and sometimes it is meant to be used [ exp1 -a exp2 ]. Most often these days the confusion is on solaris using ucb.
So the moral is, never use -a, -o, or -e in tests.
(Yes there were ancient versions of test that did not have AND and OR, so you use && and || from the shell instead.)
Re: (Score:2)
From my experience, shell scripts that use a significant amount of non-shell builtin commands are not portable. The typical shell script is highly dependent on the awk, grep, sed, etc. version on the system. And that varies not only between platforms, but between OS versions.
Simple things, like the improved mv that's floating around, tend to be easier to port. But the chance of a successful port is inversely proportional to the complexity of the script. As well, usefulness, while a loaded term, tends to be
Re: (Score:2)
Care to elaborate?
I have seen some rather complex scripts that are portable that do some useful things.
Here's one: most uses of "find" together with "xargs"...
By default, "find" prints out newline-delimited filenames, while "xargs" consumes whitespace delimited values... So if you've got filenames with spaces in them, you're in trouble.
The GNU solution is the "--print0" argument to "find", and the "-0" argument to "xargs" - which uses the zero-byte as a delimiter for both commands... The problem is, this isn't supported in other implementations of "find".
This is just one example of a command where the comm
Re:portable shell scripting is an oxymoron (Score:5, Funny)
You're in luck! I have recently heard of a book that could help with your scripting...
Re: (Score:3, Funny)
That link should really be
http://thedailywtf.com/Articles/Stupid-Coding-Tricks-A-Batch-of-Pi.aspx [thedailywtf.com]
(I should write a shell script to correct my comments.)
Bash script, ASPX page, culture clash.
Re: (Score:3, Informative)
Check out GNU autoconf. That's a good example of how a script works on *nix box. And, yes, they are useful!
Autoconf is also a horrible peace of crap. One of the better reason to hate the concept of shell scripts, actually.
Check out SCons for comparison:
Program('hello.c')
or
SharedLibrary('foo', ['f1.c', 'f2.c', 'f3.c'])
And that's pretty much it. I'm not sure all the horrors required by autoconf would fit into a slashdot posting.
Re: (Score:2)
I think it could fill an entire book.
Re: (Score:2)
Most of those issues are primarily issues with third party autoconf m4 modules, or with people who don't know what they are doing adding functionality to configure.ac. That is not really autoconf's fault. (The fact that it is easy to mess things up if you don't know what you are doing might be autoconf's fault). And proper use of autoconf can and does make portability easier, although I will admit little outside the GNU project uses autoconf properly, and there are even exceptions within the GNU project.
The
Re: (Score:2)
Sorry, but no, that is nowhere near 'pretty much it' - as soon as you want to link in with libraries in a cross platform manner, generate platform specific release mechanisms (thing mac framework and app bundles), or do anything which doesn't fall into your example there, you're back in wild and woolly territory... and yes, those things are beyond autotools, but they're also beyond the core scons too.
Re: (Score:2)
You miss the point.
Autoconf is godawful crap, no doubt about it. But people still use it. You know why?
*IT WORKS*. It works on Linux. It works on Solaris. It works on BSD. It works on Tru64. It works on Cygwin. It works all over the place. A decently written autoconf script works for cross-compiling. It doesn't require a particular language to be installed.
I've used SCons-based projects. It was a nuisance to get it set up and working. Sure, it does some stuff nicely -- but I couldn't just grab i
Re: (Score:2)
It's obvious that you can buy simplicity by special casing. What's hard is keeping a bit of it when you generalize. I had the same exp
Re: (Score:2)
Hmm... did someone re-invent imake?
No, SCons does not generate makefiles.
SCons here was just an example to illustrate how much Automake sucks. Obviously we have other systems like CMake as well.
Re:Hardly Need a Whole Book (Score:5, Informative)
Oracle, too... (Score:2)
All the DBA's I have worked with (as well as C programmers worth their salt) have tended to use ksh by default.
I think Oracle's documentation always uses korn, and maybe I have just worked with a bunch of old IBM'ers..
Re: (Score:3, Insightful)
Sounds great.
Now, off the top of your head, what happens to variables set in the last component of a pipeline in ksh? Do you know whether it's the same on systems where "/bin/ksh" is actually pdksh? ... Oh, and just for reference, about half the Linux systems our IT department installs don't have ksh. No, I don't know why. (I only know because I can't log into them because my default login shell is /bin/ksh...)
Re: (Score:2)
ksh isn't part of Linux normally. Sometime's it's pdksh and sometimes it's genuine AT&T ksh. Same with cygwin. You have to explicitly add it too.
I had to port some shell scripts from Ultrix to SunOS, Solaris, HP-UX, Irix and OSF/1. ksh wasn't part of SunOS. However, we had bought a license for ksh and put it in /bin/ksh everywhere.
I needed functions and Ultrix /bin/sh didn't have them. IIRC /bin/sh5 (?) did.
Anyways, ksh had to be licensed back then. This was Linux 1.09 era. pdksh wasn't even clos
Re: (Score:2)
Exactly -- you can't just "write for ksh", and even if you do, it's not universal.
You can do pretty well writing for a common subset of ksh88/pdksh, but I'd rather do the extra few minutes' work and write for plain old POSIX shell by default.
Re: (Score:2)
Want your shell to be portable? Write it in Korn Shell. You will find this shell on 15 year old *NIX boxes and the script will still work with bash on Linux.
Except when it doesn't.
There are differences between bash and ksh, and between different versions of ksh. In some cases the difference means syntax errors, but sometimes it can be more subtle and a script written for ksh runs just fine but doesn't do in bash what it did in ksh, or doesn't do in pdksh what it does in ksh88.
It is also not the case that ksh exists everywhere. It wasn't on MacOS until 10.4. It isn't on FreeBSD by default. It isn't on many Linux boxes, particularly the small network devices
Re: (Score:2)
But you gotta love that mic stand [hrgiger.com] H.R. Giger created for Jonathan Davis.
Re: (Score:2)
Not so much a guide, but "man bash"* or "info coreutils" are incredible resources for those common bits you routinely forget. If you stick to what's in coreutils, I think you've got a pretty good chance of portability.
*for systems that use bash as the shell, of course. Frankly, I think there might be too much in there, though and some of the builtins should have their own man pages.
Re: (Score:2)
Actually, that's useful in bash too sometimes. :)
Anyway, the reason I bother is this:
I wrote a bunch of scripts in 1992 or so. I'm still using them. I haven't touched any of them in years, except for one update to deal with Linux differing from BSD in where the actual errno definitions are.
I don't have to worry about what shells are installed, I don't have to guess whether bash is "/bin/bash" or "/usr/pkg/bin/bash", I don't have to wonder whether the sysadmin bothered to install the "GNU utilities". I ju
Re: (Score:2, Insightful)
So WHO does not? (Score:2)
I've heard that said for 15 years, that might have been true 15 years ago, but now ... ?
News flash (Score:2)
Linux is obliterating commercial Unices.
It has in part something to do with the backspace key working out of the box without typing "stty erase ^H" every time. (Have they fixed that on Solaris, yet?)
Re:Oh I hate those [ "X$var" == "X" ] (Score:5, Informative)
Why bother with portable shell scripts, seriously? Everybody has bash installed, and/or zsh that is mostly compatible, and even then you have bash anyway. I understand retro-nostalgia and all that, but necrophilia is overrated
False.
The majority of systems I work on these days and the majority of systems I have worked on since the mid 90's have not had bash installed. That includes systems running FreeBSD, NetBSD, OpenBSD, AIX, Tru64, Solaris, MacOS, and even Linux. Current versions of some of those will usually have bash in a default installation, but some still do not. Companies running stable systems as important parts of their business do not generally upgrade their OS's just for the sake of novelty. Running older systems isn't usually about nostalgia or necrophilia, it's more often about not having any compelling reason to upgrade. There is also a system hygiene practice common on the BSD's of keeping the base system minimal and only adding on what is needed, a practice that helps in keeping systems secure and stable because they are easier to fully understand. This is also common in many virtualization environments, where a running OS instance is likely to exist for a very narrow purpose and intentionally have a stripped-down set of utilities fit to that narrow purpose.
Re: (Score:2)
Recent versions of MacOSX have bash by default. By recent I mean 10.4 had bash, and probably 10.3 but I'm not sure.
All Linux distribs have had bash installed by default for ever. And by all I mean 99.999% of the installed base, I'm sure you can find a silly exception.
Recent versions of Tru64 ... do not exist.
As for the BSDs, Netcraft confirms it, .. err. I don't know, what's their default shell?
And as for Solaris, its default shell -- a Soviet-era knock off of the original Unics v1.0 -- is so fucktarded tha
Re: (Score:2)
Recent versions of MacOSX have bash by default. By recent I mean 10.4 had bash, and probably 10.3 but I'm not sure.
10.3 had it. Prior versions did not.
All Linux distribs have had bash installed by default for ever. And by all I mean 99.999% of the installed base, I'm sure you can find a silly exception.
A large fraction of the Linux systems I work with are embedded versions which use things which seem to be descended from the BSD (Almquist) sh. Talking about installed base numbers is silly, because it is probably also true that 90%+ of those systems have never been seen by a competent sysadmin who has any intention of ever using anything Unixy that isn't a major Linux distro. They might as well be Windows for all of the relevanc
Sry, I don't do archaelogoy (Score:2)
I know paleounices suck. Does it matter to 99% of shell writers, is the question.
Re: (Score:3, Insightful)
Well, I suppose because the contents are different. We're answering different questions.
I would never dream of discouraging people from looking at and using the various free guides out there. The autoconf manual is full of useful information about portability, too.
There's more than one article or book because there's more than one topic. "Shell Scripting" is not a single topic; "portable shell" is very different from "advanced bash". It solves different problems, and is useful for different circumstance
Re: (Score:2)
Re: (Score:2)
Honestly, I just never got into python. I like perl (and hate it), and I like Ruby (and love it), but Python never "clicked" for me. Python and Tcl are the "scripting languages I just don't enjoy working in".
But... I also don't have it on everything I use. And I could get it, but that's another distraction from Getting The Job Done. So I do a ton of work in shell. Especially work that's entirely built around running other commands.
Re: (Score:2)
Honestly, I just never got into python. I like perl (and hate it), and I like Ruby (and love it), but Python never "clicked" for me. Python and Tcl are the "scripting languages I just don't enjoy working in".
I believe this is a common mindset issue. Ruby borrowed lots of stuff from perl (which they thought was good, or appealing to convert - I'm not sure), including TIMTOWTDI, whereas Python community thinks of Perl as a warning example more than anything else. My theory is that the different languages balance between simplicity and 'regularity' (python) vs. 'playfullness' and 'interesting' solutions (ruby, perl).
Re: (Score:2)
That may be. I find Ruby much better than perl; perl always struck me as inherently ugly. Perl tolerates careful programming and writing for clarity; Ruby encourages it.
Python felt a bit too constrictive to me, if you'll pardon the pun. And I didn't like the indentation (although I grant the underlying intent, I just don't think it pays off enough).
Re: (Score:2)
Wherever you want to do an awful lot of things with the input and output of) system utilities and./or bash builtins.
Whenever you try to do something nontrivial in between the system utilities, you start losing the benefit of shell scripts.
Re: (Score:2)
Wherever you want to do an awful lot of things with the input and output of) system utilities and./or bash builtins. Look at the gargantuan effort that is the Knoppix boot scripts - I seriously doubt it would make sense to rewrite those in Python or Perl, since nearly every line is a pipe between utilities or redirection. And it works well.
I'm not familiar with Knoppix, but that is true for all boot script systems, and beyond being hard to do in something else, there are cases where it may be impossible. There are still systems out there that boot on very small root filesystems that do not have any shared libraries available until the rest of their storage is mounted, so you've got to have something small and statically linked to run the scripts that get your whole software edifice in place. You can have similar problems in some failure stat
Going the Other Direction (Score:2)
My first Unix shell was the Mashey shell that preceded Bourne shell, on a PDP-11 running v6. :-) Before that I'd used RSTS-11, HPUX Basic, the IBM System 34 shell (OCL was fairly powerful, though less than sh), CMS, Plato, and various things with punch cards. Unix shell seemed powerful, flexible, and really really appropriate. I later used other shells like TSO and occasionally CP/M; I forget if I used VMS and ddt before DOS or only after.
Eventually MS-DOS came around, and it was painfully clumsy. It wa
Re: (Score:2)
VMS looked far more like RSTS-11 or RSX-11 than like Unix. I don't know if CP/M was inspired by VMS or by its predecessors. DOS certainly wasn't inspired by Unix, at least in the original versions; they clearly didn't have any of the concepts, even though they had a machine that was almost as powerful as a small PDP-11 and more powerful than an IBM System/34 (which had a quite nice shell), and later versions of DOS (around 3ish?) started emulating some Unix shell syntax while still not having the underpin