Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
Data Storage Books Media Programming Java Book Reviews IT Technology

Data Crunching 94

Posted by timothy from the constructive-critique dept.
Vern Ceder writes "I really expected to love Data Crunching. The Pragmatic Bookshelf has come up with some very good and, well, "pragmatic" texts in the past so I was looking for more of the same. Even better, the subject of the book was the routine data extraction, massaging and formatting that I (and a lot of other coders) spend so much time on. I was really looking forward to adding a couple more pragmatic tools to my coding toolbox. Unfortunately (as you may have guessed), I really can't say I love Data Crunching. It's a good book, but there are several minor points that keep if from being a truly great book." Read on for the rest of Ceder's review.
Data Crunching: Solve Everyday Problems Using Java, Python, and more.
author Greg Wilson
pages 176
publisher Pragmatic Bookshelf
rating 7
reviewer Vern Ceder
ISBN 0974514071
summary A good introduction to data crunching, but watch the examples.

On the positive side, there is a lot of good stuff in this book. I would even go so far as to recommend it to everyone who writes code to extract or manipulate data, particularly those less experienced. Greg Wilson should be praised for taking the idea of data crunching seriously and for systematically dealing with its patterns and pitfalls. A lot of important work gets done every day with one-off programs and behind the scenes scripts and Wilson is right that the techniques that go into this sort of coding are different, but just as important, as those that go into full-blown application development.

The strength of this book is that it offers useful approaches and patterns for dealing with a variety of common programming situations and types of data, while also pointing out their common traps and pitfalls. Wilson starts with techniques for crunching text data, moves on to the use of regular expressions, XML, binary data, and SQL databases before concluding with a special section on "horseshoe nails," various little techniques which just might save help save the day. Quite often he uses examples in both Python, which he calls an "agile" language and Java, a "sturdy" language. The basic advice offered is sound, if not shocking -- keep things simple, test as you develop, don't duplicate code, use existing scripts and utilities when possible, and so on. The combination of such sound advice with a wealth of practical examples is makes for a very effective handbook, particularly for someone new to data crunching.

So is Data Crunching a good book? Definitely. Should you read it if you regularly do routine data manipulation and extraction? Absolutely. And yet...

And yet there are number of things that just aren't quite right. The text and binary sections are the best, while I would say that the XML and SQL sections are the weakest, partly because those topics are too broad to cover in a single slim chapter. If you already have an idea of how you might want to use XML or how to extract data from a SQL database, you're likely find something handy in those chapters. On the other hand, if you're unfamiliar with them, this book probably doesn't have enough detail to get you writing useful code. I should say it doesn't have enough detail to get you writing useful code knowing what you're doing. And data crunching without knowing what you're doing is a bad idea. Trust me on that one.

I have another problem with the section on SQL. Several of the slicker SQL recipes rely on nested queries (page 147-151). MySQL, clearly a very popular SQL database, has nested queries only in its latest versions, so many, if not the majority, of MySQL installations do not yet have that capability. Yet the text carries on as if nested queries were universal, without so much as parenthetical mention that some things might not work on all SQL implementations. It seems to me that this is exactly the sort of pitfall a book like this should inform the reader of.

There are also several coding examples that bother me. Since I tend to both learn and teach by paying close attention to examples, I get uncomfortable with examples that seem to suggest something other than what they should.

For instance, the very first pieces of sample code (pages 9-10) in the text chapter are Python and Java programs to reverse the order of lines in a text file. I don't have a problem with the exercise itself, I've often assigned it to beginning programmers. However, this book is about quick and reliable solutions to common data handling problems, not leading people through basic programming exercises. Ironically, the very same chapter discusses the advantages of using the Unix command-line and its wealth of little tools. So wouldn't it be reasonable to expect at least a brief note or example showing that the REALLY easy way to solve the problem is with a single line: $ tac filename > filename2? Yet tac is not even in the list of "Useful Commands" on page 24. If reversing lines is just a programming example, it shouldn't be the lead example in a book like this, and if it is important, then you should mention that the problem has already been solved.

In the same vein, Wilson spends a fair amount of time in the text chapter illustrating code to parse command-line parameters, before admitting that libraries for the task abound in most languages. Granted, being able to snag a parameter or two off of the command-line without using a library can sometimes be handy; but implementing a more involved command-line parser is a problem that has already been abundantly solved.

Similarly, one of the examples in the chapter on regular expressions uses a regular expression to check to see if a string contains a valid IP address (pages 65-66). After showing how to use a regular expression to scan a dotted quad of digits, the text then admits that using a regular expression alone would lead to too much complexity, since it's hard to use a regular expression to check to see if a 1 to 3 digit number is less than 255 (or 127, which is what he uses in his code). So the example on page 66 ends up compiling and matching a regular expression like this:

pat = re.compile("(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\ .(\\d{1,3})")
. . .
m = pat.match(text)
for g in m.groups():
. . .
when a Python coder would more naturally just use:

quads = text.split('.')
for number in quads:

Sure, it's a good example of how to extract matched items, but the implication is that using a regular expression is the best way to extract extract numbers separated by dots, when in fact the Python has a simpler, easier and more reliable way to deal with it. Again a quick mention of the "easy" way to solve the problem would have been appropriate.

These kinds of issues are what keeps Data Crunching from being a great book. In spite of them, it is still a very good and useful book and Mark Wilson has done a good job with a topic all too often ignored. The general idea is great, and the principles, problems and solutions are well-explained and relevant. If data crunching is something you do, I would certainly recommend that you read this book, but with a somewhat critical eye.

You can purchase Data Crunching: Solve Everyday Problems Using Java, Python, and more. from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
This discussion has been archived. No new comments can be posted.

Data Crunching More

Data Crunching

Comments Filter:

  • Reviewer catches himself. (Score:5, Insightful)

    by juuri ( 7678 ) on Monday June 20, 2005 @03:21PM (#12866360) Homepage
    Don't berate the author for his examples using nested SQL when a paragraph later you call him out for not using "tac" because you assumed it is universal.

    Like nested queries, tac, isn't standard across all unix platforms.

    • Re:Reviewer catches himself. (Score:4, Interesting)

      by FriedTurkey ( 761642 ) * on Monday June 20, 2005 @03:38PM (#12866532)
      Isn't nested SQL part of the ANSI standard? MySQL is a great database for certain purposes but every other modern database has nested SQL. I don't think an author should not use a technique native to most databases because one database's older versions don't have it.

  • quads = text.split('.') (Score:5, Insightful)

    by Evro ( 18923 ) * <evandhoffman@ g m a i l.com> on Monday June 20, 2005 @03:25PM (#12866398) Homepage Journal
    quads = text.split('.')
    This assumes valid data and not something mangled like "1.2.3" or "U.S.A.". Using the numeric regex match that the book's author suggested would be more reliable in matching IP addresses only.

    • Re: quads = text.split('.') (Score:5, Informative)

      by abigor ( 540274 ) on Monday June 20, 2005 @03:37PM (#12866513)
      quads = text.split('.')
      if len(quads) != 4:
      raise NotAnIPAddress
      for member in quads:
      try:
      quad = int(member)
      if quad < 0 or quad > 255:
      raise NotValidQuad
      except:
      raise NotValidQuad
      .
      .
      .
      etc.

      • Re: quads = text.split('.') (Score:2, Informative)

        by Anonymous Coward
        Ummm... is receiving a number less than 0 or greater than 255 an exception? No, it's abnormal input sure, but that is a nasty and poor use of exceptions.

        You get an F on programming style :(

        • Re: quads = text.split('.') (Score:4, Informative)

          by abigor ( 540274 ) on Monday June 20, 2005 @04:00PM (#12866728)
          Jesus, it's just a demo to show that calling split isn't particularly unsafe. How you handle the errors is up to you. Consider the raise statements to be pseudocode.

          Ah, but your last line explains everything: you teach programming. You don't do it for a living. Makes sense now.

        • THe whole point of python is to raise and catch exeptions instead of fucking about trying to make it all nice. So the parseing program might be called by

          ip = getuserinput()
          try: DoShitFromGrandparent()
          except NotAnIPAddress:
          print "Not an IP address, dumbass"
          except NotValidQuad:
          blah blah etc.
        • Actually, that would depend on where this code lives... if it's in the user interface, sure, using an exception is probably not the right way to do it, since you know right there how to handle it. But what if it's deep in the bowels of a library? A library should validate that its callers are following the contract, but has no way of knowing how to handle the error when the value is out of range, so it should fail early and throw an exception so the higher layers can do something about it.

          Besides, as anoth
        • Ummm... is receiving a number less than 0 or greater than 255 an exception?

          In this case, it probably is.

          No, it's abnormal input sure,

          abnormal input is an exceptional condition by definition. Normal input is expected.

          but that is a nasty and poor use of exceptions.

          No it isn't.

          You get an F on programming style

          Your teaching credential needs revoking. As anybody worth their salt as a programmer would know that whether or not to handle something as an exception depends on the severity of the prob
      • Well, sure, that'll work, but that's not what the reviewer included as an alternate example. The \d{1,3} syntax would do a lot of sanity checking right off the bat. But if there's one thing I learned from Perl, it's TMTOWTDI. If you're guaranteed that the data being passed to you is valid and clean, using a simple split on the '.' character would suffice. I usually prefer to err on the side of "never trust the data," especially when designing modular stuff, as you never really know who's going to be pas
      • "\b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5 ]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0 -9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[ 0-9][0-9]?)\b"

        This will match a valid IP address.

        --Pete
        • Maybe he's trying to be helpful... but as a perl programmer, I say, mod parent funny!!!

          My philosophy (yes, everybody seems to have one these days) is this:

          1. Define the rules for valid data.
          2. Classify types of invalid data.
          3. Break the rules down into a series of discrete steps.
          4. Write the validation code, using the simplest semantics possible.
          5. If data validation fails, try to match the problem to one of the invalid data classifications and throw an exception.

          Yes, it sounds complex, but for the type
      • or using Perl:

        if((@quads)=m:(\d+)\.(\d+)\.(\d+)\.(\d+):)
        {
        # do stuff
        }
        else
        {
        warn ( "bad address" ) ;
        }

        QED

  • nested queries are a problem? (Score:5, Insightful)

    by stoolpigeon ( 454276 ) * <bittercode@gmail> on Monday June 20, 2005 @03:26PM (#12866401) Homepage Journal
    If a book uses nested queries and some rdbms doesn't -- the problem lies with the rdbms. I've never used mysql and I've avoided the flames about it not being a real database.... but come on. That is weak.
    • Granted the ANSI SQL standard isn't followed as closely as perhaps other standards are, but if Nested Queries are in the standard, then I would have to say the RDBMS is at fault and not the book.
    • I may be wrong, but I believe that an RDBMS must support nested subqueries to be conformant to the ANSI SQL92 Entry-Level specification (maybe even SQL89?).

      Not to fan the flames of another advocacy flamewar, but if MySQL hasn't caught up to a 13-year-old standard yet, it shouldn't be treated as a fully-functional SQL RDBMS.

      If you're running MySQL you should be aware of its limitations yourself; it's not the book's job to bring them to your attention for you.

    • MySQL and data crunching (Score:3, Insightful)

      by angio ( 33504 )
      MySQL's lack of support for some of the ANSI SQL features is annoying. But, that said, I do a lot of data crunching on a terabyte or so of Internet measurement data, and MySQL remains my database of choice. In a data-mining application like mine, I need speed and a compact on-disk representation of the data and the indices before anything. Our inserts are batched a couple of times a day; having them fast is important, but having them run concurrently with queries isn't. I don't need transactions, I can

  • I really expected to love Data Crunching. (Score:4, Funny)

    by dfn5 ( 524972 ) on Monday June 20, 2005 @03:27PM (#12866418) Journal
    I also expected to love getting my teeth pulled. Trust me. It wasn't that great.

  • Matching a dotted quad (Score:1, Informative)

    by Anonymous Coward
    Shouldn't be too hard if we can use ereg() or similar. How about checking for 0-255 like so: "([1-9][0-9]{0,1}|1[0-9][0-9]|2[0-4][0-9]|25[0-5]| 0)", then it's just a matter of checking for those between dots?

  • Regex method is better (Score:2, Insightful)

    by Anonymous Coward
    Your oversimplification of his solution for validating ip addresses is a fine example of a poor review by someone who thinks he knows more than the author.

    Try passing in a string such as "I.like puppies!!!". A regex like the one the author provided will easily reject this, so there's no need to worry about checking for numericness, or any other strange characters at all. The regex in fact filters out EVERYthing so that all that has to be done is to check the actual numeric values for the right value range.

  • Reviewing the book or showing off geekiness? (Score:4, Insightful)

    by zanderredux ( 564003 ) * on Monday June 20, 2005 @03:35PM (#12866496)
    Similarly, one of the examples in the chapter on regular expressions uses a regular expression to check to see if a string contains a valid IP address (pages 65-66). After showing how to use a regular expression to scan a dotted quad of digits, the text then admits that using a regular expression alone would lead to too much complexity, since it's hard to use a regular expression to check to see if a 1 to 3 digit number is less than 255 (or 127, which is what he uses in his code). So the example on page 66 ends up compiling and matching a regular expression like this:

    pat = re.compile("(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\ .(\\d{1,3})")

    Actually, that example is safer than just invoking text.split, as that long regex can shield you from injection attacks and help you enforce numeric IPs in one single command.

    In the end, it is a matter of style, but just invoking text.split and trusting user input is... naive?!

    • I most agree with you. You should use regex for what it is for: checking if the structure of the input is correct. Leave the checking of the actual values to the program. His comments just split at '.' characters. So this means that e.g. +23.-56. 255.1e34 might evaluate to a "correct" IP address.

      The book shows the exact way I would do it; check for the maximum amount of structure in the IP adress, allowing only digits and dots, and then proceed to make sure 344.344.344.344 is not accepted. There is nothi

  • Learning concepts...bah! (Score:3, Funny)

    by Anonymous Coward on Monday June 20, 2005 @03:37PM (#12866526)
    Wilson spends a fair amount of time in the text chapter illustrating code to parse command-line parameters, before admitting that libraries for the task abound in most languages.

    You know I had that same problem with my Operating Systems class. That text by Tannenbaum goes through countless examples of what makes a good system, and then at the end he FINALLY admits that there is something called Unix that I can just go and install. What a waste learning all of those concepts!

  • It's not fair to criticize the book because you use a tarted up text file instead of something like postgres or oracle or db2 or any number of other rdbms's that managed to support subqueries and foreign keys within 30 years of their invnetion.
  • "read this book, but with a somewhat critical eye." Blindingly obvious but good advice.
  • Sounds like you're having a little too much fun with your database...

  • Not mentioning tac is not a dealbreaker (Score:4, Insightful)

    by illumin8 ( 148082 ) on Monday June 20, 2005 @04:28PM (#12866977) Journal
    I don't fault the author for not mentioning tac. It is part of the GNU textutils package, and although it might be standard on every Linux distro, it's most likely not in ANY enterprise Unix. I just checked my Sun boxes and it's not installed there, except for the ones that I've installed GNU textutils on.

    I really wish a lot of Open Source developers would stop assuming that all of us have every GNU utility ever invented on our system. I can't tell you how difficult it is to get the average GNU autoconf program to compile correctly on Solaris or any flavor of enterprise Unix, simply because most authors assume they're writing platform-independent code, without realizing that GNU's M4 is different from System V M4. Also, differences between lex, flex, tar, and GNU tar abound. Please, for the love of god, don't assume that the tools you know and love on your Linux box at home are available or even installable on enterprise kit at work. Most company policies prevent the installation of these type of tools.
    • Not installable by you, of course. But not installable? You seem to suggest that it is more difficult to install for Solaris. Doesn't Sun have a GNU toolchain site? I always thought that:

      It is a vital component in Linux kernel development, BSD development and a standard tool when developing software for embedded systems. Parts of the toolchain are also widely used in the Solaris Operating Environment (which, in the opinion of many, needs the GNU tools for reasonable usability) and Microsoft Windows

      • Not installable by you, of course. But not installable?

        Haha, yeah, I don't even know how to go to SunFreeware [sunfreeware.com] or Blastwave [blastwave.org] and download a copy of GNU textutils in Solaris package format. You can think that if you want to, but in the enterprise world, every software package I want to install has to be approved by about 3 levels of management. They want to know what it does, why we need it, how much it costs, and who else will know how to maintain it after I leave the company. The chance of providing the

  • MySQL (Score:5, Informative)

    by DogDude ( 805747 ) on Monday June 20, 2005 @04:45PM (#12867096)
    I have another problem with the section on SQL. Several of the slicker SQL recipes rely on nested queries (page 147-151). MySQL, clearly a very popular SQL database, has nested queries only in its latest versions, so many, if not the majority, of MySQL installations do not yet have that capability. Yet the text carries on as if nested queries were universal, without so much as parenthetical mention that some things might not work on all SQL implementations. It seems to me that this is exactly the sort of pitfall a book like this should inform the reader of.

    Nested queries are *basic* database functionality. This is just one of many reasons why those of us who are experienced DBAs and database developers do not consider MySQL a database. The fact that there are lots of people trying to use it as such is irrelevant. The author didn't mention that the book is also missing a section of spreadsheets. Why not? Lots of people use spreadsheets as a database!
    • Agree. It's only a toy if it doesn't support subqueries. Insert coin.

      • Re:MySQL (Score:3, Insightful)

        by DogDude ( 805747 )
        What I can't believe (and I'm replying more to myself than anything else, because I just realized...) is that if MySQL hasn't been supporting something as basic as sub-queries until recently that means that there have been tons and tons of complex applications written without subqueries! Holy mother of christ... How would something as simple as even Slashdot get written without subqueries? There must be thousands upon thousands of apps out there that were written with almost -no- understanding of what a m
        • heh, was that a very obscure mocking of typical J2EE peristence layer architecture?

        • Re:MySQL (Score:3, Insightful)

          by Matje ( 183300 )
          So from the fact that MySQL lacked subquery support you derive that there are a lot of bad programmers? me thinks there is only evidence here that you're a bad logician. Now that is a skill a good programmer must have ;). A couple of remarks:

          - if you're building a simple website, chances are you won't need any subqueries. Websites were (are?) the bread and butter of MySQL.

          - the fact that the dbms lacks subquery support does not imply that the programmer lacks knowledge about them, nor does it imply that p

    • Re:MySQL (Score:3, Insightful)

      by quasi_steller ( 539538 )

      DBAs and database developers do not consider MySQL a database.

      You have got to be kidding me. Of course MySQL is a database. A database is simply a collection of data organized so that a computer program can access pieces of that data, something a MySQL database certainly does. This would make MySQL as a whole, a DBMS (DataBase Management System), as it is a collection of programs used for managing a database. Now, Is MySQL a RDBMS (Relational DBMS)? Well, that depends on your definition of RDBMS.

      • Mysql is to a real RDBMS as Windows 3.11 is a true multitasking, multiuser, and reliable OS.

        Sure Windows 3.11 can theoretically support multiple users and multitask but I would prefer W2k thank you.

        Same is true with mysql. Mysql is popular because its free and is very multi-user friendly for ISP's with tons of user accounts so they bundle it with their hosting.

        PostgreSQL is arguably alot better and also free. In asia its what most Linux users use by default. The tools for it are finally cominging out and
  • "I really expected to love Data Crunching"

    It's interesting the way that's written, because it tells me that you didn't like the book in the first sentence. If getting people to read the entire review was an issue, which is not the case here, then that would have been moved to the last paragraph.

  • Munging Alternative (Score:3, Informative)

    by PotatoMan ( 130809 ) on Monday June 20, 2005 @05:02PM (#12867242)
    You might want to compare this book to "Data Munging With Perl" by David Cross.

    See the Slashdot Review:
    http://books.slashdot.org/article.pl?sid=01/04/26/ 1229238&tid=145&tid=6 [slashdot.org]

  • Near the beginning of the post, in the green box, we have:

    author | Greg Wilson

    And yet, in the final paragraph we see:

    In spite of them, it is still a very good and useful book and Mark Wilson has done a good job with a topic all too often ignored.

    What's going on?
  • Are there any better books about data crunching? I found at least Data Munging with Perl by David Cross. BTW: check out DataConv for a survey of data conversion tools [dataconv.org], many of them GPLed and often unix-based.

Slashdot Top Deals

Do you suffer painful hallucination? -- Don Juan, cited by Carlos Casteneda

Close