Google To Digitize Much of Harvard's Library 296
FJCsar writes "According to an e-mail sent today to Harvard students, Google will collaborate with Harvard's libraries on a pilot project to digitize a substantial number of the 15 million volumes held in the University's extensive library system, which is second only to the Library of Congress in the number of volumes it contains. Google will provide online access to the full text of those works that are in the public domain. In related agreements, Google will launch similar projects with Oxford, Stanford, the University of Michigan, and the New York Public Library. As of 9 am on December 14, a FAQ detailing the Harvard pilot program with Google will be available at hul.harvard.edu."
Nice! (Score:2)
Re:Nice! (Score:2)
No.
Re:Nice! (Score:5, Informative)
DP probably isn't threatened either - they just shift focus to books that are not in the Harvard collection to avoid duplication of effort.
Re:Nice! (Score:3, Insightful)
Are they really going to provide proofread texts? A novel might only take a couple hours to process, but math is going to take hand markup, and some of the more complex critical editions are a bear. Even at only 2 hours a book (and that's not including scanning time), 4 million volumes adds up to 8 million man-hours or a million man-days. At seven bucks an hour that's
Re:Nice! (Score:2)
Having looked at their Catalogs beta, I still suspect they just may have... not only is the text OCR'd well enough to search it, but it even highlights the words in the text. They could certainly have hand-proofed, but that strikes me as not a very Google thing to do.
Re:Nice! (Score:2)
Re:Nice! (Score:2)
I should scrounge around their forums and see if they state what the final disposition of the scans is.
Re:Nice! (Score:4, Informative)
Re:Nice! (Score:5, Informative)
I happen to work for one.
It's focused on putting otherwise one-of-a-kind materials online for preservation and ease of access, rather than Byron: The Critical Anthology or Cather on the Rye. It's kind of a mammoth, innefficient beaurocracy, though; I don't agree with some of the practices (such as sending texts off to India to be scrivened, rather than just using OCR software), they're very, very slow to incorporate data, and there are a lot of other problems which stem from the fact that most of them are not computer people, but MIMS holders (librarians).
The fact that Google is doing it gives me hope. Hell, maybe I can jump ship.
Re:Nice! (Score:2)
One more reason... (Score:2, Insightful)
ads (Score:5, Funny)
Re:ads (Score:5, Funny)
Yes, but it'll be mixed in with ads for V14gr4, male "enhancement", and Nigerian wealth opportunities. When the scientists complain, the humanities faculty will protest that spam is a perfectly valid epistemology, and that the scientists' attempt to impose an orthodoxy of "truth" in advertising is simply a power grab to extend Western, white male hegemony. At which point, the scientists will defect to MIT's library down the street.
Cheers,
IT
Re:ads (Score:3, Funny)
Google Cars (Score:2, Funny)
Will it be like google scholar? (Score:5, Interesting)
I think this is a great start, There's incredible profit here too, universities spend millions for catalogue systems. If I could use one interface to search for books, chapters, and articles on a subject, I could spend more time actually learning, and less time looking at the same damn "no results" page on GeoWeb. Grrrr.
Re:Will it be like google scholar? (Score:2, Interesting)
Re:Will it be like google scholar? (Score:5, Interesting)
Or finding that perfect article in the MLA database, only to find out that nobody in Canada subscribes to the journal, nor does anybody have the journal on fulltext. I'd rather have a more comprehensive fulltext database in plaintext rather than digitalised copies of everything anyway - makes searching a hellova lot easier.
Re:Will it be like google scholar? (Score:5, Insightful)
Sure, the company needs to get some money to cover the costs of printing, distribution, and other things, plus the associations that sponsor the journal want some money to help hold conferences, but why, oh why, must they price journals so expensively that many colleges can't even afford them?
Re:Will it be like google scholar? (Score:2)
Printing a publication is expensive. Journals are advertisement-free, which is why they cost so much. I used to work for a student newspaper and it was ridiculous how much money we were paid for ads. Without that rev
Why journals are expensive. (Score:5, Interesting)
No; the reason there are so few copies is there are so few people who want to read specialized journals. And the small audience only accounts for a small part of what many academic journals charge.
No; the problem is not overhead costs or small audiences. The problem is that the owners of much of that kind of content are greedy bastards. There is no reason for the outrageous price of some journals. Some scientific journal subscriptions are in the tens of thousands; even many liberal arts journals are far from cheap. And if you want to copy an article for your students to buy at kinkos, expect them to pay 35 cents a page or more for the copyrights alone.
And many of them are worse than the RIAA in terms of access to content electronically. Journal articles are included in databases sold to some universities You can read articles in some databases but only by loading a .gif of every page one at a time. No copy and paste, no text access at all. So much technology going into preventing the thing from being copied that the online version is actually less useful than the dead tree version rotting on the shelf.
I think this is a great move by Google and Harvard, and I like the idea behind google scholar, but I expect this kind of work to be resisted by many of journals and professional organizations, to the extent that they have in a say in it. This will be a huge boon in terms of the availability of public domain resources, but unfortunately outdated perspectives on intellectual property are likely to hold back real progress for something really useful to scholars in a systematic way. At least until those perspectives change significantly.
Re:Why journals are expensive. (Score:3, Informative)
http://math.berkeley.edu/~kirby/journals.html [berkeley.edu]
Re:Why journals are expensive. (Score:5, Informative)
As for fulltext articles, try JSTOR if you want to see how to do it wrong. Page by page in gif format, and some huge pdfs with all pictures and no ability to process text. Useless!! Yes you can print it out but then I'd just as soon get the hardcopy in the first place.
Re:Why journals are expensive. (Score:3, Insightful)
JSTOR varies in quality from journal to journal--some are actually okay, while others suck. I know that I have gotten pdf's from JSTOR, but I wonder if that is a function of JSTOR or the amount that a person/institution is paying for access.
Most journals that I have dealt with online where I had to pay (because the university wasn't a subscriber) wanted between $15 and $25 for
Re:Will it be like google scholar? (Score:3, Interesting)
Re:Will it be like google scholar? (Score:2)
Re:Will it be like google scholar? (Score:3, Interesting)
Good quality search engines have lots of qualities that Google lacks. You could search for two words located within 3 words of each other. You could search for these two words within 3 words of each other while two other words don't occur within 6 words of each other. Index
Re:Will it be like google scholar? (Score:3, Insightful)
Good quality search engines have lots of qualities that Google lacks.
One solution is to use google to locate a superset of the target articles and then use a more powerful search engine to winnow the google result set. For an individual, this approach would mean maintaining a personal index of the articles but that is a problem of storage space and bandwidth which is relatively cheap.
The two main problems that google solves is
Re:Will it be like google scholar? (Score:2, Informative)
I've been using CiteSeer [citeseer.com] for years in my research, and still prefer it over Google Scholar [google.com].
For computing research CiteSeer and the ACM DL [acm.org] are the two places to go. Scholar may obviate the need for going to both places, someday, but for now it needs to mature a bit.
How about Web of Science? (Score:2)
If you're willing to pay, this is exactly what Web of Science [isinet.com] does. It contains just about every article from every journal for the last hundred years.
WoS uses citation indexing, as ISI has done for many years, since well before Google came into existance. You can find newer articles by finding those which have cited the old article you're lo
Re:Will it be like google scholar? (Score:2)
Re:Will it be like google scholar? (Score:2)
Re:Will it be like google scholar? (Score:2)
So... (Score:4, Funny)
Google to cache the Universe (Score:3, Funny)
get your scuba gear... (Score:2, Insightful)
Re:get your scuba gear... (Score:2)
They are getting there.
15 million volumes? (Score:3, Funny)
Re:15 million volumes? (Score:3, Funny)
From Fascinating Facts About the Library of Congress: [loc.gov]
The Library of Congress is the largest library in the world, with nearly 128 million items on approximately 530 miles of bookshelves. The collections include more than 29 million books and other printed materials, 2.7 million recordings, 12 million photographs, 4.8 million maps, 5 million music items and 57 million manuscripts.
So to answer your question, it's about 0.52 LoC if you count only the books. :)
Re:15 million volumes? (Score:2)
Re:15 million volumes? (Score:5, Informative)
The British Library (www.bl.uk) has 150 million items (but fewer bookshelves) so the claim of "largest" is a bit dubious.
For
Re:15 million volumes? (Score:5, Funny)
He means just books and such. It's not fair counting umbrellas.
Re:15 million volumes? (Score:2)
Re:15 million volumes? (Score:3, Funny)
For
Sorry, I still don't understand... Could you express that in terms of how man shuttle explosions would be required to completely destroy one BL?
Re:But it's the same damn book every time (Score:2)
The penguins were developing WMD's
Re:But it's the same damn book every time (Score:2)
Re:15 million volumes? (Score:2)
How many Libraries of Congress is it?
Images and formatting? (Score:3, Insightful)
Just how much storage space will all this data consume? It seems like a massive undertaking.
Are these volumes stored as text or pictures? (Score:3, Insightful)
Re:Are these volumes stored as text or pictures? (Score:3, Insightful)
Re:Are these volumes stored as text or pictures? (Score:4, Interesting)
Re:Are these volumes stored as text or pictures? (Score:5, Insightful)
Both Images & Uncorrected OCR should be availa (Score:5, Insightful)
The uncorrected OCR is very useful for indexing (by Google or others), as the 5% or fewer typos are not enough to interfere with indexing keywords. Uncorrected OCR can also be corrected later.
The page images are tied with the uncorrected OCR so you can see exactly what's there.
For an example, see books at University of Michigan's Making of America (MoA) Exhibit [umich.edu], which has thousands of 19th century books and periodicals available.
Re:Both Images & Uncorrected OCR should be ava (Score:3, Funny)
I see they've recently added the complete run of the Journal of the U.S. Association of Charcoal Iron Workers. If I'd known that, I could've saved a bundle on gift subscriptions...
Re:Are these volumes stored as text or pictures? (Score:2)
For the vast majority of the people who will ever use the tool, that won't matter. Most of the world's libraries don't hold onto old scholarly stuff indefinately, assuming that they ever bought a lot of the obscure stuff. It seems likely that because this will be limited to public domain works, most of them will be old and hard to find, so anyone looking at them will quite likely have had no way to acc
The Fight against Plagiarism (Score:5, Interesting)
Flipside: The false positive problem (Score:3, Insightful)
False positives can be double-checked manually (Score:2, Insightful)
You'd want to do a thorough overview of any potential instance of cheating anyway. A quick run-through would determine whether or not a paper happened to contain an identical sentence clause or three identical paragraphs.
I think the bigger problem would be the second one you described -- that students could plagiarize and then go through each paragraph, changing the wording slightly so as to avoid positive matches. S
Re:Flipside: The false positive problem (Score:2, Interesting)
Using work of other people in academic work is not only possible, but greatly encouraged. Just make sure that it is very clear what comes from whom.
In many ways, science is done exactly as Open Source software. Take what you need, modify and improve it where appropriate, and make sure you give full credit where due.
As a teacher, I have given full points to a paper that has hardly any text of their own, as long as they are properly referenced, and used together to
How will the books be scanned? (Score:2, Interesting)
"Slice and scan" is used for new books only (Score:4, Informative)
For older books, most archivists use a cradle and photograph the pages. It's easier on the book, requires no slicing, and there's no scanner to clog with dust.
The disadvantage is the scanner operators need a little bit more training, but that's not a big problem.
But will you be allowed to copy the materials? (Score:2)
Re:But will you be allowed to copy the materials? (Score:2)
Reminds me of the U of Michigan and U. Microfilms (Score:3, Informative)
U Microfilms set up and ran a microfilming operation in the library system, microfilming everything that wasn't under copyright (and much that was with permission of the copyright holders, such as several large newspapers and many magazines and other periodicals), along with much of the University's records. Rare books, etc.
(If I have this right) the U got microfilm prints of the documents for free and didn't have to pay for the microfilming of its records. University Microfilms made its money by selling microfilms of the various publications (forwarding royalties, where appropriate, to the copyright holders). The rare books, for instance, could now be studied on microfilm with no further stress on the original, and their content became available at many other colleges and libraries. Good deal all around.
University Microfilms was founded by a regent, who was later slammed for conflict of interest. He dropped out of the Board of Regents but the business deal continued.
clinton (Score:2)
Homer: mmmmmm digitized google....
University of California is anti-digital (Score:5, Informative)
For books in Special Collections, they won't allow copies to be digitalized unless they are (1) paid a fee to scan the book (fair enough) and (2) paid a royalty to post the book to the web.
The royalty amounts to hundreds or thousands of dollars per book (about $100/page or image). This allows the libraries to act as a "profit center" for the universities. This policy applies to all UC campuses (I've tried UCB, UCLA, UCI, UCSD).
This is true even though the book is in the public domain (because they have physical possession and nobdy can make copies until you sign a license agreement). This is true even if you're using the book for non-commercial purposes (such as free posting to the web).
Something is wrong here. People donate to UC libraries (either books or money) for the public good. They don't donate so the library can start a business licensing public-domain books.
Despite that, I have been able to scan many books (by using books in open stacks or purchasing them). These books concern Yosemite history and are at http://www.yosemite.ca.us/history/ [yosemite.ca.us]
Re:University of California is anti-digital (Score:2)
Re:University of California is anti-digital (Score:3, Interesting)
Ever tried a Freedom of Information Act (FOIA) request? Strange as it may seem, that apparently works in the State of Washington.
FOIA fees (Score:3, Informative)
FYI, FOIA isn't free, though the fees are pretty nominal [cia.gov]. $0.10/page, $18/hr, after the first 100 pages, with a significant educational discount.
The thought of having a spook do my photocopying for me just sounds.... Hrm. Ironic?
Re:University of California is anti-digital (Score:2)
Text of Dec 13th Email (Score:5, Informative)
Dear Colleague,
I am writing today with news of an exciting new project within the Harvard libraries. As all of us know, Harvard's is the world's preeminent university library. Its holdings of over 15 million volumes are the result of nearly four centuries of thoughtful and comprehensive collecting. While those holdings are of primary importance to Harvard students and faculty, we have, for several years, been considering ways to make the collections more useful and accessible to scholars around the world. Now we are about to begin a project that can further that global goal-and, at the same time, can greatly enhance access to Harvard's vast library resources for our students and faculty.
We have agreed to a pilot project that will result in the digitization of a substantial number of volumes from the Harvard libraries. The pilot will give the University a great deal of important data on a possible future large-scale digitization program for most of the books in the Harvard collections. The pilot is a small but extremely significant first step that can ultimately provide both the Harvard community and the larger public with a revolutionary new information location tool to find materials available in libraries.
The pilot project will be done in collaboration with Google. The project will link Harvard's library collections with Google's resources and its cutting-edge technology. The pilot project, which will be announced officially tomorrow, is the result of more than a year of careful consultation at many levels of the University. We could not have achieved a meaningful pilot project without the efforts of the Harvard Corporation; the President, Provost, Chief Information Officer, and Office of General Counsel; the University Library Council; and senior managers within the College Library and the University Library.
A full description of the pilot program follows here, with further materials available on the Harvard home page tomorrow.
With best regards,
Sidney Verba
Carl H. Pforzheimer University Professor and
Director of the University Library
Project Description:
Harvard's Pilot Project with Google
Harvard University is embarking on a collaboration with Google that could harness Google's search technology to provide to both the Harvard community and the larger public a revolutionary new information location tool to find materials available in libraries. In the coming months, Google will collaborate with Harvard's libraries on a pilot project to digitize a substantial number of the 15 million volumes held in the University's extensive library system. Google will provide online access to the full text of those works that are in the public domain. In related agreements, Google will launch similar projects with Oxford, Stanford, the University of Michigan, and the New York Public Library. As of 9 am on December 14, an FAQ detailing the Harvard pilot program with Google will be available at http://hul.harvard.edu.
The Harvard pilot will provide the information and experience on which the University can base a decision to launch a large-scale digitization program. Any such decision will reflect the fact that Harvard's library holdings are among the University's core assets, that the magnitude of those holdings is unique among university libraries anywhere in the world, and that the stewardship of these holdings is of paramount importance. If the pilot is deemed successful, Harvard will explore a long-term program with Google through which the vast majority of the University's library books would be digitized and included in Google's searchable database. Google will bear the direct costs of digitization in the pilot project.
By combining the skills and library collections of Harvard University with the innovative search skills and capacity of Google, a long-term program has the potential to create an important public good. According to Harvard President Lawrence H. Summers, "Harvard has the greate
Oxford University gets every UK book published (Score:3, Informative)
The library of the University of Oxford, i.e. the Bodleian Library [ox.ac.uk], was the first "copyright" library in the UK - one of only three - which means that it automatically gets a copy of every book published in the UK [ox.ac.uk].
Aegilops
Re:Oxford University gets every UK book published (Score:5, Informative)
Re:Oxford University gets every UK book published (Score:2, Interesting)
All well and good, except (Score:2, Offtopic)
(they admit it themselves!)
Re:All well and good, except (Score:2)
Do no evil. (Score:5, Funny)
Maybe they could do something really evil to Microsoft, and then we could say: ``Well, you digitized Harvard's library, so we'll let it pass this time.''
Amen (Score:3, Informative)
Comment removed (Score:3, Informative)
It's about Time! (Score:2, Interesting)
Someone hurry up wi
Re:It's about Time! (Score:2)
Mailing Lists (Score:2, Interesting)
Re:Mailing Lists (Score:2)
I would like to see a Library section, of all the books scanned in(preferably text, with an image linked-to, rather than a image you read off of).
Also, I would think it would be neat to see a mailing-lists section either as an extension of their new google-groups2 system, if possible.
Lasly, I a blog search would be neat, though tricky. Being able to do an 'Opinion' section or
Re:Mailing Lists (Score:2)
second only to the Library of Congress. . . (Score:2, Informative)
Re:second only to the Library of Congress. . . (Score:3, Informative)
The British Library apparently has "150 million items" according to their web site, but a large number of these are not books (they claim, for instance, to have 8 million stamps). But, I'm pretty sure they have more than 15 million books.
Whether or not they have more books than the Library of Congress is an interesting question.
Re:second only to the Library of Congress. . . (Score:2)
Re:second only to the Library of Congress. . . (Score:3, Informative)
The Bodelian has only 7 million volumes [ox.ac.uk].
I would suspect that the Brish Library is substantially larger than Stanfords, but the Library Of Congress is recognised as the largest library in the world. [guinnessworldrecords.com]
Steve.
U of Michigan (Score:5, Informative)
Re:U of Michigan (Score:3, Interesting)
The size of the U-M undertaking is staggering. It involves the use of new technology developed by Google that greatly speeds the digitizing process. Without that technology -- which Google won't discuss in detail --
"second only to the Library of Congress" (Score:2)
Granted, some of these are just stamps
Championing the commons (Score:2)
I love moves forward like this. Perhaps if people understood what it meant to access knowledge and information at whim they wouldn't be so keen to keep extending privately held rights any further than is reasonable.
I live for the day when people count down the days until something e
New York Times article (Score:4, Informative)
For what it is worth, there was an article in the Painted Lady [nytimes.com] about it today.
Re:Not Just Harvard (Score:5, Funny)
Re:Not Just Harvard (Score:2)
Yes but the FS is starting to go the way of the FA as far as the number of actual readers is concerned. I admit to occasionally falling victim to this unfortunate disease myself. Sometimes I only read the headline, and with some of the YRO ones that take up nearly the whole width of my 1280px wide monitor, sometimes I can't even get through all of that.
Dead authors tell no tales . . . till now (Score:3, Insightful)
Only public-domain books will be scanned. In all or most cases the author's are dead. However, this will revive a great body of work and widen access to many.
One class of author may be pissed will be authors who take older works and just slap a foreword or introduction to the front and collect royalties. I've seen this done for many histories. But author's of todays works can count on royalties for themselves, their childr
Re:Speaking of education... (Score:2)
Re:Just what percentage... (Score:2)
With a catalogue that size, probably most of them. The number of new books published per year isn't actually all that huge -- even if you acquired everything published in the US, I would expect it to take a long time for you to reach 15 million items.
Note that, for instance, the LoC has 29 million books, which is understood to be a significant fracti
Re:I beg your pardon... (Score:2)
But of course, that doesn't come after the LoC, so whether that makes the story factually inaccurate, or just misleading, is an interesting question.
Re:how will this be better than 'grep' (Score:2)
However, grepping through 15 million volumes of text and making an attempt at ranking results by relevance through a fancy perl script probably would require a bit of time and resources