Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
×
Open Source Data Storage IT

Ask Slashdot: What Is the Best Open Document Format? 200

kramer2718 writes: I am working on a project that requires uploading and storing of documents. Although the application will need to allow uploading of .docx, doc, .pdf, etc, I'd like to store the documents in a standard open format that will allow easy search, compression, rendering, etc. Which open document format is the best? Since "best" can be highly driven by circumstances, please explain your reasoning, too.
Have a question for Slashdot's readers? Take a look at other recent questions first to see if someone else has had a similar question. And if not, ask away! The more details and context you include, the more likely your question will be selected.
This discussion has been archived. No new comments can be posted.

Ask Slashdot: What Is the Best Open Document Format?

Comments Filter:
  • by alen ( 225700 ) on Thursday May 14, 2015 @11:57AM (#49690583)

    if you use the API's supplied by their creators?

  • PDF/A (Score:5, Informative)

    by thechemic ( 1329333 ) on Thursday May 14, 2015 @12:00PM (#49690629)
    • +2

      Hundreds of people who do this for a living (they're called records managers), and have done for many years, have worked long and hard to come up with a standard format for exactly this. Doesn't do everything, but what does, but it does ensure that it will still do it in 50yrs if not longer.

      Caveat 1: OP doesn't mention editing, if he needs it editable then don't convert, or store original and PDF rendition for preservation

      Caveat 2: There is a trade off between doc size (OP mention compression) and digita

  • by PSVMOrnot ( 885854 ) on Thursday May 14, 2015 @12:01PM (#49690637)

    I would suggest, unless you have a pressing need to convert them, that you should store the documents in the formats they are uploaded in.

    Whenever you convert a document you run the risk of completely messing up the layout, style, etc.

    • by Anonymous Coward

      There's storing for later download, and then there's storing for ongoing analysis, indexing, previews, etc. For the latter, it would help a lot to have one standard format. Probably plain text.

      To properly analyze .doc / .docx, for instance, you'll probably need a Windows machine with Word installed. It will likely be significantly cheaper to have Word installed on only one or two machines, convert to text (capturing any necessary metadata on the way), and then do further processing on other machines that do

      • Re: (Score:3, Interesting)

        by darkain ( 749283 )

        All of the "X" variants of MS Office documents stand for "XML" - that is, the documents are stored in a series of XML files inside of a ZIP file that is renamed to formatX (docx, xlsx, etc). There is no real need to even have Windows or Office installed to index these documents. Just write up a basic script to extract the ZIP file and parse out the related XML documents. Note: this isn't as trivial as it sounds at first, though. This would assume that Microsoft's XML structures (yes, plural), had an easy to

        • Some of those files are just an XML wrapper around a binary format for which the documentation is not available outside of Microsoft. The wrapper meets the legal obligations but the file format in such cases is ultimately useless in the long term.
          Meanwhile I can import seismic data from the early 1970s into current software without any conversion - simply because the file format is documented instead of Microsoft's later step backwards.
          • Meanwhile I can import seismic data from the early 1970s into current software without any conversion -

            Strange, but that is exactly what I am doing at the moment. Or at least, I was doing until a few moments ago, when the task finished. Hi ho! back to the grindstone!

    • by Anonymous Coward on Thursday May 14, 2015 @12:25PM (#49690933)
      Or store both the original, and a standardized format. The place I work stores everything from engineering drawings, meeting minutes, purchase records, to manuals of old equipment in a central document library. It retains the original file, and makes a pdf of every file, and a link to both is listed in each entry. We've already had some older CAD formats no longer supported by current software we have easy access to, but the old pdfs are still readable and it is cheap enough to find some intern to re-create the document from the pdf if need be.
      • by AthanasiusKircher ( 1333179 ) on Thursday May 14, 2015 @12:46PM (#49691251)

        Or store both the original, and a standardized format. The place I work stores everything from engineering drawings, meeting minutes, purchase records, to manuals of old equipment in a central document library. It retains the original file, and makes a pdf of every file, and a link to both is listed in each entry.

        THIS.

        PDFs (or some similar standard) will ensure that the original documents can be read by everyone and viewed with the original formatting intended by the person creating them. Any differences in the version of Word or whatever is going to tweak the formatting in unpredictable ways.

        But the originals should always be retained, since it may make future editing easier. And people also won't be stuck trying to undo whatever unpredictable reformatting or editing (e.g., loss of certain features moving between formats) might go on in your conversion process.

    • by mlts ( 1038732 )

      Even with programs that can import Word/Excel/etc. documents, they do a good job, about 99% well. However, that one percent that is missed can do quite a number on a document.

      The answer for a document format... depends.

      For a document format that keeps formatting exactly, and isn't intended to be edited, PDF/A is the best thing going, since barring a major world-ending disaster, we will still have utilities that can read PDFs, and PDF/A ensures that the fonts and such are present and readable.

      For a document

  • .txt (Score:5, Interesting)

    by Anonymous Coward on Thursday May 14, 2015 @12:01PM (#49690639)

    .txt. If you need pretty formatting, fill it Latex tags.

    • Re:.txt (Score:5, Insightful)

      by jythie ( 914043 ) on Thursday May 14, 2015 @12:05PM (#49690709)
      And here I am without mod points...

      Generally when I have to worry about integration or longevity, it is still hard to compete with ASCII & LaTeX. While they do not have the every day visibility of various office document types or pdfs, renderers, search tools always know exactly what to do with them. They can even interact with version control systems cleanly since the underlying tools do not need to know anything about the formatting to manipulate it.
    • I love this answer... but sadly people aren't willing to learn things like latex. Even in academia (medical physics) many smart people refuse to learn technologies like latex.

      And, despite my agreement (I was actually going to post a similar answer if someone hadn't), there are times when I don't want to bother with the latex overhead for quick documents. Am I doing it wrong? ;-)
    • I agree. Text works. Now we get to argue over ASCII, Latin-1, ISO, Unicode, tabs vs spaces, emojis versus emoticons, CR/LF vs CR vs LF, byte stream or record format, CamelCase or pythonCase or unixcase, but thankfully we don't have to decide on vim vs emacs as those are external tools.

    • Pretty much my thought. Use the simplest format that will do the job. It it's just prose, use txt. Does anyone seriously believe that One Day in the Life of Ivan Denisovitch is somehow enhanced by saving it as .doc or .pdf or .htm or god knows what else? If the text needs some bold and italics, use .txt with markdown. If it needs lots of markup, then something more elaborate -- preferably something with standards and a DTD or equivalent indicating what standard applies. If there are flat tables, use c

      • Perhaps fine for Roman characters, not so fine if the document contains Kanji, Hiragana, Katakana, Hebrew, or any of the other character sets that don't play nice with "plain text" formats. For something that you would think would be pretty straight forward, plain text character handling is surprisingly maddening to work with.

        • Yes text handling for non-ascii characters can be surprisingly maddening to work with. (Wasn't UTF-8 supposed to fix that?). Problem is that wrapping txt in some more elaborate format like HTML often doesn't make the problem go away. With apologies to Jamie Zawinski It just means that now you have two problems.

  • by Anonymous Coward
    Either you truly wish to use something "open", at which point the only choice is ODF, or you simply want something that can be widely used. If the latter is the case, ODF is still good, perhaps PDF.
  • I am working on a project that requires uploading and storing of documents. Although the application will need to allow uploading of .docx, doc, .pdf, etc, I'd like to store the documents in a standard open format that will allow easy search, compression, rendering, etc. Which open document format is the best?

    PDF allows accurate rendering so it's the best choice. It will be a hot mess if you use anything else. Conversion of such complex formats is very error-prone for layout problems.

    • PDF is a print format, which is fine if your audience is going to print it out on a piece of A4 paper - though I think yanks have their own standard. :)

      But they don't generally reflow [wikipedia.org]. e.g. Viewing a document formatted for portrait on landscape monitor, journal articles with multiple columns, reading on a 4" smartphone are challenges for reading onscreen.

  • by xxxJonBoyxxx ( 565205 ) on Thursday May 14, 2015 @12:05PM (#49690705)

    1) Forget the Universal Format approach - your users will kill you for messing up their formatting, and you'll never get complete feature parity
    2) Store the docs in their original format
    3) Get Apache Solr to search your content
    4) You'll be spending a lot of time on #3, so leave time to tinker

    • I'll second everything but step 3. I would get some standard libraries set up to extract the plain text from each format and make that searchable. Probably much simpler.

      • by Nadir ( 805 )
        Why, Apache Solr understands various type of office documents via Apache POI. No need to get "standard" libraries, whatever you mean by that.
    • Re: (Score:2, Informative)

      by Anonymous Coward

      I work at a typography, and I get a lot of documents from a lot of different people. Those "documents" come as MSWord files with missing fonts, pdfs made with some shoddy software, strange ODTs, many more different types of doc files, the mysterious lnk files that work perfectly fine for them, but not for anyone else and my personal favorites, jpg files (not png or some other lossless format, because that would imply actual thinking).
      Strange enough, I've yet to receive any plain text files.

      To index everythi

  • by Anonymous Coward on Thursday May 14, 2015 @12:06PM (#49690715)

    Word Perfect Document, because it's been consistent for nearly 20 years. it has a simple underlying format, it's more finely granular than HTML and because I just like obsolete things.

  • I'd like to store the documents in a standard open format that will allow easy search, compression, rendering, etc. Which open document format is the best?

    Are you writing the search/compression/render capability from scratch, or are you using a library to handle that job for you?

    If you're handling more than one document type, then go for a library. I don't have a recommendation myself, but I'm sure you can find them on a search.

    Also, don't worry about compression, as modern .odf/.docx is already compressed

  • by nine-times ( 778537 ) <nine.times@gmail.com> on Thursday May 14, 2015 @12:10PM (#49690745) Homepage

    As an IT person, I hate questions like this. There's not enough information to give a solid answer. For example:

    * What kinds of documents are you talking about? Text? Photos? Spreadsheets?
    * What is the source of the documents? Are these currently printed out documents that need to be scanned back in? Are they currently digital, and in a particular file format?
    * What will people need to do with them when these documents are retrieved? Do they need to be able to edit the documents?
    * How much does formatting matter? If someone retrieves the document in 5 years, will it be important that all the line breaks and page breaks are in the same place? Does it need to have all of the correct fonts? Or are you more interested in being able to have access to the information itself?
    * When you say that the application will need to allow ".docx, doc, .pdf, etc", what formats are in "etc"?

    There may be many other relevant questions, my point is that there just isn't enough detail here. In general, if the most important thing is that you have a printable document that you want to be able to print out from any machine, maintaining the formatting as much as possible, then PDF is a pretty good choice (be sure to embed the fonts and include searchable text!). If you already have a bunch of Word documents and you want the formatting unchanged, and would like the capability to edit the document after it's retrieved, then I'd typically just recommend keeping it as a .docx. It keeps things simple, will be widely supported, and prevents the risk of something going wrong while you're converting to another format. If you like the idea of using .docx because of what I just said, but want something more "open", then ODF is probably worth looking into.

    Really, there are only so many choices, and each have advantages depending on your specific needs.

    • * What kinds of documents are you talking about? Text? Photos? Spreadsheets? Photos aren't documents. Spreadsheets tend to be proprietary.

      * What is the source of the documents? Are these currently printed out documents that need to be scanned back in? Are they currently digital, and in a particular file format? This! I tend to classify documents as "Primary Data" (Structured) and "freeform Data" (Human Readable)

      * What will people need to do with them when these documents are retrieved? Do they need to be ab

      • Photos aren't documents. Spreadsheets tend to be proprietary.

        Nonsense.

        Data needs to be organized by purpose (Record keeping = Primary / structured data) and Executive Summary Type data (human readable).

        It depends on what the data is, and what and how it's being used. There is no "correct" organization, and no "one true way" to deal with data. I would not recommend going around cramming documents into some set organization without understanding where the data is coming from and what people hope to do with it.

        Your organization may work for your purposes, within the constraints of the company or organization you work in. I've supported a lot of different types of companies over the years, and pe

  • "Best" in this case depends on your needs and resources, not on standards or common practices. Flexibility and "getting the job done" are more important than what everyone else prefers. Do what works for you.

  • I'd highly recommend leaving them in their original format, or if anything, converting them all to .pdf. Conversion is always fraught with danger, and you will be spending an awful lot of time getting to the know the intricacies of Microsoft Word if you go this route. Pdfs display equally nicely on every operating system, they archive very well, and almost every tool out there can read them - but while converting documents to this format usually works better than others, I'd still be very careful to watch

  • There are many premade document management systems. They generally will store their indices in a database format for quick searching. Why not store them in their native formats and leave it up to a document management system to handle it for you?
  • ...you can't beat bamboo strips. The oldest original versions of Lao Tzu's Tao Te Ching are written on rolls of bamboo strips. Not sure how they scan electronically, and you will have to keep your pet pandas away from them, but for document durability, you can't beat that format...
    • We carve 16 bit Unicode into stone slabs for long term backup storage.
    • by gstoddart ( 321705 ) on Thursday May 14, 2015 @12:32PM (#49691063) Homepage

      Nonsense, bamboo can't touch papyrus for longevity, and you don't need to worry about pandas.

      Damned bamboo shills.

      And don't anybody go suggesting cave paintings, it's a completely dead platform.

    • by Etherwalk ( 681268 ) on Thursday May 14, 2015 @01:55PM (#49691929)

      ...you can't beat bamboo strips. The oldest original versions of Lao Tzu's Tao Te Ching are written on rolls of bamboo strips. Not sure how they scan electronically, and you will have to keep your pet pandas away from them, but for document durability, you can't beat that format...

      Chisel it into stone tablets, then find an ignorant local. Set up a natural gas line to a nearby bush and hide behind a rock. Cub your hands to add a slight reverb effect and tell him to preach the chiselled word, then break the tablets and hide them in a box and trick nazis into looking at them.

  • And a pony too? (Score:4, Insightful)

    by gstoddart ( 321705 ) on Thursday May 14, 2015 @12:21PM (#49690879) Homepage

    Although the application will need to allow uploading of .docx, doc, .pdf, etc, I'd like to store the documents in a standard open format that will allow easy search, compression, rendering, etc. Which open document format is the best?

    Lets' see ... you want to allow uploading in a large number of formats .. you want to magically turn it into a universal format ... while retaining all of aspects of the original ... and will be easily maniuplated ... and you want it in an open, and documented format? And all for free?

    I want one of those too. And a Red Rider BB gun with a compass in the stock and this thing which tells time. And a new skateboard. And a pony.

    Honestly, you're asking for the holy grail of document management systems ... the universal, lossless document format.

    I'm not sure it exists. And I'm not sure companies like Microsoft or Adobe would allow it to exist.

  • by Phoenix Rising ( 28955 ) on Thursday May 14, 2015 @12:25PM (#49690945) Homepage

    On the conversion side... If you're taking in PDFs created by a layout/page design program, then you're not likely to get good satisfaction converting them and storing them as something other than PDFs. OTOH, if you're taking in a lot of documents created in an office suite, and they have collaborative notes, and you need to retain the documents for legal purposes, then converting them to PDF is going to lose data.

    On the future use side: PDFs are slower to render and search than most formats; they're harder to alter, but they're more reliably rendered than any other format. Office documents offer richer content and easy editing; their layout may vary depending on the output device (good and bad), and office document formats seem to change a bit more than other document types. HTML with CSS is good, and probably now stable enough that future clients will render something similar - but it's not PDF for reliable formatting, nor office docs for feature richness; editing tools for HTML aren't all that intent on preserving what came before. LaTeX is a reliable formatter wrapped around text-centric documents, but it's not something most people will be able to use and edit.

    Each document type has its reasons for being - you'll need to decide why you need to store your documents and what you need them for in the future. Retaining the original document along with a text conversion stored and indexed in a search engine may be your best bet - or not.

    • Sometimes you people make things WAY too complicated.

      In our 'best judgement' -- what's a very open standard for documents? Now, we can ask "what type of document" -- and we can also try and answer for whatever documents we know.

      So here goes;

      Documents; Try RTFD. Rich Text Formatted Document. It might not be perfect in layout -- but it's open, and accessible to a lot of apps and cross platform. If you get bad results, you might just need to switch to some other "open" app. OpenOffice on all platforms will lik

  • Docbook? (Score:4, Funny)

    by Enry ( 630 ) <enry@@@wayga...net> on Thursday May 14, 2015 @12:25PM (#49690949) Journal

    Docbook allows you to separate out the content from the presentation. You write in XML and define paragraphs, chapters, images, etc. and then leave it to the various stylesheets to drive how it looks like when it comes out the other end - PDF, HTML, Word, whatever, and the stylesheet makes sure that if some features are supported (hyperlinking from the table of contents to the chapter) it'll be included in there. Since the content is in plain 'ol XML you can use any kind of XML processor to go through it..

    • OK, grandpa, it's time for your meds again ... look, Matlock is about co start ... no, they're not on your lawn. ;-) [ Wow, and actual 3-digit id ]

      Honestly, for those of us old enough to still have a copy of Goldfarb's [wikipedia.org] book, this has been the holy grail for a very long time.

      But in practice, there's still no tools to convert all those formats to it, and most anything you do is going to be custom code.

      As a system which takes other formats as input, docbook falls into the category of wishful thinking.

      Even us

      • But in practice, there's still no tools to convert all those formats to it, and most anything you do is going to be custom code.

        I love DocBook. But I've also spent a fair portion of my lifespan persuading other formats to turn into it. Not the most fun I've ever had, to say the least.

  • There is no "best" document format, open or otherwise, for "easy search, compression, rendering, etc." because those words are too fuzzy.

    • What does rendering mean (print, screen, mobile, or ...)?
    • What is the search scope ("this" document or multiple documents)?
    • How important is compression?

    If your use case is a typical one, then you actually want, for maximum search functionality, text (perhaps with some form of markup so you can assign weights to segments, like higher weight for titles), a HTML5 based

  • Will this be for sharing information with 3rd parties? If your documents consist of a set of data you will need access to EDI [wikipedia.org] (electronic data interchange) was designed to store the data in a standard format and be able to inject that data into any document format. It's not so helpful if you are just archiving word documents or emails. There are a number of companies [covalentworks.com] that assist in converting your documents to EDI.
  • by pahles ( 701275 ) on Thursday May 14, 2015 @12:59PM (#49691385)
    Markdown? It's easy to write, read, render, compress, search.
  • Your documents will lose formating when the files are converted, if you want users to be able to download the files in any format you should just store the files in the way that the user uploaded them and convert directly. Create a metadata plain text version for search, maybe a visualization version so that the user be able to see the files inside your application, in this visualization version you should just use the easiest method.

    Of course this depends heavily on your requirements.

  • Just convert all the documents into 1200 DPI, 32-bit PNG images.

  • DNA

    Millions of years of field testing, and it still mostly works, and DNA itself is not patented.

  • Who is going to use these stored documents? How will they be used (read-only, revise and check in, etc.)? What tools are authors generating these documents with? Answers to these questions will help determine the best storage format.

    For documents intended to be downloaded and read or string searched, PDF is a good choice. There are a lot of PDF readers for different O/Ss available.

  • by BrendaEM ( 871664 ) on Thursday May 14, 2015 @03:07PM (#49692603) Homepage

    I've written several books. Because ODT's have standard compression, they are usually much smaller. For a 109,683 word book, with styles and formatting:
    ODT: 271,090 bytes
    Docx: 300,057 bytes
    Word 97: 1,379,328 bytes
    PDF: 1,050,788 bytes

    If bytes cost money to store ODT rules.

    Imagine yourself sticking with Word 97 because it's a reliable standard: imagine buying three times as much storage, as well as the backup for the storage,

  • by David_Hart ( 1184661 ) on Thursday May 14, 2015 @04:15PM (#49693333)

    No, just no....

    Store the documents in their original format.

    There are many possible reasons why you shouldn't mess with the originals such as formatting, legal implications, loss of content because one format supports stuff that the other doesn't, etc.

    The only way that I could see this working is if you converted everything to an open format but kept copies of the originals and linked to them. But if the plan is to dump the original documents, then it just isn't worth it....

  • If you can't read it in flat text, it's not long-term reliable documentatoin.

  • i usually just go with the .TXT format [wikipedia.org] but have been considering the compatible .RST format [wikipedia.org].
  • The answer's right there in the question...

Never test for an error condition you don't know how to handle. -- Steinbach

Working...