New York and Minnesota Publish Open Document Studies

Multiple readers have written to point out that New York and Minnesota have reached the end of their lengthy deliberations on open document formats. Both reports agree that an open format would be beneficial, but neither were willing to endorse a particular choice. New York's executive summary notes, "The State Legislature should not mandate in statute the use of any specific document creation and preservation technologies, as technologies can easily become outdated." Minnesota's report claims, "The marketplace is still in flux, and it is not certain that a single standard will emerge." In related news, yesterday's announcement from Microsoft that they would provide support for ODF in a future update to Office 2007 has EU antitrust investigators optimistic, but cautious. Microsoft has said that the ISO process was what prevented OOXML from receiving support in the same time frame.

Goes to show

Outdated? Really? Did they even bother to ask anyone that knows anything about opensource? Open standards means that even if it does become outdated there can still be an implementation no matter what. It's the best insurance against having outdated formats. I guess I'm off to write a few paper letters.

Outdated laws are a problem

Legislation is difficult to change once passed. Competing interests (or lack of interests) and simple inertia mean that whatever gets written into law stays there for a while. It makes sense for the law to say that the standards should be open, that they should be chosen by a particular state agency, or that they should be reviewed every X years. But writing the choice of standards into law is very inflexible. There is a reason why we have building codes and highway codes. This doesn't say that administrative rulemaking is less subject to lobbying and corruption, or that it is more transparent. But on these counts it is no worse than primary legislation.

Re:Outdated laws are a problem

I agree with you that they shouldn't state in laws that a certain standard should be used, just that they be open. It was just their wording as to the reason that I don't particularly like. Anyway, when laws get passed stating that the standards for government documents must be open are there also penalties for non-compliance? These documents are important and the best way to ensure that they survive the test of time is to make the standards open and free to download by anyone.

Re:Outdated laws are a problem

I agree with you that they shouldn't state in laws that a certain standard should be used, just that they be open.

The common way to do it is to have legislation which refers to a standard managed by an external agency. The standard can then be changed without requiring legislative change.

This works well in fields like safety, where OHS laws can reference equipment like fall prevention harnesses, and still allow manufacturers the opportunity to innovate in their products.

It will fail in an arena where the resident monopolist is willing and able to trample standards bodies in order to perpetuate its monopoly. Until the monopolist is unseated, or demonstrably changes its ways, more specific legislation, such as mandating a particular format will be needed.

Re:Goes to show

Open standards means that even if it does become outdated there can still be an implementation no matter what. It's the best insurance against having outdated formats.

Huge difference between restrictions related to a specific open standard and open standards in general. Creating a mandate for open standards in general in relation to government documents would be the best choice as opposed to any specific choice. If a standard is truely open then any document creating software can get the code written to read and output to the open standard. Newer but still open standards could be adopted under a wide mandate without having to change the mandate, but they should be provably open before adoption.

Then as you said it would be insured to be readable later, of course with one additional caveat, the documents should be stored on media with open source file structures as well, else that insurance could be blocked by the inability to read the old file structure. Old documents might be required to be moved to newer storage devices as well but as time passes on the process could become increasing expensive. How many former modes of storage hardware are no longer in use or at least no longer in wide spread use? However if such a requirement is made, then look for Microsoft and perhaps others to fight it as hard as they can.

Yes, outdated.

As in functionally outdated. What happens when active documents become the norm? (One could argue that with AJAX and magical PDF forms, they already are.) What happens when wikis are the expected way to receive complex information with cross-references? Another chance for content handling software houses to get their fingers sticky? In court, sometimes you don't appeal because you know you'll lose, and sometimes you don't appeal because you know winning would be worse than losing.

Damn that ISO

Yeah I'm sure it was the ISO process that prevented it from getting support, not the shoddy standard that is OOXML.

Re:Damn that ISO

They are talking about what's preventing *Microsoft* from supporting it in Office 2007. The problem for MS is that OOXML as submitted by MS, and OOXML as approved by ISO is not completely the same. There are minor incompatibilites, which would prevent Office 2007 SP n-1 from fully understanding documents written with Office 2007 SP n. Microsoft doesn't want this situation, they only want to introduce incompatibilities with new versions. At least with a new version, customers have learned to expect incompatible documents. So Office 14 will create OOXML documents that Office 2007 won't fully understand. Office 2007 SP n will not.

they should not choose a format

they should give a law saying that the document formats chosen by public entities should have Open Standards.

Justification for inaction

Really, the use of fear of z standard becoming outdated is just a justification for inaction. There is no answer to that in any sort of technical field. What can you say when technology will always improve, standards will always become outdated. Saying you shouldn't adopt a standard because it will become outdated is precisely akin to saying you shouldn't drive a car because it will eventually run out of gas. It's just a mask to allow them to justify to the public why they won't move forward.

If fear of a standard becoming obsolete is a reason for not adopting it, I'm curious as to how they justify any of their IT budget?

Re:Justification for inaction

What can you say when technology will always improve, standards will always become outdated. Saying you shouldn't adopt a standard because it will become outdated is precisely akin to saying you shouldn't drive a car because it will eventually run out of gas. It's just a mask to allow them to justify to the public why they won't move forward.

Actually there are different ways in which a standards could foreseeably become outdated. Only one of them is a valid reason for not adopting it.

The current version of the ODF standard will become outdated because it will be superseded by a new, improved version. That is no reason against implementing the current version of the ODF standard now, because the other players in the Marketplace (now including Microsoft) are doing the same, and when the new version comes out, your investment in support for the current version is not lost. Rather, it enables your program to continue using documents in that version of the format, and to interoperate with other implementations of that version of the format.

By contrast, it looks likely that IT reality will completely ignore ISO/IEC IS 29500:2008 (the ISO standard for OOXML) because Microsoft has said publicly that they're not going to bother implementing any of the changes until "Office 14", so for the foreseeable future there isn't going to be anyone to interoperate with via that format. And when "Office 14" comes out, it'll certainly introduce further changes to the storage format. Therefore, why would anyone want to implement the current ISO version of OOXML? Inaction regarding that is totally justified!

Clueless legislators...

Both reports agree that an open format would be beneficial, but neither were willing to endorse a particular choice.
The State Legislature should not mandate in statute the use of any specific document creation and preservation technologies , as technologies can easily become outdated.

Looks like Microsoft has effectively brainwashed these clueless legislators.

Formats and technologies are completely different things. ODF is an Open Format. Open Office is a technological implementation of this format. Microsoft has recently merely proclaimed that they will also be implementing ODF in the current version of Office.

OOXML claims to be an open ISO certified format. But as on date, there is no technological, compliant iomplementation... in fact, the specification is not yet documented, as required by ISO processes.

Why can't they simply legislate on ODF, and then go about choosing the ideal technological implementation of the same?

Re:Clueless legislators...

I think what the legislators were saying was

• We want open-standards
• Every 2/4/6/8 years we'll meet and decide what those standards are to be reviewed by a technical team.

Other good news....

• iso OOXML is not going to be implemented by Microsoft until 2010! So no rational government can use that.
• (oasis) odf 1.1 will be supported by microsoft office 2007 in early 2009
• there are numerous other ODF choices.
• OOo3/Write/Google Docs are going to give office a run for its money

its a good year for document freedom.
- Joel

Re:Clueless legislators...

You do know there isn't a fully compliant ODF implementation either right?

More spin.

This statement is misleading. Every file written by OpenOffice.org, KOffice or IBM Symmphony (to use common examples) is ODF compliant. The file may not require every tag in the full specification to describe the contents each application is capable of writing, but it will comply with the standard.

In other words, each application is fully compliant with the subset of the standard mandated by the application's content creation role.

By contrast, MS Office does NOT write compliant OOXML files at all.

Re:Clueless legislators...

Can OO.o, KOffice, or Symphony read _any_ valid ODF document properly and properly save any changes made to the document that ODF supports?

If not, they are correct in stating that there is no fully compliant ODF implementation.

Re:

Can OO.o, KOffice, or Symphony read _any_ valid ODF document properly and properly save any changes made to the document that ODF supports?

What you describe is an inescapable problem with computing in general. You can create a perfectly valid PNG that Photoshop will choke on, or an Excel spreadsheet that will not open in polynomial time. That doesn't mean that either of those applications a

Question: How does a format really get out of date

Take an XML format designed to represent a wordprocessed document. How exactly is that going to go out of date? Is there any good reason to believe that 100 years from now, someone won't be able read/parse the document or virtualize OO.org? Seems to me that Microsoft's Office formats only went out of date because they were deliberately changed from version to version. So while I can understand not specifying a single format, I don't understand the stated rationale.

Re:Question: How does a format really get out of d

Question: How does a format really get out of date

There are several ways in which a format can become outdated. Below I will list some. You will notice that from a strictly technical point-of-view they aren't even close to being show stoppers (ie. you could work around them probably in several way). However, should that outdated format be mandated for use by a law then the technical challenges and financial burdens may become serious hindrances in the least and actual show stoppers in the worst case. Changing a law will always be more difficult than changing just a standard.

An ad hoc list of how a format can become outdated (pardon the poor examples):
  1) An underlaying technology or medium becomes outdated. Example: 8-, 16-, and 32-bit integers. Another example: pre-web/xml EDI-formats.
  2) A superior (= more fit) competing technology is developed. Example: SGML vs XML.
  3) The intended use case of the format becomes outdated and/or irrelevant, which may happen due multitude of technical and non-technical reasons (the world does not stand still). Example: an early text-processing format that does not support hyperlinks or embedded pictures. Data formats for various deprecated ports.

Your particular question was about "an XML format designed to represent a wordprocessed document. How exactly is that going to go out of date?" Let's first note that in practical terms this is a format specifically designed for longevity. However, it fairly easy to imagine that a word processing format designed today does not allow for all important future use cases. Information about intention might be very important in mere twenty years, as AI and cognitive modeling applications might require it. The format might lack important security features that become necessary way you interface with data via a brain implant.

Re:Question: How does a format really get out of d

For a text based format one thing that can get out of date is charset, but then that is relatively easily converted and we also have Unicode now which should be good enough for a while to come.

Then there is of course the whole markup, that can easily get lost or forgotten in a 100 years. You still might be able to make some sense of the document, but getting an exact rendering of the document might become tricky, unless you also dig out a copy of OpenOffice and have an emulator that can run it.

In practical terms I think there are mainly two dangers of ODF. First there is the Internet and ePaper, both of which will sooner or later make paper obsolete. Why print something, when you can just email it around? And why print it, when the screen you are having already reads as easy as paper? So you might simply end up using other formats then ODF far more often and no longer end up having a copy of OpenOffice around in a few years. The second and bigger danger I think are however subtle improvements in the format. When ODF gets expanded and improved they might try to get backward compatibility going, but you likely never reach 100%. So you will have tiny little differences and when ODF1.1 is no longer used, since everybody is using ODF5.0, software might no longer support ODF1.1 properly either. It might still render, but the output might be wrong in the details. This would be basically what we have today already, you won't have much of a problem getting an old Word document to open, but you will have a very big issue if you want to get the exact rendering that the Word version it was created with produced.

Composition of those states' panels?

It appears that Microsoft is shifting its competitive strategy with regard to ODF from rejecting the ODF standard to fighting for control of the standard in the committees. Below is a statement from James Plamondon, Technical Evangelist, Microsoft Developer Relations Group [boycottnovell.com], describing his view of how to leverage Microsoft's influence over committees. One wonders to what extend Microsoft might have used these tactics in New York and Minnesota:

have mentioned before the "stacked panel." Panel discussions naturally favor alliances of relatively weak partners -- our usual opposition. For example, an "unbiased" panel on OLE vs. OpenDoc would contain representatives of the backers of OLE (Microsoft) and the Backers of OpenDoc (Apple, IBM, Novell, WordPerfect, OMG, etc.). Thus, we find ourselves outnumbered in almost every "naturally occurring" panel debate.

A stacked panel, on the other hand, is like a stacked deck: it is packed with people who, on the face of things, should be neutral, but who are in fact strong supporters of our technology. The key to stacking a panel is being able to choose the moderator. Most conference organizers allow the moderator to select die panel, so if you can pick the moderator, you win. Since you can't expect representatives of our competitors to speak on your behalf, you have to get the moderator to agree to having only "independent ISVs" on the panel. No one from Microsoft or any other formal backer of the competing technologies would be allowed -just ISVs who have to use this stuff in the "real world." Sounds marvellously independent doesn't it? In feet, it allows us to stack the panel with ISVs that back our cause. Thus, the "independent" panel ends up telling the audience that our technology beats the others hands down. Get the press to cover this panel, and you've got a major win on your hands.

It bears scrutiny as to who was on the New York and Minnesota panels, and their affiliations and histories. I don't actually know of the composition of those panels, so I am posing a question, not making a statement.

No implementations of Microsoft's OOXML

It is important to remember that there are no implementations in the real world of Microsoft's OOXML schema, as discussed in Andy Updegrove's blog, quoting David Worthington:

http://www.consortiuminfo.org/standardsblog/article.php?story=20080521092930864 [consortiuminfo.org]

Worthington's story includes quotes from Matusow and Mahugh that provide intriguing insights into how the decisions were made. After noting that saving to the OASIS ODF 1.1 format would now be possible, Worthington writes:

However, the company is not quick to embrace its own creation. Mahugh stated that Microsoft would not implement the final ISO version of OOXML until Office 14 ships at an unstated date in the future. This variant of OOXML was designated ISO/IEC 29500 at the time it was certified as an ISO International standard in April.

"One way to look at it is the prioritization of formats," Mahugh explained. "We reach a point in time where we have to decide whether to continue to invest in a previous version [of Office] or to cut the cord and move forward."

ODF support was a priority for Microsoft, Mahugh noted, adding that "real world" customers say that there is a pressing need for PDF [AU: ODF?] support. "At this point there are no products using [ISO/IEC 29500] in the marketplace."

When will Microsoft support its own file format? Worthingon quotes Gartner Research's Michael Silver on that question as follows:

"Customers that are expecting true document fidelity from XML-based, ISO-standard document formats will continue to be disappointed." Silver observed that the most compatible formats to use today are Microsoft's legacy binaries, and he believes that Microsoft will be unlikely to convince customers to move to OOXML in the foreseeable future.

Embrace Extend Extinguish in 3..2..1..

Minnesota's report claims, "The marketplace is still in flux, and it is not certain that a single standard will emerge."

Well, it looks like we standardized on MS-ODF. It has all the benefits of ODF, but with new, undocumented features that we all can't live without, such as uh, ribbontables.NET. We may not be able to read this format in 20 years by a non Microsoft application, but according to independent studies funded by Microsoft, our productivity will increase 150% in the meantime.

Executive summaries compared

The report from NY has a more thorough treatment of the issues than the report from MN. It makes a strong case for openness as a policy, not a technical choice; shows how it is just one of many such choices, not all of which can be equally satisfied in every situation; and then very ably lays out a recommendation for making sure that openness gets pushed down into the state agencies.

The report from MN is focused on relating the wide variety of opinion that exists in this area, and not on making a specific recommendation (other than the commonsense one that the market is in flux and that the legislature shouldn't be picking market winners and losers.) The report gave me a much better understanding of just how confusing it can be when people try to talk about this issue. Like many complex topics, one needs to almost insist on agreements about terminology and scope even before engaging in the real discussion.

multiple formats increases complexity and costs

This is an important statement, #9 from NY's "Key Findings", cutting through all the fog and stating unequivocally the obvious:

Increased numbers of formats for doing the same office tasks do not increase choice in any positive manner. Use of multiple formats increases complexity and ongoing costs. The use of single, standardized formats increases efficiencies and furthers compatibility and interoperability. Choice comes into play in two ways: (a) the choices made by vendors to directly support accepted standards; and (b) the ability of the State to choose among vendors who support accepted standards.

Re:A little suspicious...

Microsoft has already said they will only implement ODF 1.1 when at the same time all competitors will produce ODF 1.2 documents.

Thus Microsoft will not be able to read many ODF dokuments produced because of imcompatibilities.

Its Microsoft making a imcompatible version of Java all over again to stop Java from being adopted. Now they are doing it to stop ODF from being adopted.