Next Windows to Have New Filesystem 1008
ocipio writes: "Microsoft is currently planning a new filesystem. Its planned that the new filesystem will make searches easier, faster, and more reliable. Windows will also be less likely to break, and easier to fix when it does. The new technology will cause practically all Microsoft products to be rewritten to take advantage of it. Called Object File System, OFS will be found in the next major Windows release, codenamed Longhorn. More information can be found here at CNET."
Predictions (Score:3, Insightful)
Yes, I'm cynical. But really, why shouldn't I?
:Peter
Re:Predictions (Score:5, Insightful)
Think SQL Server 2003 = OFS
Not only do they get their new FS with nifty new features (DRM yada...yada...) but imagine this scenario...
MS Sales Rep: "You need a database?"
Potential Sucker^H^H^H^H^H^HCustomer: "We're looking at Oracle."
Rep: "Oracle's OK and all, but you know...the TCO is over the top. With Windows NG [Next Generation], a top-notch, state-of-the-art DB is included for free."
Customer: "Really? Hmm...well, if it's already in there, I might as well use it."
Don't knock it, it worked with IE.
Re:Predictions (Score:3, Insightful)
Re:Predictions (Score:3, Informative)
I think:
- enough people who know a thing or two more about this than me are talking about it
- now MS is talking about creating an Object File System
- and we know SQL Server research has been going into making it handle less structured data more effectively
I think it all points to credibility. That's not to say there won't be some add-on for additional SQL Server functionality, but I think it's believable that the core functionality of storing and retrieving objects in OFS will come from SQL Server.Re:Predictions (Score:5, Informative)
Re:Predictions (Score:3, Informative)
I think you are right, at least in part. I think that the other issue to consider is the close relationship between WNT and VMS.
The last major release of VMS with a major improvement was back in about '91 when they added in the low level support for transactions in the file system. This was a truly major breakthrough. it is a pity that it came towards the end of the VMS era and as a result was never really exploited by applications.
The advantage of a transaction based file system is that you can write updates out to disk without having to worry about partial completion. You know that the write will either succeed completely or fail completely. It is not possible for the program to crash in mid write and leave the data in a corrupted state.
This would be a major advantage for application programmers. You could get ACID properties without having to mess arround changing your data structures to use SQL. You would not need to deal with the twits who think that Entity Relationships are the latest thing in comp sci rather than what they really are, an obsolete throwback from the COBOL era. You would not need UML or any of that rubbish either. And you can hand your Oracle DBA their pink slip.
All in all a win win situation, unless that is you are an Oracle DBA or hold Oracle stock.
Re:Predictions (Score:5, Funny)
Oh wait, that email was 7 years old.
Re:Predictions (Score:4, Informative)
No, NTFS was not a rip-off of OS/2's HPFS. It was an update. Microsoft wrote HPFS (and did a good job), and updated it to NTFS. IIRC most of the updates just made it incompatable with HPFS, but there were probably one or two things done better in their defense.
Re:Predictions (Score:3, Informative)
I think the major change from HPFS to NTFS was that HPFS was only capable of locking down permssions to the directory level, and NTFS could do it to the file level.
Re:Predictions copy management (Score:3, Insightful)
An administrator (sysadmin, root, whatever) should never be denied access to anything ever, no way, no how, zip, zilch, nada.
Without full access to the machine, and every resource on it, it is impossible to properly administrate it.
"Permission Denied" shouldn't even exist for the administrator account.
They just discovered... (Score:2)
Now that would be a substantial improvement.
Re:They just discovered... (Score:5, Informative)
NTFS has features like ACLs, streams, etc that aren't in FFS or UFS. Also, support for transparent compression and encryption, also sparse files. There's support for quotas in the filesystem, and it's quite resistant to the effects of fragmentation. It's journalled and supports Unicode. It's actually a very good filesystem, once of the better parts of NT.
Re:They just discovered... (Score:4, Insightful)
Right. This begs the question of why bother ?
The push et al is just a load of hype to push the upgrade path. They are going to engineer a database into their file system "to make searches faster" because doing it the slocate way would not force another round of complete system upgrades on consumers.
You may have also noticed that Outlook and Office will need to be rewritten to "take advantage" of the new file system. So not only will they leverage OS upgrades, but Office upgrades as well. They are planning to rip out a perfectly good file system (which is called "antiquated" in the article) to make billions of dollars, and the press releases are all about consumer benefit.
And consumer benefit, as you have noted, is essentially nil.
Re:They just discovered... (Score:3, Informative)
BeFS was/is essentialy a database, which in conjunction with features like extended attributes will allow you to store your mail as files in a folder (actually any folder). The same can be done w/ mp3:s where you can store the ID3 tag as file attributes.
All this makes for lightning fast search of mail, mp3:s etc because it's all in the FS, there is no need for looking at the actually contents of a file.
So now, do not say a database as a file system will not be to any use for the consumer...
And, oh, ReiserFS seems to be heading this way too...
Re:They just discovered... (Score:3, Interesting)
Personally, I think this is the right direction to go; just like Reiser's papers on "next-generation" versions of ReiserFS (with keyword and metadata searching built-in), along with the extensive work being done of VFS abstractions for almost every OS. Hell, you could go back to Be's attribute indexing on BeFS, or even the old MacOS resource fork.
Basic UNIX filesystem trees are far from the last word in this area; if they were, no one would need MySQL to support all their webapps. (Actually, it would be interesting to get the MySQL guys involved in some filesystem design work; I'm sure they would have an interesting perspective to offer.)
Seems like I've heard this before... (Score:3, Funny)
Re:Seems like I've heard this before... (Score:2, Funny)
Other problems (Score:4, Funny)
More "security by obscurity" no doubt (Score:2, Interesting)
figuring out most of (the undocumented) NTFS. They don't want Linux,
*BSD, etc. to be able to read and write their filesystem easily, as that
would make it easier for people to dual-boot and/or migrate away from
Microsoft operating systems.
Re:More "security by obscurity" no doubt (Score:3, Insightful)
This will happen...maybe. (Score:3, Interesting)
In other words, it sounds cool. I'll believe it when I see it. (and only at that point judge whether it really makes Windows less likely to break)
Can you say DRM? (Score:3, Insightful)
To be honest, NTFS seems to be a tip-top file system to me. The only thing I can imgaine it missing is hardcore digital rights management (cant wait).
What a clever way to force DRM down every consumers throat: break every single windows program created prior to OFS.
fuckers.
I could be wrong but... (Score:2, Insightful)
BeFS (Score:4, Interesting)
well, of course! (Score:3, Funny)
:)
hawk
OFS? (Score:2, Funny)
*snort*
This is just because people finally figured out their so-secret NTFS.
'entertainment pack 1' also included (Score:2, Funny)
DB tech? OO or Relational? (Score:3, Interesting)
Replacing its antiquated file system with modern database technology...
Now, if you were going to base a file system on a DB, what would you use? An Object-Oriented DB? Where organization is key (which you want for a file system), or a Relational DB for speed (which is why they are claiming to switch)?
I'm sure they are going to make a custom system, yes, but wouldn't it have to be based on one of the two major DB designs?
For the record, I'm no DB Admin, but, as I understand it, relational is the choice of DB for almost all projects for its sheer speed, OO is only good for academic reasons to show off organization...
Metadata (Score:5, Informative)
The GNOME project has an excellent overview of some of the issues with metatata [cygnus.com] in general.
Of course, this will mean a whole new found of application incompatibilities on Windows and a whole new round of reverse engineering to determine the filesystem and metadata layout.
Re:Metadata (Score:5, Interesting)
Unfortunately, Microsoft has exactly the wrong platform to implement these ideas on. The whole motivation behind this kind of thing is to simplify the software. Microsoft needs to be backwards compatible with 20+ years of cruft, and they have an abysmal record for writing clean, simple APIs.
This will probably end up being just another popular software engineering idea that ends up being superceded by new business plans later on. It will become yet another ossified layer in the lower sediment of their future OSes (see DCOM, etc.).
Re:SQL Server _is_ the FS - Re:Metadata (Score:4, Informative)
Not if your database server has its own datastore functionality built in -- and many do. You know how Oracle can work with raw partitions directly? Think 'bout that kind of functionality, just done by MS.
Re:SQL Server _is_ the FS - Re:Metadata (Score:3, Funny)
I'd rather not. I barely tolerate vfat only because I dual boot for games.
Re:Metadata (Score:3, Insightful)
Microsoft is proposing just such a scheme-- they will control the filesystem. If applications access the filesystem through the proper API/system calls the OS can ensure that the file metadata will be kept in sync. (I.e. they will have required arguments to the API to provide input for the metadata.)
To take one of your questions as an example:
Suppose there is a metadata field for "last accessed time". The HTTP server opens the file for reading using a fictional Windows system call called "open(*FILE)". Windows then internally updates the "last accessed time" metadata field and opens the file for reading. In this simple case the OS does all the work.Suppose there is a new metadata field for "last accessed from ". In this case the "open(*FILE)" call would need a new parameter (or some other way to pass in metadata like "open_new(*FILE, *metadata_struct)") so that the HTTP server can feed in the server that accessed the file. For backward compatibility the default might be the local server if the old system call/API is used.
Of course, this is all still vaporware. We'll just have to see what really happens.
Re:What's wrong with implementing file(1) internal (Score:3, Informative)
I can see where it would be advisable to have your file manager or desktop GUI be able to update the magic(5) file by being told that one or more of the files are different and that better signatures need to be generated -- eg, find out what's the same about files a and b but isn't the same as c and d.
Anything is naturally going to have some glitches, and some choices will always need to have an arbitrary definition since the definition of what they are may vary depending on interpretation.
I just think that the overwhelming majority of files *will* be machine-typable based on contents with hints and that adding a lot of extra data to the filesystem will cripple its speed in the long run.
The rift finally breaks wide open (Score:2)
Windows: Everything is an object
I guess it all comes down to whether they can make the surrounding environment rich enough that people can do everything they can do with objects as easily as they can with files (without writing a program).
Re:The rift finally breaks wide open (Score:3, Insightful)
If only Windows Scripting Host gave you a more dead-easy way to script/tinker with the Windows objects...
Haven't we heard this before ? (Score:2, Insightful)
Windows will also be less likely to break, and easier to fix when it does.
Doesn't MS say this about all the new versions of thier products ? Not that Windows hasn't improved, it certainly has, but they also never seem to live up to the hype.
Veritas? (Score:4, Informative)
integrated built-in volume management (dynamic disks) with some abstract layer to maintain the clunky drive letter schemes.
I dont' really see the reasoning by mentioning that all Windows applications will require a rewrite; they only need to abstract the Win32::File APIs to handle the internal OFS changes... the changes that they document appear to do essentially what the Indexing services do for win2000 and winXP now. I assume it is just extra metadata strings that they associate with each file inode.
It seems a bit arrogant of MS to think they can improve (or trash and rewrite) what is essentially Veritas' domain, file systems. But then again, we're talking about MS...
Re:Veritas? (Score:3, Insightful)
The rough idea I got was that they want to make the file system a giant database, though. This would be a vast departure from NTFS, FAT32, or any other file system used today. They're saying "instead of creating database files on a hard drive, each for a specific application - and then creating all of these independent files and folders for the applications themselves, why not dump *everything* into one large database that *is* the file system?"
I would think that they wouldn't *have* to rewrite apps like Office in this scenario, but they'd *want* to - to take advantage of the new functionality possible with such a "database as filesystem" concept. Without a code rewrite, the Office apps wouldn't be able to import content via advanced search features. (EG. Import all photos on my drive related to the company I'm writing my letter to, above, and let me browse these thumbnails so I can find the ones I need.)
Wow! This would mean (Score:4, Insightful)
I'll pass. I may be running (pre-installed) XP on my Dell but I'm still using Office 97. Why?
BECAUSE IT WORKS JUST FINE.
I don't need to "upgrade" to something even more bloated and bug ridden.
How do you figure? (Score:3, Insightful)
Ummm...Why?
Changing the FS would really only affect the way the data is stored on the drive, the AppFS interface should be abstracted by the OS.
C-X C-S
Re:Wow! This would mean (Score:4, Funny)
BECAUSE IT WORKS JUST FINE.
What are you saying, Citizen? You're not meeting your consumption quota? You're not upgrading when Central Planning issues Upgrade Orders?
You're denying Microsoft their much-deserved Software Revenue! You're denying the Government their much-deserved Tax Revenue! How unAmerican! General Gates would not be pleased!
You must upgrade immediately. Our helpful Upgrade Enforcers will assist you in all aspects (or will tie you to your chair if you are uncooperative). The new version of Office will be glorious, Citizen!
(You know, the Communists had it all wrong. The best way to manage society is through central planning of Consumption!)
Re:Wow! This would mean (Score:3, Funny)
You're going to destroy capitalism!
I'll even bet you complained about the DMCA too didn't you?!?! And the All American (TM) RIAA, and the Apple Pie (TM) MPAA! Did you know you're EVIL - we have ways to deal with you!a
I'll just have to call in my thought policeman Herr Ashcroft and make sure that you're properly programmed. (That's part of that wonderful "for-the-children" legislation SSSCA - I mean I have to protect my IP, so you're required to be programmed to "respect" my IP
Just stand right there, the men in black will be here shortly - resistance is futile.
Cheers!
Re:Wow! This would mean (Score:4, Informative)
Largely due to the migration of much of the needed code into the core OS DLLs. If it's in the OS, the apps are smaller, and they load faster.
After all, wouldn't you rather allocate the memory once when the DLLs are loaded at startup...or repeatedly every time you run the app?
Re:Wow! This would mean (Score:3, Insightful)
Genius.
Re:Wow! This would mean (Score:4, Insightful)
If Microsoft ships the OS with HALF of the OFFICE CODE already EMBEDDED in SYSTEM DLLs, you still can't USE OFFICE without the other HALF...which is what you installed when you loaded Office XP.
That's why it LOADS faster, RUNS faster, and has SMALLER executables. The code for office, much like every other Microsoft product is being MIGRATED into the OS itself.
The OFS initiative will EMBED SQL Server INTO the OS itself.
Bye bye RDMBS competition.
Got a browser competing with you, embed IE into the OS. Got Citrix competing with you, embed terminal services into the OS. Got Oracle competing with you, embed the DB.
It's a proven successful tack and it makes sense.
My question (Score:3, Insightful)
This sounds familiar (Score:3, Informative)
I remember MS talking about this years ago, even before NT came out. They were calling it Cairo back then.
In the end, IIRC, the UI elements of the Cairo project were recycled into the Windows 95 shell and the Object File System concept disappeared entirely... until now, it seems.
Is no-one else disturbed at the short memories in the industry? I was at the launch of Visual Studio.NET in Ireland a few weeks back and there was a Microsoft goon waving a Tablet PC around his head like it was some completely new thing. I mentioned the Go Corp/ Windows for Pen Computing FUD from the early '90s to the guys I was there with and was met with blank stares.
yay. (Score:3, Funny)
What's it spell?
I also see my ad filter is working on the new
No built-in copy protection? (Score:5, Insightful)
Rather a good thing to know.
As long as they get rid of file extensions... (Score:4, Interesting)
Back to the original topic, I can't wait for an OFS. Just for my MP3's. Figuring out which folder hierarchy to use for genre/group/album/track is a pain. Let the file system group them for me.
Re:As long as they get rid of file extensions... (Score:3, Insightful)
right click on pretty much any file (win2k), select the summary tab, click advanced, fill in the author and revision number.
Many executable files also contain pre-filled version numbers on the version tab: \WINNT\explorer.exe has company name, internal name, language, original filename, product name and product version as well as file version, description and copyright notice.
But, yes, windows does still rely heavily on file extensions to determine type/opener. I don't mind that so much. It's easy to change a file from one format to another by changing its extension. It's I've had trouble with files on the mac that are bonafide jpegs, opening in simpletext (I didn't have PC exchange set up properly, and the file types werent set to PHSD (or whatever the magic number for photoshop is)).
Re:As long as they get rid of file extensions... (Score:5, Insightful)
First off, all kinds of things are already designed around the extension idea, redesigning everything won't work that easily. Also, users are used to the concept of extensions. MS is very much aware of this. If you take, say, Windows 98 and go to edit file type associations, the list is sorted by type description. This doesn't work, as the user is not likely to know the string attributed to the file type he is thinking of. For example, recently my fiancee wanted to change the default viewer for
Secondly, these extended attributes are not portable. Many widely used protocols would be unable to automatically notify client machine of this information, forcing the user on every file downloaded to set the type of the file manually. Sure you can embed them directly in the file, but who gets to dictate the format? I can bet you that MS would extend any standard to break compatibility with other systems if it existed. By tying in the reading of these extended attributes more tightly with the opening of files, you are inviting MS to come and make life harder on non-MS platforms, as well as technologies such as DRM to have more success...
Finally, what about performance? As it stands, a system based on
The system as it stands now works quite well. Windows explorer already works to "protect the user from himself" by not allowing renaming of extension on a file easily. We have an established, cross-platform standard for identifying file types, we don't need to blow that...
Re:As long as they get rid of file extensions... (Score:3, Insightful)
But the idea of the filesystem and files being basically one beast -- that's scary. Reminds me of a compressed volume file a la DoubleSpace/Stacker.
I'd expect the minimal effect would be a complete lack of access to my data except thru "approved" channels.
Worst case, when the OS does go titsup, my data goes with it (a la DoubleSpace), rather than being left in messy but recoverable files per older filesystems.
This may be great for big databases (billions of records and up) but I'll stick to discretely accessable files for my own systems, thank you.
Re:Ill debunk that myth (Score:5, Informative)
The problem is that it's inflexible, and individual file types include their own formats for additional metadata (like id3 tags). Hate to break it to 'ya, but having a whole bunch of little incompatible metadata formats sucks. It sucks a lot. If I want to write a search tool that can look through files' metadata, I don't want to teach it one format for MP3s, one format for Word documents, one format for Joe's Accounting System datafiles... well, you get the point.
Files sharing names but with differing metadata are Bad. That you generally aren't allowed to have them is a Good Thing -- if the file is different enough to need different metadata (like foo.txt and foo.rtf), they should have different names (as foo.txt and foo.rtf do, incidentally -- how handy! -- but doing that for *all* metadata is inappropriate).
You've "debunked" exactly nothing.
Re:Ill debunk that myth (Score:3, Interesting)
I share your concerns about interoperability and the upgrade cycle. However, this move is a good thing. All the major OSes out there need to think about file systems as more than just a filename and data. This is because humans are capable of doing more than that, and we shouldn't be limited by what computers used to be able to do.
For example, think of a boring filing cabinet. While you can just create dividers and add more filing cabinets, humans do much more than that. For example, at my doctor's office, they color code each folder. In addition, they apply stickers to the tab on the folders for certain indicators (last update of the file by year, insurance type, which doctor I usually see, since there are several there, etc). As a result, simply by scanning the shelves, they can tell a number of things about the files without having to pull them out and open them up.
Same thing about the FS. It would be nice to be able to tell something about the file without having to issue the open call. That's a good thing. Currently, most apps limit themselves to one hint (the extension). What's wrong with more?
Everything you pointed out in your message, while valid questions, are mostly elementary engineering problems. For example, the two file/different extension problem can be solved a number of ways (MacOS already has to deal with this condition, for example).
In the future, it might be better to have RFCs for an new, standard FTP or whatever that allows a metadata section as part of the DATA transfer. This wouldn't be too hard either (HTTP already could get away with this, since you can define whatever headers, more or less, that you want).
The real concern is interoperability. I can imagine a "compatibility mode" for network aware services, like file sharing or FTP/Web, that present file names to the remote user in the old filename.extension format. That's actually almost trivial.
Namesys is already working on reiserfs (which does something similar). BeOS had something similar, too. NTFS already ran into part of the problem (which stream do you want). We shouldn't hold back because it might require some compatibility for a short period...
Realize that I'm a big skeptic when it comes to Microsoft... I'm worried that they won't do anything to help non-Windows interop. But as an idea, I'm all for the updated FS. To borrow your own phrasing, a file is a file, but I want more hints to help user applications like searching through files (how do you know a file contains an ID3 tag?)
But I like this concept.
Sujal
Yikes! (Score:4, Funny)
I hope they haven't discovered the unified file system, something which all the unixes use to store their data. We might be looking at the demise of unix!
oh wait, where do we put Progra~1 then?
- runs away to see if it's patentable
This has been planned for a while now ... (Score:3, Interesting)
... right before the sledgehammer hits.
Searching by content (Score:4, Funny)
You can achieve this goal by the following process:
1. Store all of your documents in a simple, text based format, and not in some overly complex propriatary format such as ".DOC", ".PDF", etc.
1. This text based format is known as "American Standard Code for Information Interchange" (AKA "ASCII")
2. If you require more complex presentation of information, you might want to use something called HyperText Markup Language. (Which doesn't do much markup these days... but I digress)
3. There is a program built in to Windows 98 and above called "Find" (usually accessed by hitting F3), and in other environments known as "grep" which can search by content.
Use the tools you have, you won't have to "upgrade" to the latest bugs, and the computer remains useful.
--Mike--
Re:Searching by content (Score:5, Funny)
Re:Searching by content (Score:3, Interesting)
Seriously, though: how are you going to search the content of one of those files, anyway? AFAIK, searching images for content is very rudimentary (try Google's Image Search feature, which is the best thing out there but still pretty bad), and searching audio or video....forget about it. The only moderately successful approach I've seen is the metadata that Fasttrack clients (KaZaA, Grokster, and formerly Morpheus) track, but I'm pretty sure all that has to be entered in by hand, and it's usually wildly inaccurate.
If you want text to be easily searchable, you're best off sticking with plain text. For binaries, the best scheme for now is probably some sort of embedded metadata scheme like ID3 tags for MP3's, but ultimately, that metadata has to be added manually (although you could store such metadata in a database like CDDB to automate metadata creation when ripping CDs, for example).
Re:Searching by content (Score:3, Informative)
As do TeX, LaTeX and various other formats that are stored in plain text.
Obsolescence (Score:3, Interesting)
It's been tried before (Score:5, Interesting)
In the end Be developed BFS which is basically a standard file system with support for indexes and attributes, an overall much better performing system with most of the benefits of an object orientated file system.
All Other Microsoft Bugs Have Been Fixed!!!! (Score:3, Funny)
According to what Microsoft had announced some time ago, they are HALTING further new development while they focus on bug fixes. Can I then take this as a sign they've fixed all the problems with Windows?
So now they are working on new developments again? I'm pretty sure existing Windows is not completely stabiized...I know mine isn't. So what does it mean? Micrsoft can't maintain focus? They can't tell the truth? What?
Specific tech info (Score:5, Informative)
Now for some speculation. The implementation is likely to be based on the same "pluggable" FS driver architecture first introduced in Windows 2000, where the NTFS and FAT32 drivers are just a layer on top of the kernel (you can actually buy a devkit from Microsoft that will allow you to implement, say, ReiserFS for Windows 200). This however poses an interesting question: do you make this newfangled FS to sit on top of tried-and-true NTFS, or do you implement it at the kernel level and make NTFS a layer on top of that? Either way, I think the article is overstating the devastating effect on existing apps. Microsoft is not about to shoot itself in the foot so massively. Whatever this ends up being it's a good bet it will be fully backwards compatible. Kinda like Win32, which can still run 16-bit apps, albeit slower (in 2K and XP). But this will make companies more likely to either port existing code or release newer versions that take advantage of the redesigned FS.
There's a interesting Register article here [theregister.co.uk]
I wonder about this stuff.... (Score:3, Interesting)
Now, my point is, I've actually thought about setting up some form of database so that my Wife can find her info for years to come. But my biggest question is NOT would a database help, I'm sure it would. What I would like to know, is how would the interface for that database look?
Considering what I have seen of XP (I got a copy sitting on a 2GB drive that sits on my shelf), MS knows very little about information management in the UI, and I would expect this problem to not get any better for the majority of PC uses, even if the entire file system was one big database.
The Point (Score:5, Interesting)
The underlying technology is to replace the NTFS filesystem driver with SQL Server, with a few tweaks. SQL Server already supports using a RAW partition as a data store, so essentially you just have to move the transaction log and descriptive info for the databases into a specific area of the disk. Add a little bit of bootstrap code to ntldr, and slap the SQL Server stuff into the startup driver list, and it's a done deal.
The next step is creating an NTFS compatibility layer -- it would allow you to mount tables as drive letters or network shares. A lot of the information wouldn't be useful when viewed in that fashion, but it would give you a way to run older programs.
Once all your data is in a common data store and can be manipulated as such, it opens up a world of new possibilities. The change will be long and slow; no need to kid about that. It will take years for all the 3rd party programs (and even Microsoft's own apps) to catch up and start taking full advantage of it. It's the same situation Plug & Play was in back in 1995; it sorta worked sometimes, but you couldn't really take full advantage of it. But here in 2002, you really can expect to grab a piece of hardware and slap it in your box without hassles. It took some time, but it eventually paid off.
But... are you having trouble, as I did, thinking of ways to make use of this common data store? Part of that comes from the fact that we've been conditioned and trained to think of data storage in terms of files; it's hard to shift gears... to think outside of the "filesystem" box so to speak.
For one thing, I could see someone emailing me a project. Not some word documents, an excel spreadsheet, and a database zipped into a ZIP file; they just email me the project. When I get it, and open the message, the project opens up presenting me with the various documents (linked to the database of phone numbers for example), and a little yellow stickynote window that has the project leader's actual email text. I didn't have to deal with unzipping the data, rearranging it, then opening the documents separately. Since the "rows" are linked, they open and act as a unit until I tell them to do otherwise.
It gets better though... imagine if I could run a query such as "SELECT f.*, s.filename FROM Folder1 f INNER JOIN folder2 s ON f.datetime = s.datetime"
It can get even more useful because you now have full SQL syntax available to you for manipulating the filesystem, with queries that are lightning fast. Throw in some Stored Procedures, Functions, Views, etc and I can see real possibilities.
Re:The Point (Score:5, Interesting)
Version control a-la VMS anyone? the FS is a full RDBMS, it could potentially store transaction history so that you could have multi-level undo at the FS level (eat that, Veritas).
Separation of file content from metadata? Sync your word file to your PocketPC and that device only gets the data it "knows" about (a "consumer" which only understands certain "interfaces", if it helps to think of this that way).
Virtual directories? screw the strict hierarchical view of the world. Directories can finally be SQL queries! I mean they are now, but they only depend on the filename and path. Imagine a directory that literally is the result of the query "all files that were sent to this customer in the last 2 months". Seamless.
Network transparency. I posted this in another comment. This pushes the windows object orientation down to the FS. Dot-NET pushes it up to the network. RDBMS can already support redundancy and clustering. Take that concept to the NFS/distributed computing level.
This is a huge technological leap forward. We've been working on super-powerful DBs for years, but we were limited to a stupid tree when it came to our own personal data management. This is big.
Baby, Bath water (Score:3, Insightful)
Don't throw the baby out with the bath water.
A "file" expresses a fundamentally useful idea: a clear demarcation of data that lives independent of the host filesystem. Once you start tieing and interweaving data tightly with the host filesystem, how do you export it without a significant, altering transformation?
That is, when someone "just emailed you the project", what did you get? How much of the filesystem did or didn't come along with it? Have we openned the door for Version Hell? Also, can the data be compressed without having to know that it is?
Let's just be careful to clearly define what we want and how we get it.
A "file" lets us abstract the data from the filesystem. It is then trivial for that data to live on Ext2, Ext3, FAT16, NTFS, Juliet, in a zip, in a tar, as an email attachment, or in a pipe to an arbitrary process.
With a "common data storage", it sounds like what is really wanted is for each "object" to emit a standard, common interface. Once everything has that interface, we can wrap a database system around it to transform the data in lots of unique, interesting ways. Is there something implicit about this new abstraction that it has to live in the filesystem instead of on it (Is-A versus Has-A inheritence)? Does it require that we to throw out other, existing, useful abstractions ("files") to get it?
It sounds like an equivalent solution is to encapsulate each file in a platform independent, self-describing data structure. Then, impose the database query system on top of that. That both maintains the separation between file and filesystem provides all the features of the "common data storage".
ReiserFS Already does this (Score:3, Interesting)
We've Heard About This Before... (Score:3, Interesting)
The submission form is acting weird so here is the link again: http://www.pbs.org/cringely/pulpit/pulpit20010802. html [pbs.org].
This New File system sounds to me like something similar.
Thank IBM, DB is the way to go for FS (Score:3, Insightful)
I'm personally glad MS is finally changing their OS. Now that my workstation has 70GB of files, searches are taking an incredibly long time.
I have less than 100,000 files on my workstation. Each has maybe 10 searchable attributes. A full search on this can take over five minutes. (Athlon 800Mhz w/ 7200 RPM IBM drives on a Promise controller)
I know from experience that querying an Oracle database (on a cheap 500mhz linux box) on 100,000 records with 30 non-indexed columns/attributes generally takes around 2-3 seconds.
Imagine if MS were able to build a file system with such capabilities.
Extending the Unix doctrin. (Score:3, Interesting)
But that was 30 years ago. Perhaps its time to extend the unix-doctrin: Everthing is a file and a directory.
Why? Metadata.
Todays file-formats store informations about the file inside the file (thing id3-tags) or abuse file-attributes (such as the filename(.html)).
With files as file and directory, there would be no need for that. Imagine: you store informations about the authors of a file inside metadata-attributes. There would be simple possibilities to search for these informations, so one could easy pick up all "draft" files inside a direcory (ls -al *../status=draft, maybe)
p.
Re:Extending the Unix doctrin. (Score:3, Informative)
And 30 years ago the bleeding-edge filesystems (non-UNIX) did have metadata and resource forks and other whizbang ideas. Guess what? They sucked. You can't pipe a multistream file. You can't send it over the network. You can't dump it to tape. First you have to convert your whiz-bang metadata enabled file into a "bag of bytes" file. This means you need to manually add an additional "convert to bag of bytes" step before you do anything unusual with a file. UNIX removes the complexity by forcing the applications to save into the "bag of bytes" file format from the start. The metadata is still within the file, but the application put it there, and the application knows how to get it out.
The UNIX people chose simplicity over complexity. History has shown that if people want metadata then they can implement it inside their applications (ID3 tags, TAR, GZIP headers). This is one of the many reasons why UNIX still survives: it dictates the means to store data, without enforcing a policy about what data is stored. Applications can then grow and evolve to include new-and-unthought-of metadata. This would not be possible if the UNIX filesystem had forced a restricted set of metadata onto everybody from the start.
People who don't learn history are doomed to repeat it.
Good. (Score:5, Interesting)
The idea of having to rewrite the apps is interesting though. That tells me this is at least 5 years off, and longer before it would be used widescale. But I guess that makes sense, would you be the first shop to put your big fileserver on a new filesystem like that? Not me.
reasonable article with some silliness mixed in (Score:3, Interesting)
In the process, the plan could boost Microsoft's high-profile .Net Web services plan and pave the way to enter new markets for document management and portal software, while simultaneously dealing a blow to competitors."
OK I know FAT is antiquated, but NTFS is modern. In fact I recall it was announced at some point 3-4 years ago that OFS wasn't necessary because all the relevant features were being merged into NTFS? Maybe that was an internal announcement, one of the annual "we are finally merging our data stores" emails the top Microsoft brass would send out to the troops.
Anyway I don't see why this would make Windows less likely to break or easier to fix, or what it has to do with .NET...why does that kind of marketing fluff have to be included in a pretty reasonable article (and the sidebar is very nice)?
- adam
Is Linux too busy catching-up to innovate? (Score:4, Interesting)
It is funny, we accuse Microsoft of using other people's ideas - but are we really any better? How much of Open Source development is really just reimplementations of other people's ideas?
Explain how this is a risk... (Score:3, Insightful)
I wonder how many R&D projects get canned in MS. Probably a whole lot. And considering building a new FS is nowhere near innovative. Are you suggesting that MS thought up the idea of this new type of FS? Puhleeze.
Re:Is Linux too busy catching-up to innovate? (Score:4, Insightful)
It is possible, there are examples of Open Source projects which really do innovative new things, but they are quite rare. Part of the reason they are so rare is that a developer needs a thick skin to not be disheartened by the countless numbers of people around the O.S community who would rather nit-pick other people's efforts than contribute themselves.
You know, they're right... (Score:5, Interesting)
Thought 1: "You know, they're right" Current file systems are outdated and are not really serving the needs of modern applications. Take for example, Microsoft Outlook (and Outlook Express). The programming teams for these pieces of software were forced to implement a "filesystem within a file" in order to achieve their design goals (I believe the files are called DBX files). Or take for instance, the Windows Registry, or, even better, the Gnome registry, GConf. Why do programmers have to implement dozens of different abstract filesystems in order to achieve their design goals? Simple, the present filesystems are not sufficient.
Thought 2: "Another way of attacking the Free Software Movement." By creating a new filesystem, Microsoft achieves many goals. First, they make Linux filesystem developers start from scratch again. I mean, the NTFS driver isn't even done, and this means we would have to start over. It gets even worse: From the sound of this article, it seems that OFS would be fundamentally incompatible with our conception of a filesystem today (possibly including features such as resource branches, GUID tags, and other metadata forks, ad nauseum). This would make it difficult to write a usable Linux driver for OFS. And finally, to top it off, my gut tells me that the POSIX file access calls would _not_ be sufficient to access such a rich filesystem. The introduction of a new, richer file access API by Microsoft would make writing cross-platform software much more difficult.
Microsoft can kill two birds with one stone here.
Ben
BeOS lives! (Score:3, Interesting)
If you take a look at the XP interface, it feels (to me at least) a lot like a candied up BeOS -- a lot of the icons have a similar look, there's the grouped taskbar items a la the BeOS tracker, etc. And seeing as BeOS has been around for years, it makes a lot more sense that the Microsoft engineers would have been able to start reimplementing ideas like this by this point.
And now we start seeing articles like this one, and it becomes clear that just as the XP interface has started to resemble BeOS, the XP native filesystem is starting to resemble BFS. This isn't the first time in recent months that we've seen reports of this -- not long ago there were articles saying that MS wanted to ditch Access and it's Jet engine (or whatever it runs now), and turn the SQLServer engine into the core of the next generation filesystem. This is of course exactly what Be wanted to do, but couldn't due to performance constraints, so they went with the scaled back object oriented system instead. Hey look at that, now we hear that Microsoft is also going with an OO-FS instead of a full SQL-FS.
Microsoft already ran Be out of the market, and are rightfully getting sued now for doing so. I wonder if Be would be willing to use this increasingly familiar evolution for Windows as evidence that Microsoft wanted to eliminate their strongest OS competition while ripping off all their good ideas. As much as it's vindicating to see that BeOS's best features will live on in new versions of Windows, I'd rather have the chance to see the original around today...
Registry Redux (Score:5, Insightful)
But as everyone knows, with totally undisaplined usage of the registry, the registry is a nightmare. In some cases it is impossible to clean it up and the only solution is a reinstall.
Ask any dba. Even with the most heavy duty industrial strength db, somebody can come up with a schema and application that will bring that db to its knees. Prepare for deja vu.
Too bad other things got in the way (Score:5, Interesting)
While the Cairo project eventually resulted in Microsoft's Windows 2000 operating system, the file system work was abandoned because of complexity, market forces and internal bickering. "It never went away. We just had other things that needed to be done," Jim Allchin, the group vice president in charge of Windows development, told News.com.
Those other things most likely included battling "Netscape and Java and the challenge of the Internet and the Department of Justice," Gartner Group analyst David Smith said--issues that continue to persist today.
<snip>
The more important reasons for the renewed development effort, however, are strategic. If the plan succeeds, it will give Microsoft a huge technological advantage over the competition by making its products more attractive to buyers and giving large companies another reason to install Windows-based servers.
So if they hadn't been trying so hard to kill off Netscape, they would have had the time to spend on creating this. Something that seems to offer actual advantages to the user, and that would be "a huge technological advantage over the competition by making its products more attractive to buyers."
I wonder how many other genuine advances have been put on hold in the name of detroying someone else first.
Clearing up misconceptions (Score:5, Informative)
A lot of people seem to've totally missed the point of what would be different about a database-oriented filesystem. File extensions? Not bloody well relevant! Let's consider the issue of searching. A database-oriented filesystem might allow you to create directories that are basically "views" of your filesystem, perhaps including all files that meet certain name, attribute or content criteria (like Evolution's vFolders but available to any app). These views would be up-to-the-instant accurate at all times, with no dead links and no problem with apps replacing links with actual files instead of updating the file that the link pointed to. Filesystems could also benefit from other things like referential-integrity checks, triggers, and cross-file transactional behavior. In fact, there has been a lot of work in the kernel-hacker community to figure out how just that last feature could be added to Linux filesystems. Basing a filesystem on a database also allows you to leverage all of the tools (e.g. efficient snapshots and replication) that have been developed for the database. There's a lot more here than just journaling and BeOS-style metadata.
It's not that I think basing a filesystem on a database is a great idea. For one thing, it's a pretty good bet that performance is going to suck because of all the extra DB-related overhead. Administration might become more of a PITA too. I'm just trying to explain that the idea of a database-oriented filesystem has much broader implications than the trivial crap (much of which is relevant to neither filesystems nor databases) that people seem to be focusing on in this thread so far.
Summary (Score:4, Funny)
All Slashdotters: Burn it! Burn it!
Rational observer: Well, how do you know it's so bad?
First Slashdotter: Well, it's from...MS!
All Slashdotters: Yeah! Yeah! Burn it! Yeah!
Re:funky fat32 tip! (Score:5, Informative)
Yes, moving the documents _will_ stop this process, but it's not the most practical method is it? Better, by a long, long way, to turn off automatic indexing of office documents.
Cheers
Chris
Re:funky fat32 tip! (Score:2)
It's really weird, and annoying, but it's worked on two of the Win98 boxes I've tried it on. (home & work).
Re:funky fat32 tip! (Score:4, Insightful)
Yes, I know you are not supposed to store things on the desktop, but windows makes it far too easy to do so. Plus it has the advantage that once you have downloaded a file, you can see it immediately without having to navigate to the right directory.
Re:OT: Refreshing! (Score:2, Interesting)
Re:OT: Refreshing! (Score:2)
My question is: am I allowed to bash them for recent past failings? I'm still doing major clean up, and since the entire registry was among those files that were forever lost, I'm still kind of bitter.
Re:OT: Refreshing! (Score:3, Insightful)
XP is an NT-based OS...why were you using FAT32 at all when NTFS is available?
C-X C-S
Re:OT: Refreshing! (Score:3, Insightful)
Look at it this way - some of us may wish that the whole world used Linux, but it doesn't. It uses Windows. So when MS announces that they're taking a big risk (and it is a big risk) to try and make such an enormous upgrade to Windows, I think we should be happy that a few years down the road, if Windows is still dominant then at least people will be benefiting from this technology.
But ... this doesn't mean we should just sit back and go - well done Microsoft! After all, I recall reading about something similar over at the reiserFS page... how long until Linux users get this technology also?
Re:You realize why they are doing this...right? (Score:5, Insightful)
Yes, you are a troll. Is it wrong for Microsoft to advance File systems and state specific reasons and right to preach about the many choices in file systems linux/unix has?
When your talking
Say when you open a folder of 5,000+ mp3's and it searches the ID3 tags of every song and displays artist/title as part of the description, having an optimized file system for quicker searches of data on the disk will only streamline this more.
so yes, this is cool, and yes, there is alot more then just "fast searches" as you put it.
I can see it now... (Score:3, Funny)
[user] Open Excel document "Personal_Finances"
[windows] Please log into Passport so the $0.50 file usage fee can be charged to your credit card.
[user] But this is my file, I created it.
[windows] NO, I store your crap file, it is mine.
[user] THAT's IT, I'm formatting you!!
[windows] Please log into Passport so the $999.99 reformatting fee can be charged to your credit card
[windows] And have a nice day!
Escalation (Score:5, Funny)
[windows] A priority alert has been dispatched to the BSA.
[Debian Boot Disk] So Boss, do ya want to just rough him up a little or completely murdalize the bum?
Someone had to say it... (Score:3, Funny)
Me: What happen?
Windows XP: Somebody set up us the upgrade.
Me: What !
Me: New operating system turn on !
Me: It's you !!
Windows NG (next generation): How are you gentlemen !!
Windows NG: All your file are belong to OFS.
Windows NG: You are on the way to .NET and DRM.
Me: What you say !!
Windows NG: Your mp3 have no chance to survive make your time
Windows NG: Have a nice day.
Me: Take off every boot disk
Me: Load Ranish Partition Manager
Me: Move Mandrake 12.0 install DVD
Me: For great justice
--
Re:You realize why they are doing this...right? (Score:5, Insightful)
Firstly, there is no such thing as "Microsoft Digital Rights Management Software" (Media Player supports DRM, but only for WMA's) and Microsoft has nothing what-so-ever to gain from including DRM features into the file system. They know and we know that Longhorn with DRM will go down the toilet, while Longhorn without DRM will sell just as well as WinXP, probably better.
The second thing you got wrong is that this system is not (just) about speeding up searches. It's about replacing an antiquated system that's been around since MS-DOS with something future-proof, faster, and more reliable. Considering they've been working on this for 10+ years, they'll probably succeed eventually. And when they do
Now, for something constructive. When will we see this in Linux? Surely, if Microsoft can do this, so can the people working on Linux. Riiight?
Re:You realize why they are doing this...right? (Score:5, Interesting)
> Management Software to infest every aspect of
> their OS as possible.
Right. You keep throwing your FUD about while the rest of us looks at things seriously.
> Do you honestly believe that the benifit of a
> faster search is enough incentive to rewrite
> such a major part of the OS?
Short Answer: Yes.
Long Answer: Filesystems haven't changed much in the past few decades, but one of the things they have tended to gain is arbitrary metadata. Adding indexing to that metadata is a natural progression of that.
Now your filenames are just a part of the metadata you'll want to play with different views of the system, which suddenly becomes much much cheaper. Believe it or not, lots of users have trouble understanding the current basic filesystem concepts and using them to organise their data; well, now you can do it automatically for them.
Of course, you want your other stuff to make use of these new ways of looking at the system, especially when you're MS and are running out of new features to put in (come on, what are they going to add to Word XP now? A paperclip with speech recognition? Yet another GUI redesign?), so you want to do something that provides a visible difference (and maybe even an advantage) for those expensive upgrade programmes.
So, yes, I think they do have a very good reason for such a major change, like they had good reason to introduce '95 and start dropping DOS, or NT and start dropping Win16, or
The biggest issue I have with it is that it's going to be a bitch to use in other OS's. Hopefully they'll do detailed specs and stick to them fairly closely (ah haha), which will at least make it easier.
Re:Cross platform compatibility (Score:3, Insightful)
Now if we were talking about Microsoft coming out with a new obfuscated replacement for SMB (which is an evil hack and needs to be replaced with something less thoroughly bletcherous anyway), then you might have a point. But we're not talking about that at all.