Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
×
United States Data Storage The Internet

What Kind of Data Center Can You Build With $500M? 204

coondoggie writes "So, if the government gave your company $500 million to spend on building a new data center what would you buy and how would you build it? Well, the Social Security Administration is about to find out. As part of the stimulus bill, or the American Recovery and Reinvestment Act of 2009, the SSA got the tidy little sum to replace its National Computer Center. The SSA in fact says it will need closer to $800 million to fund a new IT infrastructure, including the new data center — the physical building, power and cooling infrastructure, IT hardware, and systems applications. (This is addition to a $72 million backup facility currently under construction in Durham, North Carolina)."
This discussion has been archived. No new comments can be posted.

What Kind of Data Center Can You Build With $500M?

Comments Filter:
  • I will start with the assumption that this data center must be non-homogeneous. Get an assessment of all the projects that are using the current system you're going to replace (you know, the one with 36 million lines of COBOL code?). Because the number one priority of the customer (other projects) is going to be the lengthy transition from that to current technology. Prepare yourselves for this: Some of the projects aren't going to have any funding to do jackshit. Which means that the awesome spaghetti coded current system that's held together with COBOL duct tape needs to remain intact in some form. Not ideal situation but an uncomfortable truth. I'm thinking you would want to set aside 10% or $50 million or so for this (just throwing out a figure).
    • by Anonymous Coward on Thursday April 30, 2009 @07:14AM (#27770997)
      As someone who maintains (part of) that 36 million lines of COBOL code, it's not as spaghetti as you might think. Many parts of it are quite clean, they just haven't been touched in years but they're simple and they work. The problem is, we're facing a demographic time bomb. The folks who wrote the system got hired in a huge spree in the 60s and 70s, and are reaching retirement age with full government pensions. Lots of them retire and come back as contractors, but we're still losing a lot of them.
      • by jellomizer ( 103300 ) on Thursday April 30, 2009 @07:42AM (#27771247)

        I think more colleges as part of their Computer Science Elective they should be a class on Legacy System Maintenance. Legacy systems are not as bad as everyone makes them out to be. As well for the most part languages such as COBOL and FORTRAN are really not that hard to follow and learn. In many ways they are easier then the newer languages, as they are designed to do particular things and do them well, and not like C,C++,Java,.NET try to be the end all be all language. The old code is usually focused on the business logic while newer code seems to be working more with trying to get the formatting correct, and being well organized and modularized (which is a good thing too). But the old code you get PROCESS_CLAIM.CBL all the code that you need to process the claims are there, so when you fix the code there the problem is fixed. Having done a lot of work in FORTRAN myself I have found that there isn't that much spaghetti going on. Yes there is the GOTO statement but it is usually limited to ERROR cases where if something critical failes it GOTO ErrorNum and displays the error and quits the app.

        For these legacy apps normally when something needs to be changed is because there was a change in the business process not because of a bug.

        • As someone who maintains COBOL driven legacy systems, I disagree. Why teach that stuff in college? Those jobs are much more rare than other programming gigs, and they tend to be held by lifers (I inhereted mine when 2 people hit retirement age, and the third decided she didn't want to do it alone). If you DO get one of those jobs, the learning code for the obscure hand-coded systems is going to be vastly higher than the language.

          COBOL isn't that hard to pick up. Maintaining legacy crap code is the same whether it's VB or COBOL or RPG, and the vast vast majority of your headaches will come from environmental quirks (old school databases with fixed width data, packed binary decimal numbers that no one uses anymore, etc).

          The biggest problem with COBOL in the modern world is that its designed in reverse. It treats CPU cycles and RAM like they're the most precious things on earth, so a program will make live changes as it goes along (to conserve RAM and minimize disk IO), and is designed to fail in a dirty state (in the middle of everything, so you can't re-run it) on the chance that it'll preserve cycles. It's a real maintenance headache.

          • Learning curve...

            Coffee's sitting right in front of me, but it's too hot to drink...

          • by acvh ( 120205 )

            re: your "designed in reverse" observation. Right on. And not only that, but many organizations still use those metrics of cpu cycles and I/O to bill internal clients for running reports. Which explains why people here in my current office are afraid to request reports.

            • Heh. One of the systems I maintain is an old MPE/iX server. Real P.O.S, and it's from an era so far back that when you ran a job, what it usually output was a print queue file.

              This stuff is extremely difficult to explain to people. The system wasn't written to output files...Why would it be? What would users do with them? Their terminals didn't even have disk drives! It's only real output went to a printer!

              So now, people say things like, "Well can you give me this in excel?" and we say things like, "Well, I

          • by khallow ( 566160 )
            I disagree here. A number of counterarguments spring to mind. First, code maintenance is a key part of any successful software's lifecycle (and even some failures too). While jobs maintaining ancient legacy code are relatively rare, jobs upgrading from legacy code are probably much more common. A course in maintaining legacy code also would give a programmer some more appreciation both for writing maintainable code for future generations (after all, it is a demonstration that your code may still be around i
        • Re: (Score:2, Interesting)

          by mmaniaci ( 1200061 )

          NO! They already teach us enough superfluous crap to make me wish I was a business major.

          What colleges need to do is stop teaching history lessons and begin teaching students how to learn on their own. Face it, your job will not mimic what you learned in college no matter how good the school was, or how good of a student you were. Real life != school life.

          Don't give a man a fish. Don't teach a man to fish. Teach a man to learn how to fish.

        • Re: (Score:3, Interesting)

          Classic COBOL is something of a special case:

          No real user types. There were things you could do to fake this but its hard to maintain.

          Very limited procedure capability. There is a perform A through B capability (so the extent of the routine is defined at the call site! - but there is a way to avoid that using SECTIONs or internal labels) but there is no way to pass parameters - other than as global data. You can also write a subroutine as a separate compilation unit.

          Modifiable (at run time) GOTO statements.

      • Re: (Score:3, Funny)

        by MobyDisk ( 75490 )

        The problem is not that the COBOL coders are retiring. The problem is that the government made the decision to rewrite the code 20 years too late. Everybody else knew the language was dead.

        If this were a commercial company, they would go out of business for making such a stupid decision.

        • Re: (Score:2, Interesting)

          by AlecC ( 512609 )

          There are plenty of not-out-of business companies still running COBOL for good reason. Admittedly, many of them are banks who are now going out of business, but not because of their COBOL. Look how much money IBM's mainframe division is still quietly making: many of these are COBOL engines.

        • Re: (Score:3, Informative)

          Comment removed based on user account deletion
          • I will tell you from hard-fought experience that rewriting massive DP type applications that worked mostly perfectly in COBOL, RPG and PL/SQL to Java is not necessarily the easiest thing in the world. Too much involved in frameworks, and the asynchronous nature of the web. I've seen enough such projects fail.
        • If this were a commercial company, they would go out of business for making such a stupid decision.

          More likely they'd get a bailout.

        • Everybody* else knew the language was dead.**

          *everybody: the disturbingly large subset of programmers who are unaware that COBOL and FORTRAN still power the vast majority of the financial and healthcare infrastructures of the world.

          **dead: working quietly in the background without need of constant updates and maintenance

          • by MobyDisk ( 75490 )

            The original poster was talking how they had COBOL code to maintain, and no one to work on it.

            As someone who maintains (part of) that 36 million lines of COBOL code...

        • by bdenton42 ( 1313735 ) on Thursday April 30, 2009 @10:07AM (#27773157)

          There is nothing wrong with a functional 20 year out of date COBOL application. It is a bigger waste of company/governmental resources to rewrite it just because you want to use the language flavor of the month.

          So they should have rewritten it in 'C' 20 years ago, then of course you have to go OO and rewrite it to C++ 15 years ago, then Java was really cool so 10 years ago rewrite again, and now M$ has taken over the world so we better port it to .NET today, but you've just wasted 4 projects to simply get the existing functionality and your company is out of business (or added 0.0001% more to the national debt if you are the government).

          There are very good reasons to port old projects but just doing it because it's 20 years out of date is not one of them.

          • To some extent you are right. WHen you rewrite it there is a 50% chance you will stuff it up worse, and it costs a whole hell of a lot of money.

            On the other hand there could be competitive advantages and/or productivity advantages to moving from a batch COBOL world to an online, always-on, web enabled modern infrastructure. How many times have we rung some company or ggovernment department, and they couldn't do something half rational because their computer system was way out of date?

          • by MobyDisk ( 75490 )

            There is nothing wrong with a functional 20 year out of date COBOL application.

            Tell that to the manager who can't hire anyone because they refuse to work on COBOL code. Which, was the issue the original poster brought up.

            That is like saying "There's nothing wrong with leaving data on these 8-inch floppy disks! There are perfectly fine!" Until you find out one day that there's no hardware that can read them.

            There is something very wrong with leaving something in a language so old that no one is willing to maintain it. If they rewrote it in C 20 years ago, then they would have a poo

            • Tell that to the manager who can't hire anyone because they refuse to work on COBOL code. Which, was the issue the original poster brought up.

              I doubt that very much. I'd work on COBOL for the right price.

              That is like saying "There's nothing wrong with leaving data on these 8-inch floppy disks! There are perfectly fine!" Until you find out one day that there's no hardware that can read them.

              Easy enough to virtualize hardware.

    • by elrous0 ( 869638 ) *
      Screw all that complicated stuff. I have two words for them: gold toilets.
    • 1.) Word

      BIG IRON

  • by Anonymous Coward

    At $8.95/45 specimens (http://sciencekit.com/harvester-ants%2C-live/p/IG0034483/), the newly constructed Hex would have the most powerful neural net we've ever seen!

    But that wouldn't leave any money for the clacks...

  • by eldavojohn ( 898314 ) * <eldavojohnNO@SPAMgmail.com> on Thursday April 30, 2009 @07:06AM (#27770943) Journal
    Could you approach Google and ask them to license their ideas on server [slashdot.org] and data 'pod' [slashdot.org] technology for your sharded databases? I'm not saying build the whole thing like this but with $500 million, you could probably have a large section to search and sharded databases that mimics Google. I don't think there's anything wrong with following the leader in that department. This probably isn't the best solution for relational databases so I would think another architecture would be in place for your MySQL and Postgres traditional database layouts. And that would be just huge centralized servers running virtualized instances of Linux with MySQL or Postgres.
    • by Zemplar ( 764598 )
      MySQL for critical financial data, are you mad? MySQL is fine for simple stuff, but if you need critical reliability with MySQL and have to use InnoDB ($$) and other performance killers on MySQL to make it reliable, just use Postgres if you want the best open source database. Otherwise, use Oracle, or hell, even SQL Server.
    • "I don't think there's anything wrong with following the leader in that department. This probably isn't the best solution for relational databases so I would think another architecture would be in place for your MySQL and Postgres traditional database layouts. And that would be just huge centralized servers running virtualized instances of Linux with MySQL or Postgres."

      This is the govt. You're not gonna see mysql (really not gonna see this one for such a large datastore) nor postgres. It'll either be Orac

    • This probably isn't the best solution for relational databases...

      Actually... that's exactly the architectures many enterprise scale relational databases take (not the transactional type ones, but data warehousing ones); which conceptually are mostly SQL interfaces to map-reduce...

  • Die Hard 4 (Score:3, Funny)

    by Anonymous Coward on Thursday April 30, 2009 @07:06AM (#27770945)
    Die Hard 4 pretty much set the standard for design here.
  • Why build one... (Score:5, Interesting)

    by Bandman ( 86149 ) <bandman@nOsPAM.gmail.com> on Thursday April 30, 2009 @07:09AM (#27770955) Homepage

    ...when you could have two for twice as much?

    OK, actually two 250 million dollar datacenters, but I love that quote from Contact.

    Why tie up that much money in one site when you could build two world class structures and have full redundancy?

    • Re:Why build one... (Score:5, Interesting)

      by Anonymous Coward on Thursday April 30, 2009 @07:12AM (#27770977)

      I concur. There are 3 empty data centers along the I-81 corridor (within 2-3 hours of DC) that stand empty, reminders of the dot com bust. Why build new when you can retrofit an existing center.

      • by timelorde ( 7880 ) on Thursday April 30, 2009 @07:41AM (#27771237)

        Because they're not in my Congressional district?

      • Re: (Score:3, Interesting)

        by necro81 ( 917438 )

        reminders of the dot com bust. Why build

        I could think of three reasons:
        * Because those data centers are probably 10 years old already?
        * Because government data centers may have different requirements than internet startups?
        * Because 2-3 hours is too far away for social security administrators to drop by for a quick visit?

        • There's nothing wrong inherently with a 10 year old datacenter. Guys, 2000 was 10 years ago. It isn't like we're talking about datacenters from the paleolithic era.

          If you already have a building with 5 or 10 megawatts of power run to it from the utility company, and already having some generators hanging around, why not use it? It's not like there have been quantum leaps in cooling or fire suppression or generator technology or how raised floor is put together.

          So what, you have to go and buy 30 million

    • I believe it's actually Why build one when you can have two for twice the price. The twice and the price kinda rhyme so it rolls off the tongue a bit better and is more movieesque.
    • I'd go for 5 @ $100M apiece - A-F in Vermont, G-K in Ohio....
  • by Anonymous Coward
    Build? I'd take it as a retention bonus and retire. I just want to immulate our most sucessful bankers.
    • by Ogive17 ( 691899 )

      Build? I'd take it as a retention bonus and retire. I just want to immulate our most sucessful bankers.

      I wonder if the emphasis was intended...

      • Re: (Score:3, Funny)

        by Bandman ( 86149 )

        the typo was really close to immolate...probably psychological

        • I think it's really time to get religious again, but I tend to think more along the lines of the Maya and such. Let's immolate a manager a day until the gods of economy look favorable on us again.

          I'm damn sure that would work really, really quickly...

    • emulate? Anyway, I'd say it would be more of a telecom industry emulation. Promise one thing, say nevermind but pocket the cash anyway, never be held accountable.
  • by schwit1 ( 797399 ) on Thursday April 30, 2009 @07:15AM (#27771003)
    Putting the operation in a location that is cost effective would make the taxpayers very happy. The DC area is too expensive. Maybe an old missile facility in Wyoming or the Dakotas.
    • by duckintheface ( 710137 ) on Thursday April 30, 2009 @07:39AM (#27771217)
      Look at the locations where Google is putting it's new data centers. They are in rural parts of the Carolinas on property adjacent to power plants. Google negotiates uninterruptable power contracts in fast growing states on cheap land.
      • Re: (Score:3, Insightful)

        by SirKron ( 112214 )
        You can also negotiate allowing the power company use of your backup generators for their needs if during peak load times.
    • Re: (Score:3, Informative)

      by JoeMerchant ( 803320 )
      Your location also needs some competent staffing....
    • by G-Man ( 79561 )

      As of 2007, based on median income 10 out of the 20 wealthiest counties in the nation [wikipedia.org] were located in the DC metro area. As recently as 2000, it was only five. So now that the economy has tanked, where should we spend stimulus money? Why, in the wealthiest part of the country, where all our 'public servants' live. All hail Versailles on the Potomac!

    • Or an old abandoned salt mine. Perhaps they could do that at Yucca Mountain. There is already sunk cost there.

      We have some brand new abandoned data centers here too.

      I think up the road from DC into WV and not Northern VA would be the way to go. Cheap and located close to the decision makers.

       

  • http://en.wikipedia.org/wiki/Mantrap [wikipedia.org] Mantraps...?
    • by Bandman ( 86149 )

      I would definitely include mantraps in any new datacenter design, but I know of other datacenters that have them already. Some of them even have sensors to ensure that only one person at a time is in the trap. Crazy stuff.

      • Hopefully not just weight sensors. Most NOC employees I've seen are at either end of the weight spectrum, very seldom in the middle.
    • by Lehk228 ( 705449 )
      gotta watch out for those, consult admiral akbar as needed.
  • by IWantMoreSpamPlease ( 571972 ) on Thursday April 30, 2009 @07:20AM (#27771031) Homepage Journal

    Then spend the rest on hookers and blow.

    Hey, the gov't does that all the time, why are you staring at me like that??

    • by Miseph ( 979059 )

      Yeah, I've heard that Congressional party leaders are traditionally selected by a meeting of the senior critters over lines of coke on a stripper's tits.

  • Ummm.... (Score:3, Insightful)

    by Wanon ( 808109 ) on Thursday April 30, 2009 @07:20AM (#27771035)
    This is a really really bad place to ask how to spend $500000000....
  • by paul.schulz ( 75696 ) on Thursday April 30, 2009 @07:20AM (#27771039) Homepage

    Find an industry that would otherwise need a gas fired boiler and on-sell the heat.

    Other options:
    - Heated public swimming pool
    - Source of community/public heating

  • Ask Google (Score:3, Informative)

    by mcwop ( 31034 ) on Thursday April 30, 2009 @07:21AM (#27771041) Homepage
    I bet they could do it for less, and on schedule.
    http://www.engadget.com/2009/04/02/googles-data-center-secrets-revealed/ [engadget.com]
  • by Rik Sweeney ( 471717 ) on Thursday April 30, 2009 @07:22AM (#27771053) Homepage

    Data Center: $10M*

    Hookers and Beer: $490M

    *I reckon I could get that down to $6M by cutting corners though, so that's another $4M for the beer :)

    • by Pitr ( 33016 )

      I don't have any mod points at the moment, but if I did, I'm not sure if I should mod this "Funny" or "Insightful"...

  • Simple (Score:5, Funny)

    by Opportunist ( 166417 ) on Thursday April 30, 2009 @07:26AM (#27771087)

    200m for me, 200m for you, 100m for the guy in India who builds a data center.

  • Comment removed (Score:5, Insightful)

    by account_deleted ( 4530225 ) * on Thursday April 30, 2009 @07:31AM (#27771129)
    Comment removed based on user account deletion
    • Here. [ssa.gov]

      Trustees Reports issued over the last several years have indicated that Social Security's Old-Age, Survivors, and Disability Insurance (OASDI) Trust Funds would become insolvent sometime in the next 30 to 40 years under the intermediate set of economic and demographic assumptions provided in each report. Various proposals have addressed this long-range solvency problem. These proposals are generally intended to restore, or largely restore, solvency for the long-range period (the next 75 years).

      So, we

      • Comment removed based on user account deletion
        • Re: (Score:3, Funny)

          Or they could do something really novel like let *me* handle *my* own money bypassing SS entirely.

          Careful, you try not paying your taxes and Obama will tap you as a cabinet member so fast your head will spin.

    • Wait... you expect your social security contributions to be going anywhere BUT a black hole? Silly person.
  • OR,,,, (Score:3, Insightful)

    by phrostie ( 121428 ) on Thursday April 30, 2009 @07:31AM (#27771131)

    or they could contract it out to google.
    someone who knows how to manage large data centers correctly.

    • by Bandman ( 86149 )

      It would be nice, but I think there's policy in place to put out for bidders, and I doubt google would bid or be the lowest, if they did.

      • Lowest bid isn't usually the selection criteria. It's usually 'best value to the government', which leaves some wiggle room.
  • by Anonymous Coward on Thursday April 30, 2009 @07:33AM (#27771145)

    Dislaimer: I have worked for a contractor to the Air Force, and I have some insight as to government bidding, contracting, and results.

    Well, if I were responsible for results, I would get requirements from staff, send out RFPs, hire the best people, manage the project, and deliver on time and on budget a state-of-the-art data center.

    If I worked for the government, I would do the following:
      - Find a company I would like to work for as a six-figure lobbyist, and hire them without regard to experience or practicality. I will have personal contacts with the CEO, and if I don't, I soon will.
      - Get my "requirements" from that company, and have them provide the solution they specialize in without looking at my environment.
      - I would not supervise them. I am too important for that.
      - I would ask for more money as the project spirals out of control. The government would give it to me.
      - The project would drag on for a decade, would never finish, and would ultimately get scrapped. I wouldn't care, because I now work for the vendor, lobbying the government for more projects. I get my own private limo and driver, and I don't have to declare it on my taxes, unless I want a very visible government job again.

    Someone might raise a fuss in the public about this, but all that proves is that the government need more money to fix it.

    P.S. The contractor I worked for beat out a lower-bidding, "women-owned", "development-zoned", and much more local company. By any government calculus, the local company should have won the bid hands-down. But there were, shall we say, non-written reasons the local company lost and the gigantic out-of-state, double-the-bid, next door to DC company won.

    • Politics, politics, politics.

      We'll I would like to start a company that you are trying to find. Six figures seems like a bargain so when do we have a deal?

      Or everyone could have their hands in the till, make money, even if it gets scraped like that Ultimate Warfighter scheme. They couldn't cancel one thing because everything was supposed to be able to work together. Is that what your talking about?

      It really seems like there is a lot of sunk cost without much to show for it, like the F-22. We could make quit

  • by spike2131 ( 468840 ) on Thursday April 30, 2009 @07:40AM (#27771225) Homepage

    Commodity hardware
    Full virtualization at the OS level
    And a second, mirrored data center on the other side of the country

  • For $500M I'd go for a nice laptop, on a beach, on my own private island. Maybe a NAS system with a few TB of storage and WiFi coverage for the island, couple of big screens on dedicated computers here and there, and possibly a cluster of Core i7 boxes to play with some heavy crunching.

    All told, $499.5M could go toward the real-estate and construction costs, staff, transportation, etc. $500K would cover my compute hardware needs quite nicely.

    When you analyze what the SSA ends up doing with $500M, y
  • Comment removed based on user account deletion
  • by ciellarg ( 899150 ) on Thursday April 30, 2009 @07:54AM (#27771373)
    Congress has already bankrupted the SSA fund, so all they need is one computeer that can add and subtract "0" very quickly....unless they need something that can divide by zero :)
  • and yes, they'll play doom.

    seriously though, I'd buy google data crates. one to match each US representative and then I'd place them across the US in those areas, which is the best indicator of population coverage.

    Then I'd spend the leftovers on an underground lair.

  • by steveb964 ( 727054 ) on Thursday April 30, 2009 @08:06AM (#27771523) Homepage
    Cardboard box, tin cans and string.

    I'd then proceed to give the rest to support the bailout of the severely ailing and deserving auto and financial institutions.

    sb
    • Re: (Score:3, Funny)

      by Opportunist ( 166417 )

      Once you're done bailing out the deservingauto and financial institutions, you should still have about 499m left (let's assume you bought a LOT of boxes, tin cans and a few miles of string...).

  • http://www.server-sky.com/ [server-sky.com]

    The idea is to launch a lot (zillions) of tiny (grams) solar-powered servers to orbit. This means you have no power of cooling issues. It sounds pretty crazy on the face of it, but if it costs ~$1G to build a data center, it may actually be economical. There are a ton of practical issues, of course - the site goes into them in some detail.

  • Seriously I would give my business to the vendors and hardware manufacturers who are committed to America. Force the equipment to be made here, with U.S. labor start to finish and supported by U.S. labor until the contract expires.

    You want to get the country back on track, rather than just giving handouts force the change to happen with the money already being spent.

    That said, I would ensure it is heavy on virtualization and solid blade servers/chassis. A nod needs to be given to "green-ness" and every effo

  • Or around '99. Anyway, one of our jobs was replacing token ring with ethernet! I'm talking replacing the fat cables. Ugh, that was fun. Their closets were horrendous. We had to rewire the phones, too, for some reason I forget now. The server room was completely under-AC'd. (At least they had a raised floor). All kinds of conduit issues, electrical issues (underpowered circuits, non-grounded circuit boxes). The building itself was ok, but would have to be gutted, and I'm not sure if the infrastructu

  • This is a project they should get right. Social Security's data centers maintain earning and benefits information for nearly every American worker, processing 75 million transactions per day. "All of our plans depend upon a strong, 21st century data center," the agency said in a recent document describing its needs. See Data Center Knowledge [datacenterknowledge.com] for more on this. Regarding the location, the agency says the new facility would need to be within 40 miles of its current location due to "data linkages" - presumably
  • As this is a government project, for that much money they can buy ten laptops with the required three-button Teflon-ball mice and ergonomic anti-Teflon mousepads.
  • ... on management's competence. I used to work for a company that burned through about $300 million one one IT project. Effectively, it was all lost (I say 'effectively' because the project exists in the form of a couple of servers, just so they don't have to write down 0.3 billion on an annual report and have the BOD see it). Hire these people and $500 million will get the footings poured.

    "What? You want the walls and roof? That's gonna cost you more. By the way, that's a nice little Social Security Admi

  • Was it cheaper when they did it all with paper files?

    • by davburns ( 49244 )

      Was it cheaper when they did it all with paper files?

      Yes, but mostly because there were a lot fewer people back then. (Remember, that even sorting is O(N log N) -- and you have to do that to get the right papers in the right files. I would guess that there needs to be lots of O(N^2) operations to catch fraud. 300,000 Americans is a big N.

Do you suffer painful hallucination? -- Don Juan, cited by Carlos Casteneda

Working...