Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!


Forgot your password?
Businesses Communications Network Networking Software News IT

Ask Slashdot: How Do News Organizations Keep Track of So Much Information? 119

dryriver writes: Major news organizations from CNN, BBC, ABC to TIME magazine, the New York Times and the Economist publish a tremendous amount of information, especially now that almost everybody runs a 24/7 updated website alongside their TV channel, magazine or newspaper. Question: How do news organizations actually keep track of what must be 1000s of pieces of incoming information that are processed into news stories every day? If they are using software to manage all this info -- which makes a lot of sense -- is it off-the-shelf software that anybody can buy, or do major news organizations typically commission IT/software contractors to build them a custom "Information Management System" or similar? If there is good off-the-shelf software for managing a lot of information, who makes it and what is it called?
This discussion has been archived. No new comments can be posted.

Ask Slashdot: How Do News Organizations Keep Track of So Much Information?

Comments Filter:
  • If it follows the narrative, they keep and publish it.
    If it doesn't, they purge it.

    • Re: (Score:3, Insightful)

      by Anonymous Coward

      If it's the right, information is not needed. Just shout it loudly and it must be true.

    • True for all outlets more than 5 years old and 90% less than 5.

      Both sides have 'stories', which is all they need.

      • by ShanghaiBill ( 739463 ) on Tuesday June 06, 2017 @09:28PM (#54565261)

        If you have ever read a news story where you have first hand knowledge of what is being reported, then you should know that most articles get a lot of facts wrong, and sometimes are wildly inaccurate. So the premise of the questions is wrong.

        Q: How do news organizations keep track of so much info?
        A: They don't.

        • by dcw3 ( 649211 )

          Mod parent up.

          I've had direct knowledge of a few stories over the years. What gets reported is often wildly inaccurate. Reporters often don't understand the topic at hand, or modify the story to make it more interesting.

        • by koavf ( 1099649 )
          Your point is fair but irrelevant: inaccurate information is still information and arguably much harder to keep track of since you can be wrong an infinite number of ways but only correct one way. (And for what it's worth, I have been involved in a few news stories first-hand.)
    • Spew hate, false accusations and never admit you are wrong.

      If a fact proves you wrong, call it fake news and build a conspiracy theory with no basis to distract your base with nonsense long enough for their tiny minds to forget the fact that would have changed their world view.
    • Re: (Score:1, Insightful)

      by KermodeBear ( 738243 )

      There is truth to this.

      You don't have to keep or manage information if you can just make up whatever you want and cite "a source familiar with the situation." I have never before in my life seen news organizations rely so heavily on anonymous sources. CNN, ABC, MSNBC, etc., they slap whatever they want up onto their sites, say that "someone told them so," and when the story is later proven false, eh, maybe they issue a correction. Maybe. Mostly they just let it sit out there.

      Pretty sad, but news organizatio

      • They should restore the restriction where some nameless oligarch could only monopolize one type of media in a given area. When they took this restriction away they said the Internet would keep the discussion open, boy were they wrong.
      • I believe you just "mistakenly" described the Ministry of Truth. Also, we're at war with Eastasia, we've always been at war with Eastasia. The troops need boots.
      • Trump's version of "a source familiar with the situation" is "Some people say" or "Many people say". Even if the some people is a single person named Trump. Technically he isn't wrong.

        From a Donald J. Trump Tweet:


        Many people are saying that the Iranians killed the scientist who helped the U.S. because of Hillary Clinton's hacked emails.
        4:45 PM - 8 Aug 2016

        No evidence ever presented.

        • Or this:
          "You know, some people say that was not his birth certificate," he told ABC in August 2013, more than two years after President Obama released the document. "I'm saying I don't know. Nobody knows and you don't know either."

    • Vs the right who just makes crap up and puts in a few sound bites to confirm it.

      Does that sound unfair too? Well it should. Comment like your use to be funny however we now live in a world where the president will troll news anchors in real time on twitter. And seems to be having a tendency of trusting tv news over the CIA and FBI.

      There is news then there is commentary. We need to teach the people the difference as it is often blurred.
      News: person x and y did this.
      Commatary: person x was unjustified whil

    • a large news organization will always have specialists, used to be they were called "something" desks in the inky cigar-littered past. the crime desk reporters covered the cops. the business desk reporters covered the business wires, ticker-tape, and wires. sports desk, you had reporters assigned to each team. and so on. Desk Editors rode herd.

      reporter's desks were a mess of folders and papers, and older information was filed in the news morgue, a wall of file cabinets. Facts On File, an annual compendium

    • I assume (hope?) you meant that to be funny, but even so, the irony of your answer is remarkable.

      The OP asked a serious question, looking for facts from people who actually know about a subject. You replied with an answer that contained no facts at all. In fact, you said things that are obviously, objectively false. You did that to score political points. Or to put it differently, you just posted made up "facts" based on the narrative you want to promote.

      Now reread your post. Do you see the irony?

  • by sootman ( 158191 ) on Tuesday June 06, 2017 @07:56PM (#54564757) Homepage Journal

    We're not doing your legwork for you.

    • Gosh, I wish there was something like, say, a global search engine where you could type this in as a query, and it'd point to articles on how actual news organisations do this. Pity there's nothing like this around.
      • ...and it'd point to articles on how actual news organisations do this

        Nice try, except for the fact that those articles don't exist.

        • Well, I could point you to quite a number of commercial and open-source projects designed to manage the kind of data that goes into news stories. Or mention that Django was developed in large part for news organizations (https://en.wikipedia.org/wiki/Django_(web_framework)). And that a quick query on a global search engine could turn up this and more.

          But those facts are obviously alternative to your facts. So Fake, I guess.

  • by Anonymous Coward on Tuesday June 06, 2017 @07:56PM (#54564761)

    Excel spreadsheets tens of thousands of lines long.

  • by jasonla ( 211640 ) on Tuesday June 06, 2017 @07:58PM (#54564775)
    There is an industry software that gets used a lot called iNews [avid.com]. There's a reddit thread [reddit.com] with comments from people who work at news orgs. Vox Media (The Verge, SBNation, Curbed, Polygon) built its own CMS called Chorus [pfauth.com]. The NYTimes uses WordPress for some of its blogs. And I assume the Washington Post built their own since, well, Bezos.
    • I'm not sure if this is industry-specific enough (it probably isn't), but there is a CMS comparison matrix [cmsmatrix.org] which compares 1,300 Content Management Systems.

  • Novel idea here. (Score:5, Insightful)

    by Gravis Zero ( 934156 ) on Tuesday June 06, 2017 @08:00PM (#54564785)

    Have you tried contacting and asking such an organization this very question?

    • Re: (Score:3, Funny)

      by nospam007 ( 722110 ) *

      "Have you tried contacting and asking such an organization this very question?"

      He asked the New York Times, but none of their +1500 reporters had time because each had a real news job to do.

    • by Anonymous Coward

      lol, this is "news" sites not "information".

    • They lie about everything else, so you probably wouldn't get a true answer.

      • They lie about everything else, so you probably wouldn't get a true answer.

        Paranoid schizophrenic spotted. Proceed with caution.
        He might try to push further conspiracy theories to you or force you to wear tinfoil on your head.

  • The Python based Django [wikipedia.org] framework was originally designed for this purpose. No doubt others use a different system, but a few use it.
  • by Ben Sullivan ( 3644925 ) on Tuesday June 06, 2017 @08:02PM (#54564801)
    Newsrooms depend on well-informed editors and reporters who are often notoriously paper-based. I've worked in six newspaper and magazine newsrooms and it's generally a central CMS/publishing/workflow system, plus a person-by-person armory of solutions (reporter's notebooks, things like Evernote, spreadsheets, etc.) There are systems used in intelligence that could find use, but journalists are kind of sensitive about doing things their own way -- in my experience. The real lifeblood of a newsroom is the channels of incoming info: wires, cable tv, Google News, etc.
    • by rwa2 ( 4391 ) *

      Woo, had to scroll down FAR to find any mention of a CMS (content management system), but yes.

      The closest I've ever worked for a news site was Disney, with the ESPN folks. They had bought go.com (formerly starwave.com, a Steve Ballmer venture capital spinoff from Microsoft). So they had some in-house thing in Java called GoPublish, which ran on Windows Server back in the day (they had just finished porting it to Linux when I left a few years ago), and all of the content was stored in Oracle DBs and indexe

  • by Anonymous Coward

    At least, not a one-stop centralized system in most/many newsrooms in the US. This generally falls under the category of a reporter's responsibility to maintain this information in the way that works best for his/her/the team (of course, situations vary). It is not unusual to see a reporter storing all of their data exclusively on a personal Dropbox/OneDrive/etc account.

    As for collaboration, organizations may use a product like SharePoint, but I'd be willing to wager that 90%+ of organizations in the US a

  • When news organizations have needed to see what coverage existed on a subject in past decades at least, they'd find the guy who had access to LexisNexis and get some results from that.

    At least that's what always comes up in inside-baseball discussions on news gathering stuff I've seen.

    Ryan Fenton

  • Maybe there is someone else. These are wire services who publish a lot of the raw stories the other news organizations pick up and republish and pay to do so. If it's a really big story CNN and others will send their own reporters out.

  • The questions seems weird to me. The media organizations I've been involved with have all gathered, filtered, and kept track of information using a loosely networked system of devices known as trained human brains. Much of information-gathering is subjective; there are many "pieces of information" that cross your desk each day which ultimately can and should be discarded, often because the "information" is simply inaccurate. I imagine it would be very difficult to train any kind of computer to make value ju

  • by Anonymous Coward on Tuesday June 06, 2017 @08:55PM (#54565113)

    I know this space well. My consulting/integration company works with many, many media companies including the majors on this exact area. AMA? I've been doing this for 13 years, and literally work with many of the largest media companies on the planet.

    There are two layers to the answer to this question. The first is storage and networking infrastructure, which is evolving very quickly for many reasons. Object storage, cloud (public/private/hybrid) -- all of these trends are having a massive impact on how the industry does things, but media is 5-10 years behind many other industries in adopting IT to solve particular challenges (our data needs are very, very high). So the move to object and cloud storage, taking advantage of 10GigE much less 40 and 100, seeing where fibre channel goes (SANs are used very extensively), the changing cost environment for all this stuff -- all these things are hitting int he media space big time.

    The next layer is the software management layer. We call this "MAM" for Media Asset Management. It's a bit of a catch-all term, and sort of folds up to DAM, or Digital Asset Management, and contains within it PAM, or Production Asset Management. It is sort of a shorthand term that refers to:

      Getting your media and other data behind a database

      Utilize software automation and integration technologies to orchestrate all sorts of interesting workflows

    MAM too is taking more and more advantage of the cloud and hybrid deployments. There are dozens of MAM vendors, with a handful of leaders. For instance Avid has PAM and MAM platforms they brand as "Interplay" (it's two different things). There are dozens of others, and I know many of them quite well. Again, my company does major MAM and workflow deployments for top-tier global M&E companies (among others). If I can answer questions, shoot 'em over.

    • Re: (Score:2, Informative)

      by Anonymous Coward

      Ah, I should add that many news agencies have another layer for information management called a newsroom computing system.

      AP, the news-gathering organization, actually sells one of the main ones, called ENPS.

      Avid has one a lot of people use, called iNews.

      And then there is one a lot of organizations use called Octopus.

      MAMs often integrate with ENPS, using a protocol called MOS. This allows you to associate assets in the MAM with placeholders in a rundown put together in the newsroom system, which is used for

  • by bugs2squash ( 1132591 ) on Wednesday June 07, 2017 @12:08AM (#54566023)
    I don't know what the news organizations use, but governments have some pretty big data sets and they use platforms like ckan [ckan.org] and OGPL [wikipedia.org].
  • Once approved by the govt of the day it becomes 'news' if not its 'fake news'
  • Join SMPTE. Get articles from back issues of their "Motion Imaging Journal" that deal with IT in the production workplace. MOS, Media Object Server, is one of the key acronyms. SDI, Serial Digital Interface, is the specification for the video pipeline hardware in many installations.

    MOS leads you to ENPS. Follow that down the rabbit hole to as much knowledge as most people would want if the motivation is only curiosity. The whole system is quite flexible and complex. (MOS is a relatively modest part of the g

  • by Anonymous Coward

    Just thought the OP might be interested in an actual answer rather than endless ill-informed snark.

    Obviously approaches vary between different organisations. When I worked at the BBC they used a system called ENPS - Electronic News Preparation System. It was developed by the Associated Press, and it was geared largely towards broadcast operations. It collated information from journalists on the ground, agency reports, broadcast scripts and contact information for sources and subjects. Over time, more and mo

  • by Bender Unit 22 ( 216955 ) on Wednesday June 07, 2017 @03:49AM (#54566585) Journal

    They dont, it is very obvious these days that they are not as fantastic as one once thought.

  • It is easy. They outsource it all. No, not to India. They just outsource it to the companies who then send them press releases. That is about 90% of the work done
    The other 10% they copy and paste from Reuters.

  • That's a job for systemd, right?

  • by SharpFang ( 651121 ) on Wednesday June 07, 2017 @06:33AM (#54566921) Homepage Journal

    Their editors scour the news agencies, like Associated Press for what they deem "news-worthy". These are standarized gateways, web api for importing purchased articles, which get pushed into local CMS, then manually, or half-automatically laid out. Duplicates are avoided through marking all purchases. If anything newsworthy is announced ahead of time, and the "higher ups" want something exclusive, reporters are send to provide own scoop - but great most of data comes from the agencies.

    Generally, a reporter working for a newspaper or media outlet directly is a much more rare sight than a reporter working for a news agency; news aggregated in the agencies and then distributed to news outlets.

    Source: worked at a news portal. The token reporter team existed only so that the portal would be still protected by press law, as mere "news aggregation" media can't get that around here.

  • by yelvington ( 8169 ) on Wednesday June 07, 2017 @07:01AM (#54567075) Homepage

    I do this for a living, so my answer is somewhat detailed.

    Newspapers were using content management systems for this purpose beginning around 1970, before PCs. Previous to that, stories were transmitted electronically, stored on punch tape in a 6-bit format, but edited on paper and re-keyboarded as necessary.

    If you wanted to use a story as-is, without editing, you could have a copyboy go find the right punch tape and hand-carry it to the typesetting department.

    Computerizing the editing process/approval process allowed written material to be stored, edited on screens, and output directly to electronic typesetters (which were already computerized; a major use of the PDP-8 was automated hyphenation and justification). The story "files" were typically organized in "queues" or "baskets."

    The earliest CMS were bespoke, but they quickly became standardized -- "off the shelf" with potentially a great deal of customization, produced by about a dozen companies around the world that often designed and built their own hardware components.

    Electronic page layout was pioneered on these systems. One of the first was at the Minneapolis Star and Tribune; the project leader later created founded Aldus, created Pagemaker, and the desktop publishing revolution followed.

    As desktop publishing emerged, it displaced bespoke layout systems, and networked PCs displaced proprietary terminals, and SQL databases displaced proprietary storage, but the putting them together into a usable workflow system remained a specialty. In general, the CMS companies got out of the hardware business entirely and focused on software and services.

    Photos came later. Keep in mind that the JPEG standard didn't even exist until the 1990s. The first wirephoto storage-and-editing systems were big bespoke monsters that looked like something from a 1950s sci-fi serial, but they were quickly replaced by Mac-based tools, and then the core CMS systems embraced photo management.

    Broadcasting trailed all of this in many ways. TV stations actually produce fairly little information in the common sense of the word, and have lighter requirements for handling text, but huge amounts of data in the form of video. When I first worked in TV, video was shot on film, then videotape. As video became digitized and companies like Avid created digital video editors, managing the data became a requirement there as well, and a specialty.

    It's now possible to put together a text/image/video workflow system with open source tools. For a single publication, I could do it in a few days with Drupal, and if the Web is the target, it's all pretty straightforward. But the news CMS field is still dominated by specialty vendors.

    Print is still a huge driver of revenue, and that means interfacing with advertising workflow and print page layout tools. Adobe InDesign is pretty much the standard there, although I know of one or two systems that have proprietary layout. As a result, a small (and shrinking) number of specialty vendors dominate. They integrate off-the-shelf components, including open source tools and commercial software.

    Where I work, writers are using CKEditor, but it's implemented in a proprietary Web-based workflow system that publishes to multiple Drupal sites on the Web and integrates with InDesign for print. Wire service information, agency photos, etc., all come into the CMS.

    Because most of the older legacy systems are utterly print-focused, they can be extremely frustrating in a digital world. Some news companies have assemble parallel production systems for the Web, stitching together any number of off-the-shelf components, or writing proprietary code. If you use Django, you should know that it was created at a newspaper company. The Washington Post has created its own system called Arc that it is peddling to other news companies.

  • If it follows the narrative, they keep and publish it.
    If it doesn't, they purge it to keep the narrative.

  • The news is so partisan that they only need to keep track of half of the facts. I find that reality doesn't fit neatly into the right / left paradigm, or Dem / Rep if you prefer, but many like to act as if this binary representation doesn't have substantial quantization error.
  • I worked at the AP as a software engineer for seven years at HQ in Manhattan. One system I worked on for a few years was the "Desk" system, which is a set of 3 OpenVMS Alpha clusters (NY, London, Tokyo). This was the primary news collection and dissemination system known to outsiders as "The AP Wire". It accepted thousands of stories per hour from contributors, and transmitted thousands to paying clients. Clients were typically newspapers that received various "feeds" from the AP such as Business, Sport

Mausoleum: The final and funniest folly of the rich. -- Ambrose Bierce