Ask Slashdot: How Do News Organizations Keep Track of So Much Information? 119
dryriver writes: Major news organizations from CNN, BBC, ABC to TIME magazine, the New York Times and the Economist publish a tremendous amount of information, especially now that almost everybody runs a 24/7 updated website alongside their TV channel, magazine or newspaper. Question: How do news organizations actually keep track of what must be 1000s of pieces of incoming information that are processed into news stories every day? If they are using software to manage all this info -- which makes a lot of sense -- is it off-the-shelf software that anybody can buy, or do major news organizations typically commission IT/software contractors to build them a custom "Information Management System" or similar? If there is good off-the-shelf software for managing a lot of information, who makes it and what is it called?
If it's the left, just a narrative will do. (Score:1, Troll)
If it follows the narrative, they keep and publish it.
If it doesn't, they purge it.
Re: (Score:3, Insightful)
If it's the right, information is not needed. Just shout it loudly and it must be true.
Re: (Score:2)
True for all outlets more than 5 years old and 90% less than 5.
Both sides have 'stories', which is all they need.
Re:If it's the left, just a narrative will do. (Score:5, Insightful)
If you have ever read a news story where you have first hand knowledge of what is being reported, then you should know that most articles get a lot of facts wrong, and sometimes are wildly inaccurate. So the premise of the questions is wrong.
Q: How do news organizations keep track of so much info?
A: They don't.
Re: (Score:1)
Mod parent up.
I've had direct knowledge of a few stories over the years. What gets reported is often wildly inaccurate. Reporters often don't understand the topic at hand, or modify the story to make it more interesting.
Re: (Score:2)
And conform to 'the story'.
Re: (Score:2)
Re: (Score:1, Interesting)
Re: (Score:2)
How about "Seth Rich Murder" on CNN, Politico, MSNBC, Salon, NYT, and WaPo? You are a fucking tool...
You made me curious, so I searched for "seth rich murder" on foxnews.com (because that's the only outlet mentioned by the GP you referred to as "a fucking tool"). There is only ONE hit on foxnews.com, and it is a retraction of the story, stating:
On May 16, a story was posted on the Fox News website on the investigation into the 2016 murder of DNC Staffer Seth Rich. The article was not initially subjected to the high degree of editorial scrutiny we require for all our reporting. Upon appropriate review, the article was found not to meet those standards and has since been removed.
We will continue to investigate this story and will provide updates as warranted.
So what was your point exactly?
Re: (Score:2)
And of course they don't have the balls or honor to admit they are wrong or to apologize to the family for turning the death of their son into another fake news conspiracy theory like the pizza pedos.
Re: If it's the right.... (Score:2, Troll)
If a fact proves you wrong, call it fake news and build a conspiracy theory with no basis to distract your base with nonsense long enough for their tiny minds to forget the fact that would have changed their world view.
Re: (Score:3)
Re: (Score:3)
Re: (Score:1, Insightful)
There is truth to this.
You don't have to keep or manage information if you can just make up whatever you want and cite "a source familiar with the situation." I have never before in my life seen news organizations rely so heavily on anonymous sources. CNN, ABC, MSNBC, etc., they slap whatever they want up onto their sites, say that "someone told them so," and when the story is later proven false, eh, maybe they issue a correction. Maybe. Mostly they just let it sit out there.
Pretty sad, but news organizatio
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
Trump's version of "a source familiar with the situation" is "Some people say" or "Many people say". Even if the some people is a single person named Trump. Technically he isn't wrong.
From a Donald J. Trump Tweet:
@realDonaldTrump
Many people are saying that the Iranians killed the scientist who helped the U.S. because of Hillary Clinton's hacked emails.
4:45 PM - 8 Aug 2016
No evidence ever presented.
Re: (Score:2)
Or this:
"You know, some people say that was not his birth certificate," he told ABC in August 2013, more than two years after President Obama released the document. "I'm saying I don't know. Nobody knows and you don't know either."
Re: (Score:2)
Vs the right who just makes crap up and puts in a few sound bites to confirm it.
Does that sound unfair too? Well it should. Comment like your use to be funny however we now live in a world where the president will troll news anchors in real time on twitter. And seems to be having a tendency of trusting tv news over the CIA and FBI.
There is news then there is commentary. We need to teach the people the difference as it is often blurred.
News: person x and y did this.
Commatary: person x was unjustified whil
teamwork and specialists (Score:2)
a large news organization will always have specialists, used to be they were called "something" desks in the inky cigar-littered past. the crime desk reporters covered the cops. the business desk reporters covered the business wires, ticker-tape, and wires. sports desk, you had reporters assigned to each team. and so on. Desk Editors rode herd.
reporter's desks were a mess of folders and papers, and older information was filed in the news morgue, a wall of file cabinets. Facts On File, an annual compendium
Re: (Score:2)
I assume (hope?) you meant that to be funny, but even so, the irony of your answer is remarkable.
The OP asked a serious question, looking for facts from people who actually know about a subject. You replied with an answer that contained no facts at all. In fact, you said things that are obviously, objectively false. You did that to score political points. Or to put it differently, you just posted made up "facts" based on the narrative you want to promote.
Now reread your post. Do you see the irony?
Re: (Score:2, Funny)
Please note that it's the Republican Party more than the political right. The Republicans are a centrist party. The closest thing you will get to a politically "right" party these days are the Libertarian or Federalist parties.
Re: (Score:2)
Please note that it's the Republican Party more than the political right. The Republicans are a centrist party. The closest thing you will get to a politically "right" party these days are the Libertarian or Federalist parties.
on scale of left to right to 0-100 democrat party are maybe 60 and republicans are 80.
don't pretend you have a centrist party. if republicans are centrist then democrats are communists and you know very well that they aren't even close..
thats the scale if you look at globally, like, if you look in terms of what kind of politics you could be running in the country it's all right from the center.
I mean you could be just as well be trying to argue that democrats are a socialist party...
Re: (Score:2)
Re: (Score:1)
Correct. Over here in Europe the Democrats would be a right wing party. The US leans very much to the right. Which is not a bad thing, but just an observation.
Or, if you put it in relative terms, over here, Europe leans very much to the left. Just an observation.
Re: (Score:2)
Please define left and right before making these assertions.
And being libertarian is far right? What the hell does that even mean? Libertarians are predominantly pro-freedom on basically everything. Is freedom a far right concept, or a far left concept? And why?
For example, for libertarians, free speech is absolute. In Europe however, there are plenty of censorship laws, especially for ones that people label as hate speech with some somewhat vague rules. So which one is right, and which one is left? Again,
Re: If it's the left, just a narrative will do. (Score:2)
Pulling numbers like 60 and 80 out of your ass is just the kind of superficial and tendentious analysis we have come to expect from the American left.
In fact the very assumption of a strict left right direction is b.s. For example, although communists/socialists and fascists are conventionally located at the far left and far right, they are ideologically quite close. Progressives are a little more moderate, but also close to either.
Re: (Score:2)
No they are not. We have abandoned centrists, what ever party we most align to seems the most centrist. But the odd dicominary is while most people wants to seem like they are a centrist they normally reject voting for the more centrist person. Why because you can be a polar opposite of a centrist. You may be pro choice but wants less gun owner restrictions. While the centrist who is running would be pro life and wants more gun restrictions.
The parties to survive need to be non centrist. Because then t
Figure it out on your own, Bezos. (Score:5, Funny)
We're not doing your legwork for you.
Re: (Score:2)
Re: (Score:2)
...and it'd point to articles on how actual news organisations do this
Nice try, except for the fact that those articles don't exist.
Re: (Score:2)
Well, I could point you to quite a number of commercial and open-source projects designed to manage the kind of data that goes into news stories. Or mention that Django was developed in large part for news organizations (https://en.wikipedia.org/wiki/Django_(web_framework)). And that a quick query on a global search engine could turn up this and more.
But those facts are obviously alternative to your facts. So Fake, I guess.
Excel pivot tables (Score:4, Funny)
Excel spreadsheets tens of thousands of lines long.
Depends on what you mean by "keep track" (Score:5, Informative)
Re: (Score:2)
I'm not sure if this is industry-specific enough (it probably isn't), but there is a CMS comparison matrix [cmsmatrix.org] which compares 1,300 Content Management Systems.
Novel idea here. (Score:5, Insightful)
Have you tried contacting and asking such an organization this very question?
Re: (Score:3, Funny)
"Have you tried contacting and asking such an organization this very question?"
He asked the New York Times, but none of their +1500 reporters had time because each had a real news job to do.
Re: (Score:1)
lol, this is "news" sites not "information".
Re: Novel idea here. (Score:2)
They lie about everything else, so you probably wouldn't get a true answer.
Re: (Score:2)
They lie about everything else, so you probably wouldn't get a true answer.
Paranoid schizophrenic spotted. Proceed with caution.
He might try to push further conspiracy theories to you or force you to wear tinfoil on your head.
Re: (Score:2)
Yes: you looked in the mirror again.
Some use Python/Django (Score:2)
Some tech, much brain (Score:5, Informative)
Re: (Score:2)
As I get older I realize that progress usually comes with a cost; when something is gained, something is inevitably lost. The access that search engines and social media have given us to volumes of data is invaluable; and yet for many people their mental life has been reduced to an endless cycle: skim and react, skim and react. Just as it is possible to become more informed than any human being of any past generation, it has also become easy to be plugged into current events, and yet ignorant.
To someone of
Re: (Score:2)
Thank you for a wonderful and thoughtful response.
Re: (Score:3)
Woo, had to scroll down FAR to find any mention of a CMS (content management system), but yes.
The closest I've ever worked for a news site was Disney, with the ESPN folks. They had bought go.com (formerly starwave.com, a Steve Ballmer venture capital spinoff from Microsoft). So they had some in-house thing in Java called GoPublish, which ran on Windows Server back in the day (they had just finished porting it to Linux when I left a few years ago), and all of the content was stored in Oracle DBs and indexe
Short Answer: They Don't. (Score:1)
At least, not a one-stop centralized system in most/many newsrooms in the US. This generally falls under the category of a reporter's responsibility to maintain this information in the way that works best for his/her/the team (of course, situations vary). It is not unusual to see a reporter storing all of their data exclusively on a personal Dropbox/OneDrive/etc account.
As for collaboration, organizations may use a product like SharePoint, but I'd be willing to wager that 90%+ of organizations in the US a
Historically? LexisNexis... (Score:2)
When news organizations have needed to see what coverage existed on a subject in past decades at least, they'd find the guy who had access to LexisNexis and get some results from that.
At least that's what always comes up in inside-baseball discussions on news gathering stuff I've seen.
Ryan Fenton
Reuters and AP (Score:2)
Maybe there is someone else. These are wire services who publish a lot of the raw stories the other news organizations pick up and republish and pay to do so. If it's a really big story CNN and others will send their own reporters out.
I dunno, how do YOU keep track of information? (Score:2)
The questions seems weird to me. The media organizations I've been involved with have all gathered, filtered, and kept track of information using a loosely networked system of devices known as trained human brains. Much of information-gathering is subjective; there are many "pieces of information" that cross your desk each day which ultimately can and should be discarded, often because the "information" is simply inaccurate. I imagine it would be very difficult to train any kind of computer to make value ju
I know this space well (Score:4, Interesting)
I know this space well. My consulting/integration company works with many, many media companies including the majors on this exact area. AMA? I've been doing this for 13 years, and literally work with many of the largest media companies on the planet.
There are two layers to the answer to this question. The first is storage and networking infrastructure, which is evolving very quickly for many reasons. Object storage, cloud (public/private/hybrid) -- all of these trends are having a massive impact on how the industry does things, but media is 5-10 years behind many other industries in adopting IT to solve particular challenges (our data needs are very, very high). So the move to object and cloud storage, taking advantage of 10GigE much less 40 and 100, seeing where fibre channel goes (SANs are used very extensively), the changing cost environment for all this stuff -- all these things are hitting int he media space big time.
The next layer is the software management layer. We call this "MAM" for Media Asset Management. It's a bit of a catch-all term, and sort of folds up to DAM, or Digital Asset Management, and contains within it PAM, or Production Asset Management. It is sort of a shorthand term that refers to:
Getting your media and other data behind a database
Utilize software automation and integration technologies to orchestrate all sorts of interesting workflows
MAM too is taking more and more advantage of the cloud and hybrid deployments. There are dozens of MAM vendors, with a handful of leaders. For instance Avid has PAM and MAM platforms they brand as "Interplay" (it's two different things). There are dozens of others, and I know many of them quite well. Again, my company does major MAM and workflow deployments for top-tier global M&E companies (among others). If I can answer questions, shoot 'em over.
Re: (Score:2, Informative)
Ah, I should add that many news agencies have another layer for information management called a newsroom computing system.
AP, the news-gathering organization, actually sells one of the main ones, called ENPS.
Avid has one a lot of people use, called iNews.
And then there is one a lot of organizations use called Octopus.
MAMs often integrate with ENPS, using a protocol called MOS. This allows you to associate assets in the MAM with placeholders in a rundown put together in the newsroom system, which is used for
Re: (Score:1)
www.chesa.com -- I hear these guys are great, if I do say so myself. ;)
I dunno (Score:3)
They just make it up. (Score:2)
Join SMPTE (Score:1)
Join SMPTE. Get articles from back issues of their "Motion Imaging Journal" that deal with IT in the production workplace. MOS, Media Object Server, is one of the key acronyms. SDI, Serial Digital Interface, is the specification for the video pipeline hardware in many installations.
MOS leads you to ENPS. Follow that down the rabbit hole to as much knowledge as most people would want if the motivation is only curiosity. The whole system is quite flexible and complex. (MOS is a relatively modest part of the g
Journalist here (Score:1)
Just thought the OP might be interested in an actual answer rather than endless ill-informed snark.
Obviously approaches vary between different organisations. When I worked at the BBC they used a system called ENPS - Electronic News Preparation System. It was developed by the Associated Press, and it was geared largely towards broadcast operations. It collated information from journalists on the ground, agency reports, broadcast scripts and contact information for sources and subjects. Over time, more and mo
They don't (Score:3)
They dont, it is very obvious these days that they are not as fantastic as one once thought.
Re: (Score:2)
Systemd? (Score:2)
That's a job for systemd, right?
Associated Press (Score:3)
Their editors scour the news agencies, like Associated Press for what they deem "news-worthy". These are standarized gateways, web api for importing purchased articles, which get pushed into local CMS, then manually, or half-automatically laid out. Duplicates are avoided through marking all purchases. If anything newsworthy is announced ahead of time, and the "higher ups" want something exclusive, reporters are send to provide own scoop - but great most of data comes from the agencies.
Generally, a reporter working for a newspaper or media outlet directly is a much more rare sight than a reporter working for a news agency; news aggregated in the agencies and then distributed to news outlets.
Source: worked at a news portal. The token reporter team existed only so that the portal would be still protected by press law, as mere "news aggregation" media can't get that around here.
Content management systems, of course (Score:5, Informative)
I do this for a living, so my answer is somewhat detailed.
Newspapers were using content management systems for this purpose beginning around 1970, before PCs. Previous to that, stories were transmitted electronically, stored on punch tape in a 6-bit format, but edited on paper and re-keyboarded as necessary.
If you wanted to use a story as-is, without editing, you could have a copyboy go find the right punch tape and hand-carry it to the typesetting department.
Computerizing the editing process/approval process allowed written material to be stored, edited on screens, and output directly to electronic typesetters (which were already computerized; a major use of the PDP-8 was automated hyphenation and justification). The story "files" were typically organized in "queues" or "baskets."
The earliest CMS were bespoke, but they quickly became standardized -- "off the shelf" with potentially a great deal of customization, produced by about a dozen companies around the world that often designed and built their own hardware components.
Electronic page layout was pioneered on these systems. One of the first was at the Minneapolis Star and Tribune; the project leader later created founded Aldus, created Pagemaker, and the desktop publishing revolution followed.
As desktop publishing emerged, it displaced bespoke layout systems, and networked PCs displaced proprietary terminals, and SQL databases displaced proprietary storage, but the putting them together into a usable workflow system remained a specialty. In general, the CMS companies got out of the hardware business entirely and focused on software and services.
Photos came later. Keep in mind that the JPEG standard didn't even exist until the 1990s. The first wirephoto storage-and-editing systems were big bespoke monsters that looked like something from a 1950s sci-fi serial, but they were quickly replaced by Mac-based tools, and then the core CMS systems embraced photo management.
Broadcasting trailed all of this in many ways. TV stations actually produce fairly little information in the common sense of the word, and have lighter requirements for handling text, but huge amounts of data in the form of video. When I first worked in TV, video was shot on film, then videotape. As video became digitized and companies like Avid created digital video editors, managing the data became a requirement there as well, and a specialty.
It's now possible to put together a text/image/video workflow system with open source tools. For a single publication, I could do it in a few days with Drupal, and if the Web is the target, it's all pretty straightforward. But the news CMS field is still dominated by specialty vendors.
Print is still a huge driver of revenue, and that means interfacing with advertising workflow and print page layout tools. Adobe InDesign is pretty much the standard there, although I know of one or two systems that have proprietary layout. As a result, a small (and shrinking) number of specialty vendors dominate. They integrate off-the-shelf components, including open source tools and commercial software.
Where I work, writers are using CKEditor, but it's implemented in a proprietary Web-based workflow system that publishes to multiple Drupal sites on the Web and integrates with InDesign for print. Wire service information, agency photos, etc., all come into the CMS.
Because most of the older legacy systems are utterly print-focused, they can be extremely frustrating in a digital world. Some news companies have assemble parallel production systems for the Web, stitching together any number of off-the-shelf components, or writing proprietary code. If you use Django, you should know that it was created at a newspaper company. The Washington Post has created its own system called Arc that it is peddling to other news companies.
If it's the left, just a narrative will do. (Score:1)
If it follows the narrative, they keep and publish it.
If it doesn't, they purge it to keep the narrative.
It's much easier in the US (Score:2)
At The AP 2000 - 2007 (Score:2)
I worked at the AP as a software engineer for seven years at HQ in Manhattan. One system I worked on for a few years was the "Desk" system, which is a set of 3 OpenVMS Alpha clusters (NY, London, Tokyo). This was the primary news collection and dissemination system known to outsiders as "The AP Wire". It accepted thousands of stories per hour from contributors, and transmitted thousands to paying clients. Clients were typically newspapers that received various "feeds" from the AP such as Business, Sport