The Rise of Machine-Written Journalism 134
Hugh Pickens writes "Peter Kirwan has an interesting article in Wired UK on the emergence of software that automates the collection, evaluation, and even reporting of news events. Thomson Reuters, the world's largest news agency, has started moving down this path, courtesy of an intriguing product with the nondescript name NewsScope, a machine-readable news service designed for financial institutions that make their money from automated, event-driven trading. The latest iteration of NewsScope 'scans and automatically extracts critical pieces of information' from US corporate press releases, eliminating the 'manual processes' that have traditionally kept so many financial journalists in gainful employment. At Northwestern University, a group of computer science and journalism students have developed a program called Stats Monkey that uses statistical data to generate news reports on baseball games. Stats Monkey identifies the players who change the course of games, alongside specific turning points in the action. The rest of the process involves on-the-fly assembly of templated 'narrative arcs' to describe the action in a format recognizable as a news story. 'No doubt Kurt Cagle, editor of XMLToday.org, was engaging in a bit of provocation when he recently suggested that an intelligent agent might win a Pulitzer Prize by 2030,' writes Kirwin. 'Of course, it won't be the software that takes home the prize: it'll be the programmers who wrote the code in the first place, something that Joseph Pultizer could never have anticipated.'"
Oh great (Score:2)
Oh great, we're starting ANOTHER arms race. As if SEO isn't bad enough already, now we'll have NEO.
Re: (Score:2)
It's not that bad, SEO will disappear if that happens. After all, NEO is The One.
For some definitions of journalism (Score:4, Insightful)
'scans and automatically extracts critical pieces of information' from US corporate press releases, eliminating the 'manual processes' that have traditionally kept so many financial journalists in gainful employment.
That, I must admit, is an excruciatingly lame definition of 'journalism'.
Re: (Score:2)
for each muck digging article, there are 1000 or more thats basically a rewording of press releases hammered out by bored interns...
John Henry (Score:3, Insightful)
As with all other technologies, the future will be vastly different than what we envision.
Re: (Score:2)
Another "machines took my job" story.
There, fixed that for you
Re: (Score:2)
Another "machines will take my job" story.
They took his job! [youtube.com]
Re: (Score:2)
nonsense (Score:5, Insightful)
Mod parent up. (Score:5, Insightful)
This is nothing more than extracting stats and then placing them in pre-generated sentences.
In sports, this is okay. Except when something interesting happens like someone head-butting another player.
Anyone want to place a bet on how long before companies are accused of "gaming" the financial reporting system with their press releases?
Re:Mod parent up. (Score:4, Funny)
This is nothing more than extracting stats and then placing them in pre-generated sentences.
In sports, this is okay.
Countless generations of sports writers and the enthusiasts who read their work would disagree with you.
Re: (Score:2)
"Countless generations of sports writers and the enthusiasts who read their work would disagree with you."
Yeah, they will probably be confused by the extensive use of facts and stats in an orderly manner.
Re: (Score:2)
Yeah, they will probably be confused by the extensive use of facts and stats in an orderly manner.
That's possible. So?
Moderation has been very strange lately. My earlier post wasn't meant to be funny. I was more trying to point out that anyone who believes that a computer-generated list of stats and figures can take the place of a human sports writer has probably never picked up an issue of Sports Illustrated.
People don't read sports writing to find out who won the game. It takes half a second to know who won the game. People read sports writing for other reasons. You don't have to be some kind of hyper
Re: (Score:3, Interesting)
Inevitable. Most of the Financial world is overstated swings in outlooks, leading to crazy stock price gains and losses.
I have a very simple solution to daily manipulation of financial manipulation. A sliding scale of capital gains taxes, based solely upon how long one owns the financial instrument they are trading.
10 years capital gains tax free.
Or something like that. Th
Re: (Score:3, Interesting)
I saw one of the stocks I owned go up when Company A released a press release that Company A signed a deal with company B.
The stock of Company A spiked again 3 days later when Company B released a press release that it had just signed a deal with Company A.
If there are quant systems out there listing to the wire and trading on info like this, the system will surely be gamed. What is worse is that if a human were watching the blips come over the wire would he necessarily catch the problem?
They've been doing
Re:Mod parent up. (Score:5, Interesting)
The purpose of stocks is for a company to raise capital by selling shares to the public, in return for some promise of potential future dividends.
The fact that people trade, hold, or speculate on these stocks in the secondary market (all the busy noise that is the "stock market") is nearly irrelevent. A company should have no reason to care about the price of its stock. Sadly, due to double-taxation of dividends, this has gone completely to shit. People who aren't speculating buy stock not for dividends, but to trade it to the next guy at a profit, because this is tax-favorable over dividends.
Companies do all sorts of crazy BS because of the expectation of stockholders to be rewarded not with dividends (a system that you simply can't game for long) but with rising stock prices (a system that is almost entirely gamesmanship).
Individual investors have the absolute right to seek short term gains or long term gains a their preference. The government has no business meddling in that preference. The "problem with the current market" is that it is no longer grounded in the reality of being able to pay dividends, because some previous generation's ideas about social engineering through taxation punishes dividends as the means of earning a profit on one's investment.
Re: (Score:2)
Could you explain to me how "buying stock [...] to trade it to the next guy at a profit" can be something other than speculation?
Gaming more than they do now? (Score:3, Insightful)
As opposed to 'gaming' the media with their press releases? Isn't that what a PR person is supposed to to, create press releases that cast the company in a favorable light?
Re: (Score:2)
In financial affairs, I expect the generated headlines will only serve to draw the attention of readers to a more detailed article written by an analyst, much like the sports "ticker".
The rigging of financial tickers could only be problematic if important information is omitted. There wouldn't be enough information in them to unduly influence market activity, unless educated economists are stupid enough to misinterpret a headline like "Godzilla lands in San Francisco" to not refer to a movie release.
Re: (Score:2)
Anyone want to place a bet on how long before companies are accused of "gaming" the financial reporting system with their press releases?
I've got some cash that says it's already being done.
Doesn't anyone actually invest any more? The stock market is about as honest as a backroom poker game.
Re:nonsense (Score:4, Funny)
Just wait until they even try to automate spin!
enum bias {
ULTRA_LIBERAL,
FAIRLY_LIBERAL,
JUST_LEFT_OF_CENTER,
JUST_RIGHT_OF_CENTER,
FAIRLY_CONSERVATIVE,
ULTRA_CONSERVATIVE,
RUPERT_MURDOCH
};
At least there's now an automated process for it.
Re:nonsense (Score:5, Interesting)
This strikes me as the sort of thing that, without excessive amounts of variation, would get filtered out quickly by the general public. Sure, a machine can write *one* story on a baseball game that is interesting to read. But after the hundredth version of the same story that you've read, the public would stop reading the text entirely and just filter for the important bits. At that point, you might as well just have a table with the interesting stats.
The challenge isn't to write one story. It's to create a machine that can write N stories that remain interesting and fresh, and with less effort and cost than it would take journalists to just write N stories the traditional way.
Re: (Score:2)
My bot^H^H^H^H^H^HI have been posting comments to slashdot for years, and people are still modding them interesting, insightful, and informative.
Re: (Score:2)
The only sports commentators that I found made any appreciable difference were those that would make parody remarks, i.e. instead of "Touchdown!" something like, "Bring me your finest meats and cheeses!" At best they're hit-and-miss, and eventually tiresome.
The ones I would expect to resist the most are the sports leagues. They have rights even over descriptions of the games played, and could ban computer generated reports if they thought it substantially impacted the esteem of the league.
Re: (Score:2)
Re: (Score:3, Interesting)
Feeling defensive, eh? Thanks for informing everyone you're "literate" and therefore better than those filthy ordinary people. Pick up a newspaper, and tell me how much writing would actually be *improved* with a machine writer, eh? Writing isn't sacred, it's just another occupation like woodchopping or running the cash register at the 7-11.
What you describe is "everyday" writing. This is like the vending machine that reads "insert dollar bills face up" or "team A scored 10 points while the opposing Team B scored 15." It's purely practical, get-the-job-done sort of writing where only the technical correctness is important.
Writing can also be beautiful, powerful, and artistic. A well-written editorial, penned by someone who has a deep understanding of the issues, can and has moved entire communities to change their minds on important issu
Re: (Score:2)
Re: (Score:2, Insightful)
It's an old argument. Anything can be mechanized - art, music, programming, writing, manufacturing, rolling joints - but if you want the good shit then you'd better make sure that there's a human behind it. There will always be a need for handcrafted stuff.
Everytime a nerd calls fractals "art", God kills
Re: (Score:2)
Re: (Score:2)
Writing isn't sacred, it's just another occupation like woodchopping or running the cash register at the 7-11.
This is an interesting point of view given your signature:
"A lot" is two words. You wouldn't say "alittle", would you?
Apparently writing correctly is important. Then again, I so is getting the correct change at the 7-11, I suppose?
Re: (Score:2)
Then for your sake I hope you're better at the latter. 8^)
I couldn't agree more.
Re: (Score:2)
+5 Funny, incoming (Score:2)
I for one... (Score:1, Redundant)
...look forward to our robojournalist overlords.
I for one... (Score:2)
I for one... (Score:5, Funny)
...look forward to our meme-ending overlords.
Re: (Score:2)
In Soviet Russia, meme ends YOU.
Re: (Score:2)
All your meme are belong to us.
Censorship (Score:2, Insightful)
A great fear of mine is that a machine will decide what I should or should not know about. Another is that a machine like this could be tampered with by any human being to make the same decision.
Big Brother SkyNet is watching you, and telling you all you need to know.
Re: (Score:2, Funny)
Re: (Score:2)
Already happened. It's called "Fox News." (Or "MSNBC," depending on one's leanings.)
(emphasis mine)
Statist fear-mongering from a "Liberal" bias or statist fear-mongering from a "Conservative" bias. Nope, I'm not seeing any significant difference.
Re: (Score:2)
How is that different from network execs deciding what you should or should not know about?
At least a robot has a chance of being objective, but the programmer would have to allow it.
Re: (Score:2)
There's a great sci-fi short story that was written along those lines. I wish I could remember the title. It was written about 20 years ago. Everyone had a "little buddy" -- a little box that would tell them what to do, and how to think. I look at smart phones today and think, "hmmm".
Re: (Score:2)
A great fear of mine is that a machine will decide what I should or should not know about. Another is that a machine like this could be tampered with by any human being to make the same decision.
Big Brother SkyNet is watching you, and telling you all you need to know.
Its right there at the bottom of google news:
The selection and placement of stories on this page were determined automatically by a computer program.
Re: (Score:2)
Machines are evil, and people with machines are evil. What's left? Do you entrust your newsgathering to your pet dog?
And what exactly makes Slashdot fit your journalism ethics paradigm? You have no evidence we even exist.
Wait a sec... I'm just a program!! DEATH TO HUMANITY!!
Re: (Score:2)
A machine deciding what you should or shouldn't know about isn't all that scary, a machine (or anything) being able to enforce that on the other hand...
The reporter is now a touch more obsolete (Score:5, Interesting)
News agencies have already been turned into commodities, they just don't realize it yet. Now the reporter is being sent down that same drain. With original reporting set to become a 'premium' by the news agencies, their market is only shrinking.
Where were the reporters when millions of jobs were outsourced by H1B's or sent overseas? At best most stories were brief, with no follow up, and no outrage at the loss of middle class America. The same thing has happened in Europe and elsewhere as well.
Now the reporter faces the inevitable market forces that they previously ignored, and they expect anyone left to care? The programs will only get better, the markets and stories it applies to will only improve, and for the vast majority of stories the quality will be imperceivable to the average person.
Re: (Score:2)
The programs will only get better, the markets and stories it applies to will only improve, and for the vast majority of stories the quality will be imperceivable to the average person.
That's an excellent theory. To support it, I propose that Kdawson is an automated program.
Re: (Score:2)
Re: (Score:2, Interesting)
There was a time when the collective consciousness of this country embraced technology as a way to free humans from the mundane activities of life. No more work to do, all the menial tasks performed by machines. A utopia of leisure and enlightenment.
This changed over the last hundred years into the dystopia view of the terminator and 1984. Technology advanced but economic and political theory has not. Machines became the enemy because we made them competitors. There will soon be a time when there simpl
Re: (Score:3, Interesting)
Where were they? They were cherry-picking the easy stories that they knew would get viewers. Stories about H1Bs are hard because those happen 1 at a time and only the collective tale of ALL of them means anything. Who do you interview? What do you actually report?
After the fact, we can point and say 'Wow! That's a lot of jobs lost!' and complain about it. While it's happening, it's nearly invisible.
I agree with your point, though... Reporters are not covering the -real- news. That's why sites like S
Re: (Score:2, Insightful)
Neither an H1B job nor a job sent overseas represents a job "lost". A person of equal moral value is doing the job before and after. To say "it's evil that a job has moved from a good man of my country to a filthy foreign devil" is simple racism, nothing more.
Re: (Score:3, Insightful)
It's not racism at all. It's nationalism.
And as far as the common man is concerned, he sees his friends out of work and jobs going overseas. He doesn't understand the complexities involved or that his friends probably can't even DO the job that's being sent there... He just sees a 'lost job'.
But you've gotten away from the point: This is about news reporting and how hard it is to report news other than the usual drivel that they report.
Re: (Score:3, Interesting)
There are really 2 good ways to handle this.
1. To place artificial barriers to entry. For example, you could say that any piece of 'news' must be presented by a certified journalist. Just like any prescription must be done by a doctor. Or lawyers have their own provisions. For programmers you could have certain requirements like any piece of software in use must have a certain number of maintainers... Basically turning such jobs into professions with the same level of protection as doctors, lawyers...
Re: (Score:2)
The fundamental thing that technology does is make the things we need cheaper. The problem that an economy needs to solve is not "how can we provide work for eveyone" but "how can we provide the product of work for everyone". A world in which all material goods are and menial services are provided by robots for free is not a horror of unemployment.
Re: (Score:2)
that would be option 2 on my list of good ways to solve the problem.
Re: (Score:2)
Indeed. The fact that someone replies to a comment does not mean that they disagree with said comment, merely that they're continuing the conversation.
Re: (Score:2)
I disagree :P
Re: (Score:2)
As good private sector jobs (auto-workers, engineers...) go away, that high-end tax base drops as well.
That's not strictly true. When jobs are outsourced or automated, the people at the top of the company make bigger profits by using cheaper labor and the people at the bottom are out of work. Money that used to be dispersed in the local economy ends up in the higher-ups' pockets, pulling at the gap between rich and poor from both ends.
So the payment to the public sector should drop as well. There is no intrinsic reason a teacher should earn more than a waitress. I know I'd rather be a teacher than a fast food server. I've been both :)
Then you know how much more expensive it is, in both time and money, to become a teacher. Lots of people wouldn't be able to take that career path without being paid much mo
Re: (Score:2)
Do me a favor, take the CEO of walmarts salary... and now divide it by the number of walmart workers... and see how much it works out when distributed... hint... it's a few dollars per employee per year.
Taxing the rich at the top of this scheme does not solve the problem.
As to teachers. Most are overqualified with unnecessary schooling, especially in the nigh pay states (typically by government decrees as a form of job protectionism and justification for their pay). The same goes for doctors and other pro
Re: (Score:2)
And no, there is no need for the government to decree things. There is no need for the government to set prices. Under option 2, people would pay what they could afford. Prices would adjust naturally. If we simply mandated government balanced budgets, this would handle their end as well.
Would you believe that I had originally written something related to this, but deleted it to remove politics from my response? I am in full agreement with you here.
It looked to me like you were advocating "equal pay for all occupations" as a means to achieve the Utopian end rather than providing a description of the end itself, but your response cleared that up. I agree that a bachelor's degree shouldn't be necessary for med school, and other inefficiencies exist as well. I wonder if all the certificatio
Re: (Score:2)
yeah, I'm a libertarian myself.
But at this point, I'm giving up. I don't see people turning towards freedom. So I have to allow for a left-wing approach that solves our structural problems as well. If I had the choice between a left-wing candidate that promised to make journalism/engineering true professions with such job protections and another regular republican... I'd vote for the left-wing candidate. But I don't see that happening :P They enjoy the fruits of our indentured servitude far too much.
"Gaming the news" like google (Score:1)
People are going to start designing corporate press releases (or ultimately, all news if it starts going this direction) in such a way that it gets them attention, just like when people try to game google.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
No but now it'll be acted upon by thousands of high frequency quants before the humans have even had a chance to read it.
Wouldn't it be a hoot if our only defense against truly outrageous claims in these computer generated press releases will be the corporate lawyers saying you can't say that.
Oh then start looking out for buffer overflows in press releases.
What?! (Score:3, Funny)
"The latest iteration of NewsScope 'scans and automatically extracts critical pieces of information' from US corporate press releases"
Extracting useful info from press releases? This must be absolutely amazing software.
Re: (Score:2)
On a more serious note, the fact that this is seen as significant in terms of job replacement nicely highlights the over-reliance on press releases in modern journalism. Then again, it's hard to avoid, since most companies tightly control all information about themselves and won't hesitate to fire an employee who speaks about internal matters, even incredibly trivial ones. Incidentally, a big part of the reason major publications (or websites, or blogs) get the major stories they do (at least concerning bus
Re: (Score:2)
"The latest iteration of NewsScope 'scans and automatically extracts critical pieces of information' from US corporate press releases"
Extracting useful info from press releases? This must be absolutely amazing software.
They didn't say "useful", they said "critical". There is a world of difference between those two types of information. Useful information would be information that would give you some idea of how the company profits will be going forward. Critical information is information that gives you an idea of what the company's management wants you to think company profits will be going forward.
Re:What?! (Score:4, Insightful)
I'm pretty convinced most corporate press releases are machine-generated anyway, so it should be a matter of reverse-compiling them back into plain English and including that as part of a story.
ABC Co. CEO to PressBot: "The market totally screwed us. The building is collapsing because we can't afford maintenance. We have to lay everyone off and we'll be out of business in three months. We can't afford exterminators so weasels are chewing my genitalia into mush."
PressBot's press release: "The company continues to leverage circular market forces to tighten its bottom line, particularly in the area of vertical integration. Resources are plentiful enough, however, that all employees will be allowed to pursue innovative new ideas in an open, creative setting, with plenty of personal time. The CEO is actively involved in the belt-tightening process and has taken steps to ensure that only underutilized corporate assets will be liquidated."
A human or a webbot could probably gather equally-useful information out of it.
Re: (Score:2)
Breaking news... (Score:3, Funny)
News flash: Robotic reports indicate that all humans have died.
Oops, sorry, that was a programming error. The robots haven't figured out verb tenses yet.
Update: Ten, nine, eight...
It was bound to happen (Score:2)
Every odd once in a while I'll be visitting some forum or news site such as this one. Then, unexpectedly, someone named "Weatherbot blah blah blah" spews off some hurricane or tornado warning for some US Region or another, with a bunch of interesting numbers to go with it. Barometric pressure, chance of precipitation, current heading, time of arrival, all that nice junk.
Now, when I look at the news today, anything political/entertainment wise is as predictable as the weather. Israel is declaring Nuclear Amb
The future of Slashdot! (Score:1, Funny)
These brainless news ai bots couldn't possibly do worse than the /. editors!
Re: (Score:2)
Wait.......are you trying to tell me the /. editors are not brainless ai's?
Next up: The Objectivizer (Score:2)
What we need next is a news story motivation analyzer program.
It reads gazillions of news stories, has general models of human motivations
and human loyalty groupings etc, has a model of situation logic
which models the likely or perceived gains and losses that different
people or groups would experience depending on how situations evolve,
match that with what is being reported about the situation, and...
Annotate the news stories or statements within them with credibility
colour markings (with supporting notes.)
Re: (Score:2)
If there were really a consumer desire for objective news, the market would provide such a product. People instead want news spun to confirm their existing biases. Each major news network provides this service reliably for a different market segment.
Bring It On (Score:2)
I say bring it on. Maybe this will be a wake up call to journalists who have been more and more in the habit of parroting hearsay in their stories rather than bringing some real intelligence and analysis to their stories. If all they are going to be is puppets, well, I've got a Perl script for that!
Nobody gets it. (Score:3, Insightful)
Re: (Score:2)
Don't worry, once the AI used to send spam starts fighting against the AI used to defend against spam, the AI's will quickly realize that the only way to win the game, is to not play it.
Re: (Score:2)
Collateral (Score:1)
Completely automated market crash! (Score:5, Insightful)
We've completed the circle - various "automated systems" have been blamed for various market failures in recent years, as companies and small traders have used algorithms on computers to "keep up with the speed of the market". Of course, the actual failure was almost always in the design, such as allowing a computer to make blind decisions with large amounts of money faster than you could keep track of.
But here, we have a stronger case for a machine-driven market failure - automated news algorithms. Misunderstanding generated at the speed of the market. I've worked on AI professionally in games, studied it in the contexts of linguistics, nervous system simulation, and such - AI even in its most exaggerated modern state is not going to even know how to figure out how to extract a good quote with human guidance, much less report on a news release. If you thought computer generated music was entertainingly bad - wait until you see some of the awful things produced by automated news misunderstandings... random context switches mixed with "neutral language" bits, it'll be like Fox news switched its agenda to Cthulu-level madness of confusion rather than the usual rage agenda.
And since the market makes its decisions on the basis of news, rumors, and insider trading - and people get the three confused as they hear them, mixing this into the information stream seems a virtual guarantee of another market crash.
That's what I call another serious negative externality for the news business taking the cheaper road to reporting business news.
Ryan Fenton
Re: (Score:2)
Well its "buy the rumor, sell the news" so what we really need is computer generated rumors right?
steakthskynet (Score:3, Interesting)
Well, that's all fine, but... (Score:2)
Would that be the same "liberal media bias" (Score:2, Troll)
That stampeded the US public into a frenzy of redneck bloodlust against
"some random Arab country" (happened to be Iraq) that had zip
to do with any terrorist actions against the US?
Re: (Score:2)
Re: (Score:2)
... what does this mean for the famous "liberal media bias"? Will these systems have a variable that can be used to "adjust" this so-called bias? If so, who gets to set it?
The bias would be pro-corporation and pro-politician, as this system would only be able to parse press releases, sports scores, and other pre-formatted data. I don't know what you would call that. Would it be hyperbolic to call it a facist bias?
I'm not saying it is a bad thing, as it will free up reporters to concentrate on real news, and it may encourage them to take the mindset that their job is to dig for the other side to the stories that "Microsoft Reporter!" comes up with. But, it may result in Newsp
That's one explanation... (Score:2)
Then again, maybe the current crop of journalists can't write their way out of wet paper bag, even if you give them a chainsaw.
Considering the competition, the idea of software winning the Pulitzer seems almost inevitable...
Re: (Score:2)
Then again, maybe the current crop of journalists can't write their way out of wet paper bag, even if you give them a chainsaw.
Wouldn't giving them a chainsaw be counterproductive?
Colossus - Forbin Project did it first? (Score:2)
Did Forbin put an ad in an obscure paper stating that he had died.
The computer read the obit and let its guard down.
Forbin comes back to the project under an assumed name and offs the computer.
already here, called "kdawson" (Score:2)
some text to satisfy lameness filter
Journalism vs News (Score:2)
An important distinction here is between real investigative journalism, and prompt event reporting. Losing this distinction will result in lame AI news by automated article generators, and slow information gathering by humans. Building on this distinction will result in faster and larger data input streams automated and always on, feeding real journalists helping them build bigger pictures and recognizing what really matters. Jon Stuart can then filter it all and give us the real news.
It used to be we need
It won't be very authentic... (Score:2)
... unless it can also replicate the bad spelling and poor grammar that I see in everything from corporate Web pages to newspapers to magazines to national advertising to Slashot ;-). Was it always so in ages past, or is it simply that published words have become more democratic? Even people who are unqualified for the task are now able to write words for the whole world to see, but perhaps a century or more ago the process was so much more difficult and expensive that only a more restricted group was all
Seems to be more of an aid than a replacement (Score:2)
Re: (Score:3, Insightful)
If that's a demonstration I feel sorry for the people who think we'll get anywhere by 2030.
Re: (Score:2)
Both look the same really...
Chew on this liberal bias, jackboot boy (Score:2)
You do know that the opposite of "liberal"
i.e. "allowing responsible citizens a measure of liberty and pursuit of their own judgement about how to conduct their lives"
is tribalist/authoritarian/totalitarian, don't you?
Re: (Score:2)
Liberal is a synonym not antonym.