Zap2It Labs Discontinuing Free TV Guide Service

QuijiboIsAWord writes "Zap2It Labs, which provides free TV listing data for personal use, has long been the main source of program guide information for users in the US and beyond. They've announced via their webpage that, due to abuse of the service, data will no longer be available after September 1st. There is no other direct source, and no option to pay for the service even if the users wanted to. Without a data feed of this type, users will be reduced to scraping websites at best. Is this going to be a killing blow for MythTV?"

This is troubling

[quanticle]

I don't use MythTV, and so I was surprised to see that it relies on a private third-party source for TV listings. Isn't there any way to obtain this information in an "open-source" manner?

Anybody know what the "abuse" was?

[Palmyst]

Their website does not explain. Is just using the data in MythTV, "abuse"?

Achilles Heel

[L. VeGas]

I first became aware of MythTV some years ago from a developer that was excitedly working on the project. With all the effort that was going on at the time, nobody seemed to have a clear-cut idea of a long-term, stable way of getting TV listings. "Scraping web pages" was the only plan.

Looks like five years later, it's still the only plan.

Re:This is troubling

[ajs]

I don't use MythTV, and so I was surprised to see that it relies on a private third-party source for TV listings. Isn't there any way to obtain this information in an "open-source" manner?

I can't imagine how that would work. Ultimately, you need information from the studios, and that's going to require a business relationship. TV Guide has such a relationship, as did these folks it seems (or perhaps TV Guide and these folks have a common feed).

I'm a little shocked that these guys didn't just go commercial, though, and build a MythTV add-on that allows you to subscribe to their product.

Re:Anybody know what the "abuse" was?

[ajs]

I think the "abuse" they're talking about is programs that are pre-configured to hit their service so that everyone on the Net who tries them out hammers their servers.

It seems fair to start charging, but odd that they're just shutting it off. They say they're willing to license to other companies, so presumably they're hoping someone will come along and offer a package to the MythTV folks by licensing the data and re-selling subscriptions.

Going, going, gone?

[scribblej]

I've been a Zap2it subscriber for at least three years for my MythTV.

At first, they made me fill out a big online survey as "payment" for the service. The first time it was about 30 questions.

The third time (this is like every 3-6 months) they only asked one question.

For the last year, the survey has been "click here to renew."

What's with that? I'm willing to give up some personal time and info to pay for this service, and they can't even think of a way to leverage that?

I don't see the problem

[Grishnakh]

What's wrong with just "scraping" web pages (I assume that means writing code which automatically downloads the webpages and captures the data of interest, rather than requiring a human to do it. I do this all the time with Perl code.)?

There's multiple sites out there with TV listings: Yahoo TV, Zap2It, MeeTV (the one I use), etc. Just write perl scripts to capture the listing information from these sites, and modify MythTV to allow the user to choose any service he wishes. Of course, some of these sites may (stupidly) screw with their HTML in order to throw off these scripts, but that's easily worked around with regular updates. So we just need to have a "myth-scripts" distribution site where your Myth box automatically checks for updates to the perl scripts every day and downloads them if necessary, just like we already do with many other things.

No, it's not quite as reliable and efficient as a static interface to this data, but if these companies are stupid enough to remove static interfaces, thinking we're just going to go back to doing everything manually and looking at all the ads, this seems like a reasonable solution. There's no way of preventing automated scripts from downloading webpages.

Re:Myth will survive

[garcia]

FWIW, Myth beats the shorts off of TIVO any day.

How? I had my DirecTivo installed in 2005 and haven't touched it since. When it was installed I didn't have to do anything, someone came out and hooked it up and turned it on. Ever since it's been working like it should w/o me having to fuck with it in the least.

Until MythTV boxes come ready to plug and play for less than $100 (mine was free b/c DirecTV's new firmware (at the time) put up a screensaver that my standalone TiVo recorded instead of the show) it won't "beat the pants off of TiVo".

Re:This is quite bad :(

[Lumpy]

Tribune put new management in Zap2it. They have been unresponsiveand treated the Data Direct service like a redheaded stepchild for a year now. The data has been bad, with long outages on it from time to time for a while now. Many of us that have used myth and other xmltv systems have tried to pay for a subscription for a couple of years now and they refuse.

it's the new management, they hate that OSS people are getting access to the data and want to stop it.

What if I was to write a web service?

[rehtonAesoohC]

What if I was to write a web service that exposed the data garnered from website-scraping? You could just write a standard XML request, wrap it in SOAP tags and send it to the web service, and you'd be returned whatever information you requested- by channel, time, or show name...

Any takers?

Re:Anybody know what the "abuse" was?

[Anonymous Coward]

> I'm not sure how people abused it, other than by maybe hitting too hard. Once a day should be sufficient, you'd think - pick up info for the "new" day as well as changes since the last time you checked. But, if you instead picked up a full two weeks every time you hit it, thrice daily, I can see how that'd be a problem for them...

Maybe they abused it by not hitting it often enough.

Remember, a few months ago, after a long beta, the website version of zap2it finally killed its old tables-and-HTML look to force everyone to use the styles-and-CSS look? And how the biggest difference wasn't the increased standards compliance, but the fact that you suddenly needed to "register" to get a 6-hour view rather than the default 3-hour view, and how all the things that had formerly been done with form submit buttons stopped working unless Javashit was turned on? Gosh, what a coincidence.

It's all about the ad impressions. 6-hour-view website users, hitting SaveAs four times a day (perhaps as infrequently as 28 hits, once a week)... don't make enough impressions to sell banner ads. So no 6-hour-view unless you give 'em an email address they can spam. MythTV users were only hitting the feed once a day, and without banner ads, and producing even less ad revenue.

Why not use Guide+?

[maillemaker]

My TV automatically downloads, somehow (over the air? cable?) channel lineup listings through the Guide+ system.

Could a computer not do the same thing?

Re:What if I was to write a web service?

[soxos]

The hard part isn't writing the code to make the data available, the hard part is getting the hardware, securing a data center location, and paying for bandwidth, all without a revenue stream.

The difference comes down to OSS and Free Services. The same rules don't really apply.

However... we could create a project that would allow for smaller ad-hoc communities and not have to have one site serving the entire internet, just 100 or so users per site...

I'm on board if you want to discuss more (and you got the skillz to pay the billz).

Re:Anybody know what the "abuse" was?

[jez9999]

Who woulda thought that freeloading would result in the collapse of a company's business model? Oh well, doesn't matter, right? I mean the Slashdot model is, "your business model doesn't have a right to succeed", yeah?

Re:I don't see the problem

[MS-06FZ]

What's wrong with just "scraping" web pages (I assume that means writing code which automatically downloads the webpages and captures the data of interest, rather than requiring a human to do it. I do this all the time with Perl code.)?

There's multiple sites out there with TV listings: Yahoo TV, Zap2It, MeeTV (the one I use), etc. Just write perl scripts to capture the listing information from these sites, and modify MythTV to allow the user to choose any service he wishes. Of course, some of these sites may (stupidly) screw with their HTML in order to throw off these scripts, but that's easily worked around with regular updates. So we just need to have a "myth-scripts" distribution site where your Myth box automatically checks for updates to the perl scripts every day and downloads them if necessary, just like we already do with many other things.

No, it's not quite as reliable and efficient as a static interface to this data, but if these companies are stupid enough to remove static interfaces, thinking we're just going to go back to doing everything manually and looking at all the ads, this seems like a reasonable solution. There's no way of preventing automated scripts from downloading webpages.

They can't prevent it - but if they decide they don't like you doing it, they can come up with all kinds of ways to make it hard.

For instance, they could replace their nice little HTML table with some flash code - that's not going to be impossible for you to read with scripts, but it'll be a lot harder.
Another option would be to use images instead of text - possibly even breaking up the images into smaller images, to make it harder for automated tools to pull it and OCR it.
They could load their page with bogus invisible text.
They could provide the listing data in a funky character encoding.

So if these TV listing sites decide that the impact from web scrapers is becoming significant, they have options that can make it very difficult for you.

What happened to Mythtv's paid service, TechnoVera

[digitalderbs]

A year ago it was announced that an alternative paid service through TechnoVera [slashdot.org] was available to replace Zap2It with part of the funds going back to OSS and mythtv -- and no periodic surveys. Couldn't users switch to this service? -- or is it no longer available? (I've never used this service myself.. any users care to respond?)

Re:TV stations are hardly interested in helping...

[Cro Magnon]

Without a DVR, there is zero chance that I'll see any show that doesn't air between 7-10PM on weekdays. For that matter, the odds of me seeing shows that ARE between 7-10PM is slim, since I'm often doing other stuff too. From their POV, they'd probably be better off with me seeing bits of pieces of commercials at FF speed than with me seeing nothing at all.

Re:Myth will survive

[teknotus]

I think the solution is to pressure TV stations to provide listings in something like an RSS feed. Then all you need is a database of RSS feed URL's. Any idea's on how to sell TV stations on how this will benefit them?

Re:Make it a paid service

[Buelldozer]

I was happy to find your clear, concise, comment down here all by itself. It makes it easy for a clean response.

Let's say that MythTV implemented your paid service plan and began charging the princely sum of $2 per month for the data.

I would give it all of 7 days before that paid for data became available for free. Someone, somewhere, would buy the data for $2 per month and load it up for others to have free of charge. It would be a daily torrent that you could pull, or a streamed RSS feed, a static layout site with a downloadable screen scraper, or any one of a dozen other ways I can think.

So now instead of a million dollar revenue stream you'd get a thousand dollar revenue stream coming from the 500 users who would actually be wiling to pay when a free source is available.

If you can answer the question of how to prevent the above scenario from happening I can put you in touch with some content providers who will pay REAL money for your idea. The kind of money that allows people to retire for life...at the age of twenty.

Re:This is troubling

[Reverend528]

Isn't there any way to obtain this information in an "open-source" manner?

The best possible "open source" solution would be to create some sort of wiki for tv listing data.

Re:So what?

[grommit]

The TV market will be transformed by IPTV in 10 years? That's cute. Really. I have no doubt that there will be more people using IPTV in 10 years than there are now (all dozen of them) but to say that it will transform the market is stupidly optimistic. You seem to forget that for better or worse, media companies control the TV market currently and just like they control the music market (hear of RIAA?) they are not willing to change how they do business. Yes, I know there are plenty of places to get indie music but unless you're one of those people that think so highly of themselves that they refuse to listen to any music that becomes mainstream, you are missing out on a lot of music.

It'll be the same way with IPTV for a very long time to come.

Re:Myth will survive

[BobPaul]

MythTV doesn't fast forward, it does commercial skip. Automatically. It hits a commercial break and just jumps past.

Tivo used to allow a really nice FF feature to skip commercials. Now they overlay advertisements on top of the advertisements you are fast forwarding through. Not to mention the advertising in the rest of the UI. If you own one, you should know what I'm talking about. If not, Google found me someone's blog with pictures [blogspot.com]

Re:Myth will survive

[mysidia]

Every time a feed is lost and a new one is found, every time the guide feed changes, every user has to manually upgrade the software, and it's sure to happen many times in the future..

That is inefficient.

What would be ideal would be if myth boxes could connect to each other to form a peer-to-peer network and share some of their guide data with each other.

Let the authenticity of an entry update (in case of conflicting stories) be determined by the reported source of the update, a vote, and the "age" and reputation of the mythtv installations reporting the listing.

In that manner, when one feed dies, any mythtv user who can would be able to scrape or find listings data and provide it for the benefit of all mythtv users.

Also, no one mythtv node would necessarily need the full listings, they would only need to submit updates randomly based on newly acquired data, and to submit queries based on listings they're interested in.

I.E. channels the user is currently trying to lookup listings for, or channels in the channel lineup for the next few days, and searches for program names the user is interested in or wants recorded every time.

Re: how to prevent data "stealing"?

[Anonymous Coward]

Then they'd happily decrypt the data and share it with the world. You're not very informed about cryptography are you? That little module has to have the mechanism (the key and decryption algorithm) to decrypt the data. It's sitting on your computer. You OWN that little module. Source code or not, an intelligent hacker could easily automate the reverse engineering of the module, stealing of the key, decryption of the data, and redistribution of the data. Sorry pal, your plan is full of holes you can't fill. Much larger entities have tried to keep content encrypted (see...oh...THE ENTIRE MOVIE INDUSTRY), but ultimately the consumer has to be able to see the data you're stashed away and thus there's always a hook for an intelligent hacker to subvert your data security model.

Your argument then relies only on the idea that the hacker's effort wouldn't be worth it. That's quite debatable, but flip it around. Where is the incentive for zap2it to sell this data to you and me? Well they'd have to hire some programmers to interface it with their new database system (which is probably the main reason for discontinuation, they refuse to spend money to keep a free service running), programmers to create the subscription payment and distribution framework, programmers for the encryption and mythtv modules. All for what? To have someone within a few days subvert their hard work? Not going to happen. It's obvious the new management over at zap2it (TMS) wants nothing to do with the community or it's free use of their data. They see no incentive to create a paid service for individuals when they obviously already fear piracy of their data. They wish to continue making money from businesses and businesses alone.

The real reason why TMS is killing Zap2It

[Anaerin]

Taken from a post above:

Some reasons and other comments given for the scheduled discontinuence, copied from the forum, copied from the mailing list:

• Continued use of the service to support commercial products, in violation of the agreement.
• Commercial products continued to grow despite previous appeals that this activity stop.
• There are significant changes to the supporting data structure forthcoming and we could not devote resources to the continued upkeep and enhancements of the service.
• Maintenance of the service is impacting our resource pool for other projects.
• We sought alternative options but were unable to find a solution.
• We recognize the hardship this creates for the user community. We are open to alternative solutions and would consider proposals that met the needs of the user community and our company.
• We looked into options to turn this into a paid service however we do not have the infrastructure at this time.

(NB: Emphasis mine)

Basically, what they're saying is that they will be changing their database structure, and can't be bothered to re-code the Labs.Zap2It part. They'd rather spend the time on other things, like increasing ad revenue.

Bear in mind that TMS charges each station (Somewhere in the region of $75/day) to collect their data, and charges their commercial end-users (somewhere in the region of $500/market/month) to provide the data. Quite a lucrative trade, wouldn't you say? Add to that advertising revenue from their site, and subtract bandwidth fees, and they still make a tidy profit.

It has been theorised that the main reason labs.zap2it was set up was to save bandwidth costs after XMLTV scrapers started hitting their public site. As the data was served up in chunks in the midst of a sea of other information (Links, advertisements, commentary, navigation etc) it cost considerably more in bandwidth, so it made more fiscal sense to offer the data for free, so they could track who was using it and where, along with only having to serve the data itself, with compression and selectivity. This is only a theory, mind, but considering how the Myth community (On it's own, without any of the other projects that were/are using Zap2It feeds) has grown, I think removing this option will drastically increase Zap2It's bandwidth bills without adding any ad revenue back into the pot (A scraper doesn't see ads, and doesn't care about them).

In other words, this could be a costly mistake for TMS. Here's hoping they see sense, and work out a way to work with MythTV and others.

Re:Rather than a Million Screen Scrapers...

[cpt kangarooski]

Even if a compilation is not creative, it can still be subject to contract or trade secret law.

True. However, in the case of tv listings which have been published online, they're clearly not trade secrets. A contract is possible, but AFAIK none of the sites that have the listings bother with them. Specht v. Netscape would be instructive in such a case.