Zap2It Labs Discontinuing Free TV Guide Service

QuijiboIsAWord writes "Zap2It Labs, which provides free TV listing data for personal use, has long been the main source of program guide information for users in the US and beyond. They've announced via their webpage that, due to abuse of the service, data will no longer be available after September 1st. There is no other direct source, and no option to pay for the service even if the users wanted to. Without a data feed of this type, users will be reduced to scraping websites at best. Is this going to be a killing blow for MythTV?"

How is MythTV dead?

[PolyDwarf]

... Without a data feed of this type, users will be reduced to scraping websites at best. Is this going to be a killing blow for MythTV?"

I fail to see how, unless the screenscrapers specifically mentioned don't work. I wrote a scraper for my MythTV box to pull PPV information from somewhere, since it wasn't coming from Zap2it. It's not that hard, really, and took me all of an hour to do.

TitanTV

[rlp]

I use a Windows based PVR that works with the Web site titantv.com Clicking on the 'record' button on the Web site sends a small file that a helper app converts to a Windows scheduler entry. I can't believe it would be that hard to adapt to Linux.

The Titan TV web site includes advertising and also does tracking. While personally, I don't care if anyone knows I watch StarGate and Myth Busters, privacy issues may be a concern for some.

Re:This is troubling

[QuijiboIsAWord]

The Zap2It Labs service receives all the data from the networks, typically directly, and aggregates it all into a customizable feed which users set up by signing up for an account and selecting the cable provider or broadcast area of their choice.
Many different applications, besides just MythTV, use this data. Some people have individually negotiated with their local stations to get the data they need, but that's extremely hit or miss, and about as convenient as attempting to negotiate with your Cable company to get them to carry broadcast networks in HD unencrypted - Not very, and most people in most areas won't bother.

Re:Anybody know what the "abuse" was?

[ChaosDiscord]

Based on their previous complaints and this message, I think the problem was people were using the free data set, then redistributing it, probably for profit, possibly indirectly (say, selling devices that used Zap2It's free service). Zap2It makes money selling their data set and they were very generously offering it for free to individuals. But you weren't allow to redistribute it.

The main listings

[DaveWick79]

It doesn't appear that this will affect their main listings on their website, just the programs that tap into their database.
I've always just setup an easy shortcut to their web page to get listings, so I really don't think this will affect my usage in the least.

Re:Anybody know what the "abuse" was?

[Minwee]

According to a posting on mythtv.org... [mythtv.org]

Some reasons and other comments given for the scheduled discontinuence, copied from the forum, copied from the mailing list:

• Continued use of the service to support commercial products, in violation of the agreement.
• Commercial products continued to grow despite previous appeals that this activity stop.
• There are significant changes to the supporting data structure forthcoming and we could not devote resources to the continued upkeep and enhancements of the service.
• Maintenance of the service is impacting our resource pool for other projects.
• We sought alternative options but were unable to find a solution.
• We recognize the hardship this creates for the user community. We are open to alternative solutions and would consider proposals that met the needs of the user community and our company.
• We looked into options to turn this into a paid service however we do not have the infrastructure at this time.

MythTV devs are working on this...

[cesman]

http://www.gossamer-threads.com/lists/mythtv/users /275533#275533 [gossamer-threads.com]
And it isn't just MythTV that uses the guide data provided...

Re:Anybody know what the "abuse" was?

[Applekid]

While no longer linked from the front page, this link [tribune.com] still works.

Basically, the content was provided free provided it remained for non-commercial use. After all, commercial ventures have to pay for those listings and if they could get it for free, nobody would pay.

I hope they at least tried to weed out the abusers before just cutting the cable.

It has another source

[TinheadNed]

MythTV can read the broadcast schedules on the airwaves - see EIT [wikipedia.org]. At least that's what I use in the UK. I can also still scrape the Radio Times website in XMLtv as well.

Re:Myth will survive

[gurps_npc]

MYTH TV offers everything TIVO does plus:

Not having to pay a monthly fee to automatically record shows you watched previously

Not having to deal with commercials inserted when you fast forward 30 seconds.

Now, if they could only find a way to strip thos FREAKIN' RUDE graphics that certain vile channels overlay on top of the movie/show. I can deal with a tiny little station identification, but those rude obnoxious ads obscure vital parts of the show about one out of every 3 times. I swear my Blood pressure goes through the roof.

Re:Myth will survive

[Anonymous Coward]

You forgot being able to watch a recording on your laptop while on the toilet. I'm puzzled as to why MythTV doesn't advertise this feature a bit more, since it's one of my favorites.

You can do this with Tivo out of the box.

Re:No it doesn't

[norton_I]

Yes. Also, myth has content-based commercial detection using blank frames, audio dynamic range, and scene cut rate. For most programs, a single button jumps an entire block of commercials regardless of the length. You can set myth to automatically cut commercials, and literally never see another commercial at any speed. It doesn't do very well catching very short segments such as the "moment of zen"/credits at the end of the daily show, so I don't do that, but the auto-skip is far better than tivo's FF overshoot compensation.

Re:What if I was to write a web service?

[soxos]

No, that seems like a pretty good start. Like I was talking about, if we don't try to create a Zap2It replacement, but instead, small communities that can run the same codebase, then this could work without having to have a humongous server. Check this blog entry where we can create a discussion about creating a new project for this [patf.net] Pat F.

But EIT has limitations

[LionMage]

Excellent link. However, using EIT has at least one drawback:

Unlike DVB systems, an EIT in ATSC covers a limited time span. [...] It's possible to transmit up to 128 EIT tables for any virtual channel, but it's very rare that so many are transmitted - terrestrial systems have to transmit EITs covering at least the current time slot and the next three time slots.

So at least in the U.S., the EIT would be of limited utility to MythTV users, since it is unlikely that one would be able to use this data to schedule multiple days in advance. Also, the article you linked only indicates that EIT is a standard part of ATSC (the terrestrial digital TV broadcast standard), and wouldn't necessarily be present in analog broadcasts (which are going to be around for a little while longer) or in digital cable transmissions (which use QAM here in the U.S.). It's entirely possible that a broadcaster might supply EIT data, and a cable operator might strip it out to save some bandwidth.

I've seen some of the newer LCD HDTVs sold here in the U.S. which have the ability to display program guide data, no doubt culled from available EIT data, but this is really intended to allow viewers to see what's on right now or in the immediate future without resorting to channel flipping. I bought a Sharp Aquos one generation before this feature became standard, so I missed out, but I really don't feel like I'm missing too much because of the described limitations.

Re:Google should provide a WebService

[Anonymous Coward]

There are yahoo listings [yahoo.com] and msn tv listings [msn.com]. However, if Google could do it better (which they can) and provide it as a service - they would again win the day. I second your motion sir, Snap2it Google!

Re:No it doesn't

[sfbiker]

To be fair, your Tivo costs $100 + $179 = $279 if you keep it for a year. If you keep it for 2 years, then it's at least $100 + $299 = $399. Prices are higher if you pay monthly.

Subscription rates are here: http://dynamic.tivo.com/2.0.plans.step.2.asp?box=s eries280hrDTDVR [tivo.com]

I doubt you can build an equivalent MythTV box for $279, but perhaps for $399? A quick google search couldn't find any for sale -- does anyone sell preconfigured MythTV boxes?

Re:No it doesn't

[jedidiah]

...for small values of "just works".

I can tweak my recording rules in ways on the Myth that can't be matched on my Tivo. Really easy and cheap to add storage too. Plus I can compress using MODERN compression algorithms and avoid the blockiness of Basic quality.

Lack of an mpeg4 unit is another instance of that laurel resting I mentioned earlier.

Re:This is troubling

[Line_Fault]

They are commercial!!

Zap2It is "Tribune Media Services, Inc", they provide "bulk" guide data to large TV services, such as Telco's(IPTV) and Satellite providers.
http://www.tms.tribune.com/products.html [tribune.com]

P2P Listings

[Doc Ruby]

TV listings seems like a great P2P app for MythTV. Listings for future broadcasts can be read by users from their local media.

In small entries (enforced by the GUI) that don't trip the copyright of the original publisher. Or, like most P2P users, they'll ignore the copyrights.

It's like a Napster index that doesn't point to copyright violated media, but to fair use of one's own media subscriptions.

So you can automate scanning the data.

[DrHow]

A number of the comments here seem to take the view that one source of listings is as good as any other. However, the power of getting the listings data (as opposed to a look at formatted listings) is that you can then use software to read that data and find all the programs that might interest you. The XMLTV package itself offers a number of tools for scanning the data and selecting programs based on rules you specify. There also exist other application programs which have sophisticated search capabilities for treating XML program data. The number of different TV stations I receive now is staggering. I would waste much more time finding the programs I want to watch if I did not have software that automatically weeds out most of what I would not care to consider. Furthermore, there are programs, which would interest me and which I would enjoy watching, but which I would miss if I did not have the tool to spot them for me because it would just take too much time for me to scan for such shows in the listings for all the stations which I rarely watch.

Re:No it doesn't

[toleraen]

I don't suppose you expect to get your TiVo service for free, do you?

You might want to read the summary, or even the title. From what our fine /. editors have implied, you won't be getting your MythTV feeds free either.

TCO indeed...when you figure in TCO for MythTV, you are taking into account the risk you take in your reliance on a free service that you have absolutely no control over, right? Because that's kind of necessary.

Re:But EIT has limitations

[martin_b1sh0p]

Actually, if you use TV Guide On Screen, it's 8 days worth of info here in the states (not just the next three time slots). Not as good as Tivo's 2-3 weeks but still better than nothing. I know this because I have an LG LCD TV with a DVR built in downstairs which has the TV Guide as well as a Tivo in the bedroom upstairs.

http://en.wikipedia.org/wiki/TV_Guide_On_Screen [wikipedia.org]

http://www.tvgos.com/ [tvgos.com]

Re:No it doesn't

[pakar]

http://www.xyzcomputing.com/index.php?option=conte nt&task=view&id=589&Itemid=0&limit=1&limitstart=6 [xyzcomputing.com]

There you have a short story about someone that built a box... and about the prices..

VIA EPIA board - $100 (these are a bit expensive, so it might even be cheaper to go with some low-end motherboard with built-in gfx card and a AMD-Sempron..)
128Mb RAM - $30 (128Mb is quite enough for a pure mythtv box)
HD 160Gb - $40
TV-Card (DVB-T) - $80 (With a CI-Slot)
Case - $50

hmm... so that's $300, and with this you have full control...... Want to have a HDTV capable box, then just go with a bit more expensive EPIA board with DVI output... Want to have more than one PCI card, maybe 2 hybrid tuners, then go with a case that supports a raiser card.

So there you go... with $300 you can have a simple tv-box, add a few more $ and get dual-tuners, add a few more and have a HDTV capable box... add a few more and control you X10 home-automation system..
Don't like the remote? Well, get a bluetooth dongle for the system and use your cellphone, or just buy some other remote...

This is the charm with mythtv.. You can do just about anything if you want to.

And a tip if you are going to build a system like this. Do keep in mind that you might want upgrade the system in the future so try to think of what you might want to add in the future.. And if you are going to have it as the main entertainment system try to keep everything on the system at stable-versions and no SVN checkouts :)

Re:Google should provide a WebService

[Frenchy_2001]

Actually, Microsoft already provides such a service for their Media Center Editions users.
The media center shell will get that data automatically, with a 2 weeks forward visibility.

I suppose that microsoft either pays for that service or just provides it from a known good stream and you pay for it in the cost of the OS (either Win XP MCE or some of the Vistas).

As all those streams, they are not always correct (programming DOES change unexpectedly), but usually, works just fine.

Re:Oh no!

[Anonymous Coward]

Here is something for linux that provides tv listings in a variety of formats. Unfotunately XML is not one of them. Looks like the software costs $29 but can give text info and html. Seems that it gets the data from TVNow.
http://unihedron.com/projects/tvnow/tvnow.php [unihedron.com]
http://www.tv-now.com//index.html [tv-now.com]

Re:Myth will survive

[roye]

I never see ads when I use Tivo's 30 sec skip. That you have to redo the hack after a reboot (admittedly almost never) is bullshit. But I can take the 5 seconds to press Select, Play, Select, 3, 0, Select. Instant 30 second skip anytime you want. This is also good to use during a boring minute or so of a show.

Re:This is troubling

[Vengeance_au]

In Australia, I've been using reliably for over 2 years (it has been around longer) the Oztivo [tuhs.org] TV Guide [tuhs.org] data, which is community created, supported and maintained. It was originally set up by a group of people who imported Tivo devices to Australia, where there is no Tivo data available. This data has been extended so there are now Mythtv and MCE scripts to allow access to the data. I've found the data to be better than some of the Aussie pay-services, purely because the guys who maintain the data are using it as well, so have a vested interest in making it as useful as possible. This service supports TV across all the states and regional centers in Australia, as well as the cable and satellite channels.

I'm guessing it's down to a need - in the US, there has not been a need for this kind of community data source, as Zap2It has been a free source. Now that it's gone, I'd be guessing there will be multiple US community based solutions springing up within weeks, and within 6-12 months most will fall away or aggrigate, leaving one or two that everyone uses.

Re:Make it a paid service

[pavera]

well, sure, but lets say you want to provide this "free" service, to even be marginally popular/useful, you would need at least the 50 largest markets, and the top 3 providers in those 50 markets, so you're going to have to cover at least OTA/local, incumbent cable, dish, and directv. So that is 4 accounts for 1 market, times 50. That is 200 accounts. You really think there are tons of people out there just dieing to pay $400/month to give away free tv guide data? I really don't think so. At first I thought the original reply was spot on with the criticism of the idea of trying to make money from this, but your parent made me rethink it. The market is entirely too fragmented, different localities, different channels, different providers. Plus for sports you've got market blackouts, stuff that is available in Nevada might not be available in California...

There is no way to efficiently pirate this data. It only makes sense for a centralized provider to charge a small fee and distribute the information directly from there.

Re:The killing blow? I think not.

[Cramer]

How is writing the tool in the first place mooching off everyone else?

Writing it isn't -- and I didn't say so. Throwing it to the community hoping someone else will fix the problem before you have to invest any more of your expensive time. In your own words:

If I wrote one and threw it out to the community to maintain it, the cumulative cost to me for helping to maintain it when I notice something wrong is nearly zero, as chances are, someone will already have fixed it before I notice the problem 9 times out of 10.

The "$50" figure was an estimate. The cheapest selection is $39.99, or so. Add in tax, tag, and the other BS and 50 is good round number. Plus, it's a good bet your subscribed to more than the absolute dirt cheapest minimum package. I see I'm wrong.

Billing is simple if you only do it once a year...

Hah. For someone who makes as much as you claim, you should have better sense. You do realize how businesses actually work? You're going to be picking up new customers one at a time spread out over time. Do you plan to stack up everyone's bill and process them all on one day a year? And presumablly not allow them access until then? See. Laughable.

The service will be charging people the instant they sign up. And there must be confirmation that a bill has been paid. There has to a channel ("CUSTOMER SUPPORT") for dealing with billing errors -- there will be errors... even the teenager taking your order at McDonald's messes up occassionally and they're 2 f'ing feet in front of you. If you think you can use paypal without any such support, and a "no refund" policy, I will warn you not to hang your entire business on it. Paypal is a very dangerous way to fund a business -- paypal has and will continue to close accounts for unspecified reasons (and you may never get your money out of the account.)

Customer service is only mandatory if you're trying to make money or if you're charging enough that you might reasonably get sued if things don't work.

Making money: Check. Charging for something: Check. You are vastly underestimating people's willingness to sue, esp. when a $60 trip to small claims court can put you on the hook for much more than that. (and ruin your credit rating. etc. etc. If you won't spend 5$ for guide data, why would you waste a day for to defend yourself in small claims court?) There's a great deal of satisfaction in screwing the one who screwed you.

As for servers and bandwidth, ... would probably be pretty minimal if you designed the distribution protocol correctly...

A bittorrent inspired push model is doomed to fail, for a miriade of reasons. Not the least of which is the port forwarding for an inbound connection from nearly random outside hosts. Granted, MythTV users tend not to be complete idiots, so the expectation of the users getting the port forwarding setup correctly is better than average, but still not 100%. (btw, the spider web distribution you appear to be suggesting is patented. it's been around as long at bittorrent.)

The simplest method with the least complexity that will Just Work(tm), is a simple http request. Each user asks for the data of interest to them. That's how tivo does it; it's how everybody does it. Tivo's been doing this for a decade. If there were a better way to do it reliablly, others would already be doing it.

As an expensive hobby, it might be doable for a single DMA. I know of torrent sites that cost as much (and more) per month to operate. But even that can be a non-trivial amount of work with the variations in cable lineups. But I still think this is a perfect project for Google!

Re:But EIT has limitations

[Anonymous Coward]

http://atscap.sourceforge.net/ [sourceforge.net]
http://www.penlug.org/twiki/bin/view/Main/DigitalT elevisionAtscap [penlug.org]

EIT's update every 3 hours and they carry at a minimum 12 hours of event programming information within 4 EITs.

It takes about 1 minute to fetch how ever many hours of programming the station is sending in their ATSC PSIP. For the 7 English speaking stations in my area, all of the program guides are updated in about 5 minutes.

atscap automatically updates the EPGs once every 3 hours. If a capture is in progress and it's unable to run the EPG update for all the stations, it's automatically postponed until it can run. The EPG for the station that is currently being captured is already updated, so that's one less EPG that has to be updated when the postponed EPG updates do eventually run. You're not very likely to watch a 12-hour event, so it's not very likely you will miss an event.

As for recording a specific event an unknown time in the future, a simple search event entry will add the event to the timer list when the EIT updates, if it finds a match in the updated EITs.

I don't understand what you mean when you say "So at least in the U.S., the EIT would be of limited utility to MythTV users, since it is unlikely that one would be able to use this data to schedule multiple days in advance."

Perhaps you aren't being as imaginative as you could be. "24" sat on my atscap search event list for 7 months before it finally found the events in the EIT in January and onward. I didn't miss a single episode of "24". The channel filter in the search event prevented capturing all of the NBC re-runs of 2 year old "24" episodes.

Granted, atscap is for over-the-air broadcasts only, and it's only for digital television, not analog. Some of us are looking forward to the day when all of the NTSC transmitters go dark, because some of us can't receive over-the-air NTSC if our very lives depended on it, yet ATSC comes in fine with less than 1/5th the power output of the NTSC transmitters.

Unlike MythTV, atscap doesn't heat up your CPU compressing the video into some lower-quality format. If it's HD, most people will want the original picture quality, not some weirdware NippleVideo compressed low-quality version.

The flip-side of that coin means because it stays in the standard ATSC format, it's quite possible to play back the event with [blech!] Windows. I know someone who is right at this moment sitting on their couch with their Centrino Windows Vista laptop, watching captures made by atscap via their WRT54G wireless router, happier than the proverbial bug-in-the-rug. I suppose they could sit on the toilet and do the same, but I don't recommend that. You might get hemorrhoids!

atscap is painless to install, compared to MythTV, and also requires much less resources than MythTV. atscap doesn't need an SQL server for the EPG. atscap will do captures fine on a 300MHz CPU, and will probably work even on a 100MHz CPU.

atscap doesn't care what you use to play back the events. You could use it as a replacement for the MythTV backend, and still use MythTV for all the bells and whistles that you feel you need. Some people are using the Roku HD1000, some are using VLC, some are using Xine and some are using mplayer. The player decision your decision. You can choose what works best for you.

The drawbacks to atscap? You're *almost* at the mercy of the broadcasters to send proper PSIP information, much like MythTV is at the mercy of Zap2It XML. Also, some stations have buggy ATSC PSIP generators that send garbage, or something worthless, like "Normal Programming - 180 minutes". The Late Show with David Letterman is listed on the local CBS station EIT as 60 minutes. If I were to depend on the EIT event length, it's likely the musical guests would be truncated.

Weekday event timers to the rescue. I use a 65 minute weekday timer

Re:This is troubling

[NoMaster]

As the OP said, he's in .au. Here, due to some legal oddities involving telephone numbers, it is copyrighted information. There's an ongoing case between Nine (who now own HWW, the TV guide aggregators) and IceTV [icetv.com.au] over IceTV's provision of a downloadable guide service for PVRs. IceTV claim that their guide is based on history, common knowledge, and the network's own advertisements (i.e. facts & public knowledge); Nine claim that it's not - and even if it was, that doesn't matter, because it's the facts, not the collation of them, that's copyrighted. And strictly speaking, under Australian law, they're right.

As an aside, one of the other networks here has announced the introduction of Tivo. Many people seem to think that this will guarantee a free, hopefully EIT-based, EPG for everybody, based on conditions in the Broadcasting Services Act. To them, I'd say "go and read the Act - particularly, focus on the differences between the requirements for commercial broadcasters and national broadcasters (i.e. the ABC & SBS)". You'll see the Act was modified last year, and doesn't require commercial broadcasters to share guide information at all...

I haven't got it on hand, and it's been a while, but I seem to remember it being Sections 2 & 3, subsection 20-something?