iTunes Sales Not 'Collapsing' After All 122
john82 writes "Earlier this month we had a report from Forrester, based on a random sampling of 2,000 credit card accounts, that purported to show that iTunes sales were crashing. Now comes another survey from Reston, VA-based ComScore which indicates the exact opposite. ComScore's report which is based on actual iTunes sales shows a 84% increase during the first nine months of this year compared to the same period last year. Meanwhile the author of the Forrester report, Josh Bernoff, noted in his blog yesterday that they shouldn't be pummeled just because everyone took what he wrote and ran with it."
Own up to your reporting (Score:5, Insightful)
Well, that is why people should be responsible for their reporting. In my business, when you report something, you stand by it. If you present data or a theory with the suspicion that it is incorrect, that is fraud in my line of work. Seriously though, did you *really* think that a sample size of just over 1000 purchases on credit cards obtained through a back channel source is a reliable sample size for the number of iTunes purchases? If I correctly recall, Apple announced back in February that they were selling about 3 million songs/day and if the current estimates of increases on the order of 84% are correct, your sample size is woefully under-representative. Thats just high school statistics by the way...
I am not saying that you should lose your job over this one, but this should be a tacit reminder of how important good reporting is and if you are beyond your means or competence on a particular story or analysis, go find some help before you publish it, do some fact checking and be more careful with stories that can have a significant impact on companies and individuals.
Re:Own up to your reporting "I stand..." (Score:5, Funny)
Re:Own up to your reporting (Score:5, Insightful)
Re:Own up to your reporting (Score:5, Funny)
Re:Own up to your reporting (Score:4, Insightful)
Re: (Score:2, Funny)
Re:Own up to your reporting (Score:5, Funny)
Re: (Score:1)
Personally, I think Forrester should stick to advice and should outsource all their research to "Apple should outsource hardware to Dell" Gartner.
It's tough being a research analyst - you can press your face to the window, you can hear the music and see the guests, but they'll never let you into the party.
Re: (Score:2)
Based on this, I give Apple 1, perhaps 2 more centuries of profitable operations.
Re: (Score:2)
Re:Own up to your reporting (Score:5, Insightful)
Sadly this seems to be the deal in journalism at the moment. Everything is sacrificed in order to be first to publish or, if not first then, not too far behind. Accuracy appears to be sacrificed in the race to publish.
Re: (Score:2)
I agree. At the newspaper I work for, we once had to reprint over 600 copies of our A-Section. Why? One of our reporters took a rumour about a missing girl as fact. Despite that most of us had dismissed it (as the story was being covered nationally), he chose to go with the story. When he was chewed out by the Publisher, he was venting at someone in
Re: (Score:1)
Re: (Score:2)
Re:Own up to your reporting (Score:5, Insightful)
Naturally they didn't collect enough data to calculate a p-value... THAT was their mistake. Of course, nobody seems to do that, so really, it's par for the course.
Re:Own up to your reporting (Score:5, Informative)
The sample size you need to take for doing the study is dependent upon the probability that you expect the event to occur. So for example, out of 1000 random purchases, how many do you expect to be iTunes purchases? Most people buy a lot of things on their credit cards. So my guess is that only maybe 5 out of 1000 purchases would be iTunes purchases. The rest would be clothes, gas, groceries, restaurants meals, movies, gifts, etc, etc, etc.
Let's say I'm right. If the expected value is 5 out of 1000, what are the odds that I might find 6 or 4 purchases in that sample? Well, depending on the distribution, it's not going to be that unusual. Remember, the *average* number you find will probably be about 5. If you actually look at 1000 random purchases, the actual amount you find will vary.
So you might find 4 or even 3 with a pretty high probability (don't know off the top of my head what that probability is -- especially since I don't know the distribution of the data). So you have a pretty high probability of reporting something like 0.3% of purchases are iTunes purchases, when the real value is 0.5%. That's a *huge* error.
But as others have said, the guys that do these studies know their stats. They don't put out crap reports by accident. They are intentionally misleading. Any reputable report that is based on statistical analysis will give you the error bars (i.e. + or - 5% 9 times out of 10). If this report had done this it would have said something like 65% reduction in sales +- 10% 1 time out of 10 (i.e. they aren't confident about their interval) or 65% reduction in sales +- 150% 9 times out of 10 (i.e. the error bars are totally crazy). And then it would be obvious the study was totally bogus.
Note: All numbers I've used are fictional. I took stats 20 years ago and I *really* don't remember any of the actual numbers...
Re:Own up to your reporting (Score:5, Insightful)
The proper place to start is by taking last year's data (or last month's, or whatever) and measuring that value, then measuring it again today. Then you can ask the question whether iTunes sales have changed or not. Once you've shown there's a high probability that they've decreased (that might take ten samples, or it might take millions), THEN you can talk about how much they've decreased, and what sort of error bars go on that value.
According to the original story they DID that, collecting data from 27 months... the difference between +80% and -60% is pretty huge... either they didn't do a simple t-test on their data, this was a VERY rare fluke or they decided to release their numbers anyway.
It definitely sounds like they were paid for their result... I wonder if maybe they didn't expect someone with much better data to come around so quickly to slap them down.
I also like the quote in the article about iTunes 1 billion dollars in sales not making up for the 2.5 billion dollar decrease in CD sales. Sounds about right to me. I doubt I'd purposely pay for more than a third of most albums.
Re: (Score:2)
Sample size is BIG deal. Without a significantly large enough sample size you cannot generalize your results to your target population. If we are talking about a population of one million people and you have a sample size of one thousand, that's 0.1%.
You would have useless information. You would need at least 10% to have meaningful results.
The p-value is not how likely you are to be wrong. It is a measure of how likely your results are do to chance. The lower the number the less likely, the higher the n
Re: (Score:3, Insightful)
As the number of samples increases, your estimates improve. However, you can't just say "you need a 10% sample" to be accurate. It doesn't work that way. The size of the sample you need depends
Re: (Score:2)
Now, most analysts are completely wron
Re: (Score:2)
Re: (Score:1)
I never understood why Webster didn't replace 'lose' with 'looze' while he was busy mangling, sorry, standardising the English language - I suspect you Americans would have much less trouble with the word if he had.
Incidentally, my favourite example of this nugget is the IIS 6.0 restart dialogue, warning me that I will 'loose' all current sessions if I proceed. I guess it could technically be correct...
cut him loose (Score:2)
Re: (Score:1)
Far be it from me to defend the atrocity of that spelling mistake, but I blame this on inconsistencies in English spelling (or, at best, regional differences in pronunciation).
Repeat after me - 'nose', 'rose', 'hose', 'lose'
and then 'floozie', 'doozie', 'choose', 'loose'.
Now explain to me why lose is spelled 'lose' instead of 'loose'.
Maybe I'm just a horrible speller, but when I see 'lose' it doesn't even look like a real word because I think 'lõz' and not 'loõz' and when I see 'choose',
Re: (Score:3, Insightful)
"Our credit card transaction data shows a real drop between the January post-holiday peak and the rest of the year, but with the number of transactions we counted it's simply not possible to draw this conclusion . . . as we pointed out in the report."
Seems to me like a pretty clear admission that the sample size is too small to be reliable. He took the data he had available, analyzed it, and presented the results while noting the deficiencies in the method. Doesn't sound much like fraud to me
Re: (Score:2, Redundant)
Re:Own up to your reporting (Score:5, Insightful)
"Our credit card transaction data shows a real drop (my emphasis) between the January post-holiday peak and the rest of the year, but with the number of transactions we counted it's simply not possible to draw this conclusion . . . as we pointed out in the report."
There is no way that he can use the words "...real drop..." in the same sentence as "...it's simply not possible to draw this conclusion...". Whilst those who uncritically took the information from this 'research' and used it (doubtless with some sensationalistic agenda in mind) deserve scorn, that very sentence itself demonstrates the research to be nothing more than PR to flog the thing at $249.00 a pop. If you take out the words "real drop" and substitute "no meaningful change" then this report was clearly worth fuck-all: at least in terms of the author's now visible desire to have something sexy to sell!
You're taking it out of context (Score:2)
Re: (Score:3)
Re: (Score:2)
I believe the "this conclusion" he is referring to is the conclusion that iTunes sales are "collapsing" (whatever "collapsing" means.) There is a drop over time in the data they examined (which is therefore a "real drop", as opposed to the other kind, I guess) but one cannot make any strong inference about the overall state of iTunes sales from this.
Abstraction has al
Re: (Score:2)
The real reason (Score:1)
No surprise there. As has been stated multiple times already, this isn't a valid comparison. They should be comparing the equivalent period this year, one year ago, two years ago, etc., not peak sales and slow sales.
I predict somebody's goin
Can the editors while your at it. (Score:2, Interesting)
Seems to me like a pretty clear admission that the sample size is too small to be reliable. He took the data he had available, analyzed it, and presented the results while noting the deficiencies in the method. Doesn't sound much like fraud to me. That's just grade school reading by the way...
So in other words, he knew from the beginning that he was spewing out bullshit. The article never should have gotten past the editors. One can argue back and forth whether the journalist should be disciplined, I'd argue for it and for an investigation of possible conflict of interest, but there's no way the editors should have let the article through as it stood. They should have been canned.
Sample size was probably ok (Score:3, Informative)
Re:Own up to your reporting (Score:5, Informative)
I'm a college professor of statistics. I don't think you can actually quote a high school statistics book which says that sample size is too small. In general, a sample size of 1,000 gives 95% confidence that your result is within +/-3% of the actual result. This is *regardless* of population size - that's how statisatics work, due to the Central Limit Theorem.
http://en.wikipedia.org/wiki/Sample_size [wikipedia.org]
http://en.wikipedia.org/wiki/Central_limit_theore
Now, the first thing that pops into my head is why only credit-card purchases? And even more fundamentally, why would the same people need to buy music, after they just went on a music-buying spree? I would think the opposite. That was the thing that made me skeptical of the report yesterday in the first place.
Re: (Score:3, Informative)
As one professor to another I am sure you also teach sampling error and experimental design, right? Additionally, it should be noted that the actual samples used for the analysis out of the total records pulled
That's not the case for rare phenomena (Score:5, Informative)
For example, if your sample is 1000, your 95% confidence interval is 1/sqrt(1000) = +/-3%. So if your 1000 samples showed 250 occurrences, you would know that it's 95% likely that the frequency of occurrence is between 22% and 28%. So the real frequency could be between 220 occurrences or 280 occurrences per thousand. No big deal for year to year comparison purposes. Worst case a 50% drop in sales is measurable because one year you could've been low (220), and the next year high (280/2 = 140), and the change is still statistically significant (outside your confidence interval).
For rare phenomena, this runs into a problem. Say the frequency of occurrence is 0.1%. You take 1000 samples and you measure 1 occurrence. The neophyte statistics student will say "Cool, I meansured 1 occurrence +/- 3%, so I have 95% confidence that the actual rate of occurrences is between 0.97 per thousand and 1.03 per thousand." Unfortunately, that's wrong.
The confidence interval is based on the percentage you measured. Your confidence interval says there's a 95% chance that the actual frequency of occurrence lies between 0% and 3.1%. There is a huge, huge difference between 1 incident in a thousand and 31 incidents in a thousand, especially if you're trying to compare between two samples. One sample (year 2005) you might get 25. Next sample (year 2006) you might get 5. These are both within your confidence interval, but if you're not careful you would erroneously conclude that you have 95% confidence that sales plummeted to just 20% that of the previous year.
Put simply, if you want to accurately measure rare phenomenon, your sample size has to be large enough that your confidence interval is significantly smaller than the rate at which that phenomenon occurs. If iTunes sales account for 0.1% of all credit card sales (which I think is a very high estimate) and you want to compare year to year changes, you probably want an accuracy of at least 1/10th the 0.1%, or a margin of error of +/- 0.01%. Your sample size needs to be large enough that your confidence interval is around the 0.01% range. That is, you need a sample size of a 100 million credit card transactions.
Re: (Score:2)
In general, a sample size of 1,000 gives 95% confidence that your result is within +/-3% of the actual result.
True, but how many iTunes purchases will you get in a sample of 2,000 credit card accounts? I don't know, but it should be a few percents, if even a percent. Because of that +/- 3% thing (although it's not 3% anymore for 2,000 samples) you'll hardly get a very meaningful result.
It's as if you polled 1,000 random persons twice, compared how much Ralph Nader got each time and say that he became sudd
Re: (Score:3, Interesting)
And you're a professor for which university?
Re: (Score:2, Informative)
Re: (Score:2, Insightful)
Dude, it's a think tank "report." They deal in the amorphous and write it in weasel; 'cause it's a living paid for by the brainless. Put it under a rhetorical microscope and there's little there to be responsible for.
The real title of this story should be "Think Tanker admits he shits for money."
KFG
*Yawn* (Score:2)
Re: (Score:2, Funny)
Re: (Score:2)
Anywhere there are numbers, there is going to be a group of people wanting reinterpret them. This is classic sales approach. Reminds me of the advert I saW the other day: "we will pay the first three months for you", when in actual fact they are just paying the equivalent of the amount they jacked the price up by - doh!
the field day was on purpose (Score:1)
ComScore (Score:5, Informative)
Re: (Score:3, Funny)
Re: (Score:2)
Re: (Score:2)
Yeah, the press blows stuff out of proportion, as they almost always do with statistics. Apple's stock loses 3%.
Then, Comscore comes out with an equally ridiculous se
Why shouldn't he be faulted? (Score:2)
It's his story. If he didn't mean it, he shouldn't have put it out there.
Typical of these "research" companies, though. Completely typcial.
Re: (Score:2)
Re: (Score:1)
liars, crooks, and analysts (Score:5, Insightful)
Find a big analyst company that will admit that Itanium is a colossal disaster, that businesses don't want and don't need Vista, that HP's supply line trouble and incompetent management are sinking the company (particularly during the Carly years), that Oracle is terribly insecure. You won't, because they all have contracts with Intel, Microsoft, HP, Oracle, etc. But they won't hesitate to beat up on Sun (how many times have they called for McNealy's resignation), AMD, Apple, and predict their doom*, and others that don't spend the kind of money on various analysis contracts.
So sure, iTunes sales are collapsing (according to Forrester), but nobody will call Zune a turd. It's all in a day's work.
*disclaimer: I might be considered a fanboy of one of these companies, and it's not Apple
Re: (Score:3, Funny)
You're a fanboy of Forrester and Gartner, too?!?
iTunes sales up! (Score:2, Funny)
Re: (Score:1)
Gift Cards (Score:5, Insightful)
It totally ignored the little lime-green $15 gift cards that litter the checkout stands of every Target, Best Buy, CVS Pharmacy, and Kroger in the US. Each one of those is 15 songs, and fifteen purchases that don't register as credit card transactions.
Re: (Score:1)
Re: (Score:2)
Re: (Score:1)
They are comparing two different things here. Forrester is saying, "Since January 1st, iTunes sales have fallen 65%." and ComScore is saying, "Compared to last year, iTunes sales are up 84%." Both can absolutely be correct. The ComScore numbers look like they were put together by Apple as a quick cover-up for the Forrester report. I agree that the numbers need to be looke
See Spyware IS useful after all! (Score:1)
Which is apparently ok because they are getting REAL data from it!
The only part that matters. (Score:1, Interesting)
I ask again: from whom did they get this data?
Is buying, selling, or redistributing such data a crime in the US, or CA in particular?
If not, why not? I'd like to make it crime with a nasty punishment.
If so, has the investigation started yet?
Grrrr.
from Josh Bernoff's blog (Score:5, Insightful)
He concludes with this statement in his blog:
To a degree, he has a point. With Apple's secrecy, articles like these are run without having all the facts. Sensationalism becomes rampant. Then he has to go and say "In the research business we like facts." All too often we read more about speculation rather than facts from these research companies. They complain secretive companies like Apple or Google don't give them enough information, but I wonder where the actual "research" in research business has gone.
Re: (Score:3, Insightful)
Re: (Score:2)
Sure, for a little while. But eventually people get tired of people yelling "the sky is falling". I think Google and Apple, keeping their business their business, is a good practice.
AAPL facts are available... (Score:2)
Here are some more facts; I know several people who own iPods who have never purchased anything from iTunes. Maybe someone should extrapolate from there and say that iTunes has zero sales.
Re:from Josh Bernoff's blog (Score:4, Insightful)
Seriously, why would Apple release real data to these people when they see what they do with the data that they can get hold of?
Re: (Score:3, Insightful)
"Look at the trends here"
Yes, there was a massive spike last Xmas that hasn't been exceeded during the 11.5 months that followed. Indeed, if you draw a line from that peak to the present, iTunes queries are down from a year ago. It's proof positive - especially if you don't know a fucking thing about statistics!!
I can't find my ass with both hands around statistics and even I can see what's wrong with Forrester's report. So, Forrester my ass.
Credit Card data (Score:3, Interesting)
So, I would like to ask, how was the data obtained and is this level of detailed information avilable for legal purchase? I'm just curious as to how much information is available about credit card puchases.
Re: (Score:2)
Welcome to America (Score:4, Funny)
This data was willingly given by the owner (Score:1)
Re: (Score:1)
Forrester Research, Inc. announced the launch of Forrester's Ultimate Consumer Panel(TM) (Ultimate), a single-source, opt-in, highly secure panel that electronically captures online and offline behavior from a representative group of more than 10,000 US households.
This panel, by the way, was recently sold to another market research company.http://www.forrester.com/ER/Press/Release/ 0,1769,1101,00.html [forrester.com]
Sorry guys -- nothing scandalous. This was totally legit research. They didn't have access to *everyone's* data or even *your* data, just those who opted in.
Re: (Score:2)
I can now see why the group chose a sample of 1,000 persons. It was obviously a pool from their 10,000 member sample and they sould be able to calculate the validity from the larger sample.
He should not be pummled (Score:3, Funny)
Re: (Score:1)
Re: (Score:2)
Mark-et Their Words (Score:4, Insightful)
Which is, of course, why everyone just takes what they write and run with it. That's the measure of success at marketing research peddlers. It's the CIO self-perpetuation. One reason why so little ever gets done right, but so much does get done without being called wrong. To blame their own market for taking them seriously when they ought not be is finally a whisper of honesty from these chattering weasels. I expect them to fix that in the next release.
Forrester's new Corporate Motto (Score:1, Redundant)
Testimony of Mr. Marc E. Kasowitz... (Score:5, Interesting)
One particularly effective illegal strategy involves the
following scenario: the short-selling hedge fund selects a
target company; the hedge fund then colludes with a so-called
independent stock analyst firm to prepare a false and negative
"research report" on the target; the analyst firm agrees not to
release the report to the public until the hedge fund
accumulates a significant short position in the target's stock;
once the hedge fund has accumulated that large short position,
the report is disseminated widely, causing the intended decline
in the price of the target company's stock. The report that is
disseminated contains no disclosure that the analyst was paid to
prepare the report, or that the hedge fund dictated its
contents, or that the hedge fund had a substantial short
position in the target's stock. Once the false and negative
research report -- misrepresented as "independent" -- has had
its intended effect, the hedge fund then closes its position and
makes an enormous profit, at the expense of the proper
functioning of the markets, harming innocent investors who were
unaware that the game was rigged, and damaging the target
company itself and its employees.
http://judiciary.senate.gov/testimony.cfm?id=1972
Student exercise: Compare and contrast with the movement of AAPL stock shares before and after this report came out.
Re: (Score:1)
It's also interesting that this "report" came out a couple days before December stock options expiration, right when options are the cheapest so any stock moves give the greatest profit. The SEC could investigate large block options transactions in the past week, but probably won't. Too many things to do before xmas, y'know.
Re: (Score:2)
Actually, the report was issued on December 6, presumably to whoever had ordered it. The report was then released to the press three trading days later, on Dec 11, with the first headline reports proclaiming a fall crossing the wire services later Monday.
Hypothetically one could easily establish a large short position over three days without too much impact on the stock price, and then cover that short d
The "New Responsibility" (Score:1, Funny)
This guy isn't responsible for getting his data completely wrong on the iTunes story.
The government isn't responsible for getting it wrong about WMD, having enough armor for our troops, for Katrina relief efforts or for covering the defecit that they keep increasing.
The editor isn't responsible for his writer faking articles.
And Microsoft isn't responsible for all of the holes in their security.
Heck, I'm not even respo
Martha Stewart (Score:1)
Unless he, or people he knows, bought stock in Apple after spreading that "information".
I think we'll find out... (Score:2)
Secondary Reporting (Score:2)
This shows two faults. The first is the lack of checking that goes on by web based servic
Wait, they did this *how*? (Score:4, Insightful)
Ummm... Now, I harbor no delusions that my credit card history really counts as a secret - Obviously my CC company has it and uses it to market bizarre crap to me, and they'd turn it over to the government without thinking twice about it.
But how does some guy just go and "randomly sample" 2000 cards' histories? If I wanted to validate his study, could I do the same?
Something doesn't seem right here, and I don't think most people would like the "how" either way.
RTFA: the credit card holders gave permission (Score:1)
This just in... (Score:2, Funny)
And in another shocking revelation... (Score:2)
Well duh! (Score:2)
FUD (Score:1)
According to the Records Department (Score:1)
Death to Goldstein!
truism once again proven (Score:2, Funny)
Re: (Score:1)
Thought it would be fun to compare (Score:3, Insightful)
Re: (Score:1, Insightful)