British ISP Bombards Users With Deleted Emails 134
judgecorp writes "For three days, customers at British ISP Sky have been receiving a flood of old and deleted messages. The problem started when the company switched its email provider from Google to Yahoo. As it began to move accounts from one provider to another, it became obvious that the new provider could not tell which emails in the old system had been sent or deleted. Some users had up to 8000 old messages. The incident has been going for three days, as users are migrated. Sky is apparently unable to fix the problem — its best advice been to suggest users delete the old messages."
Re:I'd be pretty pissed (Score:5, Interesting)
Actually, I'm pretty certain I know exactly what happened, because I just handled a major migration to google and dealt with an issue like this. It's due to the way google uses labels instead of folders, and how they (mostly-transparently) expose them as folder via imap (though this is one of the few non-transparent side effects).
In google, when you "delete" a message via imap, it doesn't get deleted. Instead, google just removes the label. That message still exist with all of the other labels, and it also exists in "All Mail" (which is exposed via IMAP through the "[Gmail]/All Mail" folder). So, if you have new mail come it, it is by default in your INBOX and your "[Gmail]/All Mail" folders. When you then delete it from the INBOX via imap, it's still in All Mail.
The way to deal with this is to move the message into "[Gmail]/Trash" instead of deleting it. That will truely delete it. However, since that wasn't done all along, those "deleted" messages are now orphaned in "[Gmail]/All Mail". There is a potential way to resolve even this problem, but it depends on how the account has been used. If users have logged into Gmail directly and taken advantage of the "Archive" feature to remove a message from the inbox (without truely deleting it) then all bets are off. There is no way to differentiate intentionally saved messages from deleted-via-imap messages. However, if it has only been accessed via imap (and users haven't intentionally been trying to take advantage of the All Mail folder), then you can do the following via a script:
Go through every message in "[Gmail]/All Mail".
For each message, try to find that same message in another folder.
If you don't find it in another folder, then that message only exists in "[Gmail]/All Mail". You can then move it to "[Gmail]/Trash" to get rid of it.
Searching for messages 1 at a time is a bit slow, so you can optimize this by first building a list of all messages in other folders. If you just retrieve a few headers from every message, it's actually fairly fast. The "Message-ID" field is usually sufficient for this, but there may be messages here and there that don't have that header, so you'll have to have other headers to fall back on.
All the speculation and desceiptions are bunk (Score:5, Interesting)
Had they actually *tested* this, in advance, would we be discussing the various flaws in each mail systems' implementations, or their real/imagined problems?
No. We would not even know it happened.
Don't blame Google's use of IMAP flags and folders, blame Yahoo!'s apparent lack of planning. Or Sky. Or whoever. Plenty of blame at the receiving end.
If you're moving mail from some system into yours, the responsibility is yours to make it right.
And I've done this. Wait till ya hose up the passwords, my friend. Fun times.