The Great WU Spam Debacle—Some Further Clarification

Thanks to Paul for giving a heads-up about the unfortunate loss of some archived comments. I thought I'd add some words as the resident WU tech-support guy to try to paint a slightly fuller picture of what happened, and hopefully allay some concerns.

First of all, the recent site redesign had nothing to do with the loss of some of the archived comments. The two are totally separate issues.

The story really begins during my two-year hiatus from active participation in the site, during which I failed to pay the close attention I should have to the functioning of the site, and so I didn't notice a massive, tumor-like mass of spam growing on the site. The problem wasn't the spam that appeared in the comments. That was only the tip of the iceberg. The real problem was the member list and the member profile pages. Many of you might not even be aware these existed... but the spammers were very aware of them. And thousands of them were registering as members every day so they could get a member profile page on which they could post spammy links in their "bios". By the time I noticed the problem, a few weeks ago, we had a quarter-million spammers registered as members.

Having that many spam members was having serious, negative consequences for the site. First, having up to 10,000 spammers registering a day was putting a big strain on the server and was slowing down the entire site. Second, the google bot was busily trying to index all these spam member pages, but was getting increasingly confused about what was the real content on the site, and was frantically indexing and re-indexing, provoking our webhost, at some point, to block the google bot entirely from the site.

So we were kind of hosed. Blocked from google and slowly being drowned in spam.

To fix the problem, the first thing I did was to block new member registration. Then I made the member profile pages non-public. Finally, I decided that all the spam members had to be deleted. The problem was how to delete the spammers while preserving the legitimate members and their comments.

I figured that the one unifying feature of all the spam members was that they hadn't left comments on the site. Instead, they had been posting the spam on the member profile pages. So I instructed the server to delete all members who had left no comments. The server churned and churned for a looooonnngg time. Finally, when it was all done, there were a couple hundred legitimate members left. And that, I thought, was the end of the matter. We were free to move forward with a clean slate.

Until Dumbfounded pointed out that a lot of old comments left by unregistered members had disappeared.

It was never my intention to delete any comments. In fact, I thought I had told the server specifically to preserve all the comments. But I'm guessing I wasn't specific enough in my instructions. I told the server to delete members who had left no comments, but I didn't say anything about NON-MEMBERS. So maybe the server interpreted my command to mean that it should delete everything EXCEPT members with comments. So all the non-registered guests got purged. That's my best guess as to what happened. Though I'm really not certain of this. The lesson is that whenever you give the server a broad, system-wide command to delete stuff, strange things can happen!

So there you have it. The Great WU Spam Debacle of 2012. At least the spam problem is far more under control than it was. So we've been left a little bruised and battered by the whole experience, but we're still standing. My apologies for the screw-up. Hopefully we can accumulate lots of great new comments to replace the old ones that are gone.
     Posted By: Alex - Fri Mar 16, 2012
     Category: Weird Universe

Life, as we know it, will go on at least for another 279 days or so.
Posted by Expat47 in Athens, Greece on 03/16/12 at 01:18 PM
Oh I have no doubt we will be receiving plenty of interesting comments as we go along. WU has always been a lot of fun but now that you've done all that clean up and given us a great new look I am sure we will attract lots of new people as well as all our tried and true regulars. Its really great to have you back taking care of us again Alex.
Posted by Patty in Ohio, USA on 03/16/12 at 01:20 PM
Brilliantly and clearly recounted, Alex!
Posted by Paul on 03/16/12 at 02:41 PM
That's the best OOPS! explanation I've seen in a long time, Alex. That's the way you learn about these machines. If I could give a piece of advice - Think of a computer as a very eager dog, trainable but very literal once instructions are given. Programming is more of an art form and requires a lot of thought as to consequences.
Posted by KDP on 03/16/12 at 03:35 PM
As a non-member, I assume all my previous comments have been removed as well.
I am pretty sure that is not the end of the world, since as expat pointed out, that is still 279 days away.
I cannot imagine what kind of time is involved in keeping a site like this working and clean. So, I personally feel that the loss of a number of comments for the sake of ridding the site of spammers (even if temporarily) is more than worth it.
Be assured that Alex, Paul and Chuck's hard work is continually appreciated.
Posted by Terry on 03/16/12 at 04:11 PM
End of the world? Can I get front row seats?
Posted by KDP on 03/16/12 at 04:54 PM
Huh? I've been commenting for months and didn't know there was a "membership". Serves me right.
Posted by Robert on 03/16/12 at 07:44 PM
Is anyone else getting a "forbidden message now when trying to log in?
Posted by MohawkWarrior on 03/16/12 at 10:05 PM
I always remain logged in sweetie, but I haven't noticed anything amiss.
Posted by Patty in Ohio, USA on 03/16/12 at 10:27 PM
Fun with databases.
Posted by jswolf19 in Japan on 03/17/12 at 12:40 AM
@KDP: You're right about programming being an art, I can vouch for that.

@KDP: Sorry, all front row seats taken up by dolphin watchers. The restaurant, I've heard, still has a table or two open.

@MHW: I alerted Alex about the log-in problem yesterday. He's going bald over it now.
Posted by Expat47 in Athens, Greece on 03/17/12 at 01:24 AM
Well, It looks like all this effort was for naught since there are still tons of spam comments getting posted.
Posted by IHateSpam on 03/17/12 at 08:26 AM
You don't have a backup??? (to restore all comments and try again to get it right)!!!
Posted by Na Yeo on 03/17/12 at 09:14 AM
@IHateSpam: Yea, but it's NEW & IMPROVED SPAM.
Posted by Expat47 in Athens, Greece on 03/17/12 at 10:15 AM
As a comment poster without a profile, I don't mind if my comments were deleted. Acutally, a few of mine are still there -- the ones I remember posting recently.

As long as the articles are still around, and the spam is gone, that's what matters!

Don't sweat it! 😊

Thanks for keeping this site up and running.
Posted by Kyle Morgan on 03/17/12 at 03:56 PM
Na Yeo -- The webhost should have a backup. The problem is that if we roll everything back to Jan 17, we'd lose everything since then. In theory it should be possible to extract the missing comments from the backup and somehow re-insert them into the database. But that's way above my skill level.
Posted by Alex on 03/17/12 at 08:04 PM
All I can say is t
Posted by Mark on 03/18/12 at 12:05 AM
WOOOOOOOOO!! Log-in works again for me. As for the spam-a-lam-a-ding-dong. Well, we all get that you put your best efforts into it. I, for one, will continue to assist in deleting those that I catch when I am on, as I'm sure the others will. Spam is always a problem in a community-interactive website, and it dose not detract from the quality of the articles as well as my enjoyment, even if I have been a little busy to post anything of my own for a VERY long time. Believe it or not, I have to spend alot of time in research trying to fight to get some kind of educational benefits. A lot of military personnel are getting their benefits taken away.
Posted by MohawkWarrior on 03/18/12 at 09:19 AM
That is a crying shame MW. When you serve you should get your rightful benefits.
Posted by Patty in Ohio, USA on 03/18/12 at 07:57 PM
Everyday theres a new bunch of Spam, Its getting pretty bad guys. I;m waiting for one of the spammers to post some spam here, now that would be ironic. I think it would look something like this: Its very nice of you to post this I will use this in my blog Thank you for posting the great spam debacle
Posted by I Hate Spam on 03/20/12 at 08:04 AM
Page 1 of 2 pages  1 2 > 
Commenting is not available in this channel entry.