Spamhaus ZEN is a composite blocking list run by the Spamhaus Project. This UK-based organization was created in 1998 by Steve Linford, and is maintained by a group of employees spread across the globe.
Status of completewhois.com: IN FLUX
Update 9/30/2007: The website www.completewhois.com is operational again, but some links appear to be broken. My attempts to query their DNSBLs have all timed out. While CompleteWhois may be on the mend, it seems that it may be too soon to give the all clear.
Previous updates follow.
Previous updates follow.
APEWS: Doing the Math
I'm guilty. I admit it. I've called APEWS listings "random," which isn't quite right. Arbitrary would be a better word for it. Not to mention broad, and questionable.
APEWS, the "anonymous" blocking list meant to be an early warning system for spam, generates a lot of worry from administrators and end users who find themselves listed by way of plugging their IP address into an online lookup tools like DNSStuff. Though it doesn't result in much (if any) of anyone's mail being rejected, as it's not widely used, some people still think they're being labeled a spammer, and don't know what to do about it.
They've usually done nothing to warrant the listing; the simple fact of the matter is that they happen to have an IP address on the internet, and there's more than a 1/3 chance that this IP address will be on the APEWS blocklist.
As I've indicated previously, APEWS has IP address entries accounting for about 42% of the raw numerical depth of V4 IP address space, though I'm not excluding non-routable space and overlap between some listings. When one takes those factors into consideration, APEWS seems to list somewhere around 38% of currently routable IP4 network space.
Time for an experiment. What if I take a large chunk of address space, say, 42%, and list it all? I've got detailed records of spam and ham, and it's easy to bump my corpus up against an imaginary blocklist I've just made up right here on the back of this napkin.
Here's what happens when I do that: Over the past ten days or so, my 42% listing of IP space would've captured 62.8% of spam, but also incorrectly captured non-spam 31.5% of the time.
When I skinny my imaginary blocklist down to 38% of IP4 space, I get a 62.21% hit rate on spam and 31.15% false positive rate against non-spam. (In other words, just about the same numbers.)
To me, this is evidence that APEWS seems to be blocking some spam based on the "stopped clock is right twice a day" principle. List a large chunk of IP address space, and you're going to catch a significant amount spam, though inaccurately.
It further suggests to me that if I added a few rules to start my focus points with a bit of accuracy, I could probably tune this to get a hit rate close to what I see from APEWS, with its 73% hit rate against spam, and 26% false positive rate against non-spam (21 day average ending on 9/2/2007).
The conclusion I draw from this exercise is that only the barest thought has been given to the processes by which APEWS decides which IP addresses to list and for what reason. If I can get more than halfway there with a couple hours of sloppy bar napkin math, then perhaps they haven't thought it through too deeply.
APEWS, the "anonymous" blocking list meant to be an early warning system for spam, generates a lot of worry from administrators and end users who find themselves listed by way of plugging their IP address into an online lookup tools like DNSStuff. Though it doesn't result in much (if any) of anyone's mail being rejected, as it's not widely used, some people still think they're being labeled a spammer, and don't know what to do about it.
They've usually done nothing to warrant the listing; the simple fact of the matter is that they happen to have an IP address on the internet, and there's more than a 1/3 chance that this IP address will be on the APEWS blocklist.
As I've indicated previously, APEWS has IP address entries accounting for about 42% of the raw numerical depth of V4 IP address space, though I'm not excluding non-routable space and overlap between some listings. When one takes those factors into consideration, APEWS seems to list somewhere around 38% of currently routable IP4 network space.
Time for an experiment. What if I take a large chunk of address space, say, 42%, and list it all? I've got detailed records of spam and ham, and it's easy to bump my corpus up against an imaginary blocklist I've just made up right here on the back of this napkin.
Here's what happens when I do that: Over the past ten days or so, my 42% listing of IP space would've captured 62.8% of spam, but also incorrectly captured non-spam 31.5% of the time.
When I skinny my imaginary blocklist down to 38% of IP4 space, I get a 62.21% hit rate on spam and 31.15% false positive rate against non-spam. (In other words, just about the same numbers.)
To me, this is evidence that APEWS seems to be blocking some spam based on the "stopped clock is right twice a day" principle. List a large chunk of IP address space, and you're going to catch a significant amount spam, though inaccurately.
It further suggests to me that if I added a few rules to start my focus points with a bit of accuracy, I could probably tune this to get a hit rate close to what I see from APEWS, with its 73% hit rate against spam, and 26% false positive rate against non-spam (21 day average ending on 9/2/2007).
The conclusion I draw from this exercise is that only the barest thought has been given to the processes by which APEWS decides which IP addresses to list and for what reason. If I can get more than halfway there with a couple hours of sloppy bar napkin math, then perhaps they haven't thought it through too deeply.
What to do if you are listed on APEWS
If you are listed on the APEWS blocking list, as confirmed by checking their website, here's how I would recommend that you handle the situation. (Who the heck am I?)
Note: This isn’t guidance on how to avoid a blocklisting or sidestep anti-spam groups. If you have a spam issue, fix it. Don't spam, ever, for any reason. This is information is regarding how to address an issue with a list that is very aggressive at listing non-abusing IP addresses and networks, with no published, attainable path to resolution.
Note: This isn’t guidance on how to avoid a blocklisting or sidestep anti-spam groups. If you have a spam issue, fix it. Don't spam, ever, for any reason. This is information is regarding how to address an issue with a list that is very aggressive at listing non-abusing IP addresses and networks, with no published, attainable path to resolution.
- Don't despair. Be calm.
- Do NOT post to a USENET newsgroup or to Google Groups, asking for assistance. Any replies you get will be from people who do NOT work for or with APEWS, and most of those replies will be unhelpful.
- I can't stress this point strongly enough: Posting requests for help on the Internet will not get you any assistance. The APEWS FAQ directs people to post questions, but the only thing that happens is that discussion groups are overrun with questions, and the only people who answer those questions are (a) not involved with APEWS and (b) rarely polite or helpful.
- APEWS ability to be used as a spam filter has been greatly reduced and restricted due to perceived malfeasance on the part of the APEWS maintainer(s). UCEPROTECT and SORBS, blocklist groups who used to publish the APEWS data, are no longer doing so as of August 13, 2007. This means that the two main channels available to administrators to use APEWS as a spam filter have been revoked. This means that if your mail bounced due to an APEWS listing before or on August 13, 2007, you might want to try to send your mail again – it would likely get through, as the list is even LESS widely used than it was up until August 13, 2007.
- APEWS is very aggressive (meaning its use as a spam filter drives a lot of false positive blocking) and as measured by me on August 11, 2007, lists approximately 42% of the Internet. (By “the Internet” I mean IP4 address space.) In other words, they list nearly half the Earth, suggesting that anybody who actually wants to receive mail probably cannot use APEWS as a spam filter. This strongly suggests that very few people are going to block your mail because of the APEWS listing.
- Anyone using APEWS as a spam filter is going against the advice of a multitude of other anti-spam advocates and email professionals. See my news and commentary roundup for more information and links to feedback from others in the anti-spam arena.
- Since APEWS is not widely used, your next step should be a review of your bounce data. Have you received any bounces that reference an APEWS block?
- If not, don’t worry about it. You just determined that you’re not having blocking issues that you can trace back to APEWS. It’s annoying that you’re listed on the website, but there’s little easy recourse available to you to address that.
- If yes, you have received a bounce message that references APEWS, contact the site that blocked your mail. Call them on the phone or email them from a different email account (Hotmail, Gmail, etc.) Show them that APEWS is problematic and not widely used. Explain to them that you do not spam, and that APEWS has listed you even though you do not spam. Provide them links to this page on DNSBL.com with more information about APEWS.
If you want to learn more about APEWS, I've collected everything I know about this “anonymous” blocking list here on DNSBL.com.
I hope you find this information helpful. Please feel free to contact me with your comments or feedback. But, please note that I'm unable to consult with you regarding your specific situation -- I've already got a full time day job, and I'm not looking for consulting clients.
I hope you find this information helpful. Please feel free to contact me with your comments or feedback. But, please note that I'm unable to consult with you regarding your specific situation -- I've already got a full time day job, and I'm not looking for consulting clients.
APEWS News and Commentary Roundup
APEWS, the Anonymous Postmasters Early Warning System, is an “anonymous” blocking list that claims to run in the style of SPEWS. That is to say, its goal is to be an “early warning system,” catching and stopping spam before other lists or filters have the opportunity to do so.
The APEWS blocking list was first announced by way of an anonymous posting to the newsgroup news.admin.net-abuse.blocklisting on January 12, 2007. Though this newsgroup post originated from the IP address 149.9.0.57 (registered to US provider PSI/Cogent), the list is widely believed to be run from Germany.
If you are listed on APEWS and wondering what to do, visit this page for my suggestions.
Accuracy
A quick review of the past thirteen weeks of my own stats.dnsbl.com data shows that the list has been ramping up in aggressiveness the entire time that I've been tracking it. What was barely a 20% effectiveness rate against spam eleven weeks ago is up to 80+ percent on a week-by-week basis. However, false positives have risen similarly.
The rising spam match rate is based on what I would characterize as the “stopped clock is right twice a day” principle. List enough IP addresses, and eventually you're going to stop some spam. The side effect is that you're going to block legitimate mail (and lots of it) at the same time. Against my personal hamtrap data, APEWS blocks two out of ten of every legitimate piece of newsletter or list mail that I've signed up for.
I'm not kidding about "listing enough IP addresses," either. As of today (August 11, 2007), APEWS lists just about 1.8 billion IP addresses - by the raw numbers alone, this is 42% of the entire IP4 networking space. Much of the IP space listed isn't even routable; suggesting little attention is being paid to what IP addresses are actually able to transmit traffic (email or otherwise). Also, APEWS has been growing at a very fast rate. From July 20th through today, they have added an additional 7.5 million IP addresses. These are data points that, in my opinion, suggest that the list is bloated, questionably targeted, and inaccurate.
09/30/2007 update: Click here to read about how I can similarly block around 60% of spam just by arbitrarily listing 42% of the internet.
Based on this data, and the recommendations of other trusted blocklist operators and anti-abuse folks, I personally would not use APEWS to filter incoming mail.
Controversy and Commentary
The blocklist is considered controversial by many other blocklist operators, ISP abuse staff, and anti-spam advocates.
Matthew Sullivan, SORBS maintainer, indicates that as of August 9, 2007, SORBS will no longer be publishing the APEWS blocklist zones via DNS.
Claus V. Wolfhausen, maintainer of UCEPROTECT, another German-run blocklist, indicates that UCEPROTECT will no longer publish the APEWS blocklist zones. (Previously: Claus warned that unless APEWS were to make immediate, significant changes to its policies, UCEPROTECT will no longer publish the APEWS blocklist zones.)
Suresh Ramasubramanian, respected anti-abuse manager for large mailbox provider Outblaze, categorizes APEWS as “meant to be used by fools.”
Steve Linford, Spamhaus maintainer, has suggested numerous times on newsgroups and elsewhere that APEWS is poorly run and is not widely used.
- Kevin Liston and others from the Internet Storm Center have indicated that APEWS is using the ISC "top source" data to support blocklist entries, in violation of the data's license, and against the wishes of those who provide this data. ISC says that the data "is not supposed to be used as a blocklist as it is bound to include false positives" and that "APEWS may be a useful 'anti-spam" list if you do not mind losing a lot of valid e-mail as well."
Misplaced Newsgroup Discussion
If you read either of the two popular anti-spam newsgroups (news.admin.net-abuse.blocklisting and news.admin.net-abuse.email), you already know that both groups are often overrun with requests (example) from people who find that they are listed by APEWS. I find over 2,000 messages on these groups relating to APEWS remove requests, which is a high number considering that the blocklist is less than a year old. The blocklist group is run “anonymously.” Question 41 of the APEWS FAQ asks how one contacts APEWS. The answer includes the following: One does not. APEWS does not accept removal request by email, fax, voicemail or letters.” [...] “General blocklist related issues can be discussed in the public forums mentioned above. The newsgroups news.admin.net-abuse.blocklisting (NANABL) and news.admin.net-abuse.email (NANAE) are good choices.
This is likely why many administrators post to these newsgroups, asking for assistance, when finding their IP addresses are listed. The FAQ does warn that “abusing these newsgroups & lists by posting removal request you will make a fool of yourself,” but that doesn't seem to be a deterrent. I would theorize that this is because a lot of the people on the wrong side of listings do not understand why they are listed and do not now how to “fix” whatever issue led to the listing, as the listings are often broad and vague.
ISP Perspective
Vincent Schönau, an ISP abuse adminstrator, has related his APEWS experiences to me in email, and given me permission to share them here.
Other blacklists have employed the 'escalations' strategy in the past, but APEWS has taken it to a whole new level; a few spams from a providers ip ranges will cause all or most of the providers ip space to be listed in APEWS, with comments such as 'unprofessional / negligent provider'. What this means is that if your provider is a noticeable source of e-mail, sooner or later, it's going to get listed. Several providers of 'blacklist checks','blacklist comparisons', 'e-mail reputation checks' and include APEWS data. Apparently this is causing systems administrators who are desperate to reduce the amount of spam they're receiving to think that using it might work - perhaps because not all of those sources include the data on false positives for the blacklists. In practice, this means that several times a week, I'm spending time explaining to my users how they should work around the e-mail delivery-problems they're seeing which may or may not be related to APEWS. I could be spending this time taking action against compromised hosts in our network instead. This hurts providers who do take action against the abuse from their network more than providers who didn't care in the first place.
Others have related similar stories to me, of how long after spammers were booted, that a listing still persists. In one instance, a provider had a compromised machine, which was identified and disconnected within two hours of sending spam. Three days later APEWS listed it, and six weeks later, the listing persists, even though the issue is long since addressed.
If you are listed on APEWS and wondering what to do, visit this page for my suggestions.
Subscribe to:
Posts (Atom)