[Asrg] FeedBack loops

[Chris Lewis]

Rich Kulawiec wrote:

> Incidentally, I recently concluded an analysis of nearly 5 years worth of
> feedback loop traffic from AOL.  (Which is the first one I started using
> on a site of appreciable volume.)  This analysis, partially automated
> and partially manual, arrived at the following interesting conclusion:
> the FP rate is 100.000%.  Every single feedback loop report identifying
> traffic as spam was wrong.

On a similar note, we average about 2-5 spamcop reports a week.  Only
two of them have been right in the past 5 or more years.

Rich's "result" is largely because he doesn't spam, his reports are by
fat-fingered TIS buttons.

Our "result" is because we have a /8, and therefore somewhere around 1%
of all forged Received headers are in our /8.  Even spamcop's header
parsing goofs, and when it does, we get dinged. [Yes, we deal with the
true positives, and Spamcop is made aware of the parsing goofs.  We are
satisfied with the current situation, despite the vast majority of the
reports being wrong.]

Our AOL FBL had a similar experience: 100% of our reports were wrong for
one of two reasons: fat-fingered TIS buttons, or, the fact that their
FBL generator couldn't cope with /8 declarations, and gave us reports
for someone else's allocation ;-)  AOL eventually turned it off because
we mutually decided it wasn't worth the bits.

That is by no means to imply that FBLs are always wrong, or even wrong
most of the time.  I'm sure that the vast majority of AOL's FBL reports
are absolutely right.  It's just that neither Rich nor us see a
"typical" picture.

> (By the way, has implications for the dubious approach of quarantining,
> which also relies on the ability of end users to make spam/not-spam,
> phish/not-phish, etc. decisions.  There is no reason to think that
> they're any better at that.)

Our experience has been, given the number of people who fall for phish
and other things, that the filters are more consistently accurate than
the humans in spam/non-spam determinations.  Even when the human is me
:-(  My FP rate of the quarantine folder (for an unfiltered account
relying on Tbird Bayes for local quarantine) really sucks.  Probably >50%.