view release on metacpan or  search on metacpan

share/SpamAssassin/easy_ham/01715.30f57f8851044a464064eec4c938963d  view on Meta::CPAN

> ...
> No, sorry.  These were all of the following structure:
>
>   multipart/mixed
>       text/plain        (brief text plus URL(s))
>       text/html         (long HTML copied from website)

Ah!  That explains why the HTML tags didn't get stripped.  I'd again offer
to add an optional argument to tokenize() so that they'd get stripped here
too, but if it gets glossed over a third time that would feel too much like
a loss <wink>.

>> This seems confused: Jeremy didn't use my trained classifier pickle,
>> he trained his own classifier from scratch on his own corpora.
>> ...

> I think it's still corpus size.

I reported on tests I ran with random samples of 220 spams and 220 hams from
my corpus (that means training on sets of those sizes as well as predicting
on sets of those sizes), and while that did harm the error rates, the error
rates I saw were still much better than Jeremy reported when using 500 of
each.


Ah, a full test run just finished, on the

   tokenization scheme that folds case, and ignores punctuation, and strips
a
   trailing 's' from words, and saves both word bigrams and word unigrams

This is the code:

            # Tokenize everything in the body.
            lastw = ''
            for w in word_re.findall(text):
                n = len(w)
                # Make sure this range matches in tokenize_word().
                if 3 <= n <= 12:
                    if w[-1] == 's':
                        w = w[:-1]
                    yield w
                    if lastw:
                        yield lastw + w
                    lastw = w + ' '

                elif n >= 3:
                    lastw = ''
                    for t in tokenize_word(w):
                        yield t

where

    word_re = re.compile(r"[\w$\-\x80-\xff]+")

This at least doubled the process size over what's done now.  It helped the
f-n rate significantly, but probably hurt the f-p rate (the f-p rate is too
low with only 4000 hams per run to be confident about changes of such small
*absolute* magnitude -- 0.025% is a single message in the f-p table):

false positive percentages
    0.000  0.000  tied
    0.000  0.075  lost  +(was 0)
    0.050  0.125  lost  +150.00%
    0.025  0.000  won   -100.00%
    0.075  0.025  won    -66.67%
    0.000  0.050  lost  +(was 0)
    0.100  0.175  lost   +75.00%
    0.050  0.050  tied
    0.025  0.050  lost  +100.00%
    0.025  0.000  won   -100.00%
    0.050  0.125  lost  +150.00%
    0.050  0.025  won    -50.00%
    0.050  0.050  tied
    0.000  0.025  lost  +(was 0)
    0.000  0.025  lost  +(was 0)
    0.075  0.050  won    -33.33%
    0.025  0.050  lost  +100.00%
    0.000  0.000  tied
    0.025  0.100  lost  +300.00%
    0.050  0.150  lost  +200.00%

won   5 times
tied  4 times
lost 11 times

total unique fp went from 13 to 21

false negative percentages
    0.327  0.218  won    -33.33%
    0.400  0.218  won    -45.50%
    0.327  0.218  won    -33.33%
    0.691  0.691  tied
    0.545  0.327  won    -40.00%
    0.291  0.218  won    -25.09%
    0.218  0.291  lost   +33.49%
    0.654  0.473  won    -27.68%
    0.364  0.327  won    -10.16%
    0.291  0.182  won    -37.46%
    0.327  0.254  won    -22.32%
    0.691  0.509  won    -26.34%
    0.582  0.473  won    -18.73%
    0.291  0.255  won    -12.37%
    0.364  0.218  won    -40.11%
    0.436  0.327  won    -25.00%
    0.436  0.473  lost    +8.49%
    0.218  0.218  tied
    0.291  0.255  won    -12.37%
    0.254  0.364  lost   +43.31%

won  15 times
tied  2 times
lost  3 times

total unique fn went from 106 to 94