Eudora in the News
Delete: Bathwater. Undelete: Baby.
By KATIE HAFNER
Published: August 5, 2004
LIKE many people these days, Jason Kim and Linda Crasco rely heavily on e-mail for their work, running a small educational research and evaluation company in Norwood, Mass. And like many people, they get plenty of spam, some 400 pieces of unwanted e-mail daily.
So when their company, Systemic Research, first installed a spam filter 18 months ago, they were impressed by the noticeable reduction in the amount of spam they received.
Several months ago, Dr. Kim and Mrs. Crasco were at a meeting when they ran into a program director they knew from the American Association for the Advancement of Science. She greeted them coolly. Puzzled, Dr. Kim and Mrs. Crasco asked what they might have done to offend her.
As it turned out, she had sent Dr. Kim and Mrs. Crasco an e-mail message suggesting that they work together on a grant application. The application deadline had since passed, and the acquaintance was more than a little miffed that she had gotten no response from them.
The two entrepreneurs were flabbergasted. Not only did they have no idea the e-mail had been sent, they had no idea that it had been snuffed out as junk.
As spam has become the bane of in-boxes, spam filters have come to the rescue. Added on to or built into popular e-mail services and programs like Yahoo Mail, Hotmail, Eudora and Outlook, the filters are helping many people feel less besieged by spam.
Some people who despaired of spam to the point where they were ready to change their e-mail addresses or give up on e-mail altogether are so happy with their filters that they now consider e-mail to have re-emerged as a viable form of communication.
Yet filters can also create more work, requiring constant vigilance and frequent tuning in order to teach the software to recognize certain types of junk.
And as the experience of Dr. Kim and Mrs. Crasco shows, filters can also make mistakes, siphoning important e-mail messages off to the electronic slag heap.
Some e-mail users are learning to make do with an imperfect system while others remain frustrated, even angry, that better mechanisms have not emerged.
The current gold standard in filtering is called Bayesian scoring. The method, which is used in the latest versions of Eudora, among other programs, works by looking at words in an e-mail message and assigning each a probability of being part of a spam message. The filter then analyzes the cumulative probabilities and comes up with the total likelihood that the message is spam, delivering a score of 1 to 100 - the higher the number, the more likely it is that the message is junk.
"It is fabulous," said Jean Armour Polly, who lives in Jamesville, N.Y., and is the author of several Internet-related books. Ms. Polly depends more on e-mail than most people she knows, and since she gets about 900 junk messages a day, it is worth it to her to fine-tune her filters.
"You have to spend some time training it as to what is junk," she said. "Then you set your tolerance level for junk, using a slider bar." Ms. Polly said she tells the filter to consider anything with a score of 45 or above as junk.
Matt Dudziak, Eudora project manager at Qualcomm, the company that sells the Eudora program, said that when version 6.0 of Eudora was released in September 2003, the filtering method caught 97 percent of spam e-mail.
Though still very good, Mr. Dudziak said, the filter is less accurate now. "Spam has changed in the last nine months," as spammers move to outwit filters, he said.
As spam evolves, the filters need to be trained to recognize new indicators. "When you 'junk" or 'not junk' a message, this is what you are doing," Mr. Dudziak said. If a legitimate message gets put into the junk mailbox by mistake, the user can find it, tag it as "not junk" and have it moved to the in-box. In the process, the filter learns to recognize similar future messages as legitimate.
Mr. Dudziak examined a few of the messages that got past Ms. Polly's filter and said they all illustrated the clever evolution of spam. One ad for pharmaceuticals, for instance, was surrounded by poetry. "Spammers have started using words the Eudora filter wouldn't consider junk in the first place," he said, "overloading the dictionary with those types of words, so when you calculate the total probability, the few spammy words aren't likely to outweigh the hundreds of valid words."
He said the Eudora anti-spam team was trying to devise new methods for spotting such messages, like scrutinizing just the first few lines.
Other messages that got past Ms. Polly's filter were filled with graphics. "They'll send you a message that just has a big graphic in it, and a graphic alone doesn't necessarily mean the message is spam," Mr. Dudziak said. "The filter doesn't know if it's an ad for Viagra or a picture of your kids."
One advantage of a filtering program like Eudora's is that it is under the user's control. If a legitimate message lands in the junk mailbox when it should not, the user can rescue it. This requires periodic perusing of the junk mail folder, but many people seem to be willing to do that.
Ms. Polly, for instance, knows that she must scan her junk folder every few days, looking for items scored below 60. "I occasionally 'not junk' those," she said.
But if e-mail messages get scrutinized at the server level - that is, at a company's central computer - the user never sees if a valid message gets mistaken for junk and thrown away.
That's what happened to Dr. Kim and Mrs. Crasco when the e-mail messages about the grant proposal were dispatched. Mrs. Crasco's best guess was that some of the words likely to have been in the lost e-mail message, like "opportunity," "potential," and "funding," had been spotted by the filter as potential spam.
More disturbing than having missed the chance to apply for the grant was the potential setback in the company's relationship with the science association. "If you get a reputation for ignoring people, that could kill us," Mrs. Crasco said.
So the company decided to stop filtering e-mail, and to have messages that the filter suspects are spam labeled as such but not removed. Now employees are back to wading through hundreds of messages at a time in search for the legitimate ones. If a valid message is tagged as spam, the user can notify the system administrator.
Mrs. Crasco, who spends 90 minutes a day wrestling with her in-box, is as unhappy with the unexpurgated approach as she was with the system that ate the message from the science association.
"We have an extremely busy small company," she said. "This is interfering with what I'm supposed to be doing."
Not everyone at Systemic agrees on the junk filter. "I'd rather lose a couple of important e-mails than go through 500 e-mails," said Dong-Hoon Lee, the company's director of information technology.
Few people have grown so frustrated by spam that they have given up on e-mail altogether. But Bernhard Schmidt came close. Mr. Schmidt, vice president for finance at Avalon Publishing in Emeryville, Calif., has gone from severe disgruntlement to nearly total satisfaction with his e-mail life.
Until recently, Mr. Schmidt was so overwhelmed by spam that he received on his personal Mindspring account that he considered dropping it as a form of communication. "I would dread opening the e-mail at home,'' he said. "It would fill up, day after day. And the longer you'd wait to check it, the worse it was."
Then Mr. Schmidt started using the filter program that came with Mail, the e-mail program for the Macintosh computer's OS X operating system. He said the Mail filter catches about 250 junk messages a week, and misses 15 to 20. And he occasionally checks for messages filtered by mistake and finds about two a month. "This junk filter has been a lifesaver for me," he said.
Three billion messages a day arrive at the computers that run Hotmail, a Microsoft service with 170 million subscribers. The service blocks 2.7 billion of those messages, said George Webb, business manager for Microsoft's anti-spam technology and strategy group. Of the 300 million messages that make it to people's in-boxes, Mr. Webb said, he does not know what percentage is spam.
But it is enough to keep Ed Kim, Dr. Kim's 24-year-old son, from relying on either of his two Hotmail accounts to keep his in-box free of spam. "This is life now," said Mr. Kim, who juggles five e-mail accounts in a constant effort to outrun the spammers. Whenever he buys something online, for instance, or takes an online quiz, he uses a Hotmail address. Mr. Kim guards his Harvard alumni account, on the other hand, with his life, giving it out only to people he trusts will not abuse it.
Rather than set a Hotmail account to a level of filtering that accepts e-mail only from pre-authorized senders, Mr. Kim lets the messages in but seldom checks the account. "They delete things older than five days old, so it's pretty good," he said.
Whenever it looks like the filters are catching up with the spam, the spammers find a way around them. "It's pretty much a cat-and-mouse game," said Miles Libbey, anti-spam product manager for Yahoo. "We make a change to the filters and as little as five minutes to two hours later, the spammers have adapted to our change."
For example, one "leading-edge spammer," he said, was evading Yahoo's "buy Viagra" detector by inserting random text printed in white between the two words. To the recipient, the white text was like invisible ink, but "a filter looking for 'Viagra' wouldn't see the word any more," he said.
Yahoo's basic filter, Spamguard, provides users with a button, for reporting spam after it enters the in-box. Spamguard Plus, for which users pay a premium, works as Eudora and Macintosh filters do, allowing users to tailor it to their habits.
Spam filters will keep evolving, of course. "Every time we come up with a new strategy, one of the questions we always ask ourselves is, How will the spammers get around it?" Mr. Libbey said.
And people will continue to cope in their own ways. Mrs. Crasco, Dr. Kim and their company will wade through unwanted messages until they have trained the system to their satisfaction. Ms. Polly will check her junk mailbox periodically to rescue a piece of e-mail mistaken for junk.
Manipulation is common: a piece of spam will arrive in an in-box with "sorry I missed you," or "regarding your application" in the subject line, in an attempt to pique just enough curiosity that someone will click on the message.
"I got tricked just today," Mrs. Crasco said. She got a piece of mail with "Mail Delivery Notification" in the subject line, and opened it to see if something she had sent had bounced back. It was an invitation to apply for a mortgage. "I thought, 'Got me,' " she said. "I've got to give them credit."
| Home | Online Support | Open Source Development | User Forums | Contact Webmaster |
© 1999-2009 QUALCOMM Incorporated. All rights reserved. QUALCOMM and Eudora are registered trademarks of QUALCOMM Incorporated. All other trademarks are the property of their respective owners.