Ultimate Guide to Deliverability
How Spam Filters Work
There are countless articles on the web entitled, “How to Avoid Spam Filters.” The problem with most of these articles and their advice is that they’re based on the false premise that it’s even possible to avoid spam filters.
Spam filters are part of the process. If you send email, it will be filtered—either to the inbox, a categories tab, a spam folder, or it will be blocked completely. Filter technology plays a massive role in the success of your email campaigns. That’s why, at Return Path, we encourage our clients to embrace spam filters, learn how they work, and understand how mailbox providers use them.
What is a filter?
Email filters organize email according to specified criteria. Originally, filters were designed primarily to identify spam and block it or place it in the spam folder. Today, some mailbox providers use email filters to categorize messages for inbox organization purposes (e.g., Gmail categories or Microsoft’s Focused inbox).
Why do mailbox providers filter email?
Mailbox providers have strong motivations to use spam filters, whether they build their own system, leverage third party spam filter technology, or use a combination of home grown and partner anti-spam solutions.
Spam is annoying, no doubt, but it can also be dangerous. Malware and phishing are hugely profitable for scammers and can be costly for mailbox providers’ customers, as well as the mailbox providers who face intense market competition. Practically speaking, spam filters drastically reduce the load on server resources, considering that 70 percent of all mail sent globally is spam.
Types of Filters
As a message traverses from the sender to the subscriber’s inbox, various types of filters can influence deliverability and inbox placement:
- Gateway spam filters are physical servers that are installed at the border of a company’s network, and serve as a mailbox provider’s first line of defense in preventing spam. All mail attempting to come into the company must pass through this “gate” before it can enter the system. This spam filter learns what to deem as spam based on all the email coming into the company, which means it has less email data to learn from than a hosted spam filter (discussed below). Examples of gateway spam filters include Cisco’s IronPort and Barracuda.
- Third party (or hosted) spam filters are companies that have developed a proprietary method of using content and reputation metrics to distinguish spam from legitimate mail. Third party spam filters can influence filtering decisions at the gateway or after a message is accepted through the gateway (i.e., spam or inbox placement). Since these spam filters have a large book of clients using their service, they have a broad scope of information to use in determining whether to deliver to the inbox, spam, quarantine folder, or block the message completely. Third party spam filters can include technology that is integrated by vendors directly into their own products. Examples of third party filters spam filters include Cloudmark and MessageLabs.
- Desktop spam filters are a version of third party spam filters that live on the end user’s computer. They are highly customizable by the individual, so they can be among the more difficult filters to pass through. An example of a desktop spam filter is Outlook, which uses Microsoft’s anti-spam filter SmartScreen to help filter email. SmartScreen uses the feedback from Windows Live Hotmail users to help distinguish legitimate emails from spam.
To find out more about how the top three webmail providers filter email, download the Marketers Guide to Gmail, Outlook.com, and Yahoo.
The basics of filter technology
Spam filter technology may be placed on both inbound email (email entering the system) or outbound email (email leaving the system). Mailbox providers use both methods to help protect their customers. Senders may encounter both types of filters but are mostly concerned with inbound filters.
Both outbound and inbound filter methods use algorithms, heuristics, and the more advanced form of heuristics known as Bayesian as part of their filtering technology. Algorithms in this context are rules that tell a program what to do. Heuristics work by subjecting email messages to thousands of predefined rules (algorithms). Each rule assigns a numerical score to the probability of the message being spam.
The main types of filtering analysis
Mailbox providers look at four main aspects of mail when making filtering decisions:
- The source of the mail
- The reputation of the sender
- The content of the mail they send
- Subscriber engagement
Spammers often attempt to “game” reputation systems by using multiple (and constantly changing) IP addresses or domains. However, spam filters look at factors such as sender authentication, sending permanence, and the age of the IP addresses and domains in their filtering decisions.
To that end, email sent from new IP addresses and domains is treated with caution by mailbox providers. Senders that change IP addresses and domains infrequently and use authentication techniques can be seen as more trustworthy, which may lead to a stronger sending reputation.
The reputation of the sender is calculated using algorithms and heuristics, leveraging millions of data points and hundreds of parameters. Based on the strength of the reputation score, mailbox providers will make filtering decisions about the email coming from a sender.
Reputation based filters can automatically apply the mailbox providers’ mail flow policies based on the reputation score of the sender. As the filter receives inbound mail, a threat assessment of the sender is performed. Some of the parameters leveraged to generate a reputation score are:
- Spam traps
- Message composition
Each mailbox provider uses their own undisclosed reputation formula, however they all incorporate similar elements into the calculation. Return Path’s Sender Score uses data points and reputation formulas similar to what most mailbox providers use to give a relatively accurate representation of how mailbox providers view your email.
Content analysis technology scans every part of an email, including the header, footer, code, HTML markup, images, text color, timestamp, URLs, subject line, text-to-image ratio, language, attachments, and more. For some content filters, there is not one part of the message that is ignored. Other content filters may look at only the structure of an email, or they might simply parse URLs out of the message and then reference them against blacklists.
In addition to the source, sender reputation, and message content of incoming mail, mailbox providers have begun to look at historical engagement metrics to determine whether or not to place incoming mail into the inbox. This allows them to not only judge whether incoming mail is legitimate, but also whether it is desired by their users.
Proving “desirability” is not as simple as it might sound. While you might believe your content is doing well and generating a valuable ROI, mailbox providers might not agree with you. This is because they use different metrics to measure your desirability than then ones you use to measure your email’s effectiveness. They are also evaluating you against other senders.
Mailbox providers define engagement as the positive and negative actions that users take in their mailboxes. Metrics such as read rate, reply rate, and forward rate show that users are opening, responding to, and sharing your content with others while negative metrics like complaint rate and delete before reading rate are strong indications of disinterest.
Spam filters are an integral part of the email ecosystem. Without them, email simply wouldn’t work—billions of spam messages would overload the system. Filters are our friends; as users we appreciate when they keep unwanted mail out of our inbox but we’re also pleased to get the mail we do want. And filters are the reason that important transactional emails land in our inbox, where we can find them when we need them.