Reputation Block Lists: Protecting Users Everywhere

Cybercrime is a constant problem. What many Internet users don’t know is that the vast majority of users are being protected all the time by Reputation Block Lists (RBLs). RBLs are built into the Internet services we all use every day to keep harmful material out of your email inbox, warn you away from phishing attempts, keep viruses off your organization’s network, and more.

RBLs are lists of domain names, Universal Resource Locators (URLs), and/or Internet Protocol (IP) addresses that have been investigated and subsequently identified as posing security threats. Security systems throughout the Internet examine traffic constantly to try to keep harmful material from reaching victims. For example, some security systems ensure all traffic that arrives from a blocklisted IP address or all requests that match a blocklisted domain name or URL are rejected or quarantined.

RBLs are created and maintained by commercial service providers, researchers, and public interest communities that operate the means to detect or receive notification of security threats. The lists vary in focus and in detection methods, and many lists specialize in one kind of threat. But well-run RBLs typically have well-defined criteria for listing an identifier as a threat as well as the process for removing it from the list. With these RBLs, accuracy is a priority; RBLs have an obvious interest in the reliability and quality of their work – if a list is not acceptably accurate and trustworthy, no one will use it.

Reputation Block Lists are ubiquitous, and are a proven way to protect Internet users. RBLs have been in use for twenty years. During that time, they have been one of the most widely deployed and effective security solutions on the Internet. It is likely every type of entity relies on RBLs, including companies, governments, nongovernmental organizations (NGOs), mobile networks, Internet service providers, email service providers, and social networking sites. In this blog, we take a look at the these often-unseen but indispensable systems.

 

I Don’t See That Much Spam! Where Does It Go and Why?

RBLs are a big reason why Internet users are not buried beneath a daily avalanche of spam and the threats it spreads, which include malware, phishing attacks, and scams. The RBL providers examine links in email messages and identify the Internet sources of this malicious material.

Users may be surprised to learn just how much spam is being blocked. Cisco’s Talos email reputation system combs through billions of emails a day. Cisco reports that 80 to 85 percent of all email sent in the world is spam. On any given day, Cisco observes between 300 and 400 billion spam messages transmitted. And Cisco is seeing and blocking only a portion of the world’s spam. RBLs are a main reason all that garbage doesn’t end up in our inboxes.

Figure 1: Spam Volume (Source, Cisco Talos Email & Web Traffic Reputation Center)

Since users don’t see most of the spam messages that are sent to them, they sometimes conclude that the spam doesn’t exist, or that spam is no longer a threat. These are dangerous conclusions; the harm done by spam-borne threats and the cost of dealing with spam are huge. MailCleaner reports that £1.6 billion is lost worldwide in productivity annually due to unsolicited emails. In recent comments to the U.S. Federal Trade Commission, a small ISP in Utah, USA, with 37 employees testified that:

• It spends more than $280,000 a year to deal with spam and related harm.
• It has 13 servers dedicated to filtering and protecting against spam.
• It has 2 full-time employees, plus 13 part-time employees, dedicated to dealing with the problem.

RBLs in the Browser: Gone Phishing

Web browsers almost universally employ RBLs to protect users when they surf the World Wide Web. For example, Internet Explorer and Google Chrome use the Anti-Phishing Working Group (APWG) RBL (among others) to display phishing warnings if you navigate to a dangerous URL.

Below: Phishing Warning Page in Google Chrome

Google Safe Browsing is an RBL that further protects Chrome users by displaying malware warnings in Google’s search results. Google also makes its Safe Browsing RBL available to anyone who wants to check URLs.

RBLs in the Cloud and Content-Serving Systems

Content distribution networks like Akamai use blocklists such as SURBL, Symantec, and ThreatSTOP, and sometimes build and maintain their own RBLs. These companies use RBLs to make sure that they are not delivering malicious material to their customers, and that their customers are not sending malicious material to other people. Cloud provider Amazon Web Services (AWS) has built a web application firewall (WAF) system to help its users integrate multiple blocklists to block requests from hostile or misbehaving IP addresses.

Another kind of content-serving network—online advertising—uses RBLs to prevent their systems from accepting and serving ads that take people to fraudulent or malware sites. For example, Google uses Google Safe Browsing to block harmful or fraudulent advertising in the Google AdWords program.

RBLs in Your Social Media Tools

Companies such as Facebook and Twitter create and/or use RBLs, too. Facebook makes its ThreatExchange platform available for other companies to use – it safeguards itself and its users with malware and phishing URLs shared by other trusted participants and makes that information available to others via ThreatExchange.

RBLs and the DNS

Increasingly, Internet service providers and other network operating organizations are protecting their users from security threats by configuring what are known as resource policy zones (RPZs) at their recursive resolvers. As the name suggests, a resource policy zone is literally a zone file. The data in that zone file contains Domain Block Lists (DBLs) – lists of domain names that are known or suspected to resolve to IP addresses that host malicious content or control botnets. RPZs essentially create DNS firewalls at recursive resolvers. Before attempting to resolve a domain name for a DNS query, RPZ-defended resolvers first check an RPZ; if the requested domain name is present in the RPZ data, the resolver can return a negative response. The resolver may also redirect web users to pages that warn those who attempted to look up the domain name of possible infection and may provide directions for remediation. Infoblox, Spamhaus, SURBL, and others provide RPZs, and an entire security market segment now supplies DNS firewalls.

Security systems such as firewalls, application proxies, and antivirus and antispam gateways typically accommodate the inclusion of internal and external RBLs through administrative configuration. We peeked at the administrator guides of Palo Alto Networks, Barracuda Networks, SonicWall, Check Point, Fortigate, Cisco IronPort, and WatchGuard. All provide their own RBLs and allow customer administrators to include custom or external RBLs such as Spamhaus, SURBL, SpamCop, Invaluement, abuse.ch, Open Relay Database Lists (ORDBL), Spam and Open Relay Blocking System (SORBS), Squidblacklist.org, and others to block malicious IP addresses, URLs, and/or domain names.

Corporate email and messaging systems such as Microsoft Exchange can also be configured to include RBLs. For example, administrators use server-side spam filter software such as GFI MailEssentials, SpamAssassin, or Vamsoft ORF to include Spamhaus or SpamCop RBLs in their antispam measures. Products in the antispam, antivirus, and unified threat management market such as TitanHQ SpamTitan, Sophos UTM, or Proofpoint also provide RBL-based filtering to protect users from visiting malicious URLs.

RBLs and Third-Party Service Providers

RBLs aren’t exclusively used by organizations that control their IT infrastructures and do filtering themselves. Many organizations outsource their security, email, and DNS needs to external services providers. These outsourced solutions can be less expensive and more secure than if managed internally. Organizations that choose outsourcing often receive RBL filtering as part of the service package, and may not even aware that their service providers are using RBLs, or which ones.

For example, email service providers use RBLs in two ways. They use RBLs to protect their customers from problems like incoming spam and malware. They also monitor blocklists to make sure they and their customers are not being added to blocklists (using tools such as Amazon Simple Email Service RBL or DNS block lists). All reputable email senders want to maintain the good reputation of their domains and IPs so that their mail is accepted by other parties. And all reputable service providers want to make sure that their services are not being used maliciously by their customers.

Companies in the domain industry are typical – they often use an outsourced mail service to run their corporate email systems, ensure the deliverability of their outgoing mail, or both. Readers who are familiar with DNS resource records can see this by looking at Mail Exchange (MX) and Sender Policy Framework (SPF) DNS resource records: the labels in bold identify outsourced mail service providers.

 

Customer	Provider	DNS Resource Record
GoDaddy	Microsoft Outlook	godaddy.com. 3599 IN MX 0 godaddy-com.mail.protection.outlook.com.
Neustar	Proofpoint	neustar.biz. 299 IN MX 10 mxb-0018ba01.gslb.pphosted.com.
Donuts	Google	donuts.co. 3599 IN MX 30 ASPMX5.GOOGLEMAIL.com.
Afilias	Google	afilias.info. 299 IN TXT "v=spf1 include:_spf.corp.afilias.info include:_spf.india.afilias.info include:_spf.afilias.tech include:_spf.google.com include:_spf.cision.afilias.info include:_spf.intcorp.afilias.info include:_spf.servicenow.afilias.info mx ptr ~all"
Uniregistry	SendGrid, SMTP.com	uniregistry.com. 299 IN TXT "v=spf1 ip4:162.221.214.42 ip4:64.96.176.36 ip4:64.96.176.37 ip4:64.96.210.63 ip4:64.96.162.114 include:sharpspringspf.smtp.com include:mail.uniregistry.net include:_spf.createsend.com include:sendgrid.net ~all"
Tucows	Zendesk	tucows.com. 1199 IN TXT "v=spf1 ip4:216.40.44.0/24 ip4:216.40.42.17/32 include:_spf.zdsys.com ?all " "google-site-verification=vR8JTNtUguZQwmRUcm7P9cGg5fHSOU_ILZ7-lz-MzMw" "MS=ms54709270"
eNom	mailchannels.com, Google, Zendesk	enom.com. 299 IN TXT "v=spf1 a mx ip4:69.64.144.0/20 ip4:98.124.192.0/18 ip4:216.40.44.0/24 ip4:216.40.42.17/32 include:relay.mailchannels.net include:_spf.google.com include:stspg-customer.cominclude:mail.zendesk.com ~all"
SGNIC	Outlook	sgnic.sg. 299 IN MX 0 sgnic-sg.mail.protection.outlook.com.

Figuring out which email providers use which RBLs is often a simple search exercise. For example, we easily found citations of articles that study RBL use by Gmail, Yahoo! Mail, and Microsoft Outlook (e.g., "Which Blacklists Does Gmail Use?"). Dyn uses Spamhaus and SpamCop, among other solutions.

The Price Is Right!

Of companies that manage their IT infrastructures themselves, some use RBLs that are free of charge. This is a suitable solution for small businesses, not-for-profits, and NGOs. Spamhaus is free if you do less than 300,000 DNS-based Blackhole List (DNSBL) lookups per day. SURBL has a similar policy for its Free Query Service (FQS). abuse.ch runs Ransomware Tracker and several other free RBLs – its partners include Dyn, Neustar, Spamhaus, and SoftLayer. So for several market segments, rather than asking Who uses RBLs?, perhaps a better question would be: Why aren’t you using RBLs?

The Invisible Shield

In the Internet today, RBLs are seen by many as a critical element of cybersecurity – an effective and preventive response to persistent forms of cyberattacks. They’re a first line of defense in the services you use on the Internet. And they’re one of those things that we don’t think too much about until they fail. So, the next time you notice that you haven’t seen much spam, or haven’t had your bank account drained, thank the RBLs and those who maintain and implement them.

 

Disclaimer: This blog post includes many references to companies and products but is not intended to recommend any particular product. Where possible, we cite product literature for accuracy, but acknowledge the possibility of errors and of information changing after the publication date. Readers should conduct their own research and consult their own experts before making any changes to their systems.