Executive Summary: This report discusses Microsoft 365 and its Defender product for email. In general, Microsoft 365 is a very secure service. That is a result of a massive and continuous investment from Microsoft. It is one of the most secure SaaS services on the market. This report does not indicate otherwise.
What this report does note is the challenge that Microsoft has. As the default security for most organizations, many hackers think of email and Microsoft 365 as their initial points of compromise. A good example of how hackers focus on Microsoft 365 comes in a series of blogs from Microsoft that details the attempts of a state-sponsored group to compromise their services.
Before unleashing an attack, hackers will test and verify that they are able to bypass Microsoft’s default security. In other words, they are crafting attacks that are specifically designed to take advantage of getting around Microsoft and landing in the user’s inbox. This is likely why we’re seeing a higher percentage of attacks bypassing Microsoft security. In fact, Harmony Email & Collaboration does not see the attacks that Microsoft block. Instead, we only see the attacks that bypass Microsoft’s security.
Email security is essential across all businesses, especially here in India, given that 70% of malicious files in India were delivered via Email in the last 30 days, according to Check Point Threat Intelligence.
In this context, when our analysis demonstrates that a higher percentage of attacks bypass Microsoft’s security, it’s important to note that this does not mean Microsoft’s security got worse. It means that the hackers got better and faster and learned more methods to obfuscate and bypass the default security.
For this reason, Check Point chose architecture for cloud email security that adds a layer on top of Microsoft’s security rather than replacing it. It’s why our security experts recommend that our customers add an extra layer of security on top of their default cloud email service.
The Results
Harmony Email & Collaboration (HEC) found that 18.8% of phishing emails bypassed Microsoft Exchange Online Protection (EOP) and Defender and made it to a user’s inbox.
This increase may be due to a few factors. For one, we conducted our previous analysis before COVID-19. Since then, the number of attacks has increased by almost 100%, focusing on sophisticated phishing campaigns that bypass built-in security. Further, the increase in phishing emails caught by HEC is due to many factors, including the acquisition by Check Point in 2021, furthering our AI and ML capabilities.
Additionally, HEC’s unique architecture allows it to learn from the specific emails that Microsoft misses. These are often highly targeted attacks that are designed specifically to bypass Microsoft’s protections. This increase is partly due to hackers aggressively targeting Microsoft with specific, sophisticated attacks. Since Microsoft is the default for so many, hackers know they can target most users by aiming their weapons at Microsoft. The sheer volume and sophistication is unmatched. Thus, the higher number of bypassed emails reflects a concrete, focused effort by hackers worldwide to develop tools that will get in front of Microsoft users. The emails that bypass Microsoft, then, are incredibly sophisticated and evasive.
For example, Microsoft Defender released the real-world email shown below and delivered it to an end user’s inbox. It describes a business proposal. While this email doesn’t look like a traditional phishing attack, the file attached to the document is a macro-infested Excel document. Check Point Research’s analysis of the infected file shows it’s malicious and a critical security risk.
In the balance of this report, we will offer data on Defender’s effectiveness and explore how Defender fares in organizations of different sizes. We’ll closely examine how the “Dumpster Diving” phenomenon still hurts organizations and show how Defender fares against various types of phishing. And finally, we’ll summarize how these numbers affect your SOC and your business.
This report is not necessarily representative of all Defender environments as it’s derived from analyzing customers that also have the Avanan/Check Point Email Security Solution. We believe the data set is a statistically significant sample of Microsoft 365 customers with some variance in the results for different organizations. The results should be taken as a general guide to Microsoft’s efficacy.
How We Analyzed the Data
HEC analyzed nearly 3,000,000 emails scanned by Microsoft and HEC security products over one week. The organizations in our sample ranged in size from 500 to 20,000 users. These organizations were from different industries and located in all parts of the United States.
We picked these organizations because they run both Microsoft Defender and Harmony Email and Collaboration Security. Many companies choose this deployment model with Harmony Email and Collaboration Security as the last layer before an end user’s inbox. Because HEC runs inline behind Defender and connects via API, we can look at the following to help with classifications:
- Email headers are added by Defender after files have been scanned but before they are released into quarantine, junk folders, or user inboxes
- Defender event logs
We then looked at messages that HEC’s SmartPhish classified as phishing, but Microsoft Defender did not. Without HEC, these emails would’ve been delivered to the inbox or junk folder. That number of emails classified as phishing by HEC divided by the total email volume gives us the percentage of missed phishing emails.
The missed phishing rate for Microsoft Defender is 18.8%. In HEC’s last analysis in 2020, we found that 10.8% of malicious emails reach the user’s inbox. Since 2020, Defender’s missed phishing rates have increased by 74%.
Defender’s Missed Phishing Rate is Higher in Larger Organizations
Our analysis found that Microsoft Defender missed more phishing emails when deployed in larger organizations. The missed phishing rate for two large organizations in our research was 50-70%.
This is a change from our 2020 analysis. In 2020, we found no correlation between company size and missed phishing rate. In the 2020 analysis, one organization with 53,000 users had a miss rate of 2.6%. Another company with 279 users had a 17.4% rate.
Several factors may be at play here. We conducted our 2020 analysis just before the COVID-19 pandemic. Now, many more employees are working from home, the volume of attacks has increased, and Business Email Compromise (BEC) and ransomware demands have become headline news worldwide.
For reasons we will discuss later in the report, the type of attacks used by bad actors is changing. Our analysis found that targeted financial attacks are specifically crafted to bypass Defender. Financial-based phishing attacks refer to many email types, including fake invoice scams, fraudulent Bitcoin transfers, phony business proposals, fake wire transfer requests, and more. Defender misses 42% of these types of attacks.
In general, large organizations are inundated with financial attacks. According to the Verizon Data Breach Investigation Report, financially motivated attacks are the most common. The two most common cybercrime-related terms on criminal forums are “bank account” and “credit card”.
The Dumpster Diving Problem
Another part of our analysis included emails Defender forwarded to a user’s Junk Folder. In May 2020, Avanan coined the term Dumpster Diving. This refers to the practice where marketing emails, subscriptions, and targeted phishing attacks are commingled in the Junk folder, making them immediately accessible to the end-user. Many organizations send all Defender detections to the Junk folder. They deem this preferable to sending Defender detections to quarantine because it reduces the risk of blocking legitimate emails.
We found that, on average, Defender sends 7% of phishing messages to the Junk folder. End-users become accustomed to dumpster diving in the Junk folder for legitimate messages. Users may act on a phishing email by mistake, with many emails to root through in the junk folder without distinction between treasure and trash.
Defender Versus Different Types of Threats
Every phishing message falls into a category. Some are purely social engineering. Others try to get financial information. HEC can classify and see how Defender performs against different types of phishing.
As a quick overview, here are brief definitions of these types of phishing:
- Finance: This category refers to attacks having to do money, ranging from bitcoin scams to invoice fraud, to fake wire transfers and more
- Brand impersonation: This category refers to phishing emails that look like they are coming from a popular and legitimate brand
- Credential harvest: This is a broad category that refers to types of phishing emails with the broad goal of stealing usernames and passwords
- Social engineering: This is another broad category that refers to the idea of hackers tricking a user into doing something they don’t want to do
- Business Email Compromise: Business Email Compromise (BEC) is a popular attack form that sees a hacker impersonating an executive, asking an underling for urgent action
- Taxes: This is a subset of phishing that takes advantage of tax fears to extract money
As discussed previously, targeted financial attacks are often successful against Defender. This broad category includes anything involving money – fake invoices, bitcoin transfers, etc. These scams tend to hit the enterprise space in larger quantities.
The email below shows an example of a financial scam missed by Defender. If a user clicks the purported business proposal link, it takes them to a fake Microsoft 365 portal.
There are a few ‘tells’ in the email. In the body, notice how a zero is used in place of the letter ‘o’ in the subject. For example,
After clicking, users are directed to this page. Pay attention to the URL and the off-center text.
Notice how this spoof of a Microsoft login page has a random URL.
Brand impersonation is another method hackers choose to bypass Defender. These are emails that claim to come from a well-known brand but are a hacker trying to get information.
In this attack, hackers spoof Best Buy. However, the reply-to-address offers some clues as to its legitimacy.
Finally, credential harvesting attacks saw success against Defender. The idea is to steal something from the end user. These attacks range from attempts to harvest Microsoft 365 login credentials to something more sophisticated, like the attack seen below.
In this attack, the end-user sees an email from what appears to be PayPal. It shows a purchase of $509.49. At the bottom, it encourages recipients to call a phone number if they do not recognize the transaction. When they call the number, it gets routed to a ‘support rep’ who asks for banking information.
The Impact on the SOC
Harmony Email and Collaboration’s Security platform is the only security solution that can prevent malicious content from reaching a user’s inbox. HEC operates “Inline” meaning that the platform scans all emails and attachments before they reach a user’s account. The platform scans for potentially malicious content using AI and Machine Learning algorithms, alerts SOC staff when an issue is found, and quarantines the harmful content.
As this report has noted, the three million emails analyzed by our research team used HEC Inline with Microsoft Defender to catch what Defender missed. 95% of HEC customers use inline mode, which gives us unique visibility into Microsoft Defender.
To evaluate the impact on the SOC, we looked at companies using Monitor Only mode. In Monitor Only, SOC staff will be alerted to issues, but the problems will need to be assessed and responded to manually. Avanan research found that manually managing email problems takes up to 23% of staff time. That work involves remediating email threats, responding to end-user requests, and more. In some environments, that number is even higher.
In larger companies, SOC staff devote an even larger percentage of their time to email issues. One large company saw 910 reported phishing emails within one week. The IT team could only remediate 59 of these or less than 7%. This organization said they would need 16 full-time employees to deal with the user-reported phishing problem.
Across the last year, we’ve seen some tremendous issues for IT staff. One organization spends 879 hours or 36 full days. Another organization spends 2,500 hours a year just reviewing suspicious email reports from end-user. That’s the equivalent of 104 days. That time drainage leads to other priorities being overlooked and massive burnout among the IT and engineering staff. And with users reporting both actual and imagined phishing emails, the SOC spends far too much time sifting through the smoke, unable to locate the fire.
What Defender Does Well
In addition to the sections outlined below, we’ve observed that Defender does a good job with malware–Check Point has found that, with unknown malware, Microsoft catches 90%. Further, Microsoft Defender includes URL rewriting, a key feature to prevent time-of-click attacks. In our analysis, we’ve observed that in the environments sampled, Defender limited the amount of phishing in these categories:
DMARC Spoofing
DMARC, or Domain-based Message Authentication, Reporting & Conformance, is an email standard used to authenticate an email. It’s a way for operators to identify legitimate emails. The idea is to prevent hackers from spoofing an organization.
However, in our analysis, we found in 2.5% of cases, when the Defender finds a case of a spoofed email and DMARC failed, it still sends it to the Junk Folder.
Business Email Compromise
Business Email Compromise (BEC) is one of the fastest-growing and most successful attack vectors. Though simple in nature, it works because it utilizes vast social engineering and doesn’t include any malware or malicious links. In general, the idea is to spoof an executive, asking an underling for an urgent favor. This can be the purchase of gift cards or other financial transactions. Since 2016, BEC-related losses have totaled over $43 billion.
Despite that, in our analysis, Microsoft did a good job of limiting these attacks from reaching the inbox, allowing just 2.0% of these attacks to reach the inbox.
Defender Vs Secure Email Gateways
In one study analyzing 300 million emails, we found that Microsoft is in the middle of the pack compared to the rest of the competition, in this case, Secure Email Gateways.
Per every 100,000 emails, Microsoft’s catch rate of phishing emails is better than some Secure Email Gateways and worse than others.
Where Microsoft does shine, as mentioned above, is malware. After analyzing 360 samples of both known and unknown malicious PDF, DOC, XLS and other executable files, we were able to determine the catch rate for all malicious file types:
Breaking this down even further, for just unknown malicious Office Docs and PDF, Microsoft’s efficacy is even higher:
When considering email security options, it’s important to consider the entirety of a solution’s offering.
Conclusion
Microsoft is the most used and most targeted email service in the world. After thoroughly analyzing nearly three million emails, HEC found that Microsoft Defender misses 18.8% of phishing emails. This represents not a decline in Microsoft’s effectiveness but rather an increase in targeted attacks designed directly to bypass Microsoft. Hackers, in other words, have stepped up their game.
HEC’s email security architecture sits behind Microsoft. When Microsoft blocks an attack, the attack is blocked, fully stopped. When it doesn’t, however, HEC sits between it and the inbox, giving a final analysis. Our AI has the unique ability to be trained on these specific, sophisticated, and evasive attacks.
Note: The author of this article is Manish Alshi, All views and data expressed in the post are the author’s, and the publication does not claim rights to it.
Also Read: Businesses Need To Invest More In Cybersecurity Training and Awareness: Kallol Sil
