One of my favorite things to do so far this year was to benchmark some security software and test how they integrate with real environments. The security software of choice this go around and the mode of testing is the next gen AV proof of concept.
NGAVs are essentially a collection of a few different security tools consolidated to provide a better picture of the state of your device and network health.
It’s worth mentioning that my notes below represent the offering of the products and the effectiveness of the customer success team in this period of time (early 2019). Your interaction may vary, especially if you see this much later. A lot can change quickly in this business.
Table of Contents
Crowdstrike POC
Here are my notes from the initial introduction and my time playing around with Crowdstrike Falcon:
- 3 main services: SaaS Endpoint Protection, Threat Intelligence, and Cloud Security
- Light installer and low memory usage
- AD integrated and can assign policies by OU
- Requires policies to be switched on to be effective, they do not setup or walkthrough policies
- Training courses and YouTube videos are available
- No references and no naming names
- Policy updates and the sensitivity of the software
- Need to restart to have machine appear in console
- Client is called Falcon and there are different package offerings
- Auto-quarantine executables
- Easy exclusions
- Keep history of scripts run, registry effects, and web origination information
- Need responder privileges as well as admin to issue cleanup, not readily apparent
- Has threat group lookup page for known threat actors
- SIEM and Syslog integration
What I Liked
Installation of the agent was a breeze. The hardest part of the process was removing the previous installation of the antivirus solution we used. You don’t have to restart the machine to get the install to finalize, however, if you want the computer to show up in the console, you need to restart.
The low resource requirements for running the agent are nice. The AV we were using was such a resource hog that ran multiple different processes. I also liked how you can run the agent (Falcon) silently. It was this way by default (I don’t recall if you could disable this).
Managing and assigning policies are a nice touch (once they are turned on). I came from manually creating policies and having to jump through hoops to edit and save them. Once we got the protection piece working properly (more on this in the next section), I enjoyed the easy exclusions for false positives and the info on the true positives. The group lookup page for known threat actors was a nice touch for investigating the background of certain strains of malware.
As far as the meat and potatoes go, I really enjoyed the history and web origination piece. This was leagues ahead of what I had seen at the time. The historical information was easy to find and laid out nicely.
I didn’t get a chance to integrate this into a SIEM but I did forward some logs to see how easy the integration would be. No disappointments here.
What I Didn’t Like
Even though the policies were easy to setup and assign, they require being switched on (this isn’t the default like other AV software). This was not readily apparent in the UI. Our contact didn’t do a good job explaining this or helping us setup anything, surprising for a customer success engineer. I eventually figured this out after seeing some of the test machines appear in the main console.
Speaking of our contact, the customer service success team did not do a good job introducing the software or how we could use it. Most of my notes above were from me asking quite a few questions. I’m sure there was more information about the product and service but getting to the point above took an hour. I’m guessing this person was new and didn’t know how to onboard people. This wasn’t the worst part though.
After going through quite a few tests and noticing that the software wasn’t stopping anything I threw at it – RATs, ransomware, scripts, and other viruses, I shared my concerns with the contact. Surely I was missing something since it didn’t catch anything. They didn’t know what the problem was (at least initially) and offered to send me their malware to see that the software does indeed work (LOL nice try).
It turns out that despite me telling this person exactly how I would test it, including replacing the current AV, they didn’t bother telling me that the endpoint protection part was not enabled by default and that the switch to turn it on was in its own settings menu. Also, after the product is fully enabled in 2 different spots, in order to cleanup found threats, the person reviewing and cleaning the threats needs the responder privilege as well as the admin privilege to issue cleanup commands. This is the first time I’ve seen the admin group not include every privilege. I don’t mind taking part of the blame on this, but if you are a hands on person with a lot on your plate like myself, you can understand my frustration.
After getting everything setup properly, I re-ran the tests. This was a lot of extra work for me, but I like to be thorough and properly test and setup things. I just feel that I shouldn’t be the one who cares more about this than the customer service success person trying to sell me the product.
Information and docs seem to be scattered. They have training courses, a YouTube channel, and the threat actors page – all of which are in different locations. Anytime I had a question, I was referred to the materials to look it up for myself. It seems like having a knowledgebase with everything together would be more beneficial.
On that note about things being scattered, there were so many clicks required to get to certain views, reports, and settings. I’m not convinced that all the info they provide is necessary, especially how they lay it out. It’s almost like the product was developed and designed by an engineer, without user interface or designer testing. I’m sure more feedback on this will help shape their offering for the better.
Final Thoughts
Despite my “Didn’t Like” section being longer than the “Liked” section, I really do like this company and their offering. They have a decorated history and their solutions have worked for many. Once I got everything setup properly things worked well. They were just difficult to get used to and use.
They wouldn’t be my pick now but I’d be happy to revisit in the future. I’m positive there is much more to this product and service than I got a chance to see. Working with a better customer service success engineer and a UI update would go a long way to making this experience better.
Update 5/19: Crowdstrike gets an IPO filing. More funding for continued research and operations is always a good thing.
Cylance POC
I had heard a lot of great things about Cylance Protect so I was excited to give them a try. This POC was a couple weeks after the Crowdstrike POC. Here are my notes from the initial introduction and my time playing around:
- Artificial Intelligence Based Advanced Threat Prevention
- They are able to recommend references from organizations of similar size
- They’re mostly able to name names
- Can do a contract, no reason to go out to bid
- Memory usage
- 116.8 MB for the Native Agent
- 4.1 MB for Cylance Protect
- There isn’t a Windows Defender requirement but they can run side by side
- Azure integrated with SSO
- Policies can be assigned by zones
- OUs and more
- Optics is more offensive
- MITRE framework capable
- Automated action
- Query packages and push exes
- Detailed threat data
- Evidence based
- License for these products are separate
- Protect and Optics
- Check their documentation area for admin guides on Protect and Optics (product names)
- They like to put it in monitor mode at first when they set it up and then switch it over as they onboard
- Watch for files and can stop new executable
- Auto-quarantine executables
- Protection with execution control
- Script control by file, folder, and group (zones)
- Seamless exclusions
- SIEM and Syslog integration
What I Liked
This customer success engineer was the complete opposite of the last one. I had 33 questions prepared and only had to ask a few of them. This person was so on the ball, I had trouble keeping up. Not only did they introduce the product and service, they also helped set everything up and gave a brief tour of the interface. Very nice. The ability to give references went a long way as well. I realize that not every organization would want to be a reference but it was nice that Cylance had a couple similarly setup organizations to refer us to.
As far as product offering, it’s very similar to Crowdstrike. The resource requirements are also low and the agent runs in the notification area. It’s kind of nice to see it there and what the agent is working on (by hovering the mouse over).
Managing and assigning policies are setup nicely through “zones.” Zones can be setup through more than just OUs, so I like the flexibility. The Optics piece is really versatile and automates a lot of the boring stuff. Of course the Protect side is fully capable, as it caught all of my tests. And was enabled by default. The engineer did ask if we wanted to disable it so we can run it side-by-side with another AV solution. We of course said no.
As far as the meat and potatoes go, the history and script controls are a nice touch. I also like the initial scan and executable control. They seem to be more flexible on the actual operation side of this newer security process. The interface is simple and it’s easy to find stuff. They don’t go too crazy trying to shove a bunch of analytics in your face.
The knowledgebase and documents were centrally located and easy to find information. When I asked questions they answered in addition to finding the docs for further reference. This was a nice touch.
I didn’t get a chance to integrate this into a SIEM but I did forward some logs to see how easy the integration would be. No disappointments here as well.
What I Didn’t Like
The initial scan was a full scan and it’s required to move forward. That in itself isn’t a bad thing, but doing this full scan was extremely slow. This is expected since this full scans are CPU and drive intensive but I’m surprised this is still a problem in 2019.
The interface looked old and dated. I think they mentioned an update was coming so this isn’t as much of a big deal.
That’s all I can think of for now. I know this section is really nitpicky but there weren’t a whole lot of negatives during this next gen AV proof of concept.
Final Thoughts
Even though Crowdstrike offers very similar products and solutions, Cylance really closed the gap for me. They excelled in everything I had previously had a problem with using Crowdstrike. Playing around with Cylance has been a blast and the customer service success engineer was awesome. After talking with a few people that use Cylance it’s clear why they are where they are. Setup was a breeze and everything worked well.
If I had to make a decision now, I would pick Cylance. The proof of concept was so smooth, it’s almost unfair that I engaged this solution right after the sub-par experience with Crowdstrike. Even though that is my experience, I would still recommend organizations looking to make the switch to a newer antivirus solution to engage in both of these companies for a next gen AV proof of concept at a minimum.
Update 3/19: As I’m publishing this article, I see that Blackberry recently acquired Cylance. This is interesting. I always keep an eye out for mergers and acquisitions. This could be because I graduate with an MBA with international honors or it could be that mergers and acquisitions are commonplace among tech startups, SaaS companies, and security companies. More funding for continued research and operations is always a good thing.
Other Next Gen AV Proof of Concept Options
If you need to find more than 2 options for an AV solution, here are a couple more options:
- Carbon Black Defense endpoint security with big data analytics.
- Update 8/19: VMware acquires Carbon Black
- Endgame Platform larger scope endpoint protection.
- Update 10/19: Elastic acquires Endgame
The next time I need to do this, I would like to test the additional options above, time permitting of course.
Update: Wow, 2019 was a good year for endpoint protection companies. So many acquisitions.
Update 2: One term that I’m starting to see more and more is EDR – Endpoint Defense and Response. If you’re searching for options to do a next Gen AV proof of concept, you may wish to search for next gen antivirus, NGAV, endpoint detection and response, or EDR.
