Published in Editorials

Product Testing for Security Tools Must Change or Things Will Go From Bad To Worse

by on24 April 2017 1055 times

If you have been paying attention to the technical news lately you might have noticed more than a few articles pointing fingers back and forth between the AntiMalware company Cylance and the… well the industry. The argument (if you have not already read about it) goes something like this; the big AV/AM companies are accusing Cylance of stacking the deck in their favor when they demo their product against the competition. Cylance, for their part, claims that they provide a realistic test in comparison to what is usually done when it comes to AV/AM testing. Both sides have their points and it calls into question something that exists in all levels of the technical press and testing bodies; real world vs scripted testing.

When I founded DecryptedTech it was with the intention of finding a better and more realistic way to represent hardware and software to the consumer. I had noticed, after many years of testing, that the industry was more than happy to run a few scripted benchmarks, give an award and publish “results” with little to know actual analysis on what anything really meant. It was maddening to see this happen time and time again from sites that are well known and considered to be experts on the topic.

The issue does not stop at motherboards and CPUs though. It exists in all verticals of the technical market. Gaming reviews are another area where the distributor has massive sway on what (and when) information is released. If a sites does not play along with those requirements, well they get cut off (We know this from actual experience). All vendors are exercising more and more control over what can and cannot be said in a review. They are more than happy to cut off a site for publishing information (factual or not) that does not fall in line with what they want. This control by the industry actually hurts the consumer in a very significant way by allowing inaccurate (or incomplete) information to be released about a product.

In the security industry the issue is even more impactful. By using scripted or canned tests a developer can plan for these tests and ensure a good score. This means that a potential buyer might make a choice that leaves them unprotected and vulnerable to attack. Although it is important to make sure known threats are protected against, security products must be able to defend against more than yesterday’s threats. The idea that simple signature and hash checks can protect an environment is no longer one that we can tolerate in our security tools.

Threat actors can be lazy, but even when they are lazy they are creative. I have observed threat actors mutate even common tools like PSExec to get around hash blocking in a target environment. The tool is the same one they wanted to use, but the hash has been changed so that it does not pop on the threat list. I have also watched a live monitor of a compromised C2 server where the malware used was mutated multiple times a day to ensure that it was not being caught. This same system was also rekeying the encryption used to transmit control data as often as twice a day. They are doing this to get around your typical consumer grade AV/AM software and it is also working against corporate AV/AM software as well.

The next generation of AV/AM software has to be smarter than what we have today. It must not only be able to recognize malware from a list, but also when normal software or tools are being abused or used in a manner they should not be. This is not an easy task, but it is a necessary one. The next generation of attacker has a very solid understanding of how anti-malware and even process/application control solutions work and can get around those limitations surprisingly quickly. The use of custom malware combined with normal debugging and network tools is becoming more and more common so the old signatures are out the window as area most heuristic systems. This also means that scripted testing based on “known” bad samples and heavy handed actions are also out of the window. Using them paints an incomplete and inaccurate picture of what a solution can do and gives the consumer a false sense of security. It also enables solution developers to continue doing the same thing without changing anything. As long as they keep cashing checks (thanks in part of inaccurate testing) there is no reason to do anything different or new. If the testing changes to expose the inherent weaknesses of their products, the money will slow down and the industry will finally make the changes needed.

Of course this change needs to take place across the entire industry and not just security, but it would be great if it changed for security tools sooner rather than later. After all buying a crappy motherboard based on an inaccurate review is not going to result in the loss of 1 million Credit Cards, buy a bad AV/AM solution could.

Last modified on 24 April 2017
Rate this item
(2 votes)

Leave a comment

Make sure you enter all the required information, indicated by an asterisk (*). HTML code is not allowed.