Online Virus / Malware News

Now that we’ve reached the middle of the year, it’s time to take a look at our malware collection. During the first half of the year, 10 million samples entered in our database. That’s certainly no decrease compared with last year.

With approximately 54,800 new samples arriving per day, the total size of our collection is almost 12 terabytes. At end of 2007, in contrast, and with only 5.8 million samples, the total size was only 1.1TB.

In June 2008, I posted a blog called And I say we are detecting between 400,000 and 10,000,000 malware! Two years later, I think we should compare the changes since this date.

First, let’s look at the main malware families–at left, my 2008 graph; at right, the current one. (Click to enlarge these and subsequent charts.)

From these we can see that malware developers have lost their creative spirit. Malware designers create their apps to make money, not for style. Because the old techniques still work, it is not necessary to be inventive, just repetitive. For example, it is not rare to see more than 10,000 Koobface variants in a single month.

My next 2008 graph concerned our figures compared to those of AV-test.org. At that time, I had only a short span, of four months, shown at left.

Two years later, we can observe another increase. Since June 2009, our collection has outgrown the AV-test database by two million. At that date, we started getting more samples from more sources.

To conclude my 2008 blog, I joked that we would detect at the end of 2008 between 450,000 (a figure related to the signature DAT readme file) and 22,000,000 malware (the count of malware samples in our collections). Between these extremes I also introduced another malware measurement scale that some anti-virus vendors use for comparison: the number of main variants.

Today when we quantify the malware world, the consensus is to use the number of unique files in our collections distinguished by their MD5 hash (or checksum). On June 30, we counted 43,337,677 unique binary files. Perhaps we’ll reach 54 million by the end of December.

The Anti-Malware Testing Standards Organization (AMTSO) is a coalition of security professionals, including many anti-malware product vendors, product testing organizations and publishers, and some interested individuals. Given the highly technical nature of its activities, it is inevitable that the organization owes some of its authority to the expertise of the security specialists within its ranks, but that doesn’t make it a vendor lobby group. As Kurt Wismer (not himself a member) points out here (http://anti-virus-rants.blogspot.com/2010/06/nss-labs-vs-amtso.html) “many of them are employed by vendors precisely because that’s one of the primary places where one with expertise in this field would find employment.” Given some recent negative publicity aimed at AMTSO (example: http://kevtownsend.wordpress.com/2010/06/27/anti-malware-testing-standards-organization-a-dissenting-view/), we want to collectively clarify the following points on behalf the anti-malware industry, where we come from, and indirectly on behalf of AMTSO.
We find it strange that expertise in the testing field is somehow seen as a disqualification, given the specialist expertise that characterizes the group.

Although some distrust anything a vendor says and accept uncritically anything a tester says, others are puzzled that different tests can vary so dramatically in their evaluation of the same product. Though this may sometimes be simply due to poor testing practices, there are other, deep-seated reasons, one being the high volume of malware and new attacks seen every day. Vendors work hard to close the gap between the ideal of 100 percent detection and what is actually achievable–by developing a range of technologies, both proactive and reactive. The capabilities of products can change, while tests using broadly similar methodology can generate dramatically “conflicting” results due to different approaches to the selection, classification, and validation of samples and URLs, among other factors.

AMTSO aims to promote precisely the kinds of tests that clearly demonstrate these variations, and its members were flying the flag for real-world testing before AMTSO ever formally existed, believing that sound testing benefits vendors and customers as well as testers. As an industry, we are all too aware that we cannot currently offer detection of all known and unknown malware. The relatively high scores achieved in established tests by major vendors do not necessarily reflect real-world performance, but real-world detection cannot be measured in product comparisons with no checks on selection, classification, and validation of malicious samples and URLs.

Another misconception is that AMTSO members simply don’t like tests done by non-AMTSO members. This is not the case: None of the undersigned have a problem with labs that intend to provide objective, real-world testing. (However, other testers are entitled to object vehemently when one company claims to be the only one doing live, Internet-connected testing, and that all other testers are doing static testing based on the WildList.)

However, charging consultancy fees for the release of any information relating to a test (even to participants) is very different to the transparency that AMTSO advocates, although we recognize that full-time testers must generate revenue like any other business. However, when a tester claims to have shared information about methodology in advance and subsequently fails to provide methodological and sample data, even to vendors prepared to pay the escalating consultancy fees required for such information, this suggests that the tester is not prepared to expose its methodology to informed scrutiny and validation. This stance compromises the tester’s aspirations to be taken seriously as a testing organization in the same league as the mainstream testing organizations committed to working with AMTSO.

No one believes that AMTSO has all the answers and can “fix” testing all by itself, but the organization has compiled and generated resources that have made good testing practices far more practicable and understandable. The way for testers (and others) to improve those resources is by talking to and working with AMTSO in a spirit of cooperation: The need for transparency is not going to go away.

Roel Schouwenberg, Kaspersky Lab
Luis Corrons, Panda Security
David Harley, ESET
Mark Kennedy, Symantec
Igor Muttik, McAfee

For a comparative review of anti-malware products to be useful to you, it has to be correct, comprehensive, and objective.

Unfortunately, producing a good test is not a simple task. You may think that it is, but it is not. It is like with cars – some are more reliable, some drink more petrol, some handle turns more confidently. But all cars normally do what they are sold for - transport you from point A to point B. Do you think that testing cars is easy? There are myriads of things to consider: safety, engine power, reliability, capacity, fuel consumption, color, size, seats’ shape, brand name, extras, trim quality. I think you may be getting the point. But surely any AV product is much simpler than a car!  Mmm, I am not so sure. There are many ways how a virus can get from point A to point B, you know. So there are myriads of technologies used to block malware propagation:

• static scanning for known malware
• generic family recognition for unknown variants of known malware
• behavioural techniques
• domain and URL blacklisting and reputation
• cloud-based protection
• frequent updating
• anti-spam
• virtualization and emulation
• programs’ reputation
• sandboxing
• blacklisting
• whitelisting
• security policies
• intrusion-prevention technologies
• …

For many years AV products did fairly well with just maybe 3-4 of these techniques. Today, we are seeing a lot more malware (which is actually partly due to how well AV products detect – when old malware is blocked that forces bad guys to create new one) and AV companies have to use more techniques to counter this tsunami. As a result, it became significantly harder to test AV products. Firstly, products are generally more complex now. Secondly, different products use different combinations of technologies and it is sometimes impossible to test them in the same way. Take, for example, retrospective testing – it requires “freezing” a product to test it against threats which appeared after the “freeze-point”. Such tests have been done for years but you cannot apply this test methodology to a product based on an active cloud-based protection database (cloud cannot be frozen as it is not under the tester’s control).

It will take time and investment of resources for tests to adapt to the new complexities of the products. Returning to our analogy with cars – you really have to build a crash-test lab to perform crash tests. Same for AV products – testers need to take entire AV suites and check how well it protects the PC from a virus “crashing” into it, so to speak.

Fortunately, there is some help available now for testers (and users who want to understand how better comparative tests can be set up). AMTSO (www.amtso.org), a non-profit organization, comprised of experts in testing matters (AV testers, AV vendors, publishers, academia) published two documents on their Wed site:

1. “AMTSO Whole Product Testing Guidelines” This paper suggests a holistic approach to verifying whether a product succeeded in blocking malware propagation – from point A (malware source) to point B (your computer).  The only important thing is whether malware is stopped at some stage – any contributing technology is allowed (even new and unknown ones!).
2. “AMTSO Performance Testing Guidelines” There are really many elephant traps on the route to a good test of speed offered by an AV product. This paper describes these pitfalls in detail.

It is likely impossible to execute a perfect test of AV products but it does not mean we should not strive for improving them. I am sure that new documents from AMTSO will provide useful information and will inch all of us a little closer to satisfying your needs for a good dependable test.

The Anti-Malware Testing Standards Organization (AMTSO) has published a paper on its website that addresses one of the most controversial subjects in anti-virus testing – Issues involved in the “creation” of samples for testing.

Many people within AMTSO (and I want to remind all our blog readers that this organization includes people from academic institutions, publishers, independent members as well as AV researchers) have traditionally felt that no malware should ever be created, period. This paper takes a balanced view on this subject and describes, in detail, what “creation” means and assesses all the pros and cons of sample creation.

If you are interested in AV testing or other ethical issues related to malware writing then I would thoroughly recommend you read this paper. The paper itself can be found in the AMTSO documents repository –  http://amtso.org/documents.html (Note:  you will need to OK the license agreement to get access). Enjoy the read!