Internet Security Suites fail to block exploits and do little to protect users against exploits, according to a recently released "test" [here] by Secunia, a Danish vulnerability notification firm. I quoted the word "test" as it's very common to see vulnerability companies use close-to-unethical tactics to oversell problems with the AV industry in order to promote their own services [another example here].

Now its Secunia's turn. In their "test" they assume that anti-virus products have poor performance in detecting vulnerability exploits because of their limited focus on traditional AV signatures. So along comes Secunia's Chief Technology Officer (CTO) Thomas Kristensen with the bright idea of testing 12 different Internet Security Suites from McAfee, Norton, Kaspersky, Panda and others against a testbed of exploit files. So far so good, it’s an interesting idea for comparing technologies and I believe it should be performed.

However when testing exploits one very important aspect is that these products don't just rely on traditional signature detection. Yet Secunia's "test methodology" only takes into consideration manually scanning 144 different inactive exploit files. This is very much like saying that you're going to test a car’s ABS breaks by throwing it down a 200 meter cliff. Absurd, sensationalist and misleading at best.

Just to clarify, if you only test 1 part of a product against exploits, which by the way is the part of the product which IS NOT designed to deal with exploits, and leave out of the test the part of the product that DOES deal with exploits and vulnerabilities, there's a very good chance the results will be misleading. Mr. Kristensen, as a Chief Technology Officer, should know this and should be very well aware of the consequences of a faulty methodology. So the question remains, why did he ignore it and just go for the yellow sensationalist approach?

But the absurd doesn't stop with Secunia's flawed testing methodology. Mr. Kristensen concludes that "… major security vendors do not focus on vulnerabilities. Instead, they have a much more traditional approach, which leaves their customers exposed to new malware exploiting vulnerabilities." Well duh, if you only test traditional signatures and neglect the other technologies included in the product which ARE designed to block exploits, what do you expect? Oh, wait, I just saw on their website that Secunia actually sells a vulnerability scanner! Hmmm, I wonder if that has something to do with the flawed conclusions of this test… Internet Security Suites do not rely on signature detection alone since many years ago. Panda's (and other) products integrate behavioral analysis, context-based heuristics, security policies, vulnerability detection, etc. However none of these technologies were tested by Secunia.

Let's just take 1 of the many protection technologies included in Panda Internet Security 2009 which DOES deal with prevention of vulnerability exploitation and see how it behaves against these exploits if tested correctly. I'm talking about Kernel Rules Engine, a security policy technology incorporated in 2004 to all Panda products which effectively prevents zero-day exploits of PDF, DOC, XLS, PPT and many other vulnerable applications. While Secunia's test grants Panda a lowly 1.59% detection rate of the important threats, if they would have tested correctly they would have found out that just with Kernel Rules Engine Panda's product is able to generically and proactively block 56% of the important threats. And this just with KRE technology. But Panda's  products also include other technologies such as TruPrevent's Behavioral Analysis, URL Filters and the Vulnerability Detection module which would prevent other exploits if Secunia cared to run their tests with a minimum level of professionalism.

Note to Secunia:
The following exploits (at least), which in your study are marked as "not detected by Panda", are actually detected generically with the correct testing methodology. Hint: have you tried actually "running" the exploits?

SA14896 CVE-2005-0944 PoC.mdb
SA20748#1 CVE-2006-3086 PoC.xls
SA21061 CVE-2006-3655 POC1.ppt
SA21061 CVE-2006-3656 POC2.ppt
SA21061 CVE-2006-3660 POC3.ppt
SA22127#1 CVE-2006-4694 PoC.ppt
SA23540 CVE-2007-0015 PoC.qtl
SA23676#2 CVE-2007-0028 Exploit1.xls
SA23676#2 CVE-2007-0028 exploit2.xls
SA23676#2 CVE-2007-0028 PoC.xls
SA23676#3 CVE-2007-0029 PoC.xls
SA23676#4 CVE-2007-0030 PoC.xls
SA23676#5 CVE-2007-0031 PoC.xls
SA24152 CVE-2006-1311 PoC.rtf
SA24359#1 CVE-2007-0711 PoC.3gp
SA24359#3 CVE-2007-0713 PoC.mov
SA24359#4 CVE-2007-0714 PoC.mov
SA24359#8 CVE-2007-0718 PoC.qtif
SA24359#9 CVE-NOMATCH PoC.jp2
SA24659 CVE-2007-0038 GameOver.ani
SA24664 CVE-2007-1735 PoC.wpd
SA24725 CVE-2007-1867 GameOver.ani
SA24784 CVE-2007-1942 Exploit.bmp
SA24784 CVE-2007-1942 PoC.bmp
SA24884 CVE-2007-2062 GameOver.cue
SA24973 CVE-2007-2194 GameOver.xpm
SA25023 CVE-2007-2244 PoC.bmp
SA25034 CVE-2007-2366 GameOver.png
SA25044 CVE-2007-2365 GameOver.png
SA25052 CVE-2007-2363 GameOver.iff
SA25089 CVE-2007-2498 PoC.mp4
SA25150#1 CVE-2007-0215 PoC1.xls
SA25150#1 CVE-2007-0215 PoC2.xls
SA25150#3 CVE-2007-1214 PoC.xls
SA25178 CVE-2007-1747 PoC.xls
SA25278 CVE-2007-2809 GameOver.torrent
SA25426 CVE-2007-2966 PoC.lzh
SA25619#1 CVE-2007-0934 PoC.vsd
SA25619#2 CVE-2007-0936 GameOver.vsd
SA25619#2 CVE-2007-0936 PoC.vsd
SA25826 CVE-2007-3375 PoC.lzh
SA25952 CVE-2007-6007 PoC1.psp
SA25952 CVE-2007-6007 PoC2.psp
SA25952 CVE-2007-6007 PoC3.psp
SA25988 CVE-2007-1754 PoC.pub
SA25995#1 CVE-2007-1756 PoC.xls
SA25995#2 CVE-2007-3029 PoC1.xls
SA25995#2 CVE-2007-3029 PoC2.xls
SA25995#3 CVE-2007-3030 PoC.xlw
SA26034#4 CVE-2007-2394 PoC.mov
SA26145 CVE-2007-3890 PoC1.xlw
SA26145 CVE-2007-3890 PoC2.xlw
SA26433 CVE-2007-3037 PoC.wmz
SA26619 CVE-2007-4343 Exploit.pal
SA26619 CVE-2007-4343 GameOver.pal
SA27000 CVE-2007-5279 PoC.bh
SA27151 CVE-2007-3899 GameOver.doc
SA27151 CVE-2007-3899 PoC.doc
SA27270 CVE-2007-5709 GameOver.m3u
SA27304#1 CVE-2007-5909 GameOver1.rtf
SA27304#1 CVE-2007-5909 GameOver2.rtf
SA27304#1 CVE-2007-5909 PoC1.rtf
SA27304#2 CVE-2007-6008 PoC1.eml
SA27304#2 CVE-2007-6008 PoC2.eml
SA27361#4 CVE-2007-2263 PoC.swf
SA27849 CVE-2007-6593 GameOver1.123
SA27849 CVE-2007-6593 GameOver2.123
SA27849 CVE-2007-6593 GameOver3.123
SA28034 CVE-2007-0064 PoC1.asf
SA28034 CVE-2007-0064 PoC2.asf
SA28034 CVE-2007-0064 PoC3.asf
SA28034 CVE-2007-0064 PoC4.asf
SA28083#2 CVE-2007-0071 PoC.swf
SA28092#1 CVE-2007-4706 PoC.mov
SA28209#10 CVE-2007-5399 PoCbcc.eml
SA28209#10 CVE-2007-5399 _PoC_cc.eml
SA28209#10 CVE-2007-5399 PoC_date.eml
SA28209#10 CVE-2007-5399 PoC_from.eml
SA28209#10 CVE-2007-5399 PoC_imp.eml
SA28209#10 CVE-2007-5399 PoC_prio.eml
SA28209#10 CVE-2007-5399 PoC_to.eml
SA28209#10 CVE-2007-5399 PoC_xmsmail.eml
SA28209#11 CVE-2007-5399 PoC.eml
SA28209#12 CVE-2007-5399 PoC.eml
SA28209#13 CVE-2007-5399 PoC.eml
SA28326 CVE-2008-0064 GameOver1.hdr
SA28326 CVE-2008-0064 GameOver2.hdr
SA28506#1 CVE-2008-0081 Exploit.xls
SA28506#1 CVE-2008-0081 PoC.xls
SA28506#2 CVE-2008-0111 PoC1.xls
SA28506#2 CVE-2008-0111 PoC2.xls
SA28506#2 CVE-2008-0111 PoC3.xls
SA28506#4 CVE-2008-0114 PoC.xls
SA28506#7 CVE-2008-0117 Exploit.xls
SA28506#7 CVE-2008-0117 GameOver.xls
SA28506#7 CVE-2008-0117 PoC.xls
SA28563 CVE-2008-0392 Exploit_CommandName.dsr
SA28563 CVE-2008-0392 GameOver_CommandName.dsr
SA28765 CVE-2008-0619 PoC.m3u
SA28765 CVE-2008-0619 PoC.pls
SA28802#1 CVE-2007-5659 GameOver.pdf
SA28802#1 CVE-2007-5659 PoC.pdf
SA28904#2 CVE-2008-0105 PoC1.wps
SA28904#2 CVE-2008-0105 PoC2.wps
SA28904#3 CVE-2007-0108 GameOver.wps
SA29293#1 CVE-2008-1581 PoC.pct
SA29321#2a CVE-2008-0118 PoC.ppt
SA29321#2b CVE-2008-0118 GameOver.ppt
SA29321#2b CVE-2008-0118 PoC.ppt
SA29620 CVE-2008-0069 GameOver.sld
SA29650#5 CVE-2008-1017 crgn_PoC.mov
SA29704#1 CVE-2008-1083 PoC.emf
SA29704#2 CVE-2008-1087 PoC.emf
SA29838 CVE-2008-1765 Exploit.bmp
SA29838 CVE-2008-1765 GameOver.bmp
SA29934 CVE-2008-1942 PoC_ExtGState.pdf
SA29934 CVE-2008-1942 PoC_Height.pdf
SA29934 CVE-2008-1942 PoC_MediaBox.pdf
SA29934 CVE-2008-1942 PoC_Width.pdf
SA29941 CVE-2008-1104 Exploit.pdf
SA29941 CVE-2008-1104 PoC.pdf
SA29972 CVE-2008-2021 PoC.ZOO
SA30143#1 CVE-2008-1091 PoC.rtf
SA30953 CVE-2008-1435 PoC.search-ms
SA30975 CVE-2008-2244 PoC1.doc
SA30975 CVE-2008-2244 PoC2.doc
SA31336#2 CVE-2008-3018 PoC.pict
SA31336#4 CVE-2008-3020 PoC.bmp
SA31336#5 CVE-2008-3460 PoC1.wpg
SA31336#5 CVE-2008-3460 PoC2.wpg
SA31336#5 CVE-2008-3460 PoC3.wpg
SA31385 CVE-2008-2245 PoC.emf
SA31441 CVE-2008-4434 PoC.torrent
SA31454#X CVE-NOMATCH PoC.xls
SA31454#2 CVE-2008-3005 Exploit.xls
SA31454#2 CVE-2008-3005 PoC.xls
SA31675#3 CVE-2008-3013 PoC.gif
SA31675#4 CVE-2008-3014 PoC.wmf
SA31675#X CVE-NOMATCH PoC.emf
SA31675#X CVE-NOMATCH PoC.wmf
SA31675#5 CVE-2008-3015 PoC.ppt
SA31821#6 CVE-2008-3626 PoC1.mp4
SA31821#6 CVE-2008-3626 PoC2.mp4