Lately I’ve been hearing a lot of the old “but we’ve never had a security problem” myth as an excuse for inaction (hmm… maybe I should add it to the list).

Usually this statement is made in relation to intrusions, so let’s look at that:

By now almost everyone has heard that attackers are not kids out for kicks and street cred anymore. Compromising systems is big business, whether it’s just to rent out as nodes on a botnet, a crypto ransom scheme, stealing bank and card data or industrial espionage. Whatever the goal, it’s worth serious money to the attackers to not be detected.

So we see targeted malware that scoot though pattern-based IDS and antivirus, rootkits that subvert the operating system, and smuggling traffic back to the attacker’s command and control center through HTTP, DNS, or ICMP.

With a skilled attacker, you won’t know they are there:

• In the massive TJX breach attackers were smuggling card numbers and personal data out of their network for 18 months before the breach was discovered.
• How many of the 100,000 victims of the Zeus Botnet knew they had been compromised before the botnet owner pulled the plug?

When I worked at a CIRT, the reports we got of web sites hosting malware most often turned out to be small sites like business “brochure” sites and personal blogs. Attackers exploited a vulnerability (usually in an out-of-date version of a CMS like WordPress or Joomla), uploaded malware, then sent out phishing email pointing to the URL.

Unlike in the old days, the sites were never defaced and there were no other signs that the site had been compromised, except perhaps lots of traffic to an unusual file showing up in the site’s web stats. In each case it took considerable effort to convince the site owner they had been compromised and were indeed serving malware.

Beyond intrusions, every organization has information security issues of some kind: vulnerability management, backups and continuity, user awareness, etc.

With a little help and honest reflection on past events, most organizations quickly realize that “we’ve never had a problem” is not really the case.

No related posts.

Among other things, he has an interesting idea that an attacker might take advantage of easier cloning and failover possible with virtual machines. I can see how the near instantaneous failover provided by features like VMWare’s "VMotion" technology could make attacks less noticable: rebooting or crashing a physical server tends to be more visible than it is with a virtual infrastructure that can shift processing to another VM in a few milliseconds. This potentially could allow an attacker to install a modified kernel or perform a few trial-and-error exploits on VMs without being noticed.

Escaping a virtual machine is not necessarily "an easy target" as he says, especially for ESX Server, but the consequences can be greater in a network with many virtual machines.

Physical compartmentalization for each security domain is a sound approach: one or more physical servers for all your VMs in a DMZ, other physical servers hosting VMs with application servers and databases on the trusted network, and a physical firewall between them. Unfortunately I still come across designs that use just one honking big box to host both inside and Internet-facing VMs (plus development VMs, QA VMs, and more). Beyond the obvious single point of failure of the physical server, the bigger danger is in the assumption (without hard evidence) that virtual machines provide the same level of isolation as actual hardware.

Update: Recently the SANS Handler’s Diary (recommended daily reading) discussed how more malware is detecting VMware, and linked to a presentation on how to avoid detection.

Related posts:

• Can virtualization be trusted for security?
• XenServer virtualization to go free today
• CIS releases virtual machine security guide

The last time they did this all awards honored stupidity in physical and administrative security. Perhaps this year IT security will be represented… there are so many stellar examples to choose from.

On second thought, the majority of infosec stupidity is systemic, so it may not be possible to find a specific organization to award.

There is also the Stupid Security web site, keeping tabs on the latest ineffective, burdensome and ill conceived security from around the world. Fun reading.

Related posts:

• Pwnie Award nominees are out
• Pwnie Award nominations close July 15

The idea is that if the virtual machine gets infected with malware or infiltrated by an attacker, it’s not possible to escape to the host operating system. Getting rid of the malware and undoing damage by an attacker is as easy as reloading the VM from a known good image.

This has become a lot more widespread now that VMWare has released no-cost versions of their desktop and server products. Even Microsoft has started giving away their Virtual Server product. The problem is virtual machine products are not designed for security. They are designed for resource partitioning and running multiple OSs on one physical computer.

People simply assume that since each VM looks like separate physically hardware, it provides the same level of isolation as physical hardware. But appearances are deceiving… that’s simply not what most of these products are designed for. An attacker can easily detect they are inside a virtual machine, and with some effort could escape into the host OS or into other VMs on the same physical server.

We dealt with one organization who had constructed an entire Internet server farm on one physical server using VMWare’s ESX server product. The one physical box had been segregated into multiple guest VMs running Linux and Windows: a firewall, a database server, a web server, an e-mail server and a Java application server. The VMs were interconnected using VMWare’s virtual LAN capability.

It was a cute idea and one that would potentially save a lot of administration headaches and costs compared to setting up and maintaining four separate physical servers. The flaw was the assumption of perfect isolation of each VM from each other, and isolation from the host OS.

Using the emulated VLANs for network separation between the VMs was also a big assumption. Like virtual machines, virtual LANs are often abused for security, though they were never designed for that purpose. There are several published techniques for escaping “real” VLANs and settings the administrator of a hardware switch can use to reduce that vulnerability. Can the emulated VLANs in ESX server be hardened in the same way? Does the emulation code that creates the VLAN illusion add additional methods for escape?

Detecting the presence of most VMs is easy. VMWare Workstation and VMWare Player can be detected by looking at the name of the IDE device (”VMware Virtual IDE” or vendor string of the SCSI driver (”VMWare”). MS Virtual Server can be detected by looking for “Microsoft Corporation” in the manufacturer string of the motherboard. One generic approach to VM detection is discussed at http://invisiblethings.org/papers/redpill.html

Once an attacker determines they are inside a virtual machine, breaking out can be accomplished multiple ways. With VMWare Workstation, for example, you can do the following:

• via the “hidden” I/O port vmware uses to provide host communication such as clipboard and drive sharing (details at http://chitchat.at.infoseek.co.jp/vmware/backdoor.html)
• via known bugs, especially when Linux is the host OS (for example, http://securitytracker.com/alerts/2005/Dec/1015401.html and http://secunia.com/advisories/13871/)

The above are well known published exploits. You can be sure that many more unpublished methods to escape specific virtual machines exist and are circulating.

A knowledgable attacker can detect they are inside a virtual machine, and can probably escape it to get at the host operating system (or the linux-based “Console OS” in ESX Server). The fundamental problem is that even if the hardware virtualization of the environment was perfect, the host OS or management console must use some form of hole to communicate with the VM, such as the undocumented I/O port in Vmware workstation. Any flaw in that channel can potentially be exploited.

That means such VMs should not be used in roles such as for honeypots or separation of Internet-facing services from more sensitive ones.

Marketing of various virtualization products often imply they can be used for security, but a little digging reveals how much the vendors really stand by the claim. For example, Vmware’s ESX Server product has undergone Common Criterial evaluation, but only to the EAL 2 level of assurance for “protection against casual breach of TOE security by attackers possessing a low attack potential”.

So what about using VMs as a security container for web browsing on desktops? After all, the first pre-built virtual environment offered with VMWare Player is the “Browser Appliance”, a version of Ubuntu Linux intended specifically for that task. Surfing the web using a VM can help stop simple viruses and spyware from gaining a foothold on your “real” desktop. When the OS in the VM becomes compromised you can just reset it to a known good image. However there are a problems with that:

1. Users want to share files between their real host desktop and the virtual machine (for example, when a user downloads a PDF they probably want to put it in their normal home directory, not keep it on the VM’s virtual hard drive). VMware Workstation provides “Shared Folders” for that purpose. However, any file moved from the virtual machine can contaminate the host OS. Having a writable shared folder on the host also allows malware to write files directly to the host.
2. You probably won’t know when they virtual machine has been compromised until it’s too late. If you access your bank using a browser in the VM, how do you know the VM doesn’t have a keylogger installed? You would have to purposely reset the VM to the” known good” image at the beginning of each session.
3. Nothing prevents a compromised VM from becoming part of a botnet or being controlled remotely to attack other computers. Most VMs also allow the virtual NIC to be placed into promiscuous mode to sniff local network traffic.
4. There is evidence that malware are starting to check whether they are running inside a virtual machines and either refuse to run, or automatically attempt to escape to the host OS.

So, are virtual machines completely useless for security purposes? No. While not as trustworthy as physical hardware for containment, a VM does provide isolation against casual intrusion and the ability to quickly recover from compromise. Denial of service attacks are also mitigated since a crashed or overloaded VM can usually just be restarted without adversely affecting the host or other VMs.

As long as the limitations of the containment offered by virtualization product are recognized, they are valuable tools. Personally I would not use a VM for tasks such as a honey pot, analysing malware, or running a firewall.

On a server, I would make sure all VM on a physical server are used for tasks within the same security domain (e.g. one physical server hosting VMs for Internet services in a DMZ, another physical server hosting VMs for intranet applications, etc) and separate each physical server with a physical firewall.

If using VM on a workstation to surf web sites, a good practice is to reset the VM to the known good snapshot each time the VM is started, and to disable host interoperability features such as clipboard sharing and shared drives.

By the way, containment environments designed for security do exist. In 2001 VMware started working with the National Security Agency in the U.S to develop a virtual machine intended specifically for security containment. That project eventually resulted in “NetTop” (http://www.trustedcs.com/products/1products1_1_4.html)

NetTop is intended for classified environments where users typically have one physical workstation for each classified network they need to access. NetTop is designed to isolate each network within hardened virtual machines running on one physical workstation.

VMware’s ACE (http://www.vmware.com/products/ace/) is a specialized version of VMWare Workstation that is designed for security, though it’s designed to protect the VM from compromise from the host (i.e. to keep malware out of the VM rather than in, the opposite of what we’ve been discussing).

In the Unix world there are also “private execution environments” (OS Virtualization) such as FreeBSD Jail and Solaris Containers. For Linux, similar execution evironments can be added using either OpenVZ or Linux-Vserver. All these are designed for security containment but share the same underlying kernel OS. You cannot use them to run different OSs like in VMware, but can have reasonable assurance that attackers cannot escape.

Further reading on detecting virtual machines:http://www.phrack.org/fakes/p62/p62-0×07.txt

Update 1: Recently the SANS Handler’s Diary (recommended daily reading, by the way) discussed how more malware is detecting VMWare, and linked to a presentation on methods of avoiding detection.

Update 2: A researcher at Symantec has released a great little paper Attacks on Virtual Machine Emulators

Related posts:

• Security of virtualization
• XenServer virtualization to go free today
• CIS releases Vmware ESX security guide

“…it’s behind a firewall.”

“…we have an IDS.”

“…we use a VPN.”

“…we finally got PKI to work.”

“…we installed a network intrusion prevention box.”

“…we installed host intrusion prevention software.”

“…it has Common Criteria certification.”

“…the product is from [insert big company name]”

“…the person who built it has a CISSP / CCSP / CISA certification!”

“…it’s encrypted with SSL / 3DES / AES / PGP / other crypto method.”

“…the source code / algorithms / network layout / passwords are kept secret.”

“…it uses two-factor authentication: a username and a password.”

“…it uses two-factor authentication: a smart card / SecureID / some other scheme”

“…I couldn’t hack into it, so no one else can.”

“…our tiger team couldn’t hack into it.”

“…no vulnerabilities were found by Nessus / Retina / Cybercop / some other tool”

Does any of this sound familiar? These are all claims we’ve heard over the years in dealing with IT security.

Often the first obstacle of IT security is overcoming what security guru Bruce Schneier calls “security dust”… the myth that security is a matter of buying the right appliance, using the right crypto, or hiring someone with the right letters after their name: Sprinkle enough “security” around and everyone can get back to business.

The thing is, security is part of the business. It’s an ongoing operational activity like accounting. No one would claim their organization’s finances have been “accounted” then eliminate the accounting staff. No one would claim that just buying an accounting package recommended by a consultant solves the “accounting problem” forever.

Yet this is what we see with security: organizations buying products recommended by a salesman or “expert” then never thinking about it again… until they get hacked or suffer some other loss then the process starts again.

Like accounting, security is a process, not a product. This may be shocking news, but there’s no product you can buy and no person you can hire that will make your organization’s “security problem” go away. IT security an ongoing operational process. It needs skilled individuals doing daily tasks, regular evaluation of processes being used, plus audits and other checks to ensure that no errors have crept in.

Security isn’t a product, but it’s not magic either: it is possible to define policies, create procedures, gain knowledgeable people and find reliable tools to make the ever evolving process of IT security manageable.

The first step in getting there is to stop believing in security dust.

Derrick Webber, Director
Advosys Consulting Inc.

No related posts.