Telematique, water and fire.

Critically Under-damped Oscillations

Chris Hoff has a great, common-sense post on security and where in the data center it will eventually end up residing.  (If you don't want me to give away the plot, go directly to the post.  Don't read the snippet I've enclosed.)

Along with the "dampened oscillation" graphic that he alludes to (but doesn't actually draw), I'd like to add my two-cents about where security resides when dealing with server virtualization, and the network.  Server virtualization, and particularly hot migration (likeVMware's VMotion), has definitely changed the relative workload and tsuris (a technical term of art) experienced within the data center by the persons responsible for, respectively, server administration, storage administration, and network administration. 

In the days before widespread adoption of server virtualization, making a new application "production ready" was a PITA (another term of art) for the server admin, who had to specify servers, install the apps, move the appropriate data for use by the apps, test, stage, re-test, etc. 

The storage admin had a modest workload, requiring attention to allocation of storage space, setting quotas, setting policies, ... but once done at the planning stage, required modest tweaking thereafter. 

The network admin had it easiest (IMHO).  Over the course of the weeks (if not months) it took to arrange for a new application to be put into production, the network admin might have to allocate ports, set VLANs, set policies, and be present when doing the lash up with the network equipment.

Fast forward to the day when a new application goes through development, test, staging and cut-over into production ... ALL using server virtualization.  Besides the fact that the time horizon for the production deployment has likely been compressed from weeks to days, the relative workloads as this cut-over approaches is radically different from the one described above. 

• The server admin has a relative cakewalk: extend VME cluster, copy the image, or use a hot migration to herd the app into the new spot. 
• The storage admin has pretty much the same level of work in allocating space, setting quotas, etc.,  and will soon be using SAN "hot migration" (e.g. VMware's Storage VMotion).
  
• The network admin, however, just got a rude awakening.  If he's got SLAs to which his organization must commit, the network admin must allocate ports, set VLANs and VLAN policies, set up NIC teaming in both the virtual switches and physical server access switch, and set up trunking on the vSwitch and pSwitch.   Oh, and by the way... it has to be "right" for every physical server in the data center to which a virtualized application MIGHT migrate in the future.
  

Holy smoke, Chris!  It's not a single, oscillating signal.  It's (at least) three of 'em.  (... and if I were a better graphics hack, I'd drop in a jpg right about now.)

Rational Survivability: Security Will Not End Up In the Network...

... Here's the reality we actually already know and should not come to you as a surprise if you've been reading my blog: we will always need a blended investment in technology, people and process in order to manage our risk effectively.  From a technology perspective, some of this will take the form of controls embedded in the information itself, some will come from the OS and applications and some will come from the network.

Anyone who tells you differently has something to sell you or simply needs a towel for the back of his or her ears...

MSFT to Craft it's own VMsafe?

Virtualization.info has a short but interesting post, which refers to a parenthetical comment from Chris Hoff which might imply that MSFT is considering / working on a VMsafe-like framework.

Is Microsoft working on a VMsafe-like framework? | virtualization.info

...
So far Microsoft didn't took an official position about the topic but virtualization.info had the opportunity to speak with several representatives who clearly stated how carefully the company is evaluating the security implications of a VMsafe-like approach.

Nonetheless Microsoft may be working to build the internal know-how needed to achieve the task.

Just two months ago in fact Microsoft acquired a small security firm focused on rootkit detection called Komoku.

As Christopher Hoff, Chief Security Architect at Unisys, recently discovered, Komoku did some research in the past, presenting a solution for Xen where virtual machines can do self-diagnosis and self-healing as well as learning to protect against subsequent attacks. ...

Virtualization Risk and the Fishtank

I missed this article in CIO (Top Ten Virtualization Risks Hiding in Your Company) when it was published about a month ago. After running through a believable list, I looked at one of the comments and couldn't help but smile:

... I think the comments above missed risk #11, the physical world outside any hypervisor host.

All hypervisors still depend on underlying physical machines being correctly connected to network and storage -- in multiple paths, to allow access for all VMs correctly.

In other words, all the above article comments apply to moving around the virtual machine "fish" inside the hypervisor OS "fishtank" -- but who moves and manages the associated fishtanks (with associated network & storage I/O plumbing etc)?

Kevin Epstein (of Scalent) then goes on to set out a number of additional problems, and although he makes sure to inform the reader that (in his opinion) Scalent solves these problems, I have to agree with the problem identification (if not completely with his solution):

1. Network connectivity matters
All hypervisor hosts in a group or "cluster" who are going to share virtual machines -must- share a LAN subnet.

2. Storage access matters
All hypervisor hosts in a group or "cluster" who are going to share virtual machines -must- share storage access.

3. Hardware failover must be anticipated
VMs will fail to another clustered hypervisor... if, and only if, one exists and has cycles! (See point 1 and 2)

4. Movement between Physical and Virtual (and back, repeatedly) is a necessity in real data centers, and is -not- usually seamless ...

5. Non-x86 Hardware
Not all hardware is x86! Sun now has LDOMs, AIX too, how to manage workloads between those and the rest of the virtual universe?

VMsafe and the Network Stack

Gabe has collected information from various sources about VMware's VMsafe API. The post makes clear VMware's approach to "opening up" access to the hypervisor for third-party security products provided by carefully vetted partners. In a comment to the post, the reader is pointed to VMCI as a the means by which VMsafe gains access to memory and other hypervisor resources. Gabe then makes this rather telling response:

I’m missing the network stack in this, because I read that the network virtual switches can also be monitored with VMsafe. I’m wondering if for example McAfee can build an appliance that does some kind of virus scanning, checkpoint builds a network inspection appliance, etc. You would then have multiple security appliances per host. Not sure if this is what is desired.

Is this the case? Does VMsafe offer up access of this kind to the vswitch? Does that mean that the as-yet-unannounced, but rumored third-party virtual distributed switch would offer up the same access through VMsafe?

Xensploit, VMotion and VM Migration

Over the course of the past week, I've seen a number of references to recently published proof-of-concept exploit of virtual machine hot migration. It's got a number of our more exciteable colleagues wrapped around the axle. Two thoughtful and well-considered posts have emerged in response, both worth reading. Scott Lowe and Warren Wu take on the issue directly, provide good advice and, in so doing, make an argument for the class of virtualization management systems in development at Replicate. It's nice be reassured that we're addressing a real source of data center pain.

VMotion and VLAN Security - blog.scottlowe.org - The weblog of an IT pro specializing in virtualization, storage, and servers

... Xensploit, as it’s called, is the recently demonstrated exploit that allows virtual machines (VMs) that are “in flight” during a live migration (XenMotion in Citrix XenServer, VMotion in VMware ESX Server) to be manipulated. If you haven’t yet read the PDF that describes Xensploit, I highly encourage that you take a look at it. It’s very enlightening as to exactly what can be done to an in-flight VM.

Naturally, the best way to protect against this particular problem is to guard the integrity of the live migration network. For simplicity’s sake, I’ll refer to this as the VMotion network from this point on, but keep in mind that it is equally applicable to any network connections on any virtualization platform that uses live migration.

The most surefire way to protect the VMotion network is to place it on its own dedicated, physically separate network, using separate physical NICs plugged into separate physical switches that do not possess any connections to production networks. This will ensure that unauthorized access to the VMotion network is prevented, but comes with disadvantages as well: this configuration requires more physical NICs and more physical switches than other configurations. ...

Keeping Your VMotion Traffic Secure

... Although impressive, this work by no means represents any new security risk in the datacenter. It should be emphasized this proof-of-concept does NOT “take over the hypervisor” nor present unencrypted traffic as a vulnerability needing patching, as some news reports incorrectly assert. Rather, it a reminder of how an already-compromised network, if left unchecked, could be used to stage additional severe attacks in any environment, virtual or physical.

On an insecure network, man-in-the-middle attacks can target both virtual and physical machines. The techniques published are novel in that they go after the contents of migrating VM memory to target credentials and data, rather than going after similar information flowing across internal network transactions. Putting aside the question of whether it’s even worthwhile to target memory instead of network traffic directly, the sensitivity of VM memory was never the question. ...

Extending Virtualization Security Issues to Configuration Management

Thanks to David Marshall at VMblog.com for drawing attention to this post by David Frith on SecurityPark.net.  The post is worth reading in its entirety, but I'm going to spoil the ending for you by giving you a few of its final paragraphs.

A couple of points worth emphasizing here which transcend the requirements of security, and address other elements of VME resilience and availability:

• VMs that require like levels of security, reliability, availability and resilience need to be grouped together, managed as an assemblage and this has to remain uniform, especially when VM migration (e.g. VMotion) is utilized.
• The security settings and configurations of the underlying virtual and physical networks need to be understandable by the administrators and network managers, which implies that unnecessary complexity be hidden and the burden assumed by automation.

Virtualisation: Why existing security measures are no longer enough - Security Park news

...

With potential attacks first compromising one VM and then spreading to others, each needs to be protected with secure policies configured and adapted as needed. Here existing vendor tools can be used in the partitioning, isolating and segmenting of each VM with resource management controls to allocate, schedule, monitor and cap resources as required. Such tools can ensure that the VMs that require like levels of security are grouped together and that controls are in place to stop any unauthorised replication.

...

Management tools are required to provision VMs as necessary together with their associated security settings, such tools also need to map interdependencies and data flows ensuring that with all the complexity administrators do not lose an understanding of their environment.  With VM’s being deployed and re-deployed, patching tools are also required.

...


The complexity and dynamic nature of virtualised environments means that new threats and vulnerabilities have appeared and will increasingly manifest themselves. Because traditional security practices only go so far new architectural models, design practices and security tools are required. The existing tools however are generally immature and not yet certified, while such vendors and their tools need to evolve, the market also needs to educate itself, raising awareness of potential issues, new vulnerabilities, evolving threats and where necessary pressuring the vendors to enhance their security offerings.

Powered by ScribeFire.