Telematique, water and fire.

Critically Under-damped Oscillations

Chris Hoff has a great, common-sense post on security and where in the data center it will eventually end up residing.  (If you don't want me to give away the plot, go directly to the post.  Don't read the snippet I've enclosed.)

Along with the "dampened oscillation" graphic that he alludes to (but doesn't actually draw), I'd like to add my two-cents about where security resides when dealing with server virtualization, and the network.  Server virtualization, and particularly hot migration (likeVMware's VMotion), has definitely changed the relative workload and tsuris (a technical term of art) experienced within the data center by the persons responsible for, respectively, server administration, storage administration, and network administration. 

In the days before widespread adoption of server virtualization, making a new application "production ready" was a PITA (another term of art) for the server admin, who had to specify servers, install the apps, move the appropriate data for use by the apps, test, stage, re-test, etc. 

The storage admin had a modest workload, requiring attention to allocation of storage space, setting quotas, setting policies, ... but once done at the planning stage, required modest tweaking thereafter. 

The network admin had it easiest (IMHO).  Over the course of the weeks (if not months) it took to arrange for a new application to be put into production, the network admin might have to allocate ports, set VLANs, set policies, and be present when doing the lash up with the network equipment.

Fast forward to the day when a new application goes through development, test, staging and cut-over into production ... ALL using server virtualization.  Besides the fact that the time horizon for the production deployment has likely been compressed from weeks to days, the relative workloads as this cut-over approaches is radically different from the one described above. 

• The server admin has a relative cakewalk: extend VME cluster, copy the image, or use a hot migration to herd the app into the new spot. 
• The storage admin has pretty much the same level of work in allocating space, setting quotas, etc.,  and will soon be using SAN "hot migration" (e.g. VMware's Storage VMotion).
  
• The network admin, however, just got a rude awakening.  If he's got SLAs to which his organization must commit, the network admin must allocate ports, set VLANs and VLAN policies, set up NIC teaming in both the virtual switches and physical server access switch, and set up trunking on the vSwitch and pSwitch.   Oh, and by the way... it has to be "right" for every physical server in the data center to which a virtualized application MIGHT migrate in the future.
  

Holy smoke, Chris!  It's not a single, oscillating signal.  It's (at least) three of 'em.  (... and if I were a better graphics hack, I'd drop in a jpg right about now.)

Rational Survivability: Security Will Not End Up In the Network...

... Here's the reality we actually already know and should not come to you as a surprise if you've been reading my blog: we will always need a blended investment in technology, people and process in order to manage our risk effectively.  From a technology perspective, some of this will take the form of controls embedded in the information itself, some will come from the OS and applications and some will come from the network.

Anyone who tells you differently has something to sell you or simply needs a towel for the back of his or her ears...

Is Co-Administration the Answer?

Rick Vanover, blogging at TechRepublic's Network Administrator site, suggests a solution to the problem of overlapping between the span of administrative control normally provided to the network admin, and that required of a VM server admin.  It's a solution that might appeal to a network administrator, but I'm dubious.  I'd very much like to hear from the network crowd as to how this might work in practice.

Here's my take.  In our investigations at Replicate, we've noted that VM admins are often unwilling to dig into the network management systems. (There are a number of reasons, which we won't go into here.)  So, how would a network admin view this solution?  These seem to be the implications of Vanover's approach:

• the network admin must be cross-trained in the use of the VME's management system (e.g. VMware's Virtual Center or Citrix' XenCenter)
• the network admin is required, at installation setup, to establish consistent configurations on the virtual switches and (in separate management system) the physical switches.
• The configuration settings on the vSwitches are supposed to remain inviolate and untouched by the VM admin in order to prevent configuration problems.
• the network admin thereafter is relegated to a passive, read-only audience for the VM management system reports, unless ...
• when there is a physical network issue (a problem or need to reconfigure), the network admin is reinstated with the necessary privileges to make those changes.

This sounds workable, at most, for a short period of time, an installation that changes almost never, or a very small installation.

Co-Administration is the new virtualization endpoint | Network Administrator | TechRepublic.com

Almost every organization has embraced some amount of virtualization, and the network has surely been a hot topic as a virtual environment scales upward. Most virtual host systems (VMware ESX, Citrix XenServer, etc.) offer host-based switches that implement 802.1Q tagging on the ports to the virtual machines. This poses a unique question: Who administers the virtual switch when the network and server administration are handled by different groups?

...
One creative way to solve this dilemma is with a co-administration approach. This would give the network engineers access to the virtual environment for configuration during a change and read-only access for ongoing checks of configuration and for assurance that a virtual machine is not breaking any network rules, such as having a virtual network adapter on two interfaces where one is a secured or external network. In most situations, the network administrator has no visibility into the configuration of the network within virtualization installations, and the co-administered approach can change that.   ...

VMware: All (ok, some) your access network are belong to us.

Colin McNamara and Scott Lowe, two bloggers from Eplus Technology, have been doing an excellent job of pointing out the network gap created by VMware. McNamara, in a recent post, sets out the problem very well:

... Your access layer is no longer a top of rack Cisco switch, or end of row aggregation chassis. It is now a virtual bridge that exists logically within your VMware ESX server.

This causes an interesting question to come up in many customers - Who is responsible for the configuration and maintenance of this Vswitch?

...

We no longer have this well defined edge at the access layer. The access layer now exists virtually inside a server. More specifically, it is a logical devices running in a Linux server. This presents a challenge because it requires cross over knowledge. Whoever is responsible for this integration has to be fluent in Linux systems administration , and also fluent in network design and operations. Frankly this is a rare skill set to come across, as it requires and engineer who has attained high proficiency in both systems and network engineering.

I see this fuzzy line of demarcation often as a failing point for many VMware integrations. Many times I see network operations teams not involved in ESX cluster design because its a “server” , and systems operations teams generally don’t have the networking skills necessary to design and implement an fully functional system.. The solution to this problem is education and collaboration. ...

I'm not convinced that the solution to the problem lies only in "education and collaboration". It's the central, defining issue around which Replicate's products are being created.

VMsafe and the Network Stack

Gabe has collected information from various sources about VMware's VMsafe API. The post makes clear VMware's approach to "opening up" access to the hypervisor for third-party security products provided by carefully vetted partners. In a comment to the post, the reader is pointed to VMCI as a the means by which VMsafe gains access to memory and other hypervisor resources. Gabe then makes this rather telling response:

I’m missing the network stack in this, because I read that the network virtual switches can also be monitored with VMsafe. I’m wondering if for example McAfee can build an appliance that does some kind of virus scanning, checkpoint builds a network inspection appliance, etc. You would then have multiple security appliances per host. Not sure if this is what is desired.

Is this the case? Does VMsafe offer up access of this kind to the vswitch? Does that mean that the as-yet-unannounced, but rumored third-party virtual distributed switch would offer up the same access through VMsafe?