Telematique, water and fire.

MyCMDB - the CMDB as a Wikipedia Plug-in to FaceBook

At the risk of piling on, I'll join the refrain regarding the recently announced MyCMDB from Managed Objects. As described, it makes no sense to me. I can't for the life of me figure out how one uses social networking and the "principles of Web 2.0" to solve the CMDB data accuracy and completeness problems.

myCMDB - Managed Objects

... Managed Objects myCMDB™ solves CMDB data accuracy and accessibility issues incumbent with today's CMDB implementations. By integrating principles of Web 2.0 and social networking into a new web-based application, myCMDB delivers role-based “communities” where users can more easily and effectively view and interact with CMDB data – and other CMDB users as well. ...

Critically Under-damped Oscillations

Chris Hoff has a great, common-sense post on security and where in the data center it will eventually end up residing.  (If you don't want me to give away the plot, go directly to the post.  Don't read the snippet I've enclosed.)

Along with the "dampened oscillation" graphic that he alludes to (but doesn't actually draw), I'd like to add my two-cents about where security resides when dealing with server virtualization, and the network.  Server virtualization, and particularly hot migration (likeVMware's VMotion), has definitely changed the relative workload and tsuris (a technical term of art) experienced within the data center by the persons responsible for, respectively, server administration, storage administration, and network administration. 

In the days before widespread adoption of server virtualization, making a new application "production ready" was a PITA (another term of art) for the server admin, who had to specify servers, install the apps, move the appropriate data for use by the apps, test, stage, re-test, etc. 

The storage admin had a modest workload, requiring attention to allocation of storage space, setting quotas, setting policies, ... but once done at the planning stage, required modest tweaking thereafter. 

The network admin had it easiest (IMHO).  Over the course of the weeks (if not months) it took to arrange for a new application to be put into production, the network admin might have to allocate ports, set VLANs, set policies, and be present when doing the lash up with the network equipment.

Fast forward to the day when a new application goes through development, test, staging and cut-over into production ... ALL using server virtualization.  Besides the fact that the time horizon for the production deployment has likely been compressed from weeks to days, the relative workloads as this cut-over approaches is radically different from the one described above. 

• The server admin has a relative cakewalk: extend VME cluster, copy the image, or use a hot migration to herd the app into the new spot. 
• The storage admin has pretty much the same level of work in allocating space, setting quotas, etc.,  and will soon be using SAN "hot migration" (e.g. VMware's Storage VMotion).
  
• The network admin, however, just got a rude awakening.  If he's got SLAs to which his organization must commit, the network admin must allocate ports, set VLANs and VLAN policies, set up NIC teaming in both the virtual switches and physical server access switch, and set up trunking on the vSwitch and pSwitch.   Oh, and by the way... it has to be "right" for every physical server in the data center to which a virtualized application MIGHT migrate in the future.
  

Holy smoke, Chris!  It's not a single, oscillating signal.  It's (at least) three of 'em.  (... and if I were a better graphics hack, I'd drop in a jpg right about now.)

Rational Survivability: Security Will Not End Up In the Network...

... Here's the reality we actually already know and should not come to you as a surprise if you've been reading my blog: we will always need a blended investment in technology, people and process in order to manage our risk effectively.  From a technology perspective, some of this will take the form of controls embedded in the information itself, some will come from the OS and applications and some will come from the network.

Anyone who tells you differently has something to sell you or simply needs a towel for the back of his or her ears...

Is Co-Administration the Answer?

Rick Vanover, blogging at TechRepublic's Network Administrator site, suggests a solution to the problem of overlapping between the span of administrative control normally provided to the network admin, and that required of a VM server admin.  It's a solution that might appeal to a network administrator, but I'm dubious.  I'd very much like to hear from the network crowd as to how this might work in practice.

Here's my take.  In our investigations at Replicate, we've noted that VM admins are often unwilling to dig into the network management systems. (There are a number of reasons, which we won't go into here.)  So, how would a network admin view this solution?  These seem to be the implications of Vanover's approach:

• the network admin must be cross-trained in the use of the VME's management system (e.g. VMware's Virtual Center or Citrix' XenCenter)
• the network admin is required, at installation setup, to establish consistent configurations on the virtual switches and (in separate management system) the physical switches.
• The configuration settings on the vSwitches are supposed to remain inviolate and untouched by the VM admin in order to prevent configuration problems.
• the network admin thereafter is relegated to a passive, read-only audience for the VM management system reports, unless ...
• when there is a physical network issue (a problem or need to reconfigure), the network admin is reinstated with the necessary privileges to make those changes.

This sounds workable, at most, for a short period of time, an installation that changes almost never, or a very small installation.

Co-Administration is the new virtualization endpoint | Network Administrator | TechRepublic.com

Almost every organization has embraced some amount of virtualization, and the network has surely been a hot topic as a virtual environment scales upward. Most virtual host systems (VMware ESX, Citrix XenServer, etc.) offer host-based switches that implement 802.1Q tagging on the ports to the virtual machines. This poses a unique question: Who administers the virtual switch when the network and server administration are handled by different groups?

...
One creative way to solve this dilemma is with a co-administration approach. This would give the network engineers access to the virtual environment for configuration during a change and read-only access for ongoing checks of configuration and for assurance that a virtual machine is not breaking any network rules, such as having a virtual network adapter on two interfaces where one is a secured or external network. In most situations, the network administrator has no visibility into the configuration of the network within virtualization installations, and the co-administered approach can change that.   ...

Hyper-V Needs a Pit Crew

One aspect of the staggered release of Hyper-V and VMM 2008 is the focus on how to manage virtualization ... any virtualization environment, but specifically Microsoft's. What this article also points out is that VMM 2008 starts out by demonstrating its capabilities to manage multi-vendor environments.

The article ends with Mitchell Ashley expressing doubt about the relative price to the customer for using VMM 2008 versus VMware's VirtualCenter. In my view, he doesn't come right out and say that the comparison is bogus, but he should. If a datacenter's use of VMware depends on ESX functionality like VMotion, DRS (an automated VMotion) or HA (high availability), the customer has to buy VirtualCenter. That makes the price of VMM 2008 an additional cost to the datacenter ... not the cost of substituting it for VirtualCenter.


Hyper-V May Cause Hyper Tension | NetworkWorld.com Community

Microsoft needs a successful Hyper-V launch out in the marketplace to begin to stave off VMware's dominance. But Hyper-V can't do it alone, that's only part of the picture. Just having Hyper-V is like having a NASCAR race car but no pit crew. Hyper-V's got to have the management tools to be successfully utilized by IT. Most agree; the hypervisor will be a commodity, it's the management capabilities enabling customers to manage virtualized environments that will win the day. ...

What's needed most to deploy software on Hyper-V at any scale is Microsoft's Virtual Machine Manager 2008 (VMM). VMM just went into beta and is expected to ship 30 to 60 days after Hyper-V's release, making the product launch likely sometime in early fall.

VMM's making some interesting claims about managing virtualization. Not only is VMM 2008 managing Hyper-V, Virtual Server, but also VMware's ESX product. Microsoft isn't claiming VMM is a full replacement for VMware's Virtual Center product, but is claiming a significant portion of what Virtual Center does can be done within VMM. ... The VMM team's blog post is claiming Microsoft does all this at one third the cost of VMware, but that's an incremental cost if you are already a VMware Virtual Center customer. And we'll have to see if Hyper-V and VMM are competitive against VMware in new accounts where VMware doesn't already have a presence.

Virtualization Risk and the Fishtank

I missed this article in CIO (Top Ten Virtualization Risks Hiding in Your Company) when it was published about a month ago. After running through a believable list, I looked at one of the comments and couldn't help but smile:

... I think the comments above missed risk #11, the physical world outside any hypervisor host.

All hypervisors still depend on underlying physical machines being correctly connected to network and storage -- in multiple paths, to allow access for all VMs correctly.

In other words, all the above article comments apply to moving around the virtual machine "fish" inside the hypervisor OS "fishtank" -- but who moves and manages the associated fishtanks (with associated network & storage I/O plumbing etc)?

Kevin Epstein (of Scalent) then goes on to set out a number of additional problems, and although he makes sure to inform the reader that (in his opinion) Scalent solves these problems, I have to agree with the problem identification (if not completely with his solution):

1. Network connectivity matters
All hypervisor hosts in a group or "cluster" who are going to share virtual machines -must- share a LAN subnet.

2. Storage access matters
All hypervisor hosts in a group or "cluster" who are going to share virtual machines -must- share storage access.

3. Hardware failover must be anticipated
VMs will fail to another clustered hypervisor... if, and only if, one exists and has cycles! (See point 1 and 2)

4. Movement between Physical and Virtual (and back, repeatedly) is a necessity in real data centers, and is -not- usually seamless ...

5. Non-x86 Hardware
Not all hardware is x86! Sun now has LDOMs, AIX too, how to manage workloads between those and the rest of the virtual universe?

Xensploit, VMotion and VM Migration

Over the course of the past week, I've seen a number of references to recently published proof-of-concept exploit of virtual machine hot migration. It's got a number of our more exciteable colleagues wrapped around the axle. Two thoughtful and well-considered posts have emerged in response, both worth reading. Scott Lowe and Warren Wu take on the issue directly, provide good advice and, in so doing, make an argument for the class of virtualization management systems in development at Replicate. It's nice be reassured that we're addressing a real source of data center pain.

VMotion and VLAN Security - blog.scottlowe.org - The weblog of an IT pro specializing in virtualization, storage, and servers

... Xensploit, as it’s called, is the recently demonstrated exploit that allows virtual machines (VMs) that are “in flight” during a live migration (XenMotion in Citrix XenServer, VMotion in VMware ESX Server) to be manipulated. If you haven’t yet read the PDF that describes Xensploit, I highly encourage that you take a look at it. It’s very enlightening as to exactly what can be done to an in-flight VM.

Naturally, the best way to protect against this particular problem is to guard the integrity of the live migration network. For simplicity’s sake, I’ll refer to this as the VMotion network from this point on, but keep in mind that it is equally applicable to any network connections on any virtualization platform that uses live migration.

The most surefire way to protect the VMotion network is to place it on its own dedicated, physically separate network, using separate physical NICs plugged into separate physical switches that do not possess any connections to production networks. This will ensure that unauthorized access to the VMotion network is prevented, but comes with disadvantages as well: this configuration requires more physical NICs and more physical switches than other configurations. ...

Keeping Your VMotion Traffic Secure

... Although impressive, this work by no means represents any new security risk in the datacenter. It should be emphasized this proof-of-concept does NOT “take over the hypervisor” nor present unencrypted traffic as a vulnerability needing patching, as some news reports incorrectly assert. Rather, it a reminder of how an already-compromised network, if left unchecked, could be used to stage additional severe attacks in any environment, virtual or physical.

On an insecure network, man-in-the-middle attacks can target both virtual and physical machines. The techniques published are novel in that they go after the contents of migrating VM memory to target credentials and data, rather than going after similar information flowing across internal network transactions. Putting aside the question of whether it’s even worthwhile to target memory instead of network traffic directly, the sensitivity of VM memory was never the question. ...