Telematique, water and fire.

MyCMDB - the CMDB as a Wikipedia Plug-in to FaceBook

At the risk of piling on, I'll join the refrain regarding the recently announced MyCMDB from Managed Objects. As described, it makes no sense to me. I can't for the life of me figure out how one uses social networking and the "principles of Web 2.0" to solve the CMDB data accuracy and completeness problems.

myCMDB - Managed Objects

... Managed Objects myCMDB™ solves CMDB data accuracy and accessibility issues incumbent with today's CMDB implementations. By integrating principles of Web 2.0 and social networking into a new web-based application, myCMDB delivers role-based “communities” where users can more easily and effectively view and interact with CMDB data – and other CMDB users as well. ...

Why Cloudware and why now?

In September of last year, as I was preparing (mentally and emotionally) to get Replicate started on its current path, I considered issues of portability and interoperability in the virtualized datacenter. I posted a few comments about OVF but one in particular drew the attention of Bert Armijo of 3tera.

At that time, Bert indicated that he thought it "... too early for a standard,...", with a (perfectly arguable) claim that standards are often "... a trade-off to gain interoperability in exchange for stifling innovation." He went on to say that "(w)e haven't adequately explored the possibilities in utility computing." He then provided a critique of OVF. (Whether I agree with that critique or not is immaterial to this post, and the subject for another time.)

At the end of June, 3tera announced their Cloudware vision for a standards-based interoperable utility infrastructure. Since the arrival of Cloudware, there have been a number of venues at which "cloud computing" and interoperability has been on the minds of the cognoscenti... Structure08 and Velocity being the most heavily covered. In the past few weeks, there have also been claims, and counter-claims of support... and to be fair, the disputed claims of support were made by others, not by 3tera.

So... what's changed, Bert? Why is "now the time" to create the standard for interoperable cloud computing? What's happened in 9 - 10 months that has so changed the field, that these efforts don't also stifle innovation?

Simon Wardley has also reiterated his position most recently at OpenTech regarding substitutability between utility providers (which includes portability and interoperability) ... an outcome which he maintains will require not just open standards but open source standards. When compared to the Cloudware initiative, I can more easily support this "pure form" of standard creation. The commercial success of pure, open source standard approach for utility computing, however, requires a reasonably well-established reference implementation or some acknowledged leader as the de facto standard. (Again, the topic for yet another post.)

That said, Simon and I could not be more in agreement when he states that "... standards will emerge through competition and adoption rather than committee." I'd probably add to that statement that such standards don't (often) emerge as a result of the smaller, fragmented commercial interests banding together to form a "composite" competitor to a market leader.

I have to agree with John Willis when he states that "...what we today call the 'cloud' will really just evolve into a complex IT infrastructure ... which will link services from a myriad of inter connected inter-operable applications spanning internal legacy applications, internal/external virtual resources, private clouds and public clouds." (Full quote provided below.)

Head In The Clouds | 3Tera

Well I’m happy to say that I think the time has come when we have enough companies in the space working on creative products and services that a standard can progress productively. We’ve begun to share our vision for what that standard can achieve, it’s called Cloudware, and covers not only AppLogic but a whole new way to approach infrastructure.

john m willis ESM Enterprise System Management Blog

It is my belief that what we today call the “cloud” will really just evolve into a complex IT infrastructure of the future, and in the end, will just be referred to as infrastructure. There is no doubt the traditional IT landscape of the last 20 years is going through a substantial transformation on the same scale as what happened in the mid 1980’s as mainframe resources shifted to distributed computing and client server architectures.

This new complex IT infrastructure of the future will link services from a myriad of inter connected inter-operable applications spanning internal legacy applications, internal/external virtual resources, private clouds, and public clouds. For example, I can envision a scenario where a business service runs internal behind-the-firewall VMware instances for parts of an application and possibly inter-operates with resources on Amazon’s EC2, Flexiscale, Google’s App Engine, or a player to be named later. These same business services might also use resources from private internal clouds running 3Tera’s Applogic, IBM’s Blue Cloud, or Cassatt’s Active Power Management. Like it or not, Microsoft will have resources involved in this new IT management infrastructure of the future. Any interoperability discussion will need to include them as well. ...

Critically Under-damped Oscillations

Chris Hoff has a great, common-sense post on security and where in the data center it will eventually end up residing.  (If you don't want me to give away the plot, go directly to the post.  Don't read the snippet I've enclosed.)

Along with the "dampened oscillation" graphic that he alludes to (but doesn't actually draw), I'd like to add my two-cents about where security resides when dealing with server virtualization, and the network.  Server virtualization, and particularly hot migration (likeVMware's VMotion), has definitely changed the relative workload and tsuris (a technical term of art) experienced within the data center by the persons responsible for, respectively, server administration, storage administration, and network administration. 

In the days before widespread adoption of server virtualization, making a new application "production ready" was a PITA (another term of art) for the server admin, who had to specify servers, install the apps, move the appropriate data for use by the apps, test, stage, re-test, etc. 

The storage admin had a modest workload, requiring attention to allocation of storage space, setting quotas, setting policies, ... but once done at the planning stage, required modest tweaking thereafter. 

The network admin had it easiest (IMHO).  Over the course of the weeks (if not months) it took to arrange for a new application to be put into production, the network admin might have to allocate ports, set VLANs, set policies, and be present when doing the lash up with the network equipment.

Fast forward to the day when a new application goes through development, test, staging and cut-over into production ... ALL using server virtualization.  Besides the fact that the time horizon for the production deployment has likely been compressed from weeks to days, the relative workloads as this cut-over approaches is radically different from the one described above. 

• The server admin has a relative cakewalk: extend VME cluster, copy the image, or use a hot migration to herd the app into the new spot. 
• The storage admin has pretty much the same level of work in allocating space, setting quotas, etc.,  and will soon be using SAN "hot migration" (e.g. VMware's Storage VMotion).
  
• The network admin, however, just got a rude awakening.  If he's got SLAs to which his organization must commit, the network admin must allocate ports, set VLANs and VLAN policies, set up NIC teaming in both the virtual switches and physical server access switch, and set up trunking on the vSwitch and pSwitch.   Oh, and by the way... it has to be "right" for every physical server in the data center to which a virtualized application MIGHT migrate in the future.
  

Holy smoke, Chris!  It's not a single, oscillating signal.  It's (at least) three of 'em.  (... and if I were a better graphics hack, I'd drop in a jpg right about now.)

Rational Survivability: Security Will Not End Up In the Network...

... Here's the reality we actually already know and should not come to you as a surprise if you've been reading my blog: we will always need a blended investment in technology, people and process in order to manage our risk effectively.  From a technology perspective, some of this will take the form of controls embedded in the information itself, some will come from the OS and applications and some will come from the network.

Anyone who tells you differently has something to sell you or simply needs a towel for the back of his or her ears...

Hyper-V Needs a Pit Crew

One aspect of the staggered release of Hyper-V and VMM 2008 is the focus on how to manage virtualization ... any virtualization environment, but specifically Microsoft's. What this article also points out is that VMM 2008 starts out by demonstrating its capabilities to manage multi-vendor environments.

The article ends with Mitchell Ashley expressing doubt about the relative price to the customer for using VMM 2008 versus VMware's VirtualCenter. In my view, he doesn't come right out and say that the comparison is bogus, but he should. If a datacenter's use of VMware depends on ESX functionality like VMotion, DRS (an automated VMotion) or HA (high availability), the customer has to buy VirtualCenter. That makes the price of VMM 2008 an additional cost to the datacenter ... not the cost of substituting it for VirtualCenter.


Hyper-V May Cause Hyper Tension | NetworkWorld.com Community

Microsoft needs a successful Hyper-V launch out in the marketplace to begin to stave off VMware's dominance. But Hyper-V can't do it alone, that's only part of the picture. Just having Hyper-V is like having a NASCAR race car but no pit crew. Hyper-V's got to have the management tools to be successfully utilized by IT. Most agree; the hypervisor will be a commodity, it's the management capabilities enabling customers to manage virtualized environments that will win the day. ...

What's needed most to deploy software on Hyper-V at any scale is Microsoft's Virtual Machine Manager 2008 (VMM). VMM just went into beta and is expected to ship 30 to 60 days after Hyper-V's release, making the product launch likely sometime in early fall.

VMM's making some interesting claims about managing virtualization. Not only is VMM 2008 managing Hyper-V, Virtual Server, but also VMware's ESX product. Microsoft isn't claiming VMM is a full replacement for VMware's Virtual Center product, but is claiming a significant portion of what Virtual Center does can be done within VMM. ... The VMM team's blog post is claiming Microsoft does all this at one third the cost of VMware, but that's an incremental cost if you are already a VMware Virtual Center customer. And we'll have to see if Hyper-V and VMM are competitive against VMware in new accounts where VMware doesn't already have a presence.

AWS EC2 gets more real(world)

The Amazon Web Services blog has announced some additions to Amazon EC2 that make it all that more viable for real world, production operations.

We just added three important new features to Amazon EC2: Elastic IP Addresses, Availability Zones, and User Selectable Kernels. The documentation, the WSDL, the AMI tools, and the command line tools have been revised to match and there's a release note as well.

...

The Elastic IP Addresses feature gives you more control of the IP addresses associated with your EC2 instances.

...

Availability Zones give you additional control of where your EC2 instances are run. We use a two level model which consists of geographic regions broken down into logical zones. Each zone is designed in such a way that it is insulated from failures which might affect other zones within the region.

...

Finally, the User Selectable Kernels feature allows users to run a kernel other than the default EC2 kernel. Anyone can run a non-default kernel, but the ability to create new kernels is currently restricted to Amazon and select vendors.

EC2 becomes more viable as a commercial grade, virtual data center utility with each new feature. I'll be interested to see what this does to the service's adoption by both corporate IT and SaaS players with SLAs to maintain.

HP's Data Center Transformation ... don't forget the network.

Reading through some of the articles commenting about HP's announcement of the their Data Center Transformation initiative, I came across this rather odd bit from Arthur Cole at IT Business Edge. He mentions that he's had the opportunity to speak with John Bennett, WW Director of Data Center Transformation Solutions at HP. After setting the stage for the conversation, he makes the point that, too often, data center refurbishments are done to meet short-term goals, and distinguishes HP's approach as being a good deal more strategic.

“Rather than think of it as one massive project, we’ll develop a strategic view first, and then use individual projects over time to build out the next-generation data center,” Bennett said. “You’ll achieve your tactical objectives on particular projects, but you’ll also lay out the foundation for years of compounded returns.”

Sounds right, and then Cole points out what, to him, seems problematic.

... About the only flaw in the plan that I can see is a lack of network support. With server, storage and virtualization as part of the mix, I was a bit surprised when Bennett said he hasn’t had many dealings with HP’s networking unit. It seems unlikely that a series of ProCurve switches couldn’t be brought in should the need arise, although that need could be substantial given the level of virtualization and consolidation that uses are likely to require. It might make sense to make networking a more integral part of the strategy.

(.... pregnant pause..... raised eyebrow.)

VMware: All (ok, some) your access network are belong to us.

Colin McNamara and Scott Lowe, two bloggers from Eplus Technology, have been doing an excellent job of pointing out the network gap created by VMware. McNamara, in a recent post, sets out the problem very well:

... Your access layer is no longer a top of rack Cisco switch, or end of row aggregation chassis. It is now a virtual bridge that exists logically within your VMware ESX server.

This causes an interesting question to come up in many customers - Who is responsible for the configuration and maintenance of this Vswitch?

...

We no longer have this well defined edge at the access layer. The access layer now exists virtually inside a server. More specifically, it is a logical devices running in a Linux server. This presents a challenge because it requires cross over knowledge. Whoever is responsible for this integration has to be fluent in Linux systems administration , and also fluent in network design and operations. Frankly this is a rare skill set to come across, as it requires and engineer who has attained high proficiency in both systems and network engineering.

I see this fuzzy line of demarcation often as a failing point for many VMware integrations. Many times I see network operations teams not involved in ESX cluster design because its a “server” , and systems operations teams generally don’t have the networking skills necessary to design and implement an fully functional system.. The solution to this problem is education and collaboration. ...

I'm not convinced that the solution to the problem lies only in "education and collaboration". It's the central, defining issue around which Replicate's products are being created.

HP and IaaS

Somewhere behind the shock of JPMorgan Chase buying Bear Stearns for $2 a share, there's the HP launch of a well considered Infrastructure as a Service (IaaS) offering.... a virtual data center for enterprise class computing. Larry Dignan and Dan Kusnetzky do a nice job of summarizing the announcements.

As for the competition, what caught my attention is Dignan's reference to the Forrester Research list of players. With the exception of Amazon's AWS (which has some heft but appeals to a long-tail part of the market), that list of Cloud Providers looks terribly thin... not exactly what I'd call competition for enterprise computing offers. Sorry... this is a not yet a competitive market ... not by a long shot. HP's stolen a march on everyone, with the possible exception of those parts of the ecosystem we rarely hear about ... the big services outfits like T-Systems and their Dynamic Services for SAP Applications. That's the competition.

I'm looking forward to digging into the details of HP Insight Dynamics - VSE and Operations Orchestration, the software enhancements HP is really going to deliver. The (rhetorical ?) question continues to be asked as to whether "... business will bite on data center in the cloud?" (to use ZDNet's turn of phrase). My simpleminded analysis says that, given its current reliance on in-house data centers, enterprise IT can't and won't rely on virtual data centers / outsourced data centers until they have a technical and operational means of integrating both forms in a (dare I say it?) form of infrastructure collaboration.

The idea is that, when reallocating load for a SAP application or scaling out to meet the requirements of the end-of-month analyses, the systems which manage the application should be blind as to whether the resources are in the cloud or in the enterprise data center. This ability to span or bridge the in-house and outsourced data center operation is almost impossible today unless the data center infrastructure is purpose-built for just that kind of operation. The solution relies on interoperability of the outsourced service and in-house data center: they must both operate as utilities and have the same or compatible infrastructures.

Perhaps, if HP's offer gets adopted by enterprise for the in-house, next generation data center, they will create a demand for their IaaS . That strikes me as a limited market. Rather than a single infrastructure technology on both service side and in-house side, the necessity is one of "compatible" or communicating infrastructures. In order for collaborative data center infrastructures to come about, a piece of network virtualization infrastructure needs to come into existence.

Virtualization Risk and the Fishtank

I missed this article in CIO (Top Ten Virtualization Risks Hiding in Your Company) when it was published about a month ago. After running through a believable list, I looked at one of the comments and couldn't help but smile:

... I think the comments above missed risk #11, the physical world outside any hypervisor host.

All hypervisors still depend on underlying physical machines being correctly connected to network and storage -- in multiple paths, to allow access for all VMs correctly.

In other words, all the above article comments apply to moving around the virtual machine "fish" inside the hypervisor OS "fishtank" -- but who moves and manages the associated fishtanks (with associated network & storage I/O plumbing etc)?

Kevin Epstein (of Scalent) then goes on to set out a number of additional problems, and although he makes sure to inform the reader that (in his opinion) Scalent solves these problems, I have to agree with the problem identification (if not completely with his solution):

1. Network connectivity matters
All hypervisor hosts in a group or "cluster" who are going to share virtual machines -must- share a LAN subnet.

2. Storage access matters
All hypervisor hosts in a group or "cluster" who are going to share virtual machines -must- share storage access.

3. Hardware failover must be anticipated
VMs will fail to another clustered hypervisor... if, and only if, one exists and has cycles! (See point 1 and 2)

4. Movement between Physical and Virtual (and back, repeatedly) is a necessity in real data centers, and is -not- usually seamless ...

5. Non-x86 Hardware
Not all hardware is x86! Sun now has LDOMs, AIX too, how to manage workloads between those and the rest of the virtual universe?

Xensploit, VMotion and VM Migration

Over the course of the past week, I've seen a number of references to recently published proof-of-concept exploit of virtual machine hot migration. It's got a number of our more exciteable colleagues wrapped around the axle. Two thoughtful and well-considered posts have emerged in response, both worth reading. Scott Lowe and Warren Wu take on the issue directly, provide good advice and, in so doing, make an argument for the class of virtualization management systems in development at Replicate. It's nice be reassured that we're addressing a real source of data center pain.

VMotion and VLAN Security - blog.scottlowe.org - The weblog of an IT pro specializing in virtualization, storage, and servers

... Xensploit, as it’s called, is the recently demonstrated exploit that allows virtual machines (VMs) that are “in flight” during a live migration (XenMotion in Citrix XenServer, VMotion in VMware ESX Server) to be manipulated. If you haven’t yet read the PDF that describes Xensploit, I highly encourage that you take a look at it. It’s very enlightening as to exactly what can be done to an in-flight VM.

Naturally, the best way to protect against this particular problem is to guard the integrity of the live migration network. For simplicity’s sake, I’ll refer to this as the VMotion network from this point on, but keep in mind that it is equally applicable to any network connections on any virtualization platform that uses live migration.

The most surefire way to protect the VMotion network is to place it on its own dedicated, physically separate network, using separate physical NICs plugged into separate physical switches that do not possess any connections to production networks. This will ensure that unauthorized access to the VMotion network is prevented, but comes with disadvantages as well: this configuration requires more physical NICs and more physical switches than other configurations. ...

Keeping Your VMotion Traffic Secure

... Although impressive, this work by no means represents any new security risk in the datacenter. It should be emphasized this proof-of-concept does NOT “take over the hypervisor” nor present unencrypted traffic as a vulnerability needing patching, as some news reports incorrectly assert. Rather, it a reminder of how an already-compromised network, if left unchecked, could be used to stage additional severe attacks in any environment, virtual or physical.

On an insecure network, man-in-the-middle attacks can target both virtual and physical machines. The techniques published are novel in that they go after the contents of migrating VM memory to target credentials and data, rather than going after similar information flowing across internal network transactions. Putting aside the question of whether it’s even worthwhile to target memory instead of network traffic directly, the sensitivity of VM memory was never the question. ...