Telematique, water and fire.

Virtual Infrastructure Optimization and Virtual Infrastructure Assurance

Jeff Boles has written a very interesting and very informative piece on the importance of virtual infrastructure management, and particularly its optimization in the virtualized datacenter.  He refers to the issue as virtual infrastructure optimization.  Being in the business I'm in ... next generation virtual system management for configuration and fault management ... I really appreciate the attention to virtual infrastructure (or, perhaps, more accurate virtualized infrastructure).  He's said all of the right things about optimization.

While he correctly identifies many of the problem sources -- interdependencies of the virtual and physical systems being key in the argument -- he stresses performance as the aspect or dimension on which to focus.  He then makes the statement: Faced with complexity and potentially catastrophic impacts from any change, administrators face the unknown.

The catastrophic impacts of which he speaks are rarely just a matter of performance optimization.  They are generally issues of infrastructure configuration errors and mismatches ... often mismatches between the virtualized infrastructure (e.g. the virtual switches, portgroups, and VLAN connectivity) and the physical infrastructure (the "hard goods" ... switches, storage systems, firewalls, etc.)  In fact, recent studies by Andi Mann at Enterprise Management Associates regarding virtual system management indicate that the greatest source of unplanned downtime in the virtualized datacenter (ranging from 60 - 70% of all outage incidents) is attributable to configuration errors and mismatch.  This single fact should bring to light the unpleasant truth about the way in which configuration design and on-going configuration management need to be rethought.  

Jeff points out that "...more than 85% of businesses today rely on their initial testing of known 'good' configurations or arbitrary rules of thumb rather than real data when they manage and make decisions about their virtual infrastructures."  Doesn't it make sense to provide analysis of the configurations before their use as the "golden masters", and CONTINUOUS monitoring of the topological and configuration changes being requested by the various players in order to reduce the likelihood of catastrophy? 

I liked the Q&A that Jeff uses to get across the key messages of VIA.  The one I have some trouble with, simply because I don't believe he treats misconfiguration in its entirety, is this one:

Can I immediately drill into the root cause of performance issues in my environment, and discover what happened or changed? 

While a VIO solution may arm an organization with the right data to avoid misconfigurations in the first place, VIO tools can also provide real-time or near real-time visibility into what is happening in an environment, enabling administrators to immediately identify performance anomalies and root causes.  VIO solutions can capture history, providing an audit trail that identifies when problems started, and what happened.

In a sense, what is missing from the article and from his list of virtual systems management products is infrastructure assurance.  It's not just the data required to avoid misconfiguration once an error is detected.  It's also that set of tools and systems deployed for the initial design, at the time there's a major change / revision to the datacenter, and (unlike conventional datacenters) revisited every time a "golden master" is called upon:

1. the starting configuration is well designed for the very fluid, dynamic environment of a virtualized datacenter,
2. that the configuration is the combined physical and virtual infrastructure elements, treated as a unified infrastructure rather than as separately designed and managed "physical configurations" and "virtual configurations"
3. that the instrumentation supports on-going monitoring, analysis and prescriptive actions to counteract "infrastructure drift" and eliminate (or at least significantly reduce) the major causes of catastrophic failure and performance problems.

To Boles' list of key technologies (instrumentation and infrastructure optimization) I would have to add the unified (virtual and physical) infrastructure analysis and directed re-configuration.  The result is virtual infrastructure assurance (VIA) to which VIO then is applied.  These are the technologies (of which Replicate's RDA is one) which provide visibility into the causes and prescriptive actions that actively and continuously reduce virtualized datacenter failure.  These fall logically into his requirement for a holistic view of the entire virtual infrastructure and into the configuration diagnostics that include remedial actions.  Needless to say, I'd like to see these types of discovery, analysis and guidance tools invited to the party.

InfoStor : What is virtual infrastructure optimization (VIO)?, March 2009 Page 1

While the IT practitioner's every day is a swim through waves of invisible bits, there has long been some comfort to be found in the "physicality" and accessibility of key devices. When problems arise, administrators have always been able to identify a switch port for examination, a server at the end of a wire that might be causing problems, an HBA for inspection, or any number of other physical things for further examination. But in today's data center, that comfort has vanished.

In part, this is due to virtualization, and while this trend is spearheaded by server virtualization, variations include application virtualization, network device virtualization, I/O virtualization, storage virtualization, and more.

VMware ESX Health Check Tools

Duncan Epping has put together a "what's-in-my-toolbox" post on his blog, Yellow Bricks, that lays out what he uses to deliver a health check engagement in his capacity as a VMware Professional Services Organization practitioner.  It's always interesting and useful to see what a pro adds to the "standard" toolkit, and we at Replicate are proud to know he considers RDA of value in ascertaining the health of a VMware installation.

Health Check tools I use » Yellow Bricks

A few days ago Scott Lowe asked me which tools I use to deliver a health check engagement. A health check is a standard VMware PSO engagement, a VMware Consultant will be on site to check the status of your environment and will draw up a report. ...

Chris Wolf Throws Down the Gauntlet

Chris Wolf of Burton Group is another voice in the group that calls for standards with respect to virtualizion's impact on security and compliance audit.  He restates the call from auditors looking for guidance, and a (typical ?) response from the vendor community to the effect: It's not my job.

So, he's much happier now that VMware has apparently made it their concern.

VMware Launches Site for Security and Compliance Auditors at ChrisWolf.com

I’m once again going to repeat my request to you - please get serious about providing guidelines and clarity for security auditors. They want your help. Without it, some will inevitably revert to enforcing full physical isolation within their organization’s virtual infrastructure, something which reduces consolidation density and undermines your TCO arguments. What do you say? If you’re serious about being a production-class virtualization platform, you need to publicly demonstrate how you are serious about security and compliance. The ball’s in your court.

VMware Gets Busy with PCI-DSS

Another interesting addition to the conversation regarding virtualization and PCI-DSS.  VMware has joined PCI, and we'll now see how they can improve the situation ...hopefully.

VMware makes the case for PCI DSS compliance

...Today, with a nod to millions of merchants worldwide that accept credit card payments, VMware Inc. announced that it has joined the Payment Card Industry Security Standards Council (PCI SSC) to incorporate awareness of virtualization into forthcoming versions of PCI regulations.

The company has also launched the VMware Compliance Center, a website dedicated to educating merchants and auditors about compliance in a virtualized environments, and the resource includes links to relevant white papers and webcasts.  ...

RDA Rocks!

After his first foray with RDA, Eric Sloof of NTPRO.NL decided to use Replicate's RDA 1.0 as part of a course on  VMware installation and configuration.  Here's a portion of the report.  Gotta love that conclusion.

Replicate Technologies | RDA Rocks - NTPRO.NL

This week I’m delivering the famous VMware Install and Configure course at XTG in the Netherlands. At the end of the third day it’s time for VMotion. Three ESX servers are added to one Virtual Center server and the students have the task to make all their virtual machines VMotion compatible. I took this opportunity to upload the Replicate Technologies Datacenter Analyzer virtual appliances....
...  In my classroom [RDA found] an issue with one of the virtual machines. This machine functions as a router and is booted from a floppy. Besides that it’s connected to an internal only virtual switch. Definitely some show stoppers for VMotion. One of the virtual machines had CPU affinity, RDA didn’t report the affinity setting, [and] I posted a feature request at the [Replicate RDA] forum. In the screen dumps you can see my findings. To conclude: easy setup, fast results, RDA rocks.

PCI DSS 1.2 and the On-going Conversation about Virtualization

While cruising through the feed-reader, I came upon Eric Sieberts recent post regarding the release of the Payment Card Industry’s Data Security Standard (PCI-DSS), version 1.2.  Eric notes that "... the specification dictates what must be done to secure a server that may store or process cardholder data, but if that server happened to be a virtual guest the host server would not be considered in the scope of the specification."  He then wonders (out loud) what could be the cause for this lack of attention (see quote below). 

This post reminded me of a conversation I had in August with Scott Loftesness of Glenbrook Partners, who arguably knows more about technology and the payment card industry than any five persons on the face of the planet.  He pointed me to this article as to why failure of PCI DSS 1.2 to address virtualization won't matter.  The author, David Taylor, is certainly no slacker.  He's the VP Data Security Strategies at Protegrity, as well as the founder of the PCI Knowledge Base, Research Director of the PCI Alliance, and a former E-Commerce & Security analyst with Gartner.  He takes a pragmatic approach, urging the reader to not wait for standards, and is pretty clear that he's a believer in the value of virtualization.  But there still seems to be some "buck passing."  He seems to be saying to the merchants who are subject to the PCI DSS standards:

• You need to prove to prove to an assessor that virtualization is secure enough to pass PCI audits.
• You need to cost-justify the amount of money required to do so.
• You need to push on your application software vendors to warrant the security and functionality of their products in virtualized environments ... something they, apparently, are often unwilling to do.

To the first point, it seems to me that best practices, standards and compliance tools or other means by which assessors can address the issue with uniformity are necessary.  There are a number of security specifications for virtual hosts (one of which Eric Siebert references in his post), which, if adopted, would be a reasonably objective basis for the standards and best practices.

With these standards in place, there seems little reason why the application vendors could not address the issues of security with respect to the use of virtualized infrastructure (the hosts and networks) as well as the virtualization of the applications themselves. 

This same tale is going to be told multiple times.  It's not just about PCI, but also will impact a standards and regulations like Sarbanes-Oxley, as well as (here it comes) the standards for data security and processing security in SaaS and IaaS environments ... Yes, I mean "cloud computing."  The PCI industry has a chance to do this right up front, without the buck passing.  I think I'm with Eric on this one.

Update:

Seems that while I was heads-down with Replicate's product launch, I missed Christofer Hoff's post on PCI, virtualization and clouds.  Once I get out from under, I'll get caught up and join the fray. 

Just to be clear -- I agree with most of the points that David Taylor has made, but to follow along with this reference to the OSI standards vs the TCP/IP development of standards ... what we're missing today is the moral equivalent of the TCP/IP definitions of best practice and standard.  If the PCI DSS folks won't step up to it, let's figure out who will.

PCI Data Security Standard updated, but still does not address virtualization — Server Virtualization Blog

I am puzzled as to why they would continue to ignore virtualization. After all, isn’t just about every company virtualizing in some fashion these days? Are the people that write the specification parameters just ignorant of what virtualization is, and that it has a direct impact on their regulations? Or are they just trusting that we are all securing our virtual hosts properly and there is no need to address them? If that’s the case then they have misplaced a critical amount of trust as I am sure there are a great many virtual environments that are not properly secured. Likewise, ignoring virtualization completely greatly reduces the effectiveness of their efforts to secure environments that deal with cardholder data. It’s essentially fortifying everything within a castle, but leaving the front gate open.

StorefrontBacktalk - Why PCI 1.2 Ignoring Virtualization Won't Matter

... The issue is more than just PCI compliance. It's about reliability, performance and data integrity. The point is that deciding whether to deploy virtualized servers broadly throughout the enterprise should not hinge on PCI compliance. Once the larger application and management issues are addressed to the satisfaction of the head of IT infrastructure, and the controls documentation is put in place, then PCI compliance becomes a minor issue by comparison.

Eric Sloof - First Impressions of RDA 1.0

Eric Sloof of NTPRO.NL has the distinction of making the first unsolicited comments on the use of Replicate Datacenter Analyzer.  Thank you, Eric.  You've expressed the kind of response we're seeking from our end-users -- surprise and delight.

Replicate Technologies | My RDA Dashboard - NTPRO.NL

One of my weekend projects :-) is the evaluation of Replicate Technologies Datacenter Analyzer. Yesterday evening I downloaded the Probe and RDA virtual appliances. This morning I started with importing the 2 virtual appliances into Virtual Center. I had to convert the Probe to a template and configure the RDA server. Everything went pretty straight forward. After a while I started my first analyses. To my great surprise RDA immediately confronted me with some faulty configured VM’s and Switches. Oren Teich is following me on Twitter and posted this tweet.

Have >10 people in the wild trying out RDA, including @scott_lowe, @depping, @esloof, & @matt_carpenter. I'm surprisingly nervous! about 10 hours ago from web.

Let’s wait and see what the others think about this new product, I’m very positive. When you click the thumbnail you will get a screen dump from my RDA Dashboard.

Clouds, the Criminal Element and V12N to the Rescue

Christofer Hoff has an interesting post today that reminded me of a conversation I had with Steve Tuecke, Carl Kesselman and Ian Foster in 2004 when we were establishing Univa.  The conversation pointed out that grid computing was, with little fanfare, a fundamental basis for the botnets being implemented by the "bad guys", and that grid computing models would be the most reasonable basis on which to protect the enterprise from a significant portion of intentional security threats.

The challenge, as Hoff makes clear, is the capability of establishing appropriate (and appropriately malleable) policies that travel with the applications and data.  The malleability required is not likely to be found today in conventional provisioning and scheduling systems, nor in conventional configuration management systems. 

The industry is at the stage in the evolution of utility computing/cloud computing/grid computing where the flexibility and dynamic nature of virtualization has to be applied generously to the solution of infrastructure problems like security.  At Replicate, we've started to apply it to 21st century datacenter fault prevention and remediation, which is a challenge big enough to last us a while.  It's likely to bring us in contact with a number of the security issues Hoff raises, though from a different starting point.  I'll certainly be joining Hoff in watching companies grappling with this ... though Hoff's more likely to be on the playing field (BJJ mat?), and I'll be in the bleachers.

Rational Survivability: Cloud Computing: Invented By Criminals, Secured By ???

...
One of the obvious benefits of cloud computing is the distribution of applications, services and information. The natural by-product of this is additional resiliency from operational downtime caused by error or malicious activity.

This benefit is a also a forcing function; it will require new security methodologies and technology to allow the security (policies) to travel with the applications and data as well as enforce it.

VCritical and "free" Hyper-V

Just uncovered VCritical, a recently started blog by Eric Gray of VMware, "to provide commentary on virtualization and virtualization management.  The various meanings of “critical” appealed to me, hence the name VCritical." 

His writing style and sense of humor appeal to me.   While he's unabashedly pro-VMware, his employer, his points seem fair and well considered on any side he's taking.

This post caught my eye, and clarifies the Microsoft approach to V12N pricing.

VCritical · When does your “free” Hyper-V Server cost $1304?

If Hyper-V Server is free, how much should you pay to manage it?

The most current information available at this time is from a Microsoft blog post: Nexus SC: The System Center Team Blog : SMSE and VMM 2008 Updated Pricing Information - Effective November 2008.

If you read through that blog post you will discover that for every hypervisor managed by SCVMM you will owe Microsoft $1304. Or, you can opt to pay $1497 and also use the other System Center features.

Just remember this when you hear the “free, free, free” and the “management, management, management” rhetoric:

It’s actually one or the other, not both.

On the Approach Path to RDA 1.0

The past few weeks at Replicate Technologies have been a combination of intense concentration and confident, regular progress.  It's a real tribute to the team and the  development process that everything has been showing up on time, and with the highest quality.  We can now state categorically that we have a product. 

We're quite pleased with the outcome.  The first-customer-ship (FCS) version of the Replicate Datacenter Analyzer (RDA 1.0) arrived on schedule last week.  The new website is up, and includes the first of a set of short webcasts to provide introductions and tutorials.  We've instituted an "early access" program, limited to a modest set of users, in order to test out our back office and support processes. Based on our experience so far, we should be in full commercial availability as scheduled in a few weeks. 

If you'd like to see what we've been up to for the past 12 months, stop by the site and, in particular, check out the short webcast introduction to RDA.  There are still openings available in the Early Access program, so contact us to find out how your virtualized datacenter can benefit from improved reliability, availability and decreased management effort.

Replicate Technologies

Unlike other products that simply track individual configuration item changes over time, Replicate Datacenter Analyzer (RDA) builds up a comprehensive model of your virtualized datacenter. Combining empirical data from Replicate probes with configuration information derived from your existing system management tools, RDA constructs a unified view of your datacenter across all your administrative domains. Leveraging industry best practices, RDA identifies and corrects configuration problems, eliminating errors before they can cause downtime.

Replicate Technologies, RDA