Steve Herrod’s super session was one of the things I enjoyed most about this year’s VMworld.  Not only were the technologies and ideas that were introduced inspiring and where I was hoping to see VMware head but there was real passion for making content accessible evident throughout the entire presentation.  VMworld coming so soon after the great one day class from Edward Tufte really increased the impact of the message.  I had been thinking of content in terms of Big Information, of how do we present information to people, how do we share and collaborate etc.  Steve’s presentation pushed that point even further, it’s all about the content really, the receptacle that it’s delivered in is irrelevant, it could be a briefcase, or a Ming vase, all the end user cares about is what’s in it.  Just take a gander at Vincent Vega there staring at Marsellus Wallace’s soul, the briefcase isn’t what was cool in Pulp Fiction, it’s what was in it.

One of the great things about the commoditization that is occurring right now in the end user compute space is that people are demanding what they want be delivered to whatever device they have handy at the moment.  I access my email and calendar via a MacBook Pro, an iPad and an iPhone depending on what I have near me at the time.  There are some things that I have to go to my MBP for every time, whether I like it or not and boy do I want that to change.  What we saw on September 1st is a direct response to that desire on the part of end users.  Virtual phones, HTML5, virtualized apps and desktops, and new ways to collaborate and share files, messages and ideas are where VMware is going and it’s all divorced from the container it is being delivered in.  What could be more exciting, and more scarey?

I’ve been interviewing several people to lead up the Virtualization group of my organization lately and one of the candidates asked me an excellent question, “Well, what do you mean by Virtualization?”. Very good question, am I talking about VMware, the hypervisor, virtualized infrastructure, what? Apparently I’m in a heretical mood these days because my answer was, “No, I don’t mean any of that, I don’t want to limit it to that. When I talk about Virtualization and what I want this team to focus on is bigger than that. Virtualization to me is technology enablement allowing IT to run the workloads you need to where you want to.”

That distinction is important to me, and more importantly my customers. They NEED to run particular workloads in order to keep the doors open, to support the key processes required to run their business. They WANT to run those workloads where it makes sense to from an availability, or performance, or security, or even functionality perspective. Where they want to run that particular workload might change based on market conditions, or price of energy, or where the user has moved to, or because the security profile associated with it changed. It’s all about mobility. Now I know, technically, that doesn’t only require virtualization, and in some cases doesn’t even require virtualization, but that’s what customers are looking for, that capability. Virtualization used to be about consolidating many workloads onto one physical machine to optimize utilization of resources. That’s that first phase that many refer to on the journey to Cloud, consolidation of workloads. It’s an interesting concept to the CFO and CIO and in many instances the CISO, it’s not all that compelling to an end user or a line of business.

In my Cloud Heresy post I posited that we need to solve the business’s problems, this is a part of that. The key attribute of virtualization that’s getting the most traction with IT’s customers is mobility, sometimes referred to as agility. It’s about more than the hypervisor, but that’s the start. Why wouldn’t you put a hypervisor on every physical system? Why chain yourself to a machine, the overhead associated with the hypervisor isn’t that much any more and the benefits to agility, recoverability, &c. associated with it would outweigh the added cost in my opinion. Virtualized networks, storage, changing the perimeter for security from the network endpoints to the hypervisor are all components of enabling mobility in your enterprise. So, that’s the charge I’ve given my new Virtualization lead, continue to build out offerings and delivery capability to enable workload mobility for our customers.

On an unrelated note I was interested to see a cool new infographic from VMware about how vSphere is everywhere. I’m not sure I would’ve released all the information they did on it as some simple math reveals an interesting point. Two pieces of the graphic are below:

So there are 20 million VMs on vSphere, that’s awesome!

And there are 798,000 vSphere admins, outstanding!

Ooooh, wait a minute, that means each admin is responsible for an average of 28 VMs. Hmmm, that doesn’t sound so great. Let’s assume that 50% of those vSphere admins are geeks who have the certification but aren’t necessarily using it as a part of their job. Well, that still leaves an average of 56 VMs. Either we’ve got way too many admins, or we need to continue to push for more virtualization in our environments, or vSphere admins are managing many, many physical systems or other things in addition to their VMs. Would be interesting to find out more.

With apologies to Chuck Hollis at EMC and James Governor at RedMonk I decided to take a crack at this whole “Why Applications are like fish and Data is like Wine” meme by extending it to posit that Information is like Cognac. Now, I’m not usually one to kick a dead horse but I think that all the talk of Big Data has maybe obscured something that I view as a problem with Big Data: knowledge workers don’t consume data, they consume information. I see Big Data as a problem quite frankly, and the IDC Digital Universe Study put it in context. If Big Data is the problem, Big Information is the goal, and to get there we need automation and analytics. So if you’ll bear with me I’ll share how I think Information is like cognac.

Cognac is made of wine, and not just one varietal or vintage, it is a carefully crafted blend, cared for and created layer upon layer to produce something otherworldly when done correctly. Cognac is crafted by real artists, masters who guide the spirit through it’s distillation and aging, they understand how to bring together the different grapes, wines, cognacs to deliver something that is greater than the sum of its parts. Information is like that, it takes in data, from many different systems, formats, time periods and through analytics is able to give the engineer, scientist, executive, whomever something meaningful and beautiful out of all that chaos. Knowledge workers and data scientists are like those artists who blend the cognac.

I was lucky enough to take a class on Knowledge Management ten years ago while I was pursuing my Master’s degree and it’s one of the classes that’s really stuck with me. One of the most intriguing concepts we studied was the Wisdom Hierarchy, which is sometimes referred to as the DIKW hierarchy: Data; Information; Knowledge; and Wisdom. Data is the raw material that this framework is built on, but ultimately the more data you have the more that stands in your way to true knowledge. Information in this system is Data + Analysis, something that has meaning attached to it. Data in and of itself isn’t really useful, Information however is, hence Big Data is a big problem, or in the very least a big distraction.

So, IDC tells us that Data is doubling every two years, it’ll be 50X what it is now in 2020 while IT Professional ranks will only grow 1.5X. So clearly we will need much more automation in our infrastructure and data management tools. We will definitely need a lot more automation in our analytics systems in order to be able to turn this huge pile of data into something useful. The number of files, or objects or containers that all this data is stored in will increase by 75X, so our solutions will need to be automated, adaptive and federated. And that’s just to turn all that data into information. Really what we want to create is Knowledge, and as someone who is engaged and enthusiastic about this space we eventually want to spread Wisdom throughout our enterprises. To enable that we’ve got to make it a lot easier to create new applications and ways of sharing not just the Information but the context and insight that makes it Knowledge. This is part of the point of Chuck and James’ posts, Applications, like fish, are best when fresh and up to date, long requirement definition and software development lifecycles pretty much guarantee that many applications are rather stinky upon arrival.

But wait, there’s hope! There are many, very cool efforts and initiatives underway to deliver on this. I’ve been very impressed with initiatives in this space from GreenPlum, R, NoSQL, Tableau and many others to make data more pliable and easier to turn into information. I am admittedly biased, but I think that what EMC’s Intelligent Information Group is doing with Documentum xCP and its Application Composition tools is outstanding. Also, I was lucky enough to attend the Data Scientist Summit in May and the work that companies like Via Science, Kaggle, 23andMe, Code for America, and Factual is really inspiring and mind blowing.

So, Big Data, Big Information, whatever we want to call it, color me all in, this is going to be a wild ride, and hopefully we’ll all get some wonderful cognac out of it. Cheers.

IDC Released their 2011 Digital Universe Study and the results are pretty amazing: data is doubling every two years! This is the fifth year that the IDC has released this study and each year I continue to be surprised by the results, just when I think things have started to reach terminal velocity around data growth they accelerate more. Currently data growth is outpacing Moore’s Law, suddenly the phrase Big Data just doesn’t seem to cut it any more. There are all sorts of findings in the study and the repercussions for our industry will require many changes.

I recently wrote here about the need for automation in security, and Christopher Hoff has suggested some practical ways to get started here and has started an initiative around Security Automata here. This is one of the ways that the growth of data is impacting security, the very framework for how we approach protecting assets needs to change in light of the deluge of data.

Another interesting finding had to do with the anticipated growth of data versus the growth of the number of IT Professionals, I think the image below sums it up rather nicely.

Over the course of the next 9 years data will grow 50x and the number of IT Professionals is expected to grow only 1.5X. Now that could sound like there’ll be good job security for IT folks, but this is a scary finding! If we don’t immediately change the way we manage our environments we aren’t going to be able to keep the best and brightest in our industry, much less be able to meet growing customer expectations around agility, flexibility and added value. The rise of consumer technologies that allow end users to have ready access to large volumes of data at an end point of their choosing has lead to an increase in end user expectations in the enterprise. This trend is only going to continue. So, higher expectations, larger amounts of data and only a marginal increase in the number of professionals curating this data sounds like a recipe for trouble.

Another interesting finding of the study is that during this same period the number of files\objects that this data is stored in will increase 75X. Not only will we have much more data but it will be spread all over the place and replicated and acted on by a number of collaborators, &c.. Thankfully it seems like object-based storage finally has some very good solutions that can readily scale to help meet these problems. We will however need engineers and architects who can help write and enforce file and object policies as a part of a holistic GRC approach to ensure that all of this data and its containers are manageable, secure, compliant and stored in the most cost efficient manner for their class.

There are many warning signs associated with the findings in this study, but it also highlights many opportunities for innovation, new career paths and smarter thinking about how we create, curate, share and protect our data.

Recently there’s been some chatter about the role of automation in Security and whether it is appropriate or not as a business strategy much less a security strategy. Jeffrey Carr states that EMC’s wrong that automation is an efficiency and security necessity and that you shouldn’t automate because “An automated solution will never stop a customized attack because the attack was designed to circumvent it!” (his emphasis). First, if there’s one thing I’ve learned over the last twenty years you should avoid absolutes when talking about security. Second, not automating something because someone may develop a solution to defeat it is like not brushing your teeth because it may not prevent all cavities. This seems like cutting off your nose to spite your face. Jeffrey seems to conflate EMC recommending automation in security as a necessity for efficiency’s sake and abandoning all other security policies and methods. It certainly makes for good headlines, but I don’t think that people would read the three articles/whitepapers quoted and really think that EMC is going with an “automation is everything” approach.

I do agree that an automated solution won’t protect against a customized attack designed to defeat it, that’s pretty much what I call self evident. To claim this is what EMC is calling for, well that’s the very definition of a straw man argument. Automation is a necessity for efficiency’s sake as well as security’s sake. Back in 2003 I was involved in the design and provisioning of a new data center for a financial services customer. The security and system administration teams turned on just about every last bit of logging, intrusion detection, firewalling and every other policy they could think of to create a secure environment. They redirected all logging to one system to do correlation and analysis and turned everything on. Before there was 1 test/dev, much less production, workload running in the data center they had generated 1TB of alerts and logs in the first three days. Now imagine there was no automation in place, would you really want an administrator or engineer to eyeball 1TB of raw data to figure out if there was a REAL event?

As the amount of data we create grows the amount of automation required to successfully manage the environment must grow. Nearly every industry that’s come before us has implemented automation to ensure that their most talented people are focused on the activities that add the most value to their business rather than mundane, trivial tasks. The auto industry went from handmaking every car to utilizing robots for the majority of assembly. Physicists back in the day used to sit under trees and write page after longhand page of tedious mathematical equations to explain the world. Now we have colliders to help us understand the universe around and within us. When I was a physics research assistant and student back in 1992 I visited Fermilab and one of their researchers shared with me that every run of the accelerator produces enough data to require 50 man years of analysis. These guys were the original data scientists if you ask me.

One of the key philosophies I have about IT, which you can extend to pretty much any business process, is that you want to focus the bulk of your resources on as little as possible in the environment. That is to say, if you treat everything like it is the most important thing you have then nothing is really important. Your most stringent policies, your most expensive hardware and software, your most valuable engineers and architects are more effective and provide more value if they are focused on what is truly most valuable to your business. Now, clearly, I can’t claim credit for this philosophy, commonly known as the 80-20 rule or the Pareto principle, although I like the term “the law of the vital few” best, but I try to extend it a lot further into IT and the enterprise than many of my peers.

Automation is a necessary component of IT management with TB and PB and ZB of data in our data centers, thousands of mobile workloads thanks to virtualization, fat pipes, multiple hot data centers, follow the sun (or moon) policies, 24/7 employees all over the globe, and customer expectations for near instantaneous response to every request. Oh yeah, and hundreds of thousands of hackers and script kiddies out there ranging from merely annoying to outright malevolent testing the perimeters and policies of just about every enterprise and citizen these days. We can’t handle this all manually, but the trick is applying automation where it makes sense from a cost, functionality and security perspective. It may not be the silver bullet but it is an important tool in the toolbox, one you, and more importantly your high value engineers and administrators, don’t want to be without.

Everybody’s talking about Private Cloud these days, and I think that’s great. There have been a number of really good posts and articles about it lately and I think the more people writing and thinking and implementing Private Cloud strategies and ideas the better. An informative and frankly tactically -in the best sense of the word-focused article I’ve enjoyed is A Private Cloud is Called IT by Mike Fratto over at Network Computing.

Mike, thankfully, begins by defining terms stating that a Private Cloud is one which is “wholly hosted in your data center”. I think this is the most realistic definition at the moment and my hope is that soon we will be able to extend that to be one that is managed, provisioned, secured and is compliant as if it was wholly hosted in your data center. I think he’s underestimating some of the benefits of the Private Cloud at this point versus an IaaS solution primarily because I’ve yet to see an apples to apples IaaS offering. The service levels, availability, performance, etc. just don’t exist to compete against a Private Cloud. The cost savings associated with Private Cloud are dramatic when done at scale, and I certainly haven’t seen many organizations doing IaaS at similar scales, it’s just not realistic at the moment. That being said the savings disparity between the solutions is a temporary one, the Public Cloud solutions will catch up, as will the bandwidth capabilities to allow massive migrations to them. In the meantime, the next 18 to 36 months in my opinion, Private Cloud certainly is the way to go, better savings, better security, better compliance, and more easily implemented and more importantly more easily migrated to. Let me add the caveat again, at scale! Taking 1 application, a set of call center users, a dev environment, etc. is not at scale. I’m talking entire lines of business, entire data center, or class of applications. Mike is absolutely on in regards to the steps required to get you to an automated data center, or Private Cloud and nails the reason for doing so: “leaving you with more time to work on more interesting tasks”. Or to put in my vernacular: allowing your engineers and architects to work on innovation and new offerings for the business rather than keeping the lights on. There are many studies out there that show that IT spend is focused mostly on keeping the lights on, some estimates are as high as 75%, and not on innovation and new services for the business.

Private Cloud is the new paradigm of IT, it’s not a sea-change, or a bolt from the blue, but I believe the next evolution of enterprise IT. Mike does a great job listing out several key steps specific to his realization of an automated data center that help enable the Private Cloud. His are very focused on the Infrastructure component of the transformation required. I think that there are two other key components in the transformation to Private Cloud: Applications, what is my right-sized Application Portfolio, what is my cloud sourcing strategy for those rationalized Applications, and how can I develop new Applications that benefit from the new paradigm; and Governance, what are the policies and processes required to manage the new paradigm, what do I automate, how do I secure the environment, what is the fewest number of IT controls I can implement to be compliant and what is the unified console that provides be the transparent insight into my environment from resource management, risk and compliance perspectives. It’s important to make progress against the Application, Infrastructure and Governance components in a relatively lock step fashion, getting too far out ahead in the maturation and implementation of one of the components leads to poor benefits realization efficiency and can actually cause the other areas to regress.

I argue, frequently and with just about anyone who will engage, that Cloud Computing is the model and there are several different types of instantiations.¬† This certainly isn’t a new or controversial idea, and not a sea change in and of itself.¬† The same could be said for Web 2.0, SOA, N-Tier, Client-Server and back to the Platonic Ideal.¬† The blogosphere and twitterdom is filled with talk of IaaS, PaaS, SaaS &c. as various forms of Cloud Computing and those are interesting forms but not necessarily new ideas or modes of computing.¬† EMC has laid out the vision for a Private Cloud, it’s rather well defined and we have gathered together a number of partners to help us enable our customers in the creation and operation of private clouds.¬† I’m certainly a proponent of Private Cloud, believe in the model and think that it is innovative and a new mode of computing, but I come here not to praise private cloud, but to enable it.

I’ve spent the last few months talking with customers all over the world about Cloud Computing in general and what EMC means by Private Cloud in particular.¬† I’ve been fortunate enough to get a lot of feedback from the CXO level down to the managers and administrators that will be tasked with running these clouds.¬† A few common themes have emerged in these conversations.¬† Rarely does the question, “Why Cloud Computing?” come up, it’s almost as if Cloud is a foregone conclusion, hyped into the mainstream.¬† I am almost consistently asked by people at every level, “So now what?”.¬† EMC and our partners, and the market in general, has done a good job of laying out the groundwork and vision for Cloud Computing and its benefits and a hardware and software portfolio to enable it.¬† The question becomes how do I actually execute against the vision with the products to make it reality, as it does with most paradigm shifts.

It seems to me that a lot of IT organizations are positioning themselves for Private Cloud, knowingly or unknowingly.  The virtualization of the data center, not just of servers, but real enterprise virtualization is a key milestone on the path to Private Cloud.  Not only does it provide the framework to build a Private Cloud on, it brings real benefits to the organization in terms of reduced Capital Expenses, Operating Expenses, time to provision, mean time to repair and improved customer satisfaction for internal and external customers.  These benefits are core to the allure of Private Cloud and IT is keen to realize them as quickly as possible.

I’ve often seen, and industry analysts seem to weekly report, that virtualization efforts seem to hit a wall when around 20-30% of the workloads in the data center have been virtualized.¬† There are many reasons for this, ranging from applicability of previous virtualization solutions to enterprise workloads, and insufficient application owner and line of business buy-in to the transformation leading to lack of approved downtimes and applications not being approved for P2V.¬† We’ve helped a number of customers push through this wall and drive towards their goals of 80-90% of workloads being virtualized through the development of enterprise virtualization programs, acceleration services, documenting the activities and processes surrounding the virtualization of servers and applications, training and comprehensive communication and marketing plans to get the buy-in of the stakeholders and application owners.

It’s not just driving enterprise virtualization that will help IT realize the benefits of Private Cloud, however.¬† A lot of outsourcing companies operated for years on the concept of “Your mess for less”.¬† For this to be a real transformation it can’t just be the same old problems running on a shiny new architecture.¬† A key component of the journey to Private Cloud has to be the rationalization of the application portfolio.¬† We are constantly adding new applications and features and functionality into the environment, and for every “server hugger” out there I’d argue there’s an “application hugger”, we all have our babies and we’re certainly not going to let them be torn from our arms.

A systematic review of the existing application portfolio to identify opportunities for retirement, feature\functionality consolidation, replatforming and virtualization on proprietary unix systems provides the roadmap for how many of the promised savings can be realized.  If you want to embrace x86 as the chosen platform you have to figure out how to get as much of your application portfolio as possible onto it.  Coupling this portfolio rationalization with a comprehensive business case for Private Cloud provides the framework for driving line of business and application team compliance and for a realistic timeline of how quickly you can actually realize Private Cloud.

So that accounts for the infrastructure and the applications, now for the trifecta, governance!¬† A new model of computing requires a new model of governance and the associated tools and processes.¬† Thousands of virtual machines crammed into a small number of cabinets dynamically allocating and deallocating resources is a daunting environment if your key governance tool is Microsoft Excel.¬† The identification of appropriate services to provide, service levels to achieve, and a chargeback model to allocate costs are required, absolutely required, to have any chance of successfully building and operating a Private Cloud successfully.¬† This requires transparency into what you have, what you’re using, where it is, who owns it, what it requires, how it is to be measured and monitored, backed up, replicated, encrypted, allowed to grow or shrink, &c.¬† Sounds scary, I’m sure.

The service catalog, an integrated management tool framework and automated processes allow you to monitor, maintain, provision and recover the costs of such an environment.¬† Your administrators, engineers and operations teams need to be trained on the technologies, service levels, communications plan and have their roles and responsibilities well documented to empower them in this kind of model.¬† New tools and proactive methods for communicating with your clients have to be developed and integrated to ensure they understand what services you are providing them, how they are being charged for them and what service levels you guarantee.¬† I personally think that self-service plays a key role in the development of a Private Cloud, or most cloud models for that matter, and integration of Change, Release and Capacity Management into a self-service portal can make the difference in your client’s adoption of this new paradigm.

We’ve packaged these services up under the umbrella of Accelerating the Journey to Private Cloud and have integrated our Technology Implementation Services, and several new EMC Proven Solutions into a holistic stack to enable our customers. It’s not a light switch or a silver bullet, it still is a journey, but we’ve worked hard to take the lessons learned from many years of data center consolidation and migrations, process automation, custom reporting and dashboards, building innovative solutions and architectures, product training and managing transformative programs and integrate them into an effective services and solutions stack to accelerate the journey to Private Cloud and realize real benefits today.

There’s been a lot of discussion lately about clouds and the future of IT across the blogosphere: Chuck is always good for a post or two; IBM spoke up the other day; and there are even reports that “Hey, this is real!”.¬† I can’t help but wonder if Cloud Computing is really just the marriage of flexible architecture, ubiquitous networks and IT Service Management?¬† As has been noted on this blog I am highly infrastructure biased, but I think it is apparent that fast, readily available networks are changing IT, your phone, laptop, Kindle, &c. are now viable end devices for application and content delivery almost anywhere on the planet.¬† Exciting times indeed!

If you scratch beneath the surface a bit the magic and mystery of the Cloud becomes a little more apparent: you have a high-performance, omnipresent network; a flexible delivery engine that is highly scalable and efficient; and a management framework that provides the appropriate Service Levels, security, compliance and communications the customer is seeking.  To truly deliver a cloud service you first have to identify and define a service that can be readily doled out to customers clamoring for it.  I can think of tons of services internal to an enterprise that would qualify for this designation, so I think the concept of a private cloud is a cogent one.  Take for example File Sharing, or Email, or Market Data, or Order Processing.

So why now?  The emergence of good allocation and resource management tools certainly makes the management of the service a lot easier, add adaptive authentication, identity management and role based access, couple that with the virtualization capabilities and infrastructure components geared to hypervirtualization and you have the recipe for easy to deploy private and public crowds.  The market adoption of frameworks like ITIL and ISO 20000 and their focus on Service Level Management provides the appropriate mindset for the IT organization looking to become service oriented.  Now ride all of that on a ubiquitous, converged, highly available fabric and you can provide these services to pretty much any client, via any platform, any where.

Suddenly Clouds aren’t so amorphous but really the next logical progression of virtualized infrastructure, Service-Oriented Architecture, and IT Service Management.