Analytics and Big Data for Infrastructure

by Edward Newman on 10/19/2012

I had an interesting conversation a few weeks ago and was asked where am I placing my big bets for the future of IT.  I’ve been very interested in Big Data and Analytics for awhile now and this question made me think about how I needed to evolve to reflect that.  I think the next big thing for me in IT is going to be applying the concepts of Big Data and Analytics to cloud services and cloud infrastructure.  This is going to be especially important for service providers, and I’ve been recommending to my customers that they become service providers and service brokers before their customers find some others in the marketplace, so it’ll be applicable to nearly ever sizable IT organization.

The explosive growth of data is a problem not just for the business, but for IT as well.  All the virtual machines and software-defined Xs out there create huge amounts of data regarding their status, utilization, power draw, throughput, etc. etc., this is definitely Big Data.  I started my master’s program on Predictive Analytics thinking that I’d want to focus mostly on some of the typical Big Data problems we all hear about in the marketplace but as I’ve learned more and interacted with more highly virtualized customers who are moving to cloud infrastructure solutions I really believe that analytics is necessary for good IT management.

The adoption rate for converged infrastructure has continued to increase, vSphere allows for genuinely huge virtual machines with insane I/O capabilities, and customers are utilizing cloud technologies for 95%+ of workloads these days.  There is a lot running on cloud infrastructure and analytics is going to be key for predictive capacity and performance management.  Every study and survey these days indicates that IT is going to continue to be asked to do more with at best the same amount of resources, but almost certainly fewer resources.  IT as a Service, and IT service management in general, is at its best when IT’s best are focused on high value areas like capacity and performance management and not things like service request and service provisioning.  Analytics is going to be the only way to get the information that these engineers and operators will need to manage the environment as it continues to grow, as workloads diversify, as the cloud infrastructure continues to converge and the expectation of the business continues to asymptotically approach real time.  Big Data isn’t just a problem for the business, it will be a huge impediment to good management of cloud infrastructure.

We are going to need good solutions that provide predictive analytics for IT, bringing together data from the converged infrastructure, virtual objects running on it, customer experience, facilities, power, and other key components of a 21st century data center and the services running from it.  We will also need to integrate information regarding the public and virtual private services available for consumption as well.  I think hybrid cloud is definitely the way to go and agree that IT will need to be the service broker to be relevant in the new enterprise.  Analytics will allow IT to be more automated, predictive and efficient.  A good analytics solution can determine what workloads can move where and when certain workloads should move in or out of the data center to take advantage of pricing, prepare for upcoming bursts, or other business events and requirements.  This requires a solution that can federate data across all the channels producing it in the data center.  We should also look to other big data solutions like real time pricing for new ways to think about cloud service optimization.

I’ve often felt that IT can be a bit like the cobbler’s children, always the last to benefit from new technologies and methodologies that we apply to business problems, we need to get in front of this now to reap the full benefits of cloud infrastructure and ITaaS.

Share
  • Ed Newman

    interesting, I’m glad I stumbled across this…