Big Data – Did You Know?

October 4, 2012

Great video highlighting big data, stats and the potential opportunity


Overcoming the Complexity of Big Data with Big Transaction Data

May 10, 2012

By Diego Lomanto

For most companies, the challenge with big data lies in making sense of the data acquired in order to apply it to real world problems when decisions matter most.  Big data is hot right now because we recognize that we are generating more data than ever before and that we might be able to do something with it.  However, much the execution of big data has been around storage of the data (think Hadoop) and search (think Splunk).  That’s a great start, but do they really solve any problems in a new way on their own?

Start a big data project and you will soon realize that the data itself is limited because it is partial (takes whatever is available), difficult to consume for analysis (because it’s unstructured) and often offers limited value use cases.  It’s complicated.

I think the evolution towards better value from the data is still in progress.  I think we’ll not only see continued progress in storage but I believe that technology will emerge to make working with big data feel a wee bit smaller.  What I mean by that is we’ll still collect the data at massive scales, but there will be technology that simplifies the big data into a model that is consumable by analytic applications.  In other words, it will transform the data to actually represent something that can be analyzed.

Big Transaction Data

Big Transaction Data (BTD) is a great example of this.   It is complete, comprehensive and correlated.  But it’s also usable.  Let’s have a quick primer on BTD.

What it is, effectively, is the data generated by transactional systems in raw form modeled to represent the unique end-to-end transaction that drove the data generation in the first place, and stored alongside millions, billions, trillions (insert your own “illion” here) of other transactions.  This is done by technology – typically business transaction management software that observes and reports on transaction performance at each tier.

This is REAL big data in action.  And that’s where business transaction data comes into play.  BTD takes the data and stores it in a consumable form for analytics.  The transaction becomes the anchor for the analytics process.

The Problem with Fragmented Data

For example, say you wanted to analyze the end to end process performance of a financial trade system.  The systems that execute financial trades are ridiculously complex.  Think of the most complex system you can think of  and then multiply it by 3.  Why?  Because they are using a mix of new and old technologies and it’s distributed across multiple tiers and managed by many different stakeholders.  So what you get his this hodgepodge of tiers to execute trades that is incredibly difficult to rationalize into a singular data set.  The unfortunate by-product of this is that your view of the trade transaction is really just fragmented data.  You can see pieces of the transaction performance but not really ALL of the transaction.

But, you still need to analyze trades across the tiers and processes as a single input into your trade effectiveness analysis.  So you do the best you can.  You go deep into the tier data and try to correlate it on your own within your own analytic model.  For example, you try to monitor cross-process fallout with a cool looking dashboard that gives you data on each process, but you don’t really do it well and miss a lot of cross-process issues.

Or you try to do a cost analysis.  Or a segmentation analysis.  Or a performance analysis.  But the work to create a singular data set is so complicated that you never really have full confidence in the results.

Big Transaction Data in Action

Here is a great opportunity to employ big transaction data.  Instead of working with billions of manually correlated data points, let’s simplify and work with millions of well-defined transactions instead.  End-to-end transactions that represent each trade across each process in full.  Now you have a data set that you can inject it into your BI platform or use simply use BI tools within the big transaction data solution itself for analysis.

So back to those 3 Cs.  The data is complete – that means all information is generated by BTM end to end one view.  It’s comprehensive – capturing ALL interactions. And, it’s correlated – it knows everything about vital meta-data such as user, tiers, etc. The result is easy to consume meaningful analytics leading to business outcomes.

So, big data is hot.  But it’s not quite there yet.  We’re waking up with more data but we’re still working to rationalize it.  Fortunately, the technology is on its way to simplify and gain more (true) value from big data.


Predictions for big data

November 22, 2011

Click to visit the original post

CenturyLink has released the following infographic that highlights the exponential increase in data, and the implications of Big Data on enterprises. It is predicted that video and mobile devices will be major contributors in driving the creation of 7.9 zetabytes of data in 2015. Big Data will entail 1.5 times more IT professionals managing 75 times more data moving through enterprise data centers. Image via CenturyLink Find the original post and image here. Related articles Infographic: Data Deluge – 8 Zettabytes of Data by 2015 (readwriteweb.com) Big Data and Little Data (forbes.com)

spotted on a little bit of this, a little bit of that


Gartner: 10 key trends for 2012

October 20, 2011

The technology that makes up many of the systems in the IT world today is at a critical juncture and in the next five years everything from mobile devices and applications to servers and social networking will impact IT in ways companies need to prepare for now, Gartner Vice President David Cearley says.

Cearley offered the following as examples of the way the tech world is changing:

  1. 30 billion pieces of content were added to Facebook this past month.
  2. Worldwide IP traffic will quadruple by 2015.
  3. More than 2 billion videos were watched on YouTube … yesterday.
  4. The average teenager sends 4,762 text messages per month.
  5. 32 billion searches were performed last month … on Twitter.

So what issues need to be on IT’s radar screen for 2012? Here’s a look at the Top 10 Tech Trends and the implications of those issues according to Gartner:

1. Media tablets and beyond: Bring-your-own-technology at work has become the norm, not the exception. With that come security and management challenges that IT needs to address. By 2015 media tablet shipments will reach around 50% of laptop shipments and Windows 8 will likely be in third place behind Android and Apple.

2. Mobile-centric applications and interfaces: Here touch, gesture and voice search is going to change the way mobile apps work in the future, Cearley says. By 2014, there will be more than 70 billion mobile application downloads from app stores every year.

3. Social and contextual user experience: According to Gartner, context-aware computing uses information about an end user’s or object’s environment, activities connections and preferences to improve the quality of interaction with that end user or object. A contextually aware system anticipates the user’s needs and proactively serves up the most appropriate and customized content, product or service. The tipping point here could be technology such as near-field communications getting into more and more devices. Some interesting facts here: By 2015, 40% of the world’s smartphone users will opt in to context service providers that track their activities with Google, Microsoft, Nokia and Apple continuously tracking daily journeys and digital habits for 10% of the world population by 2015, Cearley says.

4. Application stores and marketplace: The key here is the rise of enterprise application stores that can develop specific apps for users. This will let IT manage and control certain apps. But embracing the idea of user choice might be a difficult concept for enterprise IT to embrace, Cearley says. Enterprises should use a managed diversity approach to focus app store efforts and segment apps by risk and value. Where the business value of an app is low and the potential risk, such as the loss of sensitive data, is high, apps might be blocked entirely.

5. The Internet of everything: The idea here is that we are building on pervasive computing where cameras, sensors, microphones, image recognition — everything — is now part of the environment. Remote sensing of everything from electricity to air conditioning use is now part of the network. In addition, increasingly intelligent devices create issues such as privacy concerns. Eventually IT will need some central unified management of all these devices, Cearley says.

6. Next-generation analytics: Most enterprises have reached the point in the improvement of performance and costs where Cearley says they can afford to perform analytics and simulation for every action taken in the business. Not only will data center systems be able to do this, but mobile devices will have access to data and enough capability to perform analytics themselves, potentially enabling use of optimization and simulation everywhere. Going forward, IT can focus on developing analytics that enable and track collaborative decision making.

7. Big data: Big data has quickly emerged as a significant challenge for IT leaders. The term only became popular in 2009. By February 2011, a Google search on “big data” yielded 2.9 million hits, and vendors now advertise their products as solutions to the big data challenge. The key thing enterprises have to realize is that they just can’t store it all. There are new techniques to handle extreme data, such as Apache Hadoop, but companies will have to develop new skills to effectively use these technologies, Cearley says.

8. In-memory computing: We will see huge use of flash memory in consumer devices, entertainment devices, equipment and other embedded IT systems. In addition, flash offers a new layer of the memory hierarchy in servers and client computers that has key advantages — space, heat, performance and ruggedness among them. Unlike RAM, the main memory in servers and PCs, flash memory is persistent even when power is removed. In that way, it looks more like disk drives where we place information that must survive power-downs and reboots, yet it has much of the speed of memory, far faster than a disk drive. As lower-cost — and lower-quality — flash is used in the data center, software that can optimize the use of flash and minimize the endurance cycles becomes critical. Users and IT providers should look at in-memory computing as a long-term technology trend that could have a disruptive impact comparable to that of cloud computing, Cearley says.

9. Extreme low-energy servers: What if you could turn 10 virtual machines in one box into 40 slow physical servers that are tiny and use very low amounts of energy? There is a call for this type of computing to handle big data. For example, thousands of these little processors could work on a Hadoop process, Cearley says. Gartner says that 10%-15% of enterprise workloads are good for this. Moving the application from 10 images to 40 slower, less capable machines will only deliver on that promise if the software will perform the same. Server technologies are going to change to handle big data.

10. Cloud computing: This topic went from No. 1 last year to No. 10 this year, but it’s still an important trend. It will become the next-generation battleground for the likes of Google and Amazon. Going forward, enterprise IT will be concerned with developing hybrid private/public cloud apps, improving security and governance, Cearley says.

GARTNER: 10 key IT trends for 2012


Follow

Get every new post delivered to your Inbox.

Join 752 other followers