By Diego Lomanto
For most companies, the challenge with big data lies in making sense of the data acquired in order to apply it to real world problems when decisions matter most. Big data is hot right now because we recognize that we are generating more data than ever before and that we might be able to do something with it. However, much the execution of big data has been around storage of the data (think Hadoop) and search (think Splunk). That’s a great start, but do they really solve any problems in a new way on their own?
Start a big data project and you will soon realize that the data itself is limited because it is partial (takes whatever is available), difficult to consume for analysis (because it’s unstructured) and often offers limited value use cases. It’s complicated.
I think the evolution towards better value from the data is still in progress. I think we’ll not only see continued progress in storage but I believe that technology will emerge to make working with big data feel a wee bit smaller. What I mean by that is we’ll still collect the data at massive scales, but there will be technology that simplifies the big data into a model that is consumable by analytic applications. In other words, it will transform the data to actually represent something that can be analyzed.
Big Transaction Data
Big Transaction Data (BTD) is a great example of this. It is complete, comprehensive and correlated. But it’s also usable. Let’s have a quick primer on BTD.
What it is, effectively, is the data generated by transactional systems in raw form modeled to represent the unique end-to-end transaction that drove the data generation in the first place, and stored alongside millions, billions, trillions (insert your own “illion” here) of other transactions. This is done by technology – typically business transaction management software that observes and reports on transaction performance at each tier.
This is REAL big data in action. And that’s where business transaction data comes into play. BTD takes the data and stores it in a consumable form for analytics. The transaction becomes the anchor for the analytics process.
The Problem with Fragmented Data
For example, say you wanted to analyze the end to end process performance of a financial trade system. The systems that execute financial trades are ridiculously complex. Think of the most complex system you can think of and then multiply it by 3. Why? Because they are using a mix of new and old technologies and it’s distributed across multiple tiers and managed by many different stakeholders. So what you get his this hodgepodge of tiers to execute trades that is incredibly difficult to rationalize into a singular data set. The unfortunate by-product of this is that your view of the trade transaction is really just fragmented data. You can see pieces of the transaction performance but not really ALL of the transaction.
But, you still need to analyze trades across the tiers and processes as a single input into your trade effectiveness analysis. So you do the best you can. You go deep into the tier data and try to correlate it on your own within your own analytic model. For example, you try to monitor cross-process fallout with a cool looking dashboard that gives you data on each process, but you don’t really do it well and miss a lot of cross-process issues.
Or you try to do a cost analysis. Or a segmentation analysis. Or a performance analysis. But the work to create a singular data set is so complicated that you never really have full confidence in the results.
Big Transaction Data in Action
Here is a great opportunity to employ big transaction data. Instead of working with billions of manually correlated data points, let’s simplify and work with millions of well-defined transactions instead. End-to-end transactions that represent each trade across each process in full. Now you have a data set that you can inject it into your BI platform or use simply use BI tools within the big transaction data solution itself for analysis.
So back to those 3 Cs. The data is complete – that means all information is generated by BTM end to end one view. It’s comprehensive – capturing ALL interactions. And, it’s correlated – it knows everything about vital meta-data such as user, tiers, etc. The result is easy to consume meaningful analytics leading to business outcomes.
So, big data is hot. But it’s not quite there yet. We’re waking up with more data but we’re still working to rationalize it. Fortunately, the technology is on its way to simplify and gain more (true) value from big data.
Posted by david ricketts 
