There was a time when Quicken Loans’ business intelligence (BI) department could talk without exaggeration about the more than 400 static reports it built for its users.
Even then, the line of business complained about a lack of visibility: Canned reports limited business users’ ability to explore the data the way they wanted to, said Steve Brennan, director of BI for the Detroit mortgage lender and a speaker at The Data Warehousing Institute’s BI Executive Summit in February.
“It does hurt to hear that there’s no visibility,” he said during his presentation. “But it’s not shame on them; it’s shame on us.”
In 2007, Brennan and his team embarked on a project to build out the company’s centralized data warehouse. He focused on solving the visibility issue, which included building self-service dashboards, introducing data extraction tools and implementing training sessions.
But complaints didn’t stem from how the data could be viewed alone; the line of business also criticized what data could be viewed and when. To provide that kind of visibility, Quicken Loans delved into the world known as real-time BI.
Upsetting the apple cart
In the early days of data warehousing, technology was cruder than it is today. Businesses hit it big if they could load data on a daily basis, said Nancy Williams, BI and analytics consultant for DecisionPath Consulting in Gaithersburg, Md. Because businesses were limited by storage, they had to purge data frequently or risk degrading the database.
More on real-time BI
Operational BI gets boost from real-time data loading
Real-time BI capabilities offer benefits, add complexities for users
Co-op embraces real-time analytics with Web-based BI platform
“When we first started, data warehousing and BI were all about history and longer views of information,” Williams said. “The concept of real time kind of upset the apple cart.”
Today data can be loaded and stored almost instantaneously, dramatically reducing the latency -- the time between when an event is recorded and when that information is made available. Businesses these days are seeking ways to take advantage of these advances.
One way businesses can do this is by real-time enabling their data warehouses, according to Wayne Eckerson, BI consultant and research director for TechTarget Inc.’s business applications and architecture media group.
While designed to load data in batches, businesses can integrate them with messaging buses, or they can use change data capture, Eckerson said in a recent video interview.
“You can also increase the speed at which you do batch uploads,” he said. “Some people call these micro-batches or mini-batches.”
Eckerson noted that this approach means a company’s data will reside in one place, much like it does within Quicken Loans’ centralized data warehouse; Williams, though, sees another advantage.
“Real-time data warehousing also includes the ability to integrate data across the organization,” she said. “Two or three different events that happened [in different systems] could be pulled together and seen quickly.”
But a real-time environment can also be constructed outside of the data warehouse. While businesses can do this using different methods, Eckerson said complex event processing, also called stream-based processing, is a popular one.
“Typically, you’re feeding these systems large volumes of discrete events, sometimes tens of thousands per second,” he said. “These systems can run queries of the data as it’s flowing through the system and trigger alerts.”
Either approach may raise questions about data quality, but it shouldn’t, said David Loshin, president of the consultancy Knowledge Integrity Inc., in Silver Spring, Md.
“If businesses are concerned about the quality of the data, that’s pervasive and should be addressed at a systemic level,” he said.
Real-time BI vs. right-time BI
The concept of real-time BI is still spreading confusion, and part of the reason is semantics.
“Real time has a discrete definition,” Loshin said. “Real-time processing means there’s a time frame in which the transaction or the task has to be completed.”
Eckerson called the term misunderstood, and technologists find the label problematic as well.
“Once data theoretically leaves the system that it’s happening in, real time is out the window,” said Quicken Loans’ Brennan.
The debate has cleared the way for two additional industry terms -- near real time and right time. But beyond semantics lies another hurdle: Real-time BI is neither easy nor cheap.
“You have to determine if there’s a value proposition,” Loshin said. “Not every business needs to change on a minute-by-minute basis.”
For Quicken Loans, which operates its business almost entirely online and built its system almost entirely in-house, real-time BI helps the company monitor events. In some cases, monitoring happens on the front lines. In other cases, monitoring happens on the back end.
“One of the coolest things that’s come out of our engineering efforts is data lag monitoring,” said Brennan, adding that if the data isn’t updated regularly, engineers are alerted. “It’s been huge in us proactively chasing down data issues.”
But before going down the real-time BI road, Brennan advised that businesses build a technology vision, and the consultants agreed.
“You should have a BI strategy that’s closely aligned with your business strategy,” Williams said. “If you do that, think about how the information is needed, who needs it and how quickly they need it.”
Loshin also advised that before implementing new real-time BI technology, businesses consider the implications it will have for the teams that will be using it and the business processes.
“Just because I’m able to acquire, analyze and deliver results faster, if the people who get those results don’t know what to do with it, then it’s all just an expensive exercise,” he said.
This was first published in July 2012