FRONT PAGE STORY: The Information Layer Matters
By Joe McKendrick
eBay is a data giant. The
online auction provider currently stores more than two petabytes
of data - over 200 times the size of the Library of Congress. In addition, the
company rolls out more than 100,000 lines of new code each week.
To manage multiple data sources and applications, eBay has built what it calls a software-based “integration tier,” James Barrese, vice president of systems development at eBay, told DBTA. “This contains both a data access layer and a services framework,” he explained. “The integration tier acts as an abstraction layer for software engineers to work with many disparate back-end data sources through a consistent set of abstractions.”
eBay is not alone with the challenge of more effectively managing large reservoirs of data streaming into its organization through multiple channels. Cliff Longman, CTO for Kalido, told DBTA that the increasing data burden is the major driver behind interest in enterprise data management. “The data deluge that organizations are buried under is creating a heavy burden on the consumers of information,” he explained. “Information is often inconsistent and difficult to find, making the consumer, whether they be a partner or a customer or an employee, a victim of this data challenge. In addition, most systems require the user to know where the data is in order to find it, a daunting task for the business user.”
There was a time, about a decade ago, when “middleware” was considered passé. But in recent times, eBay has not been alone in finding it more profitable and manageable to abstract enterprise information from underlying systems. There is, however, little agreement as to what types of technologies, approaches or architecture should constitute what has been called the “information layer,” “integration tier,” “enterprise data management,” or, in some cases, an “enterprise data fabric.” Many industry watchers say service-oriented architecture (SOA) fills the bill, while others look to data extract, transfer and load (ETL) approaches as the purveyor of the information layer.
It’s also a term that has been heavily misused. In many cases, the information layer “is something that a lot of consultants have used to their advantage to mystify people about what’s out there,” observed Anthony Politano, partner of Business-Edge Solutions and author of Chief Performance Officer. “I’ve seen it referred to as everything from pure middleware, to just something sitting in between as a communication vehicle, to an integrated data layer, including operational data stores and data warehouses, to TIBCO.”
In an interview with DBTA, Politano defined the information layer “as a middle layer that removes or abstracts data - takes it away from its system of record and isolates it from there.” It doesn’t matter what the enabling architecture is, Politano noted, whether it’s “done through data movement, ETL, data warehousing, or data integration, or a combination of all of the above.”
Compelling Business Case
The business case for integrating key data sources into an information layer is compelling. "For most companies, isolated business process reengineering is no longer enough," said Ginni Rometty, senior vice president of IBM Enterprise Transformation Services. Business demands can be huge, Rometty said. She observed that in the
“Many companies are recognizing the need for consistent, shared information services that provide a comprehensive view of information while abstracting the complexity of source systems,”
said Michael Curry, Information Management SOA lead for IBM. “These services are designed and maintained by information specialists, ensuring accuracy and long-term viability. Master data management is a good starting point for this approach since these are the areas where consistency, accuracy, completeness, and reuse are at a
Curry cites an IBM survey of global CEOs, which found that “60 percent believe that their organizations need to do a better job of leveraging the information that they have in their environment. However, most companies don’t even understand what information they have, much less know how to use it. It’s just everywhere; it’s coming in through many different channels. They’re starting to put into place information architectures that allow them to deal with the complexity of their information layer in a consistent way.”
A recent survey of more than 800 enterprises conducted by Unisphere Research, the research unit of DBTA, for the Independent Oracle Users Group (IOUG), finds that middleware environments have now been almost universally adopted across enterprise database sites, with 89 percent of respondents reportedly working with one of the major brands of middleware or application servers. The question is how deeply these middleware environments are being used to support the information layer.
The tools and platforms are there, but deployments tend to be spotty,
Managing this information layer for consistency is another challenge, Ken Rugg, vice president of Progress Software, pointed out in an interview with DBTA. “The information layer, at a very basic level, is a way of providing consistent, up-to-date information across all of the functions of the business. It will vary. The level of consistency, the level of up-to-dateness, will vary as you get further from the center of where the ownership of the data lives. In many cases, that ownership is distributed, so you want to have a consistent view throughout the enterprise.”
To many industry watchers, SOA – in conjunction with strategies such as master data management – is opening the door to the first information layer truly detached from underlying systems. SOA may represent the key enabler for an information layer strategy going forward.
For example, at eBay, “SOA is part of the base infrastructure,” said Barrese. “We leverage both component-oriented and service-oriented architecture technologies. eBay has built a service architecture and uses it to enable integration across disparate technology stacks. Additionally, we see that by accelerating the integration of acquisitions such as PayPal, service-oriented approaches allow us to realize reduced cost and faster time to market. Services are part of eBay's architecture strategy.”
But companies’ approaches to developing information layers vary widely. “I am seeing two approaches – from the bottom up and from the top down,” Majid Abai, professor of Business Intelligence and Enterprise Data Architecture at UCLA and co-author of Data Strategy, told DBTA. “Some companies are utilizing Web services, and they are accessing one or two source systems utilizing these services, with plans to move the services to an enterprise level.”
According to Abai, “Other organizations are adopting a five-year type of plan, to combine the source systems and build all the common interfaces, capture data from the source system in a near real-time standpoint, put in a data layer, and then change the consumer systems to access the data layer - as opposed to going directly to the source system.”
A governance structure for managing the information flow also helps organizations get their arms around this challenge. “Given the emphasis on compliance and corporate transparency, organizations require a comprehensive understanding of what has occurred,” Kalido’s Longman explained. “Striving for one version of the truth has led companies to centralize an integrated copy of disparate data.”
Joe McKendrick is a contributing editor to Database Trends and Applications.