FRONT PAGE STORY: The Information Layer Matters

July 2006

By Joe McKendrick

eBay is a data giant. The online auction provider currently stores more than two petabytes of data - over 200 times the size of the Library of Congress. In addition, the company rolls out more than 100,000 lines of new code each week.
      To manage multiple data sources and applications, eBay has built what it calls a software-based “integration tier,” James Barrese, vice president of systems development at eBay, told DBTA. “This contains both a data access layer and a services framework,” he explained. “The integration tier acts as an abstraction layer for software engineers to work with many disparate back-end data sources through a consistent set of abstractions.”
      eBay is not alone with the challenge of more effectively managing large reservoirs of data streaming into its organization through multiple channels. Cliff Longman, CTO for Kalido, told DBTA that the increasing data burden is the major driver behind interest in enterprise data management. “The data deluge that organizations are buried under is creating a heavy burden on the consumers of information,” he explained. “Information is often inconsistent and difficult to find, making the consumer, whether they be a partner or a customer or an employee, a victim of this data challenge. In addition, most systems require the user to know where the data is in order to find it, a daunting task for the business user.”
      There was a time, about a decade ago, when “middleware” was considered passé. But in recent times, eBay has not been alone in finding it more profitable and manageable to abstract enterprise information from underlying systems. There is, however, little agreement as to what types of technologies, approaches or architecture should constitute what has been called the “information layer,” “integration tier,” “enterprise data management,” or, in some cases, an “enterprise data fabric.” Many industry watchers say service-oriented architecture (SOA) fills the bill, while others look to data extract, transfer and load (ETL) approaches as the purveyor of the information layer.
      It’s also a term that has been heavily misused. In many cases, the information layer “is something that a lot of consultants have used to their advantage to mystify people about what’s out there,” observed Anthony Politano, partner of Business-Edge Solutions and author of Chief Performance Officer. “I’ve seen it referred to as everything from pure middleware, to just something sitting in between as a communication vehicle, to an integrated data layer, including operational data stores and data warehouses, to TIBCO.”
       In an interview with DBTA, Politano defined the information layer “as a middle layer that removes or abstracts data - takes it away from its system of record and isolates it from there.” It doesn’t matter what the enabling architecture is, Politano noted, whether it’s “done through data movement, ETL, data warehousing, or data integration, or a combination of all of the above.”

Compelling Business Case
The business case for integrating key data sources into an information layer is compelling. "For most companies, isolated business process reengineering is no longer enough," said Ginni Rometty, senior vice president of IBM Enterprise Transformation Services. Business demands can be huge, Rometty said. She observed that in the U.S. alone, $14 billion is spent on warranties, but the data needed to manage these agreements is spread across call centers and customer service records. “It may take up to 10 days for a company to pull all those records together,” she said.
      “Many companies are recognizing the need for consistent, shared information services that provide a comprehensive view of information while abstracting the complexity of source systems,”
said Michael Curry, Information Management SOA lead for IBM. “These services are designed and maintained by information specialists, ensuring accuracy and long-term viability. Master data management is a good starting point for this approach since these are the areas where consistency, accuracy, completeness, and reuse are at a
      Curry cites an IBM survey of global CEOs, which found that “60 percent believe that their organizations need to do a better job of leveraging the information that they have in their environment. However, most companies don’t even understand what information they have, much less know how to use it. It’s just everywhere; it’s coming in through many different channels. They’re starting to put into place information architectures that allow them to deal with the complexity of their information layer in a consistent way.”
      A recent survey of more than 800 enterprises conducted by Unisphere Research, the research unit of DBTA, for the Independent Oracle Users Group (IOUG), finds that middleware environments have now been almost universally adopted across enterprise database sites, with 89 percent of respondents reportedly working with one of the major brands of middleware or application servers. The question is how deeply these middleware environments are being used to support the information layer.
      The tools and platforms are there, but deployments tend to be spotty, Ari Kaplan, president of the IOUG, told DBTA. “It really depends on what the companies are trying to do,” he said. “Sometimes it makes sense to keep information purely in the database, other times in the middle layer or the thick client side. Even if you do have the technology that makes it compelling, a lot of companies have an if-it’s-not-broke-don’t-fix-it mentality. It takes energy and effort and testing to migrate to a different area.”
      Managing this information layer for consistency is another challenge, Ken Rugg, vice president of Progress Software, pointed out in an interview with DBTA. “The information layer, at a very basic level, is a way of providing consistent, up-to-date information across all of the functions of the business. It will vary. The level of consistency, the level of up-to-dateness, will vary as you get further from the center of where the ownership of the data lives. In many cases, that ownership is distributed, so you want to have a consistent view throughout the enterprise.”

New Possibilities
To many industry watchers, SOA – in conjunction with strategies such as master data management – is opening the door to the first information layer truly detached from underlying systems. SOA may represent the key enabler for an information layer strategy going forward. Enterprise information management enables organizations to get at the right information at the right time. Likewise, SOA is about getting at the right information at the right time. SOA enables the integration of disparate data and application silos across the enterprise.
      For example, at eBay, “SOA is part of the base infrastructure,” said Barrese. “We leverage both component-oriented and service-oriented architecture technologies. eBay has built a service architecture and uses it to enable integration across disparate technology stacks. Additionally, we see that by accelerating the integration of acquisitions such as PayPal, service-oriented approaches allow us to realize reduced cost and faster time to market. Services are part of eBay's architecture strategy.”
But companies’ approaches to developing information layers vary widely. “I am seeing two approaches – from the bottom up and from the top down,” Majid Abai, professor of Business Intelligence and Enterprise Data Architecture at UCLA and co-author of Data Strategy, told DBTA. “Some companies are utilizing Web services, and they are accessing one or two source systems utilizing these services, with plans to move the services to an enterprise level.”
      According to Abai, “Other organizations are adopting a five-year type of plan, to combine the source systems and build all the common interfaces, capture data from the source system in a near real-time standpoint, put in a data layer, and then change the consumer systems to access the data layer - as opposed to going directly to the source system.”
      A governance structure for managing the information flow also helps organizations get their arms around this challenge. “Given the emphasis on compliance and corporate transparency, organizations require a comprehensive understanding of what has occurred,” Kalido’s Longman explained. “Striving for one version of the truth has led companies to centralize an integrated copy of disparate data.”

Joe McKendrick is a contributing editor to Database Trends and Applications.

Back to Ari Kaplan's Home Page