Perceptive Information Strategies

With margins for transactional operations getting thinner, organizations in many industries are focused on leveraging more value from their data. This could not be truer than in the financial services industry, where the onrushing spread of algorithmic trading is changing…well, everything, including the role of information in guiding trading and investment decisions. This rather unnerving article in Wired by Felix Salmon and Jon Stokes captures what’s happening quite well. The speed with which organizations need to turn data into both better business insight and marketable data services has many looking closely at data virtualization.

As an analyst, I am fortunate to be a (remote) participant in the Boulder BI Brain Trust (BBBT), the Boulder, Colorado-based brainchild of Claudia Imhoff. A couple of times a month, this illustrious group of experts gathers for half-day briefings with vendors in the business intelligence, analytics and data warehousing space; the briefings are always highly informative (on Twitter, watch for the hash tag #BBBT). On Friday, January 14, the topic was data virtualization: the BBBT met with Composite Software, a leading vendor of tools and solutions for data virtualization. Composite brought along a customer – and no small customer, either – NYSE Euronext. Emile Werr, NYSE Euronext’s VP of Global Data Services and head of Enterprise Data Architecture gave us a briefing on how the company is developing a data virtualization layer using Composite’s products.

Wikipedia has a good definition of data virtualization: “to integrate data from multiple, disparate sources – anywhere across the extended enterprise – in a unified, logically virtualized manner for consumption by nearly any front-end business solution, including portals, reports, applications, search and more.” As the Wiki entry notes, data virtualization (or “data federation”) is an alternative to data consolidation or replication into data warehouses and data marts. When this concept was first introduced, it was the cause of fiery debates at TDWI events and elsewhere. Now, it has settled in as a complement to the other more entrenched approaches.

Virtualization helps when organizations can’t really wait for standard data integration and loading into a data warehouse. That is very much the challenge facing NYSE Euronext, which is using Composite tools to develop a virtual layer to improve data access for internal executives and to establish a platform for creating data services. “We have so many companies trying to connect into us, and we want to serve standardized information out to companies around the world,” Werr said. NYSE Euronext is moving away from its old method of dumping transaction data into a warehouse; it wants to put more intelligence into the virtual layer. And to help build this layer, it is hiring people with business skills who understand processes and how to derive business value from data. “[These professionals] are the most productive people on my team right now,” he said.

The BBBT session featured an interesting debate about how data governance fits with data virtualization. Can data quality and governance rules be managed and invoked from the virtual layer? Should they be managed at the source or as part of extract, transformation and loading (ETL) processes, as many organizations do now? The discussion began to turn toward master data management and the option of creating a central hub or registry to implement governance for access to multiple sources. Highly regulated industries such as financial services and healthcare should consider this approach because of the need to invoke regulatory provisions for data access and sharing. Werr discussed these requirements and how his organization hopes to use the Composite virtual layer to support metadata governance and access from multiple BI tools.

Putting intelligence into a virtual layer fits with the IT infrastructure trend toward virtualization and cloud computing, and may become even more important because of this trend. Service-oriented applications running on cloud and virtual platforms frequently require access to multiple, disparate data sources. From a business standpoint, data virtualization is going to be critical to moving data quickly from the back office outward, to where it can be packaged into “information-as-a-service” offerings that customers will buy – and that will improve the seller’s profit margins.

Auld Lang Syne has been sung, but since it is still January, there remains time for New Year’s resolutions. Losing weight, sleeping more, eating vegetables and working less on weekends are perennial favorites; to those I now add “blogging more often.” For some, this resolution would be a cinch. Not for me. In 2010, with one project due after another, I struggled to carve out time to blog. I hope to correct that in 2011, even as I anticipate a new year full of interesting and demanding projects.

In competition with writing one single blog, of course, are all the social media options that have many of us splayed across the Web, including Facebook and Twitter. These need care and feeding, too. In my business, despite its high inanity quotient, Twitter has become essential for communicating news briefs and quick-take analyses. Facebook is many things to many people, but for me it is just a “fun” forum for sharing observations and artifacts along life’s journey. Maybe someday it will be more. Can’t forget LinkedIn. Finally, the social network experience has to include commenting on other people’s blogs, at major news source sites, on Yelp, on Amazon and so on. Have to give the text analytics engines something to chew on!

Most industry pundits have already published their predictions and prognostications for 2011. Rather than add to the pile, I would like to offer a few quick information management “resolutions”: priorities that I believe will shape what happens in 2011.

Integrate the integration. In 2010, the business value of information integration hit home to many organizations. Improved integration can lower the cost of information management and help eliminate downstream business problems caused by poor data quality and inconsistency. Yet, across enterprise departments and business functions there are usually numerous data, application and process information integration steps. With vendors such as IBM, Informatica, Kalido, Oracle and Talend beginning to provide better tools for developing and governing the use of data through master data management and “semantic” metadata layers, organizations have the opportunity to work toward comprehensive, end-to-end visions of information integration.

Don’t be blinded by automated analytics: The good news coming out of 2010 is that advanced analytics involving large (ok, “big”) data sources are becoming mainstream. More organizations than ever before will be able to afford analytics, especially as they deploy data appliances and use services, templates and other tools to shortcut the development of models, variable selection and other steps that are difficult and time-consuming. However, organizations need to “keep it real”: this is important stuff, involving critical decisions about customers, patients, pricing, demand chains, fraud prevention and other factors that are differentiators. Despite the hype, automated analytics are not entirely ready to replace wetware, gut feel or moments of irrational inspiration.

Respect “keeping the lights on.” It’s fashionable these days to dismiss non-strategic IT tasks as merely “keeping the lights on.” I found in 2010 that some of the most complicated and important decisions organizations are making these days have to do with IT infrastructure. Virtualization and cloud computing are completely remaking the map, which means that IT has to move to the next generation of tools and analysis of network optimization, application performance management, dependency mapping and more. Organizations need more sophisticated tools and analysis to make the right decisions about IT infrastructure.

Encourage vendor “coopetition.” The sensational story of Mark Hurd’s departure from HP and resurfacing at Oracle dominated headlines in 2010. The story is not over; InformationWeek’s Bob Evans offers an insightful blog about the continuing friction between HP and Oracle. In the wake of Oracle’s Sun acquisition, the two companies are in the midst of a tectonic shift away from what had been a longstanding partnership. Organizations should remind vendors such as HP and Oracle that despite competitive antagonism, they expect them to work together effectively on behalf of their interests. Customers have that kind of clout. Fasten your seatbelts, though, because I’m sure we’re in for more M&A activity, possibly involving HP and Oracle, which will further reshape the competitive landscape.

Be smart about being “dynamic.” A major watchword this year is “dynamic.” Cloud computing, virtualization, appliances, workload optimization, workforce optimization and other technologies are helping organizations flex to meet changing business requirements without the usual steps of adding people, software and hardware resources that sit idle when not needed. To be more just-in-time requires knowledge; otherwise, organizations could be caught short. In 2011, before going fully dynamic, organizations need to evaluate whether they have adequate knowledge about user and business requirements. If not, it may be time to evaluate tools and practices for understanding workloads, process requirements, dependencies and more. Old ways – and divisions between business and IT – have to change.

That’s all for now. Happy New Year to all!