Data management in an organization is focused on getting data to its data consumers (whether human or application). Whereas the goal of data quality and data governance is trusted data, the goal of data integration is available data – getting data to the data consumers in the format that is right for them.
My new book on Data Integration has been published and is now available: “Managing Data in Motion: Data Integration Best Practice Techniques and Technologies”. Of course the first part of a book on data management techniques has to answer the question of why an organization should invest time and effort and money. The drivers for data integration solutions are very compelling.
Supporting Data Conversion
One very common need for data integration techniques is when copying or moving data from one application or data store to another, either when replacing an application in a portfolio or seeding the data needed for an additional application implementation. It is necessary to format the data as appropriate for the new application data store, both the technical format and the semantic business meaning of the data.
Managing the Complexity of Data Interfaces by Creating Data Hubs – MDM, Data Warehouses & Marts, Hub & Spoke
This, I think, is the most compelling reason for an organization to have an enterprise data integration strategy and architecture: hubs of data significantly simplify the problem of managing the data flowing between the applications in an organization. The number of potential interfaces between applications in an organization is an exponential function of the number of applications. Thus, an organization with one thousand applications could have as many as half a million interfaces, if all applications had to talk to all others. By using hubs of data, an organization brings the potential number of interfaces down to be just a linear function of the number of applications.
Master Data Management hubs are created to provide a central place for all applications in an organization to get its Master Data. Similarly, Data Warehouses and Data Marts enable an organization to have one place to obtain all the data they need for reporting and analysis.
Data hubs that are not visible to the human data consumers of the organization can be used to significantly simplify the natural complexity of data interfaces. If data being passed around in the organization is formatted, on leaving the application where it has been updated, into a common data format for that type of data, then applications updating data only need to reformat data into one format, instead of a different format for every application that needs it. Applications that need to receive the data that has been updated only need to reformat the data from the one common format into their own needs. This approach to data integration architecture is called using a “hub and spoke” approach. The structure of the common data format that all applications pass their data to and from is called the “canonical model.” Applications that want a certain kind of data need to “subscribe” to that data and applications that provide a certain kind of data are said to “publish” the data.
Integrating Vendor Packages with an Organization’s Application Portfolio
Current best practice is to buy vendor packages rather than developing custom applications, whenever possible. This exacerbates the data integration problem because each of these vendor packages will have their own master data that have to be integrated with the organization’s master data and they will either have to send or receive transactional data for consolidated reporting and analytics.
Sharing Data Among Applications and Organizations
Some data just naturally needs to flow between applications to support the operational processes of the organization. These days, that flow of data usually needs to be in a real time or near real time mode, and it makes sense to solve the requirements across the enterprise or across the applications that support the supply chain of data rather than developing independent solutions for each application.
Archiving Data
The life cycle for data may not match the life cycle for the application in which it resides. Some data may get in the way if retained in the active operational application and some data may need to be retained after an application is retired, even if the data is not being migrated to another application. All enterprises should have an enterprise archiving solution available where data can be housed and from which it can still be retrieved, even if the application from which it was taken no longer exists.
Moving data out of an application data store and restructuring it for an enterprise archiving solution is an important data integration function.
Leveraging External Available Data
There is so much data now available from government and other sites external to a company’s own, for free as well as data available for a fee. In order to leverage the value of what is available the external data needs to be made available to the data consumers who can use it, in an appropriate format. The amount of data now available is so vast and so fast that it may not be warranted to store or persist the external data, rather using techniques with data virtualization and streaming data, or not to store the data within the organization, choosing instead to leverage cloud solutions that are also external.
Integrating Structured and Unstructured Data
New tools and techniques allow analysis of unstructured data such as documents, web sites, social media feeds, audio, and video data. Greatest meaning can be applied to the analysis when it is possible to integrate together structured data (found in databases) and unstructured data types listed above. Data integration techniques and new technologies such as data virtualization servers enable the integration of structured and unstructured data.
Support Operational Intelligence and Management Decision Support
Using data integration to leverage big data includes not just mashing different types of data together for analysis, but being able to use data streams with that big data analysis to trigger alerts and even automated actions. Example use cases exist in every industry but some of the ones we’re all aware of include monitoring for credit card fraud as well as recommending products.
Drivers for Managing Data Integration – from Data Conversion to Big Data
April 25, 2013
Leave a Comment » | 
Uncategorized | Tagged: Big Data, Business Intelligence, Data Archiving, Data Conversion, Data Integration, Data Management, Data Warehousing, Master Data Management, Technology Strategy | 
Permalink
Posted by thoughtsondata
Is High Availability Sexy?
April 10, 2013The subject of business continuity has grown in appeal for me as my years in IT have grown, especially as I have personally experienced disasters big and small and the need to recover systems and facilities. I became particularly interested during my training as an IT auditor.
The area of business continuity isn’t necessarily a “sexy” part of data management, but it is a franchise requirement for most organizations and corporations and certainly critical for financial services organizations. Interestingly, the responsibility for business continuity is a business responsibility and yet the knowledge and training for how to implement it is a specialty within information technology (IT). I call this “the tail wagging the dog” because the responsibility cannot be delegated to IT and yet IT needs to lead the process of how to implement it.
The way we implement business continuity is using techniques in disaster recovery and high availability. Disaster recovery is how to bring back up systems and access after the loss of power, services, or access to a facility. High availability is a similar concept except maintaining system continuity by switching to alternative resources automatically at the loss of any resource, system, connection, facility, etc.
The rule of thumb with business continuity is that the lower the amount of time of any disruption at the loss of a resource, the higher the cost. Thus, a high availability solution that has no disruption has the highest cost. Organizations that require high availability solutions are therefore frequently spending millions of dollars on their disaster recovery solutions and millions more on their high availability solutions.
EMC has recently released a new high availability services product and is now asking the question “why invest in both disaster recovery and high availability?” http://www.emc.com/about/news/press/2013/20130212-01.htm
Maybe organizations that require high availability should put their business continuity budgets into that rather than dividing between both high availability and disaster recovery. Well, it may not be such a simplistic answer. Should every single application in the organization be set up with high availability? And yet, dividing systems between continuity solutions makes testing and effecting business continuity much more difficult. As long as the organization can prove they have a high availability solution for everything that would serve any necessary disaster recovery requirements.
OK. High availability isn’t sexy. But to me, it is slightly sexier than disaster recovery. Certainly it is time to consider whether it is more cost effective to put the entire business continuity budget into high availability.
Leave a Comment » | Uncategorized | Tagged: Business Continuity, Data Management, Technology Strategy | Permalink
Posted by thoughtsondata
Master Data in Big Data Management
April 3, 2013Currently, most data management activities are segregated by data type: documents are kept in one type of file repository, emails in another, structured data in databases, etc. One of the goals and values of big data management is being able to analyze data across these repositories, but if so then how do we link the data together? A big part of the answer, I believe, is master data. Master data is the data describing the important things in the organization: customers, products, employees, organizational structure, financial reporting structure, etc.
People with appropriate access in the organization should be able to view about a customer, for example, not only the customer’s name, addresses, and other demographic information, but emails to and from and concerning the customer, documents related to them, as well as audio recordings of any calls to customer service and video of the customer visiting the organization’s offices. All the organizational information about a customer can be made appropriately available to customer service to support a customer inquiry, to identify additional products with which the customer might be interested, or to predict likely future behavior.
Standard business intelligence tools can be used to find and connect information about a customer located in databases. Tools that search text can be used to find information related to a customer in document and email repositories either because these items contain text with the customer’s identifying information or because someone has tagged the documentation with the customer’s id. Similarly, audio and video files can be searched for the customer likeness or tagged manually with customer information. Links to a customer can be made at the time the information is stored or dynamically when a query is made about the customer. Tagging files and documents with customer identifiers can be performed automatically or manually. The ability to attach the customer information automatically is critical to big data management since the volume of data is usually beyond human manageable scale and we need to move away from the concept of manually crafted metadata.
And so, if the data in databases is called “structured” with keys associated with the master data in the organization, then we can integrate that data together with the “unstructured” data in files, documents, and emails by tagging the unstructured data with the key master data information, automatically and manually, at storage and at query time.
Leave a Comment » | Uncategorized | Tagged: Big Data, Data Governance, Data Management, Master Data Management, Metadata | Permalink
Posted by thoughtsondata
Big Data and NoSQL – the problem with relational databases
September 25, 2012The NoSQL movement, where “NoSQL” stands for “Not Only SQL” is based on the concept that relational databases are not the right database solution for all problems. Relational databases are so ubiquitous in most organization these days that many people may not even be aware that there are other types of databases, let alone when using another database might be preferable. Relational databases perform transaction update functions very well, particularly handling the difficult issues of consistency during update. Production strength relational databases can handle the complexity of two phase commit capability, where one business transaction affects multiple databases and tables, and all updates have to be effected at the same moment.
However, relational databases apply much of the same overhead required for complex update operations to every activity, and that can handicap them for other functions. Relational databases struggle with the efficiency of certain operations key to big data management. Firstly, they don’t scale well to very large sizes, and although grid solutions can help with this problem, the creation of new clusters on the grid is not dynamic and large data solutions become very expensive using relational databases. Secondly, they don’t do unstructured data search very well (i.e. google type searching) nor do they handle data in unexpected formats well. Thirdly, but not lastly, it is difficult to implement certain kinds of basic queries using SQL and relational databases, such as the shortest path between two points.
Social networking and big data organizations such as Facebook, Yahoo, Google, and Amazon were among the first to decide that relational databases were not good solutions for the volumes and types of data that they were dealing with, hence the development of the Hadoop file system, the MapReduce programming language, and associated databases such as Cassandra and HBase. One of the key capabilities of a Hadoop type environment is the ability to dynamically, or at least easily, expand the number of servers being used for data storage. The cost of storing large amounts of data in a relational database gets very expensive, where cost grows geometrically with the amount of data to be stored, reaching a limit in the petabyte range. The cost of storing data in a Hadoop solution grows linearly with the volume of data and there is no ultimate limit.
I was a working programmer before relational databases were in common use. Yes, we did have electricity back then. And the databases I used were of the type called “hierarchical”. In fact, they were more efficient, in general, for high volume individual transaction processing than relational databases, although like relational databases they were not good for data that was structured inconsistently. But what we considered “high volume” then could be handled reasonably by my laptop now and those databases couldn’t handle dynamically allocating unlimited additional space, either.
In my next blog post I’ll describe some of the new classes of NoSQL databases and what problems they solve well.
1 Comment | Uncategorized | Tagged: Big Data, Data Management | Permalink
Posted by thoughtsondata
Big Data Modeling – part 1 – Defining “Big Data” and “Data Modeling”
July 15, 2012Last month I participated in a DataVersity webinar on Big Data Modeling . There are a lot of definitions necessary in that discussion. What is meant by Big Data? What is meant by modeling? Does modeling mean entity-relationship modeling only or something broader?
The term “Big Data” implies an emphasis on high volumes of data. What constitutes big volumes for an organization seems to be dependent on the organization and its history. The Wikipedia definition of “Big Data” says that an organization’s data is “big” when it can’t be comfortably handled by on hand technology solutions. Since the current set of relational database software can comfortably handle terabytes of data and even desktop productivity software can comfortably handle gigabytes of data, “big” implies many terabytes at least.
However, the consensus on the definition of “Big Data” seems to be with the Gartner Group definition that says that “Big Data” implies large volume, variety, and velocity of data. Therefore, “Big Data” means not just data located in relational databases but files, documents, email, web traffic, audio, video, and social media, as well. The various types of data provides the “variety”, and not just data in an organization’s own data center but in the cloud and data from external sources as well as data on mobile devices.
The third aspect of “Big Data” is the velocity of data. The ubiquity of sensor and global position monitoring information means a vast amount of information available at an ever increasing rate from both internal and external sources. How quickly can this barrage of information be processed? How much of it needs to be retained and for how long?
What is “data modeling”? Most people seem to picture this activity as synonymous with “entity relationship modeling”. Is entity relationship modeling useful for purposes outside of relational database design? If modeling is the process of creating a simpler representation of something that does or might exist, we can use modeling for communicating information about something in a simpler way than presenting the thing itself. So modeling is used for communicating. Entity relationship modeling is useful to communicate information about the attributes of the data and the types of relationships allowed between the pieces of data. This seems like it might be useful to communicate ideas outside of just relational databases.
Data modeling is also used to design data structures at various levels of abstraction from conceptual to physical. When we differentiate between modeling and design, we are mostly just differentiating between logical design and design closer to the physical implementation of a database. So data modeling is also useful for design.
In the next part of this blog I’ll get back to the question of “Big Data Modeling.”
Leave a Comment » | Uncategorized | Tagged: Big Data, Data Integration, Data Management, Data Modeling, Data Virtualization | Permalink
Posted by thoughtsondata
People who Tweet about Data Management
April 30, 2012Data Management & Architecture
Karen Lopez @datachick
Neil Raden @NeilRaden
Robin Bloor @robinbloor
M. David Allen @mdavidallen
Sue Geuens @suegeuens
Mehmet Orun @DataMinstrel
Alec Sharp @alecsharp
Loretta Mahon Smith @silverdata
Eva Smith @datadeva
Corine Jasonius @DataGenie
Peter Aiken @paiken
Tony Shaw @tonyshaw
Glenn Thomas @Warduke
Bonnie O’Neil @bonnieoneil
Rob Paller @RobPaller
Pete Rivett @rivettp
Charles T. Betz @CharlesTBetz
Tracie Larsen @RelatedStuff
Wayne Eckerson @weckerson
Julian Keith Loren @jkloren
Christophe @mydatanews
Steve Francia @spf13
Gorm Braavig @gormb
Jim Finwick @jimfinwick
Alexej Freund @alexej_freund
Corinna Martinez @Futureatti
Data Quality
Jim Harris @ocdqblog – blog
David Loshin @davidloshin – blog
Rich Murnane @murnane
Daragh O Brien @daraghobrien
Jacqueline Roberts @JackieMRoberts
Steve Tuck @SteveTuck
Vish Agashe @VishAgashe
Julian Schwarzenbach @jschwa1
Henrik L. Sorensen @hlsdk
MDM and Data Governance
Jill Dyche @jilldyche – blog
Charles Blyth @charlesblyth
Steve Sarsfield @stevesarsfield – blog
Dan Power @dan_power
Philip Tyler @tylep0
Business Intelligence and Analytics
Marcus Borba @marcusborba
Tamara Dull @tamaradull
Claudia Imhoff @Claudia_Imhoff – blog
Scott Wallask @BI_expert
Peter Thomas @PeterJThomas – blog
Barney Finucane @bfinucane
Matt Winkleman @mattwinkleman
Stray_Cat @Stray_Cat
Brett2point0 @Brett2point0
Risk Management
Peter Went @Bank_Risk
Joshua Corman @joshcorman
Michael Rasmussen @GRCPundit
Nenshad Bardoliwalla @nenshad
Gary Byrne @GRCexpert
Helmut Schindlwick @Schindwick
Technology Companies and Data Organizations
Oracle @Oracle
DAMA international @DAMA_I
McKinsey on BT @mck_biztech
SmartData Collective @SmartDataCo
DataFlux InSight @Datafluxinsight
Gartner @Gartner_inc
TDWI @TDWI
Scientific Computing @SciCom
Wearecloud @wearecloud
CloudCamp @cloudcamp
Panorama Software @PanoramaSW
Data Hole @datahole
BI Knowledge Base @biknowledgebase
EnterpriseArchitects @enterprisearchitects
DataQualityPro.com @dataqualitypro
RSA Archer eGRC @ArcherGRC
Exobox @Exobox_Security
EA_Consultant @EA_Consultant
Cloudbook @cloudbook
ID Experts @idexperts
IAIDQ @iaidq
EMC Forum @EMCForums
Data Junkies @datajunkies
True Finance Data @truefinancedata
Madam @TheMDMNetwork
IBM Initiate @IBMInitiate
Accelus_GRC @PaisleyGRC
DQ Asia Pacific @DQAsiaPacific
Data Guide @DataGuide
PCI PA-DSS Data @DataAssurant
DataFlux Corporation @DataFlux
Leave a Comment » | Uncategorized | Tagged: Business Intelligence, Data Governance, Data Management, Data Quality, Data Security, Master Data Management | Permalink
Posted by thoughtsondata
Data Virtualization – Part 1 – Business Intelligence
March 26, 2012The big transformation that we’re all dealing with in technology today is virtualization. There are many aspects to virtualization: infrastructure, systems, organization, office, applications. When you search on the internet for “data virtualization,” most of the references are regarding business intelligence and data warehousing uses. In part 2 of this blog I will talk about data virtualization and transaction processing.
In the day, when I used to build data warehouses (1990s+), there was a reference to a concept of “federated data warehouses”, where data in the logical warehouse would be separated physically either with the same schemas in multiple instances or different types of data in different locations. The thought was that the data would be physically separate but brought together real time for reporting. We also used to call that “data warehouses that don’t work”. After all, the reason we created data warehouses in the first place was that we needed to instantiate the data consolidation in order to make the response time reasonable when trying to report on millions of records. No, really, the response time on these “federated data warehouse” systems used to be many minutes or more.
Now, however, the technologies involved have made huge leaps in capabilities. The vendors have put thousands and thousands of man hours into how to make real time integration and reporting work. There are many techniques involving specialized software and hardware (data appliances) that enable these capabilities, query optimization, distributed processing, and other optimization techniques, and hybrid solutions between pure virtualization and instantiation. Specialized tuning is necessary, and the fastest solutions involve instantiating the consolidated data in a central place.
Ultimately, having to do a project to incorporate new data into the data warehouse physically isn’t responsive enough to the business need for information. Better to have a short term solution that allows for the quick incorporation of new data and then, if there is a continued need for the data in question and you want to speed up the response, possibly integrate the additional data into the physical data warehouse.
The problems being solved now for business intelligence and data virtualizations include real time data integration of multiple regional instances of a data warehouse, integrating data of different types and kinds, and integrating data from a data warehouse with big data and cloud data. This enables much more responsive business intelligence and analytical solutions to business requests without having to always instantiate all data for analysis into a central, single, enterprise data warehouse.
Leave a Comment » | Uncategorized | Tagged: Business Intelligence, Data Management, Data Virtualization, Data Warehousing | Permalink
Posted by thoughtsondata
The End of the Data Life Cycle – Data Archiving and Removal
February 2, 2012What we don’t currently emphasize greatly in Data Management is the end of the data life cycle, when data is archived and, possibly later, deleted. This has been in the past because we’ve wanted to have available as much data as our technical solutions could store and if we couldn’t store all of it then usually the oldest data would just be deleted after backup. Now, in the era of Big Data there is exponential growth in the amount of data being produced and the ability to move data aside (archive it) and retrieve it back is well developed.
Ultimately, all data in an organization (structured and unstructured) needs to be tagged with Metadata that indicates when the data is to be archived, when deleted, and who is responsible for the data, including ultimately approving that data should be deleted.
Although Data Archiving itself is an interesting topic, it isn’t particularly sophisticated in regards to its data movement aspects since the data being archived is usually not transformed. Data Archiving is usually a topic under Data Backup and Recovery. Archiving data assumes that the data is moved to a less costly (and possibly less accessible) platform from which it can be retrieved or accessed in the future, either brought back to the original application for access or possibly accessed from the archive environment.
Regulatory requirements in Pharmaceutical firms require that data from an application being shut down must be archived along with the application and hardware on which it runs. This is a very smart approach to recoverability since a simple data backup is not recoverable if the hardware that it ran on is no longer available. Less strict regulation may simply require that the data itself be archived. Standard data archiving capabilities associated with data backup will move the data to offline storage, but bringing the data back into the operational application can be problematic if the structure or schema of the data in the application changes after the data is archived.
In cases where the archived data need not be brought back into the operational application environment but only needs to be accessible, it may be decided to transform the data being archived into a common technical format: It may be decided to have the data in the data archive environment all be stored in the same database management system, for example.
It is critical that data being archived be done so along with its associated metadata explaining the meaning of the data and its history. If archiving data that may have to be accessed independently of the application from which it came, it may be best to use a data structure or storage solution that can both keep the data with its metadata and allow flexible and changing data structures. An XML type of data structure would allow a changing data structure, associated metadata, and still allow queries across data that had been archived from the same application, even if the source data structures had changed.
Leave a Comment » | Uncategorized | Tagged: Data Archiving, Data Management, Metadata | Permalink
Posted by thoughtsondata
What I Learned About Data Management in Japan
December 6, 2011I just returned from speaking at the DAMA Japan conference. A couple of the officers saw a presentation I gave at the DAMA International conference last April on performing Data Management maturity capability assessments and they asked me to repeat the speech at their conference in Tokyo. The Data Management Association (DAMA dama.org) created a Body of Knowledge document in 2009 (DMBOK) that was translated into Japanese in 2010. This conference had a lot of focus on the DMBOK, and so they were very interested in how I had been using it to assess Data Management maturity for our client organizations. About 90 people attended my presentation. The people at the conference and the DAMA Japan organization were very interested in measuring how well their organizations were doing with Data Management, especially against how other organizations are doing.
One thing I learned during the conference was how they were using the “three R’s” concept: “reuse, recycle, reduce” concerning Data Management and adding a fourth “R” for “respect”. In organizational data it is generally agreed that 80% or more of an organization’s data is ROT: redundant, out-dated, or trivial. More of our planning time should be spent trying to reuse or remove data structures than on trying to create new data structures. They had added an extra “R” to “reuse, recycle, and reduce” to stand for “respect”, a very Japanese consideration and always worth attention.
Another thing that I learned is that some of my Japanese colleagues will take a day to have a brainstorming session on different IT and strategy concepts, which sessions might very well happen in a hot spring or bath house. Concerning Enterprise Architecture, they had recently had such a session and decided that it was critical that Enterprise Architecture include business innovation, employee motivation, and technology innovation. This summary was shared with me after we had all consumed a great deal of sake and other alcohol and they were very willing to try to share their ideas in English.
The people I met at the Data Management conference seemed not very interested in Data Governance but, as I said extremely interested in assessing Data Management practices. And it appears that adding a banquet or hot bath makes the discussion of data management and strategy even more insightful.
Leave a Comment » | Uncategorized | Tagged: Capability Assessment, Data Management | Permalink
Posted by thoughtsondata
The Problem With Point to Point Interfaces
November 21, 2011
The average corporate computing environment is comprised of hundreds to thousands of disparate and changing computer systems that have been built, purchased, and acquired. The data from these various systems needs to be integrated for reporting and analysis, shared for business transaction processing, and converted from one system format to another when old systems are replaced and new systems are acquired. Effectively managing the data passing between systems is a major challenge and concern for every Information Technology organization.
Most Data Management focus is around data stored in structures such as databases and files, and a much smaller focus on the data flowing between and around the data structures. Yet, because of the prevalence of purchasing rather than building application solutions, the management of the “data in motion” in organizations is rapidly becoming one of the main concerns for business and IT management. As additional systems are added into an organization’s portfolio the complexity of the interfaces between the systems grows dramatically, making management of those interfaces overwhelming.
Traditional interface development quickly leads to a level of complexity that is unmanageable. If there is one interface between every system in an application portfolio and “n” is the number of applications in the portfolio, then there will be approximately (n-1)2 / 2 interface connections. In practice, not every system needs to interface with every other, but there may be multiple interfaces between systems for different types of data or needs. This means for a manager of application systems that if they are managing 101 applications then there may be something like 5,000 interfaces. A portfolio of 1001 applications may provide 500,000 interfaces to manage. There are more manageable approaches to interface development than the traditional “point to point” data integration solutions that generate this type of complexity.
The use of a “hub and spoke” rather than “point to point” approach to interfaces changes the level of complexity of managing interfaces from exponential to linear. The basic idea is to create a central data hub. Instead of the need to translate from each system to every other system in the portfolio, interfaces only need to translate from the source system to the hub and then from the hub to the target system. When a new system is added to the portfolio it is only necessary to add translations from the new system to the hub and from the hub back to the new system. Translations to all the other systems already exist. This architectural technique to interface design makes a substantial difference to the complexity of managing an IT systems portfolio, and yet it had nothing really to do with introducing a new technology.
Leave a Comment » | Uncategorized | Tagged: Data Integration, Data Management, Technology Strategy | Permalink
Posted by thoughtsondata
thoughtsondata 