In August I attended the NoSQL Conference in San Jose, California. This conference was about products and solutions that, primarily, don’t use relational databases. The recent rise of interest comes from the Big Data space and includes areas that aren’t necessarily too big for relational databases, but that just don’t lend themselves to relational database solutions. Relational databases became the ubiquitous storage solution for data in application systems around the early to late 1990s. However, I was a working programmer for more than 10 years before that and so I’ve worked with hierarchical databases and indexed file solutions, among other things. In the late 1990s I had some very good experiences working with multidimensional databases for data marts, which are also not relational. One of the keynotes for the NoSQL conference was from Robin Bloor on all the terrible things about relational databases and how it could be done better. Every sentence out of his mouth was controversial and thought provoking.
The main question in my mind during the conference was “how are these NoSQL technologies and products useful to my customers? For a large organization with a well established data management portfolio that is based on relational databases, what business problems would be better solved with something else?”
The Big Data technology movement was started around the Hadoop file system and Map Reduce applications to solve problems of searching web data. This technology solution is used by many web (Google, Yahoo, Amazon) and social media companies to manage and search vast amounts of data across multiple servers. It introduces solutions for storing and searching unstructured data cheaply distributed across many servers. How might Hadoop and Map Reduce be of interest to traditional Data Management organizations? It introduces search of unstructured data, distributed processing, and possibly Cloud technology (if the distributed servers are in the Cloud). This gets into the idea of being able to search through vast amounts of organizational data that might previously have seemed too trivial to spend money to store or too expensive to search. There are quite a few Business Intelligence solutions that don’t use relational database technology or which combine it with other databases and technologies. The most interesting aspect from a technology perspective is the move to distributed processing engines.
A problem area that doesn’t seem to lend itself to relational databases is, ironically, understanding how two people or things are related to one another. Examples of this problem include analyzing the degrees of separation of two participants in a social network or understanding the relationship between a suspected terrorist and someone who calls them on the telephone. These types of problems are better solved using a graph or node database with a recursive analytical engine. By the way, when was the last time you wrote a recursive program? Better get out your Knuth algorithms book.
Multidimensional databases pre-calculate all (or most) summaries along multiple hierarchies or taxonomies such as customer, product, organizational structure, or accounting bucket. They are blindingly FAST for responding to queries, but not dynamic and require the step where all the calculations are done after the data is loaded. Applications for these solutions are data mart cubes. My experience has been particularly good supporting the Finance organization analytical needs. The capabilities of these databases can be mimicked in a relational database using Kimball’s dimensional modeling and summary tables.
XML databases deal well with the problem of storing and searching data where the structure of the data may be unknown in advance. Applications around data messages and documents do well using XML database solutions.
Another day I’ll blog about some of the inherent problems of relational databases with speed and volumes, but the main point is that organizations are finding it worthwhile to expand their database solutions beyond just relational database management systems.
thoughtsondata 