Database Research At Yale

HadoopDB is:

  1. A hybrid of DBMS and MapReduce technologies targeting analytical query workloads
  2. Designed to run on a shared-nothing cluster of commodity machines, or in the cloud
  3. An attempt to fill the gap in the market for a free and open source parallel DBMS
  4. Much more scalable than currently available parallel database systems and DBMS/MapReduce hybrid systems
  5. As scalable as Hadoop, while achieving superior performance on structured data analysis workloads
This projects builds on our paper in VLDB 2009.


The goal of the Semantic Web vision is to free Web data from the applications that control them, so that data can be easily described and exchanged. This is accomplished by supplementing natural language and other data found on the Web with machine readable metadata in statement form (e.g., X is-a person, X has-name ``Joe'', X has-age ``35'') and enabling descriptions of data ontologies so that data from different applications can be integrated through ontology mapping. One ultimate goal is to turn the Web into a giant database, against which one could issue structured queries and receive structured answers in response.

SW-Store is a recently launched project whose goal is to manage and query Semantic Web data. We are starting from a clean-slate and designing a DBMS specifically for this type of data and the prevalent Semantic Web data model, the Resource Description Framework, or RDF. We explore how common SW queries and applications such as reasoning and biological data integration can be built into the database. This work builds on a recent publication that won "Best Paper" at VLDB.

H-Store: A High-Performance OLTP Database

Current OLTP database designs, which date largely from the 1970s, are based on several assumptions about the architecture of database applications and hardware that are less true today than they were 30 years ago. For example, all but the very largest OLTP applications can fit in main memory of a modern shared-nothing cluster of server machines. On a single node with a memory resident database, OLTP transactions take only a few microseconds to execute. Additionally, many applications carefully construct database transactions so they have no user stalls. Taken together, both of these points mean there is a large class of OLTP applications for which a single-threaded execution engine with no concurrency control performs very well, avoiding the need for high overhead, locking-based pessimistic concurrency control protocols designed to keep CPUs busy during disk and user stalls. Further, the cost of computers has dropped so dramatically in the past thirty years that paying for a dedicated database administrator has become one of the dominant costs in running a database system, such that tools that automate design and tuning have great value. Finally, the architecture of a server node has also shifted -- the number of cores available to process data is proliferating. The goal of the H-Store project is to investigate how these architectural and application shifts affect the performance of OLTP databases, and to study what performance benefits would be possible with a complete redesign of OLTP systems in light of these trends. Our early results show that a simple prototype built from scratch using modern assumptions can outperform current commercial DBMS offerings by around a factor of 80 on OLTP workloads. We are currently working to build a full-featured system that demonstrates these performance wins in a more robust prototype.


As companies increasingly use analytic data marts and data warehouses for their customer relationship management and business intelligence applications, the use of column-oriented DBMS technology is growing. Column-oriented databases store DBMS tables column-by-column (instead of row-by-row) and tend to perform better on analytical applications since these applications tend to only focus on a subset of table attributes at a time, and are thus more I/O efficient. Examples of these types of analytic applications are:

  • An application that evaluates the best offer to give a customer while they are on the phone with a call center
  • An application that looks for correlations in products that customers buy in the same transaction
  • An application that looks at customers' history to evaluate credit risk.

Due to the increasing popularity of column-stores, a number of recent venture capital backed start-up companies have formed in recent years that are built on this technology, including Vertica, ParAccel, and Calpont in addition to the increasing popularity of column-oriented databases that have been around a little longer (such as Sybase IQ, Sand/DNA Analytics, and SenSage). Column-stores have also recently seen great momentum in the research community with a number of recent publications. The C-Store project has built an academic prototype of a column-oriented database and this prototype has lead to a great deal of important research exploring the architectural design differences between row-oriented databases and column-oriented database. There are still many important questions remaining unanswered; however, early performance results are very encouraging, with data warehouse queries consistently running one to two orders of magnitude faster. This project is collaboration between Yale, MIT, Brown, Brandeis, and UMass Boston.