A Quick Look Into the Popular Graph Databases

Purpose

The 3Pillar Recommendation Engine currently uses MongoDB to store the processed recommendations for all the users in the system. We looked into graph databases because its storing mechanism of nodes and relations directly maps to the way of the recommendation engine data model. This results in a low storage footprint and also provides the capability of greater insights into the data. To ensure transparency and flexibility, we specifically looked into open source projects that are well-supported by the community.

Neo4j

Neo4j is the market leader in GraphDB. It is highly performant, scalable, and flexible. It is equipped with a rich UI, called the Neo4j browser, that is used for queries, visualization, and data interaction. It has compatibility with most programming platforms like Java, Nodejs, PHP, Python, .NET, etc. It supports the Cypher query language, whose syntax naturally depicts data and relationships. It has a good integration support with Apache Spark, Elasticsearch, MongoDB, Cassandra, and Docker. The community edition is free under GPL v3 license.

Orient DB

Orient DB is a high speed, scalable, and reliable GraphDB. It has a high and constant traversing speed that is not affected by database size. User, role, and record level security is built-in. It has a good compatibility with most programming platforms like Java, Nodejs, PHP, Python, .NET, etc. It has a strong community support and is licensed under Apache 2.

Titan DB

Titan DB is highly scalable and can store and query graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster. Titan DB is transactional and supports various backend stores like Cassandra, HBase, and BerkeleyDB. It supports integration with Elastic Search, Lucene, and Solr, and also supports the Gremlin query language. It is licensed under Apache 2.

Apache Giraph

It is an Iterative Graph processing framework backed by Apache Hadoop. Giraph jobs are highly scalable as they run on Hadoop Cluster. Giraph can greatly increase the efficiency of graph computation on a huge dataset. It is licensed under Apache 2. We wanted to take a deep dive into these graph databases to understand how they stacked against each other. At the outset, we eliminated TitanDB because a stable 1.0 version had not been released at the time. We also skipped Apache Giraph because it is backed up only by Hadoop. This left Neo4j and OrientDB for more exploration.

Test Environment

• CPU- Intel quad core i7 2.2 GHz
• RAM – 16 GB
• Hard disk type – Flash
• Database on same machine as test program (no network latency)

Test Methodology

We used the sample Spring projects and Java drivers of each of the databases. The intent is to programmatically

• Add 100k users,
• Randomly search users across 100k edges in total, and
• Come up with our average read/write speeds.

We then searched users with similar preferences to each other. We ran the suite as Java unit-tests and captured the times using timestamps. Neo4j allowed indexing capability on one of the attributes – we indexed the userID to give us indexed and unindexed speeds. Read queries were not identical in each DB, so the closest comparison was run.

Results

The results scale from 1 to 5, where the higher value is better.

*White paper: https://www.stingergraph.com/data/uploads/papers/ppaa2014.pdf

Conclusion

Overall, we agree with the market sentiment that Neo4j is the current leader in this space, specifically for recommendation engines. OrientDB is better suited for use cases like data mining and data extractions. However, its license terms make it somewhat difficult to work with. Although the community edition is licensed under GPL v3, the locking mechanism in case of concurrency is slow, making it unsuitable for high concurrency scenarios. The enterprise edition improves upon this, but the last time we checked, the pricing is quite steep! Apache Giraph and TitanDB are really promising projects and they may overshadow Neo4j some day. Our take is any graph database that can work over Apache Mesos would be the ultimate winner – will it be Apache Spark?