Solr 6.0 and Graph Traversal Support

Datetime:2016-08-23 02:02:32          Topic: Solr           Share

One of the new features that are present in the recently released Solr 6.0 is the graph traversal query that allows us to work with graphs. Having a root set and relations between documents (like parent identifier of the document) we can use a single query to get multiple levels of joins in the same request. Let’s look at this new feature working both in old fashioned Solr master/slave as well as in SolrCloud.

For the purpose of this blog post we will use a very simple data set that we can index using a single command and a configuration that can be downloaded from our github account available at: https://github.com/solrpl/blog .

Creating the Collection and Indexing the Data

What we need first is to create the collection and index the data itself. We will start Solr using the following command:

bin/solr start -c

This will launch our Solr instance in cloud mode. Now we need to send our configuration files to Zookeeper which we will do by running the following command:

bin/solr zk -upconfig -n graph_test_config -z localhost:9983 -d graph_test/conf

Next we will create our graph collection by running the following command:

curl -XGET 'http://localhost:8983/solr/admin/collections?action=CREATE&name=graph&numShards=2&replicationFactor=1&maxShardsPerNode=2&collection.configName=graph_test_config'

Now, after collection creation, we can finally index the data by running the following command:

curl -XPOST -H 'Content-Type: application/json' 'http://localhost:8983/solr/graph/update' --data-binary '{
 "add" : { "doc" : { "id" : "1", "name" : "Root document one" } },
 "add" : { "doc" : { "id" : "2", "name" : "Root document two" } },
 "add" : { "doc" : { "id" : "3", "name" : "Root document three" } },
 "add" : { "doc" : { "id" : "11", "parent_id" : "1", "name" : "First level document 1, child one" } },
 "add" : { "doc" : { "id" : "12", "parent_id" : "1", "name" : "First level document 1, child two" } },
 "add" : { "doc" : { "id" : "13", "parent_id" : "1", "name" : "First level document 1, child three" } },
 "add" : { "doc" : { "id" : "21", "parent_id" : "2", "name" : "First level document 2, child one" } },
 "add" : { "doc" : { "id" : "22", "parent_id" : "2", "name" : "First level document 2, child two" } },
 "add" : { "doc" : { "id" : "121", "parent_id" : "12", "name" : "Second level document 12, child one" } },
 "add" : { "doc" : { "id" : "122", "parent_id" : "12", "name" : "Second level document 12, child two" } },
 "add" : { "doc" : { "id" : "131", "parent_id" : "13", "name" : "Second level document 13, child three" } },
 "commit" : {}
}'

So our data has the following relations:

Let’s try searching in the structure.

Basic Graph Traversal Query Usage

In the basic form of graph traversal query, we need to provide a root set of documents, specify which is the identifier field, which is the parent identifier field, and run the query. For example, if we would like to find all the documents in relation to both the root documents and the ones with the next levels of relations we could run the following query:

http://localhost:8983/solr/graph/select?q=*:*&fq={!graph from=parent_id to=id}name:"root document"

The documents returned for such a query would look as follows:

<?xml version="1.0" encoding="UTF-8"?>
<response>
 <lst name="responseHeader">
  <bool name="zkConnected">true</bool>
  <int name="status">0</int>
  <int name="QTime">8</int>
  <lst name="params">
   <str name="q">*:*</str>
   <str name="fq">{!graph from=parent_id to=id}name:"root document"</str>
  </lst>
 </lst>
 <result name="response" numFound="8" start="0" maxScore="1.0">
 <doc>
  <str name="id">1</str>
  <str name="name">Root document one</str>
  <long name="_version_">1531331026113003520</long></doc>
 <doc>
  <str name="id">11</str>
  <str name="parent_id">1</str>
  <str name="name">First level document 1, child one</str>
  <long name="_version_">1531331026114052096</long></doc>
 <doc>
  <str name="id">12</str>
  <str name="parent_id">1</str>
  <str name="name">First level document 1, child two</str>
  <long name="_version_">1531331026115100672</long></doc>
 <doc>
  <str name="id">13</str>
  <str name="parent_id">1</str>
  <str name="name">First level document 1, child three</str>
  <long name="_version_">1531331026115100673</long></doc>
 <doc>
  <str name="id">122</str>
  <str name="parent_id">12</str>
  <str name="name">Second level document 12, child two</str>
  <long name="_version_">1531331026120343552</long></doc>
 <doc>
  <str name="id">2</str>
  <str name="name">Root document two</str>
  <long name="_version_">1531331026109857792</long></doc>
 <doc>
  <str name="id">3</str>
  <str name="name">Root document three</str>
  <long name="_version_">1531331026110906368</long></doc>
 <doc>
  <str name="id">21</str>
  <str name="parent_id">2</str>
  <str name="name">First level document 2, child one</str>
  <long name="_version_">1531331026111954944</long></doc>
 </result>
</response>

As you can see, we’ve got our root documents returned (identifiers 1, 2 and 3) and we’ve got all the leafs for the whole data set, which is very nice.

Filtering

The results can be filtered by using the traversalFilter property. The defined filter will be applied to each join iteration. For example, if we would like to filter the resulting documents to only those that have the term one in the name field we could run the following query:

http://localhost:8983/solr/graph/select?q=*:*&fq={!graph from=parent_id to=id traversalFilter=name:one}name:"root document"

The results would be as follows:

<pre class="brush:xml">
<?xml version="1.0" encoding="UTF-8"?>
<response>
 <lst name="responseHeader">
  <bool name="zkConnected">true</bool>
  <int name="status">0</int>
  <int name="QTime">7</int>
  <lst name="params">
   <str name="q">*:*</str>
   <str name="fq">{!graph from=parent_id to=id traversalFilter=name:one}name:"root document"</str>
  </lst>
 </lst>
 <result name="response" numFound="5" start="0" maxScore="1.0">
 <doc>
  <str name="id">1</str>
  <str name="name">Root document one</str>
  <long name="_version_">1531331026113003520</long></doc>
 <doc>
  <str name="id">11</str>
  <str name="parent_id">1</str>
  <str name="name">First level document 1, child one</str>
  <long name="_version_">1531331026114052096</long></doc>
 <doc>
  <str name="id">2</str>
  <str name="name">Root document two</str>
  <long name="_version_">1531331026109857792</long></doc>
 <doc>
  <str name="id">3</str>
  <str name="name">Root document three</str>
  <long name="_version_">1531331026110906368</long></doc>
 <doc>
  <str name="id">21</str>
  <str name="parent_id">2</str>
  <str name="name">First level document 2, child one</str>
  <long name="_version_">1531331026111954944</long></doc>
 </result>
</response>

As you can see, only the filtered documents were returned for each join, and of course the root document set. Seems that the filter is working. :-)

Returning the Root or Leaves

Apart from filtering, we can also tell Solr to only return leaves and to omit root set documents. For example, to omit root set documents we would add the returnRoot property equal to false (defaults to true ) in our query:

http://localhost:8983/solr/graph/select?q=*:*&fq={!graph from=parent_id to=id returnRoot=false}name:"root document"

The results are as follows:

<?xml version="1.0" encoding="UTF-8"?>
<response>
 <lst name="responseHeader">
  <bool name="zkConnected">true</bool>
  <int name="status">0</int>
  <int name="QTime">10</int>
  <lst name="params">
   <str name="q">*:*</str>
   <str name="fq">{!graph from=parent_id to=id returnRoot=false}name:"root document"</str>
  </lst>
 </lst>
 <result name="response" numFound="5" start="0" maxScore="1.0">
 <doc>
  <str name="id">11</str>
  <str name="parent_id">1</str>
  <str name="name">First level document 1, child one</str>
  <long name="_version_">1531331026114052096</long></doc>
 <doc>
  <str name="id">12</str>
  <str name="parent_id">1</str>
  <str name="name">First level document 1, child two</str>
  <long name="_version_">1531331026115100672</long></doc>
 <doc>
  <str name="id">13</str>
  <str name="parent_id">1</str>
  <str name="name">First level document 1, child three</str>
  <long name="_version_">1531331026115100673</long></doc>
 <doc>
  <str name="id">122</str>
  <str name="parent_id">12</str>
  <str name="name">Second level document 12, child two</str>
  <long name="_version_">1531331026120343552</long></doc>
 <doc>
  <str name="id">21</str>
  <str name="parent_id">2</str>
  <str name="name">First level document 2, child one</str>
  <long name="_version_">1531331026111954944</long></doc>
 </result>
</response>

As we can see, the results are without the root documents.

If we are interested in leaves only, we should add the returnOnlyLeaf parameter and set it to true (defaults to false ).

Controlling Maximum Depth

Finally, using the maxDepth property we can control the maximum depth of the traversal. By default, it is set to -1 which stands for unlimited. For example, if we are only interested in the first level of the graph, we could run the following query:

http://localhost:8983/solr/graph/select?q=*:*&fq={!graph from=parent_id to=id maxDepth=1}name:"root document"

The result includes only documents that are of one join from the documents in the root set:

<?xml version="1.0" encoding="UTF-8"?>
<response>
 <lst name="responseHeader">
  <bool name="zkConnected">true</bool>
  <int name="status">0</int>
  <int name="QTime">10</int>
  <lst name="params">
   <str name="q">*:*</str>
   <str name="fq">{!graph from=parent_id to=id maxDepth=1}name:"root document"</str>
  </lst>
 </lst>
 <result name="response" numFound="7" start="0" maxScore="1.0">
 <doc>
  <str name="id">1</str>
  <str name="name">Root document one</str>
  <long name="_version_">1531331026113003520</long></doc>
 <doc>
  <str name="id">11</str>
  <str name="parent_id">1</str>
  <str name="name">First level document 1, child one</str>
  <long name="_version_">1531331026114052096</long></doc>
 <doc>
  <str name="id">12</str>
  <str name="parent_id">1</str>
  <str name="name">First level document 1, child two</str>
  <long name="_version_">1531331026115100672</long></doc>
 <doc>
  <str name="id">13</str>
  <str name="parent_id">1</str>
  <str name="name">First level document 1, child three</str>
  <long name="_version_">1531331026115100673</long></doc>
 <doc>
  <str name="id">2</str>
  <str name="name">Root document two</str>
  <long name="_version_">1531331026109857792</long></doc>
 <doc>
  <str name="id">3</str>
  <str name="name">Root document three</str>
  <long name="_version_">1531331026110906368</long></doc>
 <doc>
  <str name="id">21</str>
  <str name="parent_id">2</str>
  <str name="name">First level document 2, child one</str>
  <long name="_version_">1531331026111954944</long></doc>
 </result>
</response>

Summary

As you can see, in addition to the Parallel SQL over Map Reduce functionality and the cross data center replication, we’ve got pretty neat graph traversal support in Solr 6.0. We haven’t had a chance to test the performance of the query on the larger data set, but we will try to come up with the larger data set and some sample queries to see what we can expect from Solr when it comes to graph traversal query performance.





About List