Skip to main content

Nuances of Spatial data and operations Apache SOLR

SOLR is a scalable, distributed and powerful search and indexing solution. SOLR supports indexing spatial data and provides fast search capabilities on spatial data.
Unlike conventional spatial data solutions like PostGIS, indexing spatial data in SOLR provides high speed search using bounding box and range queries, commonly used in spatial data exploration and processing.
Indexing points and running bounding box queries is covered extensively in Apache SOLR documentation.

We will discuss some specific use cases here, described but not explained in the documentation.

1. Indexing Fences in SOLR: SOLR provides a convenient and powerful type, location_rpt which is an implementation of solr.SpatialRecursivePrefixTreeFieldType. location_rpt indexes POINT data, consisting of latitude and longitude. Points are sufficient to index locations that need to be monitored, and that can be done by using bbox or geofile queries as in the documentation. A typical use case in geospatial search is drawing fences around areas that need to be monitored. This requires storing, indexing and monitoring two dimensional areas. In order to do that, 2D shapes can be indexed using SpatialRecursivePrefixTreeFieldType (called RPT), by defining a custom type in the schema as below. This requires using spatial4j JtsSpatialContextFactory. JTS jar should be added to SOLR classpath in order to resolve the factory. Once the type is added, data can be indexed into fields defined of type fence_rpt, with 2D shapes defined as POLYGON("x1 y1,x2 y2,x3 y3,x4 y4")

<fieldType name="fence_rpt"   class="solr.SpatialRecursivePrefixTreeFieldType"           spatialContextFactory="com.spatial4j.core.context.jts.JtsSpatialContextFactory"           distErrPct="0.025"           maxDistErr="0.000009"           units="degrees"        />
In order to monitor fences, we can determine which of the indexed fences does a point intersect using fq=geo:Intersects("x y"). A common use case being, monitoring a person's movements and identifying which fences he get in and out of.

2.  Indexing non geo data in SOLR: location_rpt type, by default, indexes geographical data containing latitude, longitude. In order to index, search and process non geographic data, RPT type can be defined with geo=false attribute. When indexing non geographic points, world bounds need to be defined. These form the coordinate system for indexing data.
Non Geographic RPT types can be combined with JTS to support indexing 2D shapes.
This is useful in indoor geographic indexing and search. This can also be used, probably not intended, to build custom solutions where shapes need to be indexed and searched upon, for non location based use case too, like image analysis.

<fieldType name="nongeorpt"
class="solr.SpatialRecursivePrefixTreeFieldType"
geo="false"
worldBounds="ENVELOPE(0, 0, 1000, 1000)"
maxDistErr="0.001"
units="degrees" />


P.S: Food for thought: Motion controlled games use point in polygon to determine movement of controller and render the movement on screen. Motion controller applications can leverage indexing of points and polygons in custom reference frame for varied applications like hand movement tracking to move robotic mechanical arms.



Comments

Popular posts from this blog

Catch hold of that Exception and hide that stacktrace!!!

E xceptions happen!!! Rules are to be followed, too. Time and again, Java developers are told the golden rule to catch specific exceptions and not to catch the generic Exception. The thought process behind that is, applications should not catch runtime exceptions. This is apt as runtime exceptions are an indicator of "bugs" in the code. However, blindly following rules, as always, can have unexpected consequences. If you are developing services that are to be exposed over the wire, it is always a good idea to break this rule and "catch that Exception". Instead, follow the below principles: Service methods should implement a generic Exception block, along with catching declared exceptions, thrown from inner layers of the code.  If needed, the service can throw another exception back to the client. What's important is that we create a new Exception instance to be thrown, along with relevant message for the client. The service can log stacktrace for the E...

Using JNDI managed JMS objects with Apache CAMEL

Apache CAMEL uses Spring JMS to work with JMS Queues or Topics. Evidently, we will need Spring to configure and use JMS capabilities provided by CAMEL. Details about how to implement JMS based routes using Apache CAMEL can be found in the CAMEL documentation. However, the documentation leaves a lot to be figured out. In a typical Java EE container, it is usually a good idea to abstract the underlying JMS resources by using JNDI. We can use the below configuration to achieve that. This configuration is tested in Websphere environment, but should work in any JEE container. Create a JMS queue connection factory in the JNDI registry. CAMEL configuration will be able to use only one queue connection factory, even if we have more than one. Create one or more JMS queue or topics, in the JNDI registry, as required. The above two steps are related to generic JNDI configuration for JMS resources. Now we come to the setup required for making these JMS resources work with CAMEL rout...

Cassandra data modelling: Redundant data, a tough decision

The biggest challenge around building an efficient data model for Cassandra is data redundancy. Though the basic rules for data modelling with Cassandra, mention the usual RDBMS modelling goals, as non goals for Cassandra (Refer:  Basic rules for C* data modelling ), it builds upon assumptions that clusters are built on commodity hardware, storage is cheap, and as data needs increase more nodes can be added to the cluster incurring very low cost. But in real life we are faced with technical as well as non technical problems. a. Keeping multiple column families in sync can be a major overhead, if the same data is spread across them. What if writes to some CF succeed and some fail? How long and how much will we retry? b. Horizontal scalability may be a truth, but think of a mundane question of where to keep all those heat producing, energy guzzling machines? So how do we model the database that does not allow joins without redundancy? The simple answer is, we do not. Wh...