Neo4j causal cluster HTTP load-balancing

Posted on Posted in DevOps & Networking

EDIT: Since I posted this Neo4j added new endpoints to monitor the causal cluster (in Neo4j version 3.1 and above) as mentioned in this guide, so this is only relevant for version 3.0

Neo4j recently introduced a new cluster architecture for their graph database product (available in the enterprise edition of neo4j). This clustering architecture is called causal cluster and is based on a concept of CORE nodes that form the cluster quorum and other nodes that are called READ REPLICAS.

At any given time one of the CORE nodes is the cluster LEADER and all writes should be performed through that node, other CORE nodes are in the FOLLOWER state and you can’t write to them.

You can see the state of the cluster using the neo4j browser using the :play sysinfo command

Neo4j nodes in the cluster listen on 3 protocols: bolt (Neo4j’s binary protocol that has official drivers for a lot of different programming languages), HTTP and HTTPS.
In order to use this new cluster architecture Neo4j introduced an additional feature to their bolt protocol implementation: Bolt + routing

Bolt + routing is implemented on the server side and the client-side (as part of the official bolt drivers).

To summarize how bolt+routing currently works, at the beginning of the session the client asks the cluster for the current routing table in order to understand which node is writable and readable.

After that the client will approach the appropriate node for read or write operations.

This is an example of how this routing table looks like:

Alright, so you might ask – what if I want to use HTTP to communicate with the Neo4j causal cluster?

The solution is simple – we will use a load-balancer in front of the cluster with 2 Virtual Servers – one for Write operations and the other for Read operations:

  • Write virtual server will contain all of the CORE nodes of the cluster.
  • Read virtual server will contain all of the cluster nodes.

In order to do that the load-balancer has to support custom HTTP health-checks, I will demonstrate this using F5 LTM.

Using HTTP you can pass a Cypher statement to the Neo4j cluster, using this functionality we will monitor which node is the LEADER and this monitor will be used for the Write operation Loadbalancer pool.
We will use the following built in Cypher procedure to determine if the node is a LEADER or not:

The next step is constructing the HTTP request for the custom monitor.
You will need a user and password for your Neo4j cluster (The only requirement is read permissions)

We will use Postman and Wireshark to create the base64 encoded username and password and to prepare the information required for the health monitor (Remember that base64 encoding is not secure)

HTTP POST

http://NEO4J-ADDRESS:7474/db/data/transaction/commit

Authorization > Type: Basic Auth, Fill in your Neo4j username and password

Next go to the Headers tab (here you should already see the encoded base64 username+password) and add the Content-Type header

Content-Type: application/json

The last part is the POST body, change to the Body tab and choose raw.

Paste this in the raw body area:

{ “statements”:[{ “statement” : “CALL dbms.cluster.role()” }]}

Now you can test, click on Send and you should get a response similar to this:

Use Wireshark to capture the content length when making this request – with this specific payload its 62

Content-Length: 62

 

After all of this tedious work we are finally ready to construct the HTTP monitor.

  • Add the Content-Length: 62 and Connection: close to the request
  • Replace the <HOSTNAME> with any hostname you want, e.g neo4j.vip.example.local
  • Replace <BASE64_ENCODED_USER_PASSWORD> with your base64 encoded credentials from Postman.
  • Escape the quotes (like I did below)
  • Create a HTTP monitor using the following Send String:

 

  • The Receive String will be LEADER
  • Use interval and timeout according to your requirement

Example command to create this monitor via TMSH:

 

Now you can use this HTTP monitor in a pool that will be used for Write only operations that will always get to the LEADER node.

2 thoughts on “Neo4j causal cluster HTTP load-balancing

Leave a Reply

Your email address will not be published. Required fields are marked *