Consul Blocking Queries: Use Cases and Best Practices

Avatar
Vaibhav Arora
  • Updated

Introduction

Consul blocking queries provide a mechanism for clients to receive immediate updates about changes in Consul's data. This is achieved through long-polling HTTP requests that keep the connection open until a change occurs or a timeout is reached.

Applications that need to react quickly to changes in Consul's data, such as those performing service discovery or health monitoring, greatly benefit from blocking queries.

Scenarios

Add a blocking query with a wait parameter

Recommendation

  1. Create one KV using the below command. 
    consul kv put k1 v1
  2. Validate the created and modified index values.
    root@ip-172-31-39-10:~# curl localhost:8500/v1/kv/k1 | jq
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 95 100 95 0 0 46890 0 --:--:-- --:--:-- --:--:-- 95000
    [
    {
    "LockIndex": 0,
    "Key": "k1",
    "Flags": 0,
    "Value": "djE=",
    "CreateIndex": 107013,
    "ModifyIndex": 107013
    }
    ]
  3. Updated the KV values and checked the modified index values should be updated.
    consul kv put k1 v2
    root@ip-172-31-39-10:~# curl localhost:8500/v1/kv/k1 | jq
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 95 100 95 0 0 39142 0 --:--:-- --:--:-- --:--:-- 47500
    [
    {
    "LockIndex": 0,
    "Key": "k1",
    "Flags": 0,
    "Value": "djI=",
    "CreateIndex": 107013,
    "ModifyIndex": 107046
    }
    ]
  4. Let's add a wait for 5 min and it will be a timeout for 5 min. Now the k1 value has a timeout of 5 minutes.
    curl -H 'X-Consul-Index: 107046' --request GET http://localhost:8500/v1/kv/k1?wait=5m|jq

 

Block a query until a change is made

Recommendation

  1. Start the consul agent in development mode.
    consul agent -dev
  2. Write a value in the key for "log level".
    consul kv put loglevel info
  3. Please run raw HTTP GET request to KV store
    curl -v http://localhost:8500/v1/kv/loglevel
    • In my case, it returned "X-Consul-Index:224312" in the response.
  4. Please run the same GET request with a new index specified in the parameters.
    curl -v http://localhost:8500/v1/kv/loglevel?index=224312
    • Output of curl command:
      root@ip-172-31-39-10:/etc/consul.d# curl -v http://localhost:8500/v1/kv/loglevel?index=224312
      * Trying 127.0.0.1:8500...
      * Connected to localhost (127.0.0.1) port 8500 (#0)
      > GET /v1/kv/loglevel?index=224312 HTTP/1.1
      > Host: localhost:8500
      > User-Agent: curl/7.81.0
      > Accept: */*
      >
      • Now the query is blocked.
  5. Please run the update for the key in another terminal window.
    consul kv put loglevel Debug
    • Output of the command
      root@ip-172-31-39-10:/etc/consul.d# curl -v http://localhost:8500/v1/kv/loglevel?index=224312
      * Trying 127.0.0.1:8500...
      * Connected to localhost (127.0.0.1) port 8500 (#0)
      > GET /v1/kv/loglevel?index=224312 HTTP/1.1
      > Host: localhost:8500
      > User-Agent: curl/7.81.0
      > Accept: */*
      >


      * Mark bundle as not supporting multiuse
      < HTTP/1.1 200 OK
      < Content-Type: application/json
      < Vary: Accept-Encoding
      < X-Consul-Default-Acl-Policy: allow
      < X-Consul-Index: 224347
      < X-Consul-Knownleader: true
      < X-Consul-Lastcontact: 0
      < X-Consul-Query-Backend: blocking-query
      < Date: Thu, 16 May 2024 09:39:59 GMT
      < Content-Length: 105
      <
      * Connection #0 to host localhost left intact
      [{"LockIndex":0,"Key":"loglevel","Flags":0,"Value":"RGVidWc=","CreateIndex":224312,"ModifyIndex":224347}]root@ip-172-31-39-10:/etc/consul.d#
      root@ip-172-31-39-10:/etc/consul.d#

       

Reset the index using the blocking query.

Recommendation

  1. Create two KV in the consul.
    consul kv put k1 v1_1

    consul kv put loglevel info_1
  2. Check their index values and get the KV data.
    root@ip-172-31-37-167:/tmp# curl http://127.0.0.1:8500/v1/kv/k1| jq
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 99 100 99 0 0 47277 0 --:--:-- --:--:-- --:--:-- 99000
    [
    {
    "LockIndex": 0,
    "Key": "k1",
    "Flags": 0,
    "Value": "djFfMQ==",
    "CreateIndex": 266344,
    "ModifyIndex": 266344
    }
    ]


    root@ip-172-31-37-167:/tmp# curl http://127.0.0.1:8500/v1/kv/loglevel| jq
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 105 100 105 0 0 66539 0 --:--:-- --:--:-- --:--:-- 102k
    [
    {
    "LockIndex": 0,
    "Key": "loglevel",
    "Flags": 0,
    "Value": "aW5mb18x",
    "CreateIndex": 266346,
    "ModifyIndex": 266346
    }
    ]

    root@ip-172-31-37-167:/tmp# consul kv get k1
    v1_1

    root@ip-172-31-37-167:/tmp# consul kv get loglevel
    info_1

  3. Please take the snapshot
    consul snapshot save backup_1.18.5.snap

    root@ip-172-31-37-167:/tmp# consul snapshot save backup_1.18.5.snap
    Saved and verified snapshot to index 266356

  4. Update the KV values and see the data/Index

    consul kv put k1 v1_2

    consul kv put loglevel info_2

    root@ip-172-31-37-167:/tmp# consul kv get k1
    v1_2
    root@ip-172-31-37-167:/tmp# consul kv get loglevel
    info_2

    root@ip-172-31-37-167:/tmp# curl http://127.0.0.1:8500/v1/kv/k1| jq
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 99 100 99 0 0 24712 0 --:--:-- --:--:-- --:--:-- 33000
    [
    {
    "LockIndex": 0,
    "Key": "k1",
    "Flags": 0,
    "Value": "djFfMg==",
    "CreateIndex": 266344,
    "ModifyIndex": 266365
    }
    ]

    root@ip-172-31-37-167:/tmp# curl http://127.0.0.1:8500/v1/kv/loglevel| jq
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 105 100 105 0 0 8935 0 --:--:-- --:--:-- --:--:-- 9545
    [
    {
    "LockIndex": 0,
    "Key": "loglevel",
    "Flags": 0,
    "Value": "aW5mb18y",
    "CreateIndex": 266346,
    "ModifyIndex": 266366
    }
    ]

  5. Restore the snapshot

    consul snapshot restore backup_1.18.5.snap

  6. Update the KV with old index values

    curl -H "X-Consul-Index: 266366" —request PUT 'http://localhost:8500/v1/kv/loglevel=log_2'

  7. Check the index values of log level KV and values
    Screenshot 2024-05-21 at 5.15.55 PM.png

 

Additional Information

Reset the index if it goes backward:

While indexes in general are monotonically increasing(i.e. they should only ever increase as time passes), there are several real-world scenarios in which they can go backwards for a given query. Implementations must check to see if a returned index is lower than the previous value, and if it is, should reset index to 0 - effectively restarting their blocking loop. Failure to do so may cause the client to miss future updates for an unbounded time, or to use an invalid index value that causes no blocking and increases load on the servers. Cases where this can occur includes:

  • If a raft snapshot is restored on the servers with older version of the data.
  • KV list operations where an item with the highest index is removed.
  • A Consul upgrade changes the way watches work to optimize them with more granular indexes.

 Implementation details of Consul index: 

  • If the index for the object you are watching (KV pair, service,node..) doesn't increment then there hasn't been any changes to that object.
  • If you are watching an object and the indexes returned after two consecutive blocking queries have the same values, then there has been no changes to this object.

Can we increase the blocking query request parameter wait=5m and wait=10m:

  1. Tailoring wait parameters to your application and Consul Setup: It's essential to customize the wait parameter to match the specific needs of your application and the configuration of your Consul environment.

  2. Swift Responses for Priority Applications:

    If your application places a premium on rapid responses, opting for a shorter wait time can be advantageous.

  3.  

    Mitigating Network Overhead in Bandwidth-Constrained Environments: Conversely, in situations where network bandwidth is limited or expensive, extending the wait time can help reduce network overhead and associated costs.
  4. Consideration of Merits and Drawbacks: Each configuration option carries its own set of benefits and drawbacks. Balancing these factors is crucial to optimize resource utilization, response times, and network activity.
  5. Optimization through Testing and Evaluation: It's recommended to fine-tune these settings within your testing environment. By assessing the outcomes and performance, you can make informed adjustments to ensure optimal configuration for your specific requirements.

Reference documents

Articles in this section

See more

Related articles