Consul blocking queries are long-polling HTTP requests made by Consul clients to the Consul server, enabling them to receive immediate notifications about changes in the Consul's data. These queries are particularly useful for applications requiring real-time updates, such as service discovery, health monitoring, and dynamic configuration management.
Reset the index if it goes backward:
While indexes in general are monotonically increasing(i.e. they should only ever increase as time passes), there are several real-world scenarios in which they can go backwards for a given query. Implementations must check to see if a returned index is lower than the previous value, and if it is, should reset index to 0 - effectively restarting their blocking loop. Failure to do so may cause the client to miss future updates for an unbounded time, or to use an invalid index value that causes no blocking and increases load on the servers. Cases where this can occur includes:
- If a raft snapshot is restored on the servers with older version of the data.
- KV list operations where an item with the highest index is removed.
- A Consul upgrade changes the way watches work to optimize them with more granular indexes.
Implementation details of Consul index:
- If the index for the object you are watching (KV pair, service,node..) doesn't increment then there hasn't been any changes to that object.
- If you are watching an object and the indexes returned after two consecutive blocking queries have the same values, then there has been no changes to this object.
Can we increase the blocking query request parameter wait=5m and wait=10m:
1. Tailoring wait parameters to your application and Consul Setup: It's essential to customize the wait parameter to match the specific needs of your application and the configuration of your Consul environment.
2. Swift Responses for Priority Applications:
If your application places a premium on rapid responses, opting for a shorter wait time can be advantageous.
3. Mitigating Network Overhead in Bandwidth-Constrained Environments: Conversely, in situations where network bandwidth is limited or expensive, extending the wait time can help reduce network overhead and associated costs.
4. Consideration of Merits and Drawbacks: Each configuration option carries its own set of benefits and drawbacks. Balancing these factors is crucial to optimize resource utilization, response times, and network activity.
5. Optimization through Testing and Evaluation: It's recommended to fine-tune these settings within your testing environment. By assessing the outcomes and performance, you can make informed adjustments to ensure optimal configuration for your specific requirements.
Lab Setup of blocking query:
Scenario 1: To add blocking query with wait parameter
1. Create one KV using the below command.
consul kv put k1 v1
2. Validate the created and modified index values.
root@ip-172-31-39-10:~# curl localhost:8500/v1/kv/k1 | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 95 100 95 0 0 46890 0 --:--:-- --:--:-- --:--:-- 95000
[
{
"LockIndex": 0,
"Key": "k1",
"Flags": 0,
"Value": "djE=",
"CreateIndex": 107013,
"ModifyIndex": 107013
}
]
3. Updated the KV values and checked the modified index values should be updated.
consul kv put k1 v2
root@ip-172-31-39-10:~# curl localhost:8500/v1/kv/k1 | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 95 100 95 0 0 39142 0 --:--:-- --:--:-- --:--:-- 47500
[
{
"LockIndex": 0,
"Key": "k1",
"Flags": 0,
"Value": "djI=",
"CreateIndex": 107013,
"ModifyIndex": 107046
}
]
4. Let's add a wait for 5 min and it will be a timeout for 5 min. Now the k1 value has a timeout of 5 minutes.
curl -H 'X-Consul-Index: 107046' --request GET http://localhost:8500/v1/kv/k1?wait=5m|jq
Scenario 2: To block the query until a change is made.
1. Start the consul agent in development mode.
consul agent -dev
2. Write a value in the key for "log level".
consul kv put loglevel info
3. Please run raw HTTP GET request to KV store
curl -v http://localhost:8500/v1/kv/loglevel
In my case, it returned "X-Consul-Index:224312" in the response.
4. Please run the same GET request with a new index specified in the parameters.
curl -v http://localhost:8500/v1/kv/loglevel?index=224312
Output of curl command:
root@ip-172-31-39-10:/etc/consul.d# curl -v http://localhost:8500/v1/kv/loglevel?index=224312
* Trying 127.0.0.1:8500...
* Connected to localhost (127.0.0.1) port 8500 (#0)
> GET /v1/kv/loglevel?index=224312 HTTP/1.1
> Host: localhost:8500
> User-Agent: curl/7.81.0
> Accept: */*
>
Now the query is blocked.
5. Please run the update for the key in another terminal window.
consul kv put loglevel Debug
Output of curl command:
root@ip-172-31-39-10:/etc/consul.d# curl -v http://localhost:8500/v1/kv/loglevel?index=224312
* Trying 127.0.0.1:8500...
* Connected to localhost (127.0.0.1) port 8500 (#0)
> GET /v1/kv/loglevel?index=224312 HTTP/1.1
> Host: localhost:8500
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Accept-Encoding
< X-Consul-Default-Acl-Policy: allow
< X-Consul-Index: 224347
< X-Consul-Knownleader: true
< X-Consul-Lastcontact: 0
< X-Consul-Query-Backend: blocking-query
< Date: Thu, 16 May 2024 09:39:59 GMT
< Content-Length: 105
<
* Connection #0 to host localhost left intact
[{"LockIndex":0,"Key":"loglevel","Flags":0,"Value":"RGVidWc=","CreateIndex":224312,"ModifyIndex":224347}]root@ip-172-31-39-10:/etc/consul.d#
root@ip-172-31-39-10:/etc/consul.d#
Scenario 3: Reset the index using the blocking query.
1. Create two KV in the consul.
consul kv put k1 v1_1
consul kv put loglevel info_1
2. Check their index values and get the KV data.
root@ip-172-31-37-167:/tmp# curl http://127.0.0.1:8500/v1/kv/k1| jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 99 100 99 0 0 47277 0 --:--:-- --:--:-- --:--:-- 99000
[
{
"LockIndex": 0,
"Key": "k1",
"Flags": 0,
"Value": "djFfMQ==",
"CreateIndex": 266344,
"ModifyIndex": 266344
}
]
root@ip-172-31-37-167:/tmp# curl http://127.0.0.1:8500/v1/kv/loglevel| jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 105 100 105 0 0 66539 0 --:--:-- --:--:-- --:--:-- 102k
[
{
"LockIndex": 0,
"Key": "loglevel",
"Flags": 0,
"Value": "aW5mb18x",
"CreateIndex": 266346,
"ModifyIndex": 266346
}
]
root@ip-172-31-37-167:/tmp# consul kv get k1
v1_1
root@ip-172-31-37-167:/tmp# consul kv get loglevel
info_1
3. Please take the snapshot
consul snapshot save backup_1.18.5.snap
root@ip-172-31-37-167:/tmp# consul snapshot save backup_1.18.5.snap
Saved and verified snapshot to index 266356
4. Update the KV values and see the data/Index
consul kv put k1 v1_2
consul kv put loglevel info_2
root@ip-172-31-37-167:/tmp# consul kv get k1
v1_2
root@ip-172-31-37-167:/tmp# consul kv get loglevel
info_2
root@ip-172-31-37-167:/tmp# curl http://127.0.0.1:8500/v1/kv/k1| jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 99 100 99 0 0 24712 0 --:--:-- --:--:-- --:--:-- 33000
[
{
"LockIndex": 0,
"Key": "k1",
"Flags": 0,
"Value": "djFfMg==",
"CreateIndex": 266344,
"ModifyIndex": 266365
}
]
root@ip-172-31-37-167:/tmp# curl http://127.0.0.1:8500/v1/kv/loglevel| jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 105 100 105 0 0 8935 0 --:--:-- --:--:-- --:--:-- 9545
[
{
"LockIndex": 0,
"Key": "loglevel",
"Flags": 0,
"Value": "aW5mb18y",
"CreateIndex": 266346,
"ModifyIndex": 266366
}
]
5. Restore the snapshot
consul snapshot restore backup_1.18.5.snap
6. Update the KV with old index values
curl -H "X-Consul-Index: 266366" —request PUT 'http://localhost:8500/v1/kv/loglevel=log_2'
7. Check the index values of log level KV and values
Reference documents:
https://developer.hashicorp.com/consul/api-docs/features/caching
https://developer.hashicorp.com/consul/api-docs/features/blocking
https://developer.hashicorp.com/consul/commands/watch