In a previous post I wrote about simple job queue in Redis. In this post I like to write a bit about the design choices. I think these choices illustrate a point about design.
The job handle in the code is create by the client (with help from the Redis server). This job handle is passed to the queue in Redis and picked up on the other side by the worker.
The worker doesn’t need to know the content of the key. The key is opaque. This means that the key could be changed to be something else. In the current case the handle is the build up out of the queue name and a unique id.
The worker only needs to know which commands it can use with that job handle. In
this case the commands are
DEL. When the job handle changes,
because it uses a hash type value, then the workers don’t need to change.
After the worker got the hash value with the job from Redis, it can perform it’s job. The value is specific to the worker and this shouldn’t change. In the content of the value uses the keys and value types then the workers keep working. Upgrading the content can be done separately for the client and the worker, if you keep the same keys in the job and only add keys. The workers can get upgraded later with the newer commands. This could also be done the other way around as long as the new worker understands the old job data.
The keys here are similar to URI in HTTP (and REST). The value of the URI is opaque. As a client you shouldn’t create URI yourself, but you should the follow the URI from the server according to its media type.
Incrementing the id of the job
Handle of the job consists of the name of the queue and a unique id. We get
this id from Redis using the
INCR command. The command returns the current
value + 1. We use this exact value in the handle. In a way we use the value in
pre increment way. We increment and then use the value. This means that we can
use the exact value that we got back from Redis.
Another way is to say that the value in Redis is id of the next job. By incrementing the value we say that we used this value. The only problem is that the returned value is one higher than the value we would like to use. So we need to decrement the returned value with 1.
The design choice here was to use the exact value from Redis. In a way this is the same as the job handle example. There the worker uses the exact value that it got. This choice makes it easier, because there is no calculation needed
Redis is a flexible solution to a lot of problems, but as always you need to make choices to get a solution the works for your problem.