This website recently got a major rebuild; if you're missing something, let Lorna know.

Using Persistent Storage with Gearman



I'm using gearman for the first time in a new project, and two things in particular were bothering me. Firstly, there doesn't seem to be a built-in way to see what's in the queue. Secondly, if the gearman server dies (which seemed quite likely when I was first getting to grips with this stuff and writing really buggy code!) you lose your queue. Therefore I decided that I would switch gearman over to running with persistent storage. My config file (this is Ubuntu 10.10) is in /etc/default/gearman-job-server and it contains the following snippet:

# Use mysql as persistent queue store
# PARAMS="-q libdrizzle --libdrizzle-host=10.0.0.1 --libdrizzle-user=gearman \
#                       --libdrizzle-password=secret --libdrizzle-db=some_db \
#                       --libdrizzle-table=gearman_queue --libdrizzle-mysql"

Since I'm already using MySQL as application storage, this seemed like a great way forward. After looking around a bit, I found this great post about using PHP and Gearman. including instructions for persistent storage. We create a mysql table like this:

CREATE TABLE gearman_queue(
`unique_key` VARCHAR(64) PRIMARY KEY,
`function_name` VARCHAR(255),
`priority` INT,
`data` LONGBLOB
);

Then we adapt the block of code above to point to our mysql instance as needed. I found that I also needed a ``--libdrizzle-port=3306`` in that configuration, along with my host, user, password and database details to make this work. Once I had changed the config file, I restarted gearman:

/etc/init.d/gearman-job-server restart

Now when I add jobs to gearman, I see them in the gearman_queue table until they have been processed, and if the job server does restart with an oustanding job queue, it won't be lost.


In: tech
Tags: #gearman #mysql #tech