Concepts
Tenants and keys
An installation serves one or more tenants. Every request carries an API key, and the key says which tenant it belongs to: no path contains a tenant, and tenants never see each other's queues or messages.
A key has scopes. publish may put messages on queues, consume may claim and report on
them, admin may do everything inside the tenant: change queue settings, inspect and
replay messages, manage keys. A key can also be limited to queues matching a pattern,
such as import-*.
Tenants themselves are managed with an operator key, which belongs to no tenant. It is created on the command line when the installation is set up, typically for a web interface that signs up and manages tenants.
Queues and their settings
A queue exists from its first message on. Its behaviour comes from three layers: built-in defaults, the tenant's defaults, and the queue's own settings, each overriding the one before.
| Setting | Default | What it decides |
|---|---|---|
lease_seconds |
300 | how long a claim holds a message |
max_lease_seconds |
3600 | the longest lease a worker may ask for at once |
max_attempts |
7 | attempts before the message is given up on |
max_attempts_limit |
25 | the most attempts a publisher may ask for |
backoff |
10 s, ×2, up to 1 h, ±10 % | the wait before the next attempt |
on_exhausted |
dead_letter |
after the last attempt: keep it, or discard it |
on_permanent |
dead_letter |
what a permanent failure does: dead-letter at once, or retry |
route_policy |
none | avoid_failed: keep messages off routes they failed on, for cooldown_seconds |
retention |
1 day completed, 7 days failed | how long finished messages are kept |
max_message_bytes |
1 MiB | the largest body plus headers |
paused |
false | a paused queue accepts messages but hands none out |
Settings change with PATCH /v1/queues/{queue} and take effect within a second.
The life of a message
| From | To | When |
|---|---|---|
| (new) | pending |
publish |
pending |
processing |
a worker claims it; an attempt starts |
processing |
completed |
ack |
processing |
pending |
fail with attempts left, after the backoff |
processing |
pending |
the lease expired; the attempt counts |
processing |
pending |
release; the attempt does not count |
processing |
failed |
fail with no attempts left, or a permanent failure |
processing |
(deleted) | as above, when the queue's on_exhausted is discard |
failed, completed |
pending |
replay |
failed, completed |
(deleted) | retention |
A message is published to a queue, optionally with a delay, a dedup key, headers, or a
different number of attempts. It waits as pending until a worker claims it.
Leases and receipts
A claim returns the message, a receipt and the time the lease ends. Only the holder of the current receipt can extend the lease, ack, fail, or release the message.
When a lease runs out, the server makes the message available again and counts the
attempt. The worker that held it may still be busy; when it finally reports, it gets
409 lease_lost and must drop the message. This is what keeps a slow worker from
completing or resetting a message that another worker now holds.
A worker with a long job extends its lease while it works. The official libraries do this on a timer.
Attempts and failures
Every claim starts an attempt. A worker that fails reports the error's type and message; the queue decides what follows:
- with attempts left, the message becomes pending again after the backoff;
- after the last attempt, it goes to the dead-letter set (or is discarded);
- an expired lease counts as a failed attempt, so a message that crashes its worker still runs out of attempts instead of coming back forever.
A worker that can tell the input itself is unusable reports the failure as permanent. The
queue's on_permanent setting decides whether that skips the remaining attempts.
Delivery is at least once: a message can run more than once, when a lease expires under a slow worker or an ack is lost on the network. Handlers should tolerate that.
The dead-letter set
Messages the server gave up on keep status failed, with their last error, for the
queue's retention. Administrators list them (filtered by error type), look at them,
replay them (back to pending, attempts reset) or delete them, one at a time or in bulk.
Deduplication
A publish with a dedup_key returns the existing message if one with that key is still
open (pending or processing) in the queue. The database enforces it, so two publishes at
the same moment still create one message. Once a message completes or fails, its key is
free again.
Routes
Some failures depend on the path a job took. A website that blocks one VPN exit will keep
refusing it; an API account that used up its quota fails until tomorrow. A worker that
knows its path passes it as route when it claims. The server records the route of every
failed attempt, and on a queue with route_policy.mode: avoid_failed it does not hand
the message to a worker on such a route until cooldown_seconds have passed.
Long polling
A claim may wait up to 20 seconds for a message instead of returning empty at once. A worker loop needs no sleep of its own, and the server is not flooded with empty polls.