Manage queues and process messages#

You can use the examples in the following sections to create message queues and process messages by using Rackspace Cloud Queues API operations. Before running the examples, review the Cloud Queues concepts to understand the API workflow, messaging patterns, and use cases.

Note

These examples use the $API_ENDPOINT, $AUTH_TOKEN, and $TENANT_ID environment variables to specify the API endpoint, authentication token, and project ID values for accessing the service. Be sure to configure these variables before running the code samples.

Creating a queue#

The queue operation creates a queue in the region of your choice.

The body of the PUT request is empty.

Following is the operation template:

PUT {endpoint}/queues/{queue_name}

The queue_name parameter specifies the name to give the queue. The name must not exceed 64 bytes in length and is limited to US-ASCII letters, digits, underscores, and hyphens.

Following are examples of a create queue request and response:

cURL create queue request

curl -i -X PUT $API_ENDPOINT/queues/samplequeue \
-H "X-Auth-Token: $AUTH_TOKEN" \
-H "Accept: application/json" \
-H "X-Project-Id: $TENANT_ID"

Create queue response

HTTP/1.1 201 Created
Content-Length: 0
Location: /v1/queues/samplequeue

Posting messages#

The post message operation inserts one or more messages in a queue.

You can submit up to 10 messages in a single request, but you must encapsulate them in a collection container (an array in JSON, even for a single message - without the JSON array, you receive an “Invalid body request” error message). You can use the resulting value of the location header or response body to retrieve the created messages for further processing if needed.

Following is the operation template:

POST {endpoint}/queues/{queue_name}/messages

The client specifies only the body and ttl attributes for the message. Metadata, such as id and age, is added.

The response body contains a list of resource paths that correspond to each message submitted in the request, in the same dfwer as they were submitted. If a server-side error occurs during the processing of the submitted messages, a partial list is returned. The partial attribute is set to true, and the client tries to post the remaining messages again. The body attribute specifies an arbitrary document that constitutes the body of the message being sent.

The following rules apply for the maximum size:

  • The size is limited to 256 KB for the entire request body (as-is), including whitespace.

  • The maximum size of posted messages is the maximum size of the entire request document (rather than the sum of the individual message body field values as it was in earlier releases). On error, the client is notified of by how much the request exceeded the limit.

The document must be valid JSON (Cloud Queues validates it).

The ttl attribute specifies the lifetime of the message. When the lifetime expires, the server deletes the message and removes it from the queue. Valid values are 60 through 1209600 seconds (14 days).

Note

The server might not actually delete the message until its age reaches (ttl + 60) seconds. So there might be a delay of 60 seconds after the message expires before it is deleted.

The following are examples of a post message request and response:

cURL post message request

curl -i -X POST $API_ENDPOINT/queues/samplequeue/messages -d \
'[{"ttl": 300,"body": {"event": "BackupStarted"}},{"ttl": 60,"body": {"play": "hockey"}}]' \
-H "Content-type: application/json" \
-H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \
-H "X-Auth-Token: $AUTH_TOKEN" \
-H "Accept: application/json" \
-H "X-Project-Id: $TENANT_ID"

Post message response

HTTP/1.1 201 Created
Content-Length: 153
Content-Type: application/json; charset=utf-8
Location: /v1/queues/samplequeue/messages?ids=51ca00a0c508f154c912b85c,51ca00a0c508f154c912b85d

{"partial": false, "resources": ["/v1/queues/samplequeue/messages/51ca00a0c508f154c912b85c", "/v1/queues/samplequeue/messages/51ca00a0c508f154c912b85d"]}

Claiming messages for processing#

The claim messages operation gets a set of messages from a specified message queue (up to the value of the limit parameter) from oldest to newest and skips any messages that are already claimed. If there are no messages available to claim, the Cloud Queues service returns an HTTP 204 No Content response code.

Following is the operation template:

POST {endpoint}/queues/{queue_name}/claims{?limit}
Content-Type: application/json

{
    "ttl": {claim_ttl},
    "grace": {message_grace}
}

The client (worker) needs to delete the message when it has finished processing it. The client deletes the message before the claim expires to ensure that the message is processed only once.

If a client needs more time, Cloud Queues provides the update claim operation to make changes.

As part of the delete operation, workers specify the claim ID (which is best done by simply using the provided href). If workers perform these actions, then if a claim simply expires, the server can return an error and notify the worker of a possible race condition. This action gives the worker a chance to roll back its own processing of the given message because another worker can claim the message and process it.

The age given for a claim is relative to the server’s clock. The claim’s age is useful for determining how quickly messages are getting processed and whether a given message’s claim is about to expire.

When a claim expires, it is released back to the queue for other workers to claim. (If the original worker failed to process the message, another client worker can then claim the message.)

The limit parameter specifies the number of messages to claim. If a limit value is not specified, Cloud Queues claims 10 messages. After the messages are successfully claimed, Cloud Queues returns 10 messages in the response.

Messages are claimed based on the number of messages available. The server might claim and return less than the requested number of messages.

If you specify a value for the limit parameter and the value is less than or equal to 100, Cloud Queues lets you claim free messages up to the number specified by the limit parameter. For example, if only 23 messages are free and limit is 100, Cloud Queues claims all 23 messages and returns 23 messages in the response. If 120 free messages are available and limit is 100, Cloud Queues claims the oldest 100 messages and returns 100 messages in response.

The ttl attribute specifies the lifetime of the claim. While messages are claimed, they are not available to other workers. The value must be between 60 and 43200 seconds (12 hours).

The grace attribute specifies the message grace period in seconds. Valid values are between 60 and 43200 seconds (12 hours). To deal with workers that have stopped responding (for up to 1209600 seconds or 14 days, including claim lifetime), the server extends the lifetime of claimed messages to be at least as long as the lifetime of the claim itself, plus the specified grace period. If a claimed message normally lives longer than the grace period, its expiration is not adjusted.

Following are examples of a claim messages request and response:

cURL claim messages request

curl -i -X POST $API_ENDPOINT/queues/samplequeue/claims -d \
'{"ttl": 300,"grace":300}' \
-H "Content-type: application/json" \
-H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \
-H "X-Auth-Token: $AUTH_TOKEN" \
-H "Accept: application/json" \
-H "X-Project-Id: $TENANT_ID"

Claim messages response

HTTP/1.1 201 OK
Content-Length: 164
Content-Type: application/json; charset=utf-8
Location: /v1/queues/samplequeue/claims/51ca011c821e7250f344efd6
X-Project-Id: ;;;;$TENANT_ID!bold;;;;

[
  {
    "body": {
      "event": "BackupStarted"
    },
    "age": 124,
    "href": "/v1/queues/samplequeue/messages/51ca00a0c508f154c912b85c?claim_id=51ca011c821e7250f344efd6",
    "ttl": 300
  }
]

Deleting messages#

The delete message operation deletes messages that have been processed by a worker.

Following is the operation template:

DELETE {endpoint}/queues/{queue_name}/messages/{message_id}{?claim_id}

The message_id parameter specifies the message to delete.

The claim_id parameter specifies that the message is deleted only if it has the specified claim ID and that claim has not expired. This specification is useful for ensuring that only one worker processes any given message.

When a worker’s claim expires before it deletes a message that it has processed, the worker must roll back any actions it took based on that message because another worker can now claim and process the same message.

Following are examples of a delete message request and response:

cURL delete message request

curl -i -X DELETE $API_ENDPOINT/queues/samplequeue/messages/51ca00a0c508f154c912b85c?claim_id=51ca011c821e7250f344efd6 \
-H "Content-type: application/json" \
-H "X-Auth-Token: $AUTH_TOKEN" \
-H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830" \
-H "Accept: application/json" \
-H "X-Project-Id: $TENANT_ID"

Delete message response

HTTP/1.1 204 No Content

Releasing a claim#

The release claim operation immediately releases all messages associated with a specified claim ID. This action makes all remaining, undeleted messages associated with the claim available to be processed by other workers.

Following is the operation template:

DELETE {endpoint}/queues/{queue_name}/claims/{claim_id}

This operation is useful when a worker is performing a graceful shutdown, fails to process one or more messages, or is taking longer than expected to process messages and wants to make the remainder of the messages available to other workers.

Following are examples of a release claim request and response:

cURL relesase claim request

curl -i -X DELETE $API_ENDPOINT/queues/samplequeue/claims/51ca011c821e7250f344efd6 \
-H "Content-type: application/json" \
-H "X-Auth-Token: $AUTH_TOKEN" \
-H "Client-ID: e58668fc-26eb-11e3-8270-5b3128d43830"  \
-H "Accept: application/json" \
-H "X-Project-Id: $TENANT_ID"

Release claim response

HTTP/1.1 204 No Content

Deleting a queue#

The delete queue operation immediately deletes a queue and all of its existing messages.

Following is the operation template:

DELETE {endpoint}/queues/{queue_name}

Following are examples of a delete queue request and response:

cURL delete queue request

curl -i -X DELETE $API_ENDPOINT/queues/samplequeue \
-H "Content-type: application/json" \
-H "X-Auth-Token: $AUTH_TOKEN" \
-H "Accept: application/json" \
-H "X-Project-Id: $TENANT_ID"

Delete queue response

HTTP/1.1 204 No Content