Bulk operations#

This section describes bulk operations that are available with Cloud Files.

Extracting archive files#

An extract archive request expands tar files into a Cloud Files account. The request is a PUT operation with the ?extract-archive=format query parameter specifying the format of the archive file.

Valid values for the format variable are tar, tar.gz, and tar.bz2.

Note

This bulk operation involves middleware that conducts many operations on a single request.

For the PUT request, use the following URI:

/v1/AUTH_Account/$UPLOAD_PATH?extract-archive=tar.gz

UPLOAD_PATH is the location where the files are expanded and can specify any of the following values:

  • A container

  • A pseudo directory within a container

  • An empty string

    If the UPLOAD_PATH is an empty string, Cloud Files automatically creates containers in which to place the files. Files in the base directory of the tar file (that is, files that are not in a folder of the unzipped tar file) are ignored.

The destination of a file in the archive is built as follows:

/v1/AUTH_Account/$UPLOAD_PATH/$FILE_PATH

FILE_PATH is the file name from the listing in the tar file.

The following example shows a request to extract an archive.

Example: Example extract archive request

$ tar cf archive.tar directory_to_be_archived
$ curl -i -XPUT -H'x-auth-token: AUTH_TOKEN'
https://storage101.iad3.clouddrive.com/v1/MossoCloudFS_aaa-aaa-aaa-aaa?extract-archive=tar -T ./archive.tar

You can create up to 1,000 new containers per extraction request. Also note that only regular files are uploaded. Objects such as empty directories and symlinks are not uploaded.

The responses from bulk operations are not like other Cloud Files responses because a short request body sent from the client could result in many operations on the proxy server and precautions need to be taken to prevent the request from timing out because of a lack of activity. To this end, the client always receives a 200 OK response, regardless of the actual success of the call. The body of the response, which can be delivered over a greater amount of time, must be parsed to determine the actual success of the operation. In addition, the client might receive whitespace characters prepended to the response body while the proxy server is completing the request.

The format of the response body defaults to text plain but can be either JSON or XML depending on the Accept header. Acceptable formats are text/plain, application/json, application/xml, and text/xml. The following example shows the response body, formatted in JSON, from a successful request.

Example: Successful extract archive response

HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Content-Type: application/json
X-Trans-Id: txa7fd1603cfe04bdb920cd-005191226d
Date: Mon, 13 May 2013 17:27:10 GMT
Transfer-Encoding: chunked

{
  "Number Files Created": 10,
  "Response Status": "201 Created",
  "Errors": [],
  "Response Body": ""
}

The following example shows a response with errors.

Example: Extract archive response with errors

HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Content-Type: application/json
X-Trans-Id: tx9f12e16f047049cc91147-005191245f
Date:  Mon, 13 May 2013 17:35:27 GMT
Transfer-Encoding: chunked

{
  "Number Files Created": 10,
  "Response Status": "400 Bad Request",
  "Errors": [
    [
      "/v1/AUTH_test/test_cont/big_file.wav",
      "413 Request Entity Too Large"
    ]
  ],
  "Response Body": ""
}

Errors are presented as a list of tuples in the form [objectPath, errorResponse]. The objectPath given is the full path where the object was to be put. The errorResponse is the HTTP status of the response for that individual PUT operation. After 1,000 errors, processing of the request ends, and the completed response is returned.

If all valid files were uploaded successfully, the Response Status in the response body is 201 Created. If any files failed to be created, the Response Status corresponds to the subrequest’s error. Possible codes are 400, 401, and 502. In both cases, the Response Body specifies the number of files successfully uploaded and a list of the files that failed. (The actual HTTP status code is always 200 OK, so the Response Status in the response body is the one you should monitor.)

Note

If you send a Content-Type header on the PUT request, the specified Content-Type applies to every object in the archive. If you set Content-Type to a blank string, Cloud Files determines the Content-Type based on the individual file type. For example, if you have Multipurpose Internet Mail Extensions (MIME) type files, use a blank string for Content-Type to set the MIME type for files within the archive.

Bulk delete#

You can delete multiple objects or containers from an account by using a single bulk delete request, which is a DELETE request with the ?bulk-delete set query parameter. The Content-Type header of the request, if set, must be set to text/plain. You direct the request to the root of the service endpoints. The body of the DELETE request is a newline-separated list of URL-encoded objects to delete. You can delete 10,000 objects per request.

Note

This bulk operation involves middleware that conducts many operations on a single request.

The objects specified in the DELETE request body must be URL-encoded and in the following form:

/containerName/objectName

Containers (which must be empty at time of delete) have the following form:

/containerName

Create a text file similar to the following example, objects_to_delete.txt, with the names of the objects that you want to delete.

Example: Create text file for bulk delete request

$ cat objects_to_delete.txt
/containerName/objectName1
/containerName/objectName2
/containerName/objectName3
/containerName/objectName4

You can use a cURL request similar to the following example for the bulk delete.

Example: cURL request for the bulk delete

$ curl -i -XDELETE -H'x-auth-token: f064c46a782c444cb4ba4b6434288f7c' https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_000000\?bulk-delete -T ./objects_to_delete.txt

An HTTP request for the bulk delete is similar to the following example.

Example: HTTP request for the bulk delete

DELETE /v1/MossoCloudFS_000000?bulk-delete HTTP/1.1
Host: storage101.dfw1.clouddrive.com
x-auth-token:f064c46a782c444cb4ba4b6434288f7c
Content-Length: 108

/containerName/objectName1
/containerName/objectName2
/containerName/objectName3
/containerName/objectName4

The response is similar to the extract archive responses in that every response is 200 OK and the response body must be parsed for actual results. The following example shows the response body, formatted in JSON, from a successful request.

Example: Successful bulk delete response

HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
X-Trans-Id: tx52fe4601dde24e2b8e7cb-0051912ca8
Date: Thu, 23 Oct 2014 15:16:41 GMT
Transfer-Encoding: chunked

{
  "Number Not Found": 0,
  "Response Status": "200 OK",
  "Errors": [],
  "Number Deleted": 4,
  "Response Body": ""
}

The following example shows a response with errors.

Example: Bulk delete response with errors

HTTP/1.1 100 Continue

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
X-Trans-Id: tx28552a8cd9cb441dad3ad-0051912d2d
Date: Mon, 13 May 2013 18:13:01 GMT
Transfer-Encoding: chunked

{
  "Number Not Found": 0,
  "Response Status": "400 Bad Request",
  "Errors": [
    [
      "/v1/AUTH_test/non_empty_container",
      "409 Conflict"
    ]
  ],
  "Number Deleted": 0,
  "Response Body": ""
}

The list of errors is a list of tuples in the form [objectPath, errorResponse]. The objectPath is the full path of where the object (or container) was to be deleted. The errorResponse is the HTTP status of the response for that individual DELETE request.

If all items were successfully deleted (or did not exist), the Response Status is 200 OK. If any items failed to delete, the Response Status code corresponds to the subrequest’s error. Possible codes are 400, 401, and 502. In all cases, the Response Body specifies the number of items successfully deleted or not found as well as a list of those that failed. The return body is formatted in the way specified in the request’s Accept header. Acceptable formats are text/plain, application/json, application/xml, and text/xml.