Resumable Media Uploads in the Google Data Protocol

Eric Bidelman, G Suite APIs team
February 2010

  1. Introduction
  2. The Resumable Protocol
    1. Initiating a resumable upload request
    2. Uploading a file
    3. Resuming an upload
    4. Canceling an upload
    5. Updating an existing resource
  3. Client library examples

Introduction

Current web standards provide no reliable mechanism to facilitate the HTTP upload of large files. As a result, file uploads at Google and other sites have traditionally been limited to moderate sizes (e.g. 100 MB). For services like the YouTube and the Google Documents List APIs which support large file uploads, this presents a major hurdle.

The Google Data resumable protocol directly addresses the aforementioned issues by supporting resumable POST/PUT HTTP requests in HTTP/1.0. The protocol was modeled after the ResumableHttpRequestsProposal suggested by Google Gears team.

This document describes how to incorporate Google Data's resumable upload feature into your applications. The examples below use the Google Documents List Data API. Note that additional Google APIs that implement this protocol may have slightly different requirements/response codes/etc. Please consult the service's documentation for the specifics.

The Resumable Protocol

Initiating a resumable upload request

To initiate a resumable upload session, send an HTTP POST request to the resumable-post link. This link is found at the feed level. The DocList API's resumable-post link looks like:

<link rel="http://schemas.google.com/g/2005#resumable-create-media" type="application/atom+xml"
    href="https://docs.google.com/feeds/upload/create-session/default/private/full"/>

The body of your POST request should be empty or contain an Atom XML entry and must not include the actual file contents. The example below creates a resumable request to upload a large PDF, and includes a title for the future document using the Slug header.

POST /feeds/upload/create-session/default/private/full HTTP/1.1
Host: docs.google.com
GData-Version: version_number
Authorization: authorization
Content-Length: 0
Slug: MyTitle
X-Upload-Content-Type: content_type
X-Upload-Content-Length: content_length

empty body

The X-Upload-Content-Type and X-Upload-Content-Length headers should be set to the mimetype and size of the file you will eventually upload. If the content length is unknown at the creation of the upload session, the X-Upload-Content-Length header can be omitted.

Here is another example request that instead uploads a word document. This time, Atom metadata is included and will be applied to the final document entry.

POST /feeds/upload/create-session/default/private/full?convert=false HTTP/1.1
Host: docs.google.com
GData-Version: version_number
Authorization: authorization
Content-Length: atom_metadata_content_length
Content-Type: application/atom+xml
X-Upload-Content-Type: application/msword
X-Upload-Content-Length: 7654321

<?xml version='1.0' encoding='UTF-8'?>
<entry xmlns="http://www.w3.org/2005/Atom" xmlns:docs="http://schemas.google.com/docs/2007">
  <category scheme="http://schemas.google.com/g/2005#kind"
      term="http://schemas.google.com/docs/2007#document"/>
  <title>MyTitle</title>
  <docs:writersCanInvite value="false"/>
</entry>

The server's response from the initial POST is a unique upload URI in the Location header and an empty response body:

HTTP/1.1 200 OK
Location: <upload_uri>

The unique upload URI will be used to upload the file chunks.

Note: The initial POST request does not create a new entry in the feed. This only happens when the entire upload operation has completed.

Note: A resumable session URI expires after one week.

Uploading a file

The resumable protocol allows, but doesn't require, content to be uploaded in 'chunks', because there are no inherent restrictions in HTTP on request sizes. Your client is free to choose its chunk size or just upload the file as a whole. This example uses the unique upload URI to issue a resumable PUT. The following example sends the first 100000 bytes of 1234567 byte PDF file:

PUT upload_uri HTTP/1.1
Host: docs.google.com
Content-Length: 100000
Content-Range: bytes 0-99999/1234567

bytes 0-99999

If the size of the PDF file was unknown, this example would use Content-Range: bytes 0-99999/*. Read more information on the Content-Range header here.

Server responds with the current byte range that has been stored:

HTTP/1.1 308 Resume Incomplete
Content-Length: 0
Range: bytes=0-99999

Your client should continue to PUT each chunk of the file until the entire file has been uploaded. Until the upload is complete, the server will respond with an HTTP 308 Resume Incomplete and the byte range it knows about in the Range header. Clients must use the Range header to determine where to start the next chunk. Therefore, do not assume that the server received all bytes originally sent in the PUT request.

Note: The server may issue a new unique upload URI in the Location header during a chunk. Your client should check for an updated Location and use that URI to send the remaining chunks to the server.

When the upload is complete, the response will be the same as if the upload had been made using the API's non-resumable upload mechanism. That is to say, a 201 Created will be returned along with the <atom:entry>, as created by the server. Subsequent PUTs to the unique upload URI will return the same response as what was returned when the upload completed. After a period of time, the response will be 410 Gone or 404 Not Found.

Resuming an upload

If your request is terminated prior to receiving a response from the server or if you receive an HTTP 503 response from the server, you can query the current status of the upload by issuing an empty PUT request on the unique upload URI.

Client polls the server to determine which bytes it has received:

PUT upload_uri HTTP/1.1
Host: docs.google.com
Content-Length: 0
Content-Range: bytes */content_length

Use * as the content_length if the length is not known.

Server responds with the current byte range:

HTTP/1.1 308 Resume Incomplete
Content-Length: 0
Range: bytes=0-42

Note: If the server has not committed any bytes for the session, it will omit the Range header.

Note: The server may issue a new unique upload URI in the Location header during a chunk. Your client should check for an updated Location and use that URI to send the remaining chunks to the server.

Finally, the client resumes where the server left off:

PUT upload_uri HTTP/1.1
Host: docs.google.com
Content-Length: 57
Content-Range: 43-99/100

<bytes 43-99>

Canceling an upload

If you want to cancel the upload and prevent any further action on it, issue a DELETE request on the unique upload URI. 

DELETE upload_uri HTTP/1.1
Host: docs.google.com
Content-Length: 0

If successful, the server responds that the session is canceled, and responds with the same code for further PUTs or query status requests:

HTTP/1.1 499 Client Closed Request

Note: If an upload is abandoned without cancelation, it naturally expires one week after creation.

Updating an existing resource

Similar to initiating a resumable upload session, you can utilize the resumable upload protocol to replace an existing file's content. To start a resumable update request, send an HTTP PUT to the entry's link with rel='...#resumable-edit-media'. Each media entry will contain such a link if the API supports updating the resource's content.

As an example, a document entry in the DocList API will contain a link similar to:

<link rel="http://schemas.google.com/g/2005#resumable-edit-media" type="application/atom+xml"
      href="https://docs.google.com/feeds/upload/create-session/default/private/full/document%3A12345"/>

Thus, the initial request would be:

PUT /feeds/upload/create-session/default/private/full/document%3A12345 HTTP/1.1
Host: docs.google.com
GData-Version: version_number
Authorization: authorization
If-Match: ETag | *
Content-Length: 0
X-Upload-Content-Length: content_length
X-Upload-Content-Type: content_type

empty body

To update a resource's metadata and content at the same time, include Atom XML instead of an empty body. See the example in the Initiating a resumable upload request section.

When the server responds with the unique upload URI, send a PUT with your payload. Once you have the unique upload URI, the process for updating the file's content is the same as uploading a file.

This particular example will update the existing document's content in one shot:

PUT upload_uri HTTP/1.1
Host: docs.google.com
Content-Length: 1000
Content-Range: 0-999/1000

<bytes 0-999>

Back to top

Client library examples

Below are samples of uploading a movie file to Google Docs (using the resumable upload protocol) in the Google Data client libraries. Note, that not all of the libraries support the resumable feature at this time.

int MAX_CONCURRENT_UPLOADS = 10;
int PROGRESS_UPDATE_INTERVAL = 1000;
int DEFAULT_CHUNK_SIZE = 10485760;


DocsService client = new DocsService("yourCompany-yourAppName-v1");
client.setUserCredentials("user@gmail.com", "pa$$word");

// Create a listener
FileUploadProgressListener listener = new FileUploadProgressListener(); // See the sample for details on this class.

// Pool for handling concurrent upload tasks
ExecutorService executor = Executors.newFixedThreadPool(MAX_CONCURRENT_UPLOADS);

// Create {@link ResumableGDataFileUploader} for each file to upload
List uploaders = Lists.newArrayList();

File file = new File("test.mpg");
String contentType = DocumentListEntry.MediaType.fromFileName(file.getName()).getMimeType();
MediaFileSource mediaFile = new MediaFileSource(file, contentType);
URL createUploadUrl = new URL("https://docs.google.com/feeds/upload/create-session/default/private/full");
ResumableGDataFileUploader uploader = new ResumableGDataFileUploader(createUploadUrl, mediaFile, client, DEFAULT_CHUNK_SIZE,
                                                                     executor, listener, PROGRESS_UPDATE_INTERVAL);
uploaders.add(uploader);

listener.listenTo(uploaders); // attach the listener to list of uploaders

// Start the upload(s)
for (ResumableGDataFileUploader uploader : uploaders) {
  uploader.start();
}

// wait for uploads to complete
while(!listener.isDone()) {
  try {
    Thread.sleep(100);
  } catch (InterruptedException ie) {
    listener.printResults();
    throw ie; // rethrow
  }

// Chunk size in MB
int CHUNK_SIZE = 1;

ClientLoginAuthenticator cla = new ClientLoginAuthenticator(
    "yourCompany-yourAppName-v1", ServiceNames.Documents, "user@gmail.com", "pa$$word");

// Set up resumable uploader and notifications
ResumableUploader ru = new ResumableUploader(CHUNK_SIZE);
ru.AsyncOperationCompleted += new AsyncOperationCompletedEventHandler(this.OnDone);
ru.AsyncOperationProgress += new AsyncOperationProgressEventHandler(this.OnProgress);

// Set metadata for our upload.
Document entry = new Document()
entry.Title = "My Video";
entry.MediaSource = new MediaFileSource("c:\\test.mpg", "video/mpeg");

// Add the upload uri to document entry.
Uri createUploadUrl = new Uri("https://docs.google.com/feeds/upload/create-session/default/private/full");
AtomLink link = new AtomLink(createUploadUrl.AbsoluteUri);
link.Rel = ResumableUploader.CreateMediaRelation;
entry.DocumentEntry.Links.Add(link);

ru.InsertAsync(cla, entry.DocumentEntry, userObject);

- (void)uploadAFile {
  NSString *filePath = @"~/test.mpg";
  NSString *fileName = [filePath lastPathComponent];

  // get the file's data
  NSData *data = [NSData dataWithContentsOfMappedFile:filePath];

  // create an entry to upload
  GDataEntryDocBase *newEntry = [GDataEntryStandardDoc documentEntry];
  [newEntry setTitleWithString:fileName];

  [newEntry setUploadData:data];
  [newEntry setUploadMIMEType:@"video/mpeg"];
  [newEntry setUploadSlug:fileName];

  // to upload, we need the entry, our service object, the upload URL,
  // and the callback for when upload has finished
  GDataServiceGoogleDocs *service = [self docsService];
  NSURL *uploadURL = [GDataServiceGoogleDocs docsUploadURL];
  SEL finishedSel = @selector(uploadTicket:finishedWithEntry:error:);

  // now start the upload
  GDataServiceTicket *ticket = [service fetchEntryByInsertingEntry:newEntry
                                                        forFeedURL:uploadURL
                                                          delegate:self
                                                 didFinishSelector:finishedSel];

  // progress monitoring is done by specifying a callback, like this
  SEL progressSel = @selector(ticket:hasDeliveredByteCount:ofTotalByteCount:);
  [ticket setUploadProgressSelector:progressSel];
}

// callback for when uploading has finished
- (void)uploadTicket:(GDataServiceTicket *)ticket
   finishedWithEntry:(GDataEntryDocBase *)entry
               error:(NSError *)error {
  if (error == nil) {
    // upload succeeded
  }
}

- (void)pauseOrResumeUploadForTicket:(GDataServiceTicket *)ticket {
  if ([ticket isUploadPaused]) {
    [ticket resumeUpload];
  } else {
    [ticket pauseUpload];
  }
}

import os.path
import atom.data
import gdata.client
import gdata.docs.client
import gdata.docs.data

CHUNK_SIZE = 10485760

client = gdata.docs.client.DocsClient(source='yourCompany-yourAppName-v1')
client.ClientLogin('user@gmail.com', 'pa$$word', client.source);

f = open('test.mpg')
file_size = os.path.getsize(f.name)

uploader = gdata.client.ResumableUploader(
    client, f, 'video/mpeg', file_size, chunk_size=CHUNK_SIZE, desired_class=gdata.docs.data.DocsEntry)

# Set metadata for our upload.
entry = gdata.docs.data.DocsEntry(title=atom.data.Title(text='My Video'))
new_entry = uploader.UploadFile('/feeds/upload/create-session/default/private/full', entry=entry)
print 'Document uploaded: ' + new_entry.title.text
print 'Quota used: %s' % new_entry.quota_bytes_used.text

For complete samples and source code reference, see the following resources:

Back to top