Multipart Requests in Go
development golangMIME and Multipart
Multipurpose Internet Mail Extensions (MIME) is an internet standard that extends the format of emails to support non-ASCII character sets, messages with multipart parts and non-text attachments like images, video, binary files etc. Multipart originates from MIME and a multipart message is a list of parts. Each part contains headers and a body. The MIME standard defines various multipart messages subtypes like mixed, digest, related, form-data, byteranges, encrypted and a few others. Form-Data is most commonly used for submitting files via HTTP and is normally used to express values submitted through a form. This is what browsers use to upload files through HTML forms. The multipart/related subtype is for compound objects where each message part is a component of an aggregate whole. This is commonly used to send a web page complete with images in a single message.
Normally, the content-type header of a multipart message contains multipart/ as a prefix followed by the appropriate subtype. For instance, a form-data multipart request will use multipart/form-data as the content type. Where as, a multipart related request will use multipart/related as the content type.
Multipart form data
Normally, when a form is submitted through the browser, it will use application/x-www-form-urlencoded content-type, which is just a list of keys and values. This is not for uploading files and therefore, that’s where multipart/form-data content-type comes in. When this content-type is used, the browser will create a multipart message where each part corresponds to a field on the form. Each part will be separated by MIME boundaries. A typical multipart/form-data request would look like this,
POST /upload HTTP/1.1
Content-Type: multipart/form-data; boundary=foo_bar_baz
Content-Length: [NUMBER_OF_BYTES_IN_ENTIRE_REQUEST_BODY]
Accept: application/json
--foo_bar_baz
Content-Disposition: form-data; name="metadata"
Content-Type: application/json; charset=UTF-8
{
"name": "file.jpg",
"title": "A new image"
}
--foo_bar_baz
Content-Disposition: form-data; name="media"; filename="file.jpg"
Content-Type: image/jpeg
[JPEG_DATA]
--foo_bar_baz--
Reading Multipart Form Data on Server
In Go, on the server side, we can use ParseMultipartForm to read a multipart/form-data message. This parses the whole request body and stores up to 32MB in memory, while the rest is stored on disk in temporary files. If it is unable to parse the multipart message, it will return an error.
// Here r is *http.Request
parseErr := r.ParseMultipartForm(32 << 20) // maxMemory 32MB
if parseErr != nil {
http.Error(w, "failed to parse multipart message", http.StatusBadRequest)
return
}
To actually retrieve the request body, we will use r.MultipartForm, which contains the parsed multipart form including files uploaded. This field is available only after ParseMultipartForm is called. r.MultipartForm.File is a hashmap, map[string][]*FileHeader, where the key is the form name. We can use this to retrieve each file that is uploaded by the user using multipart/form-data. There is another convenient method FormFile that takes the form name as an argument and returns the first file for the provided form key. Here, we will use FormFile to retrieve metadata and r.MultipartForm.File to retrieve media files. In the code snippets here, I have avoided error handling to keep it simple.
// Metadata
f, _, _ := r.FormFile("metadata")
metadata, _ := ioutil.ReadAll(f)
// Media files
for _, h := range r.MultipartForm.File["media"] {
file, _ := h.Open()
tmpfile, _ := os.Create("./" + h.Filename)
io.Copy(tmpfile, file)
tmpfile.Close()
file.Close()
}
POST Multipart Form-Data Request
It is quite easy to send a multipart/form-data request from browser using XMLHttpRequest (XHR). We can use FormData() to create a multipart/form-data request and use XHR to POST the request to the server. Here is a sample code.
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
</head>
<body>
<div>
<label for="title">Title</label>
<input type="text" id="title" name="title" /><br>
<label for="media-file">Media File</label>
<input type="file" id="media-file" name="media-file" /><br>
<button id="submit">Upload</button>
</div>
<script src="https://code.jquery.com/jquery-3.3.1.slim.min.js" integrity="sha384-q8i/X+965DzO0rT7abK41JStQIAqVgRVzpbzo5smXKp4YfRvH+8abtTE1Pi6jizo" crossorigin="anonymous"></script>
<script>
$("#submit").click(function() {
var media = document.getElementById('media-file');
if (media.files.length != 1) {
alert("select a media file to upload");
return;
}
var metadata = {
'name': media.files[0].name,
'title': document.getElementById('title').value
};
var form = new FormData();
form.append('metadata', new Blob([JSON.stringify(metadata)], {type: 'application/json'}));
form.append('media', media.files[0]);
var xhr = new XMLHttpRequest();
xhr.open('post', 'https://example.com/upload');
xhr.responseType = 'json';
xhr.setRequestHeader('Authorization', 'Bearer auth-token-if-any')
xhr.onload = () => {
console.log(xhr.response.id);
};
xhr.send(form);
})
</script>
</body>
</html>
Reading Multipart Related on Server
ParseMultipartForm only supports multipart/form-data. In case of multipart/related, we can use mime and mime/multipart packages to read the message. We will first have to parse the content-type header to determine if the request is actually a multipart message. We can do this using the ParseMediaType function in the mime package.
// Here r is *http.Request
contentType, params, parseErr := mime.ParseMediaType(r.Header.Get("Content-Type"))
if err != nil || !strings.HasPrefix(contentType, "multipart/") {
http.Error(w, "expecting a multipart message", http.StatusBadRequest)
return
}
This returns the media type converted to lowercase and a non-nil map. If there is an error while parsing, it will return an error along with the media type. Params will contain the mime boundary string among other things. We will then use this mime boundary string and mime/multipart package to read the body of the request.
multipartReader := multipart.NewReader(r.Body, params["boundary"])
defer r.Body.Close()
Now that we have the multipart reader, we can loop over each part of the multipart request and handle them accordingly. To obtain the next part, use multipartReader.NextPart() method. This method returns the next part of the request and an error. After the last part of the request is returned, this method will return io.EOF error. We will use this to exit the loop.
Note that each part will contain a content-type header of it’s own, indicating the type of content that is provided by the user in that particular part. Based on our sample request above, the server expects a part with json metadata and a part with image data. Each part will also contain a Content-Disposition header. This header field will provide the presentation style (inline or attachment) along with the file name. If each part in the multipart/related message has a Content-ID header that indicates whether the part contains metadata or media file, the server can use it to process the content accordingly.
for {
part, err := multipartReader.NextPart()
if err == io.EOF {
break
}
if err != nil {
http.Error(w, "unexpected error when retrieving a part of the message", http.StatusInternalServerError)
return
}
defer part.Close()
fileBytes, err := ioutil.ReadAll(part)
if err != nil {
http.Error(w, "failed to read content of the part", http.StatusInternalServerError)
return
}
switch part.Header.Get("Content-ID") {
case "metadata":
log.Print(string(fileBytes))
case "media":
log.Printf("filesize = %d", len(fileBytes))
f, _ := os.Create(part.Header.Get("Content-Filename"))
f.Write(fileBytes)
f.Close()
}
}
POST Multipart Related Request
On the client side, we will use mime/multipart and textproto packages to create a multipart/related request. Here is a snippet that shows how this is done.
mediaFiles := []string{"image1.jpg", "image2.png"}
// Metadata content.
metadata := `{"title": "New title", "description": "New description"}`
// New multipart writer.
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
// Metadata part.
metadataHeader := textproto.MIMEHeader{}
metadataHeader.Set("Content-Type", "application/json")
metadataHeader.Set("Content-ID", "metadata")
part, _ := writer.CreatePart(metadataHeader)
part.Write([]byte(metadata))
// Media Files.
for _, mediaFilename := range mediaFiles {
mediaData, _ := ioutil.ReadFile(mediaFilename)
mediaHeader := textproto.MIMEHeader{}
mediaHeader.Set("Content-Disposition", fmt.Sprintf("attachment; filename=\"%v\".", mediaFilename))
mediaHeader.Set("Content-ID", "media")
mediaHeader.Set("Content-Filename", mediaFilename)
mediaPart, _ := writer.CreatePart(mediaHeader)
io.Copy(mediaPart, bytes.NewReader(mediaData))
}
// Close multipart writer.
writer.Close()
// Request Content-Type with boundary parameter.
contentType := fmt.Sprintf("multipart/related; boundary=%s", writer.Boundary())
uploadURL := "http://example.com/upload"
r, _ := http.NewRequest(http.MethodPost, uploadURL, bytes.NewReader(body.Bytes()))
r.Header.Set("Content-Type", contentType)
client := &http.Client{Timeout: 180 * time.Second}
rsp, _ := client.Do(r)
if rsp.StatusCode != http.StatusOK {
log.Printf("Request failed with response code: %d", rsp.StatusCode)
}
If you prefer javascript, take a look at request library. There is an open support request for axios library which at the time of writing this post is not yet resolved.
Complete Source Code
If you are interested in referring the complete source code, please visit multipart-requests-in-golang
References
- https://en.wikipedia.org/wiki/MIME
- https://www.w3.org/Protocols/rfc1341/0_TableOfContents.html
- https://golang.org/pkg/mime/
- https://golang.org/pkg/mime/multipart/
- https://github.com/abvarun226/blog-source-code/tree/master/multipart-requests-in-go
In case you would like to get notified about more articles like this, please subscribe to my substack.