Improving Go cookie handling

Many people argue, a lot of Go's popularity comes from developers moving to Go from Ruby & Python. And in a way, that may certainly be the case. Because Go, like Python famously, also has Batteries Included ®, one of the many reasons I love the language.

But like everything in the world, Go has its little idiosyncrasies. It's desperately trying to be RFC conformant with its cookie mechanism. Which is the reason, why certain cookies are just ignored by Go's cookiejar. In particular, Go just ignores cookies that contain characters like: \ (backslash) or " (double quotes).

Now there might have been good reasons to exclude these from the set of valid characters, i.e. it's obviously necessary to exclude a semi-colon, the cookie delimiter. But nevertheless, there may be situations in which Go will have to put up with servers sending cookie headers which are, acording to the RFC, just plain wrong and invalid.

And this is exactly the situation I encountered! For a Quizduell API library I was working on, I also had to talk to their 3rd-party server. And, you guessed it, they sent invalid cookies!

So after about 3 hours, the time it took me to debug why all API calls weren't authenticated, I came across an issue on the Go issue tracker, where a similar problem to mine was explained (basically, someone wanted to be able to have commas in cookies). And though the issue was resolved and a patch to allow commas in cookies was merged, the comments clearly indicated that the Go maintainers weren't happy with the idea of loosening the cookie character restrictions (although most browsers allow practically any character in a cookie).

Knowing that my problem was an acknowleged issue, I was still left with my problem, that I needed to fix. Looking for a solution, that was not copying the whole cookiejar implementation to a new package and just modifying this tiny function that checked wether a character is valid or not, I came up with the following solution:

cookie := response.Header.Get("Set-Cookie")
if cookie != "" {
    cookie = strings.Replace(cookie, "\\", "_", -1)
    response.Header.Set("Set-Cookie", cookie)
    c.Jar.SetCookies(request.URL, response.Cookies())
}

In the code example, c is the API client that has his own cookiejar, since the http client's jar is private.

Simply put, this snippet just takes the raw cookie string and replaces, in this case, the backslash with an underscore (a character known not to occur in cookies issued by the Quizduell servers), puts the cookie back into the http response and manually tells the API client's cookiejar to store the cookies of this particular response. Notice, you need your own cookiejar (c.Jar), since the one included in http clients is private.

The following snippet, that loads the cookies stored in our cookiejar and attaches them to a new http request, is a bit more complicated, because it needs to handle the case of there already being other cookies that were set previously on the request.

cookies := c.Jar.Cookies(request.URL)
if len(cookies) > 0 {
    for _, cookie := range cookies {
        s := cookie.Name + "=\"" + cookie.Value + "\""
        s = strings.Replace(s, "_", "\\", -1)

        if c := request.Header.Get("Cookie"); c != "" {
            request.Header.Set("Cookie", c+"; "+s)
        } else {
            request.Header.Set("Cookie", s)
        }
    }
}

But the idea is the same, load the cookie as a raw string, replace the underscores with the forbidden backslashes and manually attach the cookie as a header on the request.

And it works! Perfectly fine in fact. Though as you can guess, this is not the ideal solution for all possible situations. We quickly run into a problem if there's no character where we know that it won't occur in the cookie (in our case that was the underscore, incase you haven't noticed yet).

A possible workaround would be to base64 encode the relevant cookies and store the encoded version in the cookiejar.