This is the story of how I wrote an S3 Go Proxy.
I was recently working on a project that needed a scalable storage solution. Since I planned on hosting this project on DigitalOcean I originally thought about using Block Storage. While Block Storage is fast, it isn’t easily scalable, and I don’t want to be responsible for resizing/adding storage. It didn’t take long to realize the ideal solution would be Spaces. Another requirement is restricting access to these files, which I’m still working on.
I’ve learned from years of experience that the majority of the people don’t pay attention to the address bar. Even knowing that, I still wanted the URLs to be branded with the domain of the project.
I could use the DigitalOcean Spaces CDN, but I needed to know where the files were. Restricting access to the files with DO’s Spaces CDN would have been easy as well since it is an S3 compatible Object Storage.
S3 Proxy in Go
Before I’ve mounted Object Storage as a local file system on Linux’s servers, but it’s always a headache to get it working properly. This time I decided to build a proxy in Go since I’ve yet to write anything useful in that language. It would use the AWS S3 Go SDK and act as a HTTP server.
Getting the proxy to work was fairly easy, but making the proxy efficient was a challenge. I imagine that there are still ways it can be improved. The buffers, reading, and writing could all be faster (I think).
This web app has thumbnails of thousands of images and I didn’t want the browser downloading them each time they load the page. I handled this by modifying the response headers for the request.
Checking if a browser has permission to request a file is an interesting challenge. I imagine it’ll involve some cookies or maybe a database query. A cookie will probably be faster, but it may be harder to implement. I’ve been trying to keep the number of packages I use down to as few as possible, but the Gorilla ones are helpful.
I enjoy writing Go. I do wish they would have named it something else, because it’s really hard to search for “Go.” I usually have to search for Golang. When I have the opportunity to make a solution open-source I always do.