Writing RESTful APIs in Go, 3 years later
I’ve started working with Go in early 2017, and since then most of my work has been writing RESTful APIs with it. With time I gained experience. and I constantly change the way I write APIs in Go. After a year of working with Go, I’ve released Gorsk — a Golang RESTful starter kit, and an update to it 6 months later. I get many emails and questions on how to use it properly, which means that something like Gorsk is highly needed. Over time I’d like to keep it up-to-date with my latest views on how to write REST APIs in Go, and this blog post serves as a first step in the next iteration.
Nowadays one of the most popular ways to interact with a server is via Web services that conform to the REST architectural style, called RESTful Web services. Most of the programming languages support HTTP calls out of the box, which makes integration with REST services a breeze.
With Go, you can write and organize your code in many different ways, there is no right or wrong. What matters the most is the code’s maintainability. It may be opinionated, but opinions backed by experience are way more valuable than those without.
Over the years I’ve worked with various clients that needed a REST API developed.
I’ve seen projects that used multiple ORM at the same time. Projects using git branches for differentiating user features based on tier. Projects connecting and disconnecting to a SQL database multiple times per HTTP request.
For most of the clients, I’ve implemented a project similar to Gorsk, with some changes I made over time. I’ll reflect on these changes in this blog post.
Initializing repository
As the Go Modules became the default choice for dependency management and gained some traction, I use it in all new projects should use it. Initializing a new module requires executing go init project_name, where project name is most often github.com/org/repo. After that, all the commands that are needed are go get (- u), and go mod tidy.
An important notice is that some tooling support is not yet there for projects outside of GoPath. (VSCode being the most prominent).
Project structure / package layout
The project structure I use nowadays is very similar to Gorsk’s. For projects that are starting small, I make it as flat as possible with main located in the root of the project, until it isn’t viable anymore. For larger projects, I start with main in cmd/ or cmd/serviceName/.
For a single domain, like users, I put all the code in user directory inside the root. The transport code is located in transport/http.go. If there are any other transports besides HTTP, like AMQP, RPC or SOAP it belongs here.
Code for interacting with other services, like database, email, 3rd party APIs is placed under /client/service. In Gorsk client is named platform. I’ve seen that Uber uses gateway instead of client, so you can name it whatever works for you.
Dependencies and interfaces for the application service are placed inside user/service.go, and the actual implementation is in user/user.go If the application service does more things than simple CRUD operations, I split the user.go into multiple files or even packages.
As for configuration, I keep using simple YAML files for nearly three years, and it serves me quite well. There are plenty of packages out there like Viper that support far more features than a single YAML file, but I never needed them. I store all the secrets as environment variables and manage them either manually or by a CI/CD tool.
Dependencies
Go’s standard library is quite young compared to others like Java, it enables you to write a web server without using any 3rd party library. However, it may not be the most practical way to do it, as some libraries ease up the process and reduce the amount of boilerplate code.
For the HTTP router, I’ve settled with chi. It uses Go’s standard HTTP handlers (w http.ResponseWriter, r *http.Request), unlike Echo and Gin. It comes with highly needed middleware packages (like logger, recoverer, requestID) and the provided render package, used for marshaling and unmarshaling HTTP requests works quite well.
Unless I’m using some platform-specific databases, like DynamoDB and Datastore, go-pg is my go-to library for database access. I’ve tried using database/sql and sqlx extensively, but go-pg’s feature-set and author’s support is second to none.
Logging and metrics
Having good logging is essential for every production application. It lets you quickly respond to any major and minor issues with provided error details. But, good logging is hard.
I log only requests that resulted in an unexpected error (unless the customer/client wants to log everything). All the expected outcomes, those return HTTP statuses between 200 and 500, are to be exported to a monitoring system like Prometheus, and later displayed in the dashboard such as Grafana. Good metrics provide excellent observability and indicate the usage patterns of your system.
My current solution for logging is wrapping it in middleware function, and writing to log if the HTTP status code is 500 or above.
Errors are logged in stdout and read by a service like GrayLog or LogStash from there.
As for errors, I use a small custom package to return HTTP status codes with error messages. It utilizes chi’s render package by implementing the Render interface. Also by implementing the Error interface, the Response type can be used where error is expected.
If an unexpected error happens, I wrap it using fmt.Errorf("something happened: %w", err), put it log and return 500 to the customer. All errors that are expected to happen return HTTP status codes in range 400-410, and are not being logged.
Serverless
All of the above applies if you’re writing a standard HTTP backend in Go. Lately, I’ve been writing more and more Serverless APIs, most of them running on Lambda. The structure of those, as well as dependencies, differ far from a standard Go backend. Thus this article is not applicable if for serverless backends.
