While APIs are relatively easy to build, they're hard to build well. You'll often hear developers air their grievances about APIs that are used by hundreds of thousands, if not millions of developers every day. Many of these APIs are developed by companies that are publicly traded with huge engineering teams - Facebook, LinkedIn, Microsoft, the list goes on. At the same time, you'll hear developers rave about APIs like Stripe.
There are many ingredients that go into a "great" API - clear documentation, reliability, explicit error messaging, and a simple, yet powerful interface. Companies like Stripe put a lot of thought and consideration into these details. Their users, after all, are other developers and providing a great experience gives them a competitive edge.
Once an API is public and users depend on it, the stakes get higher. Changes will need to be made carefully and with backwards compatibility in mind. It's worth the extra time and effort during the design phase to think about the ergonomics of your API and how it might evolve over time. There’s a relevant observation made by Hyrum Wright called Hyrum's Law:
With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.
Said another way:
Given enough use, there is no such thing as a private implementation. That is, if an interface has enough consumers, they will collectively depend on every aspect of the implementation, intentionally or not. This effect serves to constrain changes to the implementation, which must now conform to both the explicitly documented interface, as well as the implicit interface captured by usage. We often refer to this phenomenon as "bug-for-bug compatibility.”
It's fair to say that having a solid foundation will pave the way to a great API experience. In this guide, I'll discuss some design patterns and ideas that you can use in your public-facing APIs. As always, your mileage may vary, but this is a great starting point that should evolve with your ever-changing requirements. Examples will be given using Ruby on Rails, but the strategies are framework agnostic.
Most API implementations will need a way to authenticate clients. There are many ways to authenticate clients to your API. By far the easiest and most common is Bearer Authentication.
Bearer authentication (also called token authentication) is an HTTP authentication scheme that involves security tokens called bearer tokens. The name “Bearer authentication” can be understood as “give access to the bearer of this token.”
With Bearer Authentication a token is sent in the Authorization
header when making a request like so:
$ curl https://example.com \-H "Authorization: Bearer {token}"
Bearer Authentication should be used over HTTPS so that the bearer tokens are encrypted in transit and not exposed in plain text.
When using Bearer Authentication, it’s best to use the Authorization
header. Specifications for the internet (RFCs) specify how the Authorization
header should be treated by caches and proxies. To be more specific:
Authorization
headers must not be cached by default.So by using the Authorization
header you’ll get the benefits of the behavior defined by RFC. If you were to use a custom header, there would be no guarantees and a proxy could be inadvertantly caching or logging your tokens.
To find examples of Bearer Authentication in the wild, you only need to look to some of the most popular developer services - Stripe, Terraform Cloud, and Heroku. It’s also worth noting that the Bearer token scheme is used in the Oauth 2.0 protocol. In fact, that’s what it was designed for! It’s use has since evolved beyond just Oauth 2.0.
Bearer Authentication will be the focus of this guide, but depending on your use case, there are other options available. Another commonly used scheme is Basic Authentication.
Basic Authentication is similar to Bearer Authentication in that it also uses the Authorization
header, but it’s different in that it’s meant to authenticate using a username and password, rather than a single token. Basic Authentication also specifies a way to pass the username and password in the URL, but this is now deprecated behavior. If you plan to authenticate API clients using both a username and password, Basic Authentication can be a good option.
Since we’ll be using Bearer Authentication, we’ll need tokens for our bearers to authenticate to our API. These are often called API Keys. How we model, store, present, and format our API Keys is an important piece of the puzzle. Some of the ideas in this section are inspired by Zeke Gabrielse’s great blog post, How to Implement API Key Authentication in Rails Without Devise.
Let’s create an ApiKey
model and migration by running:
$ rails g model ApiKey
Then let’s add the following contents to the model and migration:
db/migrate/20220101230637_create_api_keys.rbclass CreateApiKeys < ActiveRecord::Migration[7.0]def changecreate_table :api_keys do |t|t.belongs_to :bearer, polymorphic: true# More columns to be added and discussed belowt.timestampsendendend
app/models/api_key.rbclass ApiKey < ApplicationRecordbelongs_to :bearer, polymorphic: trueend
We’ve created an ApiKey
model that belongs to a bearer. The relationship is polymorphic because it can be useful to have multiple types of bearers. For example, let’s say we have Organization, Team, and User models. With a polymorphic relationship, Organizations, Teams, and Users can all be bearers with different scopes and permissions applied to each. More on this below.
One thing that’s missing from the code above is the actual token that we provide the bearer in order to authenticate. Let’s adjust our migration and model like so:
db/migrate/20220101230637_create_api_keys.rbclass CreateApiKeys < ActiveRecord::Migration[7.0]def changecreate_table :api_keys do |t|t.belongs_to :bearer, polymorphic: truet.string :token, null: falset.timestampsendadd_index :api_keys, :token, unique: trueendend
app/models/api_key.rbclass ApiKey < ApplicationRecordbelongs_to :bearer, polymorphic: truebefore_create :generate_tokenprivatedef generate_tokenself.token = SecureRandom.base58(30)endend
Now we’re generating tokens which we can share with our bearer in order to authenticate. The problem, however, is that we should treat these tokens like passwords and we’re currently storing them as plaintext.
We have a couple options for storing these tokens safely - encrypting or hashing. If we ever need to retrieve the original, plaintext value, encryption would be our best bet because we can decrypt at-will. Hashing, however, is a one-way function. It’s a more secure option because it can’t be reversed. This is why it’s often used in authentication. In our case, we only need the plaintext value after creation to share with our users. After that, we can retrieve the API key from our database by querying for the hashed token value since our users will be sending the plaintext value in the Authorization
header.
Let’s tweak our ApiKey
model and migration to incorporate hashing our tokens before saving:
db/migrate/20220101230637_create_api_keys.rbclass CreateApiKeys < ActiveRecord::Migration[7.0]def changecreate_table :api_keys do |t|t.belongs_to :bearer, polymorphic: truet.string :token_digest, null: falset.timestampsendadd_index :api_keys, :token, unique: trueendend
app/models/api_key.rbclass ApiKey < ApplicationRecordHMAC_SECRET_KEY = Rails.application.credentials.api_key_hmac_secret_keybelongs_to :bearer, polymorphic: truebefore_create :generate_raw_tokenbefore_create :generate_token_digest# Attribute for storing and accessing the raw (non-hashed)# token value directly after creationattr_accessor :raw_tokendef self.find_by_token!(token)find_by!(token_digest: generate_digest(token))enddef self.find_by_token(token)find_by(token_digest: generate_digest(token))enddef self.generate_digest(token)OpenSSL::HMAC.hexdigest("SHA256", HMAC_SECRET_KEY, token)endprivatedef generate_raw_tokenself.raw_token = SecureRandom.base58(30)enddef generate_token_digestself.token_digest = self.class.generate_digest(raw_token)endend
In the example above, we’re using HMAC with the SHA-256 hash function to hash the raw token value that we generate via SecureRandom.base58. HMAC requires a secret key, which you can generate in the Rails console and store in your rails credentials file (or as an ENV variable):
$ rails c> SecureRandom.hex(32)=> "f8e9dfcf97e73529b000fffd14f43627cdd74da691fd7a7d4fa3526b73a16041"$ rails credentials:edit
config/credentials.yml.encapi_key_hmac_secret_key: f8e9dfcf97e73529b000fffd14f43627cdd74da691fd7a7d4fa3526b73a16041
Note that you’ll want to generate a different key for each environment - development, staging, production, etc.
Now we have an ApiKey
model that’s generating tokens and hashing the token before storing it in the database! You may be asking yourself though... How do we share the plaintext version with our users after creating an ApiKey
?
In the code example above, we have ApiKey#raw_token
which will give us the plaintext value of our token right after we create a new API key. This value will be available in memory within our new ApiKey
instance directly after creation. For example:
irb(main):> user = User.firstirb(main):> api_key = ApiKey.create(bearer: user)irb(main):> api_key.id=> 1irb(main):> api_key.raw_token=> 6NCTx2TxTHGkbNLVeChfp8et6cm83a# Fetch the record we created above from the Database.irb(main):> api_key = ApiKey.find(1)# raw_token will be nil because it's only available directly after# creationirb(main):> api_key.raw_token=> nil
This essentially means that in our controller, the common pattern of redirecting after creating an ApiKey
won’t work because raw_token
will no longer be available after we redirect. We have to send (render) the raw_token
value in the same request in which we create an ApiKey
.
There are many ways to handle this. You could use AJAX to make a request to ApiKeysController#create
which would respond with a JSON object including the raw_token
:
config/credentials.yml.encclass ApiKeysController < ApplicationControllerdef create@api_key = ApiKey.new(bearer: current_user)if @api_key.saverender json: { raw_token: @api_key.raw_token }elserender status: :unprocessable_entity, json: {errors: @api_key.errors.full_messages}endendend
Or if you’re using Turbo, you can respond with a Turbo Stream that injects the raw_token
into the UI. For example, if we wanted to respond with a Turbo Stream that appended the API key to the DOM, we could have a create.turbo_stream.erb
template:
app/views/api_keys/create.turbo_steam.erb<%= turbo_stream.append "api-keys" do %><%= render "api_key", api_key: @api_key %><% end %>
app/views/api_keys/_api_key.html.erb<li>Api Key: <%= @api_key.raw_token || "*******************" %></li>
After creating an ApiKey
, subsequent requests to render the ApiKey
in the UI won’t be able to display the plaintext token (raw_token
) - we’ll only have the irreversibly hashed version of our token, which we are storing in the token_digest
column. In the example above for rendering an ApiKey
using the _api_key.html.erb
partial, our raw_token
would display as *******************
on subsequent renders. This isn’t a great user experience and makes it difficult for users to identify active tokens. What if we wanted to show the prefix or suffix of a token without revealing the entire thing? Stripe does exactly this after you generate a new API key and reveal it for the one and only time:
Another nice thing Stripe does is that they format their API keys and secrets with an identifiable prefix (sk_*_
). When you’re managing multiple API keys for different services, this makes it very easy to identify which API keys belong to which service. Moreover, when there are multiple types of tokens available for an individual service, having different prefixes for each type makes it easy to identify the token’s purpose. For example, stripe uses sk_live
for live mode tokens and sk_test
for test mode tokens. If this wasn’t the case, it would be difficult, if not impossible, to distinguish a live token from a test token. As you can imagine, this might lead to some problems! Slack and Github use a similar pairadigm for their API tokens as well.
You’ll also notice the usage of _
as a separator in Stripe’s tokens. This makes the prefix clearly distinguishable because it’s not part of the character set used to generate the random part of the token. There’s another less obvious benefit of using _
as the separator - you can select the entire token for copying by double clicking on it. For example, try double clicking on my_token
with your mouse. You’ll highlight the entire token. Now try double clicking on my-token
. You can’t highlight the entire string!
Another nice thing about using an identifiable prefix is that you can use secret scanning on services like Github:
GitHub scans repositories for known secret formats to prevent fraudulent use of credentials that were committed accidentally. Secret scanning happens by default on public repositories, and can be enabled on private repositories by repository administrators or organization owners. As a service provider, you can partner with GitHub so that your secret formats are included in our secret scanning.
Basically, this means that if a user of your API accidentally commits a sensitive API token to their Github repo, your service can be notified by Github. Once you’re notified, you can auto-revoke the compromised token and notify the user, saving them from catastrophe!
With all this in mind, let’s refactor our ApiKey
example such that we have an identifiable prefix and the ability to display a handful of characters from the random part of our token while keeping the rest redacted.
db/migrate/20220101230637_create_api_keys.rbclass CreateApiKeys < ActiveRecord::Migration[7.0]def changecreate_table :api_keys do |t|t.belongs_to :bearer, polymorphic: truet.string :common_token_prefix, null: falset.string :random_token_prefix, null: falset.string :token_digest, null: falset.timestampsendadd_index :api_keys, :token_digest, unique: trueendend
app/models/api_key.rbclass ApiKey < ApplicationRecordHMAC_SECRET_KEY = Rails.application.credentials.api_key_hmac_secret_keyTOKEN_NAMESPACE = "tkn"# Encrypt random token prefix using Active Record Encryption# https://guides.rubyonrails.org/active_record_encryption.html# We're using deterministic encryption so that we can add uniqueness# validation below.encrypts :random_token_prefix, deterministic: truebelongs_to :bearer, polymorphic: truebefore_validation :set_common_token_prefix, on: :createbefore_validation :generate_random_token_prefix, on: :createbefore_validation :generate_raw_token, on: :createbefore_validation :generate_token_digest, on: :createvalidates_uniqueness_of :random_token_prefix, scope: [:bearer_id, :bearer_type]# Attribute for storing and accessing the raw (non-hashed)# token value directly after creationattr_accessor :raw_tokendef self.find_by_token!(token)find_by!(token_digest: generate_digest(token))enddef self.find_by_token(token)find_by(token_digest: generate_digest(token))enddef self.generate_digest(token)OpenSSL::HMAC.hexdigest("SHA256", HMAC_SECRET_KEY, token)enddef token_prefix"#{common_token_prefix}#{random_token_prefix}"endprivate# If you have multiple "types" of tokens# with different uses and permissions, you# can set a subprefix so that they are easily identifiabledef common_token_subprefixif bearer_type == "User""usr"elsif bearer_type == "Organization""org"endenddef set_common_token_prefixself.common_token_prefix = "#{TOKEN_NAMESPACE}_#{common_token_subprefix}_"enddef generate_random_token_prefixself.random_token_prefix = SecureRandom.base58(6)enddef generate_raw_tokenself.raw_token = [common_token_prefix, random_token_prefix, SecureRandom.base58(24)].join("")enddef generate_token_digestself.token_digest = self.class.generate_digest(raw_token)endend
In the above example, we’re namespacing all of our tokens with the ApiKey::TOKEN_NAMESPACE
prefix to make them easily identifiable and searchable for secret scanning. We’re also adding a subprefix via #common_token_subprefix
to every token. This identifies the type of token, e.g. a token for a user vs a token for an organization. We’re also storing the first 6 characters (out of 30 total) of the random part of the token in random_token_prefix
and encrypting it using Active Record Encryption. It’s not strictly necessary to encrypt random_token_prefix
but it adds an additional layer of security with minimal cost. Finally, the entire token is hashed and stored in token_digest
, just like our prior example.
Using a helper, we can display our tokens in a nice format in the UI - they’ll be masked but also identifiable.
app/helpers/application_helper.rbmodule ApplicationHelperdef token_mask(prefix, length = 30)"#{prefix}#{"•"*length}"endend
<%= label_tag :api_key, "API Key" %><%= text_field_tag :api_key, token_mask(api_key.token_prefix), disabled: true %>
Now that we have API Keys, how do we authenticate using Bearer Authentication? In Rails, we can do this fairly easily by leveraging #authenticate_or_request_with_http_token:
app/controllers/api/base_controller.rbmodule Apiclass BaseController < ApplicationControllerskip_before_action :authenticate_user!skip_before_action :verify_authenticity_tokenbefore_action :authenticate_with_api_keyattr_reader :current_bearer, :current_api_keyprotecteddef authenticate_with_api_keyauthenticate_or_request_with_http_token do |token, options|@current_api_key = ApiKey.find_by_token(token)@current_bearer = current_api_key&.bearerendend# Override rails default 401 response to return JSON content-type# with request for Bearer token# https://api.rubyonrails.org/classes/ActionController/HttpAuthentication/Token/ControllerMethods.htmldef request_http_token_authentication(realm = "Application", message = nil)json_response = { errors: [message || "Access denied"] }headers["WWW-Authenticate"] = %(Bearer realm="#{realm.tr('"', "")}")render json: json_response, status: :unauthorizedendendend
Using Api::BaseController
, let’s create a simple API endpoint that displays information about the current ApiKey
being used:
app/controllers/api/v1/api_keys_controller.rbmodule Apimodule V1class ApiKeysController < BaseController# Render info about the current API Keydef showrender json: {id: current_api_key.id,bearer_type: current_api_key.bearer_type,bearer_id: current_api_key.bearer_id}endendendend
config/routes.rbnamespace :api donamespace :v1 doresource :api_key, only: [:show]endend
Note that we’ve versioned our API endpoint by placing it in a V1 namespace. By versioning our API, we’re giving ourselves the flexibility to make backwards incompatible changes without breaking the API for existing clients by introducing new versions.
Let’s create an ApiKey
for a user and see what happens:
$ rails cirb(main)> user = User.create=> #<User id: 1,...irb(main)> api_key = ApiKey.create(bearer: user)=> #<ApiKey:0x00005589a5183bc8...irb(main)> api_key.raw_token=> tkn_usr_CawyxedZAsW24AkCo94toYPyyDbHWXirb(main)> exit$ curl http://localhost:3000/api/v1/api_key \-H "Authorization: Bearer tkn_usr_CawyxedZAsW24AkCo94toYPyyDbHWX"{"id":1,"bearer_type":"User","bearer_id":1}
Now that we have a framework for authenticating bearers, how do we implement authorization? That is, how do we permit bearers to do some actions and not others? We already have bearer_type
at our disposal, which we can leverage with something like Pundit to provide a basic level of authorization conditional on the type of bearer. Let’s give it a try with Pundit using the hypothetical example that we only want bearers of type Organization
to be able to retrieve information about their current ApiKey
from the API:
app/controllers/api/v1/bearers_controller.rbmodule Apiclass BaseController < ApplicationControllerinclude Punditrescue_from Pundit::NotAuthorizedError, with: :not_authorizedskip_before_action :authenticate_user!skip_before_action :verify_authenticity_tokenbefore_action :authenticate_with_api_keyattr_reader :current_bearer, :current_api_keydef pundit_usercurrent_api_keyendprotecteddef not_authorizedrender status: :unauthorized, json: {errors: ["You are not authorized to perform this action"]}enddef authenticate_with_api_keyauthenticate_or_request_with_http_token do |token, options|@current_api_key = ApiKey.find_by_token(token)@current_bearer = current_api_key&.bearerendend# Override rails default 401 response to return JSON content-type# with request for Bearer token# https://api.rubyonrails.org/classes/ActionController/HttpAuthentication/Token/ControllerMethods.htmldef request_http_token_authentication(realm = "Application", message = nil)json_response = { errors: [message || "Access denied"] }headers["WWW-Authenticate"] = %(Bearer realm="#{realm.tr('"', "")}")render status: :unauthorized, json: json_responseendendend
app/policies/api/base_policy.rbmodule Apiclass BasePolicyattr_reader :api_key, :recorddef initialize(api_key, record)@api_key = api_key@record = recordendendend
app/policies/api/api_key_policy.rbmodule Apiclass ApiKeyPolicy < BasePolicydef show?api_key.bearer.is_a?(Organization)endendend
app/controllers/api/v1/api_keys_controller.rbmodule Apimodule V1class ApiKeysController < BaseController# Render info about the current API Keydef showauthorize([:api, current_api_key])render json: {id: current_api_key.id,bearer_type: current_api_key.bearer_type,bearer_id: current_api_key.bearer_id}endendendend
$ curl http://localhost:3000/api/v1/api_key \-H "Authorization: Bearer tkn_usr_CawyxedZAsW24AkCo94toYPyyDbHWX"{"errors":["You are not authorized to perform this action"]}
If we wanted the ability to scope ApiKey
permissions even further, we can easily extend the example above by adding a scopes
string array column to ApiKey
or creating a new model called Scope
that belongs_to
ApiKey
. Then in our policies, we can simply check if the api_key
has the needed scope(s). A setup like this would give us the same functionality that Stripe provides with their restricted API keys or Github with their personal access tokens.
While the examples above provide a solid foundation for building a public REST API and a great starting point, there are some additional aspects of an API that you’ll likely need to consider.
Giving users a way to revoke API keys is a common requirement. One way to do this would be to allow users to delete their API keys. Another approach would be similar to a “soft delete” where you could add a revoked_at
column to ApiKey
. In this scenario, revoking an API key would involve setting revoked_at
to the current time and preventing the usage of ApiKey
s where revoked_at
is not NULL
by updating Api::BaseController#authenticate_with_api_key
like so:
def authenticate_with_api_keyauthenticate_or_request_with_http_token do |token, options|@current_api_key = ApiKey.where(revoked_at: nil).find_by_token(token)@current_bearer = current_api_key&.bearerendend
The advantage of the “soft delete” approach is that it allows you to present your users with their revoked API keys for future reference or historical purposes.
Stripe has a nice feature where they allow you to roll API keys in a way that allows the old API key to work for a certain amount of time before it is revoked. This allows users to make a smooth transition to the new API key, especially if you only allow one active API key at a time. This can be achieved by adding an expires_at
timestamp to ApiKey
and updating Api::BaseController#authenticate_with_api_key
like so:
def authenticate_with_api_keyauthenticate_or_request_with_http_token do |token, options|@current_api_key = ApiKey.where(revoked_at: nil).where("expires_at is NULL OR expires_at > ?", Time.zone.now).find_by_token(token)@current_bearer = current_api_key&.bearerendend
If possible, it’s a good idea to host your API on a subdomain like api.mydomain.com
. This gives you some more flexibility if you ever need to scale out your API independently from other parts of your application. By using a subdomain, it’s easier to isolate and handle traffic to your API separately from other traffic because you’re separating traffic at the DNS level. If you were to use a path rather than a subdomain this would be much more difficult.
Clear error messaging and consistent formatting provides a great user experience. If you can be detailed and explicit in your error messages, your users will greatly appreciate it. Moreover, it’s best if you can provide a consistent way of returning errors such that it’s easy for clients to parse and handle these situations gracefully. A simple example would be always returning a JSON object with an error
key and a string value explaining what went wrong:
{"error": "A name is required to create an Organization."}
If there could be multiple errors at once, you could consider returning an array of errors:
{"errors": ["A name is required to create an Organization","A logo is required to create an Organization"]}
As long as your consistent in all of your endpoints.
Whether it’s enforcing plan limits or preventing abuse, you may get to a point where you’ll need to throttle API requests. This is a great use-case for Redis. If you’re running Ruby apps (like Rails) there are many open source options to help with this - rack-attack, redis-throttle, rack-throttle, and ratelimit to name a few.
Like with a lot of things in software, building a basic REST API is relatively easy. Once you dive a little deeper and start to consider longevity, ease-of-use, ergonomics, and security, the topic becomes more nuanced with various tradeoffs and decisions to make. Hopefully this article saves you some time when building your next public facing REST API!