Building a Public-Facing REST API - An In Depth Guide with Rails

---

---

Ari Summer / January 23, 2022

While APIs are relatively easy to build, they're hard to build well. You'll often hear developers air their grievances about APIs that are used by hundreds of thousands, if not millions of developers every day. Many of these APIs are developed by companies that are publicly traded with huge engineering teams - Facebook, LinkedIn, Microsoft, the list goes on. At the same time, you'll hear developers rave about APIs like Stripe.

There are many ingredients that go into a "great" API - clear documentation, reliability, explicit error messaging, and a simple, yet powerful interface. Companies like Stripe put a lot of thought and consideration into these details. Their users, after all, are other developers and providing a great experience gives them a competitive edge.

Once an API is public and users depend on it, the stakes get higher. Changes will need to be made carefully and with backwards compatibility in mind. It's worth the extra time and effort during the design phase to think about the ergonomics of your API and how it might evolve over time. There’s a relevant observation made by Hyrum Wright called Hyrum's Law:

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

Said another way:

Given enough use, there is no such thing as a private implementation. That is, if an interface has enough consumers, they will collectively depend on every aspect of the implementation, intentionally or not. This effect serves to constrain changes to the implementation, which must now conform to both the explicitly documented interface, as well as the implicit interface captured by usage. We often refer to this phenomenon as "bug-for-bug compatibility.”

It's fair to say that having a solid foundation will pave the way to a great API experience. In this guide, I'll discuss some design patterns and ideas that you can use in your public-facing APIs. As always, your mileage may vary, but this is a great starting point that should evolve with your ever-changing requirements. Examples will be given using Ruby on Rails, but the strategies are framework agnostic.

Authentication

Most API implementations will need a way to authenticate clients. There are many ways to authenticate clients to your API. By far the easiest and most common is Bearer Authentication.

Bearer Authentication

Bearer authentication (also called token authentication) is an HTTP authentication scheme that involves security tokens called bearer tokens. The name “Bearer authentication” can be understood as “give access to the bearer of this token.”

With Bearer Authentication a token is sent in the Authorization header when making a request like so:

$ curl https://example.com \
    -H "Authorization: Bearer {token}"

Bearer Authentication should be used over HTTPS so that the bearer tokens are encrypted in transit and not exposed in plain text.

When using Bearer Authentication, it’s best to use the Authorization header. Specifications for the internet (RFCs) specify how the Authorization header should be treated by caches and proxies. To be more specific:

• RFC 7235, section 4.2 specifies that proxies “MUST NOT modify any Authorization fields” for a request.
• RFC7234, section 3.2 specifies that responses to requests with Authorization headers must not be cached by default.

So by using the Authorization header you’ll get the benefits of the behavior defined by RFC. If you were to use a custom header, there would be no guarantees and a proxy could be inadvertantly caching or logging your tokens.

To find examples of Bearer Authentication in the wild, you only need to look to some of the most popular developer services - Stripe, Terraform Cloud, and Heroku. It’s also worth noting that the Bearer token scheme is used in the Oauth 2.0 protocol. In fact, that’s what it was designed for! It’s use has since evolved beyond just Oauth 2.0.

Alternative Authentication Options

Bearer Authentication will be the focus of this guide, but depending on your use case, there are other options available. Another commonly used scheme is Basic Authentication.

Basic Authentication is similar to Bearer Authentication in that it also uses the Authorization header, but it’s different in that it’s meant to authenticate using a username and password, rather than a single token. Basic Authentication also specifies a way to pass the username and password in the URL, but this is now deprecated behavior. If you plan to authenticate API clients using both a username and password, Basic Authentication can be a good option.

API Keys

Since we’ll be using Bearer Authentication, we’ll need tokens for our bearers to authenticate to our API. These are often called API Keys. How we model, store, present, and format our API Keys is an important piece of the puzzle. Some of the ideas in this section are inspired by Zeke Gabrielse’s great blog post, How to Implement API Key Authentication in Rails Without Devise.

Modeling API Keys

Let’s create an ApiKey model and migration by running:

$ rails g model ApiKey

Then let’s add the following contents to the model and migration:

db/migrate/20220101230637_create_api_keys.rb
class CreateApiKeys < ActiveRecord::Migration[7.0]
  def change
    create_table :api_keys do |t|
      t.belongs_to :bearer, polymorphic: true
      # More columns to be added and discussed below

      t.timestamps
    end
  end
end

app/models/api_key.rb
class ApiKey < ApplicationRecord
  belongs_to :bearer, polymorphic: true
end

We’ve created an ApiKey model that belongs to a bearer. The relationship is polymorphic because it can be useful to have multiple types of bearers. For example, let’s say we have Organization, Team, and User models. With a polymorphic relationship, Organizations, Teams, and Users can all be bearers with different scopes and permissions applied to each. More on this below.

Generating tokens

One thing that’s missing from the code above is the actual token that we provide the bearer in order to authenticate. Let’s adjust our migration and model like so:

db/migrate/20220101230637_create_api_keys.rb
class CreateApiKeys < ActiveRecord::Migration[7.0]
  def change
    create_table :api_keys do |t|
      t.belongs_to :bearer, polymorphic: true
      t.string :token, null: false

      t.timestamps
    end

    add_index :api_keys, :token, unique: true
  end
end

app/models/api_key.rb
class ApiKey < ApplicationRecord
  belongs_to :bearer, polymorphic: true

  before_create :generate_token

  private

  def generate_token
    self.token = SecureRandom.base58(30)
  end
end

Now we’re generating tokens which we can share with our bearer in order to authenticate. The problem, however, is that we should treat these tokens like passwords and we’re currently storing them as plaintext.

Storing API Key Tokens Safely

We have a couple options for storing these tokens safely - encrypting or hashing. If we ever need to retrieve the original, plaintext value, encryption would be our best bet because we can decrypt at-will. Hashing, however, is a one-way function. It’s a more secure option because it can’t be reversed. This is why it’s often used in authentication. In our case, we only need the plaintext value after creation to share with our users. After that, we can retrieve the API key from our database by querying for the hashed token value since our users will be sending the plaintext value in the Authorization header.

Let’s tweak our ApiKey model and migration to incorporate hashing our tokens before saving:

db/migrate/20220101230637_create_api_keys.rb
class CreateApiKeys < ActiveRecord::Migration[7.0]
  def change
    create_table :api_keys do |t|
      t.belongs_to :bearer, polymorphic: true
      t.string :token_digest, null: false

      t.timestamps
    end

    add_index :api_keys, :token, unique: true
  end
end

app/models/api_key.rb
class ApiKey < ApplicationRecord
  HMAC_SECRET_KEY = Rails.application.credentials.api_key_hmac_secret_key

  belongs_to :bearer, polymorphic: true

  before_create :generate_raw_token
  before_create :generate_token_digest

  # Attribute for storing and accessing the raw (non-hashed)
  # token value directly after creation
  attr_accessor :raw_token

  def self.find_by_token!(token)
    find_by!(token_digest: generate_digest(token))
  end

  def self.find_by_token(token)
    find_by(token_digest: generate_digest(token))
  end

  def self.generate_digest(token)
    OpenSSL::HMAC.hexdigest("SHA256", HMAC_SECRET_KEY, token)
  end

  private

  def generate_raw_token
    self.raw_token = SecureRandom.base58(30)
  end

  def generate_token_digest
    self.token_digest = self.class.generate_digest(raw_token)
  end
end

In the example above, we’re using HMAC with the SHA-256 hash function to hash the raw token value that we generate via SecureRandom.base58. HMAC requires a secret key, which you can generate in the Rails console and store in your rails credentials file (or as an ENV variable):

$ rails c

> SecureRandom.hex(32)
=> "f8e9dfcf97e73529b000fffd14f43627cdd74da691fd7a7d4fa3526b73a16041"

$ rails credentials:edit

config/credentials.yml.enc
api_key_hmac_secret_key: f8e9dfcf97e73529b000fffd14f43627cdd74da691fd7a7d4fa3526b73a16041

Note that you’ll want to generate a different key for each environment - development, staging, production, etc.

Now we have an ApiKey model that’s generating tokens and hashing the token before storing it in the database! You may be asking yourself though... How do we share the plaintext version with our users after creating an ApiKey?

Sharing Plaintext Tokens After Generation

In the code example above, we have ApiKey#raw_token which will give us the plaintext value of our token right after we create a new API key. This value will be available in memory within our new ApiKey instance directly after creation. For example:

irb(main):> user = User.first
irb(main):> api_key = ApiKey.create(bearer: user)
irb(main):> api_key.id
=> 1
irb(main):> api_key.raw_token
=> 6NCTx2TxTHGkbNLVeChfp8et6cm83a

# Fetch the record we created above from the Database.
irb(main):> api_key = ApiKey.find(1)
# raw_token will be nil because it's only available directly after
# creation
irb(main):> api_key.raw_token
=> nil

This essentially means that in our controller, the common pattern of redirecting after creating an ApiKey won’t work because raw_token will no longer be available after we redirect. We have to send (render) the raw_token value in the same request in which we create an ApiKey.

There are many ways to handle this. You could use AJAX to make a request to ApiKeysController#create which would respond with a JSON object including the raw_token:

config/credentials.yml.enc
class ApiKeysController < ApplicationController
  def create
    @api_key = ApiKey.new(bearer: current_user)

    if @api_key.save
      render json: { raw_token: @api_key.raw_token }
    else
      render status: :unprocessable_entity, json: {
        errors: @api_key.errors.full_messages
      }
    end
  end
end

Or if you’re using Turbo, you can respond with a Turbo Stream that injects the raw_token into the UI. For example, if we wanted to respond with a Turbo Stream that appended the API key to the DOM, we could have a create.turbo_stream.erb template:

app/views/api_keys/create.turbo_steam.erb
<%= turbo_stream.append "api-keys" do %>
  <%= render "api_key", api_key: @api_key %>
<% end %>

app/views/api_keys/_api_key.html.erb
<li>
  Api Key: <%= @api_key.raw_token || "*******************" %>
</li>

After creating an ApiKey, subsequent requests to render the ApiKey in the UI won’t be able to display the plaintext token (raw_token) - we’ll only have the irreversibly hashed version of our token, which we are storing in the token_digest column. In the example above for rendering an ApiKey using the _api_key.html.erb partial, our raw_token would display as ******************* on subsequent renders. This isn’t a great user experience and makes it difficult for users to identify active tokens. What if we wanted to show the prefix or suffix of a token without revealing the entire thing? Stripe does exactly this after you generate a new API key and reveal it for the one and only time:

Another nice thing Stripe does is that they format their API keys and secrets with an identifiable prefix (sk_*_). When you’re managing multiple API keys for different services, this makes it very easy to identify which API keys belong to which service. Moreover, when there are multiple types of tokens available for an individual service, having different prefixes for each type makes it easy to identify the token’s purpose. For example, stripe uses sk_live for live mode tokens and sk_test for test mode tokens. If this wasn’t the case, it would be difficult, if not impossible, to distinguish a live token from a test token. As you can imagine, this might lead to some problems! Slack and Github use a similar pairadigm for their API tokens as well.

You’ll also notice the usage of _ as a separator in Stripe’s tokens. This makes the prefix clearly distinguishable because it’s not part of the character set used to generate the random part of the token. There’s another less obvious benefit of using _ as the separator - you can select the entire token for copying by double clicking on it. For example, try double clicking on my_token with your mouse. You’ll highlight the entire token. Now try double clicking on my-token. You can’t highlight the entire string!

Another nice thing about using an identifiable prefix is that you can use secret scanning on services like Github:

GitHub scans repositories for known secret formats to prevent fraudulent use of credentials that were committed accidentally. Secret scanning happens by default on public repositories, and can be enabled on private repositories by repository administrators or organization owners. As a service provider, you can partner with GitHub so that your secret formats are included in our secret scanning.

Basically, this means that if a user of your API accidentally commits a sensitive API token to their Github repo, your service can be notified by Github. Once you’re notified, you can auto-revoke the compromised token and notify the user, saving them from catastrophe!

Formatting & Displaying API Key Tokens

With all this in mind, let’s refactor our ApiKey example such that we have an identifiable prefix and the ability to display a handful of characters from the random part of our token while keeping the rest redacted.

db/migrate/20220101230637_create_api_keys.rb
class CreateApiKeys < ActiveRecord::Migration[7.0]
  def change
    create_table :api_keys do |t|
      t.belongs_to :bearer, polymorphic: true
      t.string :common_token_prefix, null: false
      t.string :random_token_prefix, null: false
      t.string :token_digest, null: false

      t.timestamps
    end

    add_index :api_keys, :token_digest, unique: true
  end
end

app/models/api_key.rb
class ApiKey < ApplicationRecord
  HMAC_SECRET_KEY = Rails.application.credentials.api_key_hmac_secret_key
  TOKEN_NAMESPACE = "tkn"

  # Encrypt random token prefix using Active Record Encryption
  # https://guides.rubyonrails.org/active_record_encryption.html
  # We're using deterministic encryption so that we can add uniqueness
  # validation below.
  encrypts :random_token_prefix, deterministic: true

  belongs_to :bearer, polymorphic: true

  before_validation :set_common_token_prefix, on: :create
  before_validation :generate_random_token_prefix, on: :create
  before_validation :generate_raw_token, on: :create
  before_validation :generate_token_digest, on: :create

  validates_uniqueness_of :random_token_prefix, scope: [:bearer_id, :bearer_type]

  # Attribute for storing and accessing the raw (non-hashed)
  # token value directly after creation
  attr_accessor :raw_token

  def self.find_by_token!(token)
    find_by!(token_digest: generate_digest(token))
  end

  def self.find_by_token(token)
    find_by(token_digest: generate_digest(token))
  end

  def self.generate_digest(token)
    OpenSSL::HMAC.hexdigest("SHA256", HMAC_SECRET_KEY, token)
  end

  def token_prefix
    "#{common_token_prefix}#{random_token_prefix}"
  end

  private

  # If you have multiple "types" of tokens
  # with different uses and permissions, you
  # can set a subprefix so that they are easily identifiable
  def common_token_subprefix
    if bearer_type == "User"
      "usr"
    elsif bearer_type == "Organization"
      "org"
    end
  end

  def set_common_token_prefix
    self.common_token_prefix = "#{TOKEN_NAMESPACE}_#{common_token_subprefix}_"
  end

  def generate_random_token_prefix
    self.random_token_prefix = SecureRandom.base58(6)
  end

  def generate_raw_token
    self.raw_token = [common_token_prefix, random_token_prefix, SecureRandom.base58(24)].join("")
  end

  def generate_token_digest
    self.token_digest = self.class.generate_digest(raw_token)
  end
end

In the above example, we’re namespacing all of our tokens with the ApiKey::TOKEN_NAMESPACE prefix to make them easily identifiable and searchable for secret scanning. We’re also adding a subprefix via #common_token_subprefix to every token. This identifies the type of token, e.g. a token for a user vs a token for an organization. We’re also storing the first 6 characters (out of 30 total) of the random part of the token in random_token_prefix and encrypting it using Active Record Encryption. It’s not strictly necessary to encrypt random_token_prefix but it adds an additional layer of security with minimal cost. Finally, the entire token is hashed and stored in token_digest, just like our prior example.

Using a helper, we can display our tokens in a nice format in the UI - they’ll be masked but also identifiable.

app/helpers/application_helper.rb
module ApplicationHelper
  def token_mask(prefix, length = 30)
    "#{prefix}#{"•"*length}"
  end
end

<%= label_tag :api_key, "API Key" %>
<%= text_field_tag :api_key, token_mask(api_key.token_prefix), disabled: true %>

Setting up Bearer Authentication with our API Keys

Now that we have API Keys, how do we authenticate using Bearer Authentication? In Rails, we can do this fairly easily by leveraging #authenticate_or_request_with_http_token:

app/controllers/api/base_controller.rb
module Api
  class BaseController < ApplicationController
    skip_before_action :authenticate_user!
    skip_before_action :verify_authenticity_token
    before_action :authenticate_with_api_key

    attr_reader :current_bearer, :current_api_key

    protected

    def authenticate_with_api_key
      authenticate_or_request_with_http_token do |token, options|
        @current_api_key = ApiKey.find_by_token(token)
        @current_bearer = current_api_key&.bearer
      end
    end

    # Override rails default 401 response to return JSON content-type
    # with request for Bearer token
    # https://api.rubyonrails.org/classes/ActionController/HttpAuthentication/Token/ControllerMethods.html
    def request_http_token_authentication(realm = "Application", message = nil)
      json_response = { errors: [message || "Access denied"] }
      headers["WWW-Authenticate"] = %(Bearer realm="#{realm.tr('"', "")}")
      render json: json_response, status: :unauthorized
    end
  end
end

Using Api::BaseController, let’s create a simple API endpoint that displays information about the current ApiKey being used:

app/controllers/api/v1/api_keys_controller.rb
module Api
  module V1
    class ApiKeysController < BaseController
      # Render info about the current API Key
      def show
        render json: {
          id: current_api_key.id,
          bearer_type: current_api_key.bearer_type,
          bearer_id: current_api_key.bearer_id
        }
      end
    end
  end
end

config/routes.rb
namespace :api do
  namespace :v1 do
    resource :api_key, only: [:show]
  end
end

Note that we’ve versioned our API endpoint by placing it in a V1 namespace. By versioning our API, we’re giving ourselves the flexibility to make backwards incompatible changes without breaking the API for existing clients by introducing new versions.

Let’s create an ApiKey for a user and see what happens:

$ rails c

irb(main)> user = User.create
=> #<User id: 1,...

irb(main)> api_key = ApiKey.create(bearer: user)
=> #<ApiKey:0x00005589a5183bc8...

irb(main)> api_key.raw_token
=> tkn_usr_CawyxedZAsW24AkCo94toYPyyDbHWX

irb(main)> exit

$ curl http://localhost:3000/api/v1/api_key \
  -H "Authorization: Bearer tkn_usr_CawyxedZAsW24AkCo94toYPyyDbHWX"

{"id":1,"bearer_type":"User","bearer_id":1}

Permissions & Authorization

Now that we have a framework for authenticating bearers, how do we implement authorization? That is, how do we permit bearers to do some actions and not others? We already have bearer_type at our disposal, which we can leverage with something like Pundit to provide a basic level of authorization conditional on the type of bearer. Let’s give it a try with Pundit using the hypothetical example that we only want bearers of type Organization to be able to retrieve information about their current ApiKey from the API:

app/controllers/api/v1/bearers_controller.rb
module Api
  class BaseController < ApplicationController
    include Pundit

    rescue_from Pundit::NotAuthorizedError, with: :not_authorized

    skip_before_action :authenticate_user!
    skip_before_action :verify_authenticity_token
    before_action :authenticate_with_api_key

    attr_reader :current_bearer, :current_api_key

    def pundit_user
      current_api_key
    end

    protected

    def not_authorized
      render status: :unauthorized, json: {
        errors: ["You are not authorized to perform this action"]
      }
    end

    def authenticate_with_api_key
      authenticate_or_request_with_http_token do |token, options|
        @current_api_key = ApiKey.find_by_token(token)
        @current_bearer = current_api_key&.bearer
      end
    end

    # Override rails default 401 response to return JSON content-type
    # with request for Bearer token
    # https://api.rubyonrails.org/classes/ActionController/HttpAuthentication/Token/ControllerMethods.html
    def request_http_token_authentication(realm = "Application", message = nil)
      json_response = { errors: [message || "Access denied"] }
      headers["WWW-Authenticate"] = %(Bearer realm="#{realm.tr('"', "")}")
      render status: :unauthorized, json: json_response
    end
  end
end

app/policies/api/base_policy.rb
module Api
  class BasePolicy
    attr_reader :api_key, :record

    def initialize(api_key, record)
      @api_key = api_key
      @record = record
    end
  end
end

app/policies/api/api_key_policy.rb
module Api
  class ApiKeyPolicy < BasePolicy
    def show?
      api_key.bearer.is_a?(Organization)
    end
  end
end

app/controllers/api/v1/api_keys_controller.rb
module Api
  module V1
    class ApiKeysController < BaseController
      # Render info about the current API Key
      def show
        authorize([:api, current_api_key])

        render json: {
          id: current_api_key.id,
          bearer_type: current_api_key.bearer_type,
          bearer_id: current_api_key.bearer_id
        }
      end
    end
  end
end

$ curl http://localhost:3000/api/v1/api_key \
  -H "Authorization: Bearer tkn_usr_CawyxedZAsW24AkCo94toYPyyDbHWX"

{"errors":["You are not authorized to perform this action"]}

If we wanted the ability to scope ApiKey permissions even further, we can easily extend the example above by adding a scopes string array column to ApiKey or creating a new model called Scope that belongs_to ApiKey. Then in our policies, we can simply check if the api_key has the needed scope(s). A setup like this would give us the same functionality that Stripe provides with their restricted API keys or Github with their personal access tokens.

Additional Considerations

While the examples above provide a solid foundation for building a public REST API and a great starting point, there are some additional aspects of an API that you’ll likely need to consider.

Revoking & Rolling Api Keys

Giving users a way to revoke API keys is a common requirement. One way to do this would be to allow users to delete their API keys. Another approach would be similar to a “soft delete” where you could add a revoked_at column to ApiKey . In this scenario, revoking an API key would involve setting revoked_at to the current time and preventing the usage of ApiKeys where revoked_at is not NULL by updating Api::BaseController#authenticate_with_api_key like so:

def authenticate_with_api_key
  authenticate_or_request_with_http_token do |token, options|
    @current_api_key = ApiKey.where(revoked_at: nil).find_by_token(token)
    @current_bearer = current_api_key&.bearer
  end
end

The advantage of the “soft delete” approach is that it allows you to present your users with their revoked API keys for future reference or historical purposes.

Stripe has a nice feature where they allow you to roll API keys in a way that allows the old API key to work for a certain amount of time before it is revoked. This allows users to make a smooth transition to the new API key, especially if you only allow one active API key at a time. This can be achieved by adding an expires_at timestamp to ApiKey and updating Api::BaseController#authenticate_with_api_key like so:

def authenticate_with_api_key
  authenticate_or_request_with_http_token do |token, options|
    @current_api_key = ApiKey
      .where(revoked_at: nil)
      .where("expires_at is NULL OR expires_at > ?", Time.zone.now)
      .find_by_token(token)
    @current_bearer = current_api_key&.bearer
  end
end

API Subdomain

If possible, it’s a good idea to host your API on a subdomain like api.mydomain.com. This gives you some more flexibility if you ever need to scale out your API independently from other parts of your application. By using a subdomain, it’s easier to isolate and handle traffic to your API separately from other traffic because you’re separating traffic at the DNS level. If you were to use a path rather than a subdomain this would be much more difficult.

Error Formatting

Clear error messaging and consistent formatting provides a great user experience. If you can be detailed and explicit in your error messages, your users will greatly appreciate it. Moreover, it’s best if you can provide a consistent way of returning errors such that it’s easy for clients to parse and handle these situations gracefully. A simple example would be always returning a JSON object with an error key and a string value explaining what went wrong:

{
  "error": "A name is required to create an Organization."
}

If there could be multiple errors at once, you could consider returning an array of errors:

{
  "errors": [
    "A name is required to create an Organization",
    "A logo is required to create an Organization"
  ]
}

As long as your consistent in all of your endpoints.

Throttling

Whether it’s enforcing plan limits or preventing abuse, you may get to a point where you’ll need to throttle API requests. This is a great use-case for Redis. If you’re running Ruby apps (like Rails) there are many open source options to help with this - rack-attack, redis-throttle, rack-throttle, and ratelimit to name a few.

Wrapping Up

Like with a lot of things in software, building a basic REST API is relatively easy. Once you dive a little deeper and start to consider longevity, ease-of-use, ergonomics, and security, the topic becomes more nuanced with various tradeoffs and decisions to make. Hopefully this article saves you some time when building your next public facing REST API!