The Amazing Blog

Referencing Stimulus Controllers as ViewComponent Sidecar Files with Propshaft Importmaps

2024-05-15T16:32:44-06:00

Referencing Stimulus Controllers as ViewComponent Sidecar Files with Propshaft Importmaps

AKA: How many Rails buzzwords can I fit in a single blog post title?

In a previous post, I described how I like to group related "Business Concept" objects together in app/aspects, rather than grouping them by type (app/jobs, app/commands, etc...)

In a sideproject I'm working on, I wanted to put my ViewComponent objects in that folder too, instead of app/components. Starting in ViewComponent 3, they support "sidecar" files, where you can put all the related files to that component in a subdirectory named the same as the component. Since I'm also using Stimulus for this project, and several of my Components have corresponding Stimulus controllers, I also wanted to put the controller.js file in that sidecar subdir, too.

For the final buzzword bingo, I'm also using Propshaft for this project, as I described in the last post. However, Propshaft only wants to look in the app/assets folder for Javascript controllers. I found a few different solutions, but none of them worked as-is, I had to cobble a few different solutions together.

To get started, here's what one of my "aspects" folder looks like:

$ tree app/aspects/scalers
app/aspects/scalers
├── form_component
│   ├── component_controller.js
│   └── form_component.html.haml
├── form_component.rb
├── target_form_component
│   ├── component_controller.js
│   └── target_form_component.html.haml
└── target_form_component.rb

I have two components, Scalers::FormComponent and Scalers::TargetFormComponent (one is a sub-form of the main, keep an eye out for an upcoming block post about that!).

First, I added the folders to the config/initializers/assets.rb:

diff --git a/config/initializers/assets.rb b/config/initializers/assets.rb
index b649f68..5311092 100644
--- a/config/initializers/assets.rb
+++ b/config/initializers/assets.rb
@@ -8,3 +8,10 @@ Rails.application.config.assets.version = "1.0"
 # Add additional assets to the asset load path.
 # Rails.application.config.assets.paths << Emoji.images_path
 # Rails.application.config.assets.paths << Rails.root.join("app/assets/fonts")
+Rails.application.config.assets.paths << "app/components"
+Rails.application.config.assets.paths << "app/aspects"
+
+Rails.application.config.importmap.cache_sweepers << Rails.root.join("app/components")
+Rails.application.config.importmap.cache_sweepers << Rails.root.join("app/aspects")

(I don't know if the "cache_sweepers" part is needed, I haven't deleted a stimulus controller in the project yet. It doesn't seem to hurt either.)

Then in my config/importmap.rb, I added those folders as controllers:

+pin_all_from "app/components", under: "controllers", to: ""
+pin_all_from "app/aspects", under: "controllers", to: ""

Finally, to reference this controller, you have to use the awkward naming scheme of replacing the / with -- in the name. In my case, the <form> element I want to attach to the controller starts like this:

<form data-controller="scalers--form-component--component" ...>

I don't actually write that out by hand. My controllers include a module that handles all the Stimulus naming for me (another upcoming post). Here's one of the helper methods:

def stimulus_controller
  "#{self.class.name.underscore.dasherize.gsub('/', '--')}--component"
end

This wasn't hard to get working, it was just fiddly to figure out which parts of the different solutions I needed.

Using Local Font Files in the Rails 7.1 Asset Pipeline

2024-04-29T13:03:40-06:00

Using Local Font Files in the Rails 7.1 Asset Pipeline

I wanted to play with the new hotness of Rails asset handling in a sideproject, and ran into some fiddliness in getting FontAwesome webfonts to be seen when loading them from an SCSS file. I'm using propshaft and Dart Sass in this app to build my CSS assets, and there wasn't any documentation I could find on how to reference your local font files within your SCSS files.

I happen to have a FontAwesome 5 Pro license from back when it was a KickStarter lifetime license, so that's what I'm using. FontAwesome 6 Pro is now a $100/yr subscription, which doesn't make sense for random side-projects, but I imagine these instructions will work the same. They provide a gem, but it wasn't working out-of-the-box for me in the new Rails asset pipeline, and I try to avoid dependencies when possible, so I installed it manually.

I started by downloading the zipfile. It includes a bunch of stuff, but we only care about what's in scss and webfonts.

I copied everything from scss into app/assets/stylesheets/fontawesome, and the font files from webfonts into app/assets/fonts. I noticed in FontAweseom6, it only includes files for ttf and woff2, since the other formats are for older browsers, so I only copied those.

Then, in my application.scss, I added the main fontawesome file, and then the regular theme, since that's what I'm using.

// app/assets/stylesheets/application.scss
@use 'fontawesome/fontawesome';
@use 'fontawesome/regular';

I also had to modify the fontawesome/regular.scss file, to use the right asset paths for the font files. (The regular.scss from FontAwesome 5 is different than this, and I was struggling to get it working. This one is based off the regular.scss from FontAwesome 6).

// app/assets/stylesheets/fontawesome/regular.scss
@import 'variables';

:root, :host {
  --fa-style-family-classic: "Font Awesome 5 Pro";
  --fa-font-regular: normal 400 1em/1 "Font Awesome 5 Pro";
}

@font-face {
  font-family: "Font Awesome 5 Pro";
  font-style: normal;
  font-weight: 400;
  font-display: block;
  // This here is the part to change
  src: url("fa-regular-400.woff2") format("woff2"), url("fa-regular-400.ttf") format("truetype");
}

.far,
.fa-regular {
  font-family: 'Font Awesome 5 Pro';
  font-weight: 400;
}

The important part I had to change was the src: url(...) bits. The way the asset pipeline works, these url() statements get replaced with the path to the digest files with the /assets/ prefix.

And with that, I get FontAwesome icons!

%a.is-active{href: dashboard_path(account_slug: params[:account_slug])}
  %span.icon
    %i.far.fa-tachometer-alt
    %span
      Dashboard

Rails File Naming Conventions

2023-03-08T11:06:15-07:00

Rails File Naming Conventions

One of the best things about Rails, that it gets right over so many other frameworks like React, is its Directory Naming Conventions. You can jump into any Rails app written in the last 15 years and no immediately where to go to start exploring. Where's the routes? Database Models? Controllers and Views? Every app keeps them in the same place, so once you learn it, you can easily transfer that knowledge to another project.

That's for types of objects that come with Rails though (Models, Controllers, Jobs, etc...). Modern best-practices have a variety of extra Service Objects like Queries, Adapters, Commands, and more. Where do you put those? Unfortunately, its not as clear.

This post represents my personal opinion on the subject, informed by my experiences since the early days, working on a number of different Rails apps, large and small. Even some subtle things can have a huge impact in developer happiness and friction. But, this is just like my opinion man, yours may differ, so feel free to steal my ideas and use the ones you like best.

Here's some general guidelines I try to follow:

Avoid being too flat.

Too many directories and files under a single parent causes the tree view in editors and file browsers to scroll, making things hard to find.

But also avoid being nested too deep.

All that hunting and clicking in a tree view is annoying, too.

Keep the fuzzy-finder in mind.

One app I used had third-party API clients in a single folder under app, so for example app/clients/salesforce.rb. But whenever I wanted to open what my brain called the "Salesforce Client" I would type something like sfcli into my editor's fuzzy-finder, and it wouldn't find it. I never ever managed to remember on the first try that I had to type clisf instead.

Group Service Objects by "business purpose" instead of "type".

Most apps I've seen that introduce Service Objects for the first time blindly follow the Rails pattern. It seems obvious, if models go in app/models and controllers go in app/controllers, then I should put my Queries in app/queries, right? However, I find that if you instead put Service Objects together by what they do instead of what they are, there's a number of advantages.

Let's give an example:

app/  
├─ aspects/  
│　├─ authorization.rb  
│　└─ authorization/  
│　　　├─ auth0_client.rb  
│　　　├─ authenticate_user.rb  
│　　　├─ clean_stale_sessions_job.rb  
│　　　├─ handle_omniauth_callback.rb  
│　　　├─ signup_form.rb  
│　　　├─ stale_sessions_query.rb  
│　　　└─ verify_user_token.rb  
├─ controllers/  
│　└─ sessions_controller.rb  
├─ jobs/  
└─ models/  
　　├─ user.rb  
　　├─ account.rb  
　　└─ membership.rb

Here's a hypothetical layout for handling Authorization. I like to keep Models and Controllers in the Rails-standard locations, because Rails gets picky about the naming of them, and its because where newly-onboarded devs expect to look.

Under app, I add a new top-level directly called aspects. I used to call this components, but if your app uses ViewComponent, it takes over app/components and things get confusing. aspects is shorter, but I'm not fully settled on this term yet. Each directory under here represents a "Business Aspect" of your application. Like in this case we have Authorization, but this could be like "Integrations", "Billing", "Admin", or "UserSettings". If you imagine an app like GMail's UI, we might have additional Aspects for "Inbox", "Filters" and "Attachments".

In this case, for our Authorization Aspect, we have an API Client (auth0_client.rb), some Commands (authenticate_user.rb, handle_omniauth_callback.rb), a FormObject (signup_form.rb), a Job (clean_stale_sessions_job.rb) and a Query (stale_sessions_query.rb).

I also create a top-level authorization.rb, which explicitly defines the module Authorization namespace. If you're using Dry::Container, it also gives you a nice place to define a Authorization::Container and AutoInject.

Here's what I like about having things grouped together like this:

Easy to find/fuzzy-find. If I'm working on Authorization, I can expand the app/aspects/authorization folder in my tree view to see all the related files. I can type prefix all my fuzzy-finder searches with auth to quickly scope the search.
Easier to cleanup. I find this particular important for Queries. If we decide we don't need to cleanup sessions any more, we delete the Authorization::CleanSessionsJob. Since the Authorization::StaleSessionsQuery is right there with it, we have an indication of the scope, and know its safe to delete too. If it lived in app/queries/stale_sessions.rb, it would be much less obvious what code is using it, and if it was safe to delete. Additionally, especially in the early days of a startup, you're trying different things, some of which don't pan out. Its easy to rm -rf app/aspects/salesforce_integration to nuke the whole thing at once, instead of having to track down all the files and inevitably leaving leftovers scattered around.
Easier to test. I can run the tests for a single component all at once. If I do a bunch of refactorings to the Commands in app/aspects/authorization, I can run all the relevant tests with rspec spec/aspects/authorization. I don't have to wait for the entire suite to run, which speeds up the red/green cycle, nor do I have to pass rspec a bunch of different filenames, which is annoying when I'm moving or renaming files as part of the refactor.

I've been iterating on this pattern for several different jobs and applications, with various team sizes, and I'm pretty happy with the tradeoffs that come from structuring app code this way. I'm always curious to learn how others approach it, if you've got some unique ideas you really like, let me know!

Using Postgres Enum in Rails ActiveRecord

2020-10-16T15:13:31-06:00

Using Postgres Enum in Rails ActiveRecord

In this post, I will provide some code to make working with an Enum data type in Postgres easier within your ActiveRecord models. Skip to the end for the code, or stick around for some verbose pontificating.

ActiveRecord Enums

ActiveRecord comes with a handy feature to specify a certain field is an Enum, or "Enumerated set of values". However, the documentation emphasizes the simple way to use it that is fraught with peril. They even note the danger in the docs:

Note that when an array is used, the implicit mapping from the values to database integers is derived from the order the values appear in the array. In the example, :active is mapped to 0 as it's the first element, and :archived is mapped to 1. In general, the i-th element is mapped to i-1 in the database.

Therefore, once a value is added to the enum array, its position in the array must be maintained, and new values should only be added to the end of the array. To remove unused values, the explicit hash syntax should be used.

If you use the Array form in your model, like this, it implicitly uses the position of the item in that array for the integer value for the column in the database:

class Message
  enum state: [
    :queued,     # 0
    :dispatched, # 1
    :delivered   # 2
  ]
end

This writes rows to the DB that look like this:

 id |  state   |         created_at
----+----------+----------------------------
  1 |        0 | 2020-10-14 21:34:40.036597
  2 |        2 | 2020-10-14 21:34:40.056437

However, if you make any change to the enum aside from adding a new value to the end of the Array, then the integer values of the fields change as well.

class Message
  enum state: [
    :queued,     # 0
    :dispatched, # 1
    :failed,     # 2
    :delivered   # 3
  ]
end

By inserting :failed in the middle of the Array, ActiveRecord will now consider 2 to be "failed" where previously it was "delivered", and so Message id:2 in our table that used to be "delivered" is now "failed". 💣

However, not all is lost! ActiveRecord Enums may also be defined as a Hash instead of an Array. That looks like this:

class Message
  enum state: {
    queued:     0,
    dispatched: 1,
    delivered:  2
  }
end

Now, when we add a new value to the enum, we can put it wherever we want in that hash, as long as we don't change the numbers.

Its still kinda tedious and annoying, though, that we have to track these numbers ourselves. It would be nice if we could just store those values directly as strings in the DB, but that would result in a much larger table, wasting storage on all those same strings over and over again.

Leveraging Postgres

The reason why Rails chooses to use Integers as values for its enums is because it has to support the lowest-common feature set of the databases it supports, and not all of them support Enums natively. Postgres, however, is one that does, and so if your app will only ever talk to Postgres, then you can take advantage of them.

Here's a simple migration to add an enum, we have to drop to raw SQL to accomplish it:

class CreateMessagingTables < ActiveRecord::Migration[6.0]
  reversible do |dir|
    dir.up do
      execute "CREATE TYPE message_state_type AS ENUM ('queued', 'dispatched', 'delivered')"

      create_table :messages do |t|
        t.column :state, :message_state_type, null: false
      end
    end

    dir.down do
      drop_table :messages
      execute "DROP TYPE message_state_type"
    end
  end
end

That's gross and annoying, though, so lets extract a helper:

# lib/migration_utils.rb

module MigrationUtils
  module CreateEnum
    def create_enum(name, values)
      reversible do |dir|
        dir.up do
          say_with_time "create_enum(:#{name})" do
            suppress_messages do
              execute "CREATE TYPE #{name} AS ENUM (#{values.map{ |v| quote(v) }.join(', ')})"
            end
          end
        end

        dir.down do
          say_with_time "drop_enum(:#{name})" do
            execute "DROP TYPE #{name}"
          end
        end
      end
    end
  end
end

# Then use it in a migration
#
# db/migrations/0000000000_create_messages.rb
class CreateMessagingTables < ActiveRecord::Migration[6.0]
  include MigrationUtils::CreateEnum

  change do
    create_enum :message_state_type, %w[queued dispatched delivered]

    create_table :messages do |t|
      t.column :state, :message_state_type, null: false
    end
  end
end

Now we can use Postgres Enum in Rails!

Instead of Integers, Postgres will expose the enum values as Strings, so we need to update the Hash in our model:

class Message
  enum state: {
    queued:     :queued,
    dispatched: :dispatched,
    delivered:  :delivered
  }
end

The values of the hash must match those of the postgres enum, but the keys can be whatever you like (but why would you do that to yourself?). Since for our app, the keys always match the values, we wrote a little helper to remove some boilerplate:

class ApplicationRecord < ActiveRecord::Base
  self.abstract_class = true

  # Provides a bit of syntactic sugar around Rails' built-in enums to map
  # them to postgres enums which expect string values instead of integer
  # values. Basically this saves you from having to pass in:
  # {
  #   foo: "foo",
  #   bar: "bar",
  #   baz: "baz"
  # }
  # to the Rails enum DSL method.
  def self.pg_enum(attribute, values, options = {})
    enum({ attribute => Hash[values.map{ |value| [value.to_sym, value.to_s] }] }.merge(options))
  end
end

Now our model looks like this:

class Message
  pg_enum state: %i[ queued dispatched delivered ]
end

You can find all the code for this, along with helpers to add and remove fields in the migration, at this gist

Using Dry::Container for Dependency Injection

2020-05-06T16:30:42-06:00

Using Dry::Container for Dependency Injection

The point of this post isn't to convince you of the usefulness of Dependency Injection there's been plenty of pixels spilled about it already. Instead, I want to talk about using Dry::Container to alleviate some of the pain points that DI introduces.

The first problem is when one object calls another, and both are using DI. For example, say you have a Command object that calls and Adapter object that finally calls the Client object. You end up with long chains of DI objects being injected into objects at a higher level, and it quickly becomes unwieldy, particularly for testing. Imagine we have something like this:

class MyCommand
  attr_reader :adapter

  def initialize(adapter: MyAdapter.new)
    @adapter = adapter
  end

  def call(args)
    # do stuff
    adapter.do_something(params)
  end
end

class MyAdapter
  attr_reader :client

  def initialize(client: MyClient.new)
    @client = client
  end

  def do_something(params)
    # do stuff
    client.make_request(url, json)
  end
end

class MyClient
  attr_reader :http

  def initialize(http: HttpClient.new(timeout: 5))
    @http = http
  end

  def make_request(url, json)
    http.auth(user, pass).post(url, json: json)
  end
end

In an integration test, you want to set up a mock client for the Client object to use, so it doesn't make any real requests. A typical solution is involves complicated test setup:

# my_command_spec.rb

RSpec.describe MyCommand do
  let(:http)    { instance_spy(HttpClient) }

  let(:client)  { MyClient.new(http: http) }
  let(:adapter) { MyAdapter.new(client: client) }
  let(:command) { described_class.new(adapter: adapter) }

  it "should make an http call" do
    command.call(:send_message)
    expect(http).to have_received(:post).with("http://myapp.example/send_message",
                                              json: { "text": "Hello!" })
  end
end

In our integration test for MyCommand, we have to set up a whole lot of other intermediate objects, just so we can inject the spy in at the lowest level. It seems strange that the test for this high-level business object needs to care about the low-level details about how the client is calling our HttpClient. Additionally, we probably have different things using the Adapter or MyCommand themselves, and the tests for those will need the same setup. Then, if we do any refactorings around how the Command -> Adapter -> Client pattern is set up, we'll have to come fix the setup for all these tests, which becomes tedious and error-prone.

Another alternative would be to set up all the intermediate objects to allow http to be injected, and pass it all the way through to the thing that cares about it.

class MyCommand
  attr_reader :adapter

  def initialize(adapter: MyAdapter, client: MyClient, http: HttpClient.new(timeout: 5))
    @adapter = adapter.new(client: client, http: http)
  end
end

class MyAdapter
  attr_reader :client

  def initialize(client: MyClient, http: HttpClient.new(timeout: 5))
    @client = client.new(http: http)
  end
end

class MyClient
  attr_reader :http

  def initialize(http: HttpClient.new(timeout: 5))
    @http = http
  end
end

This isn't great either, because now all the outer objects have to pass through a thing they don't care about at all. Its also easy to loose track of them, which object needs which dependency. Also, if MyCommand's job is to decide which of 5 Adapters it needs to send the message, it has to have 5 different clients injected.

The other issue I have with Dependency Injection is that once you start using it, it makes sense to use it for everything. The problem with that, however, is that you quickly run into very long and noisy #initialize methods:

class MyCommand
  attr_reader :http, :logger, :instrumenter, :error_handler

  def initialize(http: HttpClient.new(timeout: 5),
                logger: Rails.logger,
                instrumenter: ActiveSupport::Notifications,
                error_handler: Honeybadger)
    @http = http
    @logger, @instrumenter, @error_handler = logger, instrumenter, error_handler
  end
end

I found that nearly every object I had was injecting that triplet of [:logger, :instrumenter, :error_handler], which got fairly tedious. While this maybe could be resolved with a small object like what's used for Primitive Obsession, I don't have a good name for that object.

These two problems are particularly exacerbated when you need to pass through a mock logger or instrumenter, and test the calls made to that. Now you need to inject a whole lot of unrelated things, some of which the object doesn't care about, and it gets messy quickly.

Dry::Container

First, lets take a look at what a Dry::Container looks like:

module MyApp
  module Container
    extend Dry::Container::Mixin

    register(:error_handler) { Honeybadger }
    register(:instrumenter)  { ActiveSupport::Notifications }
    register(:logger)        { Rails.logger }

    namespace(:clients) do
      register(:http)   { HttpClient.new(timeout: 5) }
      register(:github) { Octokit::Client.new(login: config.github_user, password: config.github_password) }
      register(:heroku) { PlatformAPI.new(token: config.heroku_token) }
    end
  end
end

To use the values within a Container, its fairly simple, you can treat the container like a Hash:

http = MyApp::Container["clients.http"]
http.get("https://myapp.example/")

We can inject them into our objects be referencing them through the container, rather than directly:

class MyClient
  attr_reader :http

  def initialize(http: MyApp::Container["clients.http"])
    @http = http
  end

  def make_request(url, json)
    http.auth(user, pass).post(url, json: json)
  end
end

This, when coupled with dry-container's stub feature, lets us avoid complex test setup or deep injection:

# in spec_helper.rb or something:
require 'dry/container/stub'
MyApp::Container.enable_stubs!

# In your test:
RSpec.describe MyCommand do
  let(:http) { instance_spy(HttpClient) }

  around do |example|
    MyApp::Container.stub("clients.http") { http }
    example.run
    MyApp::Container.unstub("clients.http")
  end

  it "should make an http call" do
    command.call(:send_message)
    expect(http).to have_received(:post).with("http://myapp.example/send_message",
                                              json: { "text": "Hello!" })
  end
end

Aside: In our app, we have that wrapped in a helper: stub_container(MyContainer, http: fake_client) { ... }. Inside the block its stubbed, then un-stubbed when the block ends.

Auto-inject

A separate gem, Dry::AutoInject can work with our containers to help eliminate the boilerplate when injecting many dependencies into a class. You can set it up in your container:

module MyApp
  Import = Dry::AutoInject(Container)
end

Then, in your objects:

class MyCommand
  include MyApp::Import[:logger, :instrumenter, :error_handler]
  include MyApp::Import["clients.http"]

  def call(args)
    instrumenter.instrument("MyCommand.call") do
      logger.info("doing stuff")
      http.post("https://myapp.example/")
    end
  end
end

It looks a bit strange, but essentially include MyApp::Import[:foo] is a macro that generates code like:

attr_reader :foo

def initialize(foo: MyApp::Container[:foo])
  @foo = foo
end

When you have a lot of dependencies to inject, it really cuts down on the boilerplate. One thing to make a note of, however, if you have non-DI args passed to your #initialize method, you have to remember to call super.

class MyCommand
  include MyApp::Import[:logger, :instrumenter, :error_handler]

  def initialize(user:, **deps)
    @user = user
    super(**deps)
  end
end

Hopefully you find this helpful, we certainly have. Dependency Injection is a powerful tool that makes organizing and testing code a much cleaner experience, and dry-container and dry-auto_inject are a nice bit of polish over some of the tedious or boilerplate parts that come with it.

In-Band vs. Out-of-Band: The Advantages of Hypermedia APIs

2016-07-30T11:58:39-06:00

In-Band vs. Out-of-Band: The Advantages of Hypermedia APIs

A good way to look at the advantages of a hypermedia-based API vs an "HTTP-RPC" style one is to consider the differences between "In-Band" and "Out-of-Band" information. A hypermedia API focuses on getting as much information in-band as possible, reducing the burden on clients to deal with changes.

Note: I originally considered this to help my team decide what path to take when building V2 of our API, but decided it deserved a wider audience

"In-Band" is anything that is transmitted within the primary transmission channel. "Out-of-Band" refers to any communications that occur outside of that.

In the case of JSON APIs, I'll use "In-Band" to refer to the data in the JSON payload itself, while "Out-of-Band" will refer to everything else you need to know to consume that API, such as documentation or institutional knowledge.

Ideally, Out-of-Band knowledge should be minimized in an API, because it requires more effort on the part of the client to consume the API. Any Out-of-Band knowledge must be documented (or guessed!), and ends up being hard-coded into the client implementations. This makes it very difficult, if not completely impossible, to change in the future. One of the primary goals of the API it to be resilient to change, otherwise we'd all be content with scraping the HTML. Similarly to how changes to your HTML content or layout don't break the clients consuming it (browsers), so should clients of your API be resilient to changes to your JSON. It is the Out-of-Band knowledge that prevents this from happening.

A non-comprehensive list of things you have to know to consume any API, whether that knowledge is In-Band or Out-of-Band might include:

The data itself.
The types of data in the payload.
How to fetch more data.
How to change some of the data on this resource.
How to create a new instance of this resource.
When creating or updating, which fields are optional.

HTTP/RPC

Lets take a look at a typical JSON API:

// GET /api/v1/posts/1
{
  "id": 1,
  "title": "In-Band vs Out-of-Band",
  "author_id": 1,
  "summary": "Hypermedia is great!",
  "tags": ["json", "api", "hypermedia"],
  "created_at": "2016-08-01T04:20:00Z"
}

The information that is In-Band vs Out-of-Band is pretty simple:

In-Band	Out-of-Band
Fields & Values	Everything Else

Unfortunately, this JSON API leaves a lot of questions unanswered, and the developer consuming it will have to spend a lot of time consulting the documentation. Some pretty typical questions you'll be asking, which require Out-of-Band knowledge:

How do I found out who the author is? I see that author_id=1, do I need to GET /api/v1/authors/1? Or /users/1?
I see this post has tags. How do I get a list of all posts also tagged with one of these tags?
This post has a numeric id. If I store this post locally in a database, should I set ID to an integer? Or will it sometimes be a mongo id? Or a UUID?
If I want to amend or update some of these fields, do I need to return the full payload, or just the ones I want to change? Do I PUT it to this same URL, or POST it somewhere else, like ./revisions?
When creating a new post, do I POST it somewhere else? Which of these fields are required? Are there limits to the length of the title or summary?

Hopefully, this information is answered in the 100% complete and accurate documentation, but sadly that not always the case. If we're needing to create a client for this API ourselves, we're going to need to write a great deal of code to capture that Out-of-Band knowledge in the client itself, to make our client easier to use. If the creator of this API has provided us with a client, the we'll still need to read that documentation to know which fields we can expect from that client, and which to provide, and how to go about fetching the author given its id.

In any case, this is far from ideal, but it is the status quo. Luckily, the Hypermedia movement has gained enough traction that several major APIs are using it, such as GitHub.

Hypermedia APIs

Which Out-of-Band knowledge is made In-Band by throwing some hypermedia into the payload? Lets copy the example from a popular hypermedia API specification, JSON-API.

// GET /api/v1/posts/1
{
  "type": "article",
  "id": "1",
  "attributes": {
    "title": "JSON API paints my bikeshed!"
  },
  "relationships": {
    "author": {
      "links": {
        "self": "http://example.com/articles/1/relationships/author",
        "related": "http://example.com/articles/1/author"
      },
      "data": { "type": "people", "id": "9" }
    },
    "comments": {
      "links": {
        "self": "http://example.com/articles/1/relationships/comments",
        "related": "http://example.com/articles/1/comments"
      },
      "data": [
      { "type": "comments", "id": "5" },
      { "type": "comments", "id": "12" }
      ]
    }
  },
  "links": {
    "self": "http://example.com/articles/1"
  }
}

In-Band	Out-of-Band
Fields & Values	Required attributes
Where to find author & comments	Types of attributes
Where to update this post	Method to use to update

This is clearly an improvement of HTTP/RPC, there is quite a bit more knowledge In-Band, and this is what basic Hypermedia provides us. However, it still doesn't provide us with some basic knowledge about how to consume this API. Most notably, it says nothing about the attributes themselves:

What kind of data can we expect? Are they basic numbers or strings, or can we expect something encoded in them, like timestamps or embedded html tags?
Can we always expect all these fields on a document of type "article"? Will some show up that aren't in this example, or does this example have any that other responses might not?
If I want to create or update an "article", which fields must I provide, and which are optional?
If I need to update this "article", I can guess that I use the "self" link provided, but should I POST or PUT? Does this endpoint support PATCH?

Its not just JSON-API, most JSON Hypermedia proposals (HAL, Collection+JSON) do not provide a means to answer these questions In-Band, and as such, will have to be documented Out-of-Band. Also, most of these Hypermedia standards are far more verbose than the plain HTTP/RPC, which can dissuade developers from considering the advantages.

While this is far superior to plan HTTP/RPC, we can still do better.

JSON+LD

This usual reaction when someone brings up "RDF" and "Semantic Web" is to run screaming before the horrors of a cryptic W3C spec is rolled out. JSON+LD, however, is a breath of fresh air compared to the convolution of the technologies it is based upon. Let's take a look at an example of a JSON+LD document:

// GET http://dbpedia.org/resource/John_Lennon
{
  "@context": "http://json-ld.org/contexts/person.jsonld",
  "@id": "http://dbpedia.org/resource/John_Lennon",
  "name": "John Lennon",
  "born": "1940-10-09",
  "spouse": "http://dbpedia.org/resource/Cynthia_Lennon"
}

Simple, right? Even a lay-web developer can understand this document. But how does this move the Out-of-Band knowledge in JSON-API to In-Band JSON+LD? The secret lies in that @context attribute. When we fetch that URL ("dereference" in Semantic Web terminology) we get this:

// GET http://json-ld.org/contexts/person.jsonld
{
  "@context": {
    "Person":          "http://xmlns.com/foaf/0.1/Person",
    "xsd":             "http://www.w3.org/2001/XMLSchema#",
    "name":            "http://xmlns.com/foaf/0.1/name",
    "nickname":        "http://xmlns.com/foaf/0.1/nick",
    "affiliation":     "http://schema.org/affiliation",
    "depiction":       { "@id": "http://xmlns.com/foaf/0.1/depiction", "@type": "@id" },
    "image":           { "@id": "http://xmlns.com/foaf/0.1/img", "@type": "@id" },
    "born":            { "@id": "http://schema.org/birthDate", "@type": "xsd:dateTime" },
    "died":            { "@id": "http://schema.org/deathDate", "@type": "xsd:dateTime" },
    "child":           { "@id": "http://schema.org/children", "@type": "@id" },
    "parent":          { "@id": "http://schema.org/parent", "@type": "@id" },
    "sibling":         { "@id": "http://schema.org/sibling", "@type": "@id" },
    "spouse":          { "@id": "http://schema.org/spouse", "@type": "@id" },
    "telephone":       "http://schema.org/telephone",
    // It goes on for awhile with more attributes, about 30 total
  }
}

Whew, that's a lot of stuff. However, from this, we can see clearly what all the possible attributes we can expect to be returned from this endpoint. Further, we know what types they are (born is a schema.org birthDate, and is parsed as xsd:dateTime), and which can be links to something else (the ones that have "@type": "@id").

In-Band	Out-of-Band
Fields & Values	Required attributes
Where to find the spouse	Method to use to update
Where to update this person
Possible attributes
Types of attributes

We still don't know which attributes are required if we want to update the resource, or what method to use, but we know a great deal more about the fields and data types we can expect to be returned to us that we might need to handle. The basic JSON-LD document is also much more readable for a human than the JSON-API document, which is a big win in my book, and worth doing for that characteristic alone. Furthermore, the JSON+LD document is also easier for computers to read, through a process called expansion. This process follows a simple algortihm to expand the source document based upon the context document it links to, and there are libraries to do so in most common languages. Let's run our document through the expansion process using the tool on the JSON+LD site:

[
  {
    "@id": "http://dbpedia.org/resource/John_Lennon",
    "http://schema.org/birthDate": [
      {
        "@type": "http://www.w3.org/2001/XMLSchema#dateTime",
        "@value": "1940-10-09"
      }
    ],
    "http://xmlns.com/foaf/0.1/name": [
      { "@value": "John Lennon" }
    ],
    "http://schema.org/spouse": [
      { "@id": "http://dbpedia.org/resource/Cynthia_Lennon" }
    ]
  }
]

It is much more verbose, but now a computer can pull out the attribute that stores the person's "name", no matter what you decided to call it in your API document, whether it is "Name" or "fullname" or "personname" or whatever you can come up with. As long as you add to the context a line like "name": "http://xmlns.com/foaf/0.1/name", then a computer can expand the document from the context, and go look up the canonical "name" attribute. Similarly, it can use the @type attribute on the birthDate field to know to parse that string as an XMLSchema datatime to get a real date out of it.

But, there's still some knowledge that has to be provided Out-Of-Band, mainly which attributes to we have to provide when creating a person, and which HTTP methods to use to it. That's where Hydra comes in.

Hydra

Hydra is a draft W3C spec that adds a vocabulary for transmitting JSON-LD documents over a HTTP API. A Hydra api document looks exactly like the nice, simple JSON+LD example. The only change it makes is to add a vocab field to the context document:

    "vocab": "http://www.markus-lanthaler.com/hydra/event-api/vocab",

Dereferencing this document gives us a very large document, describing all the resources and endpoints provided by this API. (It doesn't have to be a single large document, you can break it up per-resource if you have a very large API surface.) Let's take a look at a subset of that document:

    {
      "@id": "http://xmlns.com/foaf/0.1/Person",
      "@type": "hydra:Class",
      "hydra:title": "Person",
      "hydra:description": null,
      "supportedOperation": [
        {
          "@id": "_:person_replace",
          "@type": "http://schema.org/UpdateAction",
          "method": "PUT",
          "label": "Replaces an existing Person entity",
          "description": null,
          "expects": "http://xmlns.com/foaf/0.1/Person",
          "returns": "http://xmlns.com/foaf/0.1/Person",
          "statusCodes": [
            {
              "code": 404,
              "description": "If the Person entity wasn't found."
            }
          ]
        },
        {
          "@id": "_:person_delete",
          "@type": "http://schema.org/DeleteAction",
          "method": "DELETE",
          "label": "Deletes a Person entity",
          "description": null,
          "expects": null,
          "returns": "http://www.w3.org/2002/07/owl#Nothing",
          "statusCodes": [ ]
        },
        {
          "@id": "_:person_retrieve",
          "@type": "hydra:Operation",
          "method": "GET",
          "label": "Retrieves a Person entity",
          "description": null,
          "expects": null,
          "returns": "http://xmlns.com/foaf/0.1/Person",
          "statusCodes": [ ]
        }
      ],
      "supportedProperty": [
        {
          "property": "http://schema.org/name",
          "hydra:title": "name",
          "hydra:description": "The person's name",
          "required": true,
          "readonly": false,
          "writeonly": false
        },
        {
          "property": "http://schema.org/birthDate",
          "hydra:title": "born",
          "hydra:description": "Date the person was born",
          "required": true,
          "readonly": false,
          "writeonly": false
        },
        {
          "property": "http://schema.org/spouse",
          "hydra:title": "spouse",
          "hydra:description": "Ther person's spouse, if any",
          "required": false,
          "readonly": false,
          "writeonly": false
        }
      ]
    }

This is a much larger document, but is pretty self-explanatory. Hydra is providing the supportedOperation and supportedProperty fields, which answer the last two Out-of-Band questions we had remaining.

supportedOperation lists all the HTTP verbs we can perform on this resource, what they expect to be provided, what status codes they may return, and the types of document it will give back. We can see that by performing a GET, we'll get a Person object back. We can do a DELETE to remove the object, and we don't have to provide anything, and don't expect anything back. We can do a PUT to update the person, and its expecting a Person document, and will give us one back, or a 404 if the person doesn't exist.

supportedProperty answers the question about which fields are required. We can see that if we want to update this person, we'll need to provide a name and their birthdate in the "born" field, but the spouse field is optional.

In-Band	Out-of-Band
Fields & Values
Where to find the spouse
Where & How to update this person
Possible attributes
Types of attributes
Which attributes are required
What HTTP verbs are allowed

Finally, we have an API with very little Out-of-Band knowledge required to consume it, or to write a client. In fact, if there's a Hydra client already available for your preferred language, it will probably automatically work with any Hydra API you can come up with, we won't have to waste time writing specific libraries in every language for every random API out there.

Additionally, Hydra and JSON+LD can be added to nearly every existing HTTP/RPC JSON API out there, simply by adding the @context field to link to the context and Hydra vocab. It can even be done without modifying the document itself, by adding an HTTP Link header.

The Hydra spec is getting the finishing touches now, and there's a few beginnings of client libraries out there already. It could definently use some wider attention, but in the meantime, JSON+LD is already a standard used by Google and Microsoft, and very well-supported. You can port or write your APIs in the JSON+LD style now, and get 90% of the advantages, and start enabling Hydra as it gets finalized. We could certainly use your help, come join us on the Hydra mailing list or in the HTTP-APIs Slack channel.