12. Rails – Create A Simple API Client To Fetch External Data

In the previous post, we created our sitemap and connected to Google. We’re ready to start populating some useful data.

In this chapter we will look at populating useful data to user from external sources. For most of these use-cases you need to create an API client to handle data capture and retention.

First and foremost, you need to obtain documentation for the API that you’re trying to access.

Make sure that you have the right privileges and authentication and make sure that you’re allowed to use it for whatever purpose you want to use it.

You will often have to sign up for a service in order to obtain authentication, which can usually be in form of API keys.

For this example, we will use Reed API client, you can find documentation for it here.

It’s invaluable to be able to test your API calls outside your code. There are multiple tools you can use for that but my favourite is Postman.

Whatever you decide to test with, just follow API documentation and make sure you’re able to get the responses you want. For instance, from Reed’s documentation we have:

https://www.reed.co.uk/api/{versionnumber}/search?keywords={keywords}&loc ationName={locationName}&employerId={employerId}&distanceFromLocation={distance in miles}
---
https://www.reed.co.uk/api/1.0/search?keywords=accountant&location=london&employerid=123&distancefromlocation=15

take the example and modify it to your needs:

https://www.reed.co.uk/api/1.0/search?keywords=java,ruby,developer&resultsToTake=1

Then test it with your application, in this example with Postman:

First, take the url you wish to use and put it in the request box. It will automatically extract the query params for you; or you could fill them out separately, this tool is also useful to generate (encode) the url with the params that you want to use.

Now you just want to set the headers up, but before that let’s encode your API key.

Note from the documentation it says the following:

Important:
You will need to include your api key for all requests in a basic authentication http header as the username, leaving the password empt

So the basic authentication is encrypted, since we have ruby and rails there’s a simple way we can do this in terminal:

# start rails console:
rails c

2.7.1 :002 > key =  "ENTER_YOUR_API_KEY_HERE"
2.7.1 :003 > Base64.strict_encode64 "#{key}:"
 => "This_will_be_the_encrypted_key_returned==" 

Make sure you include the : in the above command (just copy and paste the line) as I spent several minutes after I forgot about it!

Once you have this encoded key, let’s go to the headers tab of Postman and enter it, as well as expected response type.

The headers that you should enter are:

content-type: application/json
authorization: Basic <your_encrypted_api_key_pasted_here>

With this, you should be able to send the request.

You’re looking for a 200 (OK) response and the response that you expect. You may get a 500 (server error) or 401 (unauthorised). If you do, just try limit or remove some parameters and double check the headers again.

Now that the request itself is confirmed, we can go ahead and create a client to support this.

We will create a ‘generic’ client, or a ‘base’ client as well as the specific override to support this request.

So to begin, let’s create two files:

app/clients/resty/client.rb
app/clients/resty/response.rb

These will be the base client and response.

Let’s put the following content in the client.rb:

# frozen_string_literal: true

module Resty
  module Client
    def initialize(options = {})
      @base_url = options[:base_url]
    end

    def get(path, options = {})
      Response.new(RestClient.get(build_url(path), get_options(options)))
    end

    def post(path, payload, options = {})
      Response.new(RestClient.post(build_url(path), payload, options))
    end

    def put(path, payload, options = {})
      Response.new(RestClient.put(build_url(path), payload, options))
    end

    def patch(path, payload, options = {})
      Response.new(RestClient.patch(build_url(path), payload, options))
    end

    def delete(path, options = {})
      Response.new(RestClient.post(build_url(path), options))
    end

    private

    attr_reader :base_url

    def build_url(path)
      [base_url, path].join
    end

    def get_options(options)
      default_get_options.deep_merge(options.deep_symbolize_keys)
    end

    def default_get_options
      {
        accept: :json
      }
    end
  end
end

This client may support many request types, but we will just focus on the Get for the purpose of this post.

Let’s quickly cover what we have here.

First thing to note is that we’re utilising a RestClient here which will require the rest-client gem.

gem 'rest-client'

So go ahead and add that to your Gemfile and run bundle install

Then let’s look at the contents a bit more

    def initialize(options = {})
      @base_url = options[:base_url]
    end

    def get(path, options = {})
      Response.new(RestClient.get(build_url(path), get_options(options)))
    end

First off, the @base_url is not defined in this client, as it will be overridden by custom clients. The url itself is built like so:

def build_url(path)
  [base_url, path].join
end

where for example we wish for base_url to be: ‘https://www.reed.co.uk/api/1.0’

and path to be: ‘/search’ (with additional params)

The default_get_options are set to:

{
  accept: :json
}

As we will be sending JSON input. Note that we will need additional headers.

So let’s keep moving to the response.rb this one is relatively simple:

# frozen_string_literal: true

module Resty
  class Response
    delegate :code, to: :response

    def initialize(response)
      @response = response
    end

    def content
      Oj.strict_load(response.body)
    end

    private

    attr_reader :response
  end
end

The main thing is this:

def content
  Oj.strict_load(response.body)
end

We will simply put the response of the request into this module and load in in the constructor. It will load it using a gem called Oj which is optimized json.

So before you can use it you will just have to add it to your gemfile and run bundle install.

# put this in Gemfile
gem 'oj'

Ok now we’re getting to the specific details of the client itself, let’s make a file:

app/clients/reed_client.rb

and put the following into it:

# frozen_string_literal: true

class ReedClient
  include Resty::Client
  DEFAULT_RESULTS_TO_TAKE = 50

  def search(keywords:, location: nil, page: 1, per_page: DEFAULT_RESULTS_TO_TAKE)
    resultsToSkip = (page - 1) * per_page 
    params = {
      keywords: keywords.join(' '),
      locationname: location,
      distancefromlocation: 20,
      resultsToTake: per_page,
      resultsToSkip: resultsToSkip
    }

    get('/search', params: params)
  end

  def find(id)
    get("/jobs/#{id}")
  end

  private

  def default_get_options
    super.deep_merge(
      authorization: "Basic #{authorization}",
      params: {
        'content-type' => 'application/json'
      }
    )
  end

  def authorization
    Base64.strict_encode64 "#{ENV['REED_API_KEY']}:"
  end
end

Let’s go through this in a bit more detail.

  include Resty::Client
  DEFAULT_RESULTS_TO_TAKE = 50

  def search(keywords:, location: nil, page: 1, per_page: DEFAULT_RESULTS_TO_TAKE)
    resultsToSkip = (page - 1) * per_page 
    params = {
      keywords: keywords.join(' '),
      locationname: location,
      distancefromlocation: 20,
      resultsToTake: per_page,
      resultsToSkip: resultsToSkip
    }

    get('/search', params: params)
  end

So first of all, we include the base module.

We then define the search function, which is what we tested previously. This call will be responsible for making the following request: https://www.reed.co.uk/api/1.0/search

Then we define and populate the parameters that we care about here, some may be static and some passed from elsewhere. We then override the base get call.

The other private methods:

  def default_get_options
    super.deep_merge(
      authorization: "Basic #{authorization}",
      params: {
        'content-type' => 'application/json'
      }
    )
  end

  def authorization
    Base64.strict_encode64 "#{ENV['REED_API_KEY']}:"
  end

Here we override (and merge) the headers of the base GET request.

We add the specific authorization for the Reed API here with:

authorization: "Basic #{authorization}"

  def authorization
    Base64.strict_encode64 "#{ENV['REED_API_KEY']}:"
  end

Remember that to use this, in your terminal you should add the environment variable for the REED_API_KEY. You can do this simply with:

export REED_API_KEY=<your_key_here>

If you’re using Heroku, don’t forget to add this environment variable to your dyno settings.

There is a more secure way of doing it which is to add it to your Rails credentials, and it will look similar to:

  def authorization
    Base64.strict_encode64 "#{Rails.application.credentials.reed_api_key}:"
  end

Now that we have all of this in place, we can start testing and confirming that it all works as expected!

We can do this easily with rails console, so let’s do that and go through the steps.

rails c

2.7.1 :001 > reed_client = ReedClient.new(base_url: 'https://www.reed.co.uk/api/1.0')

2.7.1 :006 > reed_client.search(keywords: ['ruby'], location: 'london', page: 1, per_page: 2).content

 => {"results"=>[{"jobId"=>40796667, "employerId"=>300264, "employerName"=>"Client Server Ltd.", "employerProfileId"=>nil, "employerProfileName"=>nil, "jobTitle"=>"Senior Ruby Developer / Technical Lead - FinTech", "locationName"=>"Paddington", "minimumSalary"=>70000.0, "maximumSalary"=>80000.0, "currency"=>"GBP", "expirationDate"=>"25/09/2020", "date"=>"28/08/2020", "jobDescription"=>"Senior Ruby Developer / Technical Lead (FinTech RoR TDD). Market-disrupting FinTech is seeking an ambitious technologist to join them as they embark on an
 initiative to expand into international markets. You could take ownership of critical projects and accelerate your career. As a Senior Ruby Developer you'll play an integral role i
n a close-knit Agile team of eight developers. The development environment is test driven,... ", "applications"=>2, "jobUrl"=>"https://www.reed.co.uk/jobs/senior-ruby-developer-tech
nical-lead-fintech/40796667"}, {"jobId"=>40795921, "employerId"=>300264, "employerName"=>"Client Server Ltd.", "employerProfileId"=>nil, "employerProfileName"=>nil, "jobTitle"=>"Sof
tware Developer Ruby JavaScript Node.js", "locationName"=>"London", "minimumSalary"=>65000.0, "maximumSalary"=>80000.0, "currency"=>"GBP", "expirationDate"=>"25/09/2020", "date"=>"2
8/08/2020", "jobDescription"=>"Software Developer / Full Stack Engineer (Ruby JavaScript Node.js). Are you a technologist with a range of skills across the full stack who'd like to 
work on "tech for good" applications that affect people's everyday lives? You could be joining a scale-up technology company that is helping to drive efficiencies in patie
nt care for the NHS via a range of web based applications that enable doctors to access things like patien... ", "applications"=>8, "jobUrl"=>"https://www.reed.co.uk/jobs/software-d
eveloper-ruby-javascript-nodejs/40795921"}], "ambiguousLocations"=>[], "totalResults"=>110} 

And that works! Let’s go through the steps:

reed_client = ReedClient.new(base_url: 'https://www.reed.co.uk/api/1.0') 

The base client has an initialise function where we specify the base URL. This can be overridden from the reed_client.rb file so feel free to add that optimisation, or add it as a parameter as I have above.

The next line:

reed_client.search(keywords: ['ruby'], location: 'london', page: 1, per_page: 2).content

Don’t forget that ‘search’ function returns a response object, and we specified that def content will return the loaded Oj (json) content from the response. That’s what’s happening here.

We also specify the parameters for the search call, including keywords and location.

This lays the foundations in your project for making API calls with Rails.