Retry failed requests in Ruby

For most websites, your first requests will always be successful, however, it’s inevitable that some of them will fail. For these failed requests, the API will return a 500 status code and won’t charge you for the request.

In this case, we can make our code retry to make the requests until we reach a maximum number of retries that we set:

require 'net/http'
require 'net/https'
require 'addressable/uri'

# Classic (GET)
def send_request(user_url)
    uri = Addressable::URI.parse("https://app.scrapingbee.com/api/v1/")
    api_key = "YOUR-API-KEY"
    uri.query_values = {
      'api_key'  => api_key,
      'url' => user_url
    }
    uri = URI(uri)

    # Create client
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_PEER

    # Create Request
    req =  Net::HTTP::Get.new(uri)

    # Fetch Request
    res = http.request(req)

    # Return Response
    return res
rescue StandardError => e
    puts "HTTP Request failed (#{ e.message })"
end

max_retries = 5
for a in 1..max_retries do
    request = send_request("https://scrapingbee.com")
    if not [404, 200].include?(request.code)
        puts "Request failed - Status Code: #{ request.code }"
        puts "Retrying..."
    else
        puts "Successful request - Status Code: #{ request.code }"
        puts request.body
        break
    end
end