Getting started with ScrapingBee and C#

In this tutorial, we will see how you can use ScrapingBee’s API with C#, and use it to scrape web pages. As such, we will cover these topics:

• General structure of an API request
• Create your first API request.

Let’s get started!

1. General structure of an API request

The general structure of an API request made in C# will always look like this:

using System;
using System.IO;
using System.Net;
using System.Web;
namespace test {
  class test{

      private static string BASE_URL = @"https://app.scrapingbee.com/api/v1/";
      private static string API_KEY = "YOUR-API-KEY";

      public static string Get(string url)
      {
            string uri = BASE_URL + "?api_key=" + API_KEY + "&url=" + url;
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
            request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

            using(HttpWebResponse response = (HttpWebResponse)request.GetResponse())
            using(Stream stream = response.GetResponseStream())
            using(StreamReader reader = new StreamReader(stream))
            {
                return reader.ReadToEnd();
            }
      }
  }
}

And you can do whatever you want with the response variable! For example:

public static void Main(string[] args) {
    Console.WriteLine("Response Body:");
    Console.WriteLine(Get("YOUR-URL"));
}

2. Create your first API request:

Let’s create a tool that saves the HTML code of ScrapingBee’s blog:

using System;
using System.IO;
using System.Net;
using System.Web;
namespace test {
  class test{

      private static string BASE_URL = @"https://app.scrapingbee.com/api/v1/?";
      private static string API_KEY = "YOUR-API-KEY";

      public static string Get(string uri)
      {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
            request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

            using(HttpWebResponse response = (HttpWebResponse)request.GetResponse())
            using(Stream stream = response.GetResponseStream())
            using(StreamReader reader = new StreamReader(stream))
            {
                return reader.ReadToEnd();
            }
      }

      public static void Main(string[] args) {

        var query = HttpUtility.ParseQueryString(string.Empty);
        query["api_key"] = API_KEY;
        query["url"] = @"https://scrapingbee.com/blog";
        string queryString = query.ToString(); // Transforming the URL queries to string

        string output = Get(BASE_URL+queryString); // Make the request

        string path = @"./ScrapingBeeBlog.html"; // Output file
        if (!File.Exists(path))
        {
            // Create a file to write to.
            using (StreamWriter sw = File.CreateText(path))
            {
                sw.Write(output);
            }
        }
      }
  }
}