Building our first service – finding the fastest mirror site from a list

With the concepts we have built up to now, let's write our first REST service. Many mirror sites exist for hosting operating system images including Ubuntu and Debian. The mirror sites here are nothing but websites on which OS images are hosted to be geographically close to the downloading machines.

Let's look at how we can create our first service:

Problem:

Build a REST service that returns the information of the fastest mirror to download a given OS from a huge list of mirrors. Let's take the Debian OS mirror list for this service. You can find the list at https://www.debian.org/mirror/list.

We use that list as input when implementing our service.

Design:

Our REST API should return the URL of the fastest mirror.

The block of the API design document may look like this:

      
HTTP Verb           PATH           Action           Resource

      
GET           /fastest-mirror           fetch           URL: string 
  

Implementation:

Now we are going to implement the preceding API step by step:

1. As we previously discussed, you should set the GOPATH variable first. Let's assume the GOPATH variable is /home/user/workspace. Create a directory called mirrorFinder in the following path. git-user should be replaced with your GitHub username under which this project resides:

mkdir -p $GOPATH/src/github.com/git-user/chapter1/mirrorFinder

2. Our project is ready. We don't have any data store configured yet. Create an empty file called main.go:

touch $GOPATH/src/github.com/git-user/chapter1/mirrorFinder/main.go

3. Our main logic for the API server goes into this file. For now, we can create a data file that works as a data service for our main program. Create one more directory for packaging the mirror list data: 

mkdir $GOPATH/src/github.com/git-user/chapter1/mirrors

4. Now, create an empty file called data.go in the mirrors directory. The src directory structure so far looks like this:

github.com \
-- git-user \
    -- chapter1 
       -- mirrorFinder \
           -- main.go
       -- mirrors \
           -- data.go

5. Let's start adding code to the files. Create an input data file called data.go for our API to use:

package mirrors

// MirrorList is list of Debian mirror sites
var MirrorList = [...]string{
  "http://ftp.am.debian.org/debian/", "http://ftp.au.debian.org/debian/",
  "http://ftp.at.debian.org/debian/", "http://ftp.by.debian.org/debian/",
  "http://ftp.be.debian.org/debian/", "http://ftp.br.debian.org/debian/",
  "http://ftp.bg.debian.org/debian/", "http://ftp.ca.debian.org/debian/",
  "http://ftp.cl.debian.org/debian/", "http://ftp2.cn.debian.org/debian/",
  "http://ftp.cn.debian.org/debian/", "http://ftp.hr.debian.org/debian/",
  "http://ftp.cz.debian.org/debian/", "http://ftp.dk.debian.org/debian/",
  "http://ftp.sv.debian.org/debian/", "http://ftp.ee.debian.org/debian/",
  "http://ftp.fr.debian.org/debian/", "http://ftp2.de.debian.org/debian/",
  "http://ftp.de.debian.org/debian/", "http://ftp.gr.debian.org/debian/",
  "http://ftp.hk.debian.org/debian/", "http://ftp.hu.debian.org/debian/",
  "http://ftp.is.debian.org/debian/", "http://ftp.it.debian.org/debian/",
  "http://ftp.jp.debian.org/debian/", "http://ftp.kr.debian.org/debian/",
  "http://ftp.lt.debian.org/debian/", "http://ftp.mx.debian.org/debian/",
  "http://ftp.md.debian.org/debian/", "http://ftp.nl.debian.org/debian/",
  "http://ftp.nc.debian.org/debian/", "http://ftp.nz.debian.org/debian/",
  "http://ftp.no.debian.org/debian/", "http://ftp.pl.debian.org/debian/",
  "http://ftp.pt.debian.org/debian/", "http://ftp.ro.debian.org/debian/",
  "http://ftp.ru.debian.org/debian/", "http://ftp.sg.debian.org/debian/",
  "http://ftp.sk.debian.org/debian/", "http://ftp.si.debian.org/debian/",
  "http://ftp.es.debian.org/debian/", "http://ftp.fi.debian.org/debian/",
  "http://ftp.se.debian.org/debian/", "http://ftp.ch.debian.org/debian/",
  "http://ftp.tw.debian.org/debian/", "http://ftp.tr.debian.org/debian/",
  "http://ftp.uk.debian.org/debian/", "http://ftp.us.debian.org/debian/",
}

We create a map of strings called MirrorList. This map holds information on the URL to reach the mirror site. We are going to import this information into our main program to serve the request from the client.

6. Open main.go and add the following code:

import (
  "encoding/json"
  "fmt"
  "log"
  "net/http"
  "time"

  "github.com/git-user/chapter1/mirrors"
)


type response struct {
  FastestURL string `json:"fastest_url"`
  Latency time.Duration `json:"latency"`
}


func main() {
  http.HandleFunc("/fastest-mirror", func(w http.ResponseWriter,
  r *http.Request) {
    response := findFastest(mirrors.MirrorList)
    respJSON, _ := json.Marshal(response)
    w.Header().Set("Content-Type", "application/json")
    w.Write(respJSON)
  })
  port := ":8000"
  server := &http.Server{
    Addr: port,
    ReadTimeout: 10 * time.Second,
    WriteTimeout: 10 * time.Second,
    MaxHeaderBytes: 1 << 20,
  }
  fmt.Printf("Starting server on port %sn", port)
  log.Fatal(server.ListenAndServe())
}

We created the main function that runs an HTTP server. Go provides the net/http package for that purpose. The response of our API is a struct with two fields:

• fastest_url: The fastest mirror site
• latency: The time it takes to download the README from the Debian OS repository

7. We will code a function called findFastest to make requests to all the mirrors and calculate the fastest of all. To do this, instead of making sequential API calls to each and every URL one after the other, we use Go routines to parallelly request the URLs and once a goroutine returns, we stop there and return that data back.:

func findFastest(urls []string) response {
  urlChan := make(chan string)
  latencyChan := make(chan time.Duration)

  for _, url := range urls {
    mirrorURL := url
    go func() {
      start := time.Now()
      _, err := http.Get(mirrorURL + "/README")
      latency := time.Now().Sub(start) / time.Millisecond
      if err == nil {
        urlChan <- mirrorURL
        latencyChan <- latency
      }
    }()
  }
  return response{<-urlChan, <-latencyChan}
}

The findFastest function is taking a list of URLs and returning the response struct. The function creates a goroutine per mirror site URL. It also creates two channels, urlChan and latencyChan, which are passed to the goroutines. In the goroutines, we calculate the latency (time taken for the request).

The smart logic here is, whenever a goroutine receives a response, it writes data into two channels with the URL and latency information respectively. Upon receiving data, the two channels make the response struct and return from the findFastest function. When that function is returned, all goroutines spawned from that are stopped from whatever they are doing. So, we will have the shortest URL in urlChan and the smallest latency in latencyChan.

8. Now if you add this function to the main file (main.go), our code is complete for the task:

9. Now, install this project with the Go command, install:

go install github.com/git-user/chapter1/mirrorFinder

This step does two things:

• Compiles the package mirrors and places a copy in the $GOPATH/pkg directory
• Places a binary in the $GOPATH/bin

10. We can run the preceding API server like this:

$GOPATH/bin/mirrorFinder

The server is up and running on http://localhost:8000. Now we can make a GET request to the API using a client such as a browser or a curl command. Let's fire a curl command with a proper API GET request.

Request one is as follows:

curl -i -X GET "http://localhost:8000/fastest-mirror" # Valid request

The response is as follows:

HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 27 Mar 2019 23:13:42 GMT
Content-Length: 64

{"fastest_url":"http://ftp.sk.debian.org/debian/","latency":230}

Our fastest-mirror-finding API is working great. The right status code is being returned. The output may change with each API call, but it fetches the lowest-latency link at any given moment. This example also shows where goroutines and channels shine.

In the next section, we'll look at an API specification called Open API. An API specification is for documenting the REST API. To visualize the specification, we will use the Swagger UI tool.