[goshs] Part #1 - My take on SimpleHTTPServer in go
Tutorial Series
This is Part 1 of a tutorial series. Also see:
- Part#1 - My take on SimpleHTTPServer in go
- Part#2 - Trying to achieve code quality
- Part#3 - I can haz featurez?
- Part#4 - Eyecandy, anyone?
Motivation
There comes the time where you want to transfer files quick and easy from one host to another. And there comes the time where on the host the files reside there is no python installed. This is the reason I developed a tool called goshs which will replicate the function of python’s SimpleHTTPServer.
I want goshs to only use go’s standard libraries to prevent from any dependency issue and also I want it to ship as a single binary.
Python’s SimpleHTTPServer
Python’s SimpleHTTPServer does start a listener which will serve a folders content (the one you start it in) on TCP port 8000 bound to every interface on your machine like so:
❯ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...
You could add one argument to listen on another port like: python -m SimpleHTTPServer 8000 for example.
And it will also log accesses to said port:
❯ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...
127.0.0.1 - - [30/Sep/2020 13:15:17] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [30/Sep/2020 13:15:18] code 404, message File not found
127.0.0.1 - - [30/Sep/2020 13:15:18] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [30/Sep/2020 13:15:27] "GET /Downloads/ HTTP/1.1" 200 -
You can navigate folders (at least up and then down again to where it is executed) and can then download a file or view the content depending on mimetype.
So now let’s go and transfer it to go, shall we?
The Coding Journey
At it’s simplest form the following code snipped would work as a static file server already:
package main
import (
"fmt"
"net/http"
)
func main() {
fmt.Print("Serving HTTP on 0.0.0.0 port 8000\n")
h := http.FileServer(http.Dir("."))
http.ListenAndServe(":8000", h)
}
In the main function we output a message. In the next line we define a handler of the type http.FileServer which will server the directory .. And finally we start a http listener on TCP port 8000.
This will work just fine. But it will lack several “features” of the python web server.
So let’s break down the features in sections.
Custom Port
To give the user the possibility to choose a specific TCP port we will introduce this via the os library. The library is capable of reading the arguments provided, so we can choose a custom port.
First we define a new variable called port of type string and then we assign it depending on the length of arguments given:
func main() {
var (
port string
)
if len(os.Args) < 2 {
port = "8000"
} else {
port = os.Args[1]
}
Next we have to make sure we use that variable when printing the message and starting the server listener:
fmt.Printf("Serving HTTP on 0.0.0.0 port %s\n", port)
h := http.FileServer(http.Dir("."))
http.ListenAndServe(":"+port, h)
}
So far so good. We can now control the port the web server will be listening on.
Design
The python server will write the directory we are in at the top of the displayed page in a <h1> tag. And then there is a horizontal line (<hr />).
In the second section it will display the folders content as links but will prepend a bullet point (<ul><li>...</li></ul>). And then there is a horizontal line again.
Here is a screenshot for better reference:
The template
To achieve this in go we need to use the template engine. As we are composing html output we will use html/template over text/template.
To understand how the templates are used I will elaborate a bit about the structs I have chosen to make this work:
type directory struct {
Path string
Content []item
}
type item struct {
URI string
Name string
}
So as you can see we define a struct called directory. This struct will hold the path of the directory as string and it will hold a slice of items.
The item itself is another struct which will hold the URI and Name of the item.
This might get clearer as you read on. So let’s head over to the templates and look at how we fill in the information.
We define our html template string within our app as constants. We do not use a separate file, because we would have to ship it with our binary to the target or we would need third party libraries to include it with our binary.
const (
htmlTmp = `
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Directory listing for {{.Path}}</title>
</head>
<body>
<h1>Directory listing for {{.Path}}</h1>
<hr />
<ul>
{{range .Content}}
<li><a href="/{{.URI}}">{{.Name}}</a></li>
{{ end }}
</ul>
<hr />
</body>
</html>
`
notFoundTmp = `
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8">
<title>Error response</title>
</head>
<body>
<h1>Error response</h1>
<p>Error code: 404</p>
<p>Message: File not found.</p>
<p>Error code explanation: HTTPStatus.NOT_FOUND - Nothing matches the given URI.</p>
</body>
</html>
`
)
Let’s break down the upper template called htmlTmp. In the template you can see the expression {{.Path}} which will insert the value of Path in our directory struct when parsing the template.
And further down the lines you will see the links getting composed for every file or directory the server wants to display. They are inserted as hyperlinks within an unsorted list by iterating over the structs field Content and filling in the items URI and Name.
The second template I defined is a simple “404 Not Found” page which we will use later on. This template does not even have any variable bits and pieces.
The all new shiny handler
As the default http.FileServer handler is not capable of parsing a custom html template we need to reimplement this functionality from scratch. But fear not and bare with me, as this is not a too complicated task.
First we define a new handler function called handler, who would have guessed?
func handler(w http.ResponseWriter, r *http.Request) {
}
It will take a http.RepsonseWriter and a *http.Request as arguments. This will enable us to query request parameters, like for example the requested URL and it will make it possible to write output back to the browser.
So let’s fill in the handler. Our plan is devided in the following steps:
- Fetch some “assets” we will need
- Check if the requested URL is a directory or a file
- If it is a dir
- Read the directories content
- Iterate over it and fill an items struct with the content
- Sort the stuct
- Parse the template by providing a directory struct
- If it is a file
- Read the files content
- Write file to
http.ResponseWriter
1. Assets
So first we want to fetch some, as I call them, “assets” to work with. We need to know in which directory we are on the file system (a.k.a working directory) so we could access files on the file system with the absolute path. Once again the library os will have a function to return this information.
Next we want the path the web browser requested. The *http.Request will hold this information for us.
Finally we want to exclude the call to /favicon.ico which some browsers do by default. Otherwise this will clog our logging later on. And we do want to concatenate the working directory with the requested path. We will need this string a lot later on.
In our handler this would look something like this:
func handler(w http.ResponseWriter, r *http.Request) {
root, _ := os.Getwd() // ommitting error handling by using _
upath := r.URL.Path
if upath == "/favicon.ico" {
return
}
open := root + path.Clean(upath)
}
We also define a 404-handler function, as we would need to render 404-pages a lot. It will look like this:
func handle404(w http.ResponseWriter) {
t := template.New("404")
t.Parse(notFoundTmp)
t.Execute(w, nil)
}
This function will render our notFoundTmp to the browser when called. How this is working I will explain down the lines when we provide our directory listing to the browser.
2. Check if file or dir
The next step on our list will be to check if the requested URL is a file or a directory. I will do that with the os library as well. The function Stat will return an interface called os.FileInfo. This interface will have a boolean flag called Mode().IsDir(). So we can determine if the requested path is a file or a directory.
Let’s look at this code snippet:
fi, err := os.Stat(open)
if err != nil || os.IsNotExist(err) {
// Handle as 404
log.Printf("ERROR: cannot read file or folder: %+v", err)
handle404(w)
return
}
switch mode := fi.Mode(); {
case mode.IsDir():
case mode.IsRegular():
First we use os.Stat on the string open we defined at the beginning and recieve the os.FileInfo. If the file does not exist or there is another error we will parse our template notFoundTmp by calling the previously defined function handle404().
Then we switch over fi.Mode(). If mode IsDir() is true we will need to return our directory listing. Otherwise if mode IsRegular() is true we have to serve the files content.
3. Serve content if file
So I decided to implement this case first, as this was the easier task. The steps are basically reading the file in and writing it to http.ResponseWriter, done.
So let’s break down the following snippet:
case mode.IsRegular():
openfile, err := os.Open(open)
if err != nil || os.IsNotExist(err) {
// Handle as 404
log.Printf("ERROR: Cannot read file from disk: %+v", err)
handle404(w)
return
}
defer openfile.Close()
io.Copy(w, openfile)
To server the content of a file we first read in the file by using os.Open() and give it the absolute path from the beginning.
If there is an error reading the file we return our 404 page once again. This could be the case if the user for example has no permission to read the file or if it is not there.
Finally we use the standard library io and copy the content of the file directly to the http.ResponseWriter and therefore to the browser.
Well, and that’s it. Serving a file can be that easy.
4. Serve directory listing
So here comes the fun part. If you did read this far and you feel like you need to grab a coffee, this would be the perfect time before reading any further.
To serve the content of a directory we need to read in the directories os.FileInfo for every file or folder within the path and then we need to fill a slice of items to hold the correct information (Name and URI if you can recall from the beginning). We also want so sort the information afterwards and finally parse our template to write it back to the browser.
Looking at this snippet we could achieve it this way:
case mode.IsDir():
dir, err := os.Open(open)
if err != nil {
log.Printf("ERROR: Cannot read directory from disk: %+v", err)
handle404(w)
return
}
defer dir.Close()
fis, err := dir.Readdir(-1)
if err != nil {
log.Printf("ERROR: Cannot read directory content from disk: %+v", err)
handle404(w)
return
}
items := make([]item, 0, len(fis))
for _, fi := range fis {
itemname := fi.Name()
itemuri := url.PathEscape(path.Join(upath, itemname))
if fi.IsDir() {
itemname += "/"
}
item := item{
Name: itemname,
URI: itemuri,
}
items = append(items, item)
}
sort.Slice(items, func(i, j int) bool {
return strings.ToLower(items[i].Name) < strings.ToLower(items[j].Name)
})
t := template.New("index")
t.Parse(htmlTmp)
d := &directory{Path: upath, Content: items}
t.Execute(w, d)
Let’s break that down a little, shall we?
First we open the directory, like we did with the file using os.Open. Again we handle the error if something goes wrong.
Then we recieve the directories os.FileInfo as slice by using the library dir and it’s function Readdir. It will take one integer as argument. The key difference lies within the error handling of the function. If it is below 0 it will return just an error and an empty slice. If it is greater or equal to 0 it will try and read the FileInfo to a point where there might be an error and then return the filled slice and an error. So you might have half of the folders content FileInfo as a result. I prefer to have it all or an error. This is why I chose -1.
This here items := make([]item, 0, len(fis)) will define an empty slice with no content and the capacity of our directory content.
In a range loop we will then set the items name and the uri from the file information. Next we will append a single / if the item itself is a directory. Finally we will fill one item struct and append it to the items slice. So basically we are filling a slice named items with the folders content.
Afterwards we use the library sort to sort the items slice by Name. We force to sort it all lowercase, as otherwise the function would return a sorted block with uppercase names and then lowercase names below it. But as we wanted to replicate python’s behaviour we do it exactly this way.
Then comes the template parsing:
t := template.New("index")
t.Parse(htmlTmp)
d := &directory{Path: upath, Content: items}
t.Execute(w, d)
As promised I will elaborate a bit on how that works. We first define t to be a new template.Template by using the function New() of the template library. Then t is capable of parsing our const htmlTmp. When parsed we instantiate d as our directory struct giving it the Path and the filled slice items as Content.
When the template now renders it has access to the field {{.Path}} and {{.Content}}. The Content field itself on a per item basis has the fields {{.Name}} and {{.URI}}. And this is how we fill in the template dynamically.
Finally we just need to “execute” the template, which will write the rendered template to our http.ResponseWriter and thus deliver it to the browser. The data filled into the template will be in d.
5. Make the server use the handler
You do have to change one line in the main function, though:
func main() {
var (
port string
)
if len(os.Args) < 2 {
port = "8000"
} else {
port = os.Args[1]
}
fmt.Printf("Serving HTTP on 0.0.0.0 port %s\n", port)
http.HandleFunc("/", handler)
http.ListenAndServe(":"+port, nil)
}
At the second last line before closing the main function with a } instead of FileServer we are now using http.HandleFunc and give it our handler function to handle everything from / onwards.
Design is now finished
Now the design part is finished. Our server will display everything like the python server does:
The code looks now like this: code on gist
A full 184 lines of code application. Not bad. But we are not there, yet. We still need to log the requests to console, as the python server does.
Logging
The python server will print to stdout when someone makes a request to the server:
❯ python -m SimpleHTTPServer 9001
Serving HTTP on 0.0.0.0 port 9001
127.0.0.1 - - [02/Oct/2020 07:53:06] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [02/Oct/2020 07:53:20] "GET /temp/ HTTP/1.1" 200 -
127.0.0.1 - - [02/Oct/2020 07:53:25] code 404, message File not found
127.0.0.1 - - [02/Oct/2020 07:53:25] "GET /temp/foo HTTP/1.1" 404 -
As you can see the format is: <ip> - - [dd/mmm/yyyy hh:mm:ss] "<HTTP VERB> <path> <protocol>" <status code> -. Also this listing shows an error message if a file was not found. This message also applies to a file which the user has no access to.
We want to immitate this format with our goshs server. As we already have log.Printf statements in our code we might not do this part 1:1, but just add the formated message in addition to the logging we already have implemented.
So for our 404 handler we need to extend the arguments given by *http.Request to read the remote IP address from the request for logging. It will now look like this:
func handle404(w http.ResponseWriter, r *http.Request) {
// Log the request
timestamp := time.Now().Format("02/Jan/2006 15:04:05")
fmt.Printf("%s - - [%s] \"%s %s %s\" %+v -\n", r.RemoteAddr, timestamp, r.Method, r.URL.Path, r.Proto, "404")
fmt.Printf("%s - - [%s] code 404, message File not found\n", r.RemoteAddr, timestamp)
t := template.New("404")
t.Parse(notFoundTmp)
t.Execute(w, nil)
}
First we format the timestamp. The date shown in the code above is used to format it. Everytime you use the Second of January, 2006 at 15:04:05 you can format a datetime string with the function Format of the library time just the way you want. It is that easy.
Then to omit the timestamp given by the log library we use another standard library to write to stdout. This is fmt which stands for format if I am not mistaken.
So we feed the format given by python to the format string and just fill in the parts from the request and the timestamp.
Same applies to the logging in our handler. We will do the logging right after we know that the file requested exists.
if err != nil || os.IsNotExist(err) {
// Handle as 404
log.Printf("ERROR: cannot read file or folder: %+v", err)
handle404(w, r)
return
}
// Log the request
timestamp := time.Now().Format("02/Jan/2006 15:04:05")
fmt.Printf("%s - - [%s] \"%s %s %s\" %+v -\n", r.RemoteAddr, timestamp, r.Method, r.URL.Path, r.Proto, "200")
Here we again feed the format we want to be printed to the console to the format string and then fill in the information from the request and the timestamp.
And there you have it. The logging now looks like the one pythons server does:
❯ go run simplehttpserver.go 5000
Serving HTTP on 0.0.0.0 port 5000
127.0.0.1:34966 - - [02/Oct/2020 08:34:08] "GET / HTTP/1.1" 200 -
127.0.0.1:34966 - - [02/Oct/2020 08:34:11] "GET /go.mod HTTP/1.1" 200 -
2020/10/02 08:34:13 ERROR: cannot read file or folder: stat /tmp/foo: no such file or directory
127.0.0.1:34966 - - [02/Oct/2020 08:34:13] "GET /foo HTTP/1.1" 404 -
127.0.0.1:34966 - - [02/Oct/2020 08:34:13] code 404, message File not found
The final code can be found here: code on gist
Conclusion
I sure know that this is far from ideal and that the implemented code might lack some code quality. I really am aware of this. So think of this blog post as a first part for a really cool app. In the next chapters I want to look at the code and give it a good structure. Also I want to implement some extra features pythons SimpleHTTPServer is missing, like for example an upload function. Also I want to further design the page displayed to look somewhat more modern (I suck at this though 😱).
Thanks for baring with me and reading through all of this. I hope you enjoyed it. See you again, soon.
Patrick