[goshs] Part #4 - Eyecandy, anyone?
Tutorial Series
This is Part 4 of a tutorial series. Also see:
- Part#1 - My take on SimpleHTTPServer in go
- Part#2 - Trying to achieve code quality
- Part#3 - I can haz featurez?
- Part#4 - Eyecandy, anyone?
Summary
In this chapter I want to implement a bit of eyecandy. I was inspired by a beautiful project called updog by sc0tfree. His solution is almost the same but written in python. I decided to take his html construct and transform it to goshs
rewriting it to fit my needs.
Embedding Files
Well, to be honest I dislike the fact that I will have to use a third-party library for this purpose, but when it comes to this topic parcello
is doing its job pretty well. Also there is this proposal which might get integrated into go 16.1
soonish. So I might come back to this topic and migrate to a standard library later on.
Static folder
To get parcello working I created a new folder called static
. And within this folder I defined a package.go
with this content:
The package is basically empty but there is a comment to tell go generate
to run the command parcello -r -i *.go
everytime. We will see what this does in a second.
Referencing embedded files
Also you will need to import this package where you plan to use parcello embedded files. So in internal/myhttp/fileserver.go
we make sure to import parcello and our static package as so:
So what is going on here, you ask? The _
imports the package but omits the compiler error when not using an imported package (which we actually do not).
Workflow optimization
It is mandatory that you run go generate
before you go build
for this to work, as mentioned above. So to have a better workflow I rewrote my Makefile
:
.PHONY: build
generate:
@echo "[*] Minifying css and js"
@find static/ -type f -name "*.js" ! -name "*.min.*" -exec echo {} \; -exec uglifyjs -o {}.min.js {} \;
@find static/ -type f -name "*.css" ! -name "*.min.*" -exec echo {} \; -exec uglifycss --output {}.min.css {} \;
@echo "[OK] Done minifying things"
@echo "[*] Embedding via parcello"
@PARCELLO_RESOURCE_DIR=./static go generate ./...
@echo "[OK] Done bundeling things"
build: clean generate
@echo "[*] go mod dowload"
@go mod download
@echo "[*] Building for linux"
@GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o dist/linux_amd64/goshs
@GOOS=linux GOARCH=386 go build -ldflags="-s -w" -o dist/linux_386/goshs
@echo "[*] Building for windows"
@GOOS=windows GOARCH=amd64 go build -ldflags="-s -w" -o dist/windows_amd64/goshs.exe
@GOOS=windows GOARCH=386 go build -ldflags="-s -w" -o dist/windows_386/goshs.exe
@echo "[*] Building for mac"
@GOOS=darwin GOARCH=amd64 go build -ldflags="-s -w" -o dist/darwin_amd64/goshs
@echo "[*] Building for arm"
@GOOS=linux GOARCH=arm GOARM=5 go build -ldflags="-s -w" -o dist/arm_5/goshs
@GOOS=linux GOARCH=arm GOARM=6 go build -ldflags="-s -w" -o dist/arm_6/goshs
@GOOS=linux GOARCH=arm GOARM=7 go build -ldflags="-s -w" -o dist/arm_7/goshs
@GOOS=linux GOARCH=arm64 go build -ldflags="-s -w" -o dist/arm64_8/goshs
@echo "[OK] App binary was created!"
run:
@go run main.go
install:
@go install ./...
@echo "[OK] Application was installed to go binary directory!"
clean:
@rm -rf ./dist
@echo "[OK] Cleaned up!"
There is now going on a lot. Let’s talk about the generate
part. Besides minifying css and js (you will need to install the corresponding tools yourself), I am running this command: @PARCELLO_RESOURCE_DIR=./static go generate ./...
. This will tell go to recursively embed all files in the static directory into the build process.
Also you can see I enhanced my build method a bit to build executables for different operating systems.
When running make generate
it does look like this (I already have a lot of static files in my folder):
❯ make generate
[*] Minifying css and js
static/js/main.js
static/css/style.css
[OK] Done minifying things
[*] Embedding via parcello
Compressing 'css/bootstrap.min.css'
Compressing 'css/bootstrap.min.css.map'
Compressing 'css/style.css'
Compressing 'css/style.css.min.css'
Compressing 'fonts/FiraCode-VF.woff'
Compressing 'fonts/FiraCode-VF.woff2'
Compressing 'images/favicon.gif'
Compressing 'images/goshs-logo.png'
Compressing 'js/jquery-3.5.1.min.js'
Compressing 'js/main.js'
Compressing 'js/main.js.min.js'
Compressing 'templates/404.html'
Compressing 'templates/500.html'
Compressing 'templates/index.html'
Compressing 'vendor/datatable/jquery.dataTables.min.css'
Compressing 'vendor/datatable/jquery.dataTables.min.js'
Compressing 'vendor/fontawesome-5.15.1/css/all.min.css'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-brands-400.eot'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-brands-400.svg'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-brands-400.ttf'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-brands-400.woff'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-brands-400.woff2'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-regular-400.eot'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-regular-400.svg'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-regular-400.ttf'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-regular-400.woff'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-regular-400.woff2'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-solid-900.eot'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-solid-900.svg'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-solid-900.ttf'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-solid-900.woff'
Compressing 'vendor/fontawesome-5.15.1/webfonts/fa-solid-900.woff2'
Compressing 'vendor/images/sort_asc.png'
Compressing 'vendor/images/sort_asc_disabled.png'
Compressing 'vendor/images/sort_both.png'
Compressing 'vendor/images/sort_desc.png'
Compressing 'vendor/images/sort_desc_disabled.png'
Embedding 37 resource(s) at 'resource.go'
[OK] Done bundeling things
This process will result in a file resource.go
in our static directory:
// Code generated by parcello; DO NOT EDIT.
// Package static contains embedded resources
package static
import "github.com/phogolabs/parcello"
func init() {
parcello.AddResource([]byte{
80, 75, 3, 4, 20, 0, 8, 0, 8, 0, 243, 72, 77, 81, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 21, 0, 9, 0, 99, 115, 115, 47,
98, 111, 111, 116, 115, 116, 114, 97, 112, 46, 109, 105, 110,
46, 99, 115, 115, 85, 84, 5, 0, 1, 90, 110, 133, 95, 236,
189, 109, 143, 227, 56, 146, 32, 252, 253, 126, 133, 55, 11,
133, 234, 154, 150, 220, 146, 108, 217, 78, 39, 186, 177,
187, 131, 93, 220, 2, 221, 243, 97, 231, 14, 56, 160, 175,
[ ... ommited ...]
})
}
Basically our static content is transfered to a big byte slice and can be called and handled by the parcello package later on. We will now redesign our template and open it via parcello instead of referencing it as a constant.
New folder structure
[...omitted...]
├── internal
│ ├── myca
│ │ └── ca.go
│ ├── myhttp
│ │ └── fileserver.go
│ ├── mylog
│ │ └── log.go
│ └── myutils
│ └── utils.go
[...omitted...]
└── static
├── css
│ ├── bootstrap.min.css
│ ├── bootstrap.min.css.map
│ ├── style.css
│ └── style.css.min.css
├── fonts
│ ├── FiraCode-VF.woff
│ └── FiraCode-VF.woff2
├── images
│ ├── favicon.gif
│ └── goshs-logo.png
├── js
│ ├── jquery-3.5.1.min.js
│ ├── main.js
│ └── main.js.min.js
├── package.go
├── resource.go
├── templates
│ ├── 404.html
│ ├── 500.html
│ └── index.html
└── vendor
├── datatable
├── fontawesome-5.15.1
└── images
So these are the two folders which are most important to goshs
. And as you can see I transfered the templates to the static folder and removed the myhtml
package completely.
I will not go into detail which third-party javascript libraries and stylesheets I am using as this is not the main topic of this blog post. But I will describe how I integrated them with goshs
to work smoothly.
Error handling templates
As the templates for error handling do not have dynamic content I just copied over the html markup to the corresponding files 404.html
and 500.html
. It is still on my roadmap to design an even better error page with dynamic content displaying the actual error to the user.
Utilities package
I implemented a utility package which has one function by now:
To be honest, I found this one when searching for a way to express a human readable file size. I am quite new to go myself and do still not understand what exactly is going on here. So if you want to be a really smarta** write in the comments down below what this function exactly does. For all I know it converts the byte size (returned by os.Stat().Size()) to a humand readable string. And this is the string I will use later on to display the size of a file to the user.
fileserver.go
Well, this file was undergoing a big change for my new design to work. The point is, when embedding the static files like javascript and css you need to serve this to your html page for them to work. As goshs
serves the directory content to the user I cannot just use the root path to reference the css, right? What would happen if my html markup would do something like this?:
<html>
<head>
<style rel="stylesheet" href="/css/style.css">
</head>
<body>
...
</body>
</html>
The webserver would look into the folder we are in and try to find the css folder in there. So as long as the user does not have a css folder with a style.css in it there would be an error, right?
So I came up with a little trick to solve this issue.
The special folder
I decided to use a “special” folder to serve my static content from. So I chose a not so random sha256 hashsum to be the folders name.
echo "NameOfFirstChildAndNameOfSecondChild" | sha256sum
This is how I constructed a hashsum of the value 425bda8487e36deccb30dd24be590b8744e3a28a8bb5a57d9b3fcd24ae09ad3c
and this will be our special folder or more likely our special path as you will see now:
So in addition to distinguish between a GET and a POST request for handling the upload function as described in “Part#3 - I can haz featurez?” we now have to check if the request path will contain our “special” path and then handle the request with an additional handler serving our embedded static content.
For this purpose I coded the handler fs.static(w, req)
like so:
First of all I split the path and extract everything after the special string. This is to determine which file in which folder to serve within the static directory. So for example if the user request is /425bda8487e36deccb30dd24be590b8744e3a28a8bb5a57d9b3fcd24ae09ad3c/css/style.css staticPath
will be just css/style.css. And this is what we need next to open the File using parcello.Open()
. This function of the package parcello will open the file like os
would do with the difference that it will open it from the resources.go
we generate with go generate
.
Next up I read in the file content resulting in []bytes. Then I determine the content type by looking at the extension. I am using mime.TypeByExtension()
which will return the mimetype as string.
Finally I set the “Content-Type” Header in our response to the browser and then write the file content to the response.
This way I can serve every possible static file to the browser. So basically everytime the browser calles something in our “special” path it will just be served from our static folder. Cool, right?
But there are obvious trade-offs, you say? Well, that is perfectly correct. If a user has a folder with the name of our special path on disk, then goshs
will not be able to serve its content, as it will confuse the folder with our special path. To prevent that from happening I handled this situation before it can occur. I decided to not add a folder with the name of our special path to the dir listing displayed to the user like so:
So when iterating over the content of a folder in our processDir()
function I will just not add a folder called like our special path to the items slice we are generating. So even if there is a folder with this name it will just not be displayed and therefore nothing can be confused. This was the best solution I could think of.
Attributes, we need attributes!
As you can see from the screenshot of the final design we will need a few new fields in our item struct and in general. Those are: Size, LastModified, goshsVersion. Also what cannot be seen from the screenshot is that we also need to split our path we got into relpath and abspath for the template to work.
So from this (v0.0.3):
type directory struct {
Path string
Content []item
}
type item struct {
URI string
Name string
}
we need to change to this (v0.0.4):
And to fill in the values to this new attributes we will now look at the all new processDir
function, shall we?
Okay, I know. This is a lot to take in. So I left markers in the code to break that down in sections.
Section 1: Fill slice
In this part of the new processDir
function I am filling the items slice with the content of the directory. I do this like i did before but now I am filling in more information.
First I create an empty slice as before and then I iterate over the content of the folder. For each item I will define its Name, URI, Size (using our utils function to represent a human readable size), sort size (which is just the byte size to be able to sort), last modification time (as formatted string) and last modification time as timestamp (to be able to sort correct, too). This is this part:
Now comes the part with the special folder. I will skip this. But I have to mention that I added a item.IsDir = true
to the item, if it is a directory. This is important for the template later on.
Then I define the item with all the data and add it to the slice like this:
So now the item is in our slice with all the juicy new information, nice!
Section 2: Template parsing
In this part of the new processDir
function we now need to parse the new template and provide all the new information.
So first of all the new template is opened with parcello
as we did it in the static handler before like so:
Now instead of feeding a constant string to the template engine we feed an actual embedded html file to it. Isn’t this convenient?
Now lets construct the directory struct we feed to our template:
First we hand in the information we already have. Those are the relative path, the absolute path which is were we are on the disk, our items slice and finally the static goshs
version. So far so good.
To be able to jump back one layer without hitting the back button of the browser we implement a value called Back
to be able to link to it like so:
If we are not in the exact root of our file server we are in a subDirectory
therefore set it to true. Then we do a bit of string splitting and transformation to construct the relative path we were in before.
This means that if we are in /tmp/layer1/layer2/layer3/layer4
our backString now holds /tmp/layer1/layer2/layer3
. We assign this to d.Back
to be able to hand it to our template.
If we are at root level (/) we set subDirectory
to false and there will not be a value for Back
.
Finally we execute our template like so:
Instead of the constant we now parse the content of embedded the index file and execute the template.
This applies to the handle404
and handle500
function, as well. For example:
[...ommited...]
file, err := parcello.Open("templates/404.html")
[...ommited...]
t := template.New("404")
t.Parse(string(fileContent))
t.Execute(w, nil)
Index template
Well, speaking of which, we will now look at the important parts of the template. I will break it down and explain what is happening.
In the head section you can see how I reference all the styles we need to apply to make goshs
look like it does from our special path. Again, all credits go to sc0tfree and updog as I took his template as basis for this.
In the body there is a header section to display the logo and the directory we are in. Here (and in the title) I use the absolute path instead of the relative path.
Then there is the upload form consisting of a upload file dialog and a submit button. If you look closely you can already guess what our bonus section will be. But I will not point it out right now 😎.
I still do distinguis between the root path and any sub path to make the upload form work. Here we could have used {{.isSubdirectory}}
, too.
Next up if we are in a subdirectory I will display a back button to bring us back to the path which will be in {{ .Back }}
. See how it all connects?
In this code part I will construct the table holding the information. I will again do a {{range .Content}}
to loop over our items slice.
If {{ .IsDir }}
is true the icon will be a folder icon. Otherwise it will be a file icon.
Then there is the name with the link to the URI. And then there is the column for size and last modified.
I use a third-party javascript library to be able to sort the table later on. So for the data-order attribute I will fill in the {{.SortX}}
value, whereas for displaying information I will use the {{.DisplayX}}
value. This way I display human readable data but am able to sort correctly.
Finally there is a small footer showing the goshs
version and the scripts I need to include for everything to work.
Third-party libraries I use
- Bootstrap
- jQuery
- Datatables
- Font Awesome
- Fira Code as font
Conclusion
Compiling all this will give us a beautiful goshs
design with a lot of functionality, right?
Datatables will also add a search field and the total count of the items automatically.
As always you can view the complete code for this part at the github repository as tag v0.0.4 and I am happy to get some feedback, github stars, issues (if any), feature requests and such.
Bonus: Multiple files upload
So as promised here is the bonus. Some of you may have already seen the multiple
flag within our file upload form. So we need to implement that in our upload handler:
So to make this work we need to wrap our file save procedure into a range loop which will loop over all the provided files. So instead of just parsing one file we need to parse the fultipart Form a little differently.
This req.ParseMultipartForm(10 << 20)
and this m := req.MultipartForm
in combination with this files := m.File["files"]
will give us a handler to all files provided by the form.
So we then are able to loop over files
where i
will be the representation of every single file provided.
Finally we just treat the single file as before by constructing a savepath, creating its existance and reading the actual content and then finally writing it to the disk.
This does not differ from the previous procedure.
Thanks for taking the time to read. Comment down below. Have a nice day.
Patrick