Static File Server With Go (Part 3: Organizing the code)

by Aaron Lenoir

In my previous post I talked about how I changed some of the default behaviour of Go's http.FileServer to fit my limited needs.

Basically, I had built a number of http.Handler implementations that would only hand over some of the incoming requests to http.FileServer, and ignore others.

Obviously, this approach has some issues. Since I'm just blacklisting some requests there's a chance certain requests can be accepted that I actually did not want to accept. Also, in the whole thing I completely ignore any caching support.

But that is not the point here.

The point is that at first I had created all these handlers in one big Go file. And I was wondering how I could split this functionality up into a "re-usable" Go package. Here's what I tried ...

Go's Workspace

On the golang website, there's a section called How to Write Go Code. The first thing that's introduced is the "Workspace". Where you get dictated a very specific setup of your working directories.

Three folders are suggested:

  • src
  • pkg
  • bin

All source code in src, package objects in pkg and executables come in bin.

The name of the folder where you application or package's source code is under dictates how you can include it in other projects.

One interesting thing in the guide is this:

"If you keep your code in a source repository somewhere, then you should use the root of that source repository as your base path. For instance, if you have a GitHub account at github.com/user, that should be your base path."

So, since I have a github.com account, I created a folder in my workspace:

  • github.com/AaronLenoir

I then created a new empty repository "fileserver" on github.com. And cloned it into

  • src/github.com/AaronLenoir/fileserver

I now had a go package with no source code.

Without github

You could do this without github or any online source code repository and just create a local folder and reference that. But using github.com does have some advantages.

Importing the Package

To import that newly created but empty package into another project, I could use the following import statement:

import (
    "github.com/AaronLenoir/fileserver"
)

Of course, this will give an error:

No buildable Go source files in ~/go/src/github.com/AaronLenoir/fileserver

This is because there is no code, yet.

Breaking Up The Code

There are many ways to go about this. But I always like to start writing down how I want to use my package and work from there.

So my goal was something like this:

s := fileserver.Server("folder", {forceHttps: true});

The second argument would then have some flags to indicate what functionality I wanted to enable.

I'm not sure if it's "idiomatic Go" to pass a list of configuration settings to another function. Perhaps a better option is simply to pass actual arguments for each setting. But it worked for me at the time.

Doing it this way does mean that my single fileserver has quite some mixed tasks:

  • Block requests for "dot-file"
  • Force https
  • Disallow directory listings

Creating the Server

The hook into my package is the "fileserver.Server()" function, which loads up a "server".

To create this function, I simply had to make a file in my package: fileserver.go.

In there, I simply add one function:

func Server(root string, config Config, filters []Filter) http.Handler {
    // ...
}

It looks a bit different from my original plan, but the purpose is the same: to spin up a server with the desired characteristics.

The three arguments are:

  • root: simply the path of the directory from which to start serving files
  • config: a simple struct with several bool fields representing configuration options
  • filters: an (optional) array of "Filter" implementations, can be nil
    • I use this to allow users to add their own functionality to filter requests, next to the "built in" ones

For the config, the struct looks like this:

type Config struct {
    ForceHTTPS     bool
    AllowDirList   bool
    AllowDotPrefix bool
}

Organizing Code Inside the Package

For the Filter feature, I had to create a "Filter" interface:

type Filter interface {
    Validate(path string) bool
}

Additionally, I had some implementations of filters, like:

// Used to prevent serving of files that are private
// Currently only basic check: does the name start with a .?
type privateFileFilter struct {
}

// Check if the path refers to a file or directory whose
// names starts with a .
func (f *privateFileFilter) Validate(fullPath string) bool {
    basePath := path.Base(fullPath)
    isPrivate := (strings.HasPrefix(basePath, "."))
    if isPrivate {
        return false
    } else {
        return true
    }
}

I could just add all that code to the same fileserver.go, but that's not very convenient. So I created a second file, "Filter.go", and added it there.

I do not have to import that file anywhere in order for it to be included in the build of the package.

In the same way, I implemented the "dot-file" filter and https forcing in a seperate file called "middleware.go". I don't know if that makes much sense but it did at the time, so that's good enough for now.

Tests in Go

Go's tooling foresees some test infrastructure. If you run the command "go test", go will try to find tests in your package and run them:

~/go/src/github.com/AaronLenoir/fileserver$ go test
PASS
ok      github.com/AaronLenoir/fileserver   0.059s

I created some tests in a file "fileserver_test.go". The "go test" command will only look at "_test" files. So the filename is important.

Here's one of my tests:

// Tests the fact that a private file will yield a 404 error.
func TestGetPrivateFile(t *testing.T) {
    config := Config{ForceHTTPS: false, AllowDirList: false, AllowDotPrefix: false}
    recorder := httptest.NewRecorder()

    RunGetRequest(config, "http://localhost:4003/test/.text.txt", recorder)

    if recorder.Code != 404 {
        t.Error("Expected 404, received ", recorder.Code)
    }
}

Important to note here is that this function name must start with "Test", and that it must have one argument "t *testing.T").

Using the Package

In the end, importing the package can be done by "import" of the following package name:

  • "github.com/AaronLenoir/fileserver"

If you use this package in a project, before you build you can run the "go get" command.

This will automatically fetch the git repo from github and put it in the expected location (src/github.com/AaronLenoir/fileserver).

This is the advantage of using something like github for your package.

:~/go/src/openshift/files$ go build
main.go:24:2: cannot find package "github.com/AaronLenoir/fileserver" in any of:
    /usr/lib/go/src/pkg/github.com/AaronLenoir/fileserver (from $GOROOT)
    /home/aaron/go/src/github.com/AaronLenoir/fileserver (from $GOPATH)
:~/go/src/openshift/files$ go get
:~/go/src/openshift/files$ go build

Comments and Documentation

Go has a built-in system for generating reference documentation for packages based on the comments in the source files. It's accesible through the "godoc" command.

This command can either print documentation on the comman line or spin up a local webserver so that you can browse through the docs (like you would on golang.org for the Packages).

For this to work, comments above functions, Interfaces, ... have to be a little bit structured.

For example, my main function to create a server has the following comments:

// Creates an http file server that will serve the files present in the root
// folder and its subfolders.
//
// root: directory from which to serve static files
//
// config: some configuration settings to define the behaviour of the static file
// server
//
// filters: Optionally some fileserver.Filter implementations to add custom
// content filtering (based on the full path of the file)
func Server(root string, config Config, filters []Filter) http.Handler {
    handler := http.FileServer(http.Dir(root))

    if config.ForceHTTPS {
        handler = forceHttps(handler)
    }

    internalFilters := getFilters(config)
    copy(internalFilters, filters)

    handler = protectFiles(handler, internalFilters, root)

    return handler
}

Running the following command will let me view the docs of my package on: http://localhost:6060/pkg/github.com/AaronLenoir/fileserver/

godoc -http:6060

Note how http://localhost:6060/ is now basically running the golang website. But if you browse to "Packages" you will also see your local packages documented.

Godoc.org Package Search

One other advantage of putting your packages online on github is that you will automatically get your code documentation available on godoc.org.

As an example, visit https://godoc.org/github.com/AaronLenoir/fileserver to view the latest docs from my little experiment.

This can help you browse through the mass of go projects on github.

Conclusion

To really do idiomatic Go all the way, I probably know too little about it. But at least I was able to:

  • Get the functionality that I wanted
  • Package and documented it correctly
  • Get it online and available for re-use

And this was all relatively easy, which means it is indeed possible to quickly become productive with Go. But as always, it takes a lot longer to become an expert.

Now all I need is something to actually server through my fileserver ...