Go Module Internals

Inside the Zip Files

24 September 2019

Dmitri Shuralyov

Google, Go team

Intro

2

Go Module Services

(Source: groups.google.com/d/msg/golang-announce/S0CoOrZWDck/pQX-gnBwBAAJ)
3

Go 1.13

(Source: golang.org/cmd/go/#hdr-Module_configuration_for_non_public_modules)
4

Quick Demo

GOPATH mode:

$ export GO111MODULE=off
$ go get honnef.co/go/tools/cmd/staticcheck

Module mode:

$ export GO111MODULE=on
$ go get honnef.co/go/tools/cmd/staticcheck

In Go 1.13, can now do:

$ go version ./bin/staticcheck

Programs can use debug.ReadBuildInfo() for introspection.

(Sources: golang.org/doc/go1.13#go-command and godoc.org/runtime/debug#ReadBuildInfo)
5

Caution

6

Personal website

I have one of those!

7

What are module zip files

8

Previously in Go...

GOPATH mode:

go get [-u] import/path
9

Previously in Go...

10

Previously in Go...

golang.org/issue/19614 - proposal: cmd/go: go get should support source archives

11

What are module zip files

"[...] when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, [...]"

(Source: golang.org/cmd/go/#hdr-Module_proxy_protocol)
12

Example

Let's start simple and work our way up.

package main

import (
    // We'll create this next:
    "github.com/dmitshur-test/nyc2019sep/hi"

    "fmt"
)

func main() {
    fmt.Println(hi.Text)
}

$ cd ~/go
$ mkdir -p src/nyc.example
$ cd src/nyc.example
...
13

Step 1

14

Module zip archive

The zip archive for a specific version of a given module is a standard zip file that contains the file tree corresponding to the module's source code and related files.

The archive uses slash-separated paths, and every file path in the archive must begin with <module>@<version>/, where the module and version are substituted directly, not case-encoded.

The root of the module file tree corresponds to the <module>@<version>/ prefix in the archive.

example.com/Abc@v1.0.0
├── go.mod
├── a.go
├── b.go
└── dir
    └── c.go

(Source: golang.org/cmd/go/#hdr-Module_proxy_protocol)
15

Zip files

Look at a zip:

$ unzip -Z1 v1.0.0.zip

Create a zip:

# -r   recurse into directories
# -D   do not add directory entries

$ zip -r -D v1.0.0.zip 'example.com/project@v1.0.0'

(Maybe don't push to production right away though!)

16

Checksums

The "h1:" directory hash function uses SHA-256 and is documented at godoc.org/golang.org/x/mod/sumdb/dirhash#Hash1.

You can try it on a zip:

$ goexec 'dirhash.HashZip("v1.0.0.zip", dirhash.Hash1)'
(string)("h1:qbmjkFj7Mx6ih7qhHhptxkar8UII0/4eV6ErleN5PRM=")
(interface{})(nil)

Or a go.mod file:

$ goexec 'dirhash.Hash1(
    []string{"go.mod"},
    func (string) (io.ReadCloser, error) {
        return os.Open("v1.0.0.mod")
    })'
(string)("h1:sq+2TcA+fKd4qCU0iVD1kEphBJLlgD/0mS5W+vKb7FE=")
(interface{})(nil)

(Source: godoc.org/golang.org/x/mod/sumdb/dirhash)
17

Step 2

18

Module proxy protocol

GET $GOPROXY/<module>/@v/list
GET $GOPROXY/<module>/@v/<version>.info
GET $GOPROXY/<module>/@v/<version>.mod
GET $GOPROXY/<module>/@v/<version>.zip

(Source: golang.org/cmd/go/#hdr-Module_proxy_protocol)
19

Escaped paths

package module // import "golang.org/x/mod/module"

func EscapePath(path string) (escaped string, err error)
func EscapeVersion(v string) (escaped string, err error)

func UnescapePath(escaped string) (path string, err error)
func UnescapeVersion(escaped string) (v string, err error)

For example:

github.com/Azure/azure-sdk-for-go             <-> github.com/!azure/azure-sdk-for-go
github.com/GoogleCloudPlatform/cloudsql-proxy <-> github.com/!google!cloud!platform/cloudsql-proxy
github.com/Sirupsen/logrus                    <-> github.com/!sirupsen/logrus
github.com/shurcooL/githubv4                  <-> github.com/shurcoo!l/githubv4

(Source: golang.org/x/mod/module)
20

Step 3

21

Simple dynamic proxy

Can play with:

// List endpoint.
http.HandleFunc("/modproxy/nyc.example/@v/list", func(w http.ResponseWriter, req *http.Request) {
    w.Header().Set("Content-Type", "text/plain; charset=utf-8")
    io.WriteString(w, "v1.0.0\n")
})

// Serve the .info file.
http.HandleFunc("/modproxy/nyc.example/@v/v1.0.0.info", func(w http.ResponseWriter, req *http.Request) {
    w.Header().Set("Content-Type", "text/plain; charset=utf-8")
    io.WriteString(w, "{\n\t\"Version\": \"v1.0.0\",\n\t\"Time\": \"2019-05-04T15:44:36Z\"\n}\n")
})

// Serve the .mod file.
http.HandleFunc("/modproxy/nyc.example/@v/v1.0.0.mod", func(w http.ResponseWriter, req *http.Request) {
    w.Header().Set("Content-Type", "text/plain; charset=utf-8")
    io.WriteString(w, "module nyc.example\n")
})
22

Simple dynamic proxy

http.HandleFunc("/modproxy/nyc.example/@v/v1.0.0.zip", func(w http.ResponseWriter, req *http.Request) {
    w.Header().Set("Content-Type", "application/zip")
    z := zip.NewWriter(w)
    for _, file := range []struct {
        Name, Body string
    }{
        {"nyc.example@v1.0.0/go.mod", "module nyc.example\n"},
        {"nyc.example@v1.0.0/p.go", "package p\n\n// Life is the answer.\nconst Life = 42\n"},
    } {
        f, err := z.Create(file.Name)
        if err != nil {
            panic(err)
        }
        _, err = f.Write([]byte(file.Body))
        if err != nil {
            panic(err)
        }
    }
    err := z.Close()
    if err != nil {
        panic(err)
    }
})
23

Previously in Go... (reminder)

GOPATH mode:

go get [-u] import/path
24

Now in Go...

Module mode:

can use git, hg, ...         - direct from VCS
can use mod (zip, mod, info) - via module proxy  # very common thanks to Module Mirror!
25

Observation

26

Step 4

27

Look, no git!

export PATH="$HOME/bin/nogit:$PATH"
git version

Try to download a module:

mkmoddirect
go get -d dmitri.shuralyov.com/text/kebabcase

Again with a proxy:

export GOPROXY=https://proxy.golang.org,direct
go get -d dmitri.shuralyov.com/text/kebabcase
28

Some thoughts

Reading module zips can be useful for:

29

Some thoughts

Writing module zips can be useful for:

Right now, go mod download is the only officially supported way to create module zip files.

(Source: golang.org/cmd/go/#hdr-Download_modules_to_local_cache)
30

One day...

31

One day...

32

One day...

33

One day...

34

One day...

35

Conclusion

36

Thank you

Use the left and right arrow keys or click the left and right edges of the page to navigate between slides.
(Press 'H' or navigate to hide this message.)