Go (Golang) -- The Complete Deep Dive

1. Why Go Exists

In 2007, three engineers at Google -- Rob Pike, Ken Thompson, and Robert Griesemer -- sat down while waiting for a massive C++ build to finish. That build took 45 minutes. During that wait, they started sketching a new language. Go was born out of frustration with the real, daily pain of building software at Google's scale.

Google's Pain Points

Google's codebase in 2007 was enormous -- hundreds of millions of lines of C++, Java, and Python. The problems were acute:

• C++ compile times: A single Google binary could take 45+ minutes to compile. Engineers spent hours per day waiting for builds. Header file dependencies created cascading recompilation -- change one header, recompile the world.
• Header file hell: C++ uses textual inclusion (#include). A single translation unit might include tens of thousands of header files, most of which it doesn't need. This makes builds slow and dependency tracking fragile.
• Concurrency was painful: Writing correct concurrent code in C++ or Java required deep expertise with mutexes, condition variables, and thread pools. Most engineers got it wrong, leading to race conditions and deadlocks in production.
• Dependency management: C++ had no standard package manager. Java's classpath was a nightmare. Python's pip was unreliable for reproducible builds.
• Onboarding: C++ is so complex that new engineers at Google took months to become productive. The language spec is over 1,400 pages.

Go's Design Philosophy

Go was designed with a radical idea: less is more. The creators deliberately left out features that other languages consider essential:

• No classes or inheritance -- composition through embedding instead
• No exceptions -- explicit error values instead
• No generics (until Go 1.18 in 2022) -- simplicity over flexibility
• No implicit type conversions -- if you want to convert, say so explicitly
• No ternary operator -- just use an if statement
• One way to loop -- for is the only loop keyword
• Enforced code formatting -- gofmt is not optional, it's the culture
• Unused imports and variables are compile errors -- forces clean code

What Go Is Used For

Go dominates cloud infrastructure and backend services. The biggest projects in modern infrastructure are written in Go:

• Docker -- the container runtime that changed how we deploy software
• Kubernetes -- the container orchestration platform (100k+ lines of Go)
• Terraform -- infrastructure as code by HashiCorp
• CockroachDB -- distributed SQL database
• Prometheus -- monitoring and alerting toolkit
• Hugo -- the fastest static site generator
• etcd -- distributed key-value store used by Kubernetes
• Caddy -- modern web server with automatic HTTPS
• Grafana -- observability dashboards
• Vault -- secrets management by HashiCorp

Go vs Other Languages

Hello, Go

Go
package main

import "fmt"

func main() {
    fmt.Println("Hello, World!")
}

Run it with go run main.go. That's it. No build configuration, no Makefile, no CMakeLists.txt. Go's toolchain is batteries-included.

2. The Go Compiler

Understanding how Go turns your source code into a running binary helps you write faster code and debug performance issues. The Go compiler is one of the fastest compilers for a statically-typed language, and its design is a big reason why.

The Compilation Pipeline

When you run go build, your code passes through several stages:

Source Code (.go files)
        |
        v
  +------------+
  |   Lexer    |  -- Breaks source into tokens (keywords, identifiers, literals)
  +------------+
        |
        v
  +------------+
  |   Parser   |  -- Builds an Abstract Syntax Tree (AST) from tokens
  +------------+
        |
        v
  +------------+
  | Type Check |  -- Validates types, resolves names, checks interfaces
  +------------+
        |
        v
  +------------+
  | SSA Build  |  -- Converts AST to Static Single Assignment form
  +------------+
        |
        v
  +------------+
  | SSA Optim. |  -- Dead code elimination, constant folding, inlining
  +------------+
        |
        v
  +------------+
  | Code Gen   |  -- Generates machine code for target architecture
  +------------+
        |
        v
  +------------+
  |   Linker   |  -- Links packages into a single static binary
  +------------+
        |
        v
  Executable Binary

What Is SSA and Why Does It Matter?

Static Single Assignment (SSA) is an intermediate representation where every variable is assigned exactly once. If you need to update a variable, you create a new version of it instead.

Go
// Your code:
x := 5
x = x + 3
x = x * 2

// SSA form (conceptually):
// x1 = 5
// x2 = x1 + 3
// x3 = x2 * 2

Why bother? Because SSA makes optimizations dramatically easier for the compiler:

• Constant folding: The compiler can see that x1 = 5, so x2 = 5 + 3 = 8, and x3 = 8 * 2 = 16. The entire computation is replaced with x3 = 16 at compile time.
• Dead code elimination: If x1 and x2 are never used after being assigned, they can be removed entirely.
• Common subexpression elimination: If the same computation appears twice, it's computed once and reused.

Escape Analysis

One of Go's most important compiler features is escape analysis -- the compiler's decision about whether a variable should live on the stack (fast, automatically freed) or the heap (slower, garbage collected).

Go
// This stays on the stack -- x doesn't escape
func stackOnly() int {
    x := 42
    return x  // returning a copy of the value
}

// This escapes to the heap -- we return a pointer to x
func escapesToHeap() *int {
    x := 42
    return &x  // x must survive after the function returns
}

// This also escapes -- the slice may grow and be reallocated
func sliceEscape() {
    s := make([]int, 0)
    for i := 0; i < 1000000; i++ {
        s = append(s, i)  // may cause heap allocation
    }
}

You can see exactly what the compiler decides using build flags:

Terminal
# Show escape analysis decisions
go build -gcflags="-m" main.go

# More verbose output (two -m flags)
go build -gcflags="-m -m" main.go

# Example output:
# ./main.go:8:2: x escapes to heap
# ./main.go:4:6: stackOnly x does not escape

Inlining

The Go compiler automatically inlines small functions -- it replaces the function call with the function body directly. This eliminates the overhead of the call itself and enables further optimizations.

Go
// This function is small enough to be inlined
func add(a, b int) int {
    return a + b
}

func main() {
    result := add(3, 4)
    // After inlining, the compiler sees this as: result := 3 + 4
    // After constant folding: result := 7
    fmt.Println(result)
}

The compiler uses a "cost" heuristic -- functions with loops, many statements, or complex control flow won't be inlined. You can see inlining decisions in the -gcflags="-m" output:

Terminal
# Output shows:
# ./main.go:3:6: can inline add
# ./main.go:8:15: inlining call to add

Why Go Compiles So Fast

Go was designed from day one for fast compilation. Several language design decisions enforce this:

• No header files: Unlike C/C++, Go packages are compiled once and export a binary interface. No textual inclusion, no re-parsing.
• Dependency graph must be a DAG: Circular imports are a compile error. This means packages can be compiled in topological order, enabling massive parallelism.
• Unused imports are errors: This isn't just about clean code -- it means the compiler never parses or type-checks code you don't use.
• Simple grammar: Go's grammar is remarkably simple. No angle brackets for generics (originally), no operator overloading, no implicit conversions. The parser is fast because the language is simple.
• Export format: Compiled packages export a compact binary format, not source text. Importing a package reads a small file, not thousands of lines of headers.

Cross-Compilation

Go can cross-compile for any supported platform with just two environment variables. No separate toolchain required:

Terminal
# Build for Linux on AMD64 (most cloud servers)
GOOS=linux GOARCH=amd64 go build -o myapp-linux

# Build for macOS on Apple Silicon
GOOS=darwin GOARCH=arm64 go build -o myapp-mac

# Build for Windows
GOOS=windows GOARCH=amd64 go build -o myapp.exe

# Build for ARM (Raspberry Pi)
GOOS=linux GOARCH=arm GOARM=7 go build -o myapp-pi

# See all supported platforms
go tool dist list

This produces a statically linked binary with zero runtime dependencies. No JVM, no Python interpreter, no .so files. Just copy the binary to the target machine and run it. This is one reason Go dominates the container world -- a Go binary in a FROM scratch Docker image is often under 10MB.

Go Build Flags & Compiler Options

The go build command accepts a rich set of flags that control compilation, linking, and output. Mastering these flags is essential for production builds, debugging, and performance analysis.

Common -ldflags (Linker Flags)

Linker flags control the final binary output. The most common use cases are stripping debug info for smaller binaries and injecting version information at build time.

Go
package main

import "fmt"

// These are set at build time via -ldflags
var (
    version   = "dev"
    commit    = "none"
    buildDate = "unknown"
)

func main() {
    fmt.Printf("Version: %s\nCommit: %s\nBuilt: %s\n", version, commit, buildDate)
}

Terminal
# Build with version info injected
go build -ldflags="-X main.version=1.2.3 -X main.commit=$(git rev-parse --short HEAD) -X 'main.buildDate=$(date -u)'" -o myapp

# Combine with stripping for smallest production binary
go build -ldflags="-s -w -X main.version=1.2.3" -o myapp

Common -gcflags (Compiler Flags)

Compiler flags let you inspect how the Go compiler optimizes your code. These are invaluable for performance tuning and understanding memory allocation behavior.

Terminal
# See escape analysis -- which variables go to the heap?
go build -gcflags="-m" ./...

# Even more detail on optimization decisions
go build -gcflags="-m -m" ./...

# Disable optimizations for debugging with Delve
go build -gcflags="-N -l" -o myapp-debug

# Print the generated assembly for a specific package
go build -gcflags="-S" ./internal/parser

Build Tags (Conditional Compilation)

Build tags let you include or exclude files from compilation based on OS, architecture, or custom tags. This is how you write platform-specific code or toggle features at build time.

Go
// New syntax (Go 1.17+) -- use this
//go:build linux

// Old syntax (still works but deprecated)
// +build linux

package platform

func GetConfigPath() string {
    return "/etc/myapp/config.yaml"
}

Go
// file: debug_on.go
//go:build debug

package main

import "log"

func debugLog(msg string) {
    log.Printf("[DEBUG] %s", msg)
}

Go
// file: debug_off.go
//go:build !debug

package main

func debugLog(msg string) {
    // no-op in production builds
}

Terminal
# Development build with debug logging
go build -tags debug -o myapp-debug

# Production build -- debugLog is a no-op, zero cost
go build -o myapp

go tool Commands

The go tool subcommand gives you access to lower-level tools that ship with the Go toolchain. These are essential for performance analysis and understanding compiled output.

Terminal
# Generate a CPU profile and analyze it
go test -cpuprofile cpu.prof -bench .
go tool pprof cpu.prof

# Generate a memory profile
go test -memprofile mem.prof -bench .
go tool pprof mem.prof

# Record an execution trace and visualize in browser
go test -trace trace.out
go tool trace trace.out

# Disassemble a specific function from a binary
go tool objdump -s "main.handleRequest" myapp

CGO: Calling C from Go

CGO allows Go programs to call C code. By default, CGO_ENABLED=1 on most platforms, which means your binary may dynamically link to system C libraries (like libc).

Terminal
# Build a fully static binary with CGO disabled
CGO_ENABLED=0 go build -o myapp

# Check if your binary is truly static
file myapp
# Output: myapp: ELF 64-bit LSB executable, statically linked

ldd myapp
# Output: not a dynamic executable

3. Go Modules & Packages

Go's approach to dependency management has evolved dramatically. Understanding both the old way (GOPATH) and the modern way (Go Modules) helps you work with any Go codebase.

GOPATH: The Old Way

Before Go 1.11, all Go code lived in a single workspace defined by the GOPATH environment variable (defaulting to ~/go). The directory structure was rigid:

Terminal
$GOPATH/
  src/
    github.com/
      yourusername/
        yourproject/
          main.go
      somelib/
        otherproject/
          lib.go
  bin/     # compiled binaries
  pkg/     # compiled package objects

This was painful: every project shared a single dependency tree, you couldn't have two versions of the same library, and your code had to live inside GOPATH.

Go Modules: The Modern Way

Go Modules (introduced in Go 1.11, default since 1.16) give you proper versioned dependency management. Start any project anywhere on disk:

Terminal
# Initialize a new module
mkdir myproject && cd myproject
go mod init github.com/yourusername/myproject

# This creates a go.mod file

Anatomy of go.mod

go.mod
// The module path -- this is your module's unique identifier
// It's usually the repository URL where the code lives
module github.com/yourusername/myproject

// The minimum Go version required
go 1.22

// Direct dependencies with their versions
require (
    github.com/gin-gonic/gin v1.9.1
    github.com/lib/pq v1.10.9
    go.uber.org/zap v1.27.0
)

// Indirect dependencies (deps of your deps)
require (
    github.com/bytedance/sonic v1.11.3 // indirect
    golang.org/x/net v0.22.0 // indirect
)

// Replace a module with a local copy (useful during development)
replace github.com/somelib/broken => ../my-local-fix

// Exclude a specific version (rare, used to skip broken releases)
exclude github.com/old/lib v1.2.3

The go.sum File

The go.sum file contains cryptographic hashes (SHA-256) of every dependency:

go.sum
github.com/gin-gonic/gin v1.9.1 h1:4+fr/el88TOO3ewCmQr8cx/CtZ/umlIRIs5M4NTNjf8=
github.com/gin-gonic/gin v1.9.1/go.mod h1:hPrL/0KcuKQ32bYtDICo8+3Ynhqfmh4sDAM4TDwCP2Q=

This serves two purposes: integrity verification (ensuring you download the exact same code every time) and supply chain security (detecting tampered dependencies). Always commit go.sum to version control.

Essential Module Commands

The Module Proxy

When you run go get, Go doesn't clone Git repositories directly. Instead, it fetches modules through a module proxy (by default, proxy.golang.org). The proxy:

• Caches modules permanently: Even if a GitHub repo is deleted, the proxy still has the code
• Serves checksums: sum.golang.org provides a global, append-only checksum database to verify module integrity
• Improves speed: CDN-backed, much faster than cloning from Git
• Ensures immutability: Once a version is published, it cannot be changed or deleted from the proxy

Terminal
# Use a different proxy (e.g., for private modules)
GOPROXY=https://goproxy.cn,direct go get github.com/some/package

# Bypass the proxy for private repos
GONOSUMCHECK=github.com/mycompany/* go get github.com/mycompany/internal-lib
GOPRIVATE=github.com/mycompany/* go get github.com/mycompany/internal-lib

Semantic Import Versioning

Go has a unique rule for major versions: if you make a breaking change (v2+), the import path must change:

Go
// Importing v1 -- the import path is just the module path
import "github.com/user/mylib"

// Importing v2 -- the major version is part of the path!
import "github.com/user/mylib/v2"

// Importing v3
import "github.com/user/mylib/v3"

This means v1 and v2 of a library are treated as completely different packages. You can even import both in the same project during a migration. The module's go.mod must also reflect the major version:

go.mod
module github.com/user/mylib/v2

Workspaces for Multi-Module Development

When you're developing multiple modules simultaneously (e.g., a library and a service that uses it), Go workspaces let you use local copies without replace directives:

Terminal
# Create a workspace
mkdir workspace && cd workspace
go work init ./myservice ./mylib

# This creates a go.work file:

go.work
go 1.22

use (
    ./myservice
    ./mylib
)

Now myservice automatically uses the local mylib instead of the published version. Never commit go.work to version control -- it's a local development tool.

Package Naming and Visibility

In Go, visibility is controlled by capitalization, not keywords like public or private:

Go
package user

// Exported (public) -- starts with uppercase
type User struct {
    Name  string  // Exported field
    Email string  // Exported field
    age   int     // unexported field -- only visible within this package
}

// Exported function
func NewUser(name, email string) *User {
    return &User{Name: name, Email: email, age: 0}
}

// unexported function -- only callable within the "user" package
func validateEmail(email string) bool {
    return strings.Contains(email, "@")
}

Internal Packages

Go has a special convention for the internal directory. Code inside internal/ can only be imported by code in the parent directory tree:

Terminal
myproject/
  cmd/
    server/
      main.go         # CAN import myproject/internal/auth
  internal/
    auth/
      auth.go         # Can only be imported by code under myproject/
  pkg/
    api/
      api.go          # CAN import myproject/internal/auth

This is enforced by the compiler, not just convention. External projects that depend on your module cannot import your internal/ packages, giving you a private implementation boundary.

4. Types & Zero Values

Go's type system is one of the most important things to understand. Every type has a well-defined zero value, and there are no uninitialized variables. This eliminates an entire class of bugs common in C and C++.

The Zero Value Rule

Basic Types

Go
// Integers
var i int       // platform-dependent: 32 or 64 bit
var i8 int8     // -128 to 127
var i64 int64   // -9.2 quintillion to 9.2 quintillion

// Unsigned integers
var u uint       // platform-dependent
var b byte       // alias for uint8 -- used for raw binary data

// Floating point
var f32 float32  // ~7 digits of precision
var f64 float64  // ~15 digits of precision (use this by default)

// String -- immutable sequence of bytes (usually UTF-8)
var s string     // ""

// Rune -- alias for int32, represents a Unicode code point
var r rune       // 'A' is a rune, not a byte

// Boolean
var ok bool      // false

Composite Types

Arrays (Fixed Size, Value Type)

Go
// Arrays have a FIXED size that's part of the type
var a [5]int              // [0, 0, 0, 0, 0]
b := [3]string{"go", "is", "fun"}
c := [...]int{1, 2, 3}     // compiler counts: [3]int

// [5]int and [3]int are DIFFERENT TYPES -- you can't assign one to the other
// Arrays are VALUE types -- assigning copies the entire array
x := [3]int{1, 2, 3}
y := x           // y is a COPY -- modifying y doesn't affect x
y[0] = 99
fmt.Println(x[0]) // still 1

Arrays are rarely used directly in Go. Instead, you use slices.

Slices (Dynamic, Reference to Underlying Array)

Go
// Create a slice (no size in brackets = slice, not array)
s := []int{1, 2, 3}

// Create with make: make([]T, length, capacity)
s2 := make([]int, 5)        // [0,0,0,0,0], len=5, cap=5
s3 := make([]int, 0, 100)   // [], len=0, cap=100 (pre-allocated)

// Append (may allocate a new underlying array if capacity is exceeded)
s = append(s, 4, 5, 6)

// Slicing creates a new slice header pointing to the SAME array
sub := s[1:3]  // [2, 3] -- shares memory with s!

Slice Internals

A slice is a three-word struct (24 bytes on 64-bit) called a "slice header":

Slice Header (24 bytes)
+----------+----------+----------+
| Pointer  |  Length   | Capacity |
| (to data)|  (used)  | (alloc)  |
+----------+----------+----------+
     |
     v
+---+---+---+---+---+---+---+---+
| 1 | 2 | 3 | 4 | 5 |   |   |   |   <-- underlying array
+---+---+---+---+---+---+---+---+
  ^                   ^               ^
  |                   |               |
  pointer          length=5       capacity=8

When you call append() and the slice's capacity is exhausted, Go allocates a new, larger underlying array (roughly 2x the old capacity for small slices), copies the data, and returns a new slice header pointing to the new array. This is why you must always use s = append(s, ...) -- the returned slice might have a different pointer.

Go
a := []int{1, 2, 3, 4, 5}
b := a[0:2]        // b = [1, 2], but cap(b) = 5!
b = append(b, 99)   // b = [1, 2, 99]
fmt.Println(a)      // [1, 2, 99, 4, 5] -- a[2] was overwritten!

// Fix: use a full slice expression to limit capacity
b = a[0:2:2]        // b = [1, 2], cap(b) = 2 -- append will allocate new array

Maps

Go
// Create a map
m := make(map[string]int)
m["alice"] = 30
m["bob"] = 25

// Map literal
scores := map[string]int{
    "alice": 95,
    "bob":   87,
}

// Check if a key exists (the "comma ok" idiom)
val, ok := m["charlie"]
if !ok {
    fmt.Println("charlie not found")
}

// Delete a key
delete(m, "bob")

// Iterate (order is RANDOMIZED -- by design!)
for key, value := range m {
    fmt.Printf("%s: %d\n", key, value)
}

var m map[string]int    // m is nil
_ = m["key"]            // returns 0 (safe)
m["key"] = 1            // PANIC: assignment to entry in nil map

Structs

Go
// Define a struct type
type Server struct {
    Host     string
    Port     int
    TLS      bool
    MaxConns int
}

// Create instances
s1 := Server{"localhost", 8080, false, 100}         // positional (fragile)
s2 := Server{Host: "0.0.0.0", Port: 443, TLS: true}  // named fields (preferred)
s3 := Server{}                                         // zero value: {"", 0, false, 0}

// Access fields
fmt.Println(s2.Host)  // "0.0.0.0"
s2.Port = 8443

make() vs new()

These two built-in functions are often confused:

Go
// make -- required for slices, maps, and channels
s := make([]int, 10)             // slice of 10 zeroed ints
m := make(map[string]int)        // empty, ready-to-use map
ch := make(chan int, 5)           // buffered channel

// new -- allocates zeroed memory and returns a pointer
p := new(int)                   // p is *int, *p is 0
u := new(User)                  // u is *User, all fields zeroed

// In practice, you rarely use new(). Instead:
u2 := &User{Name: "Sean"}       // more common: composite literal with &

Struct Tags

Struct tags are string metadata attached to struct fields. They're used heavily by the standard library (especially encoding/json) and third-party packages:

Go
type User struct {
    ID        int    `json:"id" db:"user_id"`
    FirstName string `json:"first_name" validate:"required"`
    Email     string `json:"email,omitempty"`
    Password  string `json:"-"`  // "-" means: never include in JSON
}

user := User{ID: 1, FirstName: "Sean", Email: "sean@dev.com"}
data, _ := json.Marshal(user)
fmt.Println(string(data))
// {"id":1,"first_name":"Sean","email":"sean@dev.com"}
// Note: Password is excluded, Email would be omitted if empty

5. Pointers

Go has pointers, but unlike C and C++, there's no pointer arithmetic. You can't increment a pointer to walk through memory. This makes pointers in Go much safer while still giving you the performance benefits of passing references.

The Basics: & and *

Go
x := 42

p := &x        // & = "address of" -- p is a *int pointing to x
fmt.Println(p)  // 0xc000012080 (a memory address)
fmt.Println(*p) // 42 -- * = "dereference" -- the value at that address

*p = 100        // modify the value through the pointer
fmt.Println(x)  // 100 -- x was changed because p points to it

When to Use Pointers

There are three main reasons to use pointers in Go:

1. When you need to mutate a value in a function:

Go
// Without pointer -- the function gets a COPY
func doubleValue(x int) {
    x = x * 2  // modifies the copy, not the original
}

// With pointer -- the function can modify the original
func doublePointer(x *int) {
    *x = *x * 2  // modifies the value at the address
}

n := 5
doubleValue(n)
fmt.Println(n)     // still 5

doublePointer(&n)
fmt.Println(n)     // 10

2. When passing large structs (avoid copying):

Go
type BigConfig struct {
    // Imagine 50 fields, each a string or slice
    DatabaseURL    string
    RedisURL       string
    Features       []string
    // ... many more fields
}

// Bad: copies the entire struct every time
func processConfig(cfg BigConfig) { /* ... */ }

// Good: passes an 8-byte pointer instead of a huge struct
func processConfig(cfg *BigConfig) { /* ... */ }

3. When you need nil-ability (to represent "no value"):

Go
// A plain int can't distinguish "zero" from "not set"
type QueryParams struct {
    Limit  *int  // nil means "not specified", 0 means "limit to 0"
    Offset *int
}

func buildQuery(p QueryParams) string {
    q := "SELECT * FROM users"
    if p.Limit != nil {
        q += fmt.Sprintf(" LIMIT %d", *p.Limit)
    }
    return q
}

Value Receivers vs Pointer Receivers

Methods in Go can have either a value receiver or a pointer receiver. This controls whether the method gets a copy of the value or a reference to it:

Go
type Counter struct {
    count int
}

// Value receiver -- gets a COPY of Counter
func (c Counter) Value() int {
    return c.count
}

// Pointer receiver -- gets a reference to the original Counter
func (c *Counter) Increment() {
    c.count++  // modifies the original
}

c := Counter{}
c.Increment()          // Go automatically takes the address: (&c).Increment()
fmt.Println(c.Value()) // 1

When to use which:

var p *User          // p is nil
fmt.Println(p.Name)  // PANIC: runtime error: invalid memory address

// Always check for nil before dereferencing
if p != nil {
    fmt.Println(p.Name)
}

6. Interfaces & Composition

Interfaces are the single most important abstraction mechanism in Go. Unlike Java or C#, Go interfaces are satisfied implicitly -- there's no implements keyword. If your type has the right methods, it automatically satisfies the interface. This seemingly small design choice has profound consequences.

Implicit Interface Satisfaction

Go
// Define an interface
type Speaker interface {
    Speak() string
}

// Dog satisfies Speaker -- no "implements" declaration needed
type Dog struct {
    Name string
}

func (d Dog) Speak() string {
    return d.Name + " says woof!"
}

// Cat also satisfies Speaker
type Cat struct {
    Name string
}

func (c Cat) Speak() string {
    return c.Name + " says meow!"
}

// A function that accepts any Speaker
func MakeNoise(s Speaker) {
    fmt.Println(s.Speak())
}

MakeNoise(Dog{Name: "Rex"})   // "Rex says woof!"
MakeNoise(Cat{Name: "Mia"})   // "Mia says meow!"

The key insight: Dog and Cat don't know about the Speaker interface. They don't import it, don't declare that they implement it. The consumer defines the interface, not the producer. This inverts the typical dependency direction and makes Go code incredibly composable.

The Empty Interface: interface{} / any

An interface with zero methods is satisfied by every type:

Go
// interface{} accepts any value (renamed to "any" in Go 1.18)
func printAnything(v any) {
    fmt.Println(v)
}

printAnything(42)
printAnything("hello")
printAnything([]int{1, 2, 3})

Type Assertions and Type Switches

When you have an interface value, you can recover the concrete type:

Go
var s Speaker = Dog{Name: "Rex"}

// Type assertion -- "I believe this Speaker is a Dog"
d, ok := s.(Dog)
if ok {
    fmt.Println("It's a dog named", d.Name)
}

// Without the ok check -- PANICS if wrong type
d2 := s.(Dog)  // works here, but dangerous

// Type switch -- handle multiple types cleanly
func describe(s Speaker) string {
    switch v := s.(type) {
    case Dog:
        return fmt.Sprintf("Dog: %s", v.Name)
    case Cat:
        return fmt.Sprintf("Cat: %s", v.Name)
    default:
        return "Unknown animal"
    }
}

Embedding for Composition

Go doesn't have inheritance. Instead, it uses embedding -- you can include one type inside another, and the outer type "inherits" all of its methods:

Go
type Animal struct {
    Name string
    Age  int
}

func (a Animal) String() string {
    return fmt.Sprintf("%s (age %d)", a.Name, a.Age)
}

// Dog embeds Animal -- gets all its fields and methods
type Dog struct {
    Animal          // embedded (no field name = embedding)
    Breed  string
}

d := Dog{
    Animal: Animal{Name: "Rex", Age: 5},
    Breed:  "Labrador",
}

// Access embedded fields directly
fmt.Println(d.Name)     // "Rex" -- promoted from Animal
fmt.Println(d.String()) // "Rex (age 5)" -- promoted method
fmt.Println(d.Breed)    // "Labrador" -- Dog's own field

This is composition, not inheritance. There is no "is-a" relationship. A Dog is not an Animal -- it has an Animal. You can also embed interfaces to require method sets:

Go
// Embedding interfaces -- ReadWriter requires both Read and Write
type ReadWriter interface {
    io.Reader
    io.Writer
}

io.Reader and io.Writer -- The Most Important Interfaces

These two interfaces are the foundation of Go's I/O model. Almost everything in Go that produces or consumes data implements one or both:

Go
// Defined in the "io" package
type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

Just one method each. Yet these tiny interfaces connect the entire Go ecosystem:

• os.File -- reads/writes files
• net.Conn -- reads/writes network connections
• http.Response.Body -- reads HTTP response bodies
• bytes.Buffer -- reads/writes in-memory buffers
• compress/gzip.Reader -- decompresses gzip data
• crypto/tls.Conn -- encrypted network connection
• json.Encoder / json.Decoder -- JSON serialization

Go
// Read from network -> decrypt TLS -> decompress gzip -> decode JSON
conn, _ := tls.Dial("tcp", "api.example.com:443", &tls.Config{})
gzReader, _ := gzip.NewReader(conn)       // wraps conn (an io.Reader)
var data MyStruct
json.NewDecoder(gzReader).Decode(&data)   // wraps gzReader (an io.Reader)

Why Small Interfaces Are Better

The Go proverb is: "The bigger the interface, the weaker the abstraction." Here's why:

• Small interfaces are easier to implement: Any type can satisfy io.Reader (1 method). Few types can satisfy a 10-method interface.
• Small interfaces are easier to mock in tests: Implementing a 1-method interface for a test double takes one line.
• Small interfaces compose: You can combine io.Reader + io.Writer into io.ReadWriter. You can't easily decompose a 20-method interface.
• Small interfaces decouple: Your function that takes an io.Reader works with files, HTTP responses, strings, buffers, network connections -- anything. A function that takes *os.File only works with files.

Go
// BAD: defining an interface in the package that implements it
// package database
type UserStore interface {
    GetUser(id int) (*User, error)
    CreateUser(u *User) error
    UpdateUser(u *User) error
    DeleteUser(id int) error
    ListUsers() ([]*User, error)
}

// GOOD: define the interface where it's used, with only what you need
// package handler
type UserGetter interface {
    GetUser(id int) (*User, error)
}

func HandleGetUser(store UserGetter, id int) (*User, error) {
    return store.GetUser(id)
}
// Now any type with a GetUser method works -- easy to test, easy to swap

7. Goroutines & Channels

Concurrency is Go's superpower. While other languages bolt concurrency on as a library feature, Go bakes it into the language itself with two primitives: goroutines (lightweight threads) and channels (typed communication pipes). Together, they implement Tony Hoare's CSP (Communicating Sequential Processes) model: "Don't communicate by sharing memory; share memory by communicating."

The go Keyword

Launching a goroutine is as simple as putting go before a function call. That's it. No thread pools, no executor services, no async/await. Just go.

Go
package main

import (
    "fmt"
    "time"
)

func sayHello(name string) {
    for i := 0; i < 3; i++ {
        fmt.Printf("Hello from %s (iteration %d)\n", name, i)
        time.Sleep(100 * time.Millisecond)
    }
}

func main() {
    // Launch goroutines -- each runs concurrently
    go sayHello("goroutine-1")
    go sayHello("goroutine-2")
    go sayHello("goroutine-3")

    // main() is itself a goroutine. If it exits, all goroutines die.
    time.Sleep(500 * time.Millisecond)
    fmt.Println("main done")
}

How Goroutines Actually Work: The Go Scheduler

A goroutine is not an OS thread. It's a user-space thread managed by the Go runtime. The Go scheduler uses an M:N scheduling model: it multiplexes M goroutines onto N OS threads (where N defaults to the number of CPU cores, controlled by GOMAXPROCS).

The scheduler has three key concepts, known as the GMP model:

• G (Goroutine): The goroutine itself. Contains the stack, instruction pointer, and other state needed for scheduling.
• M (Machine): An OS thread. The thing that actually runs code on a CPU core.
• P (Processor): A logical processor. Each P has a local run queue of goroutines. The number of Ps equals GOMAXPROCS (default: number of CPU cores).

Scheduler Architecture
                    Go Scheduler (GMP Model)

  Goroutines (G)       Processors (P)        OS Threads (M)      CPU Cores
  +-----------+
  |   G   G   | -----> +------+
  |   G   G   |        |  P0  | ------------> M0 ------------> Core 0
  +-----------+        | LRQ  |
                       +------+
  +-----------+
  |   G   G   | -----> +------+
  |   G   G   |        |  P1  | ------------> M1 ------------> Core 1
  +-----------+        | LRQ  |
                       +------+
  +-----------+
  |   G   G   |        +------+
  |   G   G   | -----> |  P2  | ------------> M2 ------------> Core 2
  +-----------+        | LRQ  |
                       +------+

  +------------------+
  | Global Run Queue | <-- overflow goroutines go here
  | G  G  G  G  G   |     (Ps steal from here when local queue is empty)
  +------------------+

  LRQ = Local Run Queue (up to 256 goroutines per P)

  Work Stealing: When P1's LRQ is empty, it steals half of P0's LRQ.
  This keeps all cores busy even with uneven work distribution.

When Goroutines Yield

The Go scheduler is cooperatively preemptive (since Go 1.14). Goroutines yield at:

• Function calls: The compiler inserts preemption checks at function prologues.
• Channel operations: Sending/receiving on a channel may block the goroutine.
• System calls: When a goroutine makes a system call, the M (OS thread) blocks, and the P is handed off to another M to keep other goroutines running.
• Garbage collection: GC safe points cause goroutines to pause.
• Asynchronous preemption (Go 1.14+): The runtime can send a signal to preempt goroutines stuck in tight loops without function calls.

Channels: Communication Between Goroutines

Channels are typed conduits through which goroutines send and receive values. They are the primary mechanism for goroutine synchronization and communication.

Go
// Create channels
ch := make(chan int)       // unbuffered channel of ints
bch := make(chan string, 5) // buffered channel with capacity 5

// Send a value into a channel
ch <- 42

// Receive a value from a channel
value := <-ch

// Receive and check if channel is closed
value, ok := <-ch  // ok is false if channel is closed and empty

// Close a channel (only sender should close)
close(ch)

Unbuffered vs Buffered Channels

The distinction between unbuffered and buffered channels is critical for understanding Go concurrency.

Channel Behavior
  Unbuffered Channel (make(chan int))
  ====================================
  Sender blocks until a receiver is ready. This creates a synchronization
  point -- a "handshake" between goroutines.

  Goroutine A:  ch <- 42   ... BLOCKS until B is ready to receive
  Goroutine B:  v := <-ch  ... BLOCKS until A sends

  They meet at the channel and exchange the value simultaneously.


  Buffered Channel (make(chan int, 3))
  ====================================
  Sender only blocks when the buffer is FULL.
  Receiver only blocks when the buffer is EMPTY.

  Capacity: 3
  +---+---+---+
  |   |   |   |   <-- send doesn't block (buffer has space)
  +---+---+---+

  +---+---+---+
  | 1 | 2 | 3 |   <-- send BLOCKS (buffer full, must wait for receive)
  +---+---+---+

  Use cases:
  - Unbuffered: when you need synchronization (guaranteed handoff)
  - Buffered:   when you need to decouple sender/receiver timing

Go
package main

import "fmt"

func main() {
    // Unbuffered: this goroutine pattern is required,
    // because send blocks until someone receives
    ch := make(chan string)

    go func() {
        ch <- "hello"  // blocks until main receives
    }()

    msg := <-ch  // blocks until goroutine sends
    fmt.Println(msg)

    // Buffered: can send without an immediate receiver
    bch := make(chan int, 3)
    bch <- 1  // doesn't block (buffer has space)
    bch <- 2  // doesn't block
    bch <- 3  // doesn't block
    // bch <- 4  // THIS WOULD DEADLOCK -- buffer full, no receiver

    fmt.Println(<-bch)  // 1
    fmt.Println(<-bch)  // 2
    fmt.Println(<-bch)  // 3
}

Channel Direction Types

You can restrict a channel to send-only or receive-only in function signatures. This provides compile-time safety -- the compiler ensures you don't accidentally send on a receive-only channel.

Go
// producer can only SEND to the channel
func producer(out chan<- int) {
    for i := 0; i < 5; i++ {
        out <- i
    }
    close(out)
}

// consumer can only RECEIVE from the channel
func consumer(in <-chan int) {
    for v := range in {
        fmt.Println("received:", v)
    }
}

func main() {
    ch := make(chan int)  // bidirectional
    go producer(ch)         // auto-converts to chan<- int
    consumer(ch)             // auto-converts to <-chan int
}

The select Statement

The select statement lets a goroutine wait on multiple channel operations simultaneously. It's like a switch statement for channels.

Go
func main() {
    ch1 := make(chan string)
    ch2 := make(chan string)

    go func() {
        time.Sleep(100 * time.Millisecond)
        ch1 <- "one"
    }()

    go func() {
        time.Sleep(200 * time.Millisecond)
        ch2 <- "two"
    }()

    // Wait for whichever channel is ready first
    for i := 0; i < 2; i++ {
        select {
        case msg := <-ch1:
            fmt.Println("from ch1:", msg)
        case msg := <-ch2:
            fmt.Println("from ch2:", msg)
        }
    }
}

// Non-blocking select with default
select {
case msg := <-ch:
    fmt.Println("received:", msg)
default:
    fmt.Println("no message ready, moving on")
}

// Timeout pattern
select {
case result := <-longOperation:
    fmt.Println("got result:", result)
case <-time.After(3 * time.Second):
    fmt.Println("operation timed out")
}

Range Over Channels

You can use range to receive values from a channel until it's closed. This is the idiomatic way to consume all values from a channel.

Go
func generateNumbers(n int) <-chan int {
    ch := make(chan int)
    go func() {
        for i := 0; i < n; i++ {
            ch <- i * i
        }
        close(ch)  // MUST close, or range will block forever
    }()
    return ch
}

func main() {
    for num := range generateNumbers(5) {
        fmt.Println(num)  // 0, 1, 4, 9, 16
    }
}

Done Channels and Signaling

A common pattern is using channels to signal completion or cancellation. An empty struct channel (chan struct{}) is idiomatic because it uses zero memory.

Go
func worker(done chan struct{}) {
    fmt.Println("working...")
    time.Sleep(2 * time.Second)
    fmt.Println("done!")
    close(done)  // signal completion by closing the channel
}

func main() {
    done := make(chan struct{})
    go worker(done)

    <-done  // blocks until done is closed
    fmt.Println("worker finished")
}

context.Context: Cancellation and Timeouts

The context package provides a standard way to carry deadlines, cancellation signals, and request-scoped values across API boundaries and between goroutines. Every long-running or I/O operation should accept a context.Context as its first parameter.

Go
package main

import (
    "context"
    "fmt"
    "time"
)

func slowOperation(ctx context.Context) error {
    select {
    case <-time.After(5 * time.Second):
        fmt.Println("operation completed")
        return nil
    case <-ctx.Done():
        // Context was cancelled or timed out
        fmt.Println("operation cancelled:", ctx.Err())
        return ctx.Err()
    }
}

func main() {
    // Create a context that cancels after 2 seconds
    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
    defer cancel()  // always call cancel to release resources

    err := slowOperation(ctx)
    if err != nil {
        fmt.Println("error:", err)
    }
}

Go
var wg sync.WaitGroup

for i := 0; i < 5; i++ {
    wg.Add(1)  // increment counter BEFORE launching goroutine
    go func(id int) {
        defer wg.Done()  // decrement counter when goroutine finishes
        fmt.Printf("worker %d done\n", id)
    }(i)
}

wg.Wait()  // blocks until counter reaches 0
fmt.Println("all workers finished")

8. Error Handling

Go takes a radically different approach to error handling than most languages: errors are values, not exceptions. There is no try/catch. Every function that can fail returns an error as its last return value, and the caller is expected to check it immediately. This is by design -- it forces you to think about failure at every step.

The error Interface

The built-in error type is just an interface with a single method:

Go
// Built into the language -- you don't need to import anything
type error interface {
    Error() string
}

Any type that has an Error() string method satisfies the error interface. The simplest way to create an error:

Go
import "errors"

err := errors.New("something went wrong")

// Or with formatting
import "fmt"
err := fmt.Errorf("failed to load user %d: %s", userID, reason)

The Error Handling Pattern

This is the most common pattern in Go. You'll see it hundreds of times in any Go codebase:

Go
func readConfig(path string) (*Config, error) {
    data, err := os.ReadFile(path)
    if err != nil {
        return nil, fmt.Errorf("reading config: %w", err)
    }

    var cfg Config
    err = json.Unmarshal(data, &cfg)
    if err != nil {
        return nil, fmt.Errorf("parsing config: %w", err)
    }

    return &cfg, nil
}

// Caller always checks the error
cfg, err := readConfig("/etc/myapp/config.json")
if err != nil {
    log.Fatalf("startup failed: %v", err)
}

Error Wrapping with %w

Since Go 1.13, you can wrap errors to add context while preserving the original error. Use %w in fmt.Errorf:

Go
func getUser(id int) (*User, error) {
    row := db.QueryRow("SELECT * FROM users WHERE id = ?", id)
    var u User
    err := row.Scan(&u.Name, &u.Email)
    if err != nil {
        // Wrap the error with context -- %w preserves the original
        return nil, fmt.Errorf("getUser(%d): %w", id, err)
    }
    return &u, nil
}

// The wrapped error chain might look like:
// "getUser(42): sql: no rows in result set"

Checking Errors: errors.Is() and errors.As()

When errors are wrapped, you can't use == to compare them. Instead, use errors.Is() and errors.As() to unwrap the chain:

Go
import (
    "database/sql"
    "errors"
)

// errors.Is checks if ANY error in the chain matches
if errors.Is(err, sql.ErrNoRows) {
    // handle "not found" -- works even if err is wrapped
    return nil, ErrUserNotFound
}

// errors.As checks if ANY error in the chain is a specific type
var pathErr *os.PathError
if errors.As(err, &pathErr) {
    // pathErr is now set to the *os.PathError in the chain
    fmt.Println("failed path:", pathErr.Path)
}

Sentinel Errors

Sentinel errors are package-level variables that represent specific error conditions. By convention, they start with Err:

Go
// Define sentinel errors in your package
var (
    ErrNotFound     = errors.New("not found")
    ErrUnauthorized = errors.New("unauthorized")
    ErrConflict     = errors.New("conflict")
)

// Use them in functions
func FindUser(id int) (*User, error) {
    // ...
    if user == nil {
        return nil, fmt.Errorf("user %d: %w", id, ErrNotFound)
    }
    return user, nil
}

// Callers check with errors.Is
user, err := FindUser(42)
if errors.Is(err, ErrNotFound) {
    http.Error(w, "user not found", http.StatusNotFound)
    return
}

Custom Error Types

When you need errors to carry structured data (not just a string), create a custom error type:

Go
type ValidationError struct {
    Field   string
    Message string
    Value   interface{}
}

func (e *ValidationError) Error() string {
    return fmt.Sprintf("validation failed on %s: %s (got %v)",
        e.Field, e.Message, e.Value)
}

// Usage
func validateAge(age int) error {
    if age < 0 || age > 150 {
        return &ValidationError{
            Field:   "age",
            Message: "must be between 0 and 150",
            Value:   age,
        }
    }
    return nil
}

// Check with errors.As
var ve *ValidationError
if errors.As(err, &ve) {
    fmt.Printf("field %s is invalid: %s\n", ve.Field, ve.Message)
}

Panic and Recover

panic is for truly unrecoverable situations -- programmer bugs, not runtime errors. Think of it like an assertion that crashes the program.

Go
// Panic -- use ONLY for unrecoverable situations
func MustParseURL(rawURL string) *url.URL {
    u, err := url.Parse(rawURL)
    if err != nil {
        panic(fmt.Sprintf("invalid URL %q: %v", rawURL, err))
    }
    return u
}

// Recover -- catch a panic (use in deferred functions)
func safeHandler(w http.ResponseWriter, r *http.Request) {
    defer func() {
        if r := recover(); r != nil {
            log.Printf("panic recovered: %v", r)
            http.Error(w, "internal server error", 500)
        }
    }()

    // handler code that might panic...
    handleRequest(w, r)
}

9. The Standard Library

Go's standard library is one of its biggest strengths. Unlike many languages where you need third-party packages for basic tasks, Go ships with production-quality implementations of HTTP servers, JSON parsing, cryptography, testing, and more. Knowing the standard library well is what separates beginner Go developers from productive ones.

net/http: HTTP Server and Client

Go's net/http package is good enough to run in production. Many Go services at Google, Cloudflare, and other companies use it directly without any third-party framework.

Go
package main

import (
    "encoding/json"
    "log"
    "net/http"
)

// Handler function
func helloHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/plain")
    w.Write([]byte("Hello, World!"))
}

// JSON API handler
func usersHandler(w http.ResponseWriter, r *http.Request) {
    users := []struct {
        Name  string `json:"name"`
        Email string `json:"email"`
    }{
        {"Alice", "alice@example.com"},
        {"Bob", "bob@example.com"},
    }

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(users)
}

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("GET /hello", helloHandler)
    mux.HandleFunc("GET /api/users", usersHandler)

    log.Println("server starting on :8080")
    log.Fatal(http.ListenAndServe(":8080", mux))
}

Middleware Pattern

Middleware in Go is just a function that wraps an http.Handler. No magic, no framework -- just function composition:

Go
// Middleware is a function that takes a handler and returns a handler
func loggingMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        log.Printf("%s %s", r.Method, r.URL.Path)
        next.ServeHTTP(w, r)  // call the next handler
    })
}

func authMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        token := r.Header.Get("Authorization")
        if token == "" {
            http.Error(w, "unauthorized", http.StatusUnauthorized)
            return
        }
        next.ServeHTTP(w, r)
    })
}

// Chain middleware: logging -> auth -> handler
handler := loggingMiddleware(authMiddleware(mux))
http.ListenAndServe(":8080", handler)

HTTP Client

Go
// Simple GET request
resp, err := http.Get("https://api.example.com/data")
if err != nil {
    log.Fatal(err)
}
defer resp.Body.Close()

body, err := io.ReadAll(resp.Body)

// Custom client with timeout (ALWAYS set a timeout!)
client := &http.Client{
    Timeout: 10 * time.Second,
}
resp, err := client.Get("https://api.example.com/data")

// POST JSON
data := map[string]string{"name": "Alice"}
jsonBody, _ := json.Marshal(data)
resp, err := http.Post(
    "https://api.example.com/users",
    "application/json",
    bytes.NewReader(jsonBody),
)

encoding/json: JSON Serialization

Go
type User struct {
    Name     string    `json:"name"`
    Email    string    `json:"email"`
    Age      int       `json:"age,omitempty"`      // omit if zero value
    Password string    `json:"-"`                 // never include in JSON
    JoinedAt time.Time `json:"joined_at"`
}

// Struct to JSON (Marshal)
user := User{Name: "Alice", Email: "alice@example.com"}
jsonBytes, err := json.Marshal(user)
// {"name":"Alice","email":"alice@example.com","joined_at":"0001-01-01T00:00:00Z"}

// Pretty print
jsonBytes, err := json.MarshalIndent(user, "", "  ")

// JSON to struct (Unmarshal)
var u User
err := json.Unmarshal([]byte(jsonStr), &u)

// Streaming decode from an io.Reader (e.g., HTTP response body)
var u User
err := json.NewDecoder(resp.Body).Decode(&u)

// Streaming encode to an io.Writer (e.g., HTTP response writer)
err := json.NewEncoder(w).Encode(user)

Go
type Status int

const (
    StatusActive   Status = 1
    StatusInactive Status = 2
)

func (s Status) MarshalJSON() ([]byte, error) {
    var str string
    switch s {
    case StatusActive:
        str = "active"
    case StatusInactive:
        str = "inactive"
    default:
        str = "unknown"
    }
    return json.Marshal(str)
}

io: The Reader/Writer Foundation

Go
// io.Copy: stream bytes from reader to writer (no buffering in memory)
n, err := io.Copy(dst, src)  // copies until EOF

// io.ReadAll: read everything into memory (careful with large inputs!)
data, err := io.ReadAll(reader)

// io.Pipe: synchronous in-memory pipe (connects a writer to a reader)
pr, pw := io.Pipe()
go func() {
    fmt.Fprint(pw, "hello from writer")
    pw.Close()
}()
data, _ := io.ReadAll(pr)  // "hello from writer"

// io.TeeReader: reader that writes to a writer as it's read
tee := io.TeeReader(resp.Body, os.Stdout)  // prints body while reading
data, _ := io.ReadAll(tee)

// io.MultiWriter: writes to multiple writers at once
multi := io.MultiWriter(file, os.Stdout)  // write to both file and stdout
fmt.Fprint(multi, "logged to both destinations")

os: Files, Environment, and Signals

Go
// Read a file
data, err := os.ReadFile("config.json")

// Write a file (creates or truncates)
err := os.WriteFile("output.txt", []byte("hello"), 0644)

// Open for more control
f, err := os.Open("data.txt")       // read-only
f, err := os.Create("new.txt")      // create/truncate, write-only
f, err := os.OpenFile("log.txt", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
defer f.Close()

// Environment variables
home := os.Getenv("HOME")
val, ok := os.LookupEnv("API_KEY")  // ok is false if not set

// Command-line arguments
args := os.Args  // os.Args[0] is the program name

fmt: Printf Verbs

Go
user := User{Name: "Alice", Age: 30}

fmt.Printf("%v\n",  user)  // {Alice 30}           -- default format
fmt.Printf("%+v\n", user)  // {Name:Alice Age:30}  -- with field names
fmt.Printf("%#v\n", user)  // main.User{Name:"Alice", Age:30}  -- Go syntax
fmt.Printf("%T\n",  user)  // main.User            -- type

fmt.Printf("%s\n", "hello") // hello     -- string
fmt.Printf("%q\n", "hello") // "hello"   -- quoted string
fmt.Printf("%d\n", 42)      // 42        -- integer
fmt.Printf("%b\n", 42)      // 101010    -- binary
fmt.Printf("%x\n", 255)     // ff        -- hex
fmt.Printf("%f\n", 3.14)    // 3.140000  -- float
fmt.Printf("%.2f\n", 3.14)  // 3.14      -- float with precision
fmt.Printf("%p\n", &user)   // 0xc0000b4000  -- pointer address

strings and strconv

Go
import "strings"

strings.Contains("hello world", "world")    // true
strings.HasPrefix("hello", "hel")            // true
strings.HasSuffix("hello", "llo")            // true
strings.Split("a,b,c", ",")                  // ["a", "b", "c"]
strings.Join([]string{"a", "b"}, "-")       // "a-b"
strings.ReplaceAll("aaa", "a", "b")         // "bbb"
strings.TrimSpace("  hello  ")               // "hello"
strings.ToUpper("hello")                     // "HELLO"
strings.ToLower("HELLO")                     // "hello"

import "strconv"

strconv.Itoa(42)               // "42" (int to string)
strconv.Atoi("42")             // 42, nil (string to int)
strconv.FormatFloat(3.14, 'f', 2, 64)  // "3.14"
strconv.ParseBool("true")      // true, nil

sync: Concurrency Primitives

Go
import "sync"

// Mutex: mutual exclusion lock
var mu sync.Mutex
var count int

mu.Lock()
count++
mu.Unlock()

// RWMutex: multiple readers OR one writer
var rwmu sync.RWMutex
rwmu.RLock()    // multiple goroutines can read simultaneously
// read data...
rwmu.RUnlock()

rwmu.Lock()     // exclusive write access
// write data...
rwmu.Unlock()

// sync.Once: run something exactly once (thread-safe)
var once sync.Once
once.Do(func() {
    // this runs exactly once, even if called from many goroutines
    db = connectToDatabase()
})

// sync.Pool: reuse temporary objects to reduce GC pressure
var bufPool = sync.Pool{
    New: func() interface{} {
        return new(bytes.Buffer)
    },
}

buf := bufPool.Get().(*bytes.Buffer)
buf.Reset()
// use buf...
bufPool.Put(buf)  // return to pool for reuse

time: Durations, Tickers, and Timers

Go
import "time"

// Durations
d := 5 * time.Second
d2 := 100 * time.Millisecond

// Current time
now := time.Now()
formatted := now.Format("2006-01-02 15:04:05")  // Go's reference time!

// Parse a time string
t, err := time.Parse("2006-01-02", "2024-03-15")

// Timer: fires once after a duration
timer := time.NewTimer(2 * time.Second)
<-timer.C  // blocks for 2 seconds

// Ticker: fires repeatedly at an interval
ticker := time.NewTicker(500 * time.Millisecond)
defer ticker.Stop()
for t := range ticker.C {
    fmt.Println("tick at", t)
}

// time.After: one-shot timer channel (handy in select)
select {
case result := <-ch:
    fmt.Println(result)
case <-time.After(3 * time.Second):
    fmt.Println("timeout")
}

testing: Writing Tests

Go
// math_test.go -- test files end in _test.go
package math

import "testing"

// Test functions start with Test and take *testing.T
func TestAdd(t *testing.T) {
    got := Add(2, 3)
    want := 5
    if got != want {
        t.Errorf("Add(2, 3) = %d, want %d", got, want)
    }
}

// Table-driven tests -- THE idiomatic Go testing pattern
func TestAdd_TableDriven(t *testing.T) {
    tests := []struct {
        name     string
        a, b     int
        expected int
    }{
        {"positive", 2, 3, 5},
        {"negative", -1, -2, -3},
        {"zero", 0, 0, 0},
        {"mixed", -1, 5, 4},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got := Add(tt.a, tt.b)
            if got != tt.expected {
                t.Errorf("Add(%d, %d) = %d, want %d",
                    tt.a, tt.b, got, tt.expected)
            }
        })
    }
}

// Benchmarks start with Benchmark and take *testing.B
func BenchmarkAdd(b *testing.B) {
    for i := 0; i < b.N; i++ {
        Add(2, 3)
    }
}
// Run: go test -bench=. -benchmem

10. Concurrency Patterns

Now that you understand goroutines and channels, let's look at the battle-tested patterns that Go developers use to build concurrent systems. These patterns are the building blocks of every serious Go application.

Worker Pool

The worker pool is the most common concurrency pattern in Go. You spin up N goroutines that all read from a shared jobs channel. Work is distributed automatically -- whichever worker is free picks up the next job.

Go
package main

import (
    "fmt"
    "sync"
    "time"
)

type Job struct {
    ID   int
    Data string
}

type Result struct {
    JobID    int
    Output   string
    WorkerID int
}

func worker(id int, jobs <-chan Job, results chan<- Result, wg *sync.WaitGroup) {
    defer wg.Done()
    for job := range jobs {
        // Simulate work
        time.Sleep(100 * time.Millisecond)
        results <- Result{
            JobID:    job.ID,
            Output:   fmt.Sprintf("processed %q", job.Data),
            WorkerID: id,
        }
    }
}

func main() {
    const numWorkers = 3
    const numJobs = 10

    jobs := make(chan Job, numJobs)
    results := make(chan Result, numJobs)

    // Start workers
    var wg sync.WaitGroup
    for i := 1; i <= numWorkers; i++ {
        wg.Add(1)
        go worker(i, jobs, results, &wg)
    }

    // Send jobs
    for i := 1; i <= numJobs; i++ {
        jobs <- Job{ID: i, Data: fmt.Sprintf("task-%d", i)}
    }
    close(jobs)  // no more jobs -- workers will exit range loop

    // Wait for all workers then close results
    go func() {
        wg.Wait()
        close(results)
    }()

    // Collect results
    for r := range results {
        fmt.Printf("Worker %d finished job %d: %s\n",
            r.WorkerID, r.JobID, r.Output)
    }
}

Worker Pool Architecture

  Producer                    Workers                     Consumer
  +--------+                                              +--------+
  |        |    +------+     +---------+     +-------+    |        |
  | Submit |--->| Jobs |---->| Worker1 |---->|Results|--->|Collect |
  | Jobs   |    | Chan |---->| Worker2 |---->| Chan  |    |Results |
  |        |    |      |---->| Worker3 |---->|       |    |        |
  +--------+    +------+     +---------+     +-------+    +--------+
                 (buffered)   range jobs      (buffered)   range results
                              until closed                 until closed

Pipeline Pattern

A pipeline is a series of stages connected by channels. Each stage is a group of goroutines that receive values from upstream, process them, and send the results downstream.

Go
// Stage 1: Generate numbers
func generate(nums ...int) <-chan int {
    out := make(chan int)
    go func() {
        for _, n := range nums {
            out <- n
        }
        close(out)
    }()
    return out
}

// Stage 2: Square each number
func square(in <-chan int) <-chan int {
    out := make(chan int)
    go func() {
        for n := range in {
            out <- n * n
        }
        close(out)
    }()
    return out
}

// Stage 3: Filter (keep only values above threshold)
func filter(in <-chan int, threshold int) <-chan int {
    out := make(chan int)
    go func() {
        for n := range in {
            if n > threshold {
                out <- n
            }
        }
        close(out)
    }()
    return out
}

func main() {
    // Connect the pipeline: generate -> square -> filter
    nums := generate(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
    squared := square(nums)
    big := filter(squared, 20)

    for v := range big {
        fmt.Println(v)  // 25, 36, 49, 64, 81, 100
    }
}

Fan-Out / Fan-In

Fan-out: Multiple goroutines read from the same channel to distribute work. Fan-in: Merge multiple channels into one.

Go
// Fan-in: merge multiple channels into one
func merge(channels ...<-chan int) <-chan int {
    var wg sync.WaitGroup
    merged := make(chan int)

    for _, ch := range channels {
        wg.Add(1)
        go func(c <-chan int) {
            defer wg.Done()
            for v := range c {
                merged <- v
            }
        }(ch)
    }

    go func() {
        wg.Wait()
        close(merged)
    }()
    return merged
}

// Fan-out: distribute work across multiple squarers
func main() {
    nums := generate(1, 2, 3, 4, 5, 6, 7, 8)

    // Fan-out: 3 squarers reading from the same channel
    sq1 := square(nums)
    sq2 := square(nums)
    sq3 := square(nums)

    // Fan-in: merge all results into one channel
    for v := range merge(sq1, sq2, sq3) {
        fmt.Println(v)
    }
}

Semaphore Pattern

Use a buffered channel as a counting semaphore to limit concurrent operations:

Go
// Limit to 5 concurrent HTTP requests
sem := make(chan struct{}, 5)

var wg sync.WaitGroup
for _, url := range urls {
    wg.Add(1)
    go func(u string) {
        defer wg.Done()
        sem <- struct{}{}  // acquire (blocks if 5 already running)
        defer func() { <-sem }()  // release

        resp, err := http.Get(u)
        if err != nil {
            log.Println(err)
            return
        }
        defer resp.Body.Close()
        // process response...
    }(url)
}
wg.Wait()

Rate Limiting

Go
// Process events at most once per 200ms
limiter := time.NewTicker(200 * time.Millisecond)
defer limiter.Stop()

for event := range events {
    <-limiter.C  // wait for the next tick
    go process(event)
}

// Bursty rate limiter: allow bursts up to 3, then rate limit
bursty := make(chan time.Time, 3)

// Pre-fill for initial burst
for i := 0; i < 3; i++ {
    bursty <- time.Now()
}

// Refill at steady rate
go func() {
    for t := range time.Tick(200 * time.Millisecond) {
        bursty <- t
    }
}()

for event := range events {
    <-bursty
    go process(event)
}

Graceful Shutdown

A production Go service needs to handle shutdown cleanly: stop accepting new work, finish in-progress work, and release resources. Here's the complete pattern:

Go
package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    srv := &http.Server{Addr: ":8080"}

    // Start server in a goroutine
    go func() {
        log.Println("server starting on :8080")
        if err := srv.ListenAndServe(); err != http.ErrServerClosed {
            log.Fatalf("server error: %v", err)
        }
    }()

    // Wait for interrupt signal (Ctrl+C or kill)
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit
    log.Println("shutting down...")

    // Give outstanding requests 30 seconds to complete
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        log.Fatalf("forced shutdown: %v", err)
    }
    log.Println("server stopped gracefully")
}

11. Writing Good Go Code

Go has a strong culture of consistency. The community has well-established conventions, and following them isn't optional if you want to write code that other Go developers can read and maintain. This section covers the conventions, tools, and principles that define idiomatic Go.

go fmt: Canonical Formatting

gofmt automatically formats your code. In Go, formatting is not a matter of personal style -- there is exactly one correct format, enforced by the tool. Every Go project uses it. No arguments about tabs vs spaces (Go uses tabs). No arguments about brace placement (same line, always).

Shell
# Format a file
gofmt -w main.go

# Format all files in a package
go fmt ./...

# goimports = gofmt + auto-manages import statements
goimports -w main.go

go vet: Static Analysis

go vet catches common mistakes that compile but are almost certainly bugs:

Shell
go vet ./...

Things go vet catches:

• Printf format string mismatches (%d with a string argument)
• Unreachable code after return/break/continue
• Suspicious mutex lock/unlock patterns
• Copying a sync.Mutex by value (must use pointer)
• Struct tags with incorrect syntax

Naming Conventions

Go naming is terse and purposeful. The scope of a variable determines how descriptive its name should be.

Go
// Package names: lowercase, single word, no underscores
package http     // good
package httputil // good
package http_util // bad -- no underscores
package utils    // bad -- too generic

// Exported names: MixedCaps (PascalCase)
func ReadFile(path string) ([]byte, error)  // exported
func readFile(path string) ([]byte, error)  // unexported

// Variable names: short for small scopes, descriptive for large
for i := 0; i < n; i++ {}           // i is fine in a loop
for _, v := range values {}          // v is fine in short loops
var userCount int                     // descriptive for package-level

// Receiver names: 1-2 letters, consistent, NEVER "this" or "self"
func (s *Server) Start() {}   // good: s for Server
func (db *DB) Query() {}      // good: db for DB
func (this *Server) Start() {} // bad: never use "this"

// Acronyms: keep ALL CAPS
var httpClient *http.Client   // not httpClient
var userID int                // not userId
type HTMLParser struct{}      // not HtmlParser
type URLBuilder struct{}     // not UrlBuilder

Table-Driven Tests

Table-driven tests are the dominant testing pattern in Go. They reduce boilerplate and make it trivial to add new test cases:

Go
func TestIsPalindrome(t *testing.T) {
    tests := []struct {
        name  string
        input string
        want  bool
    }{
        {"empty string", "", true},
        {"single char", "a", true},
        {"palindrome", "racecar", true},
        {"not palindrome", "hello", false},
        {"even length", "abba", true},
        {"with spaces", "race car", false},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got := IsPalindrome(tt.input)
            if got != tt.want {
                t.Errorf("IsPalindrome(%q) = %v, want %v",
                    tt.input, got, tt.want)
            }
        })
    }
}

Package Design

• Small, focused packages: Each package should do one thing well. If a package name is vague (like utils, helpers, common), it's a sign the package is doing too much.
• Name by responsibility: package auth, package storage, package email -- not package authHelpers.
• Avoid circular imports: Go does not allow them. If A imports B and B imports A, you need to refactor -- usually by extracting a shared interface into a third package.
• Internal packages: Use the internal/ directory to make packages private to your module. Code in myapp/internal/db can only be imported by code inside myapp/.

Context Propagation

context.Context should be the first parameter of any function that performs I/O, is long-running, or might need to be cancelled:

Go
// GOOD: context is always the first parameter, named ctx
func GetUser(ctx context.Context, id int) (*User, error) { ... }
func SendEmail(ctx context.Context, to, body string) error { ... }
func FetchData(ctx context.Context, url string) ([]byte, error) { ... }

// BAD: don't store context in a struct
type Server struct {
    ctx context.Context  // don't do this
}

// BAD: don't use context.Background() deep in the call stack
func deepFunction() {
    ctx := context.Background()  // lost the caller's context!
}

Common Pitfalls to Avoid

• Don't use init() unless absolutely necessary. It runs before main(), has no error return, and makes code harder to test. Prefer explicit initialization in main().
• Don't use global variables. Pass dependencies explicitly. This makes code testable and the dependency graph visible.
• Don't ignore errors. _ = someFunction() is almost never correct. If you truly don't care about the error, add a comment explaining why.
• Don't use panic for control flow. It's not an exception. It's a crash.
• Don't use goroutines without knowing how they'll stop. Every goroutine should have a clear termination condition.

12. Building Cool Stuff with Go

Everything from sections 1-11 comes together here. These are practical, runnable examples that show how Go's simplicity and concurrency model make it a joy to build real software. Each example is self-contained -- you can copy-paste it into a main.go and run it.

CLI Tool with flag Package

Go's flag package makes it simple to build command-line tools with typed flags and usage messages:

Go
package main

import (
    "flag"
    "fmt"
    "os"
    "strings"
)

func main() {
    // Define flags
    upper := flag.Bool("upper", false, "convert to uppercase")
    repeat := flag.Int("repeat", 1, "number of times to repeat")
    sep := flag.String("sep", "\n", "separator between repetitions")

    flag.Usage = func() {
        fmt.Fprintf(os.Stderr, "Usage: %s [options] <text>\n\nOptions:\n", os.Args[0])
        flag.PrintDefaults()
    }

    flag.Parse()

    if flag.NArg() == 0 {
        flag.Usage()
        os.Exit(1)
    }

    text := strings.Join(flag.Args(), " ")
    if *upper {
        text = strings.ToUpper(text)
    }

    parts := make([]string, *repeat)
    for i := range parts {
        parts[i] = text
    }
    fmt.Println(strings.Join(parts, *sep))
}
// Run: go run main.go -upper -repeat 3 -sep " | " hello world
// Output: HELLO WORLD | HELLO WORLD | HELLO WORLD

HTTP Server with Routing (Go 1.22+)

Go
package main

import (
    "encoding/json"
    "log"
    "net/http"
    "sync"
)

type Todo struct {
    ID   int    `json:"id"`
    Text string `json:"text"`
    Done bool   `json:"done"`
}

type TodoStore struct {
    mu     sync.RWMutex
    todos  []Todo
    nextID int
}

func NewTodoStore() *TodoStore {
    return &TodoStore{nextID: 1}
}

func (s *TodoStore) List() []Todo {
    s.mu.RLock()
    defer s.mu.RUnlock()
    return append([]Todo{}, s.todos...)
}

func (s *TodoStore) Add(text string) Todo {
    s.mu.Lock()
    defer s.mu.Unlock()
    t := Todo{ID: s.nextID, Text: text}
    s.nextID++
    s.todos = append(s.todos, t)
    return t
}

func main() {
    store := NewTodoStore()
    mux := http.NewServeMux()

    // GET /api/todos -- list all todos
    mux.HandleFunc("GET /api/todos", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(store.List())
    })

    // POST /api/todos -- create a new todo
    mux.HandleFunc("POST /api/todos", func(w http.ResponseWriter, r *http.Request) {
        var input struct {
            Text string `json:"text"`
        }
        if err := json.NewDecoder(r.Body).Decode(&input); err != nil {
            http.Error(w, "invalid JSON", http.StatusBadRequest)
            return
        }
        if input.Text == "" {
            http.Error(w, "text is required", http.StatusBadRequest)
            return
        }
        todo := store.Add(input.Text)
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(http.StatusCreated)
        json.NewEncoder(w).Encode(todo)
    })

    log.Println("Todo API server on :8080")
    log.Fatal(http.ListenAndServe(":8080", mux))
}
// Test: curl localhost:8080/api/todos
// Test: curl -X POST -d '{"text":"learn go"}' localhost:8080/api/todos

TCP Echo Server

A raw TCP server showing goroutine-per-connection -- the classic Go networking pattern:

Go
package main

import (
    "io"
    "log"
    "net"
)

func handleConn(conn net.Conn) {
    defer conn.Close()
    log.Printf("new connection from %s", conn.RemoteAddr())

    // Echo everything back -- io.Copy streams bytes from reader to writer
    n, err := io.Copy(conn, conn)
    if err != nil {
        log.Printf("connection error: %v", err)
        return
    }
    log.Printf("connection closed, echoed %d bytes", n)
}

func main() {
    ln, err := net.Listen("tcp", ":9000")
    if err != nil {
        log.Fatal(err)
    }
    log.Println("TCP echo server on :9000")

    for {
        conn, err := ln.Accept()
        if err != nil {
            log.Println("accept error:", err)
            continue
        }
        go handleConn(conn)  // one goroutine per connection
    }
}
// Test: echo "hello" | nc localhost 9000

Concurrent File Watcher

Watch a directory for file changes using goroutines, tickers, and channels -- bringing together concepts from sections 7 and 10:

Go
package main

import (
    "context"
    "fmt"
    "log"
    "os"
    "os/signal"
    "path/filepath"
    "time"
)

type FileChange struct {
    Path    string
    ModTime time.Time
    Action  string // "modified" or "created" or "deleted"
}

func watchDir(ctx context.Context, dir string, interval time.Duration) <-chan FileChange {
    changes := make(chan FileChange)
    known := make(map[string]time.Time)

    go func() {
        defer close(changes)
        ticker := time.NewTicker(interval)
        defer ticker.Stop()

        for {
            select {
            case <-ctx.Done():
                return
            case <-ticker.C:
                current := make(map[string]time.Time)
                filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
                    if err != nil || info.IsDir() {
                        return nil
                    }
                    current[path] = info.ModTime()
                    prev, exists := known[path]
                    if !exists {
                        changes <- FileChange{path, info.ModTime(), "created"}
                    } else if info.ModTime().After(prev) {
                        changes <- FileChange{path, info.ModTime(), "modified"}
                    }
                    return nil
                })

                // Check for deletions
                for path, modTime := range known {
                    if _, exists := current[path]; !exists {
                        changes <- FileChange{path, modTime, "deleted"}
                    }
                }
                known = current
            }
        }
    }()
    return changes
}

func main() {
    dir := "."
    if len(os.Args) > 1 {
        dir = os.Args[1]
    }

    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
    defer stop()

    log.Printf("watching %s for changes (Ctrl+C to stop)", dir)
    for change := range watchDir(ctx, dir, 1*time.Second) {
        fmt.Printf("[%s] %s (%s)\n", change.Action, change.Path,
            change.ModTime.Format("15:04:05"))
    }
    log.Println("watcher stopped")
}

WebSocket Echo Server

A minimal WebSocket echo server using the golang.org/x/net/websocket package (part of Go's extended standard library):

Go
package main

import (
    "fmt"
    "log"
    "net/http"

    "golang.org/x/net/websocket"
)

func echoHandler(ws *websocket.Conn) {
    defer ws.Close()
    log.Printf("client connected: %s", ws.RemoteAddr())

    for {
        var msg string
        err := websocket.Message.Receive(ws, &msg)
        if err != nil {
            log.Printf("read error: %v", err)
            return
        }
        log.Printf("received: %s", msg)

        reply := fmt.Sprintf("echo: %s", msg)
        err = websocket.Message.Send(ws, reply)
        if err != nil {
            log.Printf("write error: %v", err)
            return
        }
    }
}

func main() {
    http.Handle("/ws", websocket.Handler(echoHandler))
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "text/html")
        fmt.Fprint(w, `<!DOCTYPE html><html><body>
<input id="msg" placeholder="type a message">
<button onclick="send()">Send</button>
<pre id="log"></pre>
<script>
var ws = new WebSocket("ws://"+location.host+"/ws");
ws.onmessage = function(e) {
  document.getElementById("log").textContent += e.data + "\n";
};
function send() {
  var input = document.getElementById("msg");
  ws.send(input.value);
  input.value = "";
}
</script></body></html>`)
    })

    log.Println("WebSocket server on :8080")
    log.Fatal(http.ListenAndServe(":8080", nil))
}
// Run: go run main.go
// Open: http://localhost:8080

13. Build Your Own X in Go

The best way to truly understand systems is to build them yourself. Go is the perfect language for this -- its simplicity means you spend time understanding the problem, not fighting the language. Below are guides for building real systems from scratch, with the reasoning behind every design decision. Inspired by build-your-own-x.

Each project follows the same structure: why you should build it, the architecture, a step-by-step breakdown of key design decisions, core code for the critical parts, and the "aha" moments you'll walk away with.

Project 1: Key-Value Store (like Redis)

  Client          TCP Server         Store            Disk
  ------          ----------         -----            ----
  SET k v  ---->  Parse command ---> map[string]string
                                     (sync.RWMutex)  ---> AOF file
  GET k    ---->  Parse command ---> Read from map
  DEL k    ---->  Parse command ---> Delete from map  ---> AOF file

  Wire protocol (text-based):
    SET key value\n   -->  +OK\n
    GET key\n          -->  +value\n  or  -ERR not found\n
    DEL key\n          -->  +OK\n

Go
package main

import (
    "bufio"
    "fmt"
    "log"
    "net"
    "os"
    "strings"
    "sync"
)

// --- Core data structure: thread-safe map ---
type KVStore struct {
    mu   sync.RWMutex
    data map[string]string
    aof  *os.File // append-only file for persistence
}

func NewKVStore(aofPath string) *KVStore {
    f, _ := os.OpenFile(aofPath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
    return &KVStore{data: make(map[string]string), aof: f}
}

func (s *KVStore) Set(key, value string) {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.data[key] = value
    // Persist to AOF -- every write is logged
    fmt.Fprintf(s.aof, "SET %s %s\n", key, value)
}

func (s *KVStore) Get(key string) (string, bool) {
    s.mu.RLock()         // Read lock -- multiple readers OK
    defer s.mu.RUnlock()
    val, ok := s.data[key]
    return val, ok
}

func (s *KVStore) Del(key string) {
    s.mu.Lock()
    defer s.mu.Unlock()
    delete(s.data, key)
    fmt.Fprintf(s.aof, "DEL %s\n", key)
}

// --- TCP server: one goroutine per connection ---
func handleConn(conn net.Conn, store *KVStore) {
    defer conn.Close()
    scanner := bufio.NewScanner(conn)
    for scanner.Scan() {
        parts := strings.Fields(scanner.Text())
        if len(parts) == 0 {
            continue
        }
        switch strings.ToUpper(parts[0]) {
        case "SET":
            if len(parts) < 3 {
                fmt.Fprint(conn, "-ERR usage: SET key value\n")
                continue
            }
            store.Set(parts[1], strings.Join(parts[2:], " "))
            fmt.Fprint(conn, "+OK\n")
        case "GET":
            if val, ok := store.Get(parts[1]); ok {
                fmt.Fprintf(conn, "+%s\n", val)
            } else {
                fmt.Fprint(conn, "-ERR not found\n")
            }
        case "DEL":
            store.Del(parts[1])
            fmt.Fprint(conn, "+OK\n")
        default:
            fmt.Fprint(conn, "-ERR unknown command\n")
        }
    }
}

func main() {
    store := NewKVStore("data.aof")
    ln, err := net.Listen("tcp", ":6380")
    if err != nil {
        log.Fatal(err)
    }
    log.Println("KV store listening on :6380")
    for {
        conn, err := ln.Accept()
        if err != nil {
            log.Println(err)
            continue
        }
        go handleConn(conn, store) // goroutine per client
    }
}
// Test: echo -e "SET name sean\nGET name" | nc localhost 6380

Project 2: Unix Shell

  REPL Loop
  ---------
  1. Print prompt ("$ ")
  2. Read line from stdin
  3. Parse into commands (split by "|" for pipes)
  4. For each command:
     - Built-in? (cd, exit, export) --> handle directly
     - External? --> fork/exec via os/exec
  5. If piped: connect stdout of cmd[i] to stdin of cmd[i+1]
  6. Wait for all processes to finish
  7. Go to step 1

  Key insight: cd MUST be a built-in because it changes
  the shell process's own working directory. An external
  program would change its own directory, then exit.

Go
package main

import (
    "bufio"
    "fmt"
    "os"
    "os/exec"
    "strings"
)

// --- Built-in commands (must run in shell process) ---
func runBuiltin(args []string) bool {
    switch args[0] {
    case "cd":
        dir := os.Getenv("HOME")
        if len(args) > 1 {
            dir = args[1]
        }
        if err := os.Chdir(dir); err != nil {
            fmt.Fprintf(os.Stderr, "cd: %s\n", err)
        }
        return true
    case "exit":
        os.Exit(0)
        return true
    case "export":
        for _, arg := range args[1:] {
            parts := strings.SplitN(arg, "=", 2)
            if len(parts) == 2 {
                os.Setenv(parts[0], parts[1])
            }
        }
        return true
    }
    return false
}

// --- Execute a pipeline of commands ---
func execPipeline(cmds [][]string) {
    if len(cmds) == 1 {
        // Simple case: no pipes
        cmd := exec.Command(cmds[0][0], cmds[0][1:]...)
        cmd.Stdin, cmd.Stdout, cmd.Stderr = os.Stdin, os.Stdout, os.Stderr
        cmd.Run()
        return
    }
    // Pipe case: connect stdout[i] -> stdin[i+1]
    var procs []*exec.Cmd
    for i, c := range cmds {
        proc := exec.Command(c[0], c[1:]...)
        if i > 0 {
            // Connect previous command's stdout to this stdin
            proc.Stdin, _ = procs[i-1].StdoutPipe()
        }
        if i == len(cmds)-1 {
            proc.Stdout = os.Stdout // last command prints to terminal
        }
        proc.Stderr = os.Stderr
        procs = append(procs, proc)
    }
    // Start all processes, then wait for all
    for _, p := range procs { p.Start() }
    for _, p := range procs { p.Wait() }
}

// --- Parse input: split by pipes, then by spaces ---
func parse(line string) [][]string {
    var cmds [][]string
    for _, segment := range strings.Split(line, "|") {
        args := strings.Fields(strings.TrimSpace(segment))
        if len(args) > 0 {
            cmds = append(cmds, args)
        }
    }
    return cmds
}

// --- REPL: Read-Eval-Print Loop ---
func main() {
    scanner := bufio.NewScanner(os.Stdin)
    for {
        fmt.Print("gosh$ ")
        if !scanner.Scan() {
            break
        }
        line := strings.TrimSpace(scanner.Text())
        if line == "" { continue }

        cmds := parse(line)
        if len(cmds) == 1 && runBuiltin(cmds[0]) {
            continue // built-in handled it
        }
        execPipeline(cmds)
    }
}
// Try: ls -la | grep go | wc -l

Project 3: Programming Language & Compiler

  Source Code ---> Lexer ---> Parser ---> ???
                  (tokens)    (AST)

  Approach 1: Tree-Walking Interpreter
    AST ---> Walk each node ---> Evaluate directly
    Simplest. Slowest. Great for learning.

  Approach 2: Bytecode Compiler + VM
    AST ---> Compile to bytecodes ---> VM executes bytecodes
    Python, Ruby, Lua work this way.

  Approach 3: Native Compiler
    AST ---> Generate machine code (or LLVM IR)
    Go, C, Rust work this way. Fastest output.

  We'll build Approach 1: an interpreter for a calculator
  language that handles: 2 + 3 * 4  (answer: 14, not 20)

Go
package main

import (
    "fmt"
    "strconv"
    "unicode"
)

// ===== STEP 1: LEXER (source code -> tokens) =====
type TokenType int
const (
    NUM TokenType = iota  // 42, 3.14
    PLUS                   // +
    MINUS                  // -
    STAR                   // *
    SLASH                  // /
    LPAREN                 // (
    RPAREN                 // )
    EOF
)

type Token struct {
    Type    TokenType
    Literal string
}

func lex(input string) []Token {
    var tokens []Token
    i := 0
    for i < len(input) {
        ch := rune(input[i])
        switch {
        case unicode.IsSpace(ch):
            i++
        case unicode.IsDigit(ch):
            start := i
            for i < len(input) && (unicode.IsDigit(rune(input[i])) || input[i] == '.') {
                i++
            }
            tokens = append(tokens, Token{NUM, input[start:i]})
        case ch == '+': tokens = append(tokens, Token{PLUS, "+"}); i++
        case ch == '-': tokens = append(tokens, Token{MINUS, "-"}); i++
        case ch == '*': tokens = append(tokens, Token{STAR, "*"}); i++
        case ch == '/': tokens = append(tokens, Token{SLASH, "/"}); i++
        case ch == '(': tokens = append(tokens, Token{LPAREN, "("}); i++
        case ch == ')': tokens = append(tokens, Token{RPAREN, ")"}); i++
        default:
            i++ // skip unknown
        }
    }
    tokens = append(tokens, Token{EOF, ""})
    return tokens
}

// ===== STEP 2: PARSER (tokens -> AST) =====
// Recursive descent with operator precedence:
//   expr   = term (('+' | '-') term)*
//   term   = factor (('*' | '/') factor)*
//   factor = NUM | '(' expr ')'
// This gives * and / higher precedence than + and -

type Node interface{ eval() float64 }

type NumNode struct{ val float64 }
func (n NumNode) eval() float64 { return n.val }

type BinOp struct{ left, right Node; op TokenType }
func (b BinOp) eval() float64 {
    l, r := b.left.eval(), b.right.eval()
    switch b.op {
    case PLUS:  return l + r
    case MINUS: return l - r
    case STAR:  return l * r
    case SLASH: return l / r
    }
    return 0
}

type Parser struct {
    tokens []Token
    pos    int
}

func (p *Parser) peek() Token  { return p.tokens[p.pos] }
func (p *Parser) advance() Token {
    t := p.tokens[p.pos]; p.pos++; return t
}

func (p *Parser) factor() Node {
    t := p.advance()
    if t.Type == NUM {
        val, _ := strconv.ParseFloat(t.Literal, 64)
        return NumNode{val}
    }
    if t.Type == LPAREN {
        node := p.expr()
        p.advance() // consume ')'
        return node
    }
    return NumNode{0}
}

func (p *Parser) term() Node {
    left := p.factor()
    for p.peek().Type == STAR || p.peek().Type == SLASH {
        op := p.advance().Type
        left = BinOp{left, p.factor(), op}
    }
    return left
}

func (p *Parser) expr() Node {
    left := p.term()
    for p.peek().Type == PLUS || p.peek().Type == MINUS {
        op := p.advance().Type
        left = BinOp{left, p.term(), op}
    }
    return left
}

// ===== STEP 3: EVALUATE =====
func main() {
    inputs := []string{
        "2 + 3 * 4",       // = 14 (not 20!)
        "(2 + 3) * 4",     // = 20
        "10 / 2 + 3",      // = 8
        "100 - 50 * 2",    // = 0
    }
    for _, input := range inputs {
        tokens := lex(input)
        parser := &Parser{tokens: tokens}
        ast := parser.expr()
        fmt.Printf("%s = %g\n", input, ast.eval())
    }
}
// Next steps: add variables (let x = 5), if/else,
// functions, and you have a real language!

Project 4: HTTP Load Balancer

  Client                Load Balancer              Backends
  ------                -------------              --------
  HTTP request ---->    Round-robin select ---->    Backend 1 (healthy)
                        Proxy request              Backend 2 (healthy)
                        Return response   <----    Backend 3 (down!)
                                                       ^
                        Health checker goroutine        |
                        (pings every 10s) -------------+
                        marks backends alive/dead

  Key decisions:
  - Round-robin: simple, fair, stateless
  - Health checks: don't send traffic to dead backends
  - httputil.ReverseProxy: Go's stdlib does the hard work

Go
package main

import (
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
    "sync"
    "sync/atomic"
    "time"
)

type Backend struct {
    URL          *url.URL
    Alive        bool
    mu           sync.RWMutex
    ReverseProxy *httputil.ReverseProxy
}

func (b *Backend) IsAlive() bool {
    b.mu.RLock()
    defer b.mu.RUnlock()
    return b.Alive
}

func (b *Backend) SetAlive(alive bool) {
    b.mu.Lock()
    defer b.mu.Unlock()
    b.Alive = alive
}

type LoadBalancer struct {
    backends []*Backend
    counter  uint64 // atomic round-robin counter
}

// Round-robin: pick next healthy backend
func (lb *LoadBalancer) nextBackend() *Backend {
    n := len(lb.backends)
    for i := 0; i < n; i++ {
        idx := atomic.AddUint64(&lb.counter, 1) % uint64(n)
        if lb.backends[idx].IsAlive() {
            return lb.backends[idx]
        }
    }
    return nil // all backends dead
}

// Health checker: goroutine pings each backend
func (lb *LoadBalancer) healthCheck(interval time.Duration) {
    for {
        for _, b := range lb.backends {
            resp, err := http.Get(b.URL.String() + "/health")
            alive := err == nil && resp.StatusCode == 200
            b.SetAlive(alive)
            if !alive {
                log.Printf("Backend %s is DOWN", b.URL)
            }
        }
        time.Sleep(interval)
    }
}

func (lb *LoadBalancer) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    backend := lb.nextBackend()
    if backend == nil {
        http.Error(w, "all backends are down", http.StatusServiceUnavailable)
        return
    }
    backend.ReverseProxy.ServeHTTP(w, r) // proxy the request
}

func main() {
    urls := []string{"http://localhost:8081", "http://localhost:8082"}
    var backends []*Backend
    for _, u := range urls {
        parsed, _ := url.Parse(u)
        backends = append(backends, &Backend{
            URL:          parsed,
            Alive:        true,
            ReverseProxy: httputil.NewSingleHostReverseProxy(parsed),
        })
    }
    lb := &LoadBalancer{backends: backends}
    go lb.healthCheck(10 * time.Second)

    log.Println("Load balancer on :8080")
    http.ListenAndServe(":8080", lb)
}

Project 5: Git (simplified)

  .gogit/
  +-- objects/          # content-addressable store
  |   +-- a1/          # first 2 chars of SHA-1
  |   |   +-- b2c3...  # remaining chars (the actual content)
  |   +-- ...
  +-- refs/
  |   +-- heads/
  |       +-- main     # file containing SHA-1 of latest commit
  +-- HEAD             # "ref: refs/heads/main"

  Three object types:
  1. Blob   = file content (compressed)
  2. Tree   = directory listing (name -> blob/tree hash)
  3. Commit = tree hash + parent hash + author + message

  commit3 --> commit2 --> commit1   (DAG: each points to parent)
    |            |           |
   tree3       tree2       tree1    (snapshot of all files)
    |            |           |
  blobs        blobs       blobs    (actual file contents)

Go
package main

import (
    "compress/zlib"
    "crypto/sha1"
    "fmt"
    "io"
    "os"
    "path/filepath"
)

const gitDir = ".gogit"

// --- init: create the .gogit directory structure ---
func initRepo() {
    for _, dir := range []string{"objects", "refs/heads"} {
        os.MkdirAll(filepath.Join(gitDir, dir), 0755)
    }
    os.WriteFile(filepath.Join(gitDir, "HEAD"),
        []byte("ref: refs/heads/main\n"), 0644)
    fmt.Println("Initialized empty gogit repository")
}

// --- hash-object: store content, return SHA-1 hash ---
// This is the CORE of Git: content-addressable storage
func hashObject(data []byte, objType string) string {
    // Git format: "type size\0content"
    header := fmt.Sprintf("%s %d\x00", objType, len(data))
    full := append([]byte(header), data...)

    // SHA-1 hash of the content (this IS the address)
    hash := fmt.Sprintf("%x", sha1.Sum(full))

    // Store: .gogit/objects/ab/cdef1234...
    dir := filepath.Join(gitDir, "objects", hash[:2])
    os.MkdirAll(dir, 0755)

    path := filepath.Join(dir, hash[2:])
    f, _ := os.Create(path)
    defer f.Close()

    // Compress with zlib (just like real Git)
    w := zlib.NewWriter(f)
    w.Write(full)
    w.Close()

    return hash
}

// --- cat-file: read an object by its hash ---
func catFile(hash string) []byte {
    path := filepath.Join(gitDir, "objects", hash[:2], hash[2:])
    f, _ := os.Open(path)
    defer f.Close()
    r, _ := zlib.NewReader(f)
    data, _ := io.ReadAll(r)
    // Skip past the "type size\0" header
    for i, b := range data {
        if b == 0 {
            return data[i+1:]
        }
    }
    return data
}

func main() {
    if len(os.Args) < 2 { fmt.Println("usage: gogit <command>"); return }
    switch os.Args[1] {
    case "init":
        initRepo()
    case "hash-object":
        data, _ := os.ReadFile(os.Args[2])
        hash := hashObject(data, "blob")
        fmt.Println(hash)
    case "cat-file":
        fmt.Print(string(catFile(os.Args[2])))
    }
}
// Try: gogit init
//      echo "hello" > test.txt
//      gogit hash-object test.txt  # -> sha1 hash
//      gogit cat-file <hash>       # -> "hello"

Project 6: Docker (simplified container runtime)

  Normal process:        Containerized process:
  - Sees all PIDs        - Sees only its own PIDs (PID namespace)
  - Sees host hostname   - Has its own hostname (UTS namespace)
  - Sees host filesystem - Sees only its root dir (chroot / pivot_root)
  - Shares network       - Has its own network (NET namespace)
  - No resource limits   - CPU/memory limited (cgroups)

  Container = process + namespaces + chroot + cgroups
  That's it. No hypervisor, no guest OS, no emulation.

  Our implementation:
  1. Clone process with new namespaces (CLONE_NEWPID, CLONE_NEWUTS)
  2. chroot into a filesystem (e.g., an Alpine Linux rootfs)
  3. Mount /proc so ps works inside the container
  4. Set hostname to prove UTS namespace isolation
  5. Exec the requested command

Go
package main

import (
    "fmt"
    "os"
    "os/exec"
    "syscall"
)

// go run main.go run /bin/sh
// This creates an isolated container process!

func main() {
    switch os.Args[1] {
    case "run":
        run()
    case "child":
        child() // called inside the new namespaces
    default:
        fmt.Println("usage: container run <command>")
    }
}

func run() {
    // Re-exec ourselves with "child" -- but in NEW namespaces
    cmd := exec.Command("/proc/self/exe", append([]string{"child"}, os.Args[2:]...)...)
    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

    // THE MAGIC: create new namespaces for the child
    cmd.SysProcAttr = &syscall.SysProcAttr{
        Cloneflags: syscall.CLONE_NEWUTS |  // own hostname
                    syscall.CLONE_NEWPID |  // own PID space (PID 1!)
                    syscall.CLONE_NEWNS,    // own mount points
    }
    cmd.Run()
}

func child() {
    // We're now inside the new namespaces!

    // Set hostname (proves UTS namespace isolation)
    syscall.Sethostname([]byte("container"))

    // chroot: isolate filesystem
    // You need an Alpine rootfs at ./rootfs
    // Download: https://alpinelinux.org/downloads/ (mini root filesystem)
    syscall.Chroot("./rootfs")
    syscall.Chdir("/")

    // Mount /proc so ps, top, etc. work inside container
    syscall.Mount("proc", "proc", "proc", 0, "")

    // Run the requested command (e.g., /bin/sh)
    cmd := exec.Command(os.Args[2], os.Args[3:]...)
    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr
    cmd.Run()

    // Cleanup
    syscall.Unmount("proc", 0)
}
// Try (as root on Linux):
//   go run main.go run /bin/sh
//   hostname     # prints "container" not your real hostname
//   ps aux       # shows only processes in this container
//   exit

14. Resource Pools & Memory Allocators

Every high-performance system eventually runs into the same bottleneck: creating resources is expensive. Whether it's a database connection that requires a TCP handshake and authentication, a buffer that pressures the garbage collector, or a page of memory from the operating system -- the cost of creation adds up fast. This section digs into two fundamental strategies for managing that cost: resource pools (reuse expensive things) and memory allocators (make allocation itself cheaper).

Part A: The Resource Pool Pattern

What is a Resource Pool?

A resource pool is a collection of pre-created, reusable resources. Instead of creating a resource every time you need one, you borrow one from the pool, use it, and return it when you're done. The pool manages the lifecycle: creation, validation, reuse, and eventual destruction.

The problem is straightforward: creating resources has a real cost.

• Database connections: TCP handshake + TLS negotiation + authentication = 50-100ms per connection
• TCP sockets: system calls to the kernel, file descriptor allocation
• Buffers: heap allocation + eventual GC pressure
• Threads/goroutines: stack allocation + scheduler overhead (cheap in Go, but not free at millions)

The solution: create them once, reuse them.

Resource Pool
┌─────────────────────────────────────────────────────────────┐
│                                                             │
│  available: [ resource₁, resource₂, resource₃ ]            │
│                                                             │
│  in-use:    [ resource₄, resource₅ ]                        │
│                                                             │
│  waiters:   [ goroutine_A, goroutine_B ]  ← blocked until  │
│                                              one is freed   │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Lifecycle:  Create → Acquire → Use → Release → (Reuse or Destroy)

Why Pools Exist -- The Cost of Creation

The same principle applies across many resource types:

Building a Generic Resource Pool in Go

Let's build a real, production-quality resource pool using Go generics (Go 1.18+). This pool supports context-based timeouts, maximum size limits, and clean shutdown.

Go
package pool

import (
    "context"
    "sync"
)

// Pool is a generic resource pool. T can be any type: a DB conn,
// a byte buffer, a gRPC client -- anything expensive to create.
type Pool[T any] struct {
    mu        sync.Mutex
    available []T
    maxSize   int
    curSize   int         // total resources created (available + in-use)
    waiters   chan T       // channel for goroutines waiting for a resource
    factory   func() (T, error) // creates a new resource
    destroy   func(T)     // cleans up a resource
    closed    bool
}

// New creates a pool. factory creates resources, destroy cleans them up.
func New[T any](maxSize int, factory func() (T, error), destroy func(T)) *Pool[T] {
    return &Pool[T]{
        maxSize: maxSize,
        waiters: make(chan T, maxSize),
        factory: factory,
        destroy: destroy,
    }
}

// Acquire gets a resource from the pool. If none are available and the
// pool isn't full, it creates a new one. If the pool IS full, it blocks
// until one is returned or the context expires.
func (p *Pool[T]) Acquire(ctx context.Context) (T, error) {
    p.mu.Lock()

    // 1. Try to grab an available resource
    if len(p.available) > 0 {
        resource := p.available[len(p.available)-1]
        p.available = p.available[:len(p.available)-1]
        p.mu.Unlock()
        return resource, nil
    }

    // 2. No available resources -- can we create a new one?
    if p.curSize < p.maxSize {
        p.curSize++
        p.mu.Unlock()
        return p.factory()
    }

    // 3. Pool is full -- wait for a resource to be returned
    p.mu.Unlock()
    select {
    case resource := <-p.waiters:
        return resource, nil
    case <-ctx.Done():
        var zero T
        return zero, ctx.Err()
    }
}

// Release returns a resource to the pool for reuse.
func (p *Pool[T]) Release(resource T) {
    p.mu.Lock()
    if p.closed {
        p.mu.Unlock()
        p.destroy(resource)
        return
    }

    // Try to hand it directly to a waiting goroutine
    select {
    case p.waiters <- resource:
        p.mu.Unlock()
    default:
        // No waiters -- put it back in available
        p.available = append(p.available, resource)
        p.mu.Unlock()
    }
}

// Drain shuts down the pool and destroys all available resources.
func (p *Pool[T]) Drain() {
    p.mu.Lock()
    p.closed = true
    resources := p.available
    p.available = nil
    p.mu.Unlock()

    for _, r := range resources {
        p.destroy(r)
    }
}

Using the pool for database connections:

Go
// Create a pool of up to 20 database connections
dbPool := pool.New[*sql.DB](20,
    func() (*sql.DB, error) {
        return sql.Open("postgres", "postgres://localhost/mydb")
    },
    func(db *sql.DB) {
        db.Close()
    },
)
defer dbPool.Drain()

// In your request handler:
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

conn, err := dbPool.Acquire(ctx)
if err != nil {
    // Timed out waiting for a connection
    return fmt.Errorf("pool exhausted: %w", err)
}
defer dbPool.Release(conn)

// Use conn normally...

Using the pool for reusable byte buffers:

Go
// Pool of 4KB byte buffers to avoid repeated allocation
bufPool := pool.New[*bytes.Buffer](100,
    func() (*bytes.Buffer, error) {
        return bytes.NewBuffer(make([]byte, 0, 4096)), nil
    },
    func(b *bytes.Buffer) {
        // Nothing to clean up for a buffer
    },
)

// In a hot path:
buf, _ := bufPool.Acquire(ctx)
buf.Reset()  // Clear previous data but keep allocated memory
buf.WriteString("response data...")
// ... use buf ...
bufPool.Release(buf)

sync.Pool -- Go's Built-in Object Pool

sync.Pool is Go's standard library pool for temporary objects. It's designed to reduce GC pressure by reusing short-lived allocations. But it has a critical property that makes it different from a connection pool:

When to use sync.Pool:

• Short-lived allocations in hot paths (byte buffers, encoding contexts)
• Objects where losing one has zero consequence (you just create another)
• Reducing GC pressure in high-throughput servers

When NOT to use sync.Pool:

• Database connections (need explicit open/close lifecycle)
• File handles (OS resource leak if not closed)
• Anything requiring cleanup or with external side effects

Real example -- HTTP handler reusing byte buffers:

Go
var bufferPool = &sync.Pool{
    New: func() any {
        // Called when the pool is empty -- create a 4KB buffer
        return bytes.NewBuffer(make([]byte, 0, 4096))
    },
}

func handleRequest(w http.ResponseWriter, r *http.Request) {
    // Get a buffer from the pool (or create one if empty)
    buf := bufferPool.Get().(*bytes.Buffer)
    buf.Reset()  // IMPORTANT: always reset before use

    // Build the response into the buffer
    json.NewEncoder(buf).Encode(responseData)

    // Write the response
    w.Write(buf.Bytes())

    // Return the buffer to the pool for reuse
    bufferPool.Put(buf)
}

Performance difference with and without sync.Pool:

Go
func BenchmarkWithoutPool(b *testing.B) {
    for i := 0; i < b.N; i++ {
        buf := bytes.NewBuffer(make([]byte, 0, 4096))
        buf.WriteString("some data to encode")
        _ = buf.Bytes()
        // buf becomes garbage -- GC must clean it up
    }
}
// BenchmarkWithoutPool-8    5000000    312 ns/op    4096 B/op    1 allocs/op

func BenchmarkWithPool(b *testing.B) {
    pool := &sync.Pool{New: func() any {
        return bytes.NewBuffer(make([]byte, 0, 4096))
    }}

    for i := 0; i < b.N; i++ {
        buf := pool.Get().(*bytes.Buffer)
        buf.Reset()
        buf.WriteString("some data to encode")
        _ = buf.Bytes()
        pool.Put(buf) // return to pool -- no GC needed
    }
}
// BenchmarkWithPool-8       20000000   89 ns/op     0 B/op       0 allocs/op
// 3.5x faster, zero allocations per operation!

Database Connection Pools in Go

Here's a fact that surprises many Go developers: sql.DB is not a single database connection. It's a connection pool. When you call sql.Open(), you get a pool manager, not a connection.

Go
// This does NOT open a connection. It creates a pool manager.
db, err := sql.Open("postgres", "postgres://localhost/mydb")

// Configure the pool:
db.SetMaxOpenConns(25)              // Max simultaneous connections
db.SetMaxIdleConns(10)              // Keep 10 idle connections warm
db.SetConnMaxLifetime(30 * time.Minute) // Recycle connections every 30min
db.SetConnMaxIdleTime(5 * time.Minute)  // Close idle connections after 5min

What each setting does:

Request arrives
     │
     ▼
┌─────────────────────┐
│ Pool has idle conn?  │──Yes──→ Reuse it (fast path)
└─────────────────────┘
     │ No
     ▼
┌─────────────────────┐
│ Under MaxOpenConns?  │──Yes──→ Create new connection (TCP+TLS+auth)
└─────────────────────┘
     │ No
     ▼
┌─────────────────────┐
│ Block until one is   │──→ Wait (or timeout with context)
│ returned to pool     │
└─────────────────────┘
     │
     ▼
  Use connection
     │
     ▼
  Return to idle pool (or close if past MaxIdleConns)

HTTP Client Connection Pooling

Go's http.Client with http.Transport has connection pooling built in. The transport maintains a pool of idle TCP connections and reuses them for subsequent requests to the same host.

Go
// Create ONE client and reuse it for all requests.
// Do NOT create a new http.Client per request!
var client = &http.Client{
    Transport: &http.Transport{
        MaxIdleConns:        100,              // Total idle connections across all hosts
        MaxIdleConnsPerHost: 10,               // Idle connections per host
        IdleConnTimeout:     90 * time.Second, // Close idle connections after 90s
    },
    Timeout: 30 * time.Second,
}

// Every call reuses connections from the pool:
resp, err := client.Get("https://api.example.com/data")
if err != nil {
    return err
}
defer resp.Body.Close()

// IMPORTANT: You MUST read and close the body, or the connection
// won't be returned to the pool.
io.ReadAll(resp.Body)

Part B: Memory Allocators -- How malloc() Works

What is a Memory Allocator?

Every time your program does make([]byte, 1024) or new(MyStruct), something has to find 1024 bytes of free memory and give it to you. That "something" is the memory allocator. It sits between your program and the operating system, solving a fundamental problem: the OS gives memory in large pages (typically 4KB), but your program needs chunks of all different sizes.

Your Go Program
   │
   │  make([]byte, 128)   new(MyStruct)   append(slice, item)
   │
   ▼
┌──────────────────────────────────────────┐
│         Go Runtime Allocator             │
│  (inspired by tcmalloc)                  │
│                                          │
│  Manages size classes, caches, spans     │
└──────────────────────────────────────────┘
   │
   │  mmap() / sbrk()   -- request pages from OS
   │
   ▼
┌──────────────────────────────────────────┐
│         Operating System                 │
│  Virtual Memory Manager                  │
│                                          │
│  Maps virtual pages → physical RAM       │
└──────────────────────────────────────────┘

How malloc() Works (Simplified)

Before understanding Go's allocator, it helps to understand the classic malloc() approach used in C. The core data structure is a free list: a linked list of available memory blocks.

Allocation strategies:

• First-fit: scan the free list, use the first block that's big enough. Fast but causes fragmentation at the front of the list.
• Best-fit: scan the entire list, use the smallest block that fits. Less wasted space but slower (must check everything).
• Buddy allocator: split memory into power-of-2 blocks. A 1024-byte request splits a 2048-byte block into two 1024-byte "buddies." When both are freed, they merge back. Fast and simple, but wastes up to 50% of memory (a 513-byte request uses a 1024-byte block).

Memory after many alloc/free cycles:

Address: 0     64    128   192   256   320   384   448   512
         ┌─────┬─────┬─────┬─────┬─────┬─────┬─────┬─────┐
         │USED │FREE │USED │FREE │FREE │USED │FREE │USED │
         │64B  │64B  │64B  │64B  │64B  │64B  │64B  │64B  │
         └─────┴─────┴─────┴─────┴─────┴─────┴─────┴─────┘

Free list: [128→64B] → [192→64B] → [256→64B] → [384→64B]

Problem: 4 free blocks of 64B each = 256B free, but you can't
allocate a single 256B block because they're not contiguous.
This is EXTERNAL fragmentation.

INTERNAL fragmentation: you need 50 bytes, allocator gives you
64 bytes (smallest available). 14 bytes are wasted inside the block.

Slab Allocator

The slab allocator, invented by Jeff Bonwick for the Solaris kernel, exploits a key insight: most allocations in a program use the same few sizes over and over. Instead of maintaining a single free list, it pre-allocates "slabs" of objects of the same size.

Slab-8    [████████ ████████ ████████ ████████ ░░░░░░░░ ░░░░░░░░]
           used     used     used     used     free     free

Slab-16   [████████████████ ████████████████ ░░░░░░░░░░░░░░░░]
           used             used             free

Slab-32   [████████████████████████████████ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]
           used                             free

Slab-64   [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]
           free (entire slab)

Request for 12 bytes → rounds up to Slab-16 → O(1) allocation
No fragmentation within the slab (all same size)
No searching needed (just pop from free list of that size class)

This approach is used in the Linux kernel, memcached, jemalloc, and directly inspired Go's runtime allocator. The insight is simple but powerful: eliminate the free list search by having a separate pool for each object size.

Arena Allocator (Bump Allocator)

The arena allocator (also called a bump allocator or linear allocator) is the simplest possible allocator. It maintains a pointer to the next free byte and just bumps it forward on each allocation.

Arena (1MB block of memory)
┌──────────────────────────────────────────────────────┐
│ alloc1 │ alloc2 │ alloc3 │ alloc4 │   free space ... │
└──────────────────────────────────────────────────────┘
                                    ↑
                                  offset (bump pointer)

Allocate(32 bytes):
  result = base + offset
  offset += 32
  return result         ← O(1), just pointer arithmetic!

Free individual object:
  NOT POSSIBLE.         ← You can only free the ENTIRE arena.

• Pro: O(1) allocation (just add to an integer), zero fragmentation, extremely cache-friendly (allocations are contiguous)
• Con: you can't free individual objects -- you free everything at once
• Perfect for: request-scoped work (parse a request, build a response, free all the temporary allocations at once)

Go 1.22+ has an experimental arena package, but you can build the concept in ~20 lines:

Go
type Arena struct {
    buf    []byte
    offset int
}

func NewArena(size int) *Arena {
    return &Arena{buf: make([]byte, size)}
}

// Alloc returns a slice of n bytes from the arena.
// O(1) -- just bumps the offset pointer.
func (a *Arena) Alloc(n int) ([]byte, error) {
    if a.offset+n > len(a.buf) {
        return nil, fmt.Errorf("arena: out of memory (need %d, have %d)",
            n, len(a.buf)-a.offset)
    }
    slice := a.buf[a.offset : a.offset+n]
    a.offset += n
    return slice, nil
}

// Reset frees ALL allocations at once -- O(1).
func (a *Arena) Reset() {
    a.offset = 0
}

// Usage: perfect for request-scoped work
func handleRequest(r *http.Request) {
    arena := NewArena(64 * 1024) // 64KB arena for this request

    // All allocations come from the arena -- no GC pressure
    headerBuf, _ := arena.Alloc(256)
    bodyBuf, _ := arena.Alloc(4096)
    tempBuf, _ := arena.Alloc(512)

    // ... process request using these buffers ...
    _ = headerBuf
    _ = bodyBuf
    _ = tempBuf

    // When the function returns, the arena (and all its allocations)
    // become garbage as a SINGLE object -- much cheaper for the GC
    // than tracking hundreds of individual allocations.
}

Go's Memory Allocator (tcmalloc-inspired)

Go's runtime allocator is based on Google's tcmalloc (thread-caching malloc). It's designed to be fast in concurrent programs by minimizing lock contention. The key idea: give each processor its own local cache so most allocations need zero locks.

Goroutine calls: make([]byte, 48)    (48 bytes → size class 48)

Level 1: mcache (per-P, NO LOCKS)
┌───────────────────────────────────┐
│  P0's mcache                      │
│  ┌─────────────────────────────┐  │
│  │ size-8:   [████░░░░]        │  │ ← mspan for 8-byte objects
│  │ size-16:  [████████░░░░]    │  │
│  │ size-32:  [████████████░░]  │  │
│  │ size-48:  [████░░░░░░░░]    │  │ ← alloc from here! Lock-free!
│  │ size-64:  [████████░░░░]    │  │
│  │ ...                         │  │
│  └─────────────────────────────┘  │
└───────────────────────────────────┘
         │ mcache empty for size-48?
         ▼
Level 2: mcentral (shared, LOCKED)
┌───────────────────────────────────┐
│  mcentral for size-48             │
│  ┌─────────┐  ┌─────────┐        │
│  │ mspan   │  │ mspan   │  ...   │ ← pool of mspans for this size
│  └─────────┘  └─────────┘        │
│  Grab a whole mspan, give to P0  │
└───────────────────────────────────┘
         │ mcentral has no free mspans?
         ▼
Level 3: mheap (global, LOCKED)
┌───────────────────────────────────┐
│  mheap                            │
│  Manages all memory pages         │
│  Allocates new spans from OS      │
│  mmap() syscall → get new pages   │
└───────────────────────────────────┘
         │
         ▼
    Operating System (virtual memory → physical RAM)

Go categorizes allocations by size:

Key concepts:

• mspan: a contiguous run of memory pages, all carved into objects of the same size class. An mspan for size-48 holds N objects of exactly 48 bytes each.
• mcache: per-P (per-processor) cache of mspans. Since each P only runs one goroutine at a time, there's no contention -- allocation is lock-free.
• mcentral: shared pool of mspans for each size class. When an mcache runs out, it grabs a whole mspan from mcentral (requires a lock, but happens infrequently).
• mheap: manages the virtual address space. When mcentral needs more mspans, mheap carves them from pages obtained via mmap().

Go's garbage collector is a concurrent, tri-color mark-and-sweep collector:

• Tri-color: objects are white (unvisited), gray (visited but children not scanned), or black (visited and all children scanned). The GC starts with roots as gray, processes gray→black, and sweeps remaining white objects.
• Concurrent: the GC runs alongside your goroutines, not stopping the world for the entire collection. Go targets sub-millisecond pause times.
• Write barrier: when the GC is running, writes to pointers are intercepted to maintain the tri-color invariant (no black object may point to a white object).

Pool Pattern vs Allocator -- When to Use What