Golang Experiments

These are just some miscellaneous code examples as I learn Go. They are almost entirely code pieced together from examples on the web.

Reading From AWS Secrets Manager

This is pretty simplistic but I'm just getting my feet wet with Golang and started from basics.

These are just a couple snippets to retrieve a secret from AWS Secrets Manager. I use the code to hold secrets for Ansible and miscellaneous Makefiles. Though Ansible has a module for Secrets Manager (aws_secret) and the AWS CLI can retrieve secrets, I needed a method that could be easily installed.

For example, in my Makefile I can do something like:

-- Makefile
GRAFANA_ADMIN_PASSWORD := $(shell get_password -name=/dhrl/dev/k8s/grafana)

install-grafana:
        kubectl create namespace grafana
        HELM_HOST=localhost:$(TILLER_PORT)  helm install stable/grafana \
            --name grafana \
            --namespace grafana \
            --set persistence.storageClassName="gp2" \
            --set adminPassword="$(GRAFANA_ADMIN_PASSWORD)" \
[snip]


I started using AWS with Assumed Roles. This was a convenient way of managing my account. I source the below script to set my AssumedRole credentials in my session. Using an assumed role with Secrets Manager makes it relatively easy to manage accounts and not worry about storing passwords.

ROLE='arn:aws:iam::123456789012:role/DIGITALHERMIT'
OUTPUT=$(aws sts assume-role --role-arn $ROLE --role-session-name "RoleArchitect" --profile test )

AWS_ACCESS_KEY_ID=$(echo $OUTPUT|jq '.Credentials.AccessKeyId'|tr -d '"')
AWS_SECRET_ACCESS_KEY=$(echo $OUTPUT|jq '.Credentials.SecretAccessKey'|tr -d '"')
AWS_SESSION_TOKEN=$(echo $OUTPUT|jq '.Credentials.SessionToken'|tr -d '"')

export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN


The code itself is just based on the example that Amazon provides after the secret is created. I added a few lines to allow passing the secret name but it's mainly boilerplate code.

package main
import (
   "github.com/aws/aws-sdk-go/aws"
   "github.com/aws/aws-sdk-go/aws/session"
   "github.com/aws/aws-sdk-go/aws/awserr"
   "github.com/aws/aws-sdk-go/service/secretsmanager"                                                                                                     
   "fmt"                                                                                                                                                                                       
   "flag"                                                                                                                                                                                      
   "os"                                                                                                                                                                                        
)                                                                                                                                                                                              



I added a few lines to accept command line parameters. The flag module makes this easy.

func main() {                                                                                                                                                                                  
   var secretName,secretVersion,region string                                                                                                                                                  
   flag.StringVar(&secretName, "name", "", "Name of secret")                                                                                                                                   
   flag.StringVar(&secretVersion, "version", "AWSCURRENT", "Version Stage (default: AWSCURRENT)")                                                                                              
   flag.StringVar(&region, "region", "us-east-1", "AWS Region (default: us-east-1)")                                                                                                           
   flag.Parse()                                                                                                                                                                                
   if len(secretName) == 0 {                                                                                                                                                                   
       fmt.Println("Error: Secret name required.")                                                                                                                                             
       os.Exit(1)
   }
   sn := secretName


I add a couple lines to pass the region.

   s, err := session.NewSessionWithOptions(session.Options{
          Config: aws.Config{
                    Region: aws.String(region),
                  },
          })

   sm := secretsmanager.New(s)

Another to specify the versionStage. This is useful when rotating secrets.

   output, err := sm.GetSecretValue(&secretsmanager.GetSecretValueInput{
        SecretId: &sn,
        VersionStage: aws.String(secretVersion),
        })

I added some help text to potential error messages.

    if err != nil {
        if aerr, ok := err.(awserr.Error); ok {
            switch aerr.Code() {
            case secretsmanager.ErrCodeDecryptionFailure:
                // Secrets Manager can't decrypt the protected secret text using the provided KMS key.
                fmt.Println(aerr.Error())
                fmt.Println("Secrets Manager could not decrypt the secret.")
            case secretsmanager.ErrCodeInternalServiceError:
                // An error occurred on the server side.
                fmt.Println( aerr.Error())
                fmt.Println("Server side error. Possible zombie apocalypse?")
            case secretsmanager.ErrCodeInvalidParameterException:
                // You provided an invalid value for a parameter.
                fmt.Println( aerr.Error())
                fmt.Println("Invalid parameter. Check inputs.")
            case secretsmanager.ErrCodeInvalidRequestException:
                // You provided a parameter value that is not valid for the current state of the resource.
                fmt.Println( aerr.Error())
            case secretsmanager.ErrCodeResourceNotFoundException:
            // We can't find the resource that you asked for.
                fmt.Println( aerr.Error())
                fmt.Println("Is your secret name correct?")
            }
        } else {
            // Print the error, cast err to awserr.Error to get the Code and
            // Message from an error.
            fmt.Println(err.Error())
        }
        os.Exit(1)
   }

Finally, just dump the message. OK, so normally we would not print the plaintext secret as you could do this from the AWS CLI but this is to show that it's available now.

   fmt.Println(*output.SecretString)
}


References:

Kafka Reader Example

This is an example of reading from an Amazon MSK Kafka cluster. It is mostly boilerplate code except for the TLS requirement.

Example code is available from github.

I started with the Segmentio Kafka libraries (github.com/segmentio/kafka-go) which were the easiest to consume. It requires Go version 1.12 or later for the version as of this writing. Main additions were adding TOML config file capability and some command line flags, but is otherwise just implementing their examples.

The import section seems rather long, but I do like the fact that the Go compiler warns if there are unused imports so I do know that every thing I pull in is used.

package main

import (
    "fmt"
    "./helpers"
    "io/ioutil"
    "context"
    "crypto/tls"
    "crypto/x509"
    "flag"
    "os"
    "os/signal"
    "strings"
    "syscall"
    "time"
    kafka "github.com/segmentio/kafka-go"
)



The error handling is from Go By Example - Reading Files. I was in the habit of throwing away errors until I spent an hour wakling down an incorrect troubleshooting path because I ignored the very clear error messages.

func check(e error) {
    if e != nil {
        panic(e)
    }
}


For most purposes, using the system root CAs are probably fine. In this case, I was testing with self-signed certs on my own Kafka cluster so needed a way to override some defaults. It's a TODO to update this to work properly. Further down I actually ignore the cert anyway.

func main () {
//  TLS Configuration
    var infile []byte
    var rootPEM string
    infile, _ = ioutil.ReadFile("certs/rootPEM")
    rootPEM = string(infile)
    fmt.Println(rootPEM)
    roots := x509.NewCertPool()

    ok := roots.AppendCertsFromPEM([]byte(rootPEM))
    if !ok {
        panic("failed to parse root certificate")
    }

The helpers/helpers.go sets up command line parsing and reads a TOML configuration file. It was cleaner to read by dropping it into a library. It just sets a default which can be over-ridden by the --config flag.

//  Kafka Configuration
    var configFile string
    flag.StringVar(&configFile, "config", "config.toml", "configuration file")
    flag.Parse()
    conf := helpers.GetConfig(configFile)
    fmt.Printf("Topic:  %s\n", conf.KafkaConfig.TopicName)

    signals := make(chan os.Signal, 1)
    signal.Notify(signals, syscall.SIGINT, syscall.SIGKILL)

Here I parse the config file


//  Setup the client
    bootstrapServers := strings.Split(conf.KafkaConfig.BootstrapServers, ",")
    topic := conf.KafkaConfig.TopicName


This is from the Segmentio documetation. I drop in the root CAs from a file and cheat by ignoring the verification. Don't do this in production.

//  TLS configuration
    dialer := &kafka.Dialer{
        Timeout:   10 * time.Second,
        DualStack: true,
        TLS: &tls.Config{
                RootCAs: roots,
                InsecureSkipVerify : true},
        }


And once we configure the dialer, we can specify that in the connection.

    r := kafka.NewReader(kafka.ReaderConfig{
        Brokers:   bootstrapServers,
        Topic:     topic,
        Partition: 0,
        MinBytes:  10, // 10KB
        MaxBytes:  10, // 10MB
        Dialer: dialer,
    })


Finally, just loop through and read the messages. You can specify offsets and partitions as part of the library, but this was just a quick test.

    for {
        m, err := r.ReadMessage(context.Background())
        if err != nil {
            break
        }
        fmt.Printf("message at offset %d: %s = %s\n", m.Offset, string(m.Key), string(m.Value))
    }

    r.Close()
}

The helpers/helpers.go library reads in the configuration file and parses it.

package helpers

import (
  "fmt"
  toml "github.com/BurntSushi/toml"
)

type tomlConfig struct {
        Title string
        KafkaConfig kafkaConfig `toml:"kafka"`
}

type kafkaConfig struct {
       TopicName string
       BootstrapServers string
       AutoOffsetReset string
       EnableAutoCommit bool
       ConsumerTimeoutMs int
       GroupID string
}

func GetConfig(configFile string) *tomlConfig {
    config := new(tomlConfig)
    if _, err := toml.DecodeFile(configFile, &config); err != nil {
         fmt.Println(err)
         return config
    }
    return config
}


Finally, the config.toml default file:

Title = "Kafka Configuration"
[kafka]
TopicName = "KafkaTest001"
BootstrapServers = "192.168.16.12:9094,192.168.16.13:9094"
AutoOffsetReset = "earliest"
EnableAutoCommit = true
ConsumerTimeoutMs = 2000
api_version = "1.1.0"
GroupID = "digitalhermit"