Golang Experiments

These are just some miscellaneous code examples as I learn Go. They are almost entirely code pieced together from examples on the web.

Querying Active Directory

Did some more experimentation with Golang. This time, I've converting from a mashup of Python and Powershell code brought about by a need to query Active Directory. Unfortunately, the Python AD libraries only worked on Windows and they were not actively maintained. The Powershell script was required because apparently only PS could make that particular AD query directly (not sure why, still figuring it out). Running the same query in Python gave permission errors. Then, after changing laptops, the Powershell scripts stopped working because of some interaction with security software. <sigh>

The Python code called the Powershell script to generate a full list of all users. Using this list, it iterated through the SAMAccountName field and individually queried Active Directory for each user. Yes, it was a kludge to get something going but ultimately was too slow. There were multiple reasons for the kludge, foremost being that Active Directory was limiting the number of results and this would not be changed. Yes, there are paging options which I explored later (https://medium.com/@alpolishchuk/pagination-of-ldap-search-results-with-python-ldap-845de60b90d2). The other constraint was that I would do the development but others would be running the code. Because of various restrictions on corporate laptops, Powershell was not viable. Python required some setup that the users would find onerous. Maybe Go was the answer? So, during the first day of my vacation I gave it a shot.

Here's the gist of the Python code:

Python

domains = ['it.disciplinux.com', 'us.disciplinux.com', 'au.disciplinux.com']

query_fields = [
    "accountExpires", "sAMAccountName", "description", "displayName", "title", "mail"]

for domain in domains:
    spreadsheet_df = pd.DataFrame(columns=query_fields)
    domain_mask = query_data['CanonicalName'].str.match(domain, na=False)
    subset_data = query_data[domain_mask]

    for samAccountName in subset_data['sAMAccountName']:
        samAccountName = str(samAccountName)
        query_string = "objectClass = '*' and CN='" + samAccountName + "'"
        # Convert 'x.y.z' to 'DC=x, DC=y, DC=z'
        base_dn = 'DC=' + ', DC='.join(domain.split(sep='.'))
        try:
            q.execute_query(attributes=query_fields, where_clause=query_string, base_dn=base_dn)
            data = q.get_results()
            for row in data:
                line = parse_query(row)
                newline = pd.DataFrame([line], columns=query_fields)
                try:
                    spreadsheet_df = spreadsheet_df.append([newline], ignore_index=False)
                except:
                    print("Error occurred on: ", line)

        except Exception as e:
            # Handle exception


Go

For the Golang version, I followed a similar approach except connected via LDAPS with a paged search. To setup this up, I pulled in packages to read a configuration file passed as a command line parameter, a dataframe package (more on this), and the LDAP package.

import (
    "fmt"
    "log"
    "flag"
    "crypto/tls"
    "strconv"
    toml "github.com/BurntSushi/toml"
    ldap "github.com/go-ldap/ldap"
    df "github.com/rocketlaunchr/dataframe-go"
)


The TOML library is straightforward to get going:

type tomlConfig struct {
        Title string
        AuthConfig authConfig `toml:"auth"`
        LdapConfig ldapConfig `toml:"ldap"`
}

type authConfig struct {
       Username string
       Password string
       Filter string
}

type ldapConfig struct {
       Filter string
       Server string
       Port int
       BaseDN string
}

func GetConfig(configFile string) *tomlConfig {
    config := new(tomlConfig)
    if _, err := toml.DecodeFile(configFile, &config); err != nil {
         fmt.Println(err)
         return config
    }
    return config
}


[...]
    //  Read the configuration file for setup
    var configFile string
    flag.StringVar(&configFile, "config", "auth.toml", "configuration file")
    flag.Parse()
    conf := GetConfig(configFile)
    username := conf.AuthConfig.Username
    password := conf.AuthConfig.Password
    filter := conf.LdapConfig.Filter
    server := conf.LdapConfig.Server
    searchbase := conf.LdapConfig.SearchBase
    port := conf.LdapConfig.Port
[...]


This allows reading a TOML file such as:

Title = Report Configuration File

[auth]
Username="jamesholden"
Password="RememberTheCant"

[ldap]
Filter="cn=it,cn=disciplinux,cn=com"
Server="win-ldap-004.disciplinux.com"
SearchBase="cn=Users,OU=IT,dc=disciplinux,dc=com"
Port=636


Now that the configuration is read in, we can start querying. We first connect then bind as an authorized user. For some applications (e.g., an authentication routine), you don't need authentication but the querying I need requires it. At the moment I'm skipping verification of the certificate but DON'T DO THIS. The fix is to add the system root certs and the AD cert if available. Check the Kafka example for how to do this.


    var tlsConf *tls.Config

    tlsConf = &tls.Config{InsecureSkipVerify: true} // FIXME
    dialurl := server + ":" + strconv.Itoa(port)

    conn, err := ldap.DialTLS("tcp", dialurl, tlsConf)
    if err != nil {
        fmt.Println("dialURL failure")
        log.Fatal(err)
    }
    defer conn.Close()

    _, err = conn.SimpleBind(&ldap.SimpleBindRequest{
        Username: username,
        Password: password,
    })

    if err != nil {
        fmt.Println("SimpleBind failure")
        log.Fatal(err)
    }


The query code is a cut/paste from the GODoc (https://godoc.org/gopkg.in/ldap.v3/v3#ControlPaging). I use the settings from the TOML file and setup which attributes to pull. The pageSize/pagingControl vars specify the number of entries to return per page. Increasing this can speed overall query, but you may run into other issues. Smaller entries seem to be better at maintaining connectivity. I can only speculate, but I ran into various issues with larger sizes (500+). This is probably exacerbated because I'm connecting over a VPN because of work-at-home because of COVID-19 restrictions.

The t01 - t04 variables are for testing with various dataframe tools. Unfortunately, dataframe support is Go is pretty miserable at the time of this wriing (2020-07-06). More on this at the end.

The code below sets up the paged search then for each page, iterates through the results and appends to a dataframe.

    var pageSize uint32 = 32
    searchBase := SearchBase
    pagingControl := ldap.NewControlPaging(pageSize)
    attributes := []string{
        "accountExpires", "distinguishedName", "sAMAccountName", "description",
    }

    controls := []ldap.Control{pagingControl}

    for {
        request := ldap.NewSearchRequest(searchBase, ldap.ScopeWholeSubtree, ldap.DerefAlways, 
                                         0, 0, false, filter, attributes, controls)
        response, err := conn.Search(request)
        if err != nil {
            log.Fatalf("Failed to execute search request: %s", err.Error())
        }
        // For a page of results, iterate through the reponse and pull the individual entries
        for index, value := range response.Entries {
            t01 := value.GetAttributeValue("accountExpires")
            t02 := value.GetAttributeValue("distinguishedName")
            t03 := value.GetAttributeValue("sAMAccountName")
            t04 := value.GetAttributeValue("description")

            //  Code to append t01-t04 to a dataframe goes here
            df = df.append(t01, t02, t03, t04)
        }

        updatedControl := ldap.FindControl(response.Controls, ldap.ControlTypePaging)
        if ctrl, ok := updatedControl.(*ldap.ControlPaging); ctrl != nil && ok && len(ctrl.Cookie) != 0 {
            pagingControl.SetCookie(ctrl.Cookie)
            continue
        }
        break
    }


Dataframe

Python Pandas really spoiled me. I was hoping for something similar in Go but was dissapointed. I tried two different packages:

The first package (dataframe-go) looked promising. I was able to build a dataframe easily enough. Unfortunately, getting the data into Excel or CSV was not yet working. After some searching I found a comment that this needed to be done by the user.

// Dataframe setup    
s01 := df.NewSeriesString( "accountExpires", nil)
s02 := df.NewSeriesString( "distinguishedName", nil)
s03 := df.NewSeriesString( "sAMAccountName", nil)
s04 := df.NewSeriesString( "description", nil)

df := df.NewDataFrame(s01, s02, s03, s04)

[...]
// Appending line to dataframe    
            df.Append(nil, t01, t02, t03, t04)


The go-gota package also looked promising at first but had not been updated in months. Creating the dataframe was easy enough, but there was no easy way to append to the dataframe. When I have some time, I'll attempt to hack something together but wouldn't be available in time for the project.


Reading From AWS Secrets Manager

This is pretty simplistic but I'm just getting my feet wet with Golang and started from basics.

These are just a couple snippets to retrieve a secret from AWS Secrets Manager. I use the code to hold secrets for Ansible and miscellaneous Makefiles. Though Ansible has a module for Secrets Manager (aws_secret) and the AWS CLI can retrieve secrets, I needed a method that could be easily installed.

For example, in my Makefile I can do something like:

-- Makefile
GRAFANA_ADMIN_PASSWORD := $(shell get_password -name=/dhrl/dev/k8s/grafana)

install-grafana:
        kubectl create namespace grafana
        HELM_HOST=localhost:$(TILLER_PORT)  helm install stable/grafana \
            --name grafana \
            --namespace grafana \
            --set persistence.storageClassName="gp2" \
            --set adminPassword="$(GRAFANA_ADMIN_PASSWORD)" \
[snip]


I started using AWS with Assumed Roles. This was a convenient way of managing my account. I source the below script to set my AssumedRole credentials in my session. Using an assumed role with Secrets Manager makes it relatively easy to manage accounts and not worry about storing passwords.

ROLE='arn:aws:iam::123456789012:role/DIGITALHERMIT'
OUTPUT=$(aws sts assume-role --role-arn $ROLE --role-session-name "RoleArchitect" --profile test )

AWS_ACCESS_KEY_ID=$(echo $OUTPUT|jq '.Credentials.AccessKeyId'|tr -d '"')
AWS_SECRET_ACCESS_KEY=$(echo $OUTPUT|jq '.Credentials.SecretAccessKey'|tr -d '"')
AWS_SESSION_TOKEN=$(echo $OUTPUT|jq '.Credentials.SessionToken'|tr -d '"')

export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN


The code itself is just based on the example that Amazon provides after the secret is created. I added a few lines to allow passing the secret name but it's mainly boilerplate code.

package main
import (
   "github.com/aws/aws-sdk-go/aws"
   "github.com/aws/aws-sdk-go/aws/session"
   "github.com/aws/aws-sdk-go/aws/awserr"
   "github.com/aws/aws-sdk-go/service/secretsmanager"                                                                                                     
   "fmt"                                                                                                                                                                                       
   "flag"                                                                                                                                                                                      
   "os"                                                                                                                                                                                        
)                                                                                                                                                                                              



I added a few lines to accept command line parameters. The flag module makes this easy.

func main() {                                                                                                                                                                                  
   var secretName,secretVersion,region string                                                                                                                                                  
   flag.StringVar(&secretName, "name", "", "Name of secret")                                                                                                                                   
   flag.StringVar(&secretVersion, "version", "AWSCURRENT", "Version Stage (default: AWSCURRENT)")                                                                                              
   flag.StringVar(&region, "region", "us-east-1", "AWS Region (default: us-east-1)")                                                                                                           
   flag.Parse()                                                                                                                                                                                
   if len(secretName) == 0 {                                                                                                                                                                   
       fmt.Println("Error: Secret name required.")                                                                                                                                             
       os.Exit(1)
   }
   sn := secretName


I add a couple lines to pass the region.

   s, err := session.NewSessionWithOptions(session.Options{
          Config: aws.Config{
                    Region: aws.String(region),
                  },
          })

   sm := secretsmanager.New(s)

Another to specify the versionStage. This is useful when rotating secrets.

   output, err := sm.GetSecretValue(&secretsmanager.GetSecretValueInput{
        SecretId: &sn,
        VersionStage: aws.String(secretVersion),
        })

I added some help text to potential error messages.

    if err != nil {
        if aerr, ok := err.(awserr.Error); ok {
            switch aerr.Code() {
            case secretsmanager.ErrCodeDecryptionFailure:
                // Secrets Manager can't decrypt the protected secret text using the provided KMS key.
                fmt.Println(aerr.Error())
                fmt.Println("Secrets Manager could not decrypt the secret.")
            case secretsmanager.ErrCodeInternalServiceError:
                // An error occurred on the server side.
                fmt.Println( aerr.Error())
                fmt.Println("Server side error. Possible zombie apocalypse?")
            case secretsmanager.ErrCodeInvalidParameterException:
                // You provided an invalid value for a parameter.
                fmt.Println( aerr.Error())
                fmt.Println("Invalid parameter. Check inputs.")
            case secretsmanager.ErrCodeInvalidRequestException:
                // You provided a parameter value that is not valid for the current state of the resource.
                fmt.Println( aerr.Error())
            case secretsmanager.ErrCodeResourceNotFoundException:
            // We can't find the resource that you asked for.
                fmt.Println( aerr.Error())
                fmt.Println("Is your secret name correct?")
            }
        } else {
            // Print the error, cast err to awserr.Error to get the Code and
            // Message from an error.
            fmt.Println(err.Error())
        }
        os.Exit(1)
   }

Finally, just dump the message. OK, so normally we would not print the plaintext secret as you could do this from the AWS CLI but this is to show that it's available now.

   fmt.Println(*output.SecretString)
}


References:

Kafka Reader Example

This is an example of reading from an Amazon MSK Kafka cluster. It is mostly boilerplate code except for the TLS requirement.

Example code is available from github.

I started with the Segmentio Kafka libraries (github.com/segmentio/kafka-go) which were the easiest to consume. It requires Go version 1.12 or later for the version as of this writing. Main additions were adding TOML config file capability and some command line flags, but is otherwise just implementing their examples.

The import section seems rather long, but I do like the fact that the Go compiler warns if there are unused imports so I do know that every thing I pull in is used.

package main

import (
    "fmt"
    "./helpers"
    "io/ioutil"
    "context"
    "crypto/tls"
    "crypto/x509"
    "flag"
    "os"
    "os/signal"
    "strings"
    "syscall"
    "time"
    kafka "github.com/segmentio/kafka-go"
)



The error handling is from Go By Example - Reading Files. I was in the habit of throwing away errors until I spent an hour wakling down an incorrect troubleshooting path because I ignored the very clear error messages.

func check(e error) {
    if e != nil {
        panic(e)
    }
}


For most purposes, using the system root CAs are probably fine. In this case, I was testing with self-signed certs on my own Kafka cluster so needed a way to override some defaults. It's a TODO to update this to work properly. Further down I actually ignore the cert anyway.

func main () {
//  TLS Configuration
    var infile []byte
    var rootPEM string
    infile, _ = ioutil.ReadFile("certs/rootPEM")
    rootPEM = string(infile)
    fmt.Println(rootPEM)
    roots := x509.NewCertPool()

    ok := roots.AppendCertsFromPEM([]byte(rootPEM))
    if !ok {
        panic("failed to parse root certificate")
    }

The helpers/helpers.go sets up command line parsing and reads a TOML configuration file. It was cleaner to read by dropping it into a library. It just sets a default which can be over-ridden by the --config flag.

//  Kafka Configuration
    var configFile string
    flag.StringVar(&configFile, "config", "config.toml", "configuration file")
    flag.Parse()
    conf := helpers.GetConfig(configFile)
    fmt.Printf("Topic:  %s\n", conf.KafkaConfig.TopicName)

    signals := make(chan os.Signal, 1)
    signal.Notify(signals, syscall.SIGINT, syscall.SIGKILL)

Here I parse the config file


//  Setup the client
    bootstrapServers := strings.Split(conf.KafkaConfig.BootstrapServers, ",")
    topic := conf.KafkaConfig.TopicName


This is from the Segmentio documetation. I drop in the root CAs from a file and cheat by ignoring the verification. Don't do this in production.

//  TLS configuration
    dialer := &kafka.Dialer{
        Timeout:   10 * time.Second,
        DualStack: true,
        TLS: &tls.Config{
                RootCAs: roots,
                InsecureSkipVerify : true},
        }


And once we configure the dialer, we can specify that in the connection.

    r := kafka.NewReader(kafka.ReaderConfig{
        Brokers:   bootstrapServers,
        Topic:     topic,
        Partition: 0,
        MinBytes:  10, // 10KB
        MaxBytes:  10, // 10MB
        Dialer: dialer,
    })


Finally, just loop through and read the messages. You can specify offsets and partitions as part of the library, but this was just a quick test.

    for {
        m, err := r.ReadMessage(context.Background())
        if err != nil {
            break
        }
        fmt.Printf("message at offset %d: %s = %s\n", m.Offset, string(m.Key), string(m.Value))
    }

    r.Close()
}

The helpers/helpers.go library reads in the configuration file and parses it.

package helpers

import (
  "fmt"
  toml "github.com/BurntSushi/toml"
)

type tomlConfig struct {
        Title string
        KafkaConfig kafkaConfig `toml:"kafka"`
}

type kafkaConfig struct {
       TopicName string
       BootstrapServers string
       AutoOffsetReset string
       EnableAutoCommit bool
       ConsumerTimeoutMs int
       GroupID string
}

func GetConfig(configFile string) *tomlConfig {
    config := new(tomlConfig)
    if _, err := toml.DecodeFile(configFile, &config); err != nil {
         fmt.Println(err)
         return config
    }
    return config
}


Finally, the config.toml default file:

Title = "Kafka Configuration"
[kafka]
TopicName = "KafkaTest001"
BootstrapServers = "192.168.16.12:9094,192.168.16.13:9094"
AutoOffsetReset = "earliest"
EnableAutoCommit = true
ConsumerTimeoutMs = 2000
api_version = "1.1.0"
GroupID = "digitalhermit"