Golang Experiments
These are just some miscellaneous code examples as I learn Go. They are almost entirely code pieced together from examples on the web.
Decoding Windows Timestamps
There's probably a better way than my hackery, so I'm adding this in case a reader can help improve this.
The objective was to decode timeststamps from Microsoft Active Directory. Online posts indicated a couple approaches in C and Java which I translated into Golang code. In the main() loop, LDAP queries return strings so I added string conversion which is less universal and ugly. I will do penance to appropriate code deities for my transgressions.
One site (StackOverflow) indicated that the Active Directory value counts by 100 nanosecond ticks, with an epoch start of 1601-01-01T00:00:00Z. Anyhoo, I'm assuming there's a timezone or the value is in UTC but this not clear from the documentation.
func convertWindowsTime(rawValue string) string {
adtime, _ := strconv.ParseInt(rawValue, 10, 64)
if (adtime == 9223372036854775807) || (adtime == 0) {
return "Not Set"
}
unixtime_int64 := adtime/(10*1000*1000) - 11644473600
unixtime := time.Unix(unixtime_int64, 0)
unixtime_timestring := unixtime.Format("2006-01-02 3:4:5 pm")
return unixtime_timestring
}
Some other date/time fields return a different format. Here I'm just carelessly dropping the 0Z (UTC offset) and parsing using the special date of January 2, 2006 at 3:04:05PM. It does seem to work, but anytime I drop data I worry.
func convertIso8601Date( adtime string ) string {
format := "20060102150405"
loc, _ := time.LoadLocation("UTC")
adtime = strings.ReplaceAll(adtime, ".0Z", "")
unixTime, _ := time.Parse(format, adtime)
unixTime = unixTime.In(loc)
unixTimeString := unixTime.String()
return unixTimeString
}
The code just reeks of inelegance. Not that my code is ever particularly elegant, but this reeks especially badly because it may not work properly. I can *feel* it.
Decoding Active Directory userAccountControl Field
I did some more recent work in Active Directory. To be honest, Python is still my go-to language but I'm getting more comfortable with Go with each new line of code. Most recently, I had to decode the UserAccountControl (UAC) field to generate a report. I found a couple sites that had information on the fields:
My goal was to decode the UAC field into readable text. Taking the decimal values from the Microsoft knowledge base, I dropped them into an integer to string mapping to allow looking up the integer value:
var uacMask = map[uint32]string{
2 : "ACCOUNT_DISABLE",
8 : "HOMEDIR_REQUIRED",
16 : "LOCKOUT",
32 : "PASSWD_NOTREQD",
64 : "PASSWD_CANT_CHANGE",
128 : "ENCRYPTED_TEXT_PASSWORD_ALLOWED",
512 : "NORMAL_ACCOUNT",
2048 : "INTERDOMAIN_TRUST_ACCOUNT",
4096 : "WORKSTATION_TRUST_ACCOUNT",
8192 : "SERVER_TRUST_ACCOUNT",
65536 : "DONT_EXPIRE_PASSWD",
131072 : "MNS_LOGON_ACCOUNT",
262144 : "SMARTCARD_REQUIRED",
524288 : "TRUSTED_FOR_DELEGATION",
1048576 : "NOT_DELEGATED",
2097152 : "USE_DES_KEY_ONLY",
4194304 : "DONT_REQUIRE_PREAUTH",
8388608 : "PASSWORD_EXPIRED",
16777216 : "TRUSTED_TO_AUTHENTICATE_FOR_DELEGATION",
33554432 : "NO_AUTH_DATA_REQUIRED",
67108864 : "PARTIAL_SECRETS_ACCOUNT",
}
Next, I had to step through the UAC and decode. This was made easier using Go's bitwise operators. In this case, I created a bitmask by successively shifting a unit value of 1 from right to left. By ANDing the mask against the UAC I could see if the bit was set. Bit manipulation? In 2020? Luckily, because of my chilling adventures with Arduino, this was not as bad as I had imagined.
func decodeUac( uac uint32 ) []string {
var mask uint32
var decodedUAC []string
decodedUAC = []string {}
for bit := 0; bit < 32; bit++ {
mask = 1 << bit
if (uac&mask == mask){
v, found := uacMask[mask]
if found {
decodedUAC = append(decodedUAC, v)
}
}
}
return decodedUAC
}
And finally calling it from my main()...
func main() {
var uac uint32
var x uint64
// [...] See https://www.digitalhermit.com/linux/golang-experiments#h.p_WZuij2rRRjvh
userAccountControl = value.GetAttributeValue("userAccountControl")
// ... (convert the string value into uint32)
x, _ := strconv.ParseUint(userAccountControl, 10, 64)
uac = uint32(x)
decoded := decodeUac(uac)
fmt.Println(decoded)
}
I'm still relatively new to Go and don't code nearly as often as I should. But, one thing that I absolutely enjoyed was the ease with which I could write and debug my code in Linux and when everything was working, do a git commit, checkout out on my Windows laptop and rebuild. And it worked flawlessly. Much as I love Python, it was not anywhere as straightforward.
A caveat: I'm very much a hack, not even a proper hacker. There are likely better ways to do this but the fact that I could put together working code with relative ease was satisfying. Please let me what can be improved!
Querying Active Directory
Did some more experimentation with Golang. This time, I've converting from a mashup of Python and Powershell code brought about by a need to query Active Directory. Unfortunately, the Python AD libraries only worked on Windows and they were not actively maintained. The Powershell script was required because apparently only PS could make that particular AD query directly (not sure why, still figuring it out). Running the same query in Python gave permission errors. Then, after changing laptops, the Powershell scripts stopped working because of some interaction with security software. <sigh>
The Python code called the Powershell script to generate a full list of all users. Using this list, it iterated through the SAMAccountName field and individually queried Active Directory for each user. Yes, it was a kludge to get something going but ultimately was too slow. There were multiple reasons for the kludge, foremost being that Active Directory was limiting the number of results and this would not be changed. Yes, there are paging options which I explored later (https://medium.com/@alpolishchuk/pagination-of-ldap-search-results-with-python-ldap-845de60b90d2). The other constraint was that I would do the development but others would be running the code. Because of various restrictions on corporate laptops, Powershell was not viable. Python required some setup that the users would find onerous. Maybe Go was the answer? So, during the first day of my vacation I gave it a shot.
Here's the gist of the Python code:
Python
domains = ['it.disciplinux.com', 'us.disciplinux.com', 'au.disciplinux.com']
query_fields = [
"accountExpires", "sAMAccountName", "description", "displayName", "title", "mail"]
for domain in domains:
spreadsheet_df = pd.DataFrame(columns=query_fields)
domain_mask = query_data['CanonicalName'].str.match(domain, na=False)
subset_data = query_data[domain_mask]
for samAccountName in subset_data['sAMAccountName']:
samAccountName = str(samAccountName)
query_string = "objectClass = '*' and CN='" + samAccountName + "'"
# Convert 'x.y.z' to 'DC=x, DC=y, DC=z'
base_dn = 'DC=' + ', DC='.join(domain.split(sep='.'))
try:
q.execute_query(attributes=query_fields, where_clause=query_string, base_dn=base_dn)
data = q.get_results()
for row in data:
line = parse_query(row)
newline = pd.DataFrame([line], columns=query_fields)
try:
spreadsheet_df = spreadsheet_df.append([newline], ignore_index=False)
except:
print("Error occurred on: ", line)
except Exception as e:
# Handle exception
Go
For the Golang version, I followed a similar approach except connected via LDAPS with a paged search. To setup this up, I pulled in packages to read a configuration file passed as a command line parameter, a dataframe package (more on this), and the LDAP package.
import (
"fmt"
"log"
"flag"
"crypto/tls"
"strconv"
toml "github.com/BurntSushi/toml"
ldap "github.com/go-ldap/ldap"
df "github.com/rocketlaunchr/dataframe-go"
)
The TOML library is straightforward to get going:
type tomlConfig struct {
Title string
AuthConfig authConfig `toml:"auth"`
LdapConfig ldapConfig `toml:"ldap"`
}
type authConfig struct {
Username string
Password string
Filter string
}
type ldapConfig struct {
Filter string
Server string
Port int
BaseDN string
}
func GetConfig(configFile string) *tomlConfig {
config := new(tomlConfig)
if _, err := toml.DecodeFile(configFile, &config); err != nil {
fmt.Println(err)
return config
}
return config
}
[...]
// Read the configuration file for setup
var configFile string
flag.StringVar(&configFile, "config", "auth.toml", "configuration file")
flag.Parse()
conf := GetConfig(configFile)
username := conf.AuthConfig.Username
password := conf.AuthConfig.Password
filter := conf.LdapConfig.Filter
server := conf.LdapConfig.Server
searchbase := conf.LdapConfig.SearchBase
port := conf.LdapConfig.Port
[...]
This allows reading a TOML file such as:
Title = Report Configuration File
[auth]
Username="jamesholden"
Password="RememberTheCant"
[ldap]
Filter="cn=it,cn=disciplinux,cn=com"
Server="win-ldap-004.disciplinux.com"
SearchBase="cn=Users,OU=IT,dc=disciplinux,dc=com"
Port=636
Now that the configuration is read in, we can start querying. We first connect then bind as an authorized user. For some applications (e.g., an authentication routine), you don't need authentication but the querying I need requires it. At the moment I'm skipping verification of the certificate but DON'T DO THIS. The fix is to add the system root certs and the AD cert if available. Check the Kafka example for how to do this.
var tlsConf *tls.Config
tlsConf = &tls.Config{InsecureSkipVerify: true} // FIXME
dialurl := server + ":" + strconv.Itoa(port)
conn, err := ldap.DialTLS("tcp", dialurl, tlsConf)
if err != nil {
fmt.Println("dialURL failure")
log.Fatal(err)
}
defer conn.Close()
_, err = conn.SimpleBind(&ldap.SimpleBindRequest{
Username: username,
Password: password,
})
if err != nil {
fmt.Println("SimpleBind failure")
log.Fatal(err)
}
The query code is a cut/paste from the GODoc (https://godoc.org/gopkg.in/ldap.v3/v3#ControlPaging). I use the settings from the TOML file and setup which attributes to pull. The pageSize/pagingControl vars specify the number of entries to return per page. Increasing this can speed overall query, but you may run into other issues. Smaller entries seem to be better at maintaining connectivity. I can only speculate, but I ran into various issues with larger sizes (500+). This is probably exacerbated because I'm connecting over a VPN because of work-at-home because of COVID-19 restrictions.
The t01 - t04 variables are for testing with various dataframe tools. Unfortunately, dataframe support is Go is pretty miserable at the time of this wriing (2020-07-06). More on this at the end.
The code below sets up the paged search then for each page, iterates through the results and appends to a dataframe.
var pageSize uint32 = 32
searchBase := SearchBase
pagingControl := ldap.NewControlPaging(pageSize)
attributes := []string{
"accountExpires", "distinguishedName", "sAMAccountName", "description",
}
controls := []ldap.Control{pagingControl}
for {
request := ldap.NewSearchRequest(searchBase, ldap.ScopeWholeSubtree, ldap.DerefAlways,
0, 0, false, filter, attributes, controls)
response, err := conn.Search(request)
if err != nil {
log.Fatalf("Failed to execute search request: %s", err.Error())
}
// For a page of results, iterate through the reponse and pull the individual entries
for index, value := range response.Entries {
t01 := value.GetAttributeValue("accountExpires")
t02 := value.GetAttributeValue("distinguishedName")
t03 := value.GetAttributeValue("sAMAccountName")
t04 := value.GetAttributeValue("description")
// Code to append t01-t04 to a dataframe goes here
df = df.append(t01, t02, t03, t04)
}
updatedControl := ldap.FindControl(response.Controls, ldap.ControlTypePaging)
if ctrl, ok := updatedControl.(*ldap.ControlPaging); ctrl != nil && ok && len(ctrl.Cookie) != 0 {
pagingControl.SetCookie(ctrl.Cookie)
continue
}
break
}
Dataframe
Python Pandas really spoiled me. I was hoping for something similar in Go but was dissapointed. I tried two different packages:
The first package (dataframe-go) looked promising. I was able to build a dataframe easily enough. Unfortunately, getting the data into Excel or CSV was not yet working. After some searching I found a comment that this needed to be done by the user. I am re-evaluating the datafrme-go package and will update (2020-10-01).
// Dataframe setup
s01 := df.NewSeriesString( "accountExpires", nil)
s02 := df.NewSeriesString( "distinguishedName", nil)
s03 := df.NewSeriesString( "sAMAccountName", nil)
s04 := df.NewSeriesString( "description", nil)
df := df.NewDataFrame(s01, s02, s03, s04)
[...]
// Appending line to dataframe
df.Append(nil, t01, t02, t03, t04)
The go-gota package also looked promising at first but had not been updated in months. Creating the dataframe was easy enough, but there was no easy way to append to the dataframe. When I have some time, I'll attempt to hack something together but wouldn't be available in time for the project.
Structs and Slices
With Dataframe support not quite there, I went back to native Go support for structs and slices. SImilar to other languages, a Go struct is a collection of fields. These structs can be assembled into a slice, which is similar to an array. In Go, arrays are statically sized whereas slices can be expanded or reduced. In the code below, I create a struct to hold the Canonical Name (cn) and Description from the LDAP query.
type person struct {
expireDate string
distinguishedName string
}
var dataSlice []person
header := person{"Account Expires", "DN"}
dataSlice = append(dataSlice, header)
As I assemble my data, I append each line to dataSlice, which is a slice consisting of person objects.
// for loop
newline = person{t1, t2}
dataSlice = append(dataSlice, newline)
Once assembled, I can iterate through the dataSlice to write to a file or print to console.
for index, line := range dataSlice {
fmt.Printf("Expire: %s D: %s\n", line.expireDate, line.distinguishedName)
excel.SetCellValue(sheetName, "A" + strconv.Itoa(index), line.expireDate)
excel.SetCellValue(sheetName, "B" + strconv.Itoa(index), line.distinguishedName)
}
excel.SaveAs("outfile.xlsx")
Reading From AWS Secrets Manager
This is pretty simplistic but I'm just getting my feet wet with Golang and started from basics.
These are just a couple snippets to retrieve a secret from AWS Secrets Manager. I use the code to hold secrets for Ansible and miscellaneous Makefiles. Though Ansible has a module for Secrets Manager (aws_secret) and the AWS CLI can retrieve secrets, I needed a method that could be easily installed.
For example, in my Makefile I can do something like:
-- Makefile
GRAFANA_ADMIN_PASSWORD := $(shell get_password -name=/dhrl/dev/k8s/grafana)
install-grafana:
kubectl create namespace grafana
HELM_HOST=localhost:$(TILLER_PORT) helm install stable/grafana \
--name grafana \
--namespace grafana \
--set persistence.storageClassName="gp2" \
--set adminPassword="$(GRAFANA_ADMIN_PASSWORD)" \
[snip]
I started using AWS with Assumed Roles. This was a convenient way of managing my account. I source the below script to set my AssumedRole credentials in my session. Using an assumed role with Secrets Manager makes it relatively easy to manage accounts and not worry about storing passwords.
ROLE='arn:aws:iam::123456789012:role/DIGITALHERMIT'
OUTPUT=$(aws sts assume-role --role-arn $ROLE --role-session-name "RoleArchitect" --profile test )
AWS_ACCESS_KEY_ID=$(echo $OUTPUT|jq '.Credentials.AccessKeyId'|tr -d '"')
AWS_SECRET_ACCESS_KEY=$(echo $OUTPUT|jq '.Credentials.SecretAccessKey'|tr -d '"')
AWS_SESSION_TOKEN=$(echo $OUTPUT|jq '.Credentials.SessionToken'|tr -d '"')
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
The code itself is just based on the example that Amazon provides after the secret is created. I added a few lines to allow passing the secret name but it's mainly boilerplate code.
package main
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/service/secretsmanager"
"fmt"
"flag"
"os"
)
I added a few lines to accept command line parameters. The flag module makes this easy.
func main() {
var secretName,secretVersion,region string
flag.StringVar(&secretName, "name", "", "Name of secret")
flag.StringVar(&secretVersion, "version", "AWSCURRENT", "Version Stage (default: AWSCURRENT)")
flag.StringVar(®ion, "region", "us-east-1", "AWS Region (default: us-east-1)")
flag.Parse()
if len(secretName) == 0 {
fmt.Println("Error: Secret name required.")
os.Exit(1)
}
sn := secretName
I add a couple lines to pass the region.
s, err := session.NewSessionWithOptions(session.Options{
Config: aws.Config{
Region: aws.String(region),
},
})
sm := secretsmanager.New(s)
Another to specify the versionStage. This is useful when rotating secrets.
output, err := sm.GetSecretValue(&secretsmanager.GetSecretValueInput{
SecretId: &sn,
VersionStage: aws.String(secretVersion),
})
I added some help text to potential error messages.
if err != nil {
if aerr, ok := err.(awserr.Error); ok {
switch aerr.Code() {
case secretsmanager.ErrCodeDecryptionFailure:
// Secrets Manager can't decrypt the protected secret text using the provided KMS key.
fmt.Println(aerr.Error())
fmt.Println("Secrets Manager could not decrypt the secret.")
case secretsmanager.ErrCodeInternalServiceError:
// An error occurred on the server side.
fmt.Println( aerr.Error())
fmt.Println("Server side error. Possible zombie apocalypse?")
case secretsmanager.ErrCodeInvalidParameterException:
// You provided an invalid value for a parameter.
fmt.Println( aerr.Error())
fmt.Println("Invalid parameter. Check inputs.")
case secretsmanager.ErrCodeInvalidRequestException:
// You provided a parameter value that is not valid for the current state of the resource.
fmt.Println( aerr.Error())
case secretsmanager.ErrCodeResourceNotFoundException:
// We can't find the resource that you asked for.
fmt.Println( aerr.Error())
fmt.Println("Is your secret name correct?")
}
} else {
// Print the error, cast err to awserr.Error to get the Code and
// Message from an error.
fmt.Println(err.Error())
}
os.Exit(1)
}
Finally, just dump the message. OK, so normally we would not print the plaintext secret as you could do this from the AWS CLI but this is to show that it's available now.
fmt.Println(*output.SecretString)
}
References:
Kafka Reader Example
This is an example of reading from an Amazon MSK Kafka cluster. It is mostly boilerplate code except for the TLS requirement.
Example code is available from github.
I started with the Segmentio Kafka libraries (github.com/segmentio/kafka-go) which were the easiest to consume. It requires Go version 1.12 or later for the version as of this writing. Main additions were adding TOML config file capability and some command line flags, but is otherwise just implementing their examples.
The import section seems rather long, but I do like the fact that the Go compiler warns if there are unused imports so I do know that every thing I pull in is used.
package main
import (
"fmt"
"./helpers"
"io/ioutil"
"context"
"crypto/tls"
"crypto/x509"
"flag"
"os"
"os/signal"
"strings"
"syscall"
"time"
kafka "github.com/segmentio/kafka-go"
)
The error handling is from Go By Example - Reading Files. I was in the habit of throwing away errors until I spent an hour wakling down an incorrect troubleshooting path because I ignored the very clear error messages.
func check(e error) {
if e != nil {
panic(e)
}
}
For most purposes, using the system root CAs are probably fine. In this case, I was testing with self-signed certs on my own Kafka cluster so needed a way to override some defaults. It's a TODO to update this to work properly. Further down I actually ignore the cert anyway.
func main () {
// TLS Configuration
var infile []byte
var rootPEM string
infile, _ = ioutil.ReadFile("certs/rootPEM")
rootPEM = string(infile)
fmt.Println(rootPEM)
roots := x509.NewCertPool()
ok := roots.AppendCertsFromPEM([]byte(rootPEM))
if !ok {
panic("failed to parse root certificate")
}
The helpers/helpers.go sets up command line parsing and reads a TOML configuration file. It was cleaner to read by dropping it into a library. It just sets a default which can be over-ridden by the --config flag.
// Kafka Configuration
var configFile string
flag.StringVar(&configFile, "config", "config.toml", "configuration file")
flag.Parse()
conf := helpers.GetConfig(configFile)
fmt.Printf("Topic: %s\n", conf.KafkaConfig.TopicName)
signals := make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGINT, syscall.SIGKILL)
Here I parse the config file
// Setup the client
bootstrapServers := strings.Split(conf.KafkaConfig.BootstrapServers, ",")
topic := conf.KafkaConfig.TopicName
This is from the Segmentio documetation. I drop in the root CAs from a file and cheat by ignoring the verification. Don't do this in production.
// TLS configuration
dialer := &kafka.Dialer{
Timeout: 10 * time.Second,
DualStack: true,
TLS: &tls.Config{
RootCAs: roots,
InsecureSkipVerify : true},
}
And once we configure the dialer, we can specify that in the connection.
r := kafka.NewReader(kafka.ReaderConfig{
Brokers: bootstrapServers,
Topic: topic,
Partition: 0,
MinBytes: 10, // 10KB
MaxBytes: 10, // 10MB
Dialer: dialer,
})
Finally, just loop through and read the messages. You can specify offsets and partitions as part of the library, but this was just a quick test.
for {
m, err := r.ReadMessage(context.Background())
if err != nil {
break
}
fmt.Printf("message at offset %d: %s = %s\n", m.Offset, string(m.Key), string(m.Value))
}
r.Close()
}
The helpers/helpers.go library reads in the configuration file and parses it.
package helpers
import (
"fmt"
toml "github.com/BurntSushi/toml"
)
type tomlConfig struct {
Title string
KafkaConfig kafkaConfig `toml:"kafka"`
}
type kafkaConfig struct {
TopicName string
BootstrapServers string
AutoOffsetReset string
EnableAutoCommit bool
ConsumerTimeoutMs int
GroupID string
}
func GetConfig(configFile string) *tomlConfig {
config := new(tomlConfig)
if _, err := toml.DecodeFile(configFile, &config); err != nil {
fmt.Println(err)
return config
}
return config
}
Finally, the config.toml default file:
Title = "Kafka Configuration"
[kafka]
TopicName = "KafkaTest001"
BootstrapServers = "192.168.16.12:9094,192.168.16.13:9094"
AutoOffsetReset = "earliest"
EnableAutoCommit = true
ConsumerTimeoutMs = 2000
api_version = "1.1.0"
GroupID = "digitalhermit"