Decrypt Rails Credentials in Go
PostedWhile EveryPost is written in Rails, in building an external API, I settled on doing it in Go. One of the larger challenges was deciding how to handle shared config (e.g. database keys, vendor credentials, etc.)
EveryPost has always used encrypted credentials to store these types of secrets, and it turned out to be a great way of sharing secrets across apps.
Let’s break down how to read these encrypted credentials files. I’m assuming you have some familiarity with the Go language, and have included all the code to accomplish this below.
The Design
At a high level, we create a CredentialsReader and have it implement io.Reader. This allows our credentials to be used transparently by libraries like Viper.
var _ io.Reader = &CredentialsReader{}
// CredentialsReader is an io.Reader which understands the encrypted Rails
// credentials file format, and given a decryption key, returns its unencrypted
// contents.
type CredentialsReader struct {
KeyPath string // path to decryption key (e.g. 'master.key')
Path string // path of encrypted file (e.g. 'credentials.yml.enc')
decrypted *bytes.Buffer
}
// Read implements io.Reader interface.
//
// This method will return an error if the associated KeyPath or Path are not valid,
// or if the encrypted file can not be parsed or deserialized.
//
// # See Also
//
// - https://docs.ruby-lang.org/en/2.1.0/marshal_rdoc.html for a description of how
// the encrypted file format is deserialized into YAML.
// - https://pkg.go.dev/github.com/spf13/viper#ReadConfig to see how the
// YAML data is handled by the server.
func (cr *CredentialsReader) Read(p []byte) (int, error) {
if cr.decrypted == nil {
var (
key []byte
encryptedData []byte
cipherText []byte
iv []byte
err error
)
if encryptedData, err = os.ReadFile(cr.Path); err != nil {
return 0, err
}
if key, err = cr.readKey(cr.KeyPath); err != nil {
return 0, err
}
if cipherText, iv, err = cr.parseData(encryptedData); err != nil {
return 0, err
}
res := cr.decrypt(key, cipherText, iv)
// Extract payload from Rails binary format
if res, err = deserialize(res); err != nil {
return 0, err
}
cr.decrypted = bytes.NewBuffer(res)
}
return cr.decrypted.Read(p)
}
readKey()
This is the function to load a decryption key from the environment. If it is not defined in the environment, we fall back to a file-based key. Finally if there is no file-based key, we return an error.
const (
// name used to read master key from execution environment
railsEnvKey = "RAILS_MASTER_KEY"
)
func (cr *CredentialsReader) readKey(keyPath string) ([]byte, error) {
if envKey := os.Getenv(railsEnvKey); len(envKey) > 0 {
return hex.DecodeString(envKey)
}
key, err := os.ReadFile(keyPath)
if err != nil {
return nil, err
}
key = bytes.TrimSpace(key)
if key, err = hex.DecodeString(string(key)); err != nil {
return nil, err
}
return key, nil
}
Let’s take a peek at how this ‘KeyPath’ field is set.
// main.go
// ...
if os.Getenv("RAILS_ENV") == "production" {
// Production (reads from environment)
credentialsReader = &config.CredentialsReader{
Path: "production.yml.enc",
}
} else {
// Local development
credentialsReader = &config.CredentialsReader{
Path: "../config/credentials.yml.enc",
KeyPath: "../config/master.key",
}
}
As you see here, KeyPath is only used for local development (i.e. this file would not exist in prod as the deployment injects this key at runtime).
parseData()
The format of the encrypted credentials contains three segments separated by two dashes. This is generated by Rails as part of persisting the encrypted file.
- Segment:0 The cipher text
- Segment:1 The initialization vector
- Segment:2 Auth data (this is appended to the ciphertext in order to decrypt)
func (cr *CredentialsReader) parseData(encryptedData []byte) (cipherText []byte, iv []byte, err error) {
var (
authData []byte
)
segments := strings.Split(string(encryptedData), "--")
if len(segments) < 3 {
return nil, nil, errors.New("invalid number of data segments: check data file")
}
if cipherText, err = base64.StdEncoding.DecodeString(segments[0]); err != nil {
return nil, nil, err
}
if iv, err = base64.StdEncoding.DecodeString(segments[1]); err != nil {
return nil, nil, err
}
if authData, err = base64.StdEncoding.DecodeString(segments[2]); err != nil {
return nil, nil, err
}
return bytes.Join([][]byte{cipherText, authData}, []byte{}), iv, nil
}
decrypt() and deserialize()
Finally this code performs more heavy lifting. To better understand the code, consult the Go ‘cipher’ package. There are a few important details to call out however:
- It can “panic” at multiple points here, halting execution.1
- We are decoding a marshaled Ruby object here purely in Go. Oh my..
func (cr *CredentialsReader) decrypt(key []byte, ciphertext []byte, iv []byte) []byte {
block, err := aes.NewCipher(key)
if err != nil {
panic(err.Error())
}
aesgcm, err := cipher.NewGCM(block)
if err != nil {
panic(err.Error())
}
plaintext, err := aesgcm.Open(nil, iv, ciphertext, nil)
if err != nil {
panic(err.Error())
}
return plaintext
}
func deserialize(buf []byte) ([]byte, error) {
const (
ASCII8bit = 0x22
OFFSET_2_BYTES = 0x02
OFFSET_4_BYTES = 0x03
OFFSET_5_BYTES = 0x04
)
var (
header = buf[:2]
objType = buf[2]
lenIndicator = buf[3]
)
if !bytes.Equal(header, []byte{0x04, 0x08}) {
return nil, errors.New("invalid serialization header")
}
if objType != ASCII8bit {
return nil, errors.New("data does not encode an ASCII-8BIT value")
}
/*
see https://docs.ruby-lang.org/en/2.1.0/marshal_rdoc.html
*/
switch lenIndicator {
case OFFSET_2_BYTES:
// Following two bytes store length
length := binary.LittleEndian.Uint16(buf[4:6])
return buf[6:(length + 6)], nil
case OFFSET_4_BYTES:
// Following four bytes store length
length := binary.LittleEndian.Uint16(buf[4:7])
return buf[7:(length + 7)], nil
case OFFSET_5_BYTES:
// Following five bytes store length
length := binary.LittleEndian.Uint16(buf[4:8])
return buf[8:(length + 8)], nil
case 0x01, 0xff, 0xfe, 0xfd, 0xfc:
return nil, errors.New("unsupported string length")
default:
// In this case, length indicator defined as "object length + 5"
// so we reduce it by one to get the byte array offset
return buf[4 : lenIndicator-1], nil
}
}
On Deserialization
I mentioned above that this Go code deserializes an a serialised Ruby object. The code above follows the specs for marshalling in Ruby. These specs spell out the format and fields embedded in a marshalled object and is the reason we need to work a bit with lengths and offsets. Amazingly it all just works. Neat, right?
Final note on Testing..
I can not promise future compatibility, but I can state this code has been in production for eight months and has not surfaced any issues in that time.
While I am not including it here, I’ll summarize my testing approach for the code above so you can replicate it on your end. I found the following to work very well:
- Generate a new, dummy credentials file and key file from Rails. Include these with the Go project.
- Read from these files in your tests, to assert values can be read appropriately.
Here’s a snippet of the tests showing what I mean. I place the test key in a comment here only to show what that ‘master.key’ file really contains.
Not shown here are tests asserting that the keys/values embedded in the test credentials can be read, as well as testing that the master key can be read from the environment.
/*
Key:
testdata/master.key
abcdef012374bad3a98846835cd6884f
File:
testdata/credentials.yml.enc
Contents:
# generated with rails 7.0.3
key_1: blue
key_2: green
section:
key_3: red
*/
func TestCredentialsReader(t *testing.T) {
testCases := []struct {
name string
setup func()
keyFile string
encryptedFile string
expectedFile string
expectErr bool
}{
{
name: "is successful",
keyFile: "./testdata/master.key",
encryptedFile: "./testdata/credentials.yml.enc",
expectedFile: "./testdata/credentials.plain.yml",
},
// ...
}
// ...
}
Hopefully, you found this article useful!
This code should be run on server initialization, so it’s a deliberate decision to fail hard and fast (i.e.) as unlikely as it is to happen, there is no fallback for this application if credentials can not be read due to encryption error). ↩︎