Deciphering the FileMaker Server keystore

Introduction

While checking out how the FileMaker Pro to Server upload feature worked, I noticed that credentials were encrypted using a RSA public key before being sent to the server.

I looked into FileMaker Server’s installation directory and found that the private key was stored in the CStore/keystore file – at least I assumed it, as the file stores keys and values, where the keys are base64-encoded strings, one of them being bWFjaGluZVByaXZhdGVLZXk=, or machinePrivateKey.

While the values are base64-encoded just like the keys, decoding them just reveals ciphertext, so you cannot really tell what you are looking at.

Next to the machinePrivateKey I saw several encoded file UUIDs which values represented the EAR (encryption at rest) “encryption password” for files hosted by the server. This became obvious when new entries appeared in the keystore after uploading encrypted files and choosing to save the password (to be able to auto-open the files after a reboot).

At this point I decided to spend some time to better understand how the keystore actually works, how secrets are protected and if I could replicate the encryption/decryption process to decipher the stored values.

In this post I want to share what I figured out, but more so show my approach (including failed attempts), so that you can hopefully benefit from it when taking on similar problems. I will talk about learning from observing the data, doing memory dumps, attaching debuggers, and a little bit of cryptography.

Some notes before you read on

I generally believe it’s useful to have a better understanding of the inner workings of the systems you’re using on a daliy basis (and maybe even depend on) – to help with troubleshooting, to be able to judge what features to use or to ignore, what risks to accept and to make sense of (sometimes odd) software behaviors. In my opinion most of these things are better publicly documented than privately/exclusively shared.

But I also talked to a couple of people and was advised to omit a few details, such that not a full decryption script is made available. In the last part I will thus skip over a few steps to keep it brief and don’t reveal the final part required to recreate the encryption key. I do, however, give pointers on where to look if you would like to try it yourself as an exercise.

If you follow along with this article, please be aware that the actions taken can damage your FMS installation and hosted databases.

Lastly, I’m not a professional reverse engineer, so there might be more straightforward ways of tackling the same problem and mine are not necessarily the best/fastest (in fact I’ve later learned many things that could have led me to the result much more quickly).

If you are still curious, please read on.

First insights by looking at the data

When you want to figure out an unknown encoding or encryption system it’s extremely helpful if you are able to observe how new data (entered by you) is encoded/encrypted.

Let’s have a look what insights can be gained just by observing data.

Opening the keystore file (which is publicly readable by default, by the way), we can identify two things quickly:

RTI2Q0M1RkJGOUEyNDQ1OEE3MzE0QjE0RUIyQkY4RTY=;9acSKigNs3IHuEikQeit1Q==
RTY2RDdBQjQ0NjgwNENCRjg2RjZCMDAxNkUwNzk3MTQ=;b442h3lSslgZ66HAwoSO7+jm1Xcw0O7DSmRlW9bkRrj7EUeRYaj8K6VEsLWSYh8x

First, it seems the ; is the separator. And second, based on the set of characters and padding at the end, the encoding looks like base64.

To verify, we can do echo -n RTY2RDdBQjQ0NjgwNENCRjg2RjZCMDAxNkUwNzk3MTQ= | base64 -d and observe the decoded data. In this case it’s E66D7AB446804CBF86F6B0016E079714, which could be a UUID (16 bytes, in hexadecimal format). Of course, it could be other things, but in this context (identifying a file) it seems likely.

Without knowing the details of the fmp12 file format, we could still quickly peak at the raw bytes of the hosted database to determine if this ID appears somewhere at the beginning of the file. And sure it does:

$ xxd my_file.fmp12 | head -n 20
00000000: 0001 0000 0002 0001 0005 0002 0002 c048  ...............H
00000010: 4241 4d65 0000 1000 0000 0000 0000 0000  BAMe............
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
... here it starts (16 bytes from position 250):
000000f0: 0000 0000 0000 0000 0000 e66d 7ab4 4680  ...........mz.F.
00000100: 4cbf 86f6 b001 6e07 9714 efb8 6c2e 2af4  L.....n.....l.*.

Let’s now look at the part after the semicolon – the one that is more interesting, but currently looks like garbage.

Base64-decoding the two samples from above, wee see:

$ echo -n '9acSKigNs3IHuEikQeit1Q==' | base64 -d | xxd
00000000: f5a7 122a 280d b372 07b8 48a4 41e8 add5  ...*(..r..H.A...

$ echo -n 'b442h3lSslgZ66HAwoSO7+jm1Xcw0O7DSmRlW9bkRrj7EUeRYaj8K6VEsLWSYh8x' | base64 -d | xxd
00000000: 6f8e 3687 7952 b258 19eb a1c0 c284 8eef  o.6.yR.X........
00000010: e8e6 d577 30d0 eec3 4a64 655b d6e4 46b8  ...w0...Jde[..F.
00000020: fb11 4791 61a8 fc2b a544 b0b5 9262 1f31  ..G.a..+.D...b.1

While the data does not look useful at first sight, we can see that the first sample is 16 bytes long and the second one is 48 bytes long.

Setting an EAR password to another file, uploading it and storing the password on the server, we can observe different ciphertext lengths for different plaintext lengths.

If our EAR password is less than 16 bytes, we still get 16 bytes of ciphertext. If it’s greater than 16 bytes, but less than 32 bytes, we get 32 bytes of ciphertext.

We can be relatively certain that the server only encrypts full blocks of 16 bytes, suggesting a block cipher with a size of 128 bits.

Next, we can check if we get the same ciphertext on a different FileMaker Server installation. This is not the case, so the key is likely unique for every server.

How about using the same EAR password for a different database file? Does this create a different ciphertext?

After trying, we see that it does not produce a different ciphertext. This means that the file identifier does not play a role in the encryption. It also means that a static key and no initialization vector (or a static one) is used, since a different initialization vector would cause the ciphertext to be different, even if the same plaintext and key is used (assuming we have an IV and use “cipher-block chaining”; see explanation in box below).

Another good test to verify this is to encrypt two EAR passwords which have the same first 16 bytes. For example:

File 1: AAAAAAAAAAAAAAAA|AAAAAAAAAAAAAAAAAAAAAAAA
File 2: AAAAAAAAAAAAAAAA|A
                       ^ 16 bytes

On my test server these two produce the following two ciphertexts:

b442h3lSslgZ66HAwoSO7+jm1Xcw0O7DSmRlW9bkRrj7EUeRYaj8K6VEsLWSYh8x
b442h3lSslgZ66HAwoSO78+E2b55R4CaKGiFSnwfUeQ=

Or as hex:

00000000: 6f8e 3687 7952 b258 19eb a1c0 c284 8eef  o.6.yR.X........
00000010: e8e6 d577 30d0 eec3 4a64 655b d6e4 46b8  ...w0...Jde[..F.
00000020: fb11 4791 61a8 fc2b a544 b0b5 9262 1f31  ..G.a..+.D...b.1

00000000: 6f8e 3687 7952 b258 19eb a1c0 c284 8eef  o.6.yR.X........
00000010: cf84 d9be 7947 809a 2868 854a 7c1f 51e4  ....yG..(h.J|.Q.

You can clearly see that the first 16 bytes are exactly the same (just like the 16 bytes of the plaintext).

Normally, you would not want your ciphertext to “leak” semantic information as shown and this is something that could certainly be improved.

If you were to encrypt every block of data with the same key, identical blocks of plaintext would produce identical blocks of ciphertexts. One strategy to prevent this is to let the results of each block influence the next block (so-called “cipher-block chaining”). Here, each plaintext block is XORed with the previous ciphertext block (and subsequently encrypted).

To kickstart this process, however, you need an additional input for the first block, and this is where the initialization vector comes in. It will be used to XOR the first plaintext.

If you were to experiment more, you would see that cipher-block chaining is indeed used for the FMS keystore values, but with an initialization vector of all zeros. This way you still output the same ciphertext blocks for identical plaintext blocks at the beginning.

If you like to test this for yourself you can play around with openssl. For example:
openssl enc -aes-256-cbc -in test.txt -k <some-key-as-hex> -iv 00000000000000000000000000000000 -out test.enc.

Just read the docs

Whenever you have documentation available, make sure to read it. It might give you pointers on how to narrow down the space of things to try next.

Claris’ documentation has a statement on how passwords are stored:

When you open an encrypted file on FileMaker Server or FileMaker Cloud, you can save the password to automatically open encrypted files when the server restarts. FileMaker employs a two-way AES-256 encryption that uses a composite key based on information from the machine to encrypt the password and stores the password securely on the server.

(Unfortunately, as we will learn later, this is not entirely correct, so it’s also a good idea to second-guess/verify official information).

What do we know so far?

Let’s recap on what we learned so far by just observing data and reading the docs:

• The keystore contains keys and values separated by semicolon
• Keys are just base64 encoded. Values are encrypted and base64 encoded
• We’re dealing with a block cipher (takes a fixed size block and outputs a block of the same size)
• The docs mention AES with 256-bit key
• A static (or no) initialization vector is used
• A static key is used
• The key is unique per installation and file IDs/properties don’t play a role here

A lazy try: just dump the memory

After looking at the data and the docs my first attempt was to test if some or all of the information of the keystore can be retrieved in unencrypted form from memory. It’s unlikely we learn anything about how data is encrypted but we can check if the server cleans up secrets after use. Also, grepping through a memory dump is not a lot of effort, so you might as well give it a shot (which is why I labeled this “a lazy try”).

My test server is running on Linux and I’m using procdump (the Linux version of the original Windows SysInternals tool).

Let’s first figure out the process ID of fmserverd with pgrep fmserverd and then dump the memory with procdump <pid>. The result is a dump with more than a gigabyte of size.

Since we know the plaintext values that the server encrypted, we can grep for these strings in the dump file.

$ strings fmserverd_time_2023-... | grep 'some EAR password'

No luck (which is a good thing, as you don’t want your secrets explosed like this). But what about the RSA private key that we expect to be the value of the key machinePrivateKey?

$ strings fmserverd_time_2023-04-23_14\:37\:25.1001 | grep 'BEGIN RSA PRIVATE' -A10

-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA27LEJB8uCqezntbENVPdrV/xTnnshR2fQPWFNr3Rcr10tjBa
...

Turns out the private key is hanging out in unencrypted form in memory. Let’s check if it matches the public key file CStore/machinePublicKey.

A quick way to do this is to extract the public key from the discovered private key and then compare it to the machinePublicKey file.

openssl rsa -in machinePrivateKey.key -pubout > machinePublicKey.extracted

This gives us a public key which – by looking at it – doesn’t match the one from the FileMaker Server installation. However, this is just because it’s not in PKCS#1 format like the original one. We can convert it and see that it’s a match now:

openssl rsa -pubin -in machinePublicKey.extracted -RSAPublicKey_out -out machinePublicKey.extracted.pkcs1

What now?

If we only wanted to know the RSA private key, we would have succeeded by now. But even if we would also have gotten our other secrets in plaintext, it would not really help us as we still don’t understand how the values are encrypted, wouldn’t know what to search for when not knowing the plaintext, and don’t understand the memory structure.

So what could be the next step? Since we are anyhow dealing with memory dumps, maybe we can find the actual AES key used to encrypt the keystore values in memory?

Extracting AES keys from memory

How could we possibly find what 16 or 32 bytes are an AES key when getting handed a dump of more than gigabyte of seemingly random data?

When we put a key into AES, the key is actually expanded into so-called round keys (11 rounds for 128-bit keys, 15 for 256-bit keys). These round keys are needed for the encryption/decryption process and might thus be hanging around in memory. By checking the memory for random-looking (i.e. high entropy) byte sequences using a sliding window of the size of the so-called key schedule (all the roundkeys), it is possible to get a list of potential keys.

Here’s a good video explaining the key expansion and here’s an article describing in more detail what options are available to extract keys from memory dumps.

Since it’s kind of a long shot anyway, I didn’t spend too much time in this phase and just ran two forensic tools over the dump. One was bulk_extractor and the other aeskeyfind.

Using the tools is straight-forward. For example: bulk_extractor -E aes my_dump -o extraction_output.

While the tools found some potential AES-128-bit keys, none of them were matching or weren’t even keys at all. They could have also been from the encryption/decryption routines that go on when the server encrypts/decrypts individual pages of the opened hosted databases.

I tried once more to dump the memory multiple times during FMS startup with a lot of encrypted files being opened (which would require keystore values being decrypted) and also during repeated opening/closing of a file in a loop, but it obviously didn’t produce any detectable keys in the dumps.

Getting the key so seemlessly by running a tool would have been nice in a way, but I’m also kind of reliefed that it wasn’t that easy :-)

Looking at what the program is doing and extracting the key

Since the “lazier” attempts didn’t result in any new insights, let’s actually try to observe what the program (fmserverd) is doing.

We obviously don’t have any source code, but as the program runs on our computer, we can attach a debugger and inspect the behavior, see loaded libraries, disassemble parts of it and step through the interesting bits.

I used gdb on Linux and lldb on macOS. Please note, that on macOS you will need to disable System Integrity Protection (or remove the program’s code signature) to attach to a running process.

On our Linux server, we can start by attaching with gdb -p <pid> and on macOS with lldb -p <pid>. Luckily, the binaries are not stripped of symbols, so we can see function names and of course the exports of the libraries.

Let’s first have a look at loaded libraries of a running server instance on Linux:

(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007f5dea393a40  0x00007f5dea3f89af  Yes (*)     /lib/x86_64-linux-gnu/libc++.so.1
0x00007f5dea32e450  0x00007f5dea348ef0  Yes (*)     /lib/x86_64-linux-gnu/libc++abi.so.1
0x00007f5de9a21ed0  0x00007f5dea09341d  Yes         /opt/FileMaker/lib/libFMSLib.so
0x00007f5de8f48560  0x00007f5de959d1f9  Yes         /opt/FileMaker/lib/libFMEngine.so
0x00007f5de6e0fb60  0x00007f5de83bdf00  Yes         /opt/FileMaker/lib/libDBEngine.so
0x00007f5de61fea20  0x00007f5de63496b9  Yes         /opt/FileMaker/lib/libSupport.so
...

Most interesting to us are the non-standard libaries located in /opt/FileMaker/lib and of course the actual fmserverd binary.

The binaries come with some symbol information, so we can try to make sense of any of the contained function names.

Below I’m using objdump to dump the symbol table and demangle the names (we want human-readable names), and pipe the result to less to easily scroll through and search in the result.

objdump -tC <some binary> | less

Symbol information allows for much easier debugging, also when you don’t have the source code available (even if explicit debug symbols would not be available). I assume the binaries are intentially not stripped (i.e. contain symbols) in the release versions such that it’s possible to troubleshoot/debug weird behaviors on deployed systems (although, I guess, you could still keep them separate from the binary and then strip all unneeded in the relase version)

If you like to see what is included in a binary and what not, you can write a small C program, compile it with gcc and once include the -g flag, once not include the -g flag and then run strip over another compiled version. For each of the three versions, you can then run the above objdump command and see the difference.

Fun fact: I later found out that I could have retrieved much more information just by inspecting all the provided symbol information, so I believe I made it a bit harder for myself and that you can arrive at the end result faster than I describe here.

Since we are interested in the keystore, we can search for names that could be related. Such terms could be keystore, EAR, crypt, password, AES, etc.

As an example, let’s say we searched for “Password” and have identified the following symbol in libSupport:

00000000000f9a50 g     F .text  0000000000000005              Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)`.

Since we still have our debugger attached, we can have a look at this function from within gdb:

(gdb) info address Draco::PasswordHash::ComputePasswordHashRawBytes
Symbol "Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)" is a function at address 0x7f5de6220a50.

If we disassemble this function we see it’s just a jump to PKCS5_PBKDF2_HMAC_SHA1.

(gdb) disas 0x7f5de6220a50
Dump of assembler code for function Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*):
   0x00007f5de6220a50 <+0>:     jmpq   0x7f5de61fdab0 <PKCS5_PBKDF2_HMAC_SHA1@plt>
End of assembler dump.

And what PKCS5_PBKDF2_HMAC_SHA1 is doing, we can just look up in the openssl manual.

We still don’t know if this function is at all related to what we want to find out (there’s a lot of crypto stuff in FileMaker Server), but we can start taking notes and piece together information.

Not every function will be as short and understandable as the above and you will either need to read a lot of assembly code or could try to figure things out using a decompiled version of the function of interest (for example with Ghidra’s decompiler). For where we want to get to, we don’t really need a decompiler and can just skim the assembly code, so we will mostly just stay in gdb.

Once we have few ideas and an overview of interesting functions that could be related to the keystore, we can set breakpoints, continue running the program and then perform an action that we want to observe. I always used the opening of a hosted file (fmsadmin open <file>) as this must read the keystore file and decrypt the EAR password.

Eventually, I arrived at Draco::Encrypt_AES_decrypt which is called by Draco::KeyStorageProvider::CryptData (both in the Support lib) and is responsible for the decryption of the keystore values. Let’s see what we can learn when setting a breakpoint at this function (still in the attached debugger):

(gdb) b Draco::Encrypt_AES_decrypt
Breakpoint 1 at 0x7f5de61f8a40 (2 locations)
(gdb) c
Continuing.

With the breakpoint set, we can now open a file that has an EAR password stored in the keystore. As soon as the open command is sent, the program hits the breakpoint and stops:

Thread 49 "fmserverd" hit Breakpoint 1, 0x00007f5de61f8a40 in Draco::Encrypt_AES_decrypt(unsigned char*, int, unsigned char const*, int, unsigned int&, unsigned char const*)@plt ()
   from /opt/FileMaker/lib/libSupport.so

Since we see the @plt at the end, we need to execute one more jmp into the actual function with si (step one instruction).

@plt stands for “procedure linkage table” and is essentially a way to point to a library function which address cannot be known in the linking stage but only dynamically be resolved at run time (as far as I understand it).

Now that we are in the actual implemented function, let’s have a look what arguments are passed to it.

To do this we first check the CPU registers:

(gdb) info registers
rax            0x7f5dbf8f6078      140040622530680
rbx            0x7f5dbf8f5af4      140040622529268
rcx            0x20                32
rdx            0x7f5dd863d6c0      140041039107776
rsi            0x30                48
rdi            0x7f5dd80e0e90      140041033485968
rbp            0x7f5dbf8f5ff0      0x7f5dbf8f5ff0
rsp            0x7f5dbf8f5ed8      0x7f5dbf8f5ed8
r8             0x7f5dbf8f5f80      140040622530432
r9             0x0                 0
r10            0x7f5dd8105b9c      140041033636764
r11            0x7f5dbf8f5cc8      140040622529736
r12            0x1                 1
r13            0x3e5               997
r14            0x7f5dbf8f6040      140040622530624
r15            0x7f5dbf8f5fb0      140040622530480
rip            0x7f5de62217f0      0x7f5de62217f0 <Draco::Encrypt_AES_decrypt(unsigned char*, int, unsigned char const*, int, unsigned int&, unsigned char const*)>

Based on x86-64 calling conventions, we know how and in what registers functions receive their arguments from their caller (see “A.2 AMD64 Linux Kernel Conventions” in AMD64 System V Application Binary Interface).

Quoted from the linked document:

User-level applications use as integer registers for passing the sequence %rdi, %rsi, %rdx, %rcx, %r8 and %r9.

We don’t have any parameter names but we do know the signature of our discovered function (unsigned char*, int, unsigned char const*, int, unsigned int&, unsigned char const*). So let’s map the arguments:

• rdi: A pointer to 0x7f5dd80e0e90
• rsi: An integer with value 48
• rdx: A pointer to 0x7f5dd863d6c0
• rcx: An integer with value 32
• r8: A pointer to 0x7f5dbf8f5f80
• r9: Null

Since we only have 6 parameters, we don’t need to look at the stack for more.

Now let’s inspect the addresses in more detail.

We assume that the second and fourth parameter indicates the size of the previous parameter, as this is quite common.

Checking 48 byes from address 0x7f5dd80e0e90 (rdi):

(gdb) x/48xb 0x7f5dd80e0e90
0x7f5dd80e0e90: 0x6f    0x8e    0x36    0x87    0x79    0x52    0xb2    0x58
0x7f5dd80e0e98: 0x19    0xeb    0xa1    0xc0    0xc2    0x84    0x8e    0xef
0x7f5dd80e0ea0: 0xe8    0xe6    0xd5    0x77    0x30    0xd0    0xee    0xc3
0x7f5dd80e0ea8: 0x4a    0x64    0x65    0x5b    0xd6    0xe4    0x46    0xb8
0x7f5dd80e0eb0: 0xfb    0x11    0x47    0x91    0x61    0xa8    0xfc    0x2b
0x7f5dd80e0eb8: 0xa5    0x44    0xb0    0xb5    0x92    0x62    0x1f    0x31

If we compare this to our encrypted EAR password that we looked at earlier, it comes out as the same sequence of bytes:

$ echo -n 'b442h3lSslgZ66HAwoSO7+jm1Xcw0O7DSmRlW9bkRrj7EUeRYaj8K6VEsLWSYh8x' | base64 -d | xxd
00000000: 6f8e 3687 7952 b258 19eb a1c0 c284 8eef  o.6.yR.X........
00000010: e8e6 d577 30d0 eec3 4a64 655b d6e4 46b8  ...w0...Jde[..F.
00000020: fb11 4791 61a8 fc2b a544 b0b5 9262 1f31  ..G.a..+.D...b.1

We can be pretty sure that this argument is the ciphertext from the keystore.

What about 0x7f5dd863d6c0 (rdx)? Let check 32 bytes from there:

(gdb) x/32xb 0x7f5dd863d6c0
0x7f5dd863d6c0: 0xa5    0x10    0xc9    0xf2    0x24    0xde    0x97    0x1b
0x7f5dd863d6c8: 0x45    0x95    0x82    0xae    0xec    0x5b    0xc8    0x84
0x7f5dd863d6d0: 0x9c    0x9f    0xb1    0xa0    0x4c    0xe4    0xba    0x27
0x7f5dd863d6d8: 0x31    0x0d    0x03    0xf0    0xef    0x67    0xc6    0x27

Since we are expecting a 256-bit key, this could be it.

Now we have the values of r8 and r9 left. We don’t know about the first, but the latter is 0 (0x00).

Since we expect AES to be used in CBC mode without an initialization vector (IV), maybe this could be the “null IV”!?

We already have potentially useful information here, but let’s check what Encrypt_AES_decrypt is actually doing and if we can get any further hints.

To do this, we can disassemble the function right from GDB:

Dump of assembler code for function Draco::Encrypt_AES_decrypt(unsigned char*, int, unsigned char const*, int, unsigned int&, unsigned char const*):
=> 0x00007f5de62217f0 <+0>:     push   %rbp
   ...
   0x00007f5de6221855 <+101>:   mov    %rax,%rbx
   0x00007f5de6221858 <+104>:   callq  0x7f5de61f9bf0 <EVP_aes_128_cbc@plt>
   ...
   0x00007f5de622186b <+123>:   callq  0x7f5de61f8d00 <EVP_DecryptInit_ex@plt>
   ...
   0x00007f5de6221881 <+145>:   callq  0x7f5de61fd870 <EVP_DecryptUpdate@plt>
   ...
   0x00007f5de62218a4 <+180>:   callq  0x7f5de61fc980 <EVP_DecryptFinal_ex@plt>

I removed some instructions above to only show the interesting calls. GDB resolves the addresses and shows us calls to OpenSSL functions, for example EVP_aes_128_cbc.

We could look up the details and source code for these functions as they are publicly known.

While not a big deal, it’s a bit of a surprise to see EVP_aes_128_cbc as we were expecting EVP_aes_256_cbc as that would match what is written in Claris’ documentation.

For now, we just set another breakpoint at the first and second call, (gdb) b EVP_aes_128_cbc and (gdb) b EVP_DecryptInit_ex, and then continue running the program (c).

At this stage we just want to verify that the function is actually called (we could have also read and understand all the disassembled instructions instead).

Both breakpoints hit. And in the second one, we can actually verify our assumptions from above as we know the parameters names and their meaning for the decryption intialization function from the OpenSSL documentation.

This means we have learned so far:

• AES is used in CBC mode
• We have a 256 bit key, but are using EVP_aes_128_cbc and thus only use the first half of the key (probably a bug)
• We know the IV is set to all zeros (16 bytes, like the block size)

Building our own decryptor

It’s time to test if the data we discovered can actually be used to independently decrypt the keystore values.

Let’s build a small Python script and re-implement what we discovered. I’m using PyCryptodome for the crypto routines.

import base64
from Cryptodome.Cipher import AES
from Cryptodome.Util.Padding import unpad


def decrypt(key, iv, ciphertext):
    decoded_ciphertext = base64.b64decode(ciphertext)

    cipher = AES.new(key, AES.MODE_CBC, iv=iv)
    plaintext = cipher.decrypt(decoded_ciphertext)

    return unpad(plaintext, 16)


def main():
    ciphertext = b'b442h3l<snip>'
    key = bytes.fromhex('a510c<snip>')
    iv = 16 * b'\x00'

    plaintext = decrypt(key[:16], iv, ciphertext)
    print(f'plaintext: {plaintext.decode()}')


if __name__ == '__main__':
    main()

Starting in main, the ciphertext is our base64-encoded string from the keystore, the key is the key we discovered via the debugging process (unique per server), and iv is just 16 null bytes (16 bytes being the block size). From the 32 byte key we only pass byte 0 to 15 to have a 128-bit key.

In the decrypt function, we first base64-decode the ciphertext, then initialize the cipher with the passed values and CBC mode.

Lastly, we decrypt the ciphertext and return the plaintext (unpadded, to remove the padding at the end that completes a full block).

Running the script successfully reveals the plaintext value (in this case the EAR password we decided to store before).

While we now have everything we need to decrypt other values on our server, our main goal was to better understand how the keystore works. And we still don’t know how the key we found was derived – it cannot be a hardcoded key, as it’s different from server to server. Let’s dig in more.

Figuring out how the key is derived

Above we saw that Encrypt_AES_decrypt already gets the key passed to it and that it is called by Draco::KeyStorageProvider::CryptData.

We can look at the CryptData function from within gdb:

(gdb) disas 'Draco::KeyStorageProvider::CryptData'

This function is certainly more involved with lots of conditional jmps and calls.

Let’s first demangle the function names to make it bit easier to browse through:

(gdb) set print asm-demangle on

Disassembling the function again and just looking at the symbols before our Encrypt_AES_decrypt and Encrypt_AES_encrypt calls, we see two names that stick out:

0x00007f77a60ec76c <+508>:   callq  0x7f77a60431d0 <Draco::Machine::GetPersistentID()@plt>
...
0x00007f77a60ec947 <+983>:   callq  0x7f77a603f120 <Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)@plt>

From the Claris documentation, we know that “FileMaker […] uses a composite key based on information from the machine to encrypt the password and stores the password securely on the server.”.

Since we know the Get(PersistentID) function from FileMaker Pro’s calculation environment returns a unique identifier for the current machine it is running on, we can assume that Draco::Machine::GetPersistentID() function does something similar or the same.

At the same time, we can assume that this ID goes into the “composite key based on information from the machine”.

While we can also make an assumption what the ComputePasswordHashRawBytes is doing, based on the name, we will go through the same debugging procedure as with the Encrypt_AES_decrypt function above to get more insights.

Let’s set a breakpoint, continue with the program and then close/open our sample database file again:

(gdb) b Draco::PasswordHash::ComputePasswordHashRawBytes
Breakpoint 1 at 0x7f77a603f120 (2 locations)
(gdb) c
Continuing.

Thread 49 "fmserverd" hit Breakpoint 1, 0x00007f77a603f120 in Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)@plt () from /opt/FileMaker/lib/libSupport.so
(gdb) disas
Dump of assembler code for function _ZN5Draco12PasswordHash27ComputePasswordHashRawBytesEPKcjPKhjiiPh@plt:
=> 0x00007f77a603f120 <+0>:     jmpq   *0x2b076a(%rip)        # 0x7f77a62ef890 <Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)@got.plt>
   0x00007f77a603f126 <+6>:     pushq  $0x10f
   0x00007f77a603f12b <+11>:    jmpq   0x7f77a603e020
End of assembler dump.

We step again into the actual function and then have a look at the arguments being passed. We know the signature, so let’s map the register values:

(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)

• rdi: 0x7f778c01a5c0
• rsi: 0x10 (or 16 in decimal)
• rdx: 0x7f779c250f70
• rcx: 0x08 (8)
• r8: 0x3e5 (997)
• r9: 0x20 (32)

The seventh argument is passed on the stack, so let’s look at the value on the stack after the return address (stack pointer is in $rsp and we skip the first 8 bytes):

(gdb) x/xg $rsp+8
0x7f779c250ed0: 0x00007f779c250f00

Let’s take a look what values can be found at the addresses pointed to by the first, third and seventh argument:

The first argument is 16 bytes in length, but is certainly not an ascii string (look at it with x/16xb 0x7f778c01a5c0).

The third argument can indeed be interpreted as a string and points to fmserver (if we spent more time looking at CryptData we learn that this string and the 997 are the username/id acquired via getpwuid).

The seventh argument contains data we don’t understand yet and might very well just be the address where the result goes.

Let’s disassemble the body of the function now:

(gdb) disas
Dump of assembler code for function Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*):
=> 0x00007f77a6066a50 <+0>:     jmpq   0x7f77a6043ab0 <PKCS5_PBKDF2_HMAC_SHA1@plt>
End of assembler dump.

We can see that the program is calling OpenSSL’s PKCS5_PBKDF2_HMAC_SHA1.

Looking at the documentation, we know what each argument is and can now also make sense of the arguments we deciphered above:

int PKCS5_PBKDF2_HMAC_SHA1(const char *pass, int passlen,
                           const unsigned char *salt, int saltlen, int iter,
                           int keylen, unsigned char *out);

The password is the 16 bytes at 0x7f778c01a5c0, the salt is fmserver, the iterations are 997 and the key length is 32.

PBKDF2 is a common key derivation function, which is used here to create the key for decrypting the keystore value.

PBKDF2 stands for Password-Based Key Derivation Function 2. It takes a pseudorandom function (such as HMAC-SHA1, as we see above), a password, a salt, a key length and an iteration count. In short, it computes a hash from the salt and a block iteration count, using the given password as key. In the first iteration, it builds another hash using the password as key and the previous hash result as message. These two hashes are then XORd. What follows is a loop of creating a new hash from the previous hash, XORing it again with the previous XOR result, and so on. This is done for the number of iterations specified (here 997) and for each necessary block of the hash length to reach the length of the desired key (here 2 as the hash output of SHA-1 is 20 bytes, but we request a 32 byte key) . For more details see the Key derivation process explained on Wikipedia.

A few additional things to note: 1) the mentioned hash is not just a SHA-1 hash but an invocation of HMAC. Thus, we’re running SHA-1 twice to come up with our hashes in each iteration. 2) The salt being used here is certainly not random (the default fmserver username) and the iteration count is rather low and also likely to be the same or in a close range for every machine (being the user ID). Since the “password” is already a hash created from a unique identifier (see later in this article), though, this might be less problematic. I assume that this was done like this to easily “break” the key once any of the variables on the system change, which would force a user to re-enter the EAR password.

Given the information we got so far, we should be able to reproduce the key (even though, we still don’t know how that password is created).

Back in Python, we can do:

import binascii
from Cryptodome.Protocol.KDF import PBKDF2
from Cryptodome.Hash import SHA1

salt = 'fmserver'
count = 997
password = bytes.fromhex('<the value at 0x7f778c01a5c0>')
key = PBKDF2(password, salt, 32, count=count, hmac_hash_module=SHA1)

print(binascii.hexlify(key))

Running the script, we can now generate the same key that we have discovered above. Cool, one step further!

Figuring out how the password is created

The last piece of information we need is how the password which is being passed to PBKDF2 is created .

Since the password which we discovered was 16 bytes long and we earlier saw a call to GetPersistentID (which, if equivalent to the Get(PersistentID) function, also returns 16 bytes sequences), we could assume that the password we saw was just said machine identifier.

If we compute the persistent ID of the server (for example via Perform Script on Server[]), however, we realize that the value doesn’t match – either because they are indeed different implementations or because the password is just created differently.

To understand how the server arrives at the previously detected password, we can continue in two ways. Either we step through the instructions of Draco::KeyStorageProvider::CryptData (which calls the previously analysed ComputePasswordHashRawBytes) or we go a couple of steps back and check again, if we can find other interesting symbols now that we have learned a bit more and know the KeyStorageProvider class is involved.

For the latter strategy Draco::KeyStorageProvider::GetInstance() seems like a good starting point.

Setting a breakpoint and entering the function, we see the following:

(gdb) disas
Dump of assembler code for function Draco::KeyStorageProvider::GetInstance():
   0x00007f3ef752b4c0 <+0>:     push   %rbx
   0x00007f3ef752b4c1 <+1>:     mov    0x20b3c9(%rip),%al        # 0x7f3ef7736890 <_ZGVZN5Draco18KeyStorageProvider11GetInstanceEvE8instance>
   0x00007f3ef752b4c7 <+7>:     test   %al,%al
   0x00007f3ef752b4c9 <+9>:     je     0x7f3ef752b4d4 <Draco::KeyStorageProvider::GetInstance()+20>
   0x00007f3ef752b4cb <+11>:    lea    0x20b2ae(%rip),%rax        # 0x7f3ef7736780 <_ZZN5Draco18KeyStorageProvider11GetInstanceEvE8instance>
   0x00007f3ef752b4d2 <+18>:    pop    %rbx
=> 0x00007f3ef752b4d3 <+19>:    retq

We have a test right at the beginning, meaning we likely return early, if the KeyStorageProvider has already been instantiated. However, once we’re in this function, we can also look at what’s in memory.

We can see that the previously discovered password a04d.. is already set here, so we might need to find where we initialize the provider for the first time (and thus where the password is being created).

As mentioned in the beginning of the article, I’ll brush over some details here to not make a “point-and-shoot” decryption script available, but leave this as an exercise for the reader.

What you have to look for is the initialization function for the KeyStorageProvider (which is executed during first start of the server) and another function that derives the password.

In it you will find that an MD5 context is built, which then gets fed the machine’s persistent ID and another secret (you know it when you see it). Eventually, a 16-byte MD5 checksum is created, which matches the password we saw earlier.

Having found this final part, we can now re-create the key and decrypt keystore values. The following Python script shows how you could do it in a program, leaving a few details out for the reasons mentioned above:

import hashlib
import base64
from Cryptodome.Cipher import AES
from Cryptodome.Util.Padding import unpad
from Cryptodome.Protocol.KDF import PBKDF2
from Cryptodome.Hash import SHA1


def decrypt(key, iv, ciphertext):
    decoded_ciphertext = base64.b64decode(ciphertext)

    cipher = AES.new(key, AES.MODE_CBC, iv=iv)
    plaintext = cipher.decrypt(decoded_ciphertext)

    return unpad(plaintext, 16)


def derive_key():
    secret = b'' # the full secret string discovered in the last part

    password = hashlib.md5(secret).digest()

    count = 997  # uid
    salt = 'fmserver'  # username

    return PBKDF2(password, salt, 16, count=count, hmac_hash_module=SHA1)


def main():
    ciphertext = b''  # base64-encoded ciphertext from keystore

    key = derive_key()
    iv = 16 * b'\x00'  # null IV

    plaintext = decrypt(key, iv, ciphertext)
    print(f'plaintext: {plaintext.decode()}')


if __name__ == '__main__':
    main()

How about macOS?

So far we’ve been working with a default Linux installation. To come to the same conclusion on macOS, you can use lldb instead of gdb.

To attach lldb to a running process, you will have to disable macOS’ System Integrity Protection or remove the code signature first, though.

The discovery process is more or less the same, although the syntax for lldb is a bit different.

After looking into it, I can tell you that the encryption/decryption process is the same, as is expected.

Naturally, on macOS the username and user id that go into the key derivation could be different. For a default installation the full name of the user is “fmserver User”. For the derivation this string is concatenated to the actual username “fmserver”.

You can find out the full user info with the following command (change fmserver for the username you run FMS under):

dscacheutil -q user | grep 'name: fmserver' -A6

How about Windows?

I only had a brief look at Windows. It seems that the encryption/decryption process works a bit different here.

I haven’t spent much time looking into this and wanted to write up my findings first. Maybe I’ll look into it in the future.

Conclusion

I hope this post was helpful to you learning how FileMaker Server handles the secrets you decide to store on the machine.

As we saw it’s not unfeasable to recover the plaintexts. While this is expected and “by design” as the server needs to do the same, you can now judge how easy or hard it is.

I reported some things that could be improved or might be a bug to Claris and hope the keystore will be improved as a result.

The decision for or against storing the EAR passwords really depends on your use-case. If uptime is very important and you need your databases to auto-open after a reboot, there’s no way around to storing them on disk (and I don’t think that’s an unreasonable decision – I have setups where I do the same).

You may also look into restricting access to the keystore file, i.e. remove permissions for other (it seems the only user/group accessing it should be, by default, fmserver and fmsadmin).

Generally, with enough time and patience you will (at least theoretically) always get to some of the data if you have root access to the live server and the databases are opened. Having the original EAR password stored and the keystore file readable by everyone just makes it much easier.

Updates:

• 2023-05-31: Corrected the macOS section to include that you can remove the code signature instead of disabling SIP. Thanks, Alex.