MacOS on David Hamann

Connecting to a host service from within a container using Docker for Mac

Sun, 11 Oct 2020 00:00:00 +0000

TL;DR

Use host.docker.internal.

When you are running Docker on Linux and want to access services on the host from within a container, you can make use of the docker0 bridge interface (ip a s docker0). This does not work when running Docker for Mac as the interface is inside a separate virtual machine (which you can confirm by getting a shell in that vm: screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty) and thus not visible on the local host.

To workaround this, a reliable way to get to the macOS host is to use the host.docker.internal DNS name which will always resolve to an IP where the host is reachable.

Example

Let’s say you want to run some script in the container that connects to a specific HTTP target and want to observe the requests through a proxy running on your macOS machine (e.g. Burp). Then you would do:

# Run some container
docker run -it alpine:latest /bin/sh

# ... install the dependencies for your script ...
# then run the script and target your local (!) intercept proxy
python3 exploit.py http://host.docker.internal:8081/whatever

# now observe the request in the proxy application running on the host

UnicodeError when running Python script via macOS LaunchAgent

Fri, 07 Dec 2018 00:00:00 +0000

If you are getting UnicodeErrors when reading/manipulating files using a Python script launched by a LaunchAgent or crontab, the problem might lie in the “current locale encoding”.

Sample script

Let’s assume you have the following code in a script set up to be launched by a LaunchAgent (also see my article on LaunchAgents):

with open(some_path_to_file) as f:
 f.read()

Let’s also assume that some_path_to_file points to a txt file containing some emojis (hey, why not? 😎) or some other unicode characters.

Running a file with the above code snippet in your terminal session will probably not cause any issues, because – most likely – everything is set up to use utf-8 as the default encoding: python3 my_snippet.py. Cool!

However, when the script is launched by a LaunchAgent, you may get an error.

UnicodeError: can’t decode byte

The error could look something like this:

Traceback (most recent call last):
 [...]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 13: ordinal not in range(128)

Why is this happening?

All about the context, or: what’s my locale?

Running the script in your terminal session and running it via a LaunchAgent might mean you/Python are reading different locale/encoding settings, as can be seen by the return value of getpreferredencoding from the locale module.

>>> locale.getpreferredencoding(False)
`UTF-8`
or
`US-ASCII`

From the Python help:

Return the encoding used for text data, according to user preferences. User preferences are expressed differently on different systems, and might not be available programmatically on some systems, so this function only returns a guess.

Now, let’s look at what happens when we don’t explicitly specify the encoding argument for the open function:

In text mode, if encoding is not specified the encoding used is platform dependent: locale.getpreferredencoding(False) is called to get the current locale encoding.

If the current locale cannot be determined, it will fall back to C locale US-ASCII. local.getlocale() in this case will likely also just return (None, None).

The solution: always specify the encoding!

If you just want to make sure that you can read your Unicode file properly, be explicit about the encoding, i.e.:

with open(some_path_to_file, encoding='utf-8') as f:
 f.read()

… and the problem should go away.

Using launchd agents to schedule scripts on macOS

Tue, 13 Mar 2018 00:00:00 +0000

Even though launchd has been around for quite some time now, I was still using crontab for scheduling some of my scripts until recently. Since launchd LaunchAgents can do much more and don’t expect your computer to be running at all times, it’s time to start using them more 😎.

Let’s see how we can easily set up a LaunchAgent to run a Python script for the current user in regular intervals:

Write a plist for your agent

Unlike crontab jobs, LaunchAgents are written in (quite verbose) plist XML. We define a Label according to the name of our agent and then go on describing how it should behave (which program to run, what arguments, when to run it, etc.).

Here’s is an example with the filename de.davidhamann.my-program.plist which we will place in ~/Library/LaunchAgents.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>Label</key>
 <string>de.davidhamann.my-program</string>
 <key>ProgramArguments</key>
 <array>
 <string>/usr/local/bin/python3</string>
 <string>/Users/user/path/to/my-program.py</string>
 </array>
 <key>StartInterval</key>
 <integer>30</integer>
</dict>
</plist>

Label

The first interesting element to look at is <key>Label</key>. It precedes a string element which provides the unique name for our agent. This name is generally the same as our plist filename without the extension (both should follow reverse-domain style). The label is required and we need it to later start our agent.

ProgramArguments

Next, we define what program to run and what arguments to pass to it. In the example we will run Python3 from /usr/local/bin and give it the path of our script to execute.

StartInterval

StartInterval defines in what interval we want the program to run and takes an integer expected to contain the number of seconds. In our case, we run my-program.py every 30 seconds.

Load and start the agent 🚀

LaunchAgents can be loaded, started and unloaded with launchctl. To do so, we need to supply the ID of our target user and the plist file we just created. You can get your user-id via id -u or id -u <the-username>, respectively.

Assuming we are in ~/Library/LaunchAgents/ we can now load our agent by executing the following command:

launchctl bootstrap gui/<your-user-id> de.davidhamann.my-program.plist

And then kick-start it with (or just wait 😊):

launchctl kickstart -k gui/<your-user-id>/de.davidhamann.my-program

Note that we use the Label in this command, not the plist filename. The -k options means that the service will first be killed, if running (see man launchctl).

Our agent should now be active and starting the program every 30 seconds.

Note: we are setting up an agent for a user in a gui session. You may want to specify other targets, e.g. system for system-wide services. See man launchctl for more.

Unloading

Unloading an agent is as easy as:

launchctl bootout gui/<your-user-id> de.davidhamann.my-program.plist

In other tutorials you may see the use of launchctl load, launchctl start and launchctl unload. While these commands still work, they are from a from a previous implementation of launchd and are classified as “Legacy”.

Troubleshooting

In case you’re having trouble with an agent, you can either look into system.log or direct the output to a desired location by adding two keys for output and errors to the <dict>.

<key>StandardOutPath</key>
<string>/log/path/out.log</string>
<key>StandardErrorPath</key>
<string>/log/path/error.log</string>

Make sure to bootout, bootstrap, kickstart again after you make your change.

Note: The Debug key, though mentioned in the launchd manual, is not supported anymore.

Other use cases

Another common use case is StartCalendarInterval for starting a program at a given time. A nice property of this is, that it will handle sleep times. So launchd would start the job the next time your computer wakes up and won’t just skip it like cron (it does not do that for StartInterval, though I didn’t find that to be a problem for my cases).

Have a look at man launchd.plist for many more options or visit developer.apple.com for a broader (but in parts outdated) overview.

VPN: Temporarily solve same subnet conflicts

Wed, 27 Sep 2017 00:00:00 +0000

I recently had the problem of needing to establish a connection to a server behind a VPN that was in the same subnet as the network I was connecting from. Every call I wanted to make to the server on the remote network wouldn’t go through as it was looking for the server on the local network.

To eventually reach the server, I temporarily added a route to the routing tables, telling the client it should transmit traffic to the destination ip through the interface of the VPN connection.

If you ever come across a similar situation, here’s how to solve it.

First, find out which interface your VPN is using:

> ifconfig

Let’s assume it runs via ppp0. Let’s further assume that the server you want to reach is at 192.168.1.10. Now, to tell your client about it, add the route via the route command (this is on macOS, but should work more or less the same on a Linux system):

> sudo route add -host 192.168.1.10 -interface ppp0
add host 192.168.1.10: gateway ppp0

Doing a quick ping, you should see that your desired server is now reachable.

Note that when you re-connect you may need to setup the route again as the setting is not persisting.

Running Flask on macOS with mod_wsgi/wsgi-express

Sat, 05 Aug 2017 00:00:00 +0000

Flask is a (micro) web development Framework for Python. It is fairly simple to get started. All you need to do is to pip install Flask into your virtualenv, give the FLASK_APP environment variable your file and run flask run (described in detail in installation and quick start). This will launch the development server and you can instantly start hacking around.

When you want to use your Apache webserver, however, you need to install and configure a WSGI module. When I first wanted to do this I tried to install mod_wsgi via brew (brew install mod_wsgi from the homebrew/apache tap), but quickly ran into some (apparently common) issues with the XCode toolchain. Then I discovered that there is a much easier way of installing mod_wsgi as a Python package.

On the PyPI page it says…

[it] will compile not only the Apache module for mod_wsgi, but will also install a Python module and admin script for starting up a standalone instance of Apache directly from the command line with an auto generated configuration.

Let’s try it out

At this point I assume you have virtualenv and the XCode cli tools (xcode-select --install) installed (and of course the standard Apache from macOS). Everything else we will do together in the following steps.

Setting up virtualenv

Let’s start by creating the directory for the application and setting up our virtual environment:

$ mkdir my_app
$ cd my_app/
$ virtualenv venv
New python executable in /your/path/my_app/venv/bin/python2.7
Also creating executable in /your/path/my_app/venv/bin/python
Installing setuptools, pip, wheel...done.

Now we activate our environment:

$ source venv/bin/activate
(venv) $ # <- new prompt

Installing mod_wsgi

Installing mod_wsgi is now easily done via pip:

(venv) $ pip install mod_wsgi

Let’s see if it worked by launching the server:

(venv) $ mod_wsgi-express start-server
Server URL : http://localhost:8000/
Server Root : /tmp/mod_wsgi-localhost:8000:501
Server Conf : /tmp/mod_wsgi-localhost:8000:501/httpd.conf
Error Log File : /tmp/mod_wsgi-localhost:8000:501/error_log (warn)
Request Capacity : 5 (1 process * 5 threads)
Request Timeout : 60 (seconds)
Startup Timeout : 15 (seconds)
Queue Backlog : 100 (connections)
Queue Timeout : 45 (seconds)
Server Capacity : 20 (event/worker), 20 (prefork)
Server Backlog : 500 (connections)
Locale Setting : de_DE.UTF-8

By pointing your browser to http://localhost:8000/, you should be greeted by this page:

Looks good. Let’s stop the server with ctrl-c.

Installing Flask and creating the web app

If you already have your Flask app, you can skip the next few commands, but for the sake of completeness, let’s install Flask and create a small sample web app (make sure you are still in your virtualenv):

(venv) $ pip install Flask

Let’s create another directory to put the code of our web application in, and fire up an editor for creating the code file:

(venv) $ mkdir my_app 
(venv) $ vim my_app/__init__.py # <- choose the cli editor you want or just create the file with a gui editor

Paste this code (from the Quick Start tutorial) into your file:

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
 return 'Hello, World!'

Creating the wsgi script

To let the server know about our application, let’s create the wsgi script which we will later point to when starting the server.

(venv) $ pwd
/your/path/my_app # <- we are here
(venv) $ vim my_app.wsgi # <- choose the cli editor you want or just create the file with a gui editor

Paste this code into the new file:

from my_app import app as application

UPDATE: I initially set sys.path.insert(0,"/your/path/my_app/") in the snippet above. This is not needed, as the directory you run mod_wsgi-express setup-server or mod_wsgi-express start-server in is added to sys.path automatically. See these three tweets.

Launching the server with the wsgi script

(venv) $ mod_wsgi-express start-server my_app.wsgi

By going to http://localhost:8000/ you should now see the “Hello, World!” from our Flask app.

Doesn’t work for you? Didn’t work for me at first either as I had a typo in the code 🙂. Check out the error logs and you will see a hint of what might be wrong: /tmp/mod_wsgi-localhost:8000:501/error_log.

Running the server in the background

If you want to continue with your shell session, but leave the server running, you can use nohup.

(venv) $ nohup mod_wsgi-express start-server my_app.wsgi &

This will leave the server running, while you can continue working in the session (output will go to the file nohup.out). To bring it back to the foreground, use fg.

UPDATE: Graham Dumpleton reached out to me and noted that the better way of running it in the background is by generating scripts via setup-server and then use apachectl to start/stop. So you might want to execute the following (as root) instead of using nohup:

(feel free to change the port)

mod_wsgi-express setup-server my_app.wsgi --port=8000 --user _www --group _www --server-root=/etc/mod_wsgi-express-8000

And then control the server state via:

/etc/mod_wsgi-express-8000/apachectl start
/etc/mod_wsgi-express-8000/apachectl stop

Watching for changes

When you develop your app and don’t want to reboot the server for each change, there is a convenient way of starting the server with the --reload-on-changes option. Now you can change your files and have the changes immediately served.

(venv) $ mod_wsgi-express start-server --reload-on-changes my_app.wsgi

More info

You can find more info about mod_wsgi (e.g. running it on a privileged port as root) on the PyPI page: mod_wsgi.

The command documentation is available here:

(venv) $ mod_wsgi-express start-server --help

Sharing a VPN connection with another device on macOS Sierra/El Capitan

Wed, 19 Apr 2017 00:00:00 +0000

There are multiple reasons why you would want to share a VPN connection from your Mac with another device. Maybe you have to install a proprietary VPN client which does not run on your main computer or you just don’t want to run/install it there. Maybe your main computer doesn’t “comply” (i.e. you would need to install additional AV software – who wants that?). Or you simply need it for streaming video to some TV box.

I recently had the problem with a proprietary VPN client which I didn’t want to install on my main machine. I solved it by using another Mac running macOS 10.11 and am going to describe the process here and list the resources that were helpful.

If you are looking for a hardware VPN router or want to share a natively supported VPN connection (L2TP, IPSec) via “Internet Sharing”, you can stop reading here. I won’t cover this topic.

I quickly found the article “Using (and sharing) a VPN connection on your Mac”, which looked like a great solution. It suggested the following little shell script:

#!/bin/sh

natd -interface tun0
ipfw -f flush
ipfw add divert natd ip from any to any via tun0
ipfw add pass all from any to any
sysctl -w net.inet.ip.forwarding=1

Unfortunately, there’s neither ipfw nor natd on El Capitan or Sierra, since they were deprecated with Yosemite. So, the “new” thing to look into was pf (packet filter).

After reading the man page of pfctl and a bit more of googling, I found the solution. To load your own rules and share your VPN connection, you can use pfctl -f and give it a list of nat entries defining the interfaces from and to. This article “Share your VPN with Mac OS X El Capitan” from roelant.net helped a lot.

My working natvpn.sh script now looked like this:

#!/bin/sh

sysctl -w net.inet.ip.forwarding=1
sysctl -w net.inet.ip.fw.enable=1
pfctl -f ./nat-rules -e

You might need a sudo pfctl -d before to disable the packet filter.

…and the nat-rules file:

nat on utun0 from en0:network to any -> (utun0)

You might need to change the interfaces utun0/en0 according to your preferences.

The next step was to go to my main machine and setup a new network service (System Preferences -> Network). After clicking on the “+” button, I chose the adapter (in my case the Display Ethernet) and manual IPv4 configuration and entered my standard local networking details with the exception of the gateway address (might also be called “router address” on your machine). The gateway address is the IP of the mac on which the VPN client (incl. the script from above) was installed.

After the service was created I just needed to change the order of the services to prioritze the newly created service (click on the action menu -> Set Service Order) and then start the script (./natvpn.sh). Et voilà, I could use the VPN connection established by the other Mac on my main machine Mac.

If you don’t like to set the service order every time you use the shared VPN, you might want to set up a new network location for your VPN work (System Preferences -> Network -> Location -> Edit Locations … -> +)

Update: Troubleshooting

If sharing does not work after executing the steps above or stops working from one day to the other (yep, had that problem :-)), you should first check the nat-rules file. It is likely that either one of the interfaces is invalid or you might just have a typo in there. If it all looks OK but still doesn’t work, try to explicitly list the subnet, i.e. nat on utun0 from 192.168.1.0/24 to any -> utun0 and then start again.

After running ./natvpn.sh you may also want to do a sanity check by listing the rules with sudo pfctl -s all. You should see your entry under TRANSLATION RULES.

Another thing to try is to flush all rules before attempting again: pfctl -F all.