# Background on applehv vs. qemu
Podman with QEMU virtualization on Mac comes with a big performance-penalty for bind-mounts, see https://github.com/containers/podman/issues/16994 . 

As of 25.1.24, virtualization with [Apple Hypervisor](https://developer.apple.com/documentation/hypervisor) is available with the [Podman Desktop pre-release 1.7.3](https://github.com/containers/podman-desktop/releases/tag/v1.7.0)

This Gist is supposed to keep track of how to install it and test it with Docker Compose and Testcontainers.

# Known problems of 4.9.0
Beware that as of 26.1.24, https://github.com/containers/podman/issues/21288 means that a **reboot of the host (e.g. laptop) will invalidate the applehv machine**, and the only known workaround is to delete and recreate it! The fix is already merged into main, but hasn't made it into 4.9.0

# Installing Podman with applehv virtualization
As recommended by @rhatdan [^1] , for testing purposes it's better to use the Podman Desktop .img installer or Podman .pkg installer to rule out issues with brew packaging. 

Beware that if you install via .dmg, and when you don't have podman installed via brew, that the .dmg will install a .pkg with Podman CLI. The latter then can only be uninstalled *manually*, https://github.com/containers/podman-desktop/discussions/5735

Installation steps[^2]:

1. Uninstall anything podman-related with brew uninstall
   - `brew uninstall podman podman-desktop`
1. Remove any traces of previous Podman installations, including VMs (**i.e. this will delete your existing VMs!**)
```
rm -rf ~/.config/containers/
rm -rf ~/.local/share/containers
rm ~/.ssh/podman*
```
3. Currently must manually install vfkit, because .dmg installer currently misses it[^3]:
```
brew tap cfergeau/crc
brew install vfkit
```
   - see https://github.com/crc-org/vfkit/releases for current vfkit release

From here you might either want to go with an official release installer, or, if you are adventurous, try a Podman 5 pre-release built from HEAD of the Podman main branch. The latter was surprisingly stabe for me, and you might evade bugs that are already fixed on the main branch.

## Use Podman 4.9.x from installer

4. Download Podman Desktop .dmg from [Podman Desktop Releases](https://github.com/containers/podman-desktop/releases) , preferrably 1.7+
   - alternatively for CLI only, download .pkg installer from [Podman releases](https://github.com/containers/podman/releases) (_not_ Desktop), preferrably 4.9.0+
4. install it

## Use Podman 5.0.0-dev via brew HEAD
This will build podman from its Github main branch, and that seems to be the only way to get a current build of Podman on Mac. The result was surprisingly stable for me!

4. `brew install podman --HEAD`

This will ony install Podman CLI. You can combine this with a recent Podman Desktop build from https://github.com/containers/podman-desktop/releases , as that will detect Podman CLI being installed already (via brew) and leave it untouched.

## Create the applehv machine

6. create applehv machine and start it:
```
export CONTAINERS_MACHINE_PROVIDER=applehv
podman machine init
podman machine start
```
7. Add applehv to Podman config
Add applehv to Podman config
```
# Add the 2 lines below below to ~/.config/containers/containers.conf
[machine]
Add applehv to Podman config
provider="applehv"
```
# Make applehv machine rootful
Around using Docker Compose + Testcontainers, it turned out to be necessary at some point to run Podman rootful
```
podman machine stop
podman machine set --rootful
podman system connection default podman-machine-default-root
podman machine start
```

# Using Docker Compose
You can use `docker-compose` installed via brew with `podman` as a substitute for `docker`.

## Preconditions

* Follow [Using `podman-mac-helper` on macOS | Podman Desktop (podman-desktop.io)](https://podman-desktop.io/docs/migrating-from-docker/using-podman-mac-helper)
* must set DOCKER_HOST, 
  - but in contrast to what [Using the `DOCKER_HOST` environment variable | Podman Desktop (podman-desktop.io)](https://podman-desktop.io/docs/migrating-from-docker/using-the-docker_host-environment-variable) says, must set it to the rootful connection reported by `podman system connection list` , e.g. `ssh://root@127.0.0.1:54328/run/podman/podman.sock` ![^4]

* must symlink docker executable, do not simply alias it or e.g. exec-maven-plugin won’t find docker command
  - `sudo ln -s /opt/podman/bin/podman /usr/local/bin/docker`

## Docker Compose Troubleshooting
### Problem: bind mount volume via Docker Compose cannot be accessed by container: 
e.g. Java container shows `Caused by: java.nio.file.AccessDeniedException: /folder-mount`

[2) Can't use volume mount, get permission denied in podman/troubleshooting.md](https://github.com/containers/podman/blob/main/troubleshooting.md#2-cant-use-volume-mount-get-permission-denied) says 
> This is sometimes caused by SELinux, and sometimes by user namespaces[^5].


SELinux can be disabled in the `docker-compose.yaml` as described in
[How to disable SE Linux from docker compose : r/docker (reddit.com)](https://www.reddit.com/r/docker/comments/15cw6w3/how_to_disable_se_linux_from_docker_compose/):

In `docker-compose.yaml`, within `services/<service-name>/`:
```
security_opt:
 - "label=disable"
```

If you don't want or aren't able to amend your `docker-compose.yaml`, then SELinux can also be disabled completely in the machine, see [How to disable SELinux in Fedora CoreOS - TechOverflow](https://techoverflow.net/2020/12/28/how-to-disable-selinux-in-fedora-coreos/) 

Unfortunately, turning SELinux off completely is necessary for Testcontainers w/ Docker Compose, see below.

### Problem: Permission denied (publickey,gssapi-keyex,gssapi-with-mic)
When issuing `docker-compose --file ... up`:
```
error during connect: Get "http://docker.example.com/v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dws-magnolia-webapp%22%3Atrue%7D%7D": command [ssh -o ConnectTimeout=30 -l root -p 49671 -- 127.0.0.1 docker --host unix:///run/podman/podman.sock system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=root@127.0.0.1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
```
Pruning all unused containers surprisingly helped me once here.

# Using Testcontainers
## Testcontainers troubleshooting
This is with a simple `org.testcontainers.containers.ComposeContainer` to run a custom `docker-compose.yaml`: 
```java
    private static ComposeContainer compose;
    
    @BeforeAll
    public static void startContainers() {
        compose = new ComposeContainer(
                new File("docker-compose.yaml"));
        compose.start();
    }
```
### Status 500: make cli opts(): making volume mountpoint for volume .. podman.sock: operation not supported

When run from Maven and debugged:
```
{"cause":"operation not supported","message":"make cli opts(): making volume mountpoint for volume /Users/joerg.frantzius/.local/share/containers/podman/machine/applehv/podman.sock: mkdir /Users/joerg.frantzius/.local/share/containers/podman/machine/applehv/podman.sock: operation not supported","response":500}
```
```
{"cause":"changing to intended-new-root directory \"/private/tmp/.testcontainers-tmp-5348608840095356915\": chdir /private/tmp/.testcontainers-tmp-5348608840095356915: not a directory","message":"error in copier subprocess: changing to intended-new-root directory \"/private/tmp/.testcontainers-tmp-5348608840095356915\": chdir /private/tmp/.testcontainers-tmp-5348608840095356915: not a directory","response":500}
```

```
	Caused by: com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.NoHttpResponseException: localhost:2375 failed to respond
		at com.github.dockerjava.zerodep.shaded.org.apache.hc.core5.http.impl.io.DefaultHttpResponseParser.createConnectionClosedException(DefaultHttpResponseParser.java:87)
```

When Junit Test is run inside Eclipse Debugger:
```
14:24:45.368 [main] ERROR tc.docker:24.0.2 - Could not start container
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"cause":"operation not permitted","message":"lsetxattr /var/run/docker.sock: operation not permitted","response":500}
```
When disabling SELinux entirely in the machine, running from Maven still yields:
```
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"cause":"operation not supported","message":"make cli opts(): making volume mountpoint for volume /Users/joerg.frantzius/.local/share/containers/podman/machine/applehv/podman.sock: mkdir /Users/joerg.frantzius/.local/share/containers/podman/machine/applehv/podman.sock: operation not supported","response":500}
```
#### Solution
Must set `DOCKER_HOST` to the rootful connection URI reported by `podman system connection list`, in contrast to what the documentation on `DOCKER_HOST` says
### Status 500: lsetxattr /var/run/docker.sock: operation not permitted
When run in Maven:
```
15:38:22.127 [main] ERROR tc.docker:24.0.2 - Could not start container
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"cause":"operation not permitted","message":"lsetxattr /var/run/docker.sock: operation not permitted","response":500}
```
#### Solution
The solution is to disable SELinux in the entire Podman machine, see [Docker Compose Troubleshooting](#docker-compose-troubleshooting) above.

[^1]: in https://github.com/containers/podman/issues/20776#issuecomment-1867633199
[^2]: mostly copied from @GabrielDillenburg's comment https://github.com/containers/podman/issues/20776#issuecomment-1887085665 )
[^3]: see [Unable to test podman on mac with provider applehv due to vfkit missing · Issue #21064 · containers/podman (github.com)](https://github.com/containers/podman/issues/21064#issuecomment-1908424951) 
[^4]: should file issue for https://github.com/containers/podman-desktop/blob/8f29800fb18e2adf6445dba1e3ee703d0b6b2d6e/website/docs/migrating-from-docker/using-the-docker_host-environment-variable.md?plain=1#L77 
[^5]: this is either same as ['Permission denied' on volumes in Mac OS with podman machine · Issue #17560](https://github.com/containers/podman/issues/17560), or SELinux has to be disabled