2717 views
owned this note
# OCI Containers on FreeBSD
> Work-in-progress notes, these will move to the handbook once ready.
The [Open Container Initiative], commonly referred to as `OCI`, provides
a vendor and OS-agnostic way to describe, distribute, and run containers.
The OCI specifications provide these in a way that can be used on many
different operating systems.
[14.3-RELEASE] and upwards, including [snapshots], now include
OCI-compatible images, and the [Podman] toolkit on FreeBSD is ready to
use them, on both amd64 and arm64 architectures.
For FreeBSD users familiar with jails, there is a loose analogy:
- FreeBSD's `base.txz` tarball is an example of a container image
- the `jail.conf` file describes the desired container properties,
or `Containerfile`
- use the `jail .. ` command to run a container, given a filesystem
path, with the `podman` suite of tools
By importing this container stack, FreeBSD users both benefit from common
tooling, but also enjoy wide support across public and private container
registries, and container-specific tooling and services.
In the [aarch64] and [amd64] download directories, you'll see 3 official
OCI-format images. The naming may be a little confusing at first, but
should make sense once you start using them.
The same images are also available through common public container
registries, including [Docker Hub], and [Github Container Registry],
but for the strongest chain of trust, you should fetch your image
directly from [Official FreeBSD Releases].
Each image comprises a subset of a standard FreeBSD `base.txz` release
tarball, for various use cases, and the usual FreeBSD `CHECKSUM.*` files,
that can be verified against the PGP-signed release announcement. This
gives a very strong chain of provenance, directly verifiable from the
FreeBSD release team.
```
CHECKSUM.SHA256
CHECKSUM.SHA512
FreeBSD-14.3-RELEASE-amd64-container-image-dynamic.txz
FreeBSD-14.3-RELEASE-amd64-container-image-static.txz
FreeBSD-14.3-RELEASE-amd64-container-image-runtime.txz
```
# Container Image Naming and Tagging
## Introduction
FreeBSD's official container images are built from base system packages,
themselves built during the FreeBSD release process, and published to
public registries.
This guide explains how images are named and tagged, helping you
choose the right image for your needs, and ensuring you understand the
implications of each choice, as new images are published, and in some
cases, mutable tags are amended.
This naming convention aligns with the new base system package naming
scheme, giving clear provenance between container images and the
corresponding FreeBSD releases.
## Available Image Types
Three types of FreeBSD container images are available, for `amd64` and
`aarch64` architectures, as well as `riscv64` and `i386` also.
### `freebsd-static`
The static image is intended as a base image, for a workload which
is entirely statically linked. It contains no libraries, nor binaries,
just the supporting files that most applications of this nature
require.
- public TLS certificates
- minimal password file
- minimal termcap
- timezone database
Note that it has, by design, a lean footprint, to make a security
compromise of the container less useful to the attacker. There’s no
UNIX shell, no command-line tools, no dynamic libraries, nor package
manager.
It is the smallest image provided by the FreeBSD project.
### `freebsd-dynamic`
The dynamic image uses the static image as a parent layer, and
supports using shared libraries, including `libc`.
Most FreeBSD software should run without issue with this image, with
minor changes.
It doesn't have a shell, rc system, nor a package manager. These
limitations are additional security, making it awkward for attackers
in a compromised container to move laterally, or make use of tools
that were never installed.
### `freebsd-runtime`
Again, runtime builds on the preceding dynamic layer, and finally
adds the minimum that a user would expect - a UNIX shell, rc system,
and the package manager.
It is the ideal base image for porting existing applications with a
minimum of changes. Users will need to include additional FreeBSD
base system libraries, as well as additional packages from the Ports
tree.
This is the closest to a typical FreeBSD system, including the `pkg(8)`
tool, allowing users to install, or do, almost anything, similar to
a non-containerised system.
## Image Tag Structure
All images follow a consistent naming pattern, derived from the
FreeBSD release they are based upon.
### Immutable Tags
Immutable tags never change and are ideal for production systems where
you need stability and predictability.
These are the most common tags used by the release process. They will
not change, even after FreeBSD security patches, or errata notices.
They are ideal for base images for software deployments where a high
degree of reproducibility is expected, even as a trade-off against
more pro-active security patching.
- `major.minor` (e.g., `14.2`): Points to a specific FreeBSD RELEASE
version
- `major.minor.pX` (e.g., `14.2p1`): Points to a specific patch level
of a RELEASE
- `major.snapYYYYMMDDHHMMSS` (e.g., `14.snap20250325035941`): a
snapshot build with timestamp
### Mutable Tags
Mutable tags are updated over time and are ideal for production, or
CI/CD systems that always want the latest updates, but still wish to
align with FreeBSD ABI compatibility.
- `major` (e.g. `14`) will always be the latest support RELEASE version
along the 14.x stable ABI
- `major.snap` (e.g., `14.snap`) follows the latest developer snapshot
along the 14.x stable ABI train, will often be in advance of
official releases, and carries a risk of breakage, incompatibility,
or other unanticipated issues. This is the most mutable tag, and
should be used with caution, exactly the same as using FreeBSD's
STABLE or CURRENT snapshots, that they are equivalent to.
## When to Use Each Tag Type
### For Production
Use immutable tags to ensure your environment stays consistent:
- `freebsd-runtime:14.2` - pinned to FreeBSD 14.2-RELEASE
When security patches are released, you must explicitly rebuild or
update with a new image tag.
### For CI/CD and Development
Use mutable tags to automatically get the latest updates:
- `freebsd-runtime:14` - always the latest official RELEASE in the 14.x series
These additional mutable tags will follow the regular published snapshots
off FreeBSD STABLE and CURRENT. Use these if you want to test against what
subsequent future releases of FreeBSD may contain, and accept occasional
breakage or unanticipated changes.
- `freebsd-runtime:14.snap` - always the latest build from 14-STABLE
branch
- `freebsd-runtime:15.snap` - always the latest build from 15-CURRENT branch
## Official Registry Locations
Architecture-independent images can be downloaded and directly imported
into your own registry from the [Official FreeBSD Releases] site, or
alternatively, pulled from the FreeBSD project's [Docker Hub], or
[Github Container Registry].
## Comparison to Other Projects
Both [Alpine Linux] and the [Debian] project publish tagged and mutable
images, using similar approaches:
[Alpine Linux]: https://hub.docker.com/_/alpine/
[Debian]: https://hub.docker.com/_/debian/
# A Brief Introduction to Podman
The podman suits comprises 3 main tools and a bunch of supporting ones:
- [podman] is a daemonless container engine using docker-compatible commands
- [buildah] generates OCI formatted container images
- [skopeo] works with remote OCI image registries
## Preparing your System for Podman containers
> If you are running this over ssh, or in a similar remote manner,
> remember to perform these commands in screen, tmux, or mosh to
> avoid losing your connection while restarting your firewall.
If you don't have zfs, switch to ufs, and skip the zfs commands:
```
# sed -I .bak -e 's/driver = "zfs"/driver = "vfs"/' \
/usr/local/etc/containers/storage.conf
```
```
# zfs create -o mountpoint=/var/db/containers zroot/containers
# zfs snapshot zroot/containers@empty
# sysctl net.pf.filter_local=1
# mount -t fdescfs fdesc /dev/fd
```
Amend `sysctl.conf`, and `/etc/fstab` as appropriate, to make the
above changes permanent.
## Installing Podman
Only the `sysutils/podman-suite` meta-package is required, but
if you install the additional `emulators/qemu-user-static` it
is possible to build or test images for other architectures
such as `arm64` on an `amd64` architecture host.
```
# pkg install -r FreeBSD -y podman-suite emulators/qemu-user-static
```
- integrate changes from `/usr/local/etc/containers/pf.conf.sample`
into your `/etc/pf.conf` setting egress macros appropriately,
then restart your firewall via `service pf restart`
- the packages install a number of template config files, none of
which need to be edited immediately. You should review and
amend the template config files. None of these are
initially required though, you can do this later:
```
# pkg list buildah podman conmon \
ocijail containers-common \
containernetworking-plugins \
|grep /etc/
/usr/local/etc/containers/containers.conf.sample
/usr/local/etc/containers/policy.json.sample
/usr/local/etc/containers/registries.conf.sample
/usr/local/etc/containers/storage.conf.sample
/usr/local/etc/containers/pf.conf.sample
```
# A Quick Terminal Tour
> Contrary to podman/docker on Linux, everything needs to run as root,
> for the moment, as all FreeBSD jails do already.
Now that we have the necessary tools and firewall rules in place, let's
take it out for a spin, fetching the officially published images from
[14.3-RELEASE], either [aarch64] or [amd64] as appropriate.
https://download.freebsd.org/releases/OCI-IMAGES/14.3-BETA4/amd64/Latest/
https://download.freebsd.org/releases/OCI-IMAGES/14.3-BETA4/aarch64/Latest/
[aarch64]: https://download.freebsd.org/releases/OCI-IMAGES/14.3-BETA4/aarch64/Latest/
[amd64]: https://download.freebsd.org/releases/OCI-IMAGES/14.3-BETA4/amd64/Latest/
## Import FreeBSD Official Tarballs
It's much easier, to pull these directly from a public container registry,
but for the moment, we will rely on checksums and signatures from official
FreeBSD Release Announcements, and download them from [FreeBSD.org]
directly -- Trust but Verify!
```shell
# export OCIBASE=https://download.freebsd.org/releases/OCI-IMAGES/14.3-BETA4/amd64/Latest
# podman load -i=$OCIBASE/FreeBSD-14.3-BETA4-amd64-container-image-runtime.txz
Getting image source signatures
Copying blob f30c10082d05 done |
Copying config ce1e186945 done |
Writing manifest to image destination
Loaded image: localhost/freebsd-runtime:14.3-BETA4-amd64
# podman load -i=$OCIBASE/FreeBSD-14.3-BETA4-amd64-container-image-dynamic.txz
Getting image source signatures
Copying blob d155f7d7a1e3 done |
Copying config f3f3912e67 done |
Writing manifest to image destination
Loaded image: localhost/freebsd-dynamic:14.3-BETA4-amd64
# podman load -i=$OCIBASE/FreeBSD-14.3-BETA4-amd64-container-image-static.txz
Getting image source signatures
Copying blob e5189708856c done |
Copying config 37bd047ffa done |
Writing manifest to image destination
Loaded image: localhost/freebsd-static:14.3-BETA4-amd64
```
## List the Images
```shell
# podman images
podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/freebsd-runtime 14.3-BETA4-amd64 ce1e1869455e 32 hours ago 33 MB
localhost/freebsd-dynamic 14.3-BETA4-amd64 f3f3912e67eb 32 hours ago 13.6 MB
localhost/freebsd-static 14.3-BETA4-amd64 37bd047ffad7 32 hours ago 3.69 MB
```
Note that:
- the image IDs are reproducible each time
- the `IMAGE_ID` column matches the hashes reported above when importing
the images. See `Copying config ...` lines to compare
> NB the hashes won't match until 14.3-RELEASE is actually released
## List the Layers
It is possible to show the layers that comprise an image, showing how the
`runtime` image has 3 components, each one a layer in its own right.
```
podman image tree c5f3e77557a9
Image ID: c5f3e77557a9
Tags: [localhost/freebsd14-runtime:14.3-RELEASE-amd64]
Size: 35.07MB
Image Layers
├── ID: cd53fb07fb66 Size: 5.449MB Top Layer of: [localhost/freebsd14-static:14.3-RELEASE-amd64]
├── ID: a01d37f7777b Size: 10.4MB Top Layer of: [localhost/freebsd14-dynamic:14.3-RELEASE-amd64]
└── ID: 36b0c80ca1f7 Size: 19.21MB Top Layer of: [localhost/freebsd14-runtime:14.3-RELEASE-amd64]
```
For the moment, there is only one `tag` associated with this image.
## Inspect the Checksums
These sha256 digests can be compared across hosts, to ensure you are getting
the expected images. In future FreeBSD release announcements, these digests
will also be published, for convenience.
```shell
# podmanminimal inspect localhost/freebsd14-runtime:14.3-RELEASE-amd64
dma
[
{
"Id": "c5f3e77557a916c124bfb74bdd982efe0a5749aaba9de0d282703d4634b28839",
"Digest": "sha256:b98b8f5438a07e000a156ecb216529d8dfa191448ac3dc1f6add9848b4461d05",
"RepoTags": [
"localhost/freebsd14-runtime:14.3-RELEASE-amd64"
],
"RepoDigests": [
"localhost/freebsd14-runtime@sha256:b98b8f5438a07e000a156ecb216529d8dfa191448ac3dc1f6add9848b4461d05"
],
"Parent": "ebf7538b22f43206d308fecc12bfebc72beb6d1f0e89ce9d95ed1979583010ee",
"Comment": "",
"Created": "2024-11-29T10:34:57.587490327Z",
"Config": {
"Labels": {
"io.buildah.version": "1.36.0"
}
},
"Version": "",
"Author": "",
"Architecture": "amd64",
"Os": "freebsd",
"Size": 35067392,
"VirtualSize": 35067392,
"GraphDriver": {
"Name": "zfs",
"Data": {
"Dataset": "zroot/containers/36b0c80ca1f71afdd5813c5fbd5b55fe686ec6361df1ff36286f2176be67a3fe",
"Mountpoint": "/var/db/containers/storage/zfs/graph/36b0c80ca1f71afdd5813c5fbd5b55fe686ec6361df1ff36286f2176be67a3fe"
}
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:cd53fb07fb66f2c92709cbc407af686fa53276397762dfb27c040a2f64508000",
"sha256:f709c8f6cb8695415033da5819b83262c51cd9512bddb64e0681c48a61b8723f",
"sha256:357c210f8575c70360e88cfcf3d0c241f7e3fa21669ec55bb935dce361a90585"
]
},
"Labels": {
"io.buildah.version": "1.36.0"
},
"Annotations": {
"org.opencontainers.image.base.digest": "sha256:7c1c45d99ffe8344021227ad651f675ac22eeb39b96c8c6abfe2d0612f4c2ccc",
"org.opencontainers.image.base.name": "localhost/freebsd14-dynamic:latest"
},
"ManifestType": "application/vnd.oci.image.manifest.v1+json",
"User": "",
"History": [
{
"created": "2024-11-29T09:55:25.405122405Z",
"created_by": "/bin/sh"
},
{
"created": "2024-11-29T10:34:08.010623771Z",
"created_by": "/bin/sh",
"comment": "FROM localhost/freebsd14-static:latest"
},
{
"created": "2024-11-29T10:34:58.10725353Z",
"created_by": "/bin/sh",
"comment": "FROM localhost/freebsd14-dynamic:latest"
}
],
"NamesHistory": [
"localhost/freebsd14-runtime:14.3-RELEASE-amd64"
]
}
]
```
## Run A Container
This tutorial has been done on a FreeBSD 15.0-CURRENT system, which
conveniently provides a simple way to demonstrate that a container
was indeed run, as the runtime and userland versions of FreeBSD will
be different. As the container image does not include the kernel,
`freebsd-version -k` fails, but the userland variation succeeds.
```
# podman run -it localhost/freebsd-runtime:14.3 /bin/sh
# uname -a
FreeBSD 3fad4ca90729 15.0-CURRENT FreeBSD 15.0-CURRENT main-n273934-3455b824a1e8 GENERIC-NODEBUG amd64
# freebsd-version -k
freebsd-version: unable to locate kernel
# freebsd-version -ru
15.0-CURRENT
14.3-RELEASE
# exit
```
## Useful Variations and Tips
Almost all of the `podman run ...` parameters will also apply to
`podman build`, refer to the excellent podman docs for [podman run](https://man.freebsd.org/podman-run) and [podman build](https://man.freebsd.org/podman-build).
Use the `--rm` flag to have ephemeral containers clean themselves up
afterwards.
Use `podman images -a` to show all downloaded images in `zroot/containers`
```shell
# podman images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
ghcr.io/freebsd/freebsd-runtime 14.3-prerelease 6faeb847b6fb 2 weeks ago 31.9 MB
```
Use `podman ps -a` to see running containers:
```shell
# podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3123623bef9b ghcr.io/freebsd/freebsd-runtime:14.3-prerelease /bin/sh 4 minutes ago Up 4 minutes admiring_brattain
```
### Environment Variables
Arbitrary environment variables can be passed in:
```
# podman run --env FOO=BAR -it --rm ghcr.io/freebsd/freebsd-runtime:14.3
# echo $FOO
BAR
```
### Mounting Files and Volumes
Both files and volumes can be passed into containers. For example
the `runtime` image doesn't contain `which(1)`, but it can be
mounted into the container easily:
```shell
# podman run -it --rm --volume /usr/bin/which \
ghcr.io/freebsd/freebsd-runtime:14.3-beta4
# which which
/usr/bin/which
# which df
/bin/df
```
Or for a very lean image, use the existing `pkg(8)` from the host
and mount it inside the container. We also share the package cache
between container and host, sneak in an environment variable,
and ensure that `/etc/hosts` is not mounted by podman, as it gets
in the way of upgrades.
```shell
# podman run -it --rm \
--env ASSUME_ALWAYS_YES=true \
--no-hosts \
--volume /usr/local/sbin/pkg-static:/bin/pkg-static \
--volume /var/cache/pkg \
ghcr.io/freebsd/freebsd-static:14.3-beta4
# pkg bootstrap -r FreeBSD
Bootstrapping pkg from pkg+https://pkg.FreeBSD.org/FreeBSD:14:amd64/latest, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[fd343e182fa5] Installing pkg-2.1.2...
[fd343e182fa5] Extracting pkg-2.1.2: 100%
# pkg update
Updating FreeBSD repository catalogue...
[fd343e182fa5] Fetching meta.conf: 100% 179 B 0.2kB/s 00:01
[fd343e182fa5] Fetching data.pkg: 100% 10 MiB 701.8kB/s 00:15
Processing entries: 100%
FreeBSD repository update completed. 36036 packages processed.
Updating FreeBSD-base repository catalogue...
[fd343e182fa5] Fetching meta.conf: 100% 179 B 0.2kB/s 00:01
[fd343e182fa5] Fetching data.pkg: 100% 47 KiB 47.7kB/s 00:01
Processing entries: 0%
Processing entries: 100%
FreeBSD-base repository update completed. 525 packages processed.
All repositories are up to date.
# pkg info
FreeBSD-caroot-14.snap20250424064417 SSL Certificates
FreeBSD-certctl-14.snap20250424064417 SSL Certificate Utility
FreeBSD-clibs-14.snap20250424064417 Core C Libraries
FreeBSD-fetch-14.snap20250424064417 Fetch Utility
FreeBSD-kerberos-lib-14.snap20250424064417 Kerberos Libraries
FreeBSD-libarchive-14.snap20250424064417 libarchive package
FreeBSD-libbsm-14.snap20250424064417 libbsm package
FreeBSD-libbz2-14.snap20250424064417 libbz2 package
FreeBSD-libexecinfo-14.snap20250424064417 libexecinfo package
FreeBSD-liblzma-14.snap20250424064417 liblzma package
FreeBSD-libsqlite3-14.snap20250424064417 libsqlite3 package
FreeBSD-libucl-14.snap20250424064417 libucl package
FreeBSD-mtree-14.snap20250424064417 MTREE Files
FreeBSD-openssl-lib-14.snap20250424064417 OpenSSL Libraries
FreeBSD-pkg-bootstrap-14.snap20250424064417 pkg bootstrap Utility
FreeBSD-rc-14.snap20250424064417 RC Scripts
FreeBSD-runtime-14.snap20250424064417 FreeBSD Base System
FreeBSD-zoneinfo-14.snap20250424064417 zoneinfo package
pkg-2.1.2 Package manager
# pkg install FreeBSD-openssl
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating FreeBSD-base repository catalogue...
FreeBSD-base repository is up to date.
All repositories are up to date.
Updating database digests format: 100%
The following 1 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
FreeBSD-openssl: 14.3.b2.20250512234546 [FreeBSD-base]
Number of packages to be installed: 1
314 KiB to be downloaded.
[fd343e182fa5] [1/1] Fetching FreeBSD-openssl-14.3.b2.20250512234546.pkg: 100% 314 KiB 321.2kB/s 00:01
Checking integrity... done (0 conflicting)
[fd343e182fa5] [1/1] Installing FreeBSD-openssl-14.3.b2.20250512234546...
[fd343e182fa5] [1/1] Extracting FreeBSD-openssl-14.3.b2.20250512234546: 100%
# pkg upgrade -y
# pkg install -qy lang/python3
#
```
```shell
# podman run -it --rm \
--volume /etc/fstab:/etc/fstab \
--volume /tmp:/var/tmp \
--volume /tmp \
ghcr.io/freebsd/freebsd-runtime:14.3-prerelease
# mount -p
zroot/containers/20fbf0bac55648163e2ee4670d5cae1e59ae9e1fbf189053b1f78bb285acea64 / zfs rw,noatime,nfsv4acls 0 0
/tmp /tmp nullfs rw 0 0
devfs /dev devfs rw 0 0
/tmp /var/tmp nullfs rw 0 0
fdescfs /dev/fd fdescfs rw 0 0
/var/run/containers/storage/zfs-containers/d62a58b64233daa4ae2ab9bf9c2782ac0ff4381a04bcceb3ae5895169597da95/userdata/resolv.conf /etc/resolv.conf nullfs rw 0 0
/var/run/containers/storage/zfs-containers/d62a58b64233daa4ae2ab9bf9c2782ac0ff4381a04bcceb3ae5895169597da95/userdata/hosts /etc/hosts nullfs rw 0 0
/var/cache/pkg /var/cache/pkg nullfs rw 0 0
/var/run/containers/storage/zfs-containers/d62a58b64233daa4ae2ab9bf9c2782ac0ff4381a04bcceb3ae5895169597da95/userdata/.containerenv /var/run/.containerenv nullfs rw 0 0
```
## Networking & Name Resolution
By default, the Podman Suite of tools will [mount_nullfs(8)] the
`/etc/resolv.conf` and a modified `/etc/hosts` from the jail host,
and a `.containerenv` file in `/var/run` as well.
For example, the previous container might have the following hosts file:
```
# /etc/hosts from jail host
...
10.88.0.1 host.containers.internal host.docker.internal
10.88.0.2 4fc13095288f magical_franklin
```
This hosts file in particular will conflict with base system package
upgrades, so it is advised to skip this during `podman build` runs
if you plan to do this, but leave it on during production deployment.
```shell
### don't mount /etc/resolv.conf from host, use a custom DNS server
# podman run --dns 1.2.3.4 ...
### Don't mount /etc/hosts from hosts, local containers will be hard to find
# podman build --no-hosts ...
```
# Using Public Registries
While the most secure provenance is downloading the [Oficial FreeBSD
Releases], there are 2 public Container Registries that are
managed by the FreeBSD Release and Cluster Admin teams. The images from
both places are identical, but the registries provide a simpler and
more container-friendly usage.
It is expected that most users will interact via these registries:
- [Docker Hub]
- [GitHub Container Registry]
## Fetching and Running from Public Registries
```
# podman run -it --rm ghcr.io/freebsd/freebsd-runtime:14.3-beta1
```
# Building Custom Images
Custom images can be built directly from tarballs (for example those
from [poudriere image], or official release tarballs), from base system
packages, from existing tagged images, or boot-strapped from [pkg(8)].
We will look at each of these options.
## Building Directly Off `base.txz`
A custom image can be made from release tarballs using [podman import].
The final tagged image can be pushed to registries, or modified, and
used for further image creation.
```shell
# podman import --os freebsd \
--arch arm64 \
--message 'Import FreeBSD 14.3-BETA3 base.txz' \
https://download.freebsd.org/ftp/releases/arm64/14.3-BETA3/base.txz
Downloading from "https://download.freebsd.org/releases/arm64/14.3-BETA3/base.txz"
Getting image source signatures
Copying blob e64068a7ec71 done |
Copying config dc9960b6e9 done |
Writing manifest to image destination
sha256:dc9960b6e971b649f933e2a5caa0a087fb81d5aa0b11d9d615e7126e5efd5d63
# podman image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> dc9960b6e971 57 seconds ago 848 MB
# podman image tag dc9960b6e971 localhost/freebsd-base:14.3-beta3
# podman image ls -a
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/freebsd-base 14.3-beta3 dc9960b6e971 About a minute ago 848 MB
```
This image can be used as a further base.
## Building from Tagged Images
```
# Containerfile
# podman build --no-hosts --squash -t localhost/freebsd-base:14.3-beta3 -f ./Containerfile
FROM ghcr.io/freebsd/freebsd-runtime:14.3-beta3
RUN sed -EI -e s/quarterly/latest/ /etc/pkg/FreeBSD.conf
RUN env ASSUME_ALWAYS_YES=yes IGNORE_OS_VERSION=yes pkg bootstrap -r FreeBSD
RUN pkg update
RUN pkg install -y FreeBSD-openssl
RUN pkg upgrade -y
RUN pkg clean -y
ENTRYPOINT /bin/sh
```
```
# podman build --no-hosts -t localhost/python-www:14.3-beta3 -f /root/containers/python/Containerfile
FROM localhost/freebsd-base:14.3beta3
RUN pkg install -y lang/python3
RUN pkg clean -y
ENTRYPOINT /usr/local/bin/python3 -m http.server
```
## Bootstrapping with Base System Packages
# Advanced Usage
## Daemonising
podman can be enabled as a service to restart automatically after a host is rebooted. You can either
```
# service enable podman
# service podman start
# service enable podman_service
# service podman_service start
```
or
```
# sysrc podman_enable=YES
# service podman start
# sysrc podman_service_enable=YES
# service podman_service start
```
additionally, podman can restart containers after a host is rebooted in which case containers need to run with `--restart=always` option set
```
# podman run -d --restart=always my-image
```
## Linux containers
It is possible to run many Linux container images using FreeBSD's Linux
emulation:
```
# service linux onestart
# podman run --rm --os=linux docker.io/alpine cat /etc/os-release | head -1
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 4abcf2066143 done |
Copying config 05455a0888 done |
Writing manifest to image destination
NAME="Alpine Linux"
```
Just don't expect miracles from anything that has systemd dependencies, or
expects to be pid1, or expects to be ... Linux.
## Running a local registry
A registry is a fancy web server for storing and sharing your OCI images.
NB use `/etc/rc.conf.d/registry` for `rc.conf` settings, the `rc.d` script
doesn't match the daemon name.
There is a [docker-registry](https://github.com/distribution/distribution)
port for FreeBSD, while it's rather old now, it's still quite sufficient
for local development or a small organisation, with trivial setup.
[Zot Registry](https://zotregistry.dev/) is a modern comprehensive
registry, providing S3 backends, cluster for redundancy, a precompiled
FreeBSD binary, but as of yet, does not exist in the FreeBSD ports tree.
```
# pkg install -r FreeBSD sysutils/docker-registry
# zfs create zroot/var/db/registry
# chown -R www:www /var/db/registry
# sysrc docker_registry_enable=YES
# sysrc docker_registry_user=www
# sed -i '' -E -e 's,rootdirectory:.+,rootdirectory: /var/db/registry,' \
/usr/local/etc/docker-registry/config.yml
# touch /usr/local/etc/docker-registry/htpasswd
# chown -R root:www /usr/local/etc/docker-registry/{config.yml,htpasswd}
# chmod 0640 /usr/local/etc/docker-registry/{config.yml,htpasswd}
```
- use SHA256 to create a nice random HTTP secret
- use http://aspirine.org/htpasswd_en.html & bcrypt to create `htpasswd` file
> There must be a native tool to create these, what options do we have?
```
# /usr/local/etc/docker-registry/config.yml
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/db/registry
http:
addr: :5000
secret: abc123sha256
headers:
X-Content-Type-Options: [nosniff]
auth:
htpasswd:
realm: basic-realm
path: /usr/local/etc/docker-registry/htpasswd
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
```
- `sudo service registry start`
- watch logs in `/var/log/docker-registry.log` for errors
- run `curl -vu admin:passwd http://localhost:5000/v2/`
you should get a `200 OK` response with `{}` body
See below for local registry usage.
# Building and Sharing Containers
## Building a new OCI container from sources
The FreeBSD release tooling provides an `oci-release` target
to build the necessary images for `podman load` to use. It
requires building world, kernel, and base system packages
first, so this can take a while.
The following example will build `amd64` and `arm64` images:
```shell
# cd /usr/src \
&& make TARGET_ARCH=aarch64 TARGET=arm64 -s -j32 buildworld \
&& make TARGET_ARCH=aarch64 TARGET=arm64 KERNCONF=GENERIC -s -j32 buildkernel \
&& make TARGET_ARCH=aarch64 TARGET=arm64 KERNCONF=GENERIC -s -j32 packages \
&& cd ./release \
&& make TARGET_ARCH=aarch64 TARGET=arm64 -j32 -s clean \
&& make KERNCONF=GENERIC TARGET_ARCH=aarch64 TARGET=arm64 WITH_OCIIMAGES=yes \
-s oci-release
# cd /usr/src \
&& make TARGET_ARCH=amd64 TARGET=amd64 -s -j32 buildworld \
&& make TARGET_ARCH=amd64 TARGET=amd64 KERNCONF=GENERIC -s -j32 buildkernel \
&& make TARGET_ARCH=amd64 TARGET=amd64 KERNCONF=GENERIC -s -j32 packages \
&& cd ./release \
&& make TARGET_ARCH=amd64 TARGET=amd64 -s -j32 clean \
&& make KERNCONF=GENERIC TARGET_ARCH=amd64 TARGET=amd64 \
WITH_OCIIMAGES=yes -s oci-release
```
- TODO where do these images sit?
## Listing Local Images
- note that `latest` and `14.0p6` tags match up with the same `image id`
```
# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/freebsd14.0-small 14.0p6 6b60451da926 5 hours ago 1.03 kB
localhost/freebsd14.0-small latest 6b60451da926 5 hours ago 1.03 kB
localhost/freebsd14.0-runtime 14.0p6 5faaecc767ad 5 hours ago 1.03 kB
localhost/freebsd14.0-runtime latest 5faaecc767ad 5 hours ago 1.03 kB
localhost/freebsd14.0-base latest-debug d57507702c2b 5 hours ago 1.03 kB
localhost/freebsd14.0-base 14.0p6-debug d57507702c2b 5 hours ago 1.03 kB
localhost/freebsd14.0-base 14.0p6 977dfe21ef15 5 hours ago 1.03 kB
localhost/freebsd14.0-base latest 977dfe21ef15 5 hours ago 1.03 kB
localhost/freebsd14.0-static latest-debug 8964f979748d 5 hours ago 1.03 kB
localhost/freebsd14.0-static 14.0p6-debug 8964f979748d 5 hours ago 1.03 kB
localhost/freebsd14.0-static latest 2018593833af 5 hours ago 1.03 kB
localhost/freebsd14.0-static 14.0p6 2018593833af 5 hours ago 1.03 kB
localhost/freebsd14.0-mtree latest b91fa618e394 5 hours ago 1.03 kB
localhost/freebsd14.0-base latest-aarch64 25e0d6010087 21 hours ago 35.1 MB
localhost/freebsd14.0-runtime latest-aarch64 7492fba91606 21 hours ago 61.5 MB
localhost/freebsd14.0-base latest-debug-amd64 28358b486bc6 21 hours ago 55.1 MB
localhost/freebsd14.0-base latest-debug-aarch64 1b6f8b32ed55 21 hours ago 52.7 MB
localhost/freebsd14.0-static latest-debug-amd64 01398919565e 21 hours ago 23.8 MB
localhost/freebsd14.0-base latest-amd64 cc353a2011ef 21 hours ago 35.7 MB
localhost/freebsd14.0-runtime latest-amd64 ca1f3dd2d649 21 hours ago 62.8 MB
localhost/freebsd14.0-mtree latest-amd64 05cfc4ff7d55 21 hours ago 1.17 MB
localhost/freebsd14.0-static latest-aarch64 b444483b7b3b 21 hours ago 6.01 MB
localhost/freebsd14.0-static latest-amd64 38a855def155 21 hours ago 6.02 MB
localhost/freebsd14.0-small latest-amd64 0af3b8c4c2e7 21 hours ago 175 MB
localhost/freebsd14.0-small latest-aarch64 31e93be59727 21 hours ago 164 MB
localhost/freebsd14.0-static latest-debug-aarch64 392a902558fc 21 hours ago 22 MB
localhost/freebsd14.0-mtree latest-aarch64 005682383153 21 hours ago 1.17 MB
quay.io/dougrabson/hello latest f81c971736c6 23 months ago 4.06 MB
```
## Create and Run
My PC is running a pre-release `15.0-CURRENT` but the built container is
based off a `14.0-RELEASE` image, so [freebsd-version(1)] will return
different results for running kernel vs installed userland. That's ok!
```
# buildah from localhost/freebsd14.0-runtime
freebsd14.0-runtime-working-container
# buildah run freebsd14.0-runtime-working-container freebsd-version -ru
15.0-CURRENT <------ my desktop's kernel
14.0-RELEASE-p6 <------ the container's userland
```
This image is made up of multiple layers, only the final one contains the main userland.
These layers are stored
```
# podman image tree localhost/freebsd14.0-small
Image ID: 4bc70197ea21
Tags: [localhost/freebsd14.0-small:latest-amd64]
Size: 174.8MB
Image Layers
├── ID: 1643796dfb88 Size: 1.172MB Top Layer of: [localhost/freebsd14.0-mtree:latest-amd64 localhost/freebsd14.0-mtree:latest-aarch64]
├── ID: 8a0cc7febb16 Size: 4.847MB Top Layer of: [localhost/freebsd14.0-static:latest-amd64]
├── ID: 3802ebf80a0f Size: 29.69MB Top Layer of: [localhost/freebsd14.0-base:latest-amd64]
├── ID: 0c5e170468cb Size: 27.13MB Top Layer of: [localhost/freebsd14.0-runtime:latest-amd64]
└── ID: 1c354111d4b5 Size: 112MB Top Layer of: [localhost/freebsd14.0-small:latest-amd64]
```
## Registries
In general, registries require authentication for push, and optionally for
pull. Credentials are saved in `/root/.config/containers/auth.json` by
default, if you login, but it is also possible to enter the password each
time manually, to avoid it being stored on the filesystem.
It is recommended that you serve your registry over TLS, behind nginx,
haproxy, or similar.
## Github
This container registry is namespaced by github accounts and organisations.
### Organisation Changes
- for each organisation, or user, that you wish to publish packages for,
enable `public` [permissions in that organisation](https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility)
- set up permissions on your [github organisation](https://github.com/organizations/freebsd/settings/packages) to enable public packages
- we will not link packages to repos at this stage, it requires automated builds
### Personal Changes
Each user who can upload, requires a personal access token, or [PAT](https://github.com/settings/tokens)
- make a PAT, with `write:packages` scope permissions, delete is not required
- this can be used to login to the Github Container registry
- after uploading each package (see below for `buildah push`, change its visibility
to public, and grant the group access
### Push An Image
```
# podman login ghcr.io -u dch
Password:
Login Succeeded!
# chown 0600 /root/.config/containers/auth.json
# buildah push ghcr.io/skunkwerks/freebsd14.0-small:14.0p6
Error: pushing image "ghcr.io/skunkwerks/freebsd14.0-small:14.0p6" to "docker://ghcr.io/skunkwerks/freebsd14.0-small:14.0p6": ghcr.io/skunkwerks/freebsd14.0-small:14.0p6: image not known
# buildah push freebsd14.0-small:14.0p6 docker://ghcr.io/skunkwerks/freebsd14.0-small:14.0p6
Getting image source signatures
Copying blob d6344e14ba8a done |
Copying blob aca51be9eadf done |
Copying blob 05a151490b6d done |
Copying blob 42509cf52690 done |
Copying blob a94388b7abab done |
Copying config 0af3b8c4c2 done |
Writing manifest to image destination
```
And now let's push all of the local ones, using fish shell:
```
# for i in (podman images -n | cut -wf 1 |grep freebsd \
sed -e 's,localhost/,,')
buildah push $i docker://ghcr.io/skunkwerks/$i
end
...
Copying blob a18920fa617e skipped: already exists
Copying blob 188e5e166924 skipped: already exists
Copying config 38a855def1 done |
Writing manifest to image destination
Getting image source signatures
Copying blob a18920fa617e skipped: already exists
Copying config 05cfc4ff7d done |
Writing manifest to image destination```
```
### Pull And Run
You don't need to authenticate to fetch public images.
```
# buildah from ghcr.io/skunkwerks/freebsd14.0-runtime
Trying to pull ghcr.io/skunkwerks/freebsd14.0-runtime:latest...
Getting image source signatures
Copying blob 31fba944001f done |
Copying blob a18920fa617e done |
Copying blob 188e5e166924 done |
Copying blob 8bf3dc12f14d done |
Copying config ca1f3dd2d6 done |
Writing manifest to image destination
freebsd14.0-runtime-working-container
# buildah run freebsd14.0-runtime-working-container freebsd-version -ru
14.0-RELEASE-p5
14.0-RELEASE-p6
# buildah run freebsd14.0-runtime-working-container ifconfig
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xffffff00
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xf
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
eth0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:55:75:36:69:0b
inet 10.88.0.6 netmask 0xffff0000 broadcast 10.88.255.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
# buildah run freebsd14.0-runtime-working-container ping -c 3 100.64.0.1
PING 100.64.0.1 (100.64.0.1): 56 data bytes
64 bytes from 100.64.0.1: icmp_seq=0 ttl=64 time=0.197 ms
64 bytes from 100.64.0.1: icmp_seq=1 ttl=64 time=0.299 ms
64 bytes from 100.64.0.1: icmp_seq=2 ttl=64 time=0.173 ms
--- 100.64.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.173/0.223/0.299/0.055 ms
# buildah run freebsd14.0-runtime-working-container df -h /
Filesystem Size Used Avail Capacity Mounted on
zroot/containers/1d9516c0e6ae2ba2b83b148d34bfafcb39a28db282778fb0928bf60fa901c23b 961G 33M 961G 0% /
```
## Using Oracle Cloud Registry
- confusingly also named `OCI` for funsies, we will call it `OCR` instead
- create a registry https://docs.oracle.com/en/middleware/fusion-middleware/12.2.1.4/ikedg/using-oci-container-registry.html
& https://docs.oracle.com/en-us/iaas/Content/Registry/Tasks/registrycreatingarepository.htm
- I recommend disabling auto-repo creation as it gets confusing if
incorrect names or paths are used for images
```
$ export CID=ocid1.tenancy.oc1..aaaaaaaaobmlhru...
$ oci artifacts container repository create \
--display-name skunkwerks \
--is-public yes \
--compartment-id $CID
```
```json
{
"data": {
"billable-size-in-gbs": 0,
"compartment-id": "ocid1.tenancy.oc1..aaaaaaaaobml",
"created-by": "ocid1.user.oc1..aaaaaaaaodt",
"defined-tags": {},
"display-name": "skunkwerks",
"freeform-tags": {},
"id": "ocid1.containerrepo.oc1.eu-amsterdam-1.0.axvxsnomswgi.aaaaaaaakr",
"image-count": 0,
"is-immutable": false,
"is-public": true,
"layer-count": 0,
"layers-size-in-bytes": 0,
"lifecycle-state": "AVAILABLE",
"namespace": "axvxsnomswgi",
"readme": null,
"system-tags": {},
"time-created": "2024-04-04T14:55:44.163000+00:00",
"time-last-pushed": null
},
"etag": "ockblikdf5cn7usf46of6dujdikfsa-1"
}
```
### Authenticate
- [region-specific hostnames](https://docs.oracle.com/en-us/iaas/Content/Registry/Concepts/registryprerequisites.htm)
- Amsterdam https://eu-amsterdam-1.ocir.io | https://ams.ocir.io
- Frankfurt https://eu-frankfurt-1.ocir.io | https://fra.ocir.io
- weird usernames `<tenancy-namespace>/oracleidentitycloudservice/<your@email>`
- password is your OCI auth token (a very dangerous secret)
```
# podman login ams.ocir.io
Username: axvxsnomswgi/oracleidentitycloudservice/dch@...
Password: .....
Login Succeeded!
```
### Push Images
As before, this is straightforwards:
```
# buildah push freebsd14.0-small:14.0p6 docker://ams.ocir.io/axvxsnomswgi/freebsd/freebsd14.0-small:14.0p6
Getting image source signatures
Copying blob a94388b7abab done |
Copying blob 05a151490b6d done |
Copying blob aca51be9eadf done |
Copying blob d6344e14ba8a done |
Copying blob 42509cf52690 done |
Copying config 0af3b8c4c2 done |
Writing manifest to image destination
# for i in (podman images -n | cut -wf 1 |grep freebsd | sed -e 's,localhost/freebsd,,'); buildah push freebsd$i docker://ams.ocir.io/axvxsnomswgi/freebsd:$i; end
...
Getting image source signatures
Copying blob 8bf3dc12f14d skipped: already exists
Copying blob a18920fa617e skipped: already exists
Copying blob 31fba944001f skipped: already exists
Copying blob 3ba10043c0e3 skipped: already exists
Copying blob 188e5e166924 skipped: already exists
Copying config 0af3b8c4c2 done |
Writing manifest to image destination
```
### Pull And Run
```
# export c=$(buildah from ams.ocir.io/axvxsnomswgi/freebsd:14.0-small)
freebsd-working-container
# buildah run $c df -h /
Filesystem Size Used Avail Capacity Mounted on
zroot/containers/1d9516c0e6ae2ba2b83b148d34bfafcb39a28db282778fb0928bf60fa901c23b 961G 33M 961G 0% /
```
########################################
# after this it's .... just messy notes
# TODO
- [x] quick setup & usage
- [x] setup & using a local repository
- [x] setup & using a remote repository (GitHub, OCI, .. others? )
- [ ] document podman runtime hooks
- [ ] export & import containers as files
- [ ] using signatures and hashes to verify container provenance
- [x] using buildah to make custom images
- [ ] keeping local & remote repositories in sync with [skopeo](https://redhat-scholars.github.io/containers-tutorial/containers-tutorial/skopeo.html)
- [ ] update [sysutils/docker-registry](https://freshports.org/sysutils/docker-registry)
- [ ] creating containers in build pipelines (what CI?)
- [ ] handling multi-arch repos and builds
- [x] naming things is hard, what makes sense, how do tags relate to updates?
- [ ] how to assign specific IPv4/IPv6 addresses to containers?
- [ ] attaching custom nested zfs datasets into containers
- [ ] Create a diagram depicting how all the pieces come together
- [podman](https://github.com/containers/podman)
- [image](https://github.com/containers/image)
- [conmon](https://github.com/containers/conmon)
- [ocijail](https://github.com/dfr/ocijail)
- [ ] Further topics?
- [ ] Redo everything using [runj](https://github.com/samuelkarp/runj) + [containerd](https://github.com/containerd/containerd)
- [ ] Redo everything using [runj](https://github.com/samuelkarp/runj) + [containerd](https://github.com/containerd/containerd) + [nerdctl](https://github.com/containerd/nerdctl)
- [ ] What about other FreeBSD distros ?
- [ ] HardenedBSD
- [ ] check licenses of podman-suite ports (GPL vs ALv2)
- [ ] should the man pages refer to FreeBSD containers? is
this a lot of effort to maintain over time? Upstream?
- [ ] make sure docs/packages/podman are consistent
for example [Installing on FreeBSD 14.0](https://podman.io/docs/installation#installing-on-freebsd-140)
- [ ] support podman rootless mode on FreeBSD (lot of work)
- [ ] document default and useful configs for [containers-common](https://cgit.freebsd.org/ports/tree/sysutils/containers-common)
so that e.g. `podman search` finds stuff in registries
- **Issues**
- ~~After [installing podman](https://podman.io/docs/installation#installing-on-freebsd-140) and applying the [initial configuration](https://podman.io/docs/installation#initial-configuration) but before configuring [networking](https://podman.io/docs/installation#networking) and [storage](https://podman.io/docs/installation#storage), running [podman search](https://docs.podman.io/en/latest/markdown/podman-search.1.html) without adding any registry prefix leads no reults~~ - See **Improvements**
```
# podman search httpd
```
## Find and Fetch
```
# podman image search --list-tags docker.io/dougrabson/hello
NAME TAG
docker.io/dougrabson/hello latest
```
## Fetch and Run It
```
podman run --rmi docker.io/dougrabson/hello
Trying to pull docker.io/dougrabson/hello:latest...
Getting image source signatures
Copying blob b13a5ec7f3d2 done |
Copying config f81c971736 done |
Writing manifest to image destination
!... Hello Podman World ...!
.--"--.
/ - - \
/ (O) (O) \
~~~| -=(,Y,)=- |
.---. /` \ |~~
~/ o o \~~~~.----. ~~
| =(X)= |~ / (O (O) \
~~~~~~~ ~| =(Y_)=- |
~~~~ ~~~| U |~~
Project: https://github.com/containers/podman
Website: https://podman.io
Documents: https://docs.podman.io
Twitter: @Podman_io
```
## Inspect the Stash
```
# fd -tf . /var/db/containers/storage/
/var/db/containers/storage/db.sql
/var/db/containers/storage/defaultNetworkBackend
/var/db/containers/storage/storage.lock
/var/db/containers/storage/userns.lock
/var/db/containers/storage/zfs-containers/containers.lock
/var/db/containers/storage/zfs-containers/dac733521be73f95b7346dbe43562aadc7266c5205c996311ce5dec483bf8725/userdata/buildah.json
/var/db/containers/storage/zfs-containers/volatile-containers.json
/var/db/containers/storage/zfs-images/f81c971736c66436f90455b90e866f7e80d09ca6abda4d818abc589cd4eb05ee/=bWFuaWZlc3Qtc2hhMjU2OjI3YTBlYzhjMmRhODZkZjE0YjRjZWI3MDEwNDE0OTlmNWEzOTBmNzJjY2RmNjc2OWM2NzdjOWZmNjEyNDhmOGU=
/var/db/containers/storage/zfs-images/f81c971736c66436f90455b90e866f7e80d09ca6abda4d818abc589cd4eb05ee/=bWFuaWZlc3Qtc2hhMjU2OmU1ZjI0Y2RmMDIyZTIzYzE5OGYwODE5NDRkMWNjNzg5NzVkY2FkYzQxOTE3MDRjOWFiYjc2MzgzMjY5NjhkZmU=
/var/db/containers/storage/zfs-images/f81c971736c66436f90455b90e866f7e80d09ca6abda4d818abc589cd4eb05ee/=c2hhMjU2OmY4MWM5NzE3MzZjNjY0MzZmOTA0NTViOTBlODY2ZjdlODBkMDljYTZhYmRhNGQ4MThhYmM1ODljZDRlYjA1ZWU=
/var/db/containers/storage/zfs-images/f81c971736c66436f90455b90e866f7e80d09ca6abda4d818abc589cd4eb05ee/manifest
/var/db/containers/storage/zfs-images/images.json
/var/db/containers/storage/zfs-images/images.lock
/var/db/containers/storage/zfs-layers/c0e5c047951147a952903acd94d5044a912a8390be3f3f26d3d066b275ae7234.tar-split.gz
/var/db/containers/storage/zfs-layers/layers.json
/var/db/containers/storage/zfs-layers/layers.lock
/var/db/containers/storage/zfs-layers/volatile-layers.json
```
- TODO how can we see whats in the layers?
## Signing Images (WIP)
**DRAFT - NOT FINAL YET**
- Registeries are not responsible for signing container images or verifying their signatures
- Who then is responsible ?
- What are the methods of Signing:
- :white_check_mark: GNU Privacy Guard ([GPG](https://gnupg.org/))
- :white_check_mark: [How to sign and distribute container images using Podman](https://github.com/containers/podman/blob/main/docs/tutorials/image_signing.md)
- [Sigstore](https://www.sigstore.dev/) (Cosign [1](https://docs.sigstore.dev/signing/quickstart/) [2](https://github.com/sigstore/cosign/tree/main/doc) ?)
- What should be then the flow of:
- Signing ?
- Distributing/Publishing Signatures ?
- Verifying Signatures ?
## Local Registry
Using the prior installation, login, and push an image:
```
### the typical release artifacts
# podman push c5f3e77557a9 ghcr.io/freebsd/freebsd-runtime:14.3p0
# podman push 7876fe59dbb3 ghcr.io/freebsd/freebsd-static:14.3p0
# podman push ebf7538b22f4 ghcr.io/freebsd/freebsd-dynamic:14.3p0
```
### Immutable Tags & Official Releases
pkg-base is the source for building OCI images, so let's keep
the nomenclature as close as possible. Peek in these URLs to
see what's what:
- https://pkg.freebsd.org/FreeBSD:14:amd64/base_release_2/
- FreeBSD-clibs-14.3p2.pkg
- https://pkg.freebsd.org/FreeBSD:14:amd64/base_latest/
- FreeBSD-clibs-14.snap20250325035941.pkg
- https://pkg.freebsd.org/FreeBSD:14:aarch64/base_release_2/
- FreeBSD-clibs-15.snap20250323122233.pkg
- https://pkg.freebsd.org/FreeBSD:15:amd64/base_latest/
- FreeBSD-clibs-15.snap20250323120318.pkg
All images can be clearly mapped by the MAJOR version to a FreeBSD
release cycle:
- no minor version -> mutable tag, latest off the RELENG train (14.3 now, soon 14.3)
- minor & major -> immutable tag, official RELEASE
- minor & major & patchlevel -> immutable tag, official RELEASE
- is that actually true? maybe emaste@ knows?
- which team owns this and is it a once-off after each patch release?
- we dont want a weekly checksum that changes
- `freebsd-runtime:14.3` is the official RELEASE
- `freebsd-runtime:14` is sliding RELENG train
- `freebsd-runtime:XX.latest` -> you're using a STABLE or CURRENT
### Fixed Tags for Deploy (Paranoid Ops)
- the major release cycle starts with `freebsd-runtime:14.0`, then `freebsd-minimal:14.1`, etc
- the`freebsd-runtime:14.x` tag does not move
- WYSIWYG
- you need to choose to patch
- this means rebuilding the image
- we get the sweet patches with `FreeBSD-14.3p1.pkg`
- rebuild your image to move from `freebsd-runtime:14.3` -> `freebsd-minimal:14.2p1`
### Mutable Tags For Releases (CI use case)
- the`freebsd-runtime:14` will slide along the RELENG train
- we start off on 14.3-RELEASE = `FreeBSD-14.2.pkg`
- `freebsd-runtime:14.3` -> `FreeBSD 14.2-RELEASE`
- then we get the sweet patches with `FreeBSD-14.3p1.pkg`
- `freebsd-runtime:14` -> `FreeBSD 14.3-RELEASE-p1`
- eventually 14.3-RELEASE arrives = `FreeBSD-14.3.pkg`
- `freebsd-runtime:14` -> `FreeBSD 14.3-RELEASE`
### Mutable Tags For STABLE & CURRENT (CI use case)
- derived from `/usr/src/Makefile.inc1`
- the `freebsd-runtime:14.snapYYYYMMDDHHMMSS` is immutable
because this matches pkg-base naming also get used to it!
- use `freebsd-runtime:14-latest` to slide along STABLE or CURRENT
```
### the latest -pX image would be tagged as 14.3
# podman push c5f3e77557a9 ghcr.io/freebsd/freebsd-runtime:14.3
# podman push 7876fe59dbb3 ghcr.io/freebsd/freebsd-static:14.3
# podman push ebf7538b22f4 ghcr.io/freebsd/freebsd-dynamic:14.3
### the latest RELEASE as cut from the 14/stable lineage
# podman push ... ghcr.io/freebsd/freebsd-runtime:14-latest
# podman push ... ghcr.io/freebsd/freebsd-static:14-latest
# podman push ... ghcr.io/freebsd/freebsd-dynamic:14-latest
### the latest RELEASE as cut from the 15/current lineage
# podman push ... ghcr.io/freebsd/freebsd-runtime:15-latest
# podman push ... ghcr.io/freebsd/freebsd-static:15-latest
# podman push ... ghcr.io/freebsd/freebsd-dynamic:15-latest
```
## Multi-architecture Builds
- RELENG builds will always be separate artefacts
- `image list` is a JSON blob comprising the image hash + some metadata
- use either [buildah] or [podman] to merge them
- `podman manifest create` to make an empty one
- `podman manifest add` includes the image
- `buildah manifest push`
- moving these around is what [skopeo] is for
```
# buildah push --tls-verify=false --creds admin \
freebsd14.0-base docker://localhost:5000/freebsd:14.0-base
Password: ...
Getting image source signatures
Copying blob 8bf3dc12f14d skipped: already exists
Copying blob a18920fa617e skipped: already exists
Copying blob 188e5e166924 skipped: already exists
Copying config cc353a2011 done |
Writing manifest to image destination
```
This time, the login will be saved in `/root/.config/containers/auth.json`.
```
# buildah login --tls-verify=false localhost:5000
Username: admin
Password:
Login Succeeded!
# buildah push --tls-verify=false freebsd14.0-small docker://localhost:5000/freebsd:14.0-small
Getting image source signatures
Copying blob 05a151490b6d done |
Copying blob 42509cf52690 done |
Copying blob d6344e14ba8a done |
Copying blob aca51be9eadf done |
Copying blob a94388b7abab done |
Copying config 0af3b8c4c2 done |
Writing manifest to image destination
```
# Building Custom Images
Essentially we "fork" a new image off an existing one, make modifications,
and eventually commit our changes, and push them to a registry for re-use.
In this section, we'll create an Apache CouchDB image. It requires a handful
of packages, but also has some tricky runtime requirements:
- `/var/db/couchdb3/` for database files
- config files in `/etc/rc.conf.d/couchdb` and `/usr/local/etc/couchdb/`.
- in my deployment, an overlay IPv6 network to allow other CouchDB
cluster nodes to network with each other
## Forking an Existing Image
To keep the image clean, we'll mount additional tmpfs volumes into the
temporary building image.
```
# buildah from \
--volume /var/cache/pkg:/var/cache/pkg \
--volume /tmp:/var/tmp \
--volume /tmp:/tmp \
--name couch-builder \
localhost/freebsd14.0-small:latest
couch-builder
# buildah containers
CONTAINER ID BUILDER IMAGE ID IMAGE NAME CONTAINER NAME
fde7287590ed * 4bc70197ea21 localhost/freebsd14.0-small:l... couch-builder
```
## Customising Our Image
```
# buildah run couch-builder pkg install -yr FreeBSD databases/couchdb3
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:14:amd64/latest, please wait...
[fde7287590ed] Installing pkg-1.20.9_1...
[fde7287590ed] Extracting pkg-1.20.9_1: 100%
Updating FreeBSD repository catalogue...
[fde7287590ed] Fetching meta.conf: 100% 163 B 0.2kB/s 00:01
[fde7287590ed] Fetching packagesite.pkg: 100% 7 MiB 7.4MB/s 00:01
Processing entries: 100%
FreeBSD repository update completed. 34057 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 8 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
couchdb3: 3.3.3_1 [FreeBSD]
erlang-runtime25: 25.3.2.10 [FreeBSD]
...
[fde7287590ed] [7/8] Installing erlang-runtime25-25.3.2.10...
[fde7287590ed] [7/8] Extracting erlang-runtime25-25.3.2.10: 100%
[fde7287590ed] [8/8] Installing couchdb3-3.3.3_1...
...
For more information see
http://docs.couchdb.org/en/latest/setup/index.html
```
The default config files installed during `pkg(1)` are not wanted, so let's
remove them. Also, add an `/etc.rc.conf.d/couchdb3` file to ensure that
Couch is started on boot.
```
# buildah run couch-builder rm -rf \
/usr/local/etc/couchdb3/local.ini \
/usr/local/etc/couchdb3/local.ini
# buildah run couch-builder /bin/sh
# pwd
/
# printf 'couchdb3_enable=YES\ncouchdb3_oomprotect=ALL\n' | tee /etc/rc.conf.d/couchdb3
couchdb3_enable=YES
couchdb3_oomprotect=ALL
# exit
```
Note that `buildah run` allows running arbitrary commands in the container,
and also just using a shell directly, similar to `jexec(1)`. Run `mount` and
grep your container id to see the mountpoint:
```
# buildah containers
CONTAINER ID BUILDER IMAGE ID IMAGE NAME CONTAINER NAME
132edd5445b9 * 4bc70197ea21 localhost/freebsd14.0-small:l... couch-builder
# jls | grep 132edd5445b9
36 132edd5445b9 /var/db/containers/storage/zfs/graph/0554...
```
## Inspecting Images
This image should be usable now. It's missing some metadata and tags,
but but we can view its properties, and eventually commit (push) to
a remote registry, to be used anywhere.
```
# buildah containers
CONTAINER ID BUILDER IMAGE ID IMAGE NAME CONTAINER NAME
fde7287590ed * 4bc70197ea21 localhost/freebsd14.0-small:l... couch-builder
# buildah inspect couch-builder
{
"Type": "buildah 0.0.1",
"FromImage": "localhost/freebsd14.0-small:latest-amd64",
"FromImageID": "4bc70197ea21c7fedb1ff0ff4e7cef3f0bc69f8d00c11fb4e976a25c0aa2a732",
"FromImageDigest": "sha256:83baa06dc1a9e826f500dfceeb8f331a2267b1959ee1bfb683207e7e5bd27c9e",
...
"History": [
{
"created": "2024-04-09T00:00:00Z",
"created_by": "/bin/sh"
},
{
"created": "2024-04-09T00:00:00Z",
"created_by": "/bin/sh",
"comment": "FROM localhost/freebsd14.0-mtree:latest-amd64"
},
{
"created": "2024-04-09T00:00:00Z",
"created_by": "/bin/sh",
"comment": "FROM localhost/freebsd14.0-static:latest-amd64"
},
{
"created": "2024-04-09T00:00:00Z",
"created_by": "/bin/sh",
"comment": "FROM localhost/freebsd14.0-base:latest-amd64"
},
{
"created": "2024-04-09T00:00:00Z",
"created_by": "/bin/sh",
"comment": "FROM localhost/freebsd14.0-runtime:latest-amd64"
}
],
"Devices": []
}
```
Most of the JSON output has been removed above, apart from `FromImage*`
keys that show its provenance, and the `History` object that shows the
dependent layers required.
## Container Metadata (WIP)
- whats the difference between metadata and tags?
- when to use one over the other?
```
# buildah containers
CONTAINER ID BUILDER IMAGE ID IMAGE NAME CONTAINER NAME
fde7287590ed * 4bc70197ea21 localhost/freebsd14.0-small:l... couch-builder
# buildah tag 4bc70197ea21 3.3.3 latest
```
## Committing and Publishing Images (WIP)
- what is difference between `commit` and `push` ?
- seems that only `push` actually shows up on remote repo
```
# buildah commit couch-builder couchdb:3.3.3
Getting image source signatures
Copying blob 1643796dfb88 skipped: already exists
Copying blob cc00286ea44e skipped: already exists
Copying blob c30672e8f04b skipped: already exists
Copying blob bdb2e98fd234 skipped: already exists
Copying blob b1698910dc02 skipped: already exists
Copying blob 7e7852284f41 skipped: already exists
Copying config 7af22b0211 done |
Writing manifest to image destination
7af22b021197c554c8a5b5a9af3d0879bad22bad543cef30d16d7d95d986eb57
```
## TODO Using Containers with ZFS and Complex Networking
- create a set of nested datasets to be mounted into the container
```
# zfs create -o jailed=on \
-o canmount=off \
-o mountpoint=/var/db/couchdb3 \
zroot/jailed/couchdb
# zfs create zroot/jailed/couchdb/data
# zfs create zroot/jailed/couchdb/views
```
- fetch container by tag and confirm its got the package inside
```
# podman pull ghcr.io/skunkwerks/couchdb:3.3.3
Trying to pull ghcr.io/skunkwerks/couchdb:3.3.3...
Getting image source signatures
Copying blob 5f8b8c3299eb done |
Copying blob 52bf5d211a1f done |
Copying blob 95e5da2e6747 done |
Copying blob ccdaedb6e555 done |
Copying blob c100350252d2 done |
Copying blob 1f86f171a4f2 done |
Copying config 7af22b0211 done |
Writing manifest to image destination
7af22b021197c554c8a5b5a9af3d0879bad22bad543cef30d16d7d95d986eb57
# podman run ghcr.io/skunkwerks/couchdb:3.3.3 pkg info couchdb\*
couchdb3-3.3.3_1
```
- run container, mounting volumes, and IPv6 address
```
... working on it!
```
## Using and Testing arm64 images on amd64
This is very convenient for validating and testing images after build.
You can install the qemu-user-static (or qemu-user-static-devel),
enable it and 'start' with service(8) - this will register image
activators for all the emulated architectures. With that, you can
run the image by mounting the relevant qemu into the container:
```
$ uname -a
FreeBSD herring.home.rabson.org 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC amd64
$ sudo podman run -ti --rm --arch=arm64 -v /usr/local/bin/qemu-aarch64-static:/usr/local/bin/qemu-aarch64-static ghcr.io/freebsd/freebsd-runtime:14.snap uname -a
FreeBSD 73d4049d10b2 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC arm64
```
It is possible to avoid the mounting nonsense but it requires a small
change to qemu (https://github.com/qemu-bsd-user/qemu-bsd-user/pull/42)
and the binmiscctl invocations in the service file need to have the
'--pre-open' flag added.
## Manipulating Images
# Issues And Problems
### how to mount tmpfs
- `podman run --mount type=tmpfs...` doesn't work
```
# podman run -t --mount type=tmpfs,destination=/var/db/couchdb ghcr.io/skunkwerks/couchdb:3.3.3 df -h
Filesystem Size Used Avail Capacity Mounted on
zroot/containers/4a3cdf1734c84263fc0aaa5e1ddea2beb0c95190fdd67f886c65497fab1312c3 961G 272M 961G 0% /
devfs 1.0K 0B 1.0K 0% /dev
fdescfs 1.0K 0B 1.0K 0% /dev/fd
/var/run/containers/storage/zfs-containers/504e8d18b56bece2d9c321b1e7fd24785fe035671b6eb80eceb4888259efb722/userdata/hosts 964G 2.7G 961G 0% /etc/hosts
/var/run/containers/storage/zfs-containers/504e8d18b56bece2d9c321b1e7fd24785fe035671b6eb80eceb4888259efb722/userdata/resolv.conf 964G 2.7G 961G 0% /etc/resolv.conf
/var/run/containers/storage/zfs-containers/504e8d18b56bece2d9c321b1e7fd24785fe035671b6eb80eceb4888259efb722/userdata/.containerenv 964G 2.7G 961G 0% /var/run/.containerenv
```
## `ipv6` doesn't work
- see https://www.redhat.com/sysadmin/podman-new-network-stack
- and https://developers.redhat.com/articles/2022/08/10/how-conifgure-podman-40-ipv6
- I have an existing ipv6 overlay network (zerotier, basically a smart tap if)
- I want to "give" a specific ip6 to the container so it can reach other couchdb
containers in the cluster, on different teams
- I manually made a `zerotier` network, it doesn't need a bridge
- hangs on tearing down the container
```
# podman run -t --ip6 fca2:927d:4d7c:5005:f22b::3 ghcr.io/skunkwerks/couchdb:3.3.3 ifconfig
Error: requested static ip fca2:927d:4d7c:5005:f22b::3 not in any subnet on network podman
# podman info |grep -i networkbackend
networkBackend: cni
networkBackendInfo:
# podman network inspect podman
[
{
"name": "podman",
"id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
"driver": "bridge",
"network_interface": "cni-podman0",
"created": "2024-04-10T11:58:10.506628503Z",
"subnets": [
{
"subnet": "10.88.0.0/16",
"gateway": "10.88.0.1"
}
],
"ipv6_enabled": false, <----------------------------
"internal": false,
"dns_enabled": false,
"ipam_options": {
"driver": "host-local"
}
}
]
# podman network create --ipv6 zerotier
zerotier
# podman network inspect zerotier
[
{
"name": "jails",
"id": "4b798d7395fb72fac619638be2fab07326ddb1cdec8c3827a22ede623b8a7af6",
"driver": "bridge",
"network_interface": "cni-podman1",
"created": "2024-04-10T11:58:51.536122Z",
"subnets": [
{
"subnet": "10.89.0.0/24",
"gateway": "10.89.0.1"
},
{
"subnet": "fdfa:b524:bd5f:51b1::/64",
"gateway": "fdfa:b524:bd5f:51b1::1"
}
],
"ipv6_enabled": true, <--------------- good
"internal": false,
"dns_enabled": false,
"ipam_options": {
"driver": "host-local"
}
}
]
```
# References
## General
- [this doc](https://docs.skunkwerks.at/s/fUiAmi4pE)
- [Containers](https://github.com/containers)
- [Container Tools Guide](https://github.com/containers/buildah/tree/main/docs/containertools)
- [Containers Image](https://github.com/containers/image/tree/main)
- [Containers Image Docs](https://github.com/containers/image/tree/main/docs)
- [Podman](https://github.com/containers/podman)
- [Installing on FreeBSD 14.0](https://podman.io/docs/installation#installing-on-freebsd-140)
- [Docker Registry](https://docs.docker.com/registry/) which has been donated to [CNCF] and became known as [(CNCF) Distribution](https://distribution.github.io/distribution/)
## Porting
- https://reviews.freebsd.org/D37324
- https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254645
- https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=267184
- https://github.com/freebsd/freebsd-ci/pull/126
- https://github.com/opencontainers/umoci/issues/364
- https://github.com/oras-project/oras looks useful, trivial patch for FreeBSD
- https://github.com/opencontainers/umoci people use this for (re)packing
## Hacking & Using
- https://akhramov.github.io/posts/2021-09-16-FreeBSD-OCI-jails.html
- https://gitlab.com/runhyve/jailer/-/snippets/2082864
- https://gitlab.com/runhyve/jailer/-/snippets/2082864
- https://freshports.org/sysutils/docker-registry
## Building Images
- https://github.com/dfr/freebsd-images has a `build.sh` script that will
auto-build all the images you require from official sources
- https://medium.com/oracledevs/building-multi-architecture-containers-on-oci-with-podman-67d49a8b965e
## Using Registries
- https://redhat-scholars.github.io/containers-tutorial/containers-tutorial/buildah.html
- https://buildah.io/blogs/2018/01/26/using-image-registries-with-buildah.html
- https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry
- https://docs.oracle.com/en/operating-systems/oracle-linux/podman/podman-BuildingImagesWithBuildah.html#buildah-containers
- https://docs.public.oneportal.content.oci.oraclecloud.com/en-us/iaas/Content/Registry/home.htm
- https://www.oracle.com/webfolder/technetwork/tutorials/obe/oci/registry/index.html
- https://enabling-cloud.github.io/oci-learning/manual/OCIRegistry.html
- https://stackoverflow.com/questions/73879886/how-to-list-all-images-from-an-account-in-github-container-registry
- https://www.douglashellinger.com/explainer/container-oci-registry/pull-a-public-container-image/
- https://podman.io/blogs/2021/10/11/multiarch
## Linux Specific
- https://projectatomic.io/blog/2018/03/the-many-ways-to-build-oci-images/
- https://computingforgeeks.com/how-to-build-oci-docker-container-images-with-buildah/
## Trusted Content
- https://docs.docker.com/trusted-content/
- https://docs.digicert.com/en/software-trust-manager/signing-tools/container-signing-with-podman.html
## Quick Notes on Signing
- you only need a *writable* registry for signing `/usr/local/etc/containers/registries.conf`
- `/usr/local/etc/containers/registries.d/default.yaml` tells us where to go to verify signatures (via external http server)
- `/usr/local/etc/containers/policy.json` defines locally what keys are accepted
- signatures are stored in files, e.g. inside container `/var/lib/containers/sigstore` and can be versioned in git
# FAQ: Reported User Issues & Solutions
- podman-nullfs-mounted files like `/etc/resolv.conf` and `/etc/hosts`
gets in the way with error `Cross-device link` or similar, during
`pkg update`
- add `--no-hosts` and `--dns 1.1.1.1` to your container
```
[fd343e182fa5] [17/18] Extracting FreeBSD-runtime-14.3.b2.20250512234546: 100%
pkg: Fail to rename /etc/.pkgtemp.hosts.VeIDrHM0sKN5 -> /etc/hosts:Cross-device link
```
- `pkg install ...` fails
- see below, it's complicated
## Checking Networking
This could be anything - keep it simple! Start off with the rules from
`pf.conf` provided in `/usr/local/etc/containers/pf.conf.sample`.
The following steps starts off with a container that specifies its own
DNS resolver, not using the host's provided one, passes an env var in
for pkg, and tries to fetch a file over http.
```
jail@host# podman run \
--no-hosts \
--dns 9.9.9.9 \
--env ASSUME_ALWAYS_YES=1
-it --rm \
ghcr.io/freebsd/freebsd-runtime:14.2 \
/bin/sh
### do we have a sensible route?
# route show 9.9.9.9
route to: 9.9.9.9
destination: default
mask: default
gateway: 10.88.0.1
fib: 0
interface: eth0
flags: <UP,GATEWAY,DONE,STATIC>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 9000 1 0
### can we ping the gateway?
# ping -c 3 10.88.0.1
PING 10.88.0.1 (10.88.0.1): 56 data bytes
64 bytes from 10.88.0.1: icmp_seq=0 ttl=64 time=0.083 ms
64 bytes from 10.88.0.1: icmp_seq=1 ttl=64 time=0.076 ms
64 bytes from 10.88.0.1: icmp_seq=2 ttl=64 time=0.089 ms
--- 10.88.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.076/0.083/0.089/0.006 ms
### does ICMP to the world work?
# ping -c 3 9.9.9.9
PING 9.9.9.9 (9.9.9.9): 56 data bytes
64 bytes from 9.9.9.9: icmp_seq=0 ttl=60 time=0.920 ms
64 bytes from 9.9.9.9: icmp_seq=1 ttl=60 time=0.882 ms
64 bytes from 9.9.9.9: icmp_seq=2 ttl=60 time=0.876 ms
--- 9.9.9.9 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.876/0.893/0.920/0.020 ms
### does simple tcp work?
# fetch http://1.1.1.1/
fetch.out 55 kB 16 MBps 00s
### does pkg work? the -d is for Debugging
# pkg -dd bootstrap -r FreeBSD
```
- if `ping` fails, check your firewall rules for blocking ICMP
- if `fetch` fails, check your firewall rules for blocking TCP
- if `pkg` fails, but ICMP & TCP work, this could be an MTU problem
### Blocked Packets, with`pflog` and `tcpdump`
Any complex firewall ruleset will have block and pass rules.
The pflog device will log src/dest ip & port, for any rule that
blocks traffic, that has `block log` listed.
- add `log` to all of your block rules
- run `service pflog onerestart` to enable pflog
- check for blocked packets
- `tcpdump -vvveni pflog0 icmp or net 10.88.0.0/16`
- also check for passing packets to your final `$DEST_IP`
- `tcpdump -vvveni vtnet0 icmp or $DEST_IP`
In this MTU failure case, `tcpdump(1)` reports:
```
# tcpdump -vvveni vtnet0 host 134.70.104.1
10.0.0.122 > 134.70.104.1: ICMP 10.0.0.122 unreachable - need to frag (mtu 1500), length 576
(tos 0x0, ttl 61, id 40019, offset 0, flags [DF], proto TCP (6), length 2948, bad cksum 9c5f (->9d5f)!)
```
Lowering MTU on bridge, external interface didn't help. So there
must be further changes required within the container itself.
## checking MTU consistency
- Read https://www.cisco.com/c/en/us/support/docs/ip/generic-routing-encapsulation-gre/25885-pmtud-ipfrag.html carefully
- remove any & all sysctls relating to `net.inet`
- reboot
In the following example, `vtnet0` has MTU 9000, but the bridge and
containers have MTU 1500. In general, coerce your internal layout
to match the external network, most often, yours is the problem.
```
### jail host
# ifconfig
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
ether 02:00:17:10:ca:45
inet 10.0.0.122 netmask 0xffffff00 broadcast 10.0.0.255
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
cni-podman0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=0
ether 58:9c:fc:00:04:63
inet 10.88.0.1 netmask 0xffff0000 broadcast 10.88.255.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet22fc8dd7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
vnet22fc8dd7: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
description: associated with jail: 06fc02aa3b8a74348a97068cdc92c8c2742860f3a692777f892ddc0ac78b78b8 as nic: eth0
options=8<VLAN_MTU>
ether 02:42:7f:a8:b9:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```
```
### in container
# podman run -it --rm ghcr.io/freebsd/freebsd-runtime:14.2 /sbin/ifconfig
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0x0
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
eth0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:bf:bb:bb:dc:0b
inet 10.88.0.8 netmask 0xffff0000 broadcast 10.88.255.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
```
## Fixing MTU inconsistencies
Bridges & epairs should automatically align MTU if ICMP4/6 traffic
is allowed to pass.
In some cloud environments, `set reassemble yes no-df` may be
required in `/etc/pf.conf` as well.
In this scenario, it was sufficient to align MTU on all 3 sides
(primary NIC, container bridge, container):
```
### already set by upstream network provider
# ifconfig vtnet0 mtu 9000
# ifconfig cni-podman0 mtu 9000
### and in the container
# ifconfig eth0 mtu 9000
```
An issue is that the epairs & bridge used by podman are ephemeral, these
will need to be set for each bridge creation, and each container.
[14.3-RELEASE]: https://www.freebsd.org/releases/14.3R/announce/
[Docker Hub]: https://hub.docker.com/u/freebsd
[FreeBSD.org]: https://freebsd.org/
[GitHub Container Registry]: https://github.com/orgs/freebsd/packages
[Official FreeBSD Releases]: https://download.freebsd.org/releases
[Open Container Initiative]: https://opencontainers.org/
[Podman]: https://podman.io/
[buildah]: https://buildah.io/
[podman]: https://podman.io/
[skopeo]: https://github.com/containers/skopeo
[snapshots]: https://download.freebsd.org/snapshots/OCI-IMAGES/
[podman import]: http://man.freebsd.org/podman-import