# OCI containers on FreeBSD with Podman
The podman suits comprises 3 main tools and a bunch of supporting ones:
- [podman] is a daemonless container engine using docker-compatible commands
- [buildah] generates OCI formatted container images
- [skopeo] works with remote OCI image registries
> Work-in-progress notes, these will move to my blog once reviewed
> at https://people.freebsd.org/~dch/posts
[buildah]: https://buildah.io/
[podman]: https://podman.io/
[skopeo]: https://github.com/containers/skopeo
> Mad props to SamuelKarp@ and dfr@ who made *all* of this work,
> from building https://github.com/samuelkarp/runj the first
> container-friendly port for FreeBSD, to porting the entire podman
> suite and getting necessary kernel changes to support it.
> Mad Mad Props!
# TODO
- [x] quick setup & usage
- [x] setup & using a local repository
- [x] setup & using a remote repository (GitHub, OCI, .. others? )
- [ ] document podman runtime hooks
- [ ] export & import containers as files
- [ ] using signatures and hashes to verify container provenance
- [x] using buildah to make custom images
- [ ] keeping local & remote repositories in sync with [skopeo](https://redhat-scholars.github.io/containers-tutorial/containers-tutorial/skopeo.html)
- [ ] update [sysutils/docker-registry](https://freshports.org/sysutils/docker-registry)
- [ ] creating containers in build pipelines (what CI?)
- [ ] handling multi-arch repos and builds
- [ ] naming things is hard, what makes sense, how do tags relate to updates?
- [ ] how to assign specific IPv4/IPv6 addresses to containers?
- [ ] attaching custom nested zfs datasets into containers
- [ ] Create a diagram depicting how all the pieces come together
- [podman](https://github.com/containers/podman)
- [image](https://github.com/containers/image)
- [conmon](https://github.com/containers/conmon)
- [ocijail](https://github.com/dfr/ocijail)
- [ ] Further topics?
- [ ] Redo everything using [runj](https://github.com/samuelkarp/runj) + [containerd](https://github.com/containerd/containerd)
- [ ] Redo everything using [runj](https://github.com/samuelkarp/runj) + [containerd](https://github.com/containerd/containerd) + [nerdctl](https://github.com/containerd/nerdctl)
- [ ] What about other FreeBSD distros ?
- [ ] HardenedBSD
# Additional Notes (Possibly Roadmap Action Items ?)
- **Cleaning**
- Some packages/ports with GPLv2 license while the packaged/ported code is ALv2. **A note has been sent to dfr@ over Discord**
- Checking some man pages, conmon(8) for example, only referring to Linux Containers. **Should we update those as well to indicate both Linux and FreeBSD Contaienrs ?**
- Consistency between packages/ports messages/descriptions, man pages and online docs, for example [Installing on FreeBSD 14.0](https://podman.io/docs/installation#installing-on-freebsd-140)
- **Improvements**
- Support Podman Rootless mode on FreeBSD
- [containers-common](https://cgit.freebsd.org/ports/tree/sysutils/containers-common) default configurations (```registries.conf```, etc ...)
- **Issues**
- ~~After [installing podman](https://podman.io/docs/installation#installing-on-freebsd-140) and applying the [initial configuration](https://podman.io/docs/installation#initial-configuration) but before configuring [networking](https://podman.io/docs/installation#networking) and [storage](https://podman.io/docs/installation#storage), running [podman search](https://docs.podman.io/en/latest/markdown/podman-search.1.html) without adding any registry prefix leads no reults~~ - See **Improvements**
```
# podman search httpd
```
# Install
- remember to run all this in screen/tmux/mosh to avoid network loss
- integrate changes from `/usr/local/etc/containers/pf.conf.sample`
into your `/etc/pf.conf` setting egress macros appropriately
- amend `sysctl.conf`, and `/etc/fstab` as below
```
# zfs create -o mountpoint=/var/db/containers zroot/containers
# zfs snapshot zroot/containers@empty
# sysctl net.pf.filter_local=1
# mount -t fdescfs fdesc /dev/fd
# pkg install -r FreeBSD -y podman-suite emulators/qemu-user-static
# service pf restart
```
If you don't have zfs, switch to ufs:
```
# sed -I .bak -e 's/driver = "zfs"/driver = "vfs"/' \
/usr/local/etc/containers/storage.conf
```
- review and amend the template config files. None of these are
required though:
```
# pkg list buildah podman conmon ocijail containers-common \
containernetworking-plugins \
|grep /etc/
/usr/local/etc/containers/containers.conf.sample
/usr/local/etc/containers/policy.json.sample
/usr/local/etc/containers/registries.conf.sample
/usr/local/etc/containers/storage.conf.sample
/usr/local/etc/containers/pf.conf.sample
```
# Usage
Contrary to podman/docker on Linux, everything needs to run as root. For
the moment.
## Find and Fetch
```
# podman image search --list-tags docker.io/dougrabson/hello
NAME TAG
docker.io/dougrabson/hello latest
```
## Fetch and Run It
```
podman run --rmi docker.io/dougrabson/hello
Trying to pull docker.io/dougrabson/hello:latest...
Getting image source signatures
Copying blob b13a5ec7f3d2 done |
Copying config f81c971736 done |
Writing manifest to image destination
!... Hello Podman World ...!
.--"--.
/ - - \
/ (O) (O) \
~~~| -=(,Y,)=- |
.---. /` \ |~~
~/ o o \~~~~.----. ~~
| =(X)= |~ / (O (O) \
~~~~~~~ ~| =(Y_)=- |
~~~~ ~~~| U |~~
Project: https://github.com/containers/podman
Website: https://podman.io
Documents: https://docs.podman.io
Twitter: @Podman_io
```
## Inspect the Stash
```
# fd -tf . /var/db/containers/storage/
/var/db/containers/storage/db.sql
/var/db/containers/storage/defaultNetworkBackend
/var/db/containers/storage/storage.lock
/var/db/containers/storage/userns.lock
/var/db/containers/storage/zfs-containers/containers.lock
/var/db/containers/storage/zfs-containers/dac733521be73f95b7346dbe43562aadc7266c5205c996311ce5dec483bf8725/userdata/buildah.json
/var/db/containers/storage/zfs-containers/volatile-containers.json
/var/db/containers/storage/zfs-images/f81c971736c66436f90455b90e866f7e80d09ca6abda4d818abc589cd4eb05ee/=bWFuaWZlc3Qtc2hhMjU2OjI3YTBlYzhjMmRhODZkZjE0YjRjZWI3MDEwNDE0OTlmNWEzOTBmNzJjY2RmNjc2OWM2NzdjOWZmNjEyNDhmOGU=
/var/db/containers/storage/zfs-images/f81c971736c66436f90455b90e866f7e80d09ca6abda4d818abc589cd4eb05ee/=bWFuaWZlc3Qtc2hhMjU2OmU1ZjI0Y2RmMDIyZTIzYzE5OGYwODE5NDRkMWNjNzg5NzVkY2FkYzQxOTE3MDRjOWFiYjc2MzgzMjY5NjhkZmU=
/var/db/containers/storage/zfs-images/f81c971736c66436f90455b90e866f7e80d09ca6abda4d818abc589cd4eb05ee/=c2hhMjU2OmY4MWM5NzE3MzZjNjY0MzZmOTA0NTViOTBlODY2ZjdlODBkMDljYTZhYmRhNGQ4MThhYmM1ODljZDRlYjA1ZWU=
/var/db/containers/storage/zfs-images/f81c971736c66436f90455b90e866f7e80d09ca6abda4d818abc589cd4eb05ee/manifest
/var/db/containers/storage/zfs-images/images.json
/var/db/containers/storage/zfs-images/images.lock
/var/db/containers/storage/zfs-layers/c0e5c047951147a952903acd94d5044a912a8390be3f3f26d3d066b275ae7234.tar-split.gz
/var/db/containers/storage/zfs-layers/layers.json
/var/db/containers/storage/zfs-layers/layers.lock
/var/db/containers/storage/zfs-layers/volatile-layers.json
```
# Advanced Usage
## Daemonising
podman can be enabled as a service to restart automatically after a host is rebooted. You can either
```
# service enable podman
# service podman start
```
or
```
# sysrc podman_enable=YES
# service podman start
```
additionally, podman can restart containers after a host is rebooted in which case containers need to run with `--restart=always` option set
```
# podman run -d --restart=always my-image
```
## Linux containers
It is possible to run many Linux container images using FreeBSD's Linux
emulation:
```
# service linux onestart
# podman run --rm --os=linux docker.io/alpine cat /etc/os-release | head -1
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 4abcf2066143 done |
Copying config 05455a0888 done |
Writing manifest to image destination
NAME="Alpine Linux"
```
Just don't expect miracles from anything that has systemd dependencies, or
expects to be pid1, or expects to be ... Linux.
## Running a local registry
A registry is a fancy web server for storing and sharing your OCI images.
NB use `/etc/rc.conf.d/registry` for `rc.conf` settings, the `rc.d` script
doesn't match the daemon name.
There is a [docker-registry](https://github.com/distribution/distribution) port for FreeBSD, but it's rather old now.
```
# pkg install -r FreeBSD sysutils/docker-registry
# zfs create zroot/var/db/registry
# chown -R www:www /var/db/registry
# sysrc docker_registry_enable=YES
# sysrc docker_registry_user=www
# sed -i '' -E -e 's,rootdirectory:.+,rootdirectory: /var/db/registry,' \
/usr/local/etc/docker-registry/config.yml
# touch /usr/local/etc/docker-registry/htpasswd
# chown -R root:www /usr/local/etc/docker-registry/{config.yml,htpasswd}
# chmod 0640 /usr/local/etc/docker-registry/{config.yml,htpasswd}
```
- use SHA256 to create a nice random HTTP secret
- use http://aspirine.org/htpasswd_en.html & bcrypt to create `htpasswd` file
```
# /usr/local/etc/docker-registry/config.yaml
version: 0.1
log:
fields:
service: registry
storage:
cache:
blobdescriptor: inmemory
filesystem:
rootdirectory: /var/db/registry
http:
addr: :5000
secret: abc123sha256
headers:
X-Content-Type-Options: [nosniff]
auth:
htpasswd:
realm: basic-realm
path: /usr/local/etc/docker-registry/htpasswd
health:
storagedriver:
enabled: true
interval: 10s
threshold: 3
```
- `sudo service registry start`
- watch logs in `/var/log/docker-registry.log` for errors
- run `curl -vu admin:passwd http://localhost:5000/v2/`
you should get a `200 OK` response with `{}` body
See below for local registry usage.
# Building and Sharing Containers
## Building a new OCI container
- you can run `build.sh` from https://github.com/dfr/freebsd-images
to create a local set of FreeBSD base images, from official packages
```
# ./build.sh -b releng/14.0:14.0
pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
Generating mtree for amd64
Creating directory structure
pkg: Warning: Major OS version upgrade detected. Running "pkg bootstrap -f" recommended
a68f8ba857101498af0ed46943109864da6d3c504126a1cfae9c0109f2614102
Getting image source signatures
Copying blob aca51be9eadf done |
Copying config 05cfc4ff7d done |
...
Copying blob fafaf4b2c1fc done |
Copying config 31e93be597 done |
Writing manifest to image destination
6b60451da926d6de4ce52423313ed89bc0eddfd4164cf5e6bbf9e1bab8c84a29
```
## Listing Local Images
- note that `latest` and `14.0p6` tags match up with the same `image id`
```
# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/freebsd14.0-small 14.0p6 6b60451da926 5 hours ago 1.03 kB
localhost/freebsd14.0-small latest 6b60451da926 5 hours ago 1.03 kB
localhost/freebsd14.0-minimal 14.0p6 5faaecc767ad 5 hours ago 1.03 kB
localhost/freebsd14.0-minimal latest 5faaecc767ad 5 hours ago 1.03 kB
localhost/freebsd14.0-base latest-debug d57507702c2b 5 hours ago 1.03 kB
localhost/freebsd14.0-base 14.0p6-debug d57507702c2b 5 hours ago 1.03 kB
localhost/freebsd14.0-base 14.0p6 977dfe21ef15 5 hours ago 1.03 kB
localhost/freebsd14.0-base latest 977dfe21ef15 5 hours ago 1.03 kB
localhost/freebsd14.0-static latest-debug 8964f979748d 5 hours ago 1.03 kB
localhost/freebsd14.0-static 14.0p6-debug 8964f979748d 5 hours ago 1.03 kB
localhost/freebsd14.0-static latest 2018593833af 5 hours ago 1.03 kB
localhost/freebsd14.0-static 14.0p6 2018593833af 5 hours ago 1.03 kB
localhost/freebsd14.0-mtree latest b91fa618e394 5 hours ago 1.03 kB
localhost/freebsd14.0-base latest-aarch64 25e0d6010087 21 hours ago 35.1 MB
localhost/freebsd14.0-minimal latest-aarch64 7492fba91606 21 hours ago 61.5 MB
localhost/freebsd14.0-base latest-debug-amd64 28358b486bc6 21 hours ago 55.1 MB
localhost/freebsd14.0-base latest-debug-aarch64 1b6f8b32ed55 21 hours ago 52.7 MB
localhost/freebsd14.0-static latest-debug-amd64 01398919565e 21 hours ago 23.8 MB
localhost/freebsd14.0-base latest-amd64 cc353a2011ef 21 hours ago 35.7 MB
localhost/freebsd14.0-minimal latest-amd64 ca1f3dd2d649 21 hours ago 62.8 MB
localhost/freebsd14.0-mtree latest-amd64 05cfc4ff7d55 21 hours ago 1.17 MB
localhost/freebsd14.0-static latest-aarch64 b444483b7b3b 21 hours ago 6.01 MB
localhost/freebsd14.0-static latest-amd64 38a855def155 21 hours ago 6.02 MB
localhost/freebsd14.0-small latest-amd64 0af3b8c4c2e7 21 hours ago 175 MB
localhost/freebsd14.0-small latest-aarch64 31e93be59727 21 hours ago 164 MB
localhost/freebsd14.0-static latest-debug-aarch64 392a902558fc 21 hours ago 22 MB
localhost/freebsd14.0-mtree latest-aarch64 005682383153 21 hours ago 1.17 MB
quay.io/dougrabson/hello latest f81c971736c6 23 months ago 4.06 MB
```
## Create and Run
My PC is running a pre-release `15.0-CURRENT` but the built container is
based off a `15.0-RELEASE` image, so [freebsd-version(1)] will return
different results for running kernel vs installed userland. That's ok!
```
# buildah from localhost/freebsd14.0-minimal
freebsd14.0-minimal-working-container
# buildah run freebsd14.0-minimal-working-container freebsd-version -ru
15.0-CURRENT <------ my desktop's kernel
14.0-RELEASE-p6 <------ the container's userland
```
This image is made up of multiple layers, only the final one contains the main userland.
These layers are stored
```
# podman image tree localhost/freebsd14.0-small
Image ID: 4bc70197ea21
Tags: [localhost/freebsd14.0-small:latest-amd64]
Size: 174.8MB
Image Layers
├── ID: 1643796dfb88 Size: 1.172MB Top Layer of: [localhost/freebsd14.0-mtree:latest-amd64 localhost/freebsd14.0-mtree:latest-aarch64]
├── ID: 8a0cc7febb16 Size: 4.847MB Top Layer of: [localhost/freebsd14.0-static:latest-amd64]
├── ID: 3802ebf80a0f Size: 29.69MB Top Layer of: [localhost/freebsd14.0-base:latest-amd64]
├── ID: 0c5e170468cb Size: 27.13MB Top Layer of: [localhost/freebsd14.0-minimal:latest-amd64]
└── ID: 1c354111d4b5 Size: 112MB Top Layer of: [localhost/freebsd14.0-small:latest-amd64]
```
- TODO how can we see whats in the layers?
## Signing Images (WIP)
**DRAFT - NOT FINAL YET**
- Registeries are not responsible for signing container images or verifying their signatures
- Who then is responsible ?
- What are the methods of Signing:
- :white_check_mark: GNU Privacy Guard ([GPG](https://gnupg.org/))
- :white_check_mark: [How to sign and distribute container images using Podman](https://github.com/containers/podman/blob/main/docs/tutorials/image_signing.md)
- [Sigstore](https://www.sigstore.dev/) (Cosign [1](https://docs.sigstore.dev/signing/quickstart/) [2](https://github.com/sigstore/cosign/tree/main/doc) ?)
- What should be then the flow of:
- Signing ?
- Distributing/Publishing Signatures ?
- Verifying Signatures ?
## Tagging Images (WIP)
Users should be able to fetch and verify signed images by tag, that
matches precisely what is published by the FreeBSD release engineering
team. This should Do The Right Thing based on your system architecture.
- [ ] `podman pull freebsd:latest` should give you the latest patched release
- [ ] `podman pull freebsd:14.0-RELEASE-p6` sometimes precision is important
- [ ] `podman pull freebsd:13.2-RELEASE` no innovation we're the govt thanks
- [ ] `podman pull freebsd:14-STABLE` CURRENT-light
- [ ] `podman pull freebsd:CURRENT` you know you like it hawt
# Registries
In general, registries require authentication for push, and optionally for
pull. Credentials are saved in `/root/.config/containers/auth.json` by
default, if you login, but it is also possible to enter the password each
time manually.
## Local Registry
Using the prior installation, login, and push an image:
```
# buildah push --tls-verify=false --creds admin \
freebsd14.0-base docker://localhost:5000/freebsd:14.0-base
Password: ...
Getting image source signatures
Copying blob 8bf3dc12f14d skipped: already exists
Copying blob a18920fa617e skipped: already exists
Copying blob 188e5e166924 skipped: already exists
Copying config cc353a2011 done |
Writing manifest to image destination
```
This time, the login will be saved in `/root/.config/containers/auth.json`.
```
# buildah login --tls-verify=false localhost:5000
Username: admin
Password:
Login Succeeded!
# buildah push --tls-verify=false freebsd14.0-small docker://localhost:5000/freebsd:14.0-small
Getting image source signatures
Copying blob 05a151490b6d done |
Copying blob 42509cf52690 done |
Copying blob d6344e14ba8a done |
Copying blob aca51be9eadf done |
Copying blob a94388b7abab done |
Copying config 0af3b8c4c2 done |
Writing manifest to image destination
```
## Github
This container registry is namespaced by github accounts and organisations.
### Organisation Changes
- for each organisation, or user, that you wish to publish packages for,
enable `public` [permissions in that organisation](https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility)
- set up permissions on your [github organisation](https://github.com/organizations/freebsd/settings/packages) to enable public packages
- we will not link packages to repos at this stage, it requires automated builds
### Personal Changes
Each user who can upload, requires a personal access token, or [PAT](https://github.com/settings/tokens)
- make a PAT, with `write:packages` scope permissions, delete is not required
- this can be used to login to the Github Container registry
- after uploading each package (see below for `buildah push`, change its visibility
to public, and grant the group access
### Push An Image
```
# podman login ghcr.io -u dch
Password:
Login Succeeded!
root@wintermute /h/dch# buildah push ghcr.io/skunkwerks/freebsd14.0-small:14.0p6
Error: pushing image "ghcr.io/skunkwerks/freebsd14.0-small:14.0p6" to "docker://ghcr.io/skunkwerks/freebsd14.0-small:14.0p6": ghcr.io/skunkwerks/freebsd14.0-small:14.0p6: image not known
root@wintermute /h/dch# buildah push freebsd14.0-small:14.0p6 docker://ghcr.io/skunkwerks/freebsd14.0-small:14.0p6
Getting image source signatures
Copying blob d6344e14ba8a done |
Copying blob aca51be9eadf done |
Copying blob 05a151490b6d done |
Copying blob 42509cf52690 done |
Copying blob a94388b7abab done |
Copying config 0af3b8c4c2 done |
Writing manifest to image destination
```
And now let's push all of the local ones, using fish shell:
```
# for i in (podman images -n | cut -wf 1 |grep freebsd \
sed -e 's,localhost/,,')
buildah push $i docker://ghcr.io/skunkwerks/$i
end
...
Copying blob a18920fa617e skipped: already exists
Copying blob 188e5e166924 skipped: already exists
Copying config 38a855def1 done |
Writing manifest to image destination
Getting image source signatures
Copying blob a18920fa617e skipped: already exists
Copying config 05cfc4ff7d done |
Writing manifest to image destination```
```
These packages are now available under the [skunkwerks](https://github.com/orgs/skunkwerks/packages) organisation.
Trust them like you would the food left out back of a nightclub after their
New Year's party, but a week later.
### Pull And Run
You don't need to authenticate to fetch public images.
```
# buildah from ghcr.io/skunkwerks/freebsd14.0-minimal
Trying to pull ghcr.io/skunkwerks/freebsd14.0-minimal:latest...
Getting image source signatures
Copying blob 31fba944001f done |
Copying blob a18920fa617e done |
Copying blob 188e5e166924 done |
Copying blob 8bf3dc12f14d done |
Copying config ca1f3dd2d6 done |
Writing manifest to image destination
freebsd14.0-minimal-working-container
# buildah run freebsd14.0-minimal-working-container freebsd-version -ru
14.0-RELEASE-p5
14.0-RELEASE-p6
# buildah run freebsd14.0-minimal-working-container ifconfig
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet 127.0.0.1 netmask 0xffffff00
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0xf
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
eth0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
options=8<VLAN_MTU>
ether 02:55:75:36:69:0b
inet 10.88.0.6 netmask 0xffff0000 broadcast 10.88.255.255
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
# buildah run freebsd14.0-minimal-working-container ping -c 3 100.64.0.1
PING 100.64.0.1 (100.64.0.1): 56 data bytes
64 bytes from 100.64.0.1: icmp_seq=0 ttl=64 time=0.197 ms
64 bytes from 100.64.0.1: icmp_seq=1 ttl=64 time=0.299 ms
64 bytes from 100.64.0.1: icmp_seq=2 ttl=64 time=0.173 ms
--- 100.64.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.173/0.223/0.299/0.055 ms
# buildah run freebsd14.0-minimal-working-container df -h /
Filesystem Size Used Avail Capacity Mounted on
zroot/containers/1d9516c0e6ae2ba2b83b148d34bfafcb39a28db282778fb0928bf60fa901c23b 961G 33M 961G 0% /
```
## Using Oracle Cloud Registry
- confusingly also named `OCI` for funsies, we will call it `OCR` instead
- create a registry https://docs.oracle.com/en/middleware/fusion-middleware/12.2.1.4/ikedg/using-oci-container-registry.html
& https://docs.oracle.com/en-us/iaas/Content/Registry/Tasks/registrycreatingarepository.htm
- I recommend disabling auto-repo creation as it gets confusing if
incorrect names or paths are used for images
```
$ export CID=ocid1.tenancy.oc1..aaaaaaaaobmlhru...
$ oci artifacts container repository create \
--display-name skunkwerks \
--is-public yes \
--compartment-id $CID
```
```json
{
"data": {
"billable-size-in-gbs": 0,
"compartment-id": "ocid1.tenancy.oc1..aaaaaaaaobml",
"created-by": "ocid1.user.oc1..aaaaaaaaodt",
"defined-tags": {},
"display-name": "skunkwerks",
"freeform-tags": {},
"id": "ocid1.containerrepo.oc1.eu-amsterdam-1.0.axvxsnomswgi.aaaaaaaakr",
"image-count": 0,
"is-immutable": false,
"is-public": true,
"layer-count": 0,
"layers-size-in-bytes": 0,
"lifecycle-state": "AVAILABLE",
"namespace": "axvxsnomswgi",
"readme": null,
"system-tags": {},
"time-created": "2024-04-04T14:55:44.163000+00:00",
"time-last-pushed": null
},
"etag": "ockblikdf5cn7usf46of6dujdikfsa-1"
}
```
### Authenticate
- [region-specific hostnames](https://docs.oracle.com/en-us/iaas/Content/Registry/Concepts/registryprerequisites.htm)
- Amsterdam https://eu-amsterdam-1.ocir.io | https://ams.ocir.io
- Frankfurt https://eu-frankfurt-1.ocir.io | https://fra.ocir.io
- weird usernames `<tenancy-namespace>/oracleidentitycloudservice/<your@email>`
- password is your OCI auth token (a very dangerous secret)
```
# podman login ams.ocir.io
Username: axvxsnomswgi/oracleidentitycloudservice/dch@...
Password: .....
Login Succeeded!
```
### Push Images
As before, this is straightforwards:
```
# buildah push freebsd14.0-small:14.0p6 docker://ams.ocir.io/axvxsnomswgi/freebsd/freebsd14.0-small:14.0p6
Getting image source signatures
Copying blob a94388b7abab done |
Copying blob 05a151490b6d done |
Copying blob aca51be9eadf done |
Copying blob d6344e14ba8a done |
Copying blob 42509cf52690 done |
Copying config 0af3b8c4c2 done |
Writing manifest to image destination
# for i in (podman images -n | cut -wf 1 |grep freebsd | sed -e 's,localhost/freebsd,,'); buildah push freebsd$i docker://ams.ocir.io/axvxsnomswgi/freebsd:$i; end
...
Getting image source signatures
Copying blob 8bf3dc12f14d skipped: already exists
Copying blob a18920fa617e skipped: already exists
Copying blob 31fba944001f skipped: already exists
Copying blob 3ba10043c0e3 skipped: already exists
Copying blob 188e5e166924 skipped: already exists
Copying config 0af3b8c4c2 done |
Writing manifest to image destination
```
### Pull And Run
```
# export c=$(buildah from ams.ocir.io/axvxsnomswgi/freebsd:14.0-small)
freebsd-working-container
# buildah run $c df -h /
Filesystem Size Used Avail Capacity Mounted on
zroot/containers/1d9516c0e6ae2ba2b83b148d34bfafcb39a28db282778fb0928bf60fa901c23b 961G 33M 961G 0% /
```
# Building Custom Images
Essentially we "fork" a new image off an existing one, make modifications,
and eventually commit our changes, and push them to a registry for re-use.
In this section, we'll create an Apache CouchDB image. It requires a handful
of packages, but also has some tricky runtime requirements:
- `/var/db/couchdb3/` for database files
- config files in `/etc/rc.conf.d/couchdb` and `/usr/local/etc/couchdb/`.
- in my deployment, an overlay IPv6 network to allow other CouchDB
cluster nodes to network with each other
## Forking an Existing Image
To keep the image clean, we'll mount additional tmpfs volumes into the
temporary building image.
```
# buildah from \
--volume /var/cache/pkg:/var/cache/pkg \
--volume /tmp:/var/tmp \
--volume /tmp:/tmp \
--name couch-builder \
localhost/freebsd14.0-small:latest
couch-builder
# buildah containers
CONTAINER ID BUILDER IMAGE ID IMAGE NAME CONTAINER NAME
fde7287590ed * 4bc70197ea21 localhost/freebsd14.0-small:l... couch-builder
```
## Customising Our Image
```
# buildah run couch-builder pkg install -yr FreeBSD databases/couchdb3
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:14:amd64/latest, please wait...
[fde7287590ed] Installing pkg-1.20.9_1...
[fde7287590ed] Extracting pkg-1.20.9_1: 100%
Updating FreeBSD repository catalogue...
[fde7287590ed] Fetching meta.conf: 100% 163 B 0.2kB/s 00:01
[fde7287590ed] Fetching packagesite.pkg: 100% 7 MiB 7.4MB/s 00:01
Processing entries: 100%
FreeBSD repository update completed. 34057 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 8 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
couchdb3: 3.3.3_1 [FreeBSD]
erlang-runtime25: 25.3.2.10 [FreeBSD]
...
[fde7287590ed] [7/8] Installing erlang-runtime25-25.3.2.10...
[fde7287590ed] [7/8] Extracting erlang-runtime25-25.3.2.10: 100%
[fde7287590ed] [8/8] Installing couchdb3-3.3.3_1...
...
For more information see
http://docs.couchdb.org/en/latest/setup/index.html
```
The default config files installed during `pkg(1)` are not wanted, so let's
remove them. Also, add an `/etc.rc.conf.d/couchdb3` file to ensure that
Couch is started on boot.
```
# buildah run couch-builder rm -rf \
/usr/local/etc/couchdb3/local.ini \
/usr/local/etc/couchdb3/local.ini
# buildah run couch-builder /bin/sh
# pwd
/
# printf 'couchdb3_enable=YES\ncouchdb3_oomprotect=ALL\n' | tee /etc/rc.conf.d/couchdb3
couchdb3_enable=YES
couchdb3_oomprotect=ALL
# exit
```
Note that `buildah run` allows running arbitrary commands in the container,
and also just using a shell directly, similar to `jexec(1)`. Run `mount` and
grep your container id to see the mountpoint:
```
# buildah containers
CONTAINER ID BUILDER IMAGE ID IMAGE NAME CONTAINER NAME
132edd5445b9 * 4bc70197ea21 localhost/freebsd14.0-small:l... couch-builder
# jls | grep 132edd5445b9
36 132edd5445b9 /var/db/containers/storage/zfs/graph/0554...
```
## Inspecting Images
This image should be usable now. It's missing some metadata and tags,
but but we can view its properties, and eventually commit (push) to
a remote registry, to be used anywhere.
```
# buildah containers
CONTAINER ID BUILDER IMAGE ID IMAGE NAME CONTAINER NAME
fde7287590ed * 4bc70197ea21 localhost/freebsd14.0-small:l... couch-builder
# buildah inspect couch-builder
{
"Type": "buildah 0.0.1",
"FromImage": "localhost/freebsd14.0-small:latest-amd64",
"FromImageID": "4bc70197ea21c7fedb1ff0ff4e7cef3f0bc69f8d00c11fb4e976a25c0aa2a732",
"FromImageDigest": "sha256:83baa06dc1a9e826f500dfceeb8f331a2267b1959ee1bfb683207e7e5bd27c9e",
...
"History": [
{
"created": "2024-04-09T00:00:00Z",
"created_by": "/bin/sh"
},
{
"created": "2024-04-09T00:00:00Z",
"created_by": "/bin/sh",
"comment": "FROM localhost/freebsd14.0-mtree:latest-amd64"
},
{
"created": "2024-04-09T00:00:00Z",
"created_by": "/bin/sh",
"comment": "FROM localhost/freebsd14.0-static:latest-amd64"
},
{
"created": "2024-04-09T00:00:00Z",
"created_by": "/bin/sh",
"comment": "FROM localhost/freebsd14.0-base:latest-amd64"
},
{
"created": "2024-04-09T00:00:00Z",
"created_by": "/bin/sh",
"comment": "FROM localhost/freebsd14.0-minimal:latest-amd64"
}
],
"Devices": []
}
```
Most of the JSON output has been removed above, apart from `FromImage*`
keys that show its provenance, and the `History` object that shows the
dependent layers required.
## Container Metadata (WIP)
- whats the difference between metadata and tags?
- when to use one over the other?
```
# buildah containers
CONTAINER ID BUILDER IMAGE ID IMAGE NAME CONTAINER NAME
fde7287590ed * 4bc70197ea21 localhost/freebsd14.0-small:l... couch-builder
# buildah tag 4bc70197ea21 3.3.3 latest
```
## Committing and Publishing Images (WIP)
- what is difference between `commit` and `push` ?
- seems that only `push` actually shows up on remote repo
```
# buildah commit couch-builder couchdb:3.3.3
Getting image source signatures
Copying blob 1643796dfb88 skipped: already exists
Copying blob cc00286ea44e skipped: already exists
Copying blob c30672e8f04b skipped: already exists
Copying blob bdb2e98fd234 skipped: already exists
Copying blob b1698910dc02 skipped: already exists
Copying blob 7e7852284f41 skipped: already exists
Copying config 7af22b0211 done |
Writing manifest to image destination
7af22b021197c554c8a5b5a9af3d0879bad22bad543cef30d16d7d95d986eb57
```
## Using Containers with ZFS and Complex Networking
- create a set of nested datasets to be mounted into the container
```
# zfs create -o jailed=on \
-o canmount=off \
-o mountpoint=/var/db/couchdb3 \
zroot/jailed/couchdb
# zfs create zroot/jailed/couchdb/data
# zfs create zroot/jailed/couchdb/views
```
- fetch container by tag and confirm its got the package inside
```
# podman pull ghcr.io/skunkwerks/couchdb:3.3.3
Trying to pull ghcr.io/skunkwerks/couchdb:3.3.3...
Getting image source signatures
Copying blob 5f8b8c3299eb done |
Copying blob 52bf5d211a1f done |
Copying blob 95e5da2e6747 done |
Copying blob ccdaedb6e555 done |
Copying blob c100350252d2 done |
Copying blob 1f86f171a4f2 done |
Copying config 7af22b0211 done |
Writing manifest to image destination
7af22b021197c554c8a5b5a9af3d0879bad22bad543cef30d16d7d95d986eb57
# podman run ghcr.io/skunkwerks/couchdb:3.3.3 pkg info couchdb\*
couchdb3-3.3.3_1
```
- run container, mounting volumes, and IPv6 address
```
... working on it!
```
# Issues And Problems
### how to mount tmpfs
- `podman run --mount type=tmpfs...` doesn't work
```
# podman run -t --mount type=tmpfs,destination=/var/db/couchdb ghcr.io/skunkwerks/couchdb:3.3.3 df -h
Filesystem Size Used Avail Capacity Mounted on
zroot/containers/4a3cdf1734c84263fc0aaa5e1ddea2beb0c95190fdd67f886c65497fab1312c3 961G 272M 961G 0% /
devfs 1.0K 0B 1.0K 0% /dev
fdescfs 1.0K 0B 1.0K 0% /dev/fd
/var/run/containers/storage/zfs-containers/504e8d18b56bece2d9c321b1e7fd24785fe035671b6eb80eceb4888259efb722/userdata/hosts 964G 2.7G 961G 0% /etc/hosts
/var/run/containers/storage/zfs-containers/504e8d18b56bece2d9c321b1e7fd24785fe035671b6eb80eceb4888259efb722/userdata/resolv.conf 964G 2.7G 961G 0% /etc/resolv.conf
/var/run/containers/storage/zfs-containers/504e8d18b56bece2d9c321b1e7fd24785fe035671b6eb80eceb4888259efb722/userdata/.containerenv 964G 2.7G 961G 0% /var/run/.containerenv
```
## `ipv6` doesn't work
- see https://www.redhat.com/sysadmin/podman-new-network-stack
- and https://developers.redhat.com/articles/2022/08/10/how-conifgure-podman-40-ipv6
- I have an existing ipv6 overlay network (zerotier, basically a smart tap if)
- I want to "give" a specific ip6 to the container so it can reach other couchdb
containers in the cluster, on different teams
- I manually made a `zerotier` network, it doesn't need a bridge
- hangs on tearing down the container
```
# podman run -t --ip6 fca2:927d:4d7c:5005:f22b::3 ghcr.io/skunkwerks/couchdb:3.3.3 ifconfig
Error: requested static ip fca2:927d:4d7c:5005:f22b::3 not in any subnet on network podman
# podman info |grep -i networkbackend
networkBackend: cni
networkBackendInfo:
# podman network inspect podman
[
{
"name": "podman",
"id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
"driver": "bridge",
"network_interface": "cni-podman0",
"created": "2024-04-10T11:58:10.506628503Z",
"subnets": [
{
"subnet": "10.88.0.0/16",
"gateway": "10.88.0.1"
}
],
"ipv6_enabled": false, <----------------------------
"internal": false,
"dns_enabled": false,
"ipam_options": {
"driver": "host-local"
}
}
]
# podman network create --ipv6 zerotier
zerotier
# podman network inspect zerotier
[
{
"name": "jails",
"id": "4b798d7395fb72fac619638be2fab07326ddb1cdec8c3827a22ede623b8a7af6",
"driver": "bridge",
"network_interface": "cni-podman1",
"created": "2024-04-10T11:58:51.536122Z",
"subnets": [
{
"subnet": "10.89.0.0/24",
"gateway": "10.89.0.1"
},
{
"subnet": "fdfa:b524:bd5f:51b1::/64",
"gateway": "fdfa:b524:bd5f:51b1::1"
}
],
"ipv6_enabled": true, <--------------- good
"internal": false,
"dns_enabled": false,
"ipam_options": {
"driver": "host-local"
}
}
]
```
# References
## General
- [this doc](https://docs.skunkwerks.at/s/fUiAmi4pE)
- [Containers](https://github.com/containers)
- [Container Tools Guide](https://github.com/containers/buildah/tree/main/docs/containertools)
- [Containers Image](https://github.com/containers/image/tree/main)
- [Containers Image Docs](https://github.com/containers/image/tree/main/docs)
- [Podman](https://github.com/containers/podman)
- [Installing on FreeBSD 14.0](https://podman.io/docs/installation#installing-on-freebsd-140)
- [Docker Registry](https://docs.docker.com/registry/) which has been donated to [CNCF] and became known as [(CNCF) Distribution](https://distribution.github.io/distribution/)
## Porting
- https://reviews.freebsd.org/D37324
- https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=254645
- https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=267184
- https://github.com/freebsd/freebsd-ci/pull/126
- https://github.com/opencontainers/umoci/issues/364
- https://github.com/oras-project/oras looks useful, trivial patch for FreeBSD
- https://github.com/opencontainers/umoci people use this for (re)packing
## Hacking & Using
- https://akhramov.github.io/posts/2021-09-16-FreeBSD-OCI-jails.html
- https://gitlab.com/runhyve/jailer/-/snippets/2082864
- https://gitlab.com/runhyve/jailer/-/snippets/2082864
- https://freshports.org/sysutils/docker-registry
## Building Images
- https://github.com/dfr/freebsd-images has a `build.sh` script that will
auto-build all the images you require from official sources
- https://medium.com/oracledevs/building-multi-architecture-containers-on-oci-with-podman-67d49a8b965e
## Using Registries
- https://redhat-scholars.github.io/containers-tutorial/containers-tutorial/buildah.html
- https://buildah.io/blogs/2018/01/26/using-image-registries-with-buildah.html
- https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry
- https://docs.oracle.com/en/operating-systems/oracle-linux/podman/podman-BuildingImagesWithBuildah.html#buildah-containers
- https://docs.public.oneportal.content.oci.oraclecloud.com/en-us/iaas/Content/Registry/home.htm
- https://www.oracle.com/webfolder/technetwork/tutorials/obe/oci/registry/index.html
- https://enabling-cloud.github.io/oci-learning/manual/OCIRegistry.html
- https://stackoverflow.com/questions/73879886/how-to-list-all-images-from-an-account-in-github-container-registry
- https://www.douglashellinger.com/explainer/container-oci-registry/pull-a-public-container-image/
- https://podman.io/blogs/2021/10/11/multiarch
## Linux Specific
- https://projectatomic.io/blog/2018/03/the-many-ways-to-build-oci-images/
- https://computingforgeeks.com/how-to-build-oci-docker-container-images-with-buildah/
## Trusted Content
- https://docs.docker.com/trusted-content/
- https://docs.digicert.com/en/software-trust-manager/signing-tools/container-signing-with-podman.html
## Quick Notes on Signing
- you only need a *writable* registry for signing `/usr/local/etc/containers/registries.conf`
- `/usr/local/etc/containers/registries.d/default.yaml` tells us where to go to verify signatures (via external http server)
- `/usr/local/etc/containers/policy.json` defines locally what keys are accepted
- signatures are stored in files, e.g. inside container `/var/lib/containers/sigstore` and can be versioned in git
## Quick Notes on Exporting & Importing Layouts
- ...
- we would end up with architecture-specific deployments on the current website initially
- no point in doing multi-arch images just yet