release: add cloudware oracle targets to package and upload .oci files

[dch]

Summary

oracle upload script for releng tooling ready for review now.

• added portinstall for qemu-img, curl
• uses base tar, & flua instead of sed, 2 dependencies less
• uses qemu-tools to reduce build requirements

Test Plan

This is how I use it. Needs the $ORACLE_PAR_URL to upload.

# as root
cd /usr/src
export SRCCONF=/dev/null
export SRC_ENV_CONF=/dev/null
make -j2C buildworld  TARGET_ARCH=amd64 TARGET=amd64 -s \
&& make -j2C buildkernel TARGET_ARCH=amd64 TARGET=amd64 KERNCONF=GENERIC -s \
&& cd ./release \
&& -make TARGET_ARCH=amd64 TARGET=amd64 -j2C clean \
&& make  KERNCONF=GENERIC TARGET_ARCH=amd64 TARGET=amd64 \
   WITH_CLOUDWARE=yes \
   CLOUDWARE=ORACLE \
   ORACLE_PAR_URL=$(vault read -field=url secret/releng/upload) \
   -s clean cloudware-release cw-oracle-upload

[dch]

dont print out PAR_URL as info, its supposed to be a secret

[cperciva]

release/Makefile.oracle

18

I don't think there's anything needing to be fixed here; we put staging directories under /usr/obj/.../release all the time.

23

Can't this just be done with sed?

31

Am I understanding this correctly to mean that our qcow2 support in mkimg doesn't do zstd compression?

Unless this is really critical, I'd prefer to omit the compression until it can be added to mkimg, rather than pulling in yet another tool from ports.

36

I would be very surprised if it didn't work with tarballs generated by bsdtar.

Also, s/OCI/Oracle/ ? Just to avoid confusion with OCI containers, which also use tarballs...

41

Don't do this; leave it for make clean to deal with.

52

Do we need to make any sort of API call to tell Oracle "hey we uploaded an image", or does anything uploaded to the magic URL get added to their listings?

release/scripts/oracle/amd64.shapeCompatibilities.json

1

Deliberately empty file?

[dch]

re-push with missing commits

[lwhsu]

release/scripts/oracle/base.json

23

extra white line?

release/scripts/oracle/imageCapabilityData.json

2

is it possible/good to format this?

[dch]

add portinstall target

[dch]

switch to qemu-nox11 as tools flavor doesnt support zstd

raw 9.0GiB vs zip at 6.0GiB and zstd at 1.6GiB

[dch]

release/Makefile.oracle

23

phab seems to have got lines confused. Are you referring to the jq invocation?

TLDR yes; I wanted to get it tested & working with the tools I already use today, before swapping out lighter ones.

This would be a good next step.

31

its a massive difference - 9GiB/image raw -> 6GiB with mkimg, vs 1.6GiB with qemu native qcow2 & zstd. This cost is not just borne by my bucket where we upload the images, it flows down to every user who spins up the resulting image.

Alternatively I can do this manually (outside build process). But then we lose the nice chain of provenance that we have presently.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263029 for the mkimg compression.

36

ok I'll call them ORACLE_QCOW_LIST then. ORACLE_ORACLE_LIST feels weird.

bsdtar testing will be next week.

41

you mean modifying CLEANFILES+= ? will make clean remove the tmpdir as well? Or did you mean something else?

52

Sadly the magical bit is Dave clicking in their web pages, its around 30-60m to do.

Its not yet automatable but they promise me 2025Q1.

For the moment, I get a notification each time something is added to the bucket. Once their automation is available I can make this happen.

release/scripts/oracle/amd64.shapeCompatibilities.json

1

yes, strictly its not JSON without [] or {}.

release/scripts/oracle/imageCapabilityData.json

2

I vaguely recall trying this last year and IIRC it wasn't happy with the extra whitespace and new lines. I will check and update if possible.

[cperciva]

release/Makefile.oracle

23

Yes, I was referring to replacing jq with sed so we don't need to pull in the extra build tool.

31

Ok, we can pull in qemu until this can be added to mkimg.

36

Sorry that's phabricator getting confused about lines. This was about the comment "# TODO does OCI only support GNU tar format".

41

CLEANDIRS += ${ORACLE_TMP} rather than the manual rm -rf.

52

Got it. If this is the best we can do, it's the best we can do. :-)

[emaste]

release/Makefile.oracle

10

what do we need gtar for?

52

Got it. If this is the best we can do, it's the best we can do. :-)

Probably worth a comment to this effect in the src

[dch]

• switch out jq for sed
• use qemu-nox11 temporarily, I sent bofh@ a patch to allow use to use qemu-tools instead
• switch to bsd tar with gnutar format
• by switching back to sed, we have to keep the JSON blobs in their raw unreadable format
• remove the tmpdir cleanups, rely on CLEANDIRS for that

amd64 images seem to boot fine
arm64 loads kernel, but don't complete booting, likely not a build issue but a kernel one

[dch]

if GITREV is blank (e.g. during releases) then skip it and a preceding -

[dch]

Is this enough for including as a block comment in the Makefile outlining the process?
I'm likely to write a blog post with more detail, I can include that future markdown
somehere in releng repo if thats useful.

# Overview:
#
# The base image is created by cloudware-release already.
# Construct the custom OCI metdata, derived by exporting official OCI images.
# It is architecture-specific but appears mostly stable over time.
# Compress the raw image and place it in the same directory as the metadata.
# Make a GNU format tarball of these files.
# Upload the tarball to Oracle Cloud via a pre-approved curl URI, into
# the FreeBSD Foundation's Oracle Cloud account.
#
# These images go into dch@ "re" bucket in us-ashburn-1 region, which
# is mounted into the FreeBSD Foundation Oracle Marketplace account.
# Once uploaded, a manual step is needed to import the images as local
# custom images. These can then be tested within the us-ashburn-1 region.
# Once tested, follow the manual Oracle Marketplace import process to
# create a new FreeBSD version, attach the images, and initiate validation
# by Oracle. This can take up to 5 working days. Once complete, a final
# manual step is needed to make the currently private images, public.
# Syncing to all sites should take 2-3 hours.

release/Makefile.oracle

10

bsd tar with --format=gnutar is adequate

31

waiting on ports commit + build of qemu-tools with zstd compression to

[dch]

switch to flua for json handling, this is 1000x nicer than sed

[emaste]

release/scripts/oracle/generate_metadata.lua

1 ↗

(On Diff #149233)

This is an odd path; flua is inherently the stripped-down-arbitrary-lua-version for /usr/libexec. How do we end up with a flua in /usr/local?

[dch]

use system flua not dch's flua

[dch]

release/scripts/oracle/generate_metadata.lua

1 ↗

(On Diff #149233)

fixed thanks!

[emaste]

This looks reasonable to me, I think it's sensible to get this into the tree and iterate on it from there later, if necessary.