fedora-createrepo-image/docs/scripts.md
XLion 964aa339d1
Set rpm's default backend to sq; sign rpm to both v4 and v6 format; import key to sq; (#525)
* Add sq

* repomd sq

* shellcheck

* fix: request both v4 and v6 RPM signatures

* fix: complete Sequoia signing contract

* fix: require RPM v4-compatible signing keys

* docs: streamline agent guidance and track downstream compatibility

* fix: reject invalid RPM verification and match explicit signing keys

* refactor: share RPM signing command across retries

* refactor: simplify key import and signing test fixtures
2026-10-28 10:32:32 +08:00

545 lines
20 KiB
Markdown

# Scripts
## Overview
Files under `scripts/` are copied into `/usr/bin` by the container image. Package repository workflows use these commands as CI building blocks.
The scripts should stay small. They should wrap one clear responsibility, expose stable flags, and avoid hiding repository-specific decisions behind a generic abstraction.
## General Design
### CLI Conventions
- Use Bash for small orchestration around existing CLI tools.
- Start scripts with `#!/bin/bash` and `set -euo pipefail` unless there is a concrete reason to use something else.
- Keep option names descriptive and stable; let callers supply environment variable names, such as `--key-env GPG_PRIVATE_KEY`.
- Validate required dependencies near startup and report missing tools clearly.
- Validate required inputs before network requests, signing, or destructive file operations.
- Prefer explicit arguments for caller-specific behavior.
### Keep Scripts Focused
Each script should have one main job.
Good examples:
- `xlion-repo-utils-gh` handles GitHub release metadata and checksum verification.
- `xlion-repo-gpg-import` imports a private key into GnuPG and Sequoia and reports the fingerprint.
- `xlion-repo-repackage-deb` repackages DEB files with consistent fpm options.
- `xlion-repo-repackage-rpm` repackages RPM files with consistent fpm options.
- `xlion-repo-rpm-createrepo` creates RPM repository metadata for one or more directories.
- `xlion-repo-rpm-sign-repomd` signs RPM repository metadata for one or more directories.
- `xlion-repo-rpm-sign-packages` signs RPM packages and verifies signatures.
Avoid scripts that do too much, such as one command that downloads assets, repackages packages, signs metadata, publishes repositories, and uploads artifacts. Those workflows are easier to reason about when they remain separate steps.
### Keep Product Rules Near the Caller
Not every repeated shell line needs a shared abstraction. For example, RustDesk release assets have product-specific naming rules around architecture, `sciter`, nightly builds, and SUSE RPMs. Simple workflow filtering with `grep` is often clearer than a generic selector command with many flags.
Use shared scripts for stable mechanics:
- GitHub release API access
- checksum verification
- GPG key import
- package reversion or repackaging when the behavior is identical
- RPM signing and metadata creation when those helpers exist
Keep caller-specific choices in the workflow:
- which upstream repository to use
- which release tag to use
- which package names or architectures to include
- how to lay out product-specific publish directories
### stdout and stderr
If a script is useful in command substitution, stdout should be machine-readable.
For example, `xlion-repo-gpg-import` prints only the fingerprint to stdout. Status messages go to stderr. This allows safe usage like:
```bash
fingerprint="$(xlion-repo-gpg-import --key-env GPG_PRIVATE_KEY)"
```
Scripts that generate human-readable reports can write normal logs to stdout if they are not expected to be used this way. Be deliberate and document the behavior.
### Secrets
Scripts must not print secrets. This includes:
- GPG private keys
- GitHub tokens
- Cloudflare R2 credentials
- full environment dumps
Accept secret material through environment variables or stdin. Prefer letting the caller choose the environment variable name with a flag such as `--key-env`.
### Temporary State
Temporary state should have an obvious owner.
For GPG, `GNUPGHOME` must remain available to later steps that run `gpg` or `aptly`. RPM signing and RPM repository metadata signing use Sequoia, so `SEQUOIA_HOME` must remain available to later steps that run `sq` or the `rpmsign` sq backend. `xlion-repo-gpg-import` may create both homes, but it intentionally does not delete either one. With `--github-env`, it writes the paths to `$GITHUB_ENV`; GitHub Actions makes them available to subsequent steps. The workflow should remove both homes in an `if: always()` cleanup step after signing and publishing are finished.
## Validation Requirements
For every changed Bash script, run `bash -n scripts/SCRIPT_NAME` and `scripts/SCRIPT_NAME --help`. Each script reference below lists its minimum commands and practical functional checks.
Use temporary local keys and packages for functional tests. Verify behavior without real signing secrets or repository publishing. For key import, confirm the fingerprint on stdout, private-key availability in both stores, and the documented GitHub environment/output entries. Keep the generated homes available for subsequent signing checks; the caller owns cleanup.
When adding or changing a script, update its reference here with its purpose, ownership boundaries, common examples, and minimum test commands.
## Script Reference
### `xlion-repo-utils-gh`
`xlion-repo-utils-gh` has two modes.
Fetch mode:
```bash
xlion-repo-utils-gh --workfolder "$WORKFOLDER" --repo rustdesk/rustdesk
```
Fetch a specific tag:
```bash
xlion-repo-utils-gh --workfolder "$WORKFOLDER" --repo rustdesk/rustdesk --tag nightly
```
Verify downloaded files:
```bash
xlion-repo-utils-gh --verify --workfolder "$WORKFOLDER"
```
Fetch mode writes:
- `api.json`
- `dl_urls.txt`
- `sha256.txt` when release asset digests are available
Verify mode reads `sha256.txt` from the workfolder, checks only files actually downloaded there, and fails if any downloaded file has a checksum mismatch.
Typical caller flow:
```bash
xlion-repo-utils-gh --workfolder "$WORKFOLDER" --repo pendulum-project/ntpd-rs
cd "$WORKFOLDER"
grep -E '\.deb$|\.rpm$' dl_urls.txt | wget2 --input-file=-
xlion-repo-utils-gh --verify --workfolder "$WORKFOLDER"
```
What this script owns:
- GitHub release API calls
- writing release URL and checksum files
- checksum verification
What this script does not own:
- deciding which packages a product repo wants
- downloading every possible asset automatically
- repository metadata generation
- package signing
Minimum tests:
```bash
bash -n scripts/xlion-repo-utils-gh
scripts/xlion-repo-utils-gh --help
```
If network access is acceptable, also test a small fetch-only scenario against a known public release.
### `xlion-repo-gpg-import`
`xlion-repo-gpg-import` imports a private key into both GnuPG and Sequoia and prints the imported fingerprint. The two key stores are separate: GnuPG consumers such as aptly use `GNUPGHOME`, while RPM signing helpers use `SEQUOIA_HOME` for `sq`.
Recommended GitHub Actions usage:
```bash
xlion-repo-gpg-import \
--key-env GPG_PRIVATE_KEY \
--create-gnupghome \
--github-env \
--name GPG_FINGERPRINT
```
This writes these variables for later workflow steps:
```bash
GNUPGHOME=/tmp/...
SEQUOIA_HOME=/tmp/...
GPG_TTY=
GPG_FINGERPRINT=...
```
`--github-env` appends these values to `$GITHUB_ENV`; GitHub Actions applies that file when it starts the next step. The importer cannot change the environment of the step that invoked it. `GNUPGHOME` is created when it is unset and `--create-gnupghome` is given. `SEQUOIA_HOME` is created when it is unset. A caller-provided home is reused.
Use stdin for local testing or when the caller does not want to store the key in an environment variable:
```bash
xlion-repo-gpg-import --stdin --create-gnupghome < private-key.asc
```
Use RPM key import when later steps call `rpmsign` or verify RPM signatures:
```bash
xlion-repo-gpg-import \
--key-env GPG_PRIVATE_KEY \
--create-gnupghome \
--github-env \
--rpmkeys-import
```
What this script owns:
- reading private key material from stdin or a caller-selected environment variable
- creating `GNUPGHOME` when requested and creating `SEQUOIA_HOME` when needed
- importing the private key with both `gpg` and `sq`
- parsing and reporting the fingerprint
- optionally importing the public key into rpm's key database
What this script does not own:
- publishing APT repositories
- signing RPM packages
- signing `repomd.xml`
- deleting `GNUPGHOME` or `SEQUOIA_HOME`; the calling workflow owns cleanup
- choosing the secret name used by a workflow
Minimum tests:
```bash
bash -n scripts/xlion-repo-gpg-import
scripts/xlion-repo-gpg-import --help
```
Functional test with a throwaway key:
```bash
tmp_src="$(mktemp -d)"
tmp_env="$(mktemp)"
chmod 700 "$tmp_src"
GNUPGHOME="$tmp_src" \
gpg --batch --passphrase '' \
--quick-generate-key "XLion Repo Test <[email protected]>" \
default default 1d >/dev/null 2>&1
key_data="$(GNUPGHOME="$tmp_src" gpg --batch --armor --export-secret-keys "[email protected]")"
fingerprint="$({
GITHUB_ENV="$tmp_env" \
GPG_PRIVATE_KEY="$key_data" \
scripts/xlion-repo-gpg-import \
--key-env GPG_PRIVATE_KEY \
--create-gnupghome \
--github-env
})"
test -n "$fingerprint"
grep -q '^GNUPGHOME=' "$tmp_env"
grep -q '^SEQUOIA_HOME=' "$tmp_env"
grep -qx 'GPG_TTY=' "$tmp_env"
grep -q '^GPG_FINGERPRINT=' "$tmp_env"
imported_home="$(grep '^GNUPGHOME=' "$tmp_env" | cut -d= -f2-)"
sequoia_home="$(grep '^SEQUOIA_HOME=' "$tmp_env" | cut -d= -f2-)"
GNUPGHOME="$imported_home" gpg --batch --list-secret-keys "$fingerprint" >/dev/null
test -d "$sequoia_home"
SEQUOIA_HOME="$sequoia_home" sq key list --cert "$fingerprint" >/dev/null
rm -rf "$tmp_src" "$imported_home" "$sequoia_home" "$tmp_env"
```
### `xlion-repo-repackage-deb`
`xlion-repo-repackage-deb` repackages `.deb` files from a source directory into a target directory.
Latest-style repackaging without changing the package version:
```bash
xlion-repo-repackage-deb \
--source-dir ori \
--target-dir . \
--deb-recommends xlion-repo-archive-keyring \
--remove-source-dir
```
Nightly-style repackaging with a date suffix:
```bash
xlion-repo-repackage-deb \
--source-dir ori \
--target-dir . \
--date-version \
--deb-recommends xlion-repo-archive-keyring \
--remove-source-dir
```
This turns an original version such as `1.2.3` into `1.2.3+YYYYMMDD`.
What this script owns:
- scanning one source directory for `.deb` files
- reading original package versions with `dpkg-deb`
- calling `fpm -t deb -s deb`
- applying `--deb-compression`
- optionally adding `--deb-recommends`
- optionally removing the source directory after success
What this script does not own:
- downloading upstream release assets
- deciding which architecture packages should exist
- creating APT repository metadata
- signing APT repositories
Minimum tests:
```bash
bash -n scripts/xlion-repo-repackage-deb
scripts/xlion-repo-repackage-deb --help
```
Functional testing should use a small throwaway `.deb` package. Confirm that the repackaged output exists, and when `--date-version` is used, confirm the output package `Version` contains `+YYYYMMDD`.
### `xlion-repo-repackage-rpm`
`xlion-repo-repackage-rpm` repackages `.rpm` files from a source directory into a target directory. By default, the target directory is the parent of the source directory. This matches common `wwwroot/channel/ori` usage.
Latest-style repackaging without changing the package version:
```bash
xlion-repo-repackage-rpm \
--source-dir wwwroot/latest/ori \
--remove-source-dir
```
Nightly-style repackaging with a date suffix:
```bash
xlion-repo-repackage-rpm \
--source-dir wwwroot/nightly/ori \
--date-version \
--remove-source-dir
```
This turns an original version such as `1.2.3` into `1.2.3+YYYYMMDD`.
What this script owns:
- scanning one source directory for `.rpm` files
- reading original package versions with `rpm -qp`
- calling `fpm -t rpm -s rpm`
- applying `--rpm-compression`
- optionally removing the source directory after success
What this script does not own:
- splitting SUSE and non-SUSE packages
- creating RPM repository metadata
- importing GPG keys
- signing RPM packages
- signing `repomd.xml`
Minimum tests:
```bash
bash -n scripts/xlion-repo-repackage-rpm
scripts/xlion-repo-repackage-rpm --help
```
Functional testing should use a small throwaway `.rpm` package. Confirm that the repackaged output exists, and when `--date-version` is used, confirm the output package version contains `+YYYYMMDD`.
### `xlion-repo-rpm-createrepo`
`xlion-repo-rpm-createrepo` runs `createrepo_c` for one or more RPM repository directories. Multiple `--dir` values are processed concurrently, and each background job is checked with `wait`.
RustDesk RPM-style usage:
```bash
xlion-repo-rpm-createrepo \
--dir wwwroot/latest \
--dir wwwroot/latest-suse \
--dir wwwroot/nightly \
--dir wwwroot/nightly-suse
```
Single-directory usage:
```bash
xlion-repo-rpm-createrepo --dir ~/.aptly/public/rpm
```
Pass extra arguments to `createrepo_c` after `--`:
```bash
xlion-repo-rpm-createrepo --dir wwwroot/latest -- --update
```
What this script owns:
- validating that each repo directory exists
- warning when a repo directory has no direct `.rpm` files
- running `createrepo_c` for each directory
- running multiple directories concurrently
- failing if any `createrepo_c` process fails
What this script does not own:
- splitting product-specific RPM directories
- signing RPM packages
- signing `repomd.xml`
- uploading repository artifacts
Minimum tests:
```bash
bash -n scripts/xlion-repo-rpm-createrepo
scripts/xlion-repo-rpm-createrepo --help
```
Functional testing should create one or more throwaway directories containing small RPM packages, run the helper, and confirm each directory contains `repodata/repomd.xml`.
### `xlion-repo-rpm-sign-repomd`
`xlion-repo-rpm-sign-repomd` signs `repodata/repomd.xml` with Sequoia's `sq` backend for one or more RPM repository directories. Multiple `--dir` values are processed concurrently, and each background job is checked with `wait`.
The helper requires `sq` and checks that it is available before signing.
RustDesk RPM-style usage:
```bash
xlion-repo-rpm-sign-repomd \
--dir wwwroot/latest \
--dir wwwroot/latest-suse \
--dir wwwroot/nightly \
--dir wwwroot/nightly-suse \
--gpg-key "$GPG_FINGERPRINT"
```
Single-directory usage:
```bash
xlion-repo-rpm-sign-repomd --dir ~/.aptly/public/rpm --gpg-key "$GPG_FINGERPRINT"
```
The `--gpg-key` value can be an OpenPGP key ID or fingerprint; a full fingerprint is recommended. The helper passes it to `sq` as `--signer`. The `--gpg-key` flag name is retained for compatibility with existing workflows.
What this script owns:
- validating that each repo directory exists
- validating that each `repodata/repomd.xml` exists
- signing each `repomd.xml` with `sq --batch --overwrite sign`
- running multiple directories concurrently
- failing if any signing process fails
What this script does not own:
- importing GPG keys
- importing the signing key into Sequoia
- choosing the signing key secret name
- signing RPM packages
- creating RPM repository metadata
Minimum tests:
```bash
bash -n scripts/xlion-repo-rpm-sign-repomd
scripts/xlion-repo-rpm-sign-repomd --help
```
Functional testing should use a throwaway OpenPGP key imported into a throwaway Sequoia key store and a throwaway RPM repo directory with `repodata/repomd.xml`. Set `SEQUOIA_HOME` to that store when needed, then confirm that `repomd.xml.asc` is created.
### `xlion-repo-rpm-sign-packages`
`xlion-repo-rpm-sign-packages` signs `.rpm` files in one or more directories with Sequoia's `sq` backend, verifies the resulting signatures with `rpmkeys -Kv`, and retries signing when verification fails. It captures each `rpmkeys -Kv` result once and rejects any non-zero exit or reported `BAD`/`NOT OK` result. A package is accepted only when a signature line reports `OK`, includes the requested key in its explicit fingerprint or key ID field, and uses a supported RPM v4 signing algorithm (`RSA`, `DSA`, `ECDSA`, or `EdDSA`); this is the signing-algorithm compatibility gate.
Signing uses the Sequoia `sq` backend and RPM 6's `--rpmv4 --rpmv6` options to request both RPM signature formats, including during retries. The signing key must already be imported into sq, and `SEQUOIA_HOME` must point to that key store when it is not the default. The `--gpg-key` value should be an OpenPGP key ID or fingerprint; a full fingerprint is recommended.
RPM v4 compatibility requires a signing algorithm supported by RPM v4. The helper uses the matching `rpmkeys -Kv` signature algorithm token as this compatibility gate and rejects other algorithms, such as ML-DSA. Key matching is case-insensitive and exact. It accepts full fingerprint equality; it derives a 16-hex-character key ID from a v4 fingerprint's final 64 bits or a v6 fingerprint's first 64 bits. An 8-hex-character legacy ID matches the final eight characters of that 16-character key ID. A v3 fingerprint is matched only in full because its key ID comes from the RSA modulus. `rpmsign --rpmv4 --rpmv6` remains responsible for RPM v4 and v6 signature storage.
RustDesk RPM-style usage:
```bash
xlion-repo-rpm-sign-packages \
--dir wwwroot/latest \
--dir wwwroot/latest-suse \
--dir wwwroot/nightly \
--dir wwwroot/nightly-suse \
--gpg-key "$GPG_FINGERPRINT"
```
Single-directory usage:
```bash
xlion-repo-rpm-sign-packages --dir ~/.aptly/public/rpm --gpg-key "$GPG_FINGERPRINT"
```
Retry options:
```bash
xlion-repo-rpm-sign-packages \
--dir wwwroot/latest \
--gpg-key "$GPG_FINGERPRINT" \
--retry 3 \
--retry-sleep 2
```
What this script owns:
- collecting direct child `.rpm` files from one or more directories
- signing each RPM with `rpmsign --key-id --rpmv4 --rpmv6` using sq
- requiring successful overall `rpmkeys -Kv` verification and a matching signature line for the requested key with an RPM v4-supported algorithm (`RSA`, `DSA`, `ECDSA`, or `EdDSA`)
- verifying signatures with `rpmkeys -Kv`
- checking that the signature line's explicit fingerprint or key ID field matches the requested `--gpg-key`
- retrying signing when verification fails
- checking that `find`, `rpmkeys`, `rpmsign`, `sleep`, `sort`, and `sq` are available
What this script does not own:
- importing the OpenPGP signing key into Sequoia
- importing public keys into rpmkeys
- creating RPM repository metadata
- signing `repomd.xml`
- deciding whether PR workflows should skip signing
Minimum tests:
```bash
bash -n scripts/xlion-repo-rpm-sign-packages
scripts/xlion-repo-rpm-sign-packages --help
tests/test-rpm-sign-packages.sh
```
The regression test runs the public CLI with temporary controlled `rpmkeys`, `rpmsign`, and `sq` commands in a temporary directory. It covers v3/v4/v6 fingerprint and key ID matching, key ID collisions, unsupported algorithms, failed verifier output/status, payload digest failures, invalid-key rejection before signing, and the retry path; all fixtures are removed when it exits.
Keep RPM package signing sequential unless there is a proven need to parallelize; signing is more sensitive to concurrent access than metadata generation.
Functional testing should use a throwaway RPM and a throwaway RSA signing key. Import the private key into sq and the public key into rpmkeys before running this helper, then confirm `rpmkeys -Kv` reports the expected key with a supported algorithm and `OK` status and the helper accepts it. Test a key whose matching output has an unsupported algorithm such as ML-DSA and confirm that the helper rejects it. Also reject a supported algorithm with the wrong fingerprint or a `NOT OK` status. Exercise each of the RPM v4-supported signing algorithms (RSA, DSA, ECDSA, and EdDSA) where available; `rpmsign --rpmv4 --rpmv6` remains responsible for signature storage.
## Adding a New Script
Before adding a new script, check whether it is actually a stable shared mechanic. If the logic is mostly product-specific filtering or directory layout, keep it in the calling workflow.
When a new script is justified:
- Add it under `scripts/` with an executable bit.
- Include `--help` output.
- Validate dependencies and inputs.
- Keep stdout behavior documented.
- Add or update documentation in this file.
- Run `bash -n` and at least one practical functional test.
## Container Integration
The `Dockerfile` currently contains:
```Dockerfile
COPY scripts/* /usr/bin
```
That means every file under `scripts/` becomes available in the image PATH. Do not add non-executable notes, fixtures, or test data under `scripts/`; put documentation under `docs/` instead.