This is a pirate version of Modularity (TM) module build
<<<<< Brainstorm dump
we'd like to ask for a help with a best way to write "module build" prototype.
As of now think of module as rpm repo with extra metadata (yaml file defining core packages of given module and also it's build requires (requires are just other modules), + module version)
so think of module webserver-1.0 (which has httpd, lighttpd) which build-requires e.g. module devel-2.0 (glibc ...) + core-1.0 (coreutils, bash) and perhaps httpd itself (webserver is probably wrong example for requiring itself but yeah ...)
What "module build webserver" means:
build package list from webserver (httpd) the sources would be referenced by their commit IDs
this will need to generate buildroot which consists of devel-2.0 and core-1.0 and will chainbuild (since some packages within module might depend on other packages from module ... nss, nspr ...). There might be some overrides involved, even when we didn't really think of them (perhaps they might not be necessary by design)
so now we have all builds ... then pungi-createrepo (or perhaps koji signed repo) takes it job and generates repository out of these build packages.
the receipt for module (yaml file) would be stored in dist-git. We don't have proper module depsolving yet but think of module as rpm metapackage ... which has all rpm depsolving capabilities.
It's not from my head so I hope that I translated it properly.
The idea is to work on some mock-based content-generator for quick prototyping. Basically we need chainbuild and generation of tags/buildroots/targets based on receipt if that makes sense.
Check out skvidal's `mockchain`. It may help?
Lubos
CURRENT PROBLEMS AND HOW TO START WITH PROTOTYPING
RPM Build separate from module build
- generate buildroots and tags based on module definition (buildrequires?)
- module -> tag mapping?
>>>>>>>
- too late to solve it today!
Architecture of build
User story
As a module creator, (referenced as The Creator) I want to rule all modules, but especially I want to create module core for some unknown reason ...
fedpkg co modules/core
cd core.git
cat > core.yaml
document: modulemd
version: 0
data:
name: core
version: 0-1
summary: The core module
description: >
The minimal set of packages for modularity.
license:
module:
- MIT
content: [] # This should be populated by other tools
components:
rpms:
dependencies: True
fulltree: False
packages:
bash: ~ # Fundamental to the Linux CLI experience, scripting
bzip2: ~ # Standard CLI tool
chrony: ~ # Plumbing for timekeeping
coreutils: ~ # Fundamental to the Linux CLI experience, scripting
dnf: ~ # Plumbing for package management
dracut: ~ # Required to boot hardware and virt guests
filesystem: ~ # Basic directories
glibc: ~ # Plumbing required for all applications
grub2: ~ # Required to boot hardware and virt guests
gzip: ~ # Standard CLI tool
initscripts: ~ # support for third-party initscripts
iproute: ~ # Required for network initialization
kernel-core: ~ # Required to boot hardware and virt guests
less: ~ # Standard CLI tool
openssh-clients: ~ # Fundamental to the Linux CLI experience, scripting
procps-ng: ~ # Fundamental to the Linux CLI experience, scripting
rpm: ~ # Plumbing for package management
sed: ~ # Fundamental to the Linux CLI experience, scripting
sudo: ~ # Fundamental to the Linux CLI experience, scripting
systemd: ~ # Plumbing required for all non-container install types
tar: ~ # Standard CLI tool
util-linux: ~ # Fundamental to the Linux CLI experience, scripting
xz: ~ # Standard CLI tool
zlib: ~ # Standard CLI tool and library
^D
fedpkg commit -pm "Initial commit"
fedpkg module-build --nowait # died as a virgin while waiting several years for kojira to regen all buildroot repos
----------------------------------------------------------
card idea: what will be installed in buildroot?
A) whole module?
B) just core set of packages defined in koji by rel-eng (koji list-groups $BUILDROOT) + build deps of packages (current behavior for rpm builds)
C) just the build deps of the packages
C is winner! (note we'll most probably need to have at least empty build and srpm-build needs to contain minimal set of default packages ... for actually being able to do rpmbuild -bs file.spec)
----------------------------------------------------------
----------------------------------------------------------
card idea: order of required modules (issue one package provided by multiple modules)
Module (build)requirements should be an ordered-set to ensure order of required modules. Build system is expected to respect such order.
Undefined is a winner? (can yaml guarantee order that will be passed to ordered dict?)
I think yaml can do that, yes?
----------------------------------------------------------
----------------------------------------------------------
card idea: define where to read default set of arches for module build
big requirement here is set of arches in koji build-tag which needs to be set to non-zero value
example:
$ koji taginfo f24-build | grep -i arches
Arches: armv7hl i686 x86_64
----------------------------------------------------------
----------------------------------------------------------
card idea: how to rebuild module without bumping rpm specs
Note: changing dist-tag (.el6, .fc24) requires:
1) rebuild of system-release (fedora-release, redhat-release ...) witch changed dist_tag macro in system-release rpm-spec
2) trigger build tag it to specific buildroot that we want to affect
3) wait for createrepo to finish.
4) rebuild desired package
idea: have $module-build metapackage containing macro affecting dist-tag in this particular module
Example disttag: httpd-2.4.18-1.webserver@2.4.18@11
----------------------------------------------------------
----------------------------------------------------------
card idea: automated generation/maintenance of targets
validation of module name/version (typos could result into generating multiple otherwise unusable targets)
automated part would be:
1. creation of target (if it doesn't exist)
2. maintenance of buildroot depenedencies (defined in yaml)
requirement of package-version which is supposed to be built as part of "building this module" would be solved by chainbuild?
do we need overrides
fedpkg module-build
...
validate name / version of module (name should be same as name of the repo)
check whether build target exists
check whether build tag inheritance matches module build requirement
check whether arches are matching expectations (card define where to read default set of arches for module build)
if needed create/change koji build-tag arches or inheritance
if needed create koji target (it's more "kojira effective" to create target after the build-tag-inheritance is in place ... triggers fewer tasks)
wait for koji createrepo tasks to finish (generates repodata above set of packages in build-tag)
perform (chain) build
----------------------------------------------------------