Tasks#
Ontology for generic tasks#
While tasks are unique in theory, we can have different tasks sharing some commonalities. In the Debian context in particular, we have different ways to build Debian packages with different helper programs (sbuild, pbuilder, etc.) and we want those tasks to reuse the same set of parameters so that they can be called interchangeably.
This public interface is materialized by a generic task that can be scheduled by the users and that will run one of the available implementations that can run on one of the available workers.
This section documents those generic tasks and their interface.
Task PackageBuild
#
A generic task to represent a package build, i.e. the act of transforming a source package (.dsc) into binary packages (.deb).
The task_data
associated to this task can contain the following keys:
input
(required): a dictionary describing the input datasource_artifact_id
(required): source_artifact_id pointing to a source package, it is used to retrieve the source package to build.
distribution
(required): name of the target distributionextra_repositories
(optional): a list of extra repositories to enable. Each repository is described by a dictionary with the following possible keys:sources_list
: a single-line for an APT’s sources.list fileauthentication_key
(optional): the ascii-armored public key used to authenticate the repository
host_architecture
(required): the architecture that we want to build for, it defines the architecture of the resulting architecture-specific .deb (if any)build_architecture
(optional, defaults to the host architecture): the architecture on which we want to build the package (implies cross-compilation if different from the host architecture). Can be explicitly set to the undefined value (Python’sNone
or JavaScript’snull
) if we want to allow cross-compilation with any build architecture.build_components
(optional, defaults toany
): list that can contain the following 3 words (cfdpkg-buildpackage --build=any,all,source
):any
: enables build of architecture-specific .deball
: enables build of architecture-independent .debsource
: enables build of the source package (.dsc)
build_profiles
: list of build profiles to enable during package build (cfdpkg-buildpackage --build-profiles
)build_options
: value ofDEB_BUILD_OPTIONS
during buildbuild_path
(optional, default unset): forces the build to happen through a path named according to the passed value. When this value is not set, there’s no restriction on the name of the path.
Task SystemBootstrap
#
A generic task to represent the bootstrapping of a Debian system out
of an APT repository. The end result of such a task is to generate
an artifact of category debian:system-tarball
.
The task_data
associated to this task can contain the following keys:
bootstrap_options
: a dictionary with a few global options:variant
(optional): maps to the--variant
command line option of debootstrapextra_packages
(optional): list of extra packages to include in the bootstrapped systemarchitecture
(required): the native architecture of the built Debian system. The task will be scheduled on a system of that architecture.
bootstrap_repositories
: a list of repositories used to bootstrap the Debian system. Note that not all implementations might support multiple repositories.types
(optional): a list of source types to enable amongdeb
(binary repository) anddeb-src
(source repository). Defaults to a list withdeb
only.mirror
(required): the base URL of a mirror containing APT repositories in$mirror/dists/$suite
suite
(required): name of the distribution’s repository to use for the bootstrapcomponents
(optional): list of components to use in the APT repository (e.g.main
,contrib
,non-free
, …)check_signature_with
(optional, defaults tosystem
): indicates whether we want to check the repository signature with the system-wide keyrings (system
), or with the external keyring documented in the in thekeyring
key (valueexternal
), or whether we don’t want to check it at all (valueno-check
).keyring_package
(optional): install an extra keyring package in the bootstrapped systemkeyring
(optional): provide an external keyring for the bootstrapurl
(required): URL of the external keyring to downloadsha256sum
(optional): SHA256 checksum of the keyring to validate the downloaded fileinstall
(boolean, defaults to False): if True, the downloaded keyring is installed and used in the target system.
customization_script
(optional): a script that is copied in the target chroot, executed from inside the chroot and then removed. It lets you perform arbitrary customizations to the generated system. You can use apt to install extra packages. If you want to use something more elaborated than a shell script, you need to make sure to install the appropriate interpreter during the bootstrap phase with theextra_packages
key.
Task SystemImageBuild
#
This generic task is an extension of the SystemBootstrap generic task: it should generate a disk image artifact complying with the debian:system-image definition. That disk image contains a Debian-based system matching the description provided by the SystemBootstrap interface.
The following additional keys are supported:
disk_image
format
(required): desired format for the disk image. Supported values areraw
andqcow2
.filename
(optional): filename of the generated disk imagekernel_package
(optional): name of the kernel package to install, default value is architecture specific.bootloader
(optional): name of the bootloader package to use, default value is architecture specific.partitions
(required): a list of partitions, each represented by a dictionary with the following keys:size
(required): size of the partition in gigabytesfilesystem
(required): filesystem used in the partition, can benone
for no filesystem,swap
for a swap partition, orfreespace
for free space that doesn’t result in any partition (it will thus just offset the position of the following partitions).mountpoint
(optional, defaults tonone
): mountpoint of the partition in the target system, can benone
for a partition that doesn’t get a mountpoint.
Specifications of tasks#
This section lists all the available tasks, with the input that they are accepting, the description of what they are doing, including the artifacts that they are generating.
The tasks listed in this section are those that you can use to submit work requests.
Autopkgtest task#
The task_data
associated to this task can contain the following keys:
input
(required): a dictionary describing the input data:source_arfifact_id
(required): the ID of the artifact representing the source package to be tested with autopkgtestbinary_artifacts_ids
(required): a list ofdebian:binary-packages
artifact IDs representing the binary packages to be tested with autopkgtest (they are expected to be part of the same source package as the one identified withsource_artifact_id
)context_artifacts_ids
(optional): a list ofdebian:binary-packages
artifact IDs representing a special context for the tests. This is used to trigger autopkgtests of reverse dependencies, wherecontext_artifacts_ids
is set to the artifacts of the updated package whose reverse dependencies are tested, and source/binary artifact IDs are one of the reverse dependency whose autopkgtests will be executed.
architecture
(required): the Debian architecture that will be used in the chroot or VM where tests are going to be run. The packages submitted ininput:binary_artifacts_ids
usually have a matching architecture (but need not in the case of cross-architecture package testing, eg. testing i386 packages in an amd64 system).environment_id
(required): Artifact ID of thedebian:system-tarball
ordebian:system-image
(as appropriate for the selected backend) that will be used to run the tests.backend
(optional): the virtualization backend to use, defaults toauto
where the task is free to use the most suitable backend. Other valid options areschroot
,lxc
,qemu
,podman
,unshare
.
Note
We are not supporting all backends as there’s a cost in supporting
the required infrastructure setup to make use of all the backends. For
schroot, we already have that due to the sbuild task. For podman
,
unshare
, and qemu
we can do all the setup without root
rights. For lxc
it’s the reference used by Debian and we want to
support it too.
include_tests
(optional): a list of the tests that will be executed. If not provided (or empty), defaults to all tests being executed. Translates into--test-name=TEST
command line options.exclude_tests
(optional): a list of tests that will skipped. If not provided (or empty), then no tests are skipped. Translates into the--skip-test=TEST
command line options.debug_level
(optional, defaults to 0): a debug level between 0 and 3. Translates into-d
up to-ddd
command line options.extra_apt_sources
(optional): a list of APT sources. Each APT source is described by a single line (deb http://MIRROR CODENAME COMPONENT
) that is copied to a file in /etc/apt/sources.list.d. Translates into--add-apt-source
command line options.use_packages_from_base_repository
(optional, defaults to False): if True, then we pass--apt-default-release=$DISTRIBUTION
with the name of the base distribution given in thedistribution
key.environment
(optional): a dictionary listing environment variables to inject in the build and test environment. Translates into (multiple)--env=VAR=VALUE
command line options.needs_internet
(optional, defaults to “run”): Translates directly into the--needs-internet
command line option. Allowed values are “run”, “try” and “skip”.fail_on
(optional): indicates whether the work request must be marked as failed in different scenario identified by the following sub-keys:failed_test
(optional, defaults to true): at least one test has failed (and the test was not marked as flaky).flaky_test
(optional, defaults to false): at least one flaky test has failed.skipped_test
(optional, defaults to false): at least one test has been skipped.
timeout
(optional): a dictionary where each key/value pair maps to the corresponding--timeout-KEY=VALUE
command line option with the exception of theglobal
key that maps to--timeout=VALUE
. Supported keys areglobal
,factor
,short
,install
,test
,copy
andbuild
.
Note
At this point, we have voluntarily not added any key for the
--pin-packages
option because that option is not explicit enough:
differences between the mirror used to schedule jobs and the mirror
used by the jobs result in tests that are not testing the version that
we want. At this point, we believe it’s better to submit all modified
packages explicitly via input:context_artifacts_ids
so that we are
sure of the .deb that we are submitting and testing with. That way we
can even test reverse dependencies before the modified package is
available in any repository.
This assumes that we can submit arbitrary .deb on the command line and that they are effectively used as part of the package setup.
autopkgtest is always run with the options --apt-upgrade
--output-dir=ARTIFACT-DIR --summary=ARTIFACT-DIR/summary --no-built-binaries
.
An artifact of category debian:autopkgtest
is generated and its
content is a copy of what’s available in the ARTIFACT-DIR
(except
for files in binaries/
, they are excluded to save space). The artifact has
“relates to” relationships for the artifacts used as input that are part of the
source package being tested.
The data
field of the artifact has the following structure:
results
: a dictionary with details about the tests that have been run. Each key is the name of the test (as shown in the summary file) and the value is another dictionary with the following keys:status
: one ofPASS
,FAIL
,FLAKY
orSKIPPED
details
: more details when available
cmdline
: the complete command line that has been used for the runsource_package
: a dictionary with some information about the source package hosting the tests that have been run. It has the following sub-keys:name
:the name of the source packageversion
: the version of the source packageurl
: the URL of the source package
architecture
: the architecture of the system where tests have been rundistribution
: the distribution of the system where tests have been run (formatted asVENDOR:CODENAME
)
Debootstrap task (NOT IMPLEMENTED YET)#
The debootstrap
task implements the SystemBootstrap interface except that it only supports a single
repository in the bootstrap_repositories
key.
On top of the keys defined in that interface, it also supports the
following additional keys in task_data
:
bootstrap_options
script
: last parameter on debootstrap’s command line
The various keys in the first entry of bootstrap_repositories
are mapped to the
corresponding command line options and parameters:
mirror
,suite
andscript
map to positional command line parameterscomponents
maps to--components
check_signature
maps to--check-gpg
or--no-check-gpg
keyring_package
maps to an extra package name in--include
keyring
maps to--keyring
The following keys from bootstrap_options
are also mapped:
* variant
maps to --variant
* extra_packages
maps to --include
Mmdebstrap task#
The mmdebstrap
task fully implements the SystemBootstrap interface.
On top of the keys defined in that interface, it also supports the
following additional keys in task_data
:
bootstrap_options
use_signed_by
(defaults to True): if set to False, then we do not pass the keyrings to APT via theSigned-By
sources.list option, instead we rely on the--keyring
command line parameter.
The keys from bootstrap_options
are mapped to command line options:
variant
maps to--variant
(and it supports more values than debootstrap, see its manual page)extra_packages
maps to--include
The keys from bootstrap_repositories
are used to build a sources.list
file that is then fed to mmdebstrap
as input.
SimpleSystemImageBuild task (NOT IMPLEMENTED YET)#
The simplesystemimagebuild
task implements the SystemImageBuild interface except that it expects a single
entry in the list of partitions: the entry for the root filesystem (thus
with a mountpoint of /
).
In terms of compliance with the SystemBootstrap
interface, the
bootstrap phase only uses a single repository but the remaining
repositories are enabled after the bootstrap.
This task is implemented with the help of the debos
tool.
Lintian task#
The task_data
associated to this task can contain the following keys:
input
(required): a dictionary of values describing the input data, one of the sub-keys is required but both can be given at the same time too.source_arfifact_id
(optional): the ID of the artifact representing the source package to be tested with lintianbinary_artifacts_ids
(optional): a list ofdebian:binary-packages
artifact IDs representing the binary packages to be tested with lintian (they are expected to be part of the same source package as the one identified withsource_artifact_id
)
Note
While it’s possible to submit only a source or only a single binary artifact, you should aim to always submit source + arch-all + arch-any related artifacts to have the best test coverage as some tags can only be emitted when lintian has access to all of them at the same time.
output
(optional): a dictionary of values controlling some aspects of the generated artifactssource_analysis
(optional, defaults to True): indicates whether we want to generate thedebian:lintian
artifact for the source packagebinary_all_analysis
(optional, defaults to True): same assource_analysis
but for thedebian:lintian
artifact related toArchitecture: all
packagesbinary_any_analysis
(optional, defaults to True): same assource_analysis
but for thedebian:lintian
artifact related toArchitecture: any
packages
target_distribution
(optional): the fully qualified name of the distribution that will provide the lintian software to analyze the packages. Defaults todebian:unstable
.min_lintian_version
(optional): request that the analysis be performed with a lintian version that is higher or equal to the version submitted. If a satisfying version is not pre-installed and cannot be installed withapt-get install lintian
, then the work request is aborted.include_tags
(optional): a list of the lintian tags that are allowed to be reported. If not provided (or empty), defaults to all. Translates into the--tags
or--tags-from file
command line option.exclude_tags
(optional): a list of the lintian tags that are not allowed to be reported. If not provided (or empty), then no tags are hidden. Translates into the--suppress-tags
or--suppress-tags-from file
command line option.fail_on_severity
(optional, defaults tonone
): if the analysis emits tags of that severity or higher, then the task will return a “failure” instead of a “success”. Valid values are (in decreasing severity) “error”, “warning”, “info”, “pedantic”, “experimental”, “overridden”. “none” is a special value indicating that we should never fail.
The lintian runs will always use the options --display-level
">=classification"
(>=pedantic
in jessie) --no-cfg
--display-experimental --info --show-overrides
to collect the full set of
data that lintian can provide.
Note
Current lintian can generate “masked” tags (with M: prefix) when you
use --show-overrides
. For the purpose of debusine, we entirely
ignore those tags on the basis that it’s lintian’s decision to hide
them (and not the maintainer’s decision) and as such, they don’t bring
any useful information. Lintian is full of exceptions to not emit some
tags and the fact that some tags rely on a modular exception mechanism
that can be diverted to generate masked tags is not useful to package
maintainers.
For those reasons, we suggested to lintian’s maintainers to entirely stop emitting those tags in https://bugs.debian.org/1053892
Between 1 to 3 artifacts of category debian:lintian
will be generated (one
for each source/binary package artifact submitted) and they will have a
“relates to” relationship with the corresponding artifact that has been
analyzed. These artifacts contain a lintian.txt
file with the raw
(unfiltered) lintian output and an analysis.json
file with the details
about all the tags discovered (in a top-level tags
key), some
statistics/summary (in a top-level summary
key) and a version
key
with the value 1.0
if the content follows the (initial) JSON structure
described below.
The summary
key is also duplicated in the data
field of the
artifact. It is a dictionary with the following keys:
tags_count_by_severity
: a dictionary with a sub-key for each of the possible severities documenting the number of tags of the corresponding severity that have been emitted by lintianpackage_filename
: a dictionary mapping the name of the binary or source package to its associated filename (will be a single key dictionary for the case of a source package lintian analysis, and a multiple keys one for the case of an analysis of binary packages)tags_found
: the list of non-overridden tags that have been found during the analysisoverridden_tags_found
: the list of overridden tags that have been found during the analysislintian_version
: the lintian version used for the analysisdistribution
: the distribution in which lintian has been run
The tags
key in analysis.json is a sorted list of tags where each tag
is represented with a dictionary. The list is sorted by the following
criteria:
binary package name in alphabetical order (if relevant)
severity (from highest to lowest)
tag name (alphabetical order)
tag details (alphabetical order)
Each tag is represented with the following fields:
tag
: the name of the tagseverity
: one of the possible severities (see below for full list)package
: the name of the binary or source package (there is no risk of confusion between a source and a binary of the same name as the artifact with the analysis is dedicated either to a source packages or to a set of binary packages, but not to both at the same time)note
: the details associated to the tag (those are printed after the tag name in the lintian output)pointer
: the optional part shown between angle brackets that gives a specific location for the issue (often a filename and a line number)explanation
: the long description shown after a tag with--info
, aka the lines prefixed withN:
(they always start and end with an empty line)comment
: the maintainer’s comment shown on lines prefixed withN:
just before a given overridden tag (those lines can be identified by the lack of an empty line between them and the tag)
Note
Here’s the ordered list of all the possible severities (from highest to lowest):
error
warning
info
pedantic
experimental
overridden
classification
Note that experimental
and overridden
are not true tag
severities, but lintian’s output replaces the usual severity field
for those tags with X
or O
and it is thus not easily possible
to capture the original severity.
And while classification
is implemented like a low-severity issue,
those tags do not represent real issues, they are just a convenient way
to export data generated while doing the analysis.
Sbuild task#
Regarding inputs, the sbuild
task is compatible with the ontology
defined for Task PackageBuild even though it implements only
a subset of the possible options at this time.
Currently unsupported PackageBuild
task keys:
extra_repositories
build_architecture
/build_profiles
build_options
build_path
Specific options:
sbuild_options
: command-line options to pass tosbuild
. Note: this is a temporary feature; to avoid arbitrary command execution on the worker, we will prevent that in the long term.
Output artifacts and relationships:
debian:package-build-log
: sbuild outputrelates-to:
source_artifact_id
relates-to: b
debian:binary-packages
: the binary packages (*.deb
) built from the source packagerelates-to:
source_artifact_id
debian:upload
: b plus the right administrative files (.changes
,.buildinfo
) necessary for its binary uploadextends: b
relates-to: b
debusine:work-request-debug-logs
: debusine-specific worker logsrelates-to:
source_artifact_id
Piuparts task#
A specific task to represent a binary package check using the
piuparts
utility.
The task_data
associated to this task can contain the following keys:
input
(required): a dictionary describing the input databinary_artifacts_ids
(required): a list ofdebian:binary-packages
artifact IDs representing the binary packages to be tested. Multiple Artifacts can be provided so as to support e.g. testing binary packages from split indep/arch builds.
distribution
(required): name of the target distribution.host_architecture
(required): the architecture that we want to test on.
The piuparts
output will be provided as a new artifact.