From 40cd6a1446b9975ec4b60cd7efcfeb7fdb99ef1d Mon Sep 17 00:00:00 2001
From: Christoph Groth <christoph.groth@cea.fr>
Date: Fri, 15 Dec 2017 11:44:26 +0100
Subject: [PATCH] add instructions on Debian/Ubuntu packaging
---
RELEASE.rst | 325 ++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 314 insertions(+), 11 deletions(-)
diff --git a/RELEASE.rst b/RELEASE.rst
index 760ee226..e294f01f 100644
--- a/RELEASE.rst
+++ b/RELEASE.rst
@@ -1,13 +1,18 @@
Making a Kwant release
======================
+
This document guides a contributor through creating a release of Kwant.
+
Preflight checks
################
+
The following checks should be made *before* tagging the release.
+
Check that all issues are resolved
----------------------------------
+
Check that all the issues and merge requests for the appropriate
`milestone <https://gitlab.kwant-project.org/kwant/kwant/milestones>`
have been resolved. Any unresolved issues should have their milestone
@@ -16,6 +21,7 @@ bumped.
Ensure that all tests pass
--------------------------
+
This should be as simple as verifying that the latest CI pipeline succeeded.
For major and minor releases we will be tagging the ``master`` branch.
For patch releases, the ``stable`` branch.
@@ -23,6 +29,7 @@ For patch releases, the ``stable`` branch.
Inspect the documentation
-------------------------
+
If the CI pipeline succeeded, then the latest docs should be available at:
https://test.kwant-project.org/doc/<branch name>
@@ -32,6 +39,7 @@ Check that there are no glaring deficiencies.
Update the ``whatsnew`` file
----------------------------
+
Check that there is an appropriate ``whatsnew`` file in ``doc/source/pre/whatsnew``.
This should be named as::
@@ -45,28 +53,313 @@ assigned to the release's milestons and get an idea of what was introduced
from there.
+Make a release, but do not publish it yet
+#########################################
+
+Various problems can surface only during the process of preparing a release and
+make it necessary to fix the codebase. It would be a pity to have to succeed
+the freshly released version by a minor release just to correct a glitch that
+was detected too late. Therefore it is a good idea to pursue the release as
+far as possible without announcing it, such that it can be undone and corrected
+if necessary. In the past tests that failed on the x86-32 architecture and
+wrongly declared dependencies have been detected in this way.
+
+
Tag the release
-###############
-Make an *annotated*, *signed* tag for the release. The tag must have the semantic
-versioning format::
+---------------
- v<major>.<minor>.<patch>
+A particularly good way to expose hidden problems is building Debian packages
+using an isolated minimal build environment (cowbuilder). This approach is
+described here.
-Once the tag has been created, push it to Kwant Gitlab.
+Make an *annotated*, *signed* tag for the release. The tag must have the name::
+ git tag -s v1.2.3 -m "version 1.2.3"
-Post-tagging
-############
-Pushing a new tag to Kwant Gitlab will kick off a few CI jobs, but there is
-still some manual work to do.
+Be sure to respect the format of the tag name (leading "v"). The tag message
+format is the one that has been used so far.
+Do *not* yet push the tag anywhere, it might have to be undone!
-Prepare the Debian package
---------------------------
+
+Build a source taball and inspect it
+------------------------------------
+
+ ./setup.py sdist
+
+This creates the file dist/kwant-1.2.3.tar.gz. It is a good idea to unpack it
+in /tmp and inspect that builds in isolation and that the tests run::
+
+ cd /tmp
+ tar xzf ~/src/kwant/dist/kwant-1.2.3.tar.gz
+ cd kwant-1.2.3
+ ./setup.py test
+
+
+Clone the repository of the Kwant Debian package
+------------------------------------------------
+
+This step needs to be performed only once. The cloned repository can be reused
+for subsequent releases.
+
+Clone the "kwant-debian" repository and go into its root directory. If you
+keep the Kwant source in "src/kwant", a good location for the Debian package
+repository is "src/debian/kwant". The packaging process creates many files
+that are placed into the parent directory of the packaging repository, hence
+having an additional directory level ("src/debian") is a good way to keep these
+files separate::
+
+ mkdir debian
+ cd debian
+ git clone ssh://git@gitlab.kwant-project.org:443/kwant/debian-kwant.git kwant
+ cd kwant
+
+Create a local upstream branch::
+
+ git branch upstream origin/upstream
+
+Add a remote for the repository that contains the previously created tag::
+
+ git remote add upstream_repo ~/src/kwant
+
+Make sure that::
+
+ git config --get user.name
+ git config --get user.email
+
+show correct information.
+
+
+Release a new version of the Kwant Debian package
+-------------------------------------------------
+
+Fetch the git tag created above into the packaging repo::
+
+ git fetch --all
+
+Now there are two options. If, as recommended above, the tarball of the new
+version has not been made public yet, it must be imported as follows::
+
+ gbp import-orig ~/src/kwant/dist/kwant-1.2.3.tar.gz
+
+Alternatively, the following commands will import the newest version from PyPI::
+
+ uscan --report # This will report if a newer version exists on PyPI
+ gbp import-orig --uscan
+
+Update the debian changelog. Add a point "New upstream release" if there was
+one, and describe any other changes to the Debian *packaging*::
+
+ DEBEMAIL=your.email@somewhere.org gbp dch -R --commit --distribution testing
+
+Now the package can be built with::
+
+ git clean -i
+ gbp buildpackage
+
+But this is *not* how the package should be built for distribution. For that,
+see the following two sections.
+
+
+Setup git-pbuilder to build Debian packages
+-------------------------------------------
+
+Pbuilder is a tool to build Debian packages in an isolated chroot. This allows
+to verify that the package indeed only has the declared dependencies. It also
+allows to cross-build packages for i386 on amd64.
+
+The following describes how to setup git-pbuilder, see also
+https://wiki.debian.org/git-pbuilder. This procedure needs to be executed only
+once for a Debian system.
+
+Install the Debian package git-buildpackage.
+
+As root, add the following lines to /etc/sudoers or /etc/sudoers.d/local
+
+ Cmnd_Alias BUILD = /usr/sbin/cowbuilder
+
+and
+
+ user ALL = SETENV: BUILD
+
+Now create pbuilder images. In the following, replace "buster" by the current
+Debian testing codename::
+
+ ARCH=i386 DIST=buster git-pbuilder create
+ ARCH=amd64 DIST=buster git-pbuilder create
+
+If the packages to be built have special dependencies, use the trick described in https://wiki.debian.org/git-pbuilder#Using_Local_Packages
+
+
+Build Kwant packages using git-pbuilder
+---------------------------------------
+
+Update the builder environment (again, replace "buster" with the name of the
+current Debian testing)::
+
+ ARCH=i386 DIST=buster git-pbuilder update
+ ARCH=amd64 DIST=buster git-pbuilder update
+
+Now build the packages. First the i386 package. The option "--git-tag" tags
+and signs the tag if the build is successful. In a second step, the package is
+built for amd64, but only the architecture-dependent files (not the
+documentation package)::
+
+ gbp buildpackage --git-pbuilder --git-arch=i386 --git-dist=buster --git-tag
+ gbp buildpackage --git-pbuilder --git-arch=amd64 --git-dist=buster --git-pbuilder-options='--binary-arch'
+
+Another example: build source package only::
+
+ gbp buildpackage --git-export-dir=/tmp -S
+
+Build packports for the current Debian stable
+---------------------------------------------
+
+Create a changelog entry for the backport::
+
+ DEBEMAIL=your.email@somewhere.org dch --bpo
+
+Build backported packages::
+
+ gbp buildpackage --git-pbuilder --git-ignore-new --git-arch=i386 --git-dist=stretch
+ gbp buildpackage --git-pbuilder --git-ignore-new --git-arch=amd64 --git-dist=stretch --git-pbuilder-options='--binary-arch'
+
+Do not commit anything.
+
+Publish the release
+###################
+
+If the Debian packages build correctly that means that all tests pass both on
+i386 and amd64, and that no undeclared dependencies are needed. We can be
+reasonable sure that the release is ready to be published.
+
+git
+---
+
+Push the tag to the official Kwant repository::
+
+ git push origin v1.2.3
+
+PyPI
+----
+
+PyPI (this requires a file ~/.pypirc with a vaild username and password)::
+
+ twine upload -s dist/kwant-1.2.3.tar.gz
+
+It is very important that the tarball uploaded here is the same (bit-by-bit,
+not only the contents) as the one used for the Debian packaging. Otherwise it
+will not be possible to build the Debian package based on the tarball from
+PyPI.
+
+Kwant website
+-------------
+
+The tarball and its signature (generated by the twine command above) should be
+also made available on the website::
+
+ scp dist/kwant-1.2.3.tar.gz* kwant-website-downloads:downloads/kwant
+
+Debian packages
+---------------
+
+Go to the Debian packaging repository and push out everything::
+
+ git push --all --tags origin
+
+Now the Debian packages that we built previously need to be added to the
+repository of Debian packages on the Kwant website. So far this the full
+version of this repository is kept on Christoph Groth's machine, so this
+instructions are for reference only.
+
+Go to the reprepro repository directory and verify that the configuration file
+"conf/distributions" looks up-to-date. It should look something like this::
+
+ Origin: Kwant project
+ Suite: stretch-backports
+ Codename: stretch-backports
+ Version: 9.0
+ Architectures: i386 amd64 source
+ Components: main
+ Description: Unofficial Debian package repository of http://kwant-project.org/
+ SignWith: C3F147F5980F3535
+
+ Origin: Kwant project
+ Suite: testing
+ Codename: buster
+ Version: 10.0
+ Architectures: i386 amd64 source
+ Components: main
+ Description: Unofficial Debian package repository of http://kwant-project.org/
+ SignWith: C3F147F5980F3535
+
+If the config had to be updated execute::
+
+ reprepro export
+ reprepro --delete createsymlinks
+
+Now the source and binary Debian packages can be added. The last line has to
+be executed for all the .deb files and may be automated with a shell loop. (Be
+sure to use the appropriate <dist>: either testing or stretch-backports.)::
+
+ reprepro includedsc <dist> ../../src/kwant_1.2.3-1.dsc
+ reprepro includedeb <dist> python3-kwant_1.2.3-1_amd64.deb
+
+Once all the packages have been added, upload the repository::
+
+ rsync -avz --delete dists pool wfw:webapps/downloads/debian
+
+Ubuntu packages
+---------------
+
+Make sure ~/.dput.cf has something like this::
+ [ubuntu-ppa-kwant]
+ fqdn = ppa.launchpad.net
+ method = ftp
+ incoming = ~kwant-project/ppa/ubuntu/
+ login = anonymous
+ allow_unsigned_uploads = 0
+
+We will also use the following script (prepare_ppa_upload)::
+ #!/bin/sh
+
+ if [ $# -eq 0 ]; then
+ echo -e "\nUsage: $(basename $0) lousy mourning2 nasty\n"
+ exit
+ fi
+
+ version=`dpkg-parsechangelog --show-field Version`
+ mv debian/changelog /tmp/changelog.$$
+
+ for release in $@; do
+ cp /tmp/changelog.$$ debian/changelog
+ DEBEMAIL=christoph.groth@cea.fr dch -b -v "$version~$release" -u low 'Ubuntu PPA upload'
+ sed -i -e "1,1 s/UNRELEASED/$release/" debian/changelog
+ debuild -S -sa
+ done
+
+ mv /tmp/changelog.$$ debian/changelog
+
+Make sure that the Debian package builds correctly and go to its directory.
+
+Check https://wiki.ubuntu.com/Releases for the current releases and execute::
+ prepare_ppa_upload trusty vivid wily
+
+(if a second upload of the same Debian version is needed, something like vivid2 can be used)
+
+Now the changes files are "put" to start the build process on the PPA servers::
+
+ cd ..
+ dput ubuntu-ppa-kwant *~*.changes
Gather the autobuilt packages from CI
-------------------------------------
+
+(This section needs to be updated. Using the sdist package as generated by
+PyPI requires pushing the tag to gitlab and as such is incompatible with doing
+the Debian packaging with an unpublished tag.)
+
CI automatically generates:
+ HTML documentation
@@ -80,6 +373,10 @@ These can be found in the artifacts of the last CI job in the pipeline,
Publish to the Kwant website
----------------------------
+
+(This section needs to be updated. The twine tool creates the signature file
+during the upload.)
+
To do the following you will need access to the webroots of ``downloads.kwant-project.org``
and ``kwant-project.org``. Ask Christoph Groth if you need to be granted access.
@@ -104,6 +401,9 @@ on ``kwant-project.org``. Point the symbolic link ``<major>`` to this directory.
Publish to PyPI
---------------
+
+(This also needs to be updated.)
+
Install `twine <https://pypi.python.org/pypi/twine>` and use it to upload
the tar archive in the ``sdist`` directory of the Ci artifacts downloaded
in the previous step::
@@ -121,6 +421,7 @@ Publish to Launchpad
Publish to Conda forge
----------------------
+
Conda forge automates build/deploy by using CI on Github repositoried containing
recipes for making packages from their source distributions.
@@ -139,6 +440,7 @@ to review and accept the pull request, so that Kwant will be rebuilt.
Announce the release
--------------------
+
Write a short post summarizing the highlights of the release on the
`Kwant website <https://gitlab.kwant-project.org/kwant/website>`, then
post this to the mailing list kwant-discuss@kwant-project.org.
@@ -146,6 +448,7 @@ post this to the mailing list kwant-discuss@kwant-project.org.
Working towards the next release
################################
+
After finalizing a release, a new ``whatsnew`` file should be created for
the *next* release, and this addition should be committed and tagged as::
--
GitLab