How to package Rust applications to RPM using vendoring

Datetime:2016-08-23 00:02:08          Topic: Rust  Git           Share

The Rust programming language was born in the Internet age. There are many useful Rust libraries available (called crates) on the Internet collected to https://crates.io/ . When you use them, the Cargo build tool can automatically download any of these dependencies as necessary. It is easy for developers, but has some drawbacks. First of all, it requires an active Internet connection. And downloading crates regularly also requires bandwidth and time.

While most developer workstations have an active Internet connection, the other two drawbacks affect them as well. Also, there are some situations when there is no Internet access available during compilation. These are mostly automatic build systems, like the openSUSE Build Service or Fedora’s Copr , but also some corporate environments.

The solution to work around these problems is vendoring. In this case dependencies are downloaded only once, and after that the local copy of crates are used for building the Rust application. I will show through the example of a syslog-ng regular expression parser written in Rust, how vendoring works, and how it can be utilized in RPM packaging.

Vendoring

Because vendoring is quite a new concept, it exists only as a proof-of-concept subcommand for Cargo. It is available from https://github.com/alexcrichton/cargo-vendor Installation is straightforward:

     
$ git clone https://github.com/alexcrichton/cargo-vendor
$ cd cargo-vendor
$ cargo build –release

At the end you need to copy the resulting binary to somewhere in $PATH, and you are ready to use it. Vendoring adds a few extra steps to your build process, at least at the beginning. First you have to start Cargo with the “vendor” option:

     
[czanik@localhost regex-parser]$ cargo vendor --verbose
Updating registry `https://github.com/rust-lang/
Downloading syslog-ng-common v0.8.0
Downloading gobject-sys v0.3.1
[...]
Downloading regex-syntax v0.3.3
Downloading kernel32-sys v0.2.2
Create a `.cargo/config` with this entry to use the vendor cache:
 
[registry]
index = "file:///home/czanik/work/regex-parser/vendor/index"
 
[czanik@localhost regex-parser]$

The instructions are on the screen: create or edit .cargo/config and add the two lines that are displayed on the screen below this instruction. From now on Cargo will use the local copies instead of downloading the files from the Internet:

     
[czanik@localhost regex-parser]$ cargo build --release 
Updating registry `file:///home/czanik/work/regex-parser/vendor/index` 
Downloading regex v0.1.71 (registry file:///home/czanik/work/regex-parser/vendor/index) 
Downloading log v0.3.6 (registry file:///home/czanik/work/regex-parser/vendor/index) 
[...] 
Downloading syslog-ng-build v0.2.0 (registry file:///home/czanik/work/regex-parser/vendor/index) 
Downloading gcc v0.3.28 (registry file:///home/czanik/work/regex-parser/vendor/index) 
Compiling regex-parser v0.2.0 (file:///home/czanik/work/regex-parser) 
[czanik@localhost regex-parser]$

If dependencies have changed, you will need to create a new local database, by running “cargo vendor” again.

RPM packaging

Setting up the build environment is a bit tricky. It involves creating a few configuration files, and as the vendor database is stored into a git repository, you have to commit the changes there. Examples below are taken frommy Fedora/EPEL package, I expect only minimal differences for an openSUSE package.

First of all: sources. I could combine them into one file, but I find it more elegant and flexible to use two source archives. The first one is an unmodified source .zip downloaded from GitHub. The second one contains the “vendor” directory and a “.cargo” directory for the Cargo configuration file:

     
# Source: https://github.com/ihrwein/regex-parser/archive/master.zip 
Source0: master.zip 
 
# Vendoring of all regex-parser dependencies from crates.io using 
# cargo-vendor from https://github.com/alexcrichton/cargo-vendor and 
# storing results together with a cargo config 
Source1: regex-parser-deps.tgz

The next lines are the BuildRequires lines required by your Rust application. There is an extra line for Git:

     
BuildRequires:  git

This is used to commit changes to the vendor Git repository.

Once all of the dependencies are installed, the build environment needs to be prepared:

     
%prep 
%setup -n regex-parser-master 
%setup -D -T -a 1 -n regex-parser-master 
 
# build directory is needed in config files 
export BDIR=`pwd` 
 
# setup cargo registry 
echo [registry] > .cargo/config 
echo index = \"file://$BDIR/vendor/index\" >> .cargo/config 
 
# setup vendor repo and commit it 
cd $BDIR/vendor/index 
echo \{\"dl\":\"file://$BDIR/vendor/cache\",\"api\":\"\"\} > config.json 
git config --global user.email "you@example.com" 
git config --global user.name "Your Name" 
git commit -a -m "this is a message"

As a first step both source archives are extracted into the same directory. Once source files are set up, the working directory is changed to the build directory. As we use this value a couple of times while creating configuration files, I put it into an environment variable. It is used in the next few lines to create a configuration for Cargo with the correct path name.

The vendor repository is not used directly, but checked out during the build process. Therefore it is not enough to edit the configuration, but you also have to commit it. Git requires some configuration, before you can do that. Because build chroots are deleted after use, dummy values are sufficient.

For the build stage you have to set an environment variable that instructs Cargo to use the configuration you have just created:

     
%build 
export CARGO_HOME=`pwd`/.cargo/ 
cargo build --release –verbose

The rest of the spec file is pretty standard and boring: copying files to the right places and adding them to the files section. If you want to take a look at the complete spec file and sources, it is available at https://copr.fedorainfracloud.org/coprs/czanik/syslog-ng38/package/syslog-ng-regex-parser/

Feedback

The Rust compiler is not yet part of Linux distributions, the packaging of Rust application is not yet formalized in policies. By the time syslog-ng 3.8 and its Rust-based parsers are released, both Fedora and openSUSE Tumbleweed are supposed to include Rust support. If you package for Fedora or openSUSE (or RHEL / SLES) I am very interested about your opinion about this work-flow. Could I include Rust-based syslog-ng modules in distro syslog-ng packaged this way?





About List