Planet Debian

Subscribe to Planet Debian feed
Planet Debian - http://planet.debian.org/
Updated: 52 min 49 sec ago

Ben Hutchings: Debian LTS work, November 2017

2 December, 2017 - 00:54

I was assigned 13 hours of work by Freexian's Debian LTS initiative and carried over 14 hours from September. I worked all 17 hours.

I prepared and released two updates on the Linux 3.2 longterm stable branch (3.2.95, 3.2.96), but I didn't upload an update to Debian. However, I have rebased the Debian package on 3.2.96 and expect to make a new upload soon.

Ben Hutchings: Mini-DebConf Cambridge 2017

2 December, 2017 - 00:51

Last week I attended Cambridge's annual mini-DebConf. It's slightly strange to visit a place one has lived in for a long time but which is no longer home. I joined Nattie in the 'video team house' which was rented for the whole week; I only went for four days.

I travelled down on Wednesday night, and spent a long time (rather longer than planned) on trains and in waiting rooms. I used this time to catch up on discussions about signing infrastructure for Secure Boot, explaining my concerns with the most recent proposal and proposing some changes that might alleviate those. Sorry to everyone who was waiting for that; I should have replied earlier.

On the Thursday and Friday I prepared for my talk, and had some conversations with Steve McIntyre and others about SB signing infrastructure. Nattie and Andy respectively organised group dinners at the Polish club on Thursday and a curry house on Friday, both of which I enjoyed.

The mini-DebConf proper took place on the Saturday and Sunday, and I presented my now annual talk on "What's new in the Linux kernel". As usual, the video team did a fine job of recording and publishing video of the talks.

Junichi Uekawa: I was trying to get selenium up and running.

2 December, 2017 - 00:20
I was trying to get selenium up and running. I wanted to try chrome headless and one that seemed to be usable seemed to be selenium but that didn't just work out of the box on Debian apt-get installed binary. hmm.

James McCoy: Monthly FLOSS activity - 2017/11 edition

1 December, 2017 - 12:02
Debian vim
  • Uploaded version 2:8.0.1257-1 to fix a flaky test and the debsources syntax highlighting
    • This round of builds revealed some more flaky tests, so I sent PR#2282 to make the tests more deterministic.
  • Uploaded version 2:8.0.1257-2
neovim
  • Uploaded versions 0.2.1-1 and 0.2.1-1
    • There were various test failures, which thankfully boiled down to a few common issues
      • Lua appears to be stricter than LuaJIT when formatting nil
      • A few potentially flaky screen tests
      • Garbage being formatted into an error string
  • Uploaded version 0.2.1-3 to fix the test failures
  • Uploaded version 0.2.2-1
libvterm
  • Uploaded version 0~bzr715-1
    • This is needed by the new pangoterm revision to report focus events
pangoterm
  • Uploaded version 0~bzr607-1
    • Focus reporting can now be enabled by applications
    • The -T switch, an alias for --title, is supported for full compliance with the requirements of an x-terminal-emulator alternative
serf
  • Uploaded version 1.3.9-4
    • Cleaned up the packaging
    • Marked a symbol as public since svn has been using it for 10 years, which let me drop a patch from svn's packaging
Subversion
  • Changed how swig is invoked so there are explicit mechanisms for passing general or language-specific options when generating the bindings. No longer are CPPFLAGS being (ab)used to handle this, so the whack-a-mole game of stripping unrecognized switches was also removed. (r1816254)
  • Since there was a bit of development activity at the hackathon and talk of another pre-release for 1.10, I performed a Coverity scan of trunk. I did a quick scrub of some of the new and existing issues, one of which I proposed for a backport to 1.9.x.
neovim

Paul Wise: FLOSS Activities November 2017

1 December, 2017 - 09:44
Changes Issues Review Administration
  • hexchat-addons: merged pull request
  • Debian: fix LDAP issue, reenable webserver on a VM, redirect users to support channels
  • Debian wiki: whitelist email addresses
  • Debian website: check on a build issue
Communication
  • Invite Slax to the Debian derivatives census
Sponsors

All work was done on a volunteer basis.

Antoine Beaupré: November 2017 report: LTS, standard disclosure, Monkeysphere in Python, flash fraud and Goodbye Drupal

1 December, 2017 - 05:33
Debian Long Term Support (LTS)

This is my monthly Debian LTS report. I didn't do as much as I wanted this month so a bunch of hours are reported to next month. I got frustrated by two difficult packages: exiv2 and libreoffice.

Exiv

For Exiv2 I first reported the issues upstream as requested in the original CVE assignment. Then I went to see if I could reproduce the issue. Valgrind didn't find anything, so I went on to test the new ASAN instructions that tell us how to build for ASAN in LTS. Turns out that I couldn't make that work either. Fortunately, Roberto was able to build the package properly and confirmed the wheezy version as non-vulnerable, so I marked the three CVEs as not-affected and moved on.

Libreoffice

Next up was LibreOffice. I started backporting the patches to wheezy which was a little difficult because any error in the backport takes hours to find because LibreOffice is so big. The monster takes about 4 hours to build on my i3-6100U processor - I can't imagine how long that would take on a slower machine. Still, I managed to get patches that mostly builds. I say mostly because while most of the code builds, the tests fail to build. Not only do they fail to build, but they even segfault the linker. At that point, I had already spent too many hours working on this frustrating loop of "work/build-wait/crash" that I gave up.

I also worked on reproducing a supposed regression associated with the last security update. Somehow, I couldn't reproduce either - the description of the regression was very limited and all suggested approaches failed to trigger the problems described.

OptiPNG

Finally, a little candy: an easy backport of a simple 2-line patch for a simple program, OptiPNG that, ironically, had a vulnerability (CVE-2017-16938) in GIF parsing. I could do hundreds of those breezy updates, they are fun and simple, and easy to test. This resulted in the trivial DLA-1196-1.

Miscellaneous

LibreOffice stretched the limits of my development environment. I had to figure out how to deal with out of space conditions in the build tree (/build) something that is really not obvious in sbuild. I ended up documenting that in a new troubleshooting section in the wiki.

Other free software work feed2exec

I pushed forward with the development of my programmable feed reader, feed2exec. Last month I mentioned I released the 0.6.0 beta: since then 4 more releases were published, and we are now at the 0.10.0 beta. This added a bunch new features:

  • wayback plugin to save feed items to the Wayback Machine on archive.org
  • archive plugin to save feed items to the local filesystem
  • transmission plugin to save RSS Torrent feeds to the Transmission torrent client
  • vast expansion of the documentation, now hosted on ReadTheDocs. The design was detailed with a tour of the source code and detailed plugin writing instructions were added to the documentation, also shipped as a feed2exec-plugins manpage.
  • major cleanup and refactoring of the codebase, including standard locations for the configuration files, which moved

The documentation deserves special mention. If you compare between version 0.6 and the latest version you can see 4 new sections:

  • Plugins - extensive documentation on plugins use, the design of the plugin system and a full tutorial on how to write new plugins. the tutorial was written while writing the archive plugin, which was written as an example plugin just for that purpose and should be readable by novice programmers
  • Support - a handy guide on how to get technical support for the project, copied over from the Monkeysign project.
  • Code of conduct - was originally part of the contribution guide. the idea then was to force people to read the Code when they wanted to contribute, but it wasn't a good idea. The contribution page was overloaded and critical parts were hidden down in the page, after what is essentially boilerplate text. Inversely, the Code was itself hidden in the contribution guide. Now it is clearly visible from the top and trolls will see this is an ethical community.
  • Contact - another idea from the Monkeysign project. became essential when the security contact was added (see below).

All those changes were backported in the ecdysis template documentation and I hope to backport them back into my other projects eventually. As part of my documentation work, I also drifted into the Sphinx project itself and submitted a patch to make manpage references clickable as well.

I now use feed2exec to archive new posts on my website to the internet archive, which means I have an ad-hoc offsite backup of all content I post online. I think that's pretty neat. I also leverage the Linkchecker program to look for dead links in new articles published on the site. This is possible thanks to a Ikiwiki-specific filter to extract links to changed files from the Recent Changes RSS feed.

I'm considering making the parse step of the program pluggable. This is an idea I had in mind for a while, but it didn't make sense until recently. I described the project and someone said "oh that's like IFTTT", a tool I wasn't really aware of, which connects various "feeds" (Twitter, Facebook, RSS) to each other, using triggers. The key concept here is that feed2exec could be made to read from Twitter or other feeds, like IFTTT and not just write to them. This could allow users to bridge social networks by writing only to a single one and broadcasting to the other ones.

Unfortunately, this means adding a lot of interface code and I do not have a strong use case for this just yet. Furthermore, it may mean switching from a "cron" design to a more interactive, interrupt-driven design that would run continuously and wake up on certain triggers.

Maybe that could come in a 2.0 release. For now, I'll see to it that the current codebase is solid. I'll release a 0.11 release candidate shortly, which has seen a major refactoring since 0.10. I again welcome beta testers and users to report their issues. I am happy to report I got and fixed my first bug report on this project this month.

Towards standard security disclosure guidelines

When reading the excellent State of Opensource Security report, some metrics caught my eye:

  • 75% of vulnerabilities are not discovered by the maintainer

  • 79.5% of maintainers said that they had no public-facing disclosure policy in place

  • 21% of maintainers who do not have a public disclosure policy have been notified privately about a vulnerability

  • 73% of maintainers who do have a public disclosure policy have been notified privately about a vulnerability

In other words, having a public disclosure policy more than triples your chances of being notified of a security vulnerability. I was also surprised to find that 4 out of 5 projects do not have such a page. Then I realized that none of my projects had such a page, so I decided to fix that and fix my documentation templates (the infamously named ecdysis project) to specifically include a section on security issues.

I found that the HackerOne disclosure guidelines were pretty good, except they require having a bounty for disclosure. I understand it's part of their business model, but I have no such money to give out - in fact, I don't even pay myself for the work of developing the project, so I don't quite see why I would pay for disclosures.

I also found that many projects include OpenPGP key fingerprints in their contact information. I find that's a little silly: project documentation is no place to offer OpenPGP key discovery. If security researchers cannot find and verify OpenPGP key fingerprints, I would be worried about their capabilities. Adding a fingerprint or key material is just bound to create outdated documentation when maintainers rotate. Instead, I encourage people to use proper key discovery mechanism like the Web of trust, WKD or obviously TOFU which is basically what publishing a fingerprint does anyways.

Git-Mediawiki

After being granted access to the Git-Mediawiki project last month, I got to work. I fought hard with both Travis and Git, and Perl, and MediaWiki, to add continuous integration in the repository. It turns out the MediaWiki remote doesn't support newer versions of MediaWiki because the authentication system changed radically. Even though there is supposed to be backwards-compatibility, it doesn't seem to really work in my cases, which means that any test case that requires logging into the wiki fails. This also required using an external test suite instead of the one provided by Git, which insists on building Git before being used at all. I ended up using the Sharness project and submitted a few tests that were missing.

I also:

I also opened a discussion regarding the versioning scheme and started tracking issues that would be part of the next release. I encourage people to get involved in this project if they are interested: the door is wide open for contributors, of course.

Rewriting Monkeysphere in Python?

After being told one too many times that Monkeysphere doesn't support elliptic curve algorithms, I finally documented the issue and looked into solutions. It turns out that a key part of Monkeysphere is a Perl program written using a library that doesn't support ECC itself, which makes this problem very hard to solve. So I looked into the PGPy project to see if it could be useful and it turns out that ECC support already exists - the only problem is the specific curve GnuPG uses, ed25519, was missing. The author fixed that but the fix requires a development version of OpenSSL, because even the 1.1 release doesn't support that curve.

I nevertheless went ahead to see how hard it would be to re-implement that key component of Monkeysphere in Python, and ended up with monkeypy, an analysis of the Monkeysphere design and first prototype of a Python-based OpenPGP / SSH conversion tool. This lead to a flurry of issues on the PGPy project to fix UTF-8 support, easier revocation checks and a few more. I also reviewed 9 pull requests that were pending in the repository. A key piece missing is keyserver interaction and I made a first prototype of this as well.

It was interesting to review Monkeysphere's design. I hope to write more about this in the future, especially about how it could be improved, and find the time to do this re-implementation which could mean wider adoption of this excellent project.

Goodbye, Drupal

I finally abandoned all my Drupal projects, except the Aegir-related ones, which I somewhat still feel attached to, even though I do not do any Drupal development at all anymore. I do not use Drupal anywhere (unless as a visitor on some websites maybe?), I do not manage Drupal or Aegir sites anymore, and in fact, I have completely lost interested in writing anything remotely close to PHP.

So it was time to go. I have been involved with Drupal for a long time: I registered on Drupal.org in March 2001, almost 17 years ago. I was active for about 12 years on the site: I made my first post in 2003 which showed I was already "hacking core" which back then was the way to go. My first commit, to the mailman_api project, was also in 2003, and I kept contributing until my last commit in 2015, on the Aegir project of course.

That is probably the longest time I have spent on any software project, with the notable exception of the Debian project. I had a lot of fun working with Drupal: I met amazing people, traveled all over the world and made powerful websites that shook the world. Unfortunately, the Drupal project has decided to go in a direction I cannot support: Drupal 8 is a monster that is beyond anything needed for me, my friends or the organizations I support. It may be very well for the startups and corporations, to create large sites, but it seems it completely lost touch with its roots. Drupal used to be a simple tool for "community plumbing" (ref), it is now a behemoth to "Launch, manage, and scale ambitious digital experiences—with the flexibility to build great websites or push beyond the browser" (ref).

When I heard a friend had his small artist website made in Drupal, my first questions were "which version?" and "who takes care of the upgrades?" Those questions still don't seem to be resolved in the community: a whole part of the project is still stuck in the old Drupal 7 world, years after the first D8 alpha releases. Drupal 8 doesn't address the fundamental issue of upgradability and cost: Drupal sites are bigger than they ever were. The Drupal community seems to be following a very different way of thinking (notice the Amazon Echo device there) than what I hope the future of the Internet will be.

Goodbye, Drupal - it's been a great decade, but I'm already gone and I'm not missing you at all.

Fixing fraudulent memory cards

I finally got around testing the interesting f3 project with a real use case, which allowed me to update the stressant documentation with actual real-world results. A friend told me the memory card he put in his phone had trouble with some pictures he was taking with the phone: they would show up as "gray" when he would open them, yet the thumbnail looked good.

It turns out his memory card wasn't really 32GB as advertised, but really 16GB. Any write sent to the card above that 16GB barrier was just discarded and reads were returned as garbage. But thanks to the f3 project it was possible to fix that card so he could use it normally.

In passing, I worked on the f3 project to merge the website with the main documentation, again using the Sphinx project to generate more complete documentation. The old website is still up, but hopefully a redirection will be installed soon.

Miscellaneous
  • quick linkchecker patch, still haven't found the time to make a real release

  • tested the goaccess program to inspect web server logs and get some visibility on the traffic to this blog. found two limitations: some graphs look weird and would benefit from a logarithmic scale and it would be very useful to have plain text reports. otherwise this makes for a nice replacement for the aging analog program I am currently using, especially as a sysadmin quick diagnostic tool. for now I use this recipe to send a HTML report by email every week

  • found out about the mpd/subsonic bridge and sent a few issues about problems I found. turns out the project is abandoned and this prompted the author to archive the repository. too bad...

  • uploaded a new version of etckeeper into Debian, and started an upstream discussion about what to do with the pile of patches waiting there

  • uploaded a new version of the also aging smokeping in Debian as well to fix a silly RC bug about the rename dependency

  • GitHub also says I "created 7 repositories", "54 commits in 8 repositories", "24 pull requests in 11 repositories", "reviewed 19 pull requests in 9 repositories", and "opened 23 issues in 14 repositories".

  • blocked a bunch of dumb SMTP bots attacking my server using this technique. this gets rid of those messages I was getting many times per second in the worst times:

    lost connection after AUTH from unknown[180.121.131.155]
    
  • happily noticed that fail2ban gained the possibility of incremental ban times in the upcoming version 0.11, which i documented on Serverfault.com. unfortunately, the Debian package is out of date for now...

  • tested Firefox Quantum with great success: all my extensions are supported and it is much, much faster than it was. i do miss the tab list drop-down which somehow disappeared and I also miss a Debian package. I am also concerned about the maintainability of Rust in Debian, but we'll see how that goes.

I spent only 30% of this month on paid work.

Chris Lamb: Free software activities in November 2017

1 December, 2017 - 00:51

Here is my monthly update covering what I have been doing in the free software world in November 2017 (previous month):

Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to allow verification that no flaws have been introduced — either maliciously or accidentally — during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

I have generously been awarded a grant from the Core Infrastructure Initiative to fund my work in this area.

This month I:

  • Presented at the Open Compliance Summit 2017 in Yokohama, Japan and had many follow-up conversations regarding using reproducible builds as a way of ensuring the long-term sustainability of civil infrastructure.
  • Created pull requests upstream for fswatch, bitz-server, stetl, nbsphinx & stardicter.
  • Updated diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, to only parse DTB's version number, not any -dirty suffix. (#880279)
  • Expanded the documentation for disorderfs, our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues, highlighting the non-intuitive recommendation to sort instead of shuffle. [...]
  • Made some brief changes to buildinfo.debian.net, my experiment into how to process, store and distribute .buildinfo files after the Debian archive software has processed them:
    • Add a by-hash API endpoint. [...]
    • Support ?key__uid=X&key__uid=Y filtering. [...]
  • Updated our website:
    • Move the "contribute" page from the Debian wiki to /contribute/. [...]
    • Add a (redirecting) /docs/source-date-epoch/ page so we have a canonical URL. [...]
    • Add recent talks to Resources page. [...]
    • Cachebust CSS files. [...]
  • In Debian:
  • Categorised a large number of packages and issues in the Reproducible Builds "notes" repository.
  • Made some changes to: jenkins.debian.net which uns our comprehensive testing framework:
    • Ignore "warning" strings in commit messages causing builds to be marked as unstable. [...]
    • Update the email subject of status change mails away from Debian-specific URI. [...]
    • Move some IRC announcements to #debian-reproducible-changes. [...]
  • Worked on publishing our weekly reports. (#132, #133, #134 & #135)


Debian

My activities as the current Debian Project Leader are covered in my "Bits from the DPL" email to the debian-devel-announce mailing list.

Patches contributed
  • dget: Please support downloading packages over gopher://. (#880649)
  • gpaw: Incorrectly creates logging files called - instead of logging to standard output. (#882638)
  • pk4: Please avoid the use of avail in package descriptions. (#881343)
Debian LTS

This month I have been paid to work 13 hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 1161-1 for the redis key-value storage database to fix cross-protocol scripting attack.
  • Issued DLA 1162-1 & DLA 1163-1 to fix out-of-bounds memory vulnerabilites in apr and apr-util, portability libraries for various Apache applications.
  • Issued DLA 1173-1 for procmail, a tool used to sort incoming mail into various directories and filter out spam messages to fix a heap-based buffer overflow.
  • Issued DLA 1174-1 to correct a denial of service vulnerability in the konversation IRC client related to parsing of color formatting codes.
  • Issued DLA 1175-1 for the lynx-cur web browser, preventing a use-after-free vulnerability in the HTML parser which could lead to memory/information disclosure.
Uploads
  • python-django:
  • redis:
    • 4.0.2-6 — Correct locations of redis-sentinel pidfiles. (#880980)
    • 4.0.2-7 — Add a redis metapackage. (#876475)
    • 4.0.2-8 — Use get_current_dir_name over a PATHMAX, etc. (#881684), don't rely on taskset existing for kFreeBSD-* (#881683), drop "memory efficiency" tests on advice from upstream (#881682) and allow the package be bin-NMUable.
    • 4.0.2-9 — Modify aof.c for MAXPATHLEN issues. (#881684)
    • 4.0.2-9~bpo9+1 — Upload to stretch-backports.
  • bfs:
    • 1.1.4-1 — New upstream release.
    • 1.1.4-2 — Use upstream's new manpage.
  • python-daiquiri:
    • 1.3.0-2 — Ensure all dependencies are available for DEP-8 tests. (#882876)
  • redisearch (0.90.0~alpha1-1, 0.90.1-1, 0.99.0-1 & 0.99.2-1) — New upstream releases.

Finally, I also made a non-maintainer upload (NMU) of cpio (2.12+dfsg-5) to the experimental distribution.

Debian bugs filed
  • cappuccino: Broken symlink in /usr/games. (#880714)
  • statsmodels: Accesses raw.github.com during build. (#882641)
  • python-lti: Please run the upstream testsuite. (#880834)
  • git-buildpackage: gbp dch needs a better workflow description. (#880552)
  • audacity: New upstream release. (#880717)
  • python-djangorestframework: New upstream release. (#880538)
  • djangorestframework: New upstream release. (#880558)

I also filed 2 FTBFS bugs against django-axes & plinth.

FTP Team

As a Debian FTP assistant I ACCEPTed 58 packages: aladin, apulse, aribb24, ayatana-indicator-printers, beads, belr, binutils, breezy-debian, brightnessctl, cupt, dino-im, evqueue-core, fdm-materials, fonts-noto-color-emoji, gcc-8-cross, gcc-8-cross-ports, gnome-shell-extension-hide-veth, gnome-shell-extension-no-annoyance, gnome-shell-extension-tilix-shortcut, gnome-shell-extension-workspaces-to-dock, goocanvasmm-2.0, intel-vaapi-driver-shaders, ldc, libaws-signature4-perl, libcdio-paranoia, libemail-address-xs-perl, libjs-jquery-file-upload, libmath-utils-perl, libosmo-abis, libosmocore, libsavitar, libsignal-protocol-c, lr, mate-window-applets, node-ms, openjdk-10, phast, pspg, python-daphne, r-cran-cardata, r-cran-cvst, r-cran-forcats, r-cran-gower, r-cran-guerry, r-cran-haven, r-cran-lava, r-cran-nortest, r-cran-rcpproll, r-cran-readr, r-cran-spatstat.data, r-cran-tidyselect, ros-geometry2, shoogle, snapd-glib, sphinx-intl, tang, ulfius & webapps-metainfo.

I additionally filed 2 RC bugs against packages that had incomplete debian/copyright files against libsavitar & fdm-materials.

Jonathan Dowland: distribution-wide projects in Debian

30 November, 2017 - 23:36

A few weeks ago, Lars and I had a discussion: wouldn't it be nice if, across the whole of Debian, applications stopped writing to arbitrary dot-files in your home directory (e.g. ~/.someapp) and instead abided by the XDG Base Directory Specification?

Assuming that there was Debian-wide consensus that this was a good idea, in theory it could be achieved. The main problem would be that many of the upstream authors of the software we package would not accept the change. Consequently, Debian would be left carrying the patches.

We generally try to remain as close to upstream's code as possible and shy away from carrying too many patches in too many packages. The ideal lifecycle for a patch is for it to be accepted upstream. Patches are a burden for packagers, and we don't have enough packagers or packager time (or both).

Contrast this to OpenBSD. I know very little about the BSDs, and what little I do know I've mostly gleaned from the excellent blog posts by Ted Unangst, but my understanding is that when they import some software into their main project, their mental model appears to be closer to that of a fork of the upstream software than the way Debian (and most Linux distributions) operate. The entire OpenBSD Operating System is in a single (CVS!) version control repository. When some software is imported to the core Operating System, that software's source is copied into place within the tree and from that point forward is managed as part of the whole.

(OpenBSD also have a separate source code collection called "ports", which is managed in a different manner which is much closer to a Linux distribution's notion of packaging, but I won't cover that further here.)

If they decided that a distribution-wide project was worthy, they would have no hesitation in making that change across all the software in their Operating System. Their focus is on consistency and they seem to maintain that focus rather than thinking about packages in relative isolation.

Although I do appreciate the practical problem of Debian managing a lot of patches, I am sometimes envious of OpenBSD's approach and corresponding perspective. Although I really do not miss CVS.

Thanks to Ryan Freeman for proof-reading this blog post. The remaining errors are mine.

Enrico Zini: Qt Creator cross-platform development in Stretch: chroot automation

30 November, 2017 - 15:35

I wrote a tool to automate the creation and maintenance of Qt cross-build environments built from packages in Debian Stretch.

It allows to:

  • Do cross-architecture development with Qt Creator, including remote debugging on the target architecture
  • Compile using native compilers and cross-compilers, to avoid running the compilers inside an emulator making the build slower
  • Leverage all of Debian as development environment, using existing Debian packages for cross-build-dependencies
Getting started
# Creates an armhf environment under the current directory
$ sudo cbqt ./armhf --create --verbose
2017-11-30 14:09:23,715 INFO armhf: Creating /home/enrico/lavori/truelite/system/cross/armhf
…
2017-11-30 14:14:49,887 INFO armhf: Configuring cross-build environment

# Get information about an existing chroot.
# Note: the output is machine-parsable yaml
$ cbqt ./armhf --info
name: armhf
path: ./armhf
arch: armhf
arch_triplet: arm-linux-gnueabihf
exists: true
configured: true
issues: []

# Create a qmake wrapper for this environment
$ sudo ./cbqt ./armhf --qmake -o /usr/local/bin/qmake-armhf

# Install the build-depends you need
# Note: :arch is added automatically to package names if no arch is explicitly
#       specified
$ sudo ./cbqt ./armhf --install libqt5svg5-dev libmosquittopp-dev qtwebengine5-dev
Building packages

To build a package, use the qmake wrapper generated with cbqt --qmake instead of the normal qmake:

$ qmake-armhf -makefile
$ make
arm-linux-gnueabihf-g++ … -I../armhf/usr/include/arm-linux-gnueabihf/… -I../armhf/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/arm-linux-gnueabihf -o browser.o browser.cpp
…
/home/enrico/…/armhf/usr/bin/moc …
…
arm-linux-gnueabihf-g++ … -Wl,-rpath-link,/home/enrico/…/armhf/lib/arm-linux-gnueabihf -Wl,-rpath-link,/home/enrico/…/armhf/usr/lib/arm-linux-gnueabihf -Wl,-rpath-link,/home/enrico/…/armhf/usr/lib/ -o program browser.o … -L/home/enrico/…/armhf/usr/lib/arm-linux-gnueabihf …
Building in Qt Creator

Configure a new Kit in Qt Creator:

  1. Tools/Options, then Build & Run, then Kits, then Add
  2. Name: armhf (or anything you like)
  3. Qt version: choose the one autodetected from /usr/local/bin/qmake-armhf
  4. In Compilers, add a GCC C compiler with path arm-linux-gnueabihf-gcc, and a GCC C++ compiler with path arm-linux-gnueabihf-g++
  5. Choose the newly created compilers in the kit
  6. Dismiss the dialog with "OK": the new kit is ready

Now you can choose the default kit to build and run locally, and the armhf kit for remote cross-development.

Where to get it

Here!

Credits

This has been done as part of my work with Truelite.

Reproducible builds folks: Reproducible Builds: Weekly report #135

30 November, 2017 - 03:45

Here's what happened in the Reproducible Builds effort between Sunday November 19 and Saturday November 25 2017:

Upcoming events

Reproducible Builds will have an assembly at 34c3, the "Galactic Congress". Currently we are discussing to informally meet there every day at 13:37 UTC.

Reproducible Arch Linux

Since November 23 2017, Arch Linux is again being continuously tested for reproducibility. However, this time a patched pacman is being used which can create reproducible packages. After 4 days of testing, 18% of all packages in the core, extra, multilib and community Arch repos has been tested, with these — very preliminary — results:

  • core: 77.1% reproducible, all 197 packages tested.
  • extra: 75.2% reproducible, 514 packages (of 2250 total) tested.
  • multilib: 82.6% reproducible, all 259 packages tested.
  • community: 76.5% reproducible, 487 packages (of 7739 total) tested.

Jelle van der Waa also wrote a blog post explaining more details detailing how this already had lead to more QA work in Arch.

So all in all, it looks like 77.2% of the tested Arch Linux packages are now reproducible! With an unreleased pacman version and without some variations we apply when testing Debian… still this is a very good start! Kudos to all involved.

Packages reviewed and fixed, and bugs filed

Patches filed upstream:

  • Bernhard M. Wiedemann:
  • Chris Lamb:
    • gpaw - (merged) embedded logging output
    • bitz-server (merged) - build path

Patches filed in Debian:

Patches filed in OpenSUSE:

Reviews of unreproducible packages

97 package reviews have been added, 56 have been updated and 42 have been removed in this week, adding to our knowledge about identified issues.

2 issue types have been added:

Weekly QA work

During our reproducibility testing, FTBFS bugs have been detected and reported by:

  • Adrian Bunk (62)
  • Gilles Filippini (1)
  • Gregor Riepl (1)
  • James Cowgill (1)
  • Laurent Bigonville (1)
  • Matthias Klose (1)
  • Sylvestre Ledru (2)
  • gregor herrmann (1)
reproducible-faketools
  • reproducible-faketools 0.3.10 was released with support for:
    • Reduced randomness (/dev/random and urandom are actually /dev/zero)
    • Disabled ASLR and
    • Building with fixed PIDs.
    • Also the tar wrapper script got a bug fix.
reprotest development reproducible-website development tests.reproducible-builds.org
  • anthraxx worked on reproducible Arch Linux (19 commits)
  • Holger Levsen did some work on reproducible Debian:
    • aa9ce22d6 - Update email subject of status change mails to use t.r-b.o/debian - thanks to lamby for #882186
  • Holger mostly worked on reproducible Arch Linux that week (56 commits).
  • Misc tests.r-b.o work by Holger:
    • 0d79ab54a - reproducible Fedora: be explicit that this is stalled atm
    • Holger also reviewed and deployed 25 commits from other people.
    • Finally, Holger moved IRC notifications for jenkins.debian.net from #debian-reproducible to #reproducible-builds (and kept them on debian-qa as well).
  • Johannes Löthberg worked on Arch Linux as well (2 commits)
  • kpcyrd also worked on Arch Linux (5 commits)

Finally there was discussion to how to generalise the database schema for supporting several projects, triggered by the recent work on reproducible Arch, but also previously discussed in the context of openSUSE, LEDE and FreeBSD.

Misc.

This week's edition was written by Ximin Luo, Bernhard M. Wiedemann, Holger Levsen and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Enrico Zini: Qt Creator cross-platform development in Stretch: chroots

29 November, 2017 - 21:53

My next step in creating a cross-platform Qt development environment is trying to set it up on a chroot and make it usable from Qt Creator, so that both buil-dependencies and cross-build-dependencies can be available even when they are not coinstallable.

Building a chroot

I built a chroot to host the armhf build-dependencies:

$ sudo cdebootstrap stretch arm-linux-gnueabihf
$ sudo systemd-nspawn -D arm-linux-gnueabihf
# dpkg --add-architecture armhf
# apt update
# apt install qtbase5-dev qtbase5-dev-tools qtbase5-dev:armhf

The cross-compilers need to be outside the chroot. I tried to use cross-compilers installed inside the chroot while building outside the chroot, and they fail to link at runtime, since they expect their shared libraries to be in /usr.

I added the custom mkspecs to the chroot:

$ cd arm-linux-gnueabihf/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/
$ sudo cp -r linux-g++ arm-linux-gnueabihf
$ sudo vi arm-linux-gnueabihf/qmake.conf

This is the content of usr/lib/arm-linux-gnueabihf/qt5/mkspecs/arm-linux-gnueabihf/qmake.conf:

#
# qmake configuration for arm-linux-gnueabihf
#

MAKEFILE_GENERATOR      = UNIX
CONFIG                 += incremental
QMAKE_INCREMENTAL_STYLE = sublib

include(../common/linux.conf)
include(../common/gcc-base-unix.conf)
include(../common/g++-unix.conf)

QMAKE_RPATHLINKDIR += /home/enrico/…/arm-linux-gnueabihf/lib/arm-linux-gnueabihf
QMAKE_RPATHLINKDIR += /home/enrico/…/arm-linux-gnueabihf/usr/lib/arm-linux-gnueabihf
QMAKE_RPATHLINKDIR += /home/enrico/…/arm-linux-gnueabihf/usr/lib/

QMAKE_COMPILER          = arm-linux-gnueabihf-gcc

QMAKE_CC                = arm-linux-gnueabihf-gcc

QMAKE_LINK_C            = $$QMAKE_CC
QMAKE_LINK_C_SHLIB      = $$QMAKE_CC

QMAKE_CXX               = arm-linux-gnueabihf-g++

QMAKE_LINK              = $$QMAKE_CXX
QMAKE_LINK_SHLIB        = $$QMAKE_CXX

load(qt_config)

Note QMAKE_RPATHLINKDIR, which was not present in the previous post: since linking needs to happen against libraries in /home/enrico/…/arm-linux-gnueabihf/…, we need to tell the linker not to go and search in /.

The extra QMAKE_RPATHLINKDIR pointing to usr/lib is a workaround for libsrtp0 installing files outside multiarch directories (#765173):

# dpkg -L libsrtp0 | grep usr/lib
/usr/lib
/usr/lib/libsrtp.so.0.0
/usr/lib/libsrtp.so.0

(libsrtp0 is a dependency of libqt5webenginecore5)

I changed the wrapper /usr/local/bin/qmake-arm-linux-gnueabihf to point to the chroot:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
#!/usr/bin/python3

import sys, os

QT_CONFIG = "/home/enrico/…/arm-linux-gnueabihf/qt.conf"

argv0 = os.path.join(os.path.dirname(sys.argv[0]), "qmake")

if len(sys.argv) == 1:
    os.execv("/usr/bin/qmake", [argv0] + sys.argv[1:])
else:
    os.execv("/usr/bin/qmake", [argv0] + sys.argv[1:] + ["-qtconf", QT_CONFIG])

And it all works. Even the test application that requires QtWebEngine builds and links fine.

Mike Gabriel: Call for Translations: Arctica Greeter and Ayatana Indicators

29 November, 2017 - 21:24

This is a quick call for help to all non-English native speakers.

Please visit projects hosted by the Arctica Project and the Ayatana Indicators project on Weblate and help localizing our projects into your native language.

Projects waiting for Your Language Expertise

The projects on Weblate are:

Arctica Project:
https://hosted.weblate.org/projects/arctica-framework/

Ayatana Indicators:
https://hosted.weblate.org/projects/ayatana-indicators/

If interested in helping with localizations for these project, please add your language for these projects to your Hosted Weblate Dashboard and stay informed when changes occur, components get added, etc.

Credits

Thanks to all those who already have contributed with translation, so far. However, more work is needed. Let's come together!!!

light+love
Mike Gabriel

Enrico Zini: Qt Creator cross-platform development in Stretch: consolidation

29 November, 2017 - 20:07

Time to consolidate the exploration done yesterday.

qmake -qtconf

Looking at the work done by the Qt/KDE team I found that there is no need to rebuild qmake, as it can be configured by using the -qtconf option.

The work recently done on debhelper provides a starting point that I can build on.

This qtconf makes qmake use the right paths, including running rcc, moc and uic from /usr/bin/, removing the need for the dirty hack:

[Paths]
Prefix=/usr
ArchData=lib/arm-linux-gnueabihf/qt5
Binaries=lib/qt5/bin
Data=share/qt5
Documentation=share/qt5/doc
Examples=lib/arm-linux-gnueabihf/qt5/examples
Headers=include/arm-linux-gnueabihf/qt5
HostBinaries=bin
HostData=lib/arm-linux-gnueabihf/qt5
HostLibraries=lib/arm-linux-gnueabihf
Imports=lib/arm-linux-gnueabihf/qt5/imports
Libraries=lib/arm-linux-gnueabihf
LibraryExecutables=lib/arm-linux-gnueabihf/qt5/libexec
Plugins=lib/arm-linux-gnueabihf/qt5/plugins
Qml2Imports=lib/arm-linux-gnueabihf/qt5/qml
Settings=/etc/xdg
Translations=share/qt5/translations

It still needs overriding the compiler settings, to make it use a compiler different than g++:

qmake -makefile -qtconf /tmp/qmake.conf QMAKE_CC=arm-linux-gnueabihf-gcc QMAKE_CXX=arm-linux-gnueabihf-g++ QMAKE_LINK=arm-linux-gnueabihf-g++

But then -m64 creeps in and breaks the cross-build (this has been fixed since qtbase 5.9.1+dfsg-12. but my target is stretch):

$ make
arm-linux-gnueabihf-g++ -c -m64 -pipe -std=c++11 -O2 -Wall -W -D_REENTRANT -fPIC -DQT_DEPRECATED_WARNINGS -DQT_NO_DEBUG -DQT_SVG_LIB -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -I. -isystem /usr/include/arm-linux-gnueabihf/qt5 -isystem /usr/include/arm-linux-gnueabihf/qt5/QtSvg -isystem /usr/include/arm-linux-gnueabihf/qt5/QtWidgets -isystem /usr/include/arm-linux-gnueabihf/qt5/QtGui -isystem /usr/include/arm-linux-gnueabihf/qt5/QtCore -I. -I/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/linux-g++-64 -o main.o main.cpp
arm-linux-gnueabihf-g++: error: unrecognized command line option ‘-m64’
Makefile:401: recipe for target 'main.o' failed
make: *** [main.o] Error 1

Where does it come from? I found that I could run qmake -query:

$ qmake -query -qtconf /tmp/qmake.conf
QT_SYSROOT:
QT_INSTALL_PREFIX:/usr
QT_INSTALL_ARCHDATA:/usr/lib/arm-linux-gnueabihf/qt5
QT_INSTALL_DATA:/usr/share/qt5
QT_INSTALL_DOCS:/usr/share/qt5/doc
QT_INSTALL_HEADERS:/usr/include/arm-linux-gnueabihf/qt5
QT_INSTALL_LIBS:/usr/lib/arm-linux-gnueabihf
QT_INSTALL_LIBEXECS:/usr/lib/arm-linux-gnueabihf/qt5/libexec
QT_INSTALL_BINS:/usr/lib/qt5/bin
QT_INSTALL_TESTS:/usr/tests
QT_INSTALL_PLUGINS:/usr/lib/arm-linux-gnueabihf/qt5/plugins
QT_INSTALL_IMPORTS:/usr/lib/arm-linux-gnueabihf/qt5/imports
QT_INSTALL_QML:/usr/lib/arm-linux-gnueabihf/qt5/qml
QT_INSTALL_TRANSLATIONS:/usr/share/qt5/translations
QT_INSTALL_CONFIGURATION:/etc/xdg
QT_INSTALL_EXAMPLES:/usr/lib/arm-linux-gnueabihf/qt5/examples
QT_INSTALL_DEMOS:/usr/lib/arm-linux-gnueabihf/qt5/examples
QT_HOST_PREFIX:/usr
QT_HOST_DATA:/usr/lib/arm-linux-gnueabihf/qt5
QT_HOST_BINS:/usr/bin
QT_HOST_LIBS:/usr/lib/arm-linux-gnueabihf
QMAKE_SPEC:linux-g++-64
QMAKE_XSPEC:linux-g++-64
QMAKE_VERSION:3.0
QT_VERSION:5.7.1

I suppose that QMAKE_XSPEC should be linux-g++ instead of linux-g++-64

Poking through Qt's sources I found that I could also tweak TargetSpec, to change it to linux-g++ from its default of linux-g++-64:

[Paths]
Prefix=/usr
ArchData=lib/arm-linux-gnueabihf/qt5
Binaries=lib/qt5/bin
Data=share/qt5
Documentation=share/qt5/doc
Examples=lib/arm-linux-gnueabihf/qt5/examples
Headers=include/arm-linux-gnueabihf/qt5
HostBinaries=bin
HostData=lib/arm-linux-gnueabihf/qt5
HostLibraries=lib/arm-linux-gnueabihf
Imports=lib/arm-linux-gnueabihf/qt5/imports
Libraries=lib/arm-linux-gnueabihf
LibraryExecutables=lib/arm-linux-gnueabihf/qt5/libexec
Plugins=lib/arm-linux-gnueabihf/qt5/plugins
Qml2Imports=lib/arm-linux-gnueabihf/qt5/qml
Settings=/etc/xdg
Translations=share/qt5/translations
TargetSpec=linux-g++

Now, I still need to override the compilers, but that's all I need to do to get a build:

$ qmake -makefile -qtconf /tmp/qmake.conf QMAKE_CC=arm-linux-gnueabihf-gcc QMAKE_CXX=arm-linux-gnueabihf-g++ QMAKE_LINK=arm-linux-gnueabihf-g++
$ make
…
$ file usr/bin/program
usr/bin/program: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=541e472a29a2b309af2fea40d9aa3439e6616bdd, not stripped

Can I get rid of needing to override the compilers? Let's try poking with TargetSpec:

# cd /usr/lib/arm-linux-gnueabihf/qt5/mkspecs/
# cp -r linux-g++ arm-linux-gnueabihf
# cd arm-linux-gnueabihf
# …tweak…
# cat qmake.conf
#
# qmake configuration for arm-linux-gnueabihf
#

MAKEFILE_GENERATOR      = UNIX
CONFIG                 += incremental
QMAKE_INCREMENTAL_STYLE = sublib

include(../common/linux.conf)
include(../common/gcc-base-unix.conf)
include(../common/g++-unix.conf)

QMAKE_COMPILER          = arm-linux-gnueabihf-gcc

QMAKE_CC                = arm-linux-gnueabihf-gcc

QMAKE_LINK_C            = $$QMAKE_CC
QMAKE_LINK_C_SHLIB      = $$QMAKE_CC

QMAKE_CXX               = arm-linux-gnueabihf-g++

QMAKE_LINK              = $$QMAKE_CXX
QMAKE_LINK_SHLIB        = $$QMAKE_CXX

load(qt_config)

Now, using arm-linux-gnueabihf as TargetSpec:

[Paths]
Prefix=/usr
ArchData=lib/arm-linux-gnueabihf/qt5
Binaries=lib/qt5/bin
Data=share/qt5
Documentation=share/qt5/doc
Examples=lib/arm-linux-gnueabihf/qt5/examples
Headers=include/arm-linux-gnueabihf/qt5
HostBinaries=bin
HostData=lib/arm-linux-gnueabihf/qt5
HostLibraries=lib/arm-linux-gnueabihf
Imports=lib/arm-linux-gnueabihf/qt5/imports
Libraries=lib/arm-linux-gnueabihf
LibraryExecutables=lib/arm-linux-gnueabihf/qt5/libexec
Plugins=lib/arm-linux-gnueabihf/qt5/plugins
Qml2Imports=lib/arm-linux-gnueabihf/qt5/qml
Settings=/etc/xdg
Translations=share/qt5/translations
TargetSpec=arm-linux-gnueabihf

It finally works:

$ qmake -makefile -qtconf /tmp/qmake.conf
$ make
$ file usr/bin/program
usr/bin/program: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=541e472a29a2b309af2fea40d9aa3439e6616bdd, not stripped

Skipping prl file '/usr/lib/arm-linux-gnueabihf/libQt5PlatformSupport.prl', because it cannot be opened (No such file or directory).

Turn this into a kit for Qt Creator

I could not find a way to configure -qtconf in a kit in Qt Creator. Since Qt Creator detects versions of Qt by pointing at their qmake, I resorted to making a custom qmake wrapper:

#!/usr/bin/python3

# /usr/local/bin/qmake-arm-linux-gnueabihf

import sys, os

QT_CONFIG = "/home/enrico/qt-arm-linux-gnueabihf.conf"

argv0 = os.path.join(os.path.dirname(sys.argv[0]), "qmake")

if len(sys.argv) == 1:
    os.execv("/usr/bin/qmake", [argv0] + sys.argv[1:])
else:
    os.execv("/usr/bin/qmake", [argv0] + sys.argv[1:] + ["-qtconf", QT_CONFIG])

Just calling qmake -qtconf /home/enrico/qt-arm-linux-gnueabihf.conf "$@" is not enough: for some invocations of qmake, providing -qtconf gives an error:

$ qmake -qtconf /home/enrico/qt-arm-linux-gnueabihf.conf
qmake: could not find a Qt installation of 'conf'

If needed I can look at qmake's sources to see what is going on, but for now the wrapper above seems to be enough to cover all the needs of Qt Creator, and if the wrapper is in the path, Qt Creator manages to autodetect it on startup.

These are the details of a working arm-linux-gnueabihf kit for Qt Creator:

  • Device type: Generic Linux Device
  • Device: my hi-fi
  • Sysroot: blank
  • C compiler: /usr/bin/arm-linux-gnueabihf-gcc
  • C++ compiler: /usr/bin/arm-linux-gnueabihf-g++
  • Qt version: /usr/local/bin/qmake-arm-linux-gnueabihf

And with this, software cross-builds locally and can be deployed and tested remotely.

Summary of the situation so far

This is what is needed to do cross-build now:

  • Depend on crossbuild-essential-armhf
  • Install :armhf versions of Qt and all build-dependencies
  • Deploy a custom mkspec
  • Deploy a custom qt.conf
  • Deploy qmake wrapper
  • Configure a kit

No dirty hacks, everything can be shipped by a single .deb package.

Further things to investigate:

  • Find out if there is a way to package and install a kit for Qt Creator
  • Find an easy way to use a chroot as the cross-build environment, to be able to use cross-build-dependencies that cannot be coinstalled together with their native version

Enrico Zini: Qt Creator cross-platform development in Stretch: consolidation

29 November, 2017 - 20:07

Time to consolidate the exploration done yesterday.

qmake -qtconf

Looking at the work done by the Qt/KDE team I found that there is no need to rebuild qmake, as it can be configured by using the -qtconf option.

The work recently done on debhelper provides a starting point that I can build on.

This qtconf makes qmake use the right paths, including running rcc, moc and uic from /usr/bin/, removing the need for the dirty hack:

[Paths]
Prefix=/usr
ArchData=lib/arm-linux-gnueabihf/qt5
Binaries=lib/qt5/bin
Data=share/qt5
Documentation=share/qt5/doc
Examples=lib/arm-linux-gnueabihf/qt5/examples
Headers=include/arm-linux-gnueabihf/qt5
HostBinaries=bin
HostData=lib/arm-linux-gnueabihf/qt5
HostLibraries=lib/arm-linux-gnueabihf
Imports=lib/arm-linux-gnueabihf/qt5/imports
Libraries=lib/arm-linux-gnueabihf
LibraryExecutables=lib/arm-linux-gnueabihf/qt5/libexec
Plugins=lib/arm-linux-gnueabihf/qt5/plugins
Qml2Imports=lib/arm-linux-gnueabihf/qt5/qml
Settings=/etc/xdg
Translations=share/qt5/translations

It still needs overriding the compiler settings, to make it use a compiler different than g++:

qmake -makefile -qtconf /tmp/qmake.conf QMAKE_CC=arm-linux-gnueabihf-gcc QMAKE_CXX=arm-linux-gnueabihf-g++ QMAKE_LINK=arm-linux-gnueabihf-g++

But then -m64 creeps in and breaks the cross-build:

$ make
arm-linux-gnueabihf-g++ -c -m64 -pipe -std=c++11 -O2 -Wall -W -D_REENTRANT -fPIC -DQT_DEPRECATED_WARNINGS -DQT_NO_DEBUG -DQT_SVG_LIB -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -I. -isystem /usr/include/arm-linux-gnueabihf/qt5 -isystem /usr/include/arm-linux-gnueabihf/qt5/QtSvg -isystem /usr/include/arm-linux-gnueabihf/qt5/QtWidgets -isystem /usr/include/arm-linux-gnueabihf/qt5/QtGui -isystem /usr/include/arm-linux-gnueabihf/qt5/QtCore -I. -I/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/linux-g++-64 -o main.o main.cpp
arm-linux-gnueabihf-g++: error: unrecognized command line option ‘-m64’
Makefile:401: recipe for target 'main.o' failed
make: *** [main.o] Error 1

Where does it come from? I found that I could run qmake -query:

$ qmake -query -qtconf /tmp/qmake.conf
QT_SYSROOT:
QT_INSTALL_PREFIX:/usr
QT_INSTALL_ARCHDATA:/usr/lib/arm-linux-gnueabihf/qt5
QT_INSTALL_DATA:/usr/share/qt5
QT_INSTALL_DOCS:/usr/share/qt5/doc
QT_INSTALL_HEADERS:/usr/include/arm-linux-gnueabihf/qt5
QT_INSTALL_LIBS:/usr/lib/arm-linux-gnueabihf
QT_INSTALL_LIBEXECS:/usr/lib/arm-linux-gnueabihf/qt5/libexec
QT_INSTALL_BINS:/usr/lib/qt5/bin
QT_INSTALL_TESTS:/usr/tests
QT_INSTALL_PLUGINS:/usr/lib/arm-linux-gnueabihf/qt5/plugins
QT_INSTALL_IMPORTS:/usr/lib/arm-linux-gnueabihf/qt5/imports
QT_INSTALL_QML:/usr/lib/arm-linux-gnueabihf/qt5/qml
QT_INSTALL_TRANSLATIONS:/usr/share/qt5/translations
QT_INSTALL_CONFIGURATION:/etc/xdg
QT_INSTALL_EXAMPLES:/usr/lib/arm-linux-gnueabihf/qt5/examples
QT_INSTALL_DEMOS:/usr/lib/arm-linux-gnueabihf/qt5/examples
QT_HOST_PREFIX:/usr
QT_HOST_DATA:/usr/lib/arm-linux-gnueabihf/qt5
QT_HOST_BINS:/usr/bin
QT_HOST_LIBS:/usr/lib/arm-linux-gnueabihf
QMAKE_SPEC:linux-g++-64
QMAKE_XSPEC:linux-g++-64
QMAKE_VERSION:3.0
QT_VERSION:5.7.1

I suppose that QMAKE_XSPEC should be linux-g++ instead of linux-g++-64

Poking through Qt's sources I found that I could also tweak TargetSpec, to change it to linux-g++ from its default of linux-g++-64:

[Paths]
Prefix=/usr
ArchData=lib/arm-linux-gnueabihf/qt5
Binaries=lib/qt5/bin
Data=share/qt5
Documentation=share/qt5/doc
Examples=lib/arm-linux-gnueabihf/qt5/examples
Headers=include/arm-linux-gnueabihf/qt5
HostBinaries=bin
HostData=lib/arm-linux-gnueabihf/qt5
HostLibraries=lib/arm-linux-gnueabihf
Imports=lib/arm-linux-gnueabihf/qt5/imports
Libraries=lib/arm-linux-gnueabihf
LibraryExecutables=lib/arm-linux-gnueabihf/qt5/libexec
Plugins=lib/arm-linux-gnueabihf/qt5/plugins
Qml2Imports=lib/arm-linux-gnueabihf/qt5/qml
Settings=/etc/xdg
Translations=share/qt5/translations
TargetSpec=linux-g++

Now, I still need to override the compilers, but that's all I need to do to get a build:

$ qmake -makefile -qtconf /tmp/qmake.conf QMAKE_CC=arm-linux-gnueabihf-gcc QMAKE_CXX=arm-linux-gnueabihf-g++ QMAKE_LINK=arm-linux-gnueabihf-g++
$ make
…
$ file usr/bin/program
usr/bin/program: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=541e472a29a2b309af2fea40d9aa3439e6616bdd, not stripped

Can I get rid of needing to override the compilers? Let's try poking with TargetSpec:

# cd /usr/lib/arm-linux-gnueabihf/qt5/mkspecs/
# cp -r linux-g++ arm-linux-gnueabihf
# cd arm-linux-gnueabihf
# …tweak…
# cat qmake.conf
#
# qmake configuration for arm-linux-gnueabihf
#

MAKEFILE_GENERATOR      = UNIX
CONFIG                 += incremental
QMAKE_INCREMENTAL_STYLE = sublib

include(../common/linux.conf)
include(../common/gcc-base-unix.conf)
include(../common/g++-unix.conf)

QMAKE_COMPILER          = arm-linux-gnueabihf-gcc

QMAKE_CC                = arm-linux-gnueabihf-gcc

QMAKE_LINK_C            = $$QMAKE_CC
QMAKE_LINK_C_SHLIB      = $$QMAKE_CC

QMAKE_CXX               = arm-linux-gnueabihf-g++

QMAKE_LINK              = $$QMAKE_CXX
QMAKE_LINK_SHLIB        = $$QMAKE_CXX

load(qt_config)

Now, using arm-linux-gnueabihf as TargetSpec:

[Paths]
Prefix=/usr
ArchData=lib/arm-linux-gnueabihf/qt5
Binaries=lib/qt5/bin
Data=share/qt5
Documentation=share/qt5/doc
Examples=lib/arm-linux-gnueabihf/qt5/examples
Headers=include/arm-linux-gnueabihf/qt5
HostBinaries=bin
HostData=lib/arm-linux-gnueabihf/qt5
HostLibraries=lib/arm-linux-gnueabihf
Imports=lib/arm-linux-gnueabihf/qt5/imports
Libraries=lib/arm-linux-gnueabihf
LibraryExecutables=lib/arm-linux-gnueabihf/qt5/libexec
Plugins=lib/arm-linux-gnueabihf/qt5/plugins
Qml2Imports=lib/arm-linux-gnueabihf/qt5/qml
Settings=/etc/xdg
Translations=share/qt5/translations
TargetSpec=arm-linux-gnueabihf

It finally works:

$ qmake -makefile -qtconf /tmp/qmake.conf
$ make
$ file usr/bin/program
usr/bin/program: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=541e472a29a2b309af2fea40d9aa3439e6616bdd, not stripped

Skipping prl file '/usr/lib/arm-linux-gnueabihf/libQt5PlatformSupport.prl', because it cannot be opened (No such file or directory).

Turn this into a kit for Qt Creator

I could not find a way to configure -qtconf in a kit in Qt Creator. Since Qt Creator detects versions of Qt by pointing at their qmake, I resorted to making a custom qmake wrapper:

#!/usr/bin/python3

# /usr/local/bin/qmake-arm-linux-gnueabihf

import sys, os

QT_CONFIG = "/home/enrico/qt-arm-linux-gnueabihf.conf"

argv0 = os.path.join(os.path.dirname(sys.argv[0]), "qmake")

if len(sys.argv) == 1:
    os.execv("/usr/bin/qmake", [argv0] + sys.argv[1:])
else:
    os.execv("/usr/bin/qmake", [argv0] + sys.argv[1:] + ["-qtconf", QT_CONFIG])

Just calling qmake -qtconf /home/enrico/qt-arm-linux-gnueabihf.conf "$@" is not enough: for some invocations of qmake, providing -qtconf gives an error:

$ qmake -qtconf /home/enrico/qt-arm-linux-gnueabihf.conf
qmake: could not find a Qt installation of 'conf'

If needed I can look at qmake's sources to see what is going on, but for now the wrapper above seems to be enough to cover all the needs of Qt Creator, and if the wrapper is in the path, Qt Creator manages to autodetect it on startup.

These are the details of a working arm-linux-gnueabihf kit for Qt Creator:

  • Device type: Generic Linux Device
  • Device: my hi-fi
  • Sysroot: blank
  • C compiler: /usr/bin/arm-linux-gnueabihf-gcc
  • C++ compiler: /usr/bin/arm-linux-gnueabihf-g++
  • Qt version: /usr/local/bin/qmake-arm-linux-gnueabihf

And with this, software cross-builds locally and can be deployed and tested remotely.

Summary of the situation so far

This is what is needed to do cross-build now:

  • Depend on crossbuild-essential-armhf
  • Install :armhf versions of Qt and all build-dependencies
  • Deploy a custom mkspec
  • Deploy a custom qt.conf
  • Deploy qmake wrapper
  • Configure a kit

No dirty hacks, everything can be shipped by a single .deb package.

Further things to investigate:

  • Find out if there is a way to package and install a kit for Qt Creator
  • Find an easy way to use a chroot as the cross-build environment, to be able to use cross-build-dependencies that cannot be coinstalled together with their native version

François Marier: Proxy ACME challenges to a single machine

29 November, 2017 - 13:10

The Libravatar mirrors are setup using DNS round-robin which makes it a little challenging to automatically provision Let's Encrypt certificates.

In order to be able to use Certbot's webroot plugin, I need to be able to simultaneously host a randomly-named file into the webroot of each mirror. The reason is that the verifier will connect to seccdn.libravatar.org, but there's no way to know which of the DNS entries it will hit. I could copy the file over to all of the mirrors, but that would be annoying since some of the mirrors are run by volunteers and I don't have direct access to them.

Thankfully, Scott Helme has shared his elegant solution: proxy the .well-known/acme-challenge/ directory from all of the mirrors to a single validation host. Here's the exact configuration I ended up with.

DNS Configuration

In order to serve the certbot validation files separately from the main service, I created a new hostname, acme.libravatar.org, pointing to the main Libravatar server:

CNAME acme libravatar.org.
Mirror Configuration

On each mirror, I created a new Apache vhost on port 80 to proxy the acme challenge files by putting the following in the existing port 443 vhost config (/etc/apache2/sites-available/libravatar-seccdn.conf):

<VirtualHost *:80>
    ServerName __SECCDNSERVERNAME__
    ServerAdmin __WEBMASTEREMAIL__

    ProxyPass /.well-known/acme-challenge/ http://acme.libravatar.org/.well-known/acme-challenge/
    ProxyPassReverse /.well-known/acme-challenge/ http://acme.libravatar.org/.well-known/acme-challenge/
</VirtualHost>

Then I enabled the right modules and restarted Apache:

a2enmod proxy
a2enmod proxy_http
systemctl restart apache2.service

Finally, I added a cronjob in /etc/cron.daily/commit-new-seccdn-cert to commit the new cert to etckeeper automatically:

#!/bin/sh
cd /etc/libravatar
/usr/bin/git commit --quiet -m "New seccdn cert" seccdn.crt seccdn.pem seccdn-chain.pem > /dev/null || true
Main Configuration

On the main server, I created a new webroot:

mkdir -p /var/www/acme/.well-known

and a new vhost in /etc/apache2/sites-available/acme.conf:

<VirtualHost *:80>
    ServerName acme.libravatar.org
    ServerAdmin webmaster@libravatar.org
    DocumentRoot /var/www/acme
    <Directory /var/www/acme>
        Options -Indexes
    </Directory>
</VirtualHost>

before enabling it and restarting Apache:

a2ensite acme
systemctl restart apache2.service
Registering a new TLS certificate

With all of this in place, I was able to register the cert easily using the webroot plugin on the main server:

certbot certonly --webroot -w /var/www/acme -d seccdn.libravatar.org

The resulting certificate will then be automatically renewed before it expires.

Dirk Eddelbuettel: Rcpp now used by 1250 CRAN packages

29 November, 2017 - 08:23

Earlier today Rcpp passed 1250 reverse-dependencies on CRAN as another big milestone. The graph is on the left depicts the growth of Rcpp usage (as measured by Depends, Imports and LinkingTo, but excluding Suggests) over time.

Rcpp cleared 300 packages in November 2014. It passed 400 packages in June 2015 (when I only tweeted about it), 500 packages in late October 2015, 600 packages last March, 700 packages last July, 800 packages last October, 900 packages early January, and 1000 packages in April. The chart extends to the very beginning via manually compiled data from CRANberries and checked with crandb. The next part uses manually saved entries. The core (and by far largest) part of the data set was generated semi-automatically via a short script appending updates to a small file-based backend. A list of packages using Rcpp is kept on this page.

Also displayed in the graph is the relative proportion of CRAN packages using Rcpp. The four per-cent hurdle was cleared just before useR! 2014 where I showed a similar graph (as two distinct graphs) in my invited talk. We passed five percent in December of 2014, six percent July of 2015, seven percent just before Christmas 2015, eight percent last summer, nine percent mid-December 2016 and then cracked ten percent this summer.

1250 user packages is staggering. We can use the progression of CRAN itself compiled by Henrik in a series of posts and emails to the main development mailing list. A decade ago CRAN itself did not have 1250 packages, and here we are approaching 12k with Rcpp at 10% and growing steadily. Amazeballs.

This puts a whole lot of responsibility on us in the Rcpp team as we continue to keep Rcpp as performant and reliable as it has been.

And with that, and as always, a very big Thank You! to all users and contributors of Rcpp for help, suggestions, bug reports, documentation or, of course, code.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Norbert Preining: Debian/TeX Live and Biber updates

29 November, 2017 - 08:02

Halloween is over, Thanksgiving is over, snow is starting to fall here in Ishikawa, time for a new update of TeX Live (2017.20171128-1) – and this time also Biber (2.9-1) – in Debian.

As usual, biblatex and Biber updates go hand in hand, so here we are with the newest version of the next-generation bibliography management for LaTeX.

Other than this, nothing spectacular here but the usual flow of updates and new packages. Notably might be the IBM Plex font family (see the CTAN packages plex and plex-otf), which replaces Helvetica Neue on official IBM communications. A complete family with sans, serif, and mono in various weights.

Enjoy.

New packages

beamertheme-saintpetersburg, bxtexlogo, cm-mf-extra-bold, ctan-o-mat, gridslides, lyluatex, plex, plex-otf, simpleinvoice, xii-lat.

Updated packages

GS1, abnt, adobemapping, afm2pl, amsldoc-it, amstex, amsthdoc-it, archaeologie, arsclassica, babel, babel-georgian, baskervillef, beamer, beebe, beilstein, besjournals, bib2gls, biber, biblatex, biblatex-abnt, biblatex-anonymous, biblatex-apa, biblatex-archaeology, biblatex-arthistory-bonn, biblatex-bookinother, biblatex-fiwi, biblatex-manuscripts-philology, biblatex-oxref, biblatex-publist, biblatex-realauthor, biblatex-sbl, biblatex-shortfields, bibtexu, c90, ccicons, cjkutils, cm, cmexb, cns, context, context-filter, context-fullpage, context-letter, context-title, contracard, crossreftools, crossrefware, cslatex, csplain, ctex, cyrillic-bin, cyrplain, datatool, datetime2-german, datetime2-spanish, dnp, ducksay, dvipos, dynkin-diagrams, eplain, etoolbox, europasscv, euxm, exam, feyn, fontspec, fontware, genmisc, glossaries, glossaries-extra, glyphlist, graphics-def, guide-to-latex, gustlib, gustprog, hyphen-base, impatient-cn, ipaex, iscram, jadetex, jlreq, jmn, kantlipsum, kluwer, kpathsea, l3build, l3experimental, l3kernel, l3packages, lambda, latex-bin, latex2e-help-texinfo-spanish, latexconfig, latexmk, limecv, lollipop, lt3graph, luatex, luatexja, lwarp, manfnt-font, mathtools, metafont, mex, mfirstuc, mflua, mltex, mptopdf, musicography, musixtex, novel, octave, omegaware, otibet, overlays, pdfpages, pdfwin, pkgloader, platex, pst-circ, pst-fractal, pst-ghsb, pst-pdgr, pst-plot, pxrubrica, qpxqtx, reledmac, repere, revtex4, roex, scsnowman, siunitx, srbook-mem, sympytexpackage, synctex, tetex, tex4ht, texdoc, texsis, tikzducks, tlshell, ttfutils, tugboat, turabian-formatting, unicode-math, uplatex, vlna, web, witharrows, xcharter, xecjk, xetex, xetexconfig, xii, xmltex, xmltexconfig, ycbook, zebra-goodies.

Lisandro Damián Nicanor Pérez Meyer: Experimental cross compiling Qt in Debian packages

29 November, 2017 - 02:35
Some time ago we the Qt/KDE team where contacted by Helmut Grohne. He was trying to cross compile Debian packages in general thanks to Ubuntu/Debian's multi-arch support, and he was having problems with Qt-based ones.

As far as we understand Qt upstreams only support cross compiling by having a toolchain for each pair of architectures involved. In Debian terms, and only considering current official architectures, that would mean building 90 cross toolchains. It clearly doesn't scales.

So we set up to discuss if somehow we could use multiarch to let debian packages using Qt to cross compile.

In the meantime Enrico Zini had the same idea. He wrote a nice summary of the situation at that time in his blog.

After many thinking some ideas were tested and we've got to the point of solving/hacking the issue. As this is not something directly supported by upstream you should take care, and file bugs whenever necessary.

Dmitry Schanev from our team's side and Helmut from the debian-cross side worked a lot on it, and I would like to present what they have done. To be fair it's mostly described in our team's gobby qt-cross page, but I would like to give it some publicity in order to let people know about it and why not, find and help solving bugs.

General stuff

The first thing that was done was to move Qt binaries from their (Debian original) multi-arch path to a non multi-arch one, providing symlinks for compatibility. In this way the path of the binaries is the same for any arch (why they where not there is a long story, but nothing to worry now).

This move needed some other touches, like qtchooser being updated with the new paths.

The other changes where related to how we do our packaging:


  • All packages containing binaries are now M-A:foreign.
  • Some packages (qt3d, qtwayland) had binaries split to allow that.
  • qttools5-dev-tools now depends on libqt5sql5-sqlite (not uploaded yet)

qmake related changes
We also needed to address qmake. To begin with we splitted the package containing it into qt5-qmake-bin (M-A:foreign) and qt5-qmake (M-A:same). The first one has the binaries and the second the relevant mkspecs for some arch.
The rest of the "magic" comes from debhelper. It generates a qt.conf file with the right paths for each cross compilation and also passes cross QMAKE_CC and QMAKE_CXX to qmake when needed.

autotools

qt5-qmake will ship /usr/bin/$(DEB_HOST_GNU_TYPE)-qmake executable for use with AC_CHECK_TOOL (not uploaded yet).

There is still work to be done, but so far we have been able to cross compile packages using for example sbuild.

Petter Reinholdtsen: Metadata proposal for movies on the Internet Archive

28 November, 2017 - 18:00

It would be easier to locate the movie you want to watch in the Internet Archive, if the metadata about each movie was more complete and accurate. In the archiving community, a well known saying state that good metadata is a love letter to the future. The metadata in the Internet Archive could use a face lift for the future to love us back. Here is a proposal for a small improvement that would make the metadata more useful today. I've been unable to find any document describing the various standard fields available when uploading videos to the archive, so this proposal is based on my best quess and searching through several of the existing movies.

I have a few use cases in mind. First of all, I would like to be able to count the number of distinct movies in the Internet Archive, without duplicates. I would further like to identify the IMDB title ID of the movies in the Internet Archive, to be able to look up a IMDB title ID and know if I can fetch the video from there and share it with my friends.

Second, I would like the Butter data provider for The Internet archive (available from github), to list as many of the good movies as possible. The plugin currently do a search in the archive with the following parameters:

collection:moviesandfilms
AND NOT collection:movie_trailers
AND -mediatype:collection
AND format:"Archive BitTorrent"
AND year

Most of the cool movies that fail to show up in Butter do so because the 'year' field is missing. The 'year' field is populated by the year part from the 'date' field, and should be when the movie was released (date or year). Two such examples are Ben Hur from 1905 and Caminandes 2: Gran Dillama from 2013, where the year metadata field is missing.

So, my proposal is simply, for every movie in The Internet Archive where an IMDB title ID exist, please fill in these metadata fields (note, they can be updated also long after the video was uploaded, but as far as I can tell, only by the uploader):
mediatype
Should be 'movie' for movies.
collection
Should contain 'moviesandfilms'.
title
The title of the movie, without the publication year.
date
The data or year the movie was released. This make the movie show up in Butter, as well as make it possible to know the age of the movie and is useful to figure out copyright status.
director
The director of the movie. This make it easier to know if the correct movie is found in movie databases.
publisher
The production company making the movie. Also useful for identifying the correct movie.
links
Add a link to the IMDB title page, for example like this: <a href="http://www.imdb.com/title/tt0028496/">Movie in IMDB</a>. This make it easier to find duplicates and allow for counting of number of unique movies in the Archive. Other external references, like to TMDB, could be added like this too.

I did consider proposing a Custom field for the IMDB title ID (for example 'imdb_title_url', 'imdb_code' or simply 'imdb', but suspect it will be easier to simply place it in the links free text field.

I created a list of IMDB title IDs for several thousand movies in the Internet Archive, but I also got a list of several thousand movies without such IMDB title ID (and quite a few duplicates). It would be great if this data set could be integrated into the Internet Archive metadata to be available for everyone in the future, but with the current policy of leaving metadata editing to the uploaders, it will take a while before this happen. If you have uploaded movies into the Internet Archive, you can help. Please consider following my proposal above for your movies, to ensure that movie is properly counted. :)

The list is mostly generated using wikidata, which based on Wikipedia articles make it possible to link between IMDB and movies in the Internet Archive. But there are lots of movies without a Wikipedia article, and some movies where only a collection page exist (like for the Caminandes example above, where there are three movies but only one Wikidata entry).

Enrico Zini: Qt Creator cross-platform development in Stretch

28 November, 2017 - 17:33

I have been asked to create a system that makes it easy to do cross-architecture Qt development with Debian Stretch.

The goal is this:

  • Do development with Qt Creator, including remote debugging on the target architecture
  • Compile using native code, to avoid running the compilers inside an emulator making the build slower
  • Leverage all of Debian as development environment, using existing Debian packages for build-dependencies
  • Set everything up just by installing a single .deb package, which ideally gives Qt Creator a new kit that just works

I did some exploration on this some time ago, and the Qt/KDE and cross-build people did some work on this, too.

Now I'm trying to build on all that. I need to target Stretch so I cannot build on the recent improvements in Qt packaging, but I can at least use the experience that went into those changes.

I have two sample Qt application to try and cross-build, one that depends on a non-Qt library (libmosquittopp-dev), and one that depends on a nasty Qt library (qtwebengine5-dev).

This post has the notes from the first day of trying out different strategies.

It begins

I imported the two projects in Qt creator, installed their amd64 dependencies and make sure they build for the current system, with the default kit, no cross-building yet.

That works, good.

Now let's see about the rest:

dpkg --add-architecture armhf
apt install crossbuild-essential-armhf
A cross-build kit for Qt Creator

I created a new armhf kit for Qt Creator:

  • Device type: Generic Linux Device
  • Device: my hi-fi
  • Sysroot: blank
  • C compiler: /usr/bin/arm-linux-gnueabihf-gcc
  • C++ compiler: /usr/bin/arm-linux-gnueabihf-g++
  • Qt version: /usr/lib/arm-linux-gnueabihf/qt5/bin/qmake

I ran qmake from Qt Creator and go this:

10:59:49: Starting: "/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake" …/project.pro -spec linux-g++ CONFIG+=debug CONFIG+=qml_debug
sh: 1: /usr/lib/arm-linux-gnueabihf/qt5/bin/rcc: not found
10:59:50: The process "/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake" exited normally.

rcc is provided by qtbase5-dev-tools, which cannot be coinstalled alongside other architectures' version of itself:

# apt install qtbase5-dev-tools:armhf
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  qtbase5-dev-tools
The following NEW packages will be installed:
  qtbase5-dev-tools:armhf
0 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.
Need to get 651 kB of archives.
After this operation, 985 kB disk space will be freed.
Do you want to continue? [Y/n] q

I'll try with a very dirty hack:

# cd /usr/lib/arm-linux-gnueabihf/qt5/bin
# ln -s `which rcc` .
# ln -s `which uic` .
# ln -s `which moc` .
# ls -la
total 1944
drwxr-xr-x 2 root root    4096 Nov 29 11:04 .
drwxr-xr-x 7 root root    4096 Nov 28 17:05 ..
lrwxrwxrwx 1 root root      12 Nov 29 11:04 moc -> /usr/bin/moc
-rwxr-xr-x 1 root root 1982208 Jan 11  2017 qmake
lrwxrwxrwx 1 root root      12 Nov 29 11:04 rcc -> /usr/bin/rcc
lrwxrwxrwx 1 root root      12 Nov 29 11:04 uic -> /usr/bin/uic

This is ugly:

  • It places files in /usr outside /usr/local that are not maintained by the package manager
  • It places amd64 executables in /usr/lib/arm-linux-gnueabihf which should contain armhf code

Let's see what happens in Qt Creator:

11:14:35: Starting: "/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake" …/project.pro -spec linux-g++ CONFIG+=debug CONFIG+=qml_debug
11:14:35: The process "/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake" exited normally.
11:14:35: Starting: "/usr/bin/make" qmake_all
make: Nothing to be done for 'qmake_all'.
11:14:36: The process "/usr/bin/make" exited normally.

and build:

11:15:29: Starting: "/usr/bin/make"
g++ -c -pipe -std=c++11 -g -Wall -W -D_REENTRANT -fPIC -DQT_DEPRECATED_WARNINGS -DQT_QML_DEBUG -DQT_SVG_LIB -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_CORE_LIB -I../project -I. -isystem /usr/include/arm-linux-gnueabihf/qt5 -isystem /usr/include/arm-linux-gnueabihf/qt5/QtSvg -isystem /usr/include/arm-linux-gnueabihf/qt5/QtWidgets -isystem /usr/include/arm-linux-gnueabihf/qt5/QtGui -isystem /usr/include/arm-linux-gnueabihf/qt5/QtCore -I. -I/usr/lib/arm-linux-gnueabihf/qt5/mkspecs/linux-g++ -o main.o ../project/main.cpp

Unfortunately, it is using g++ even though I configured the kit to use /usr/bin/arm-linux-gnueabihf-* instead.

Trying an explicit override in the .pro:

QMAKE_CXX = /usr/bin/arm-linux-gnueabihf-g++
QMAKE_LINK = /usr/bin/arm-linux-gnueabihf-g++

And the project builds and runs fine on the Raspberry Pi, using a kit and two simple overrides in the .pro, all build dependencies as packaged by Debian, and a toolchain that entirely runs on native code.

Entirely?

$ file /usr/lib/arm-linux-gnueabihf/qt5/bin/qmake
/usr/lib/arm-linux-gnueabihf/qt5/bin/qmake: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=1f5b063926570702f5568b1e5cec6c70d214fc73, stripped
$ dpkg-architecture -q DEB_HOST_ARCH
amd64
$ /usr/lib/arm-linux-gnueabihf/qt5/bin/qmake -v
QMake version 3.0
Using Qt version 5.7.1 in /usr/lib/arm-linux-gnueabihf

Indeed qmake is armhf code that runs happily on my amd64 laptop because I accidentally have qemu-user-static installed.

This strategy produces results, although it depends on a dirty hack. To summarise it:

  • Depend on crossbuild-essential-armhf, qemu-user-static, qemu-system-arm
  • Install :armhf versions of Qt and all build-dependencies
  • Symlink native moc, rcc, uic into the armhf bin directory (argh!)
  • Configure a kit as before
  • Override QMAKE_CXX and QMAKE_LINK in the .pro

Other things to investigate, besides removing the need for that dirty hack:

  • Find out if there is a way to package and install a kit for Qt Creator
  • Find out why qmake is ignoring the compiler settings from the kit and needs overrides in the .pro. Ideally the .pro should be unmodified, so that it can be used for all builds
A native qmake

Would it be possible to build a native version of qmake tweaked to point to all the right bits out of the box?

$ apt source qtbase-opensource-src-5.7.1+dfsg
$ qtbase-opensource-src-5.7.1+dfsg$
$ DEB_HOST_MULTIARCH=arm-linux-gnueabihf DEB_HOST_ARCH_BITS=32 debian/rules override_dh_auto_configure
MAKEFLAGS="-j1" ./configure \
            -confirm-license \
            -prefix "/usr" \
            -bindir "/usr/lib/arm-linux-gnueabihf/qt5/bin" \
            -libdir "/usr/lib/arm-linux-gnueabihf" \
            -docdir "/usr/share/qt5/doc" \
            -headerdir "/usr/include/arm-linux-gnueabihf/qt5" \
            -datadir "/usr/share/qt5" \
            -archdatadir "/usr/lib/arm-linux-gnueabihf/qt5" \
            -plugindir "/usr/lib/arm-linux-gnueabihf/qt5/plugins" \
            -importdir "/usr/lib/arm-linux-gnueabihf/qt5/imports" \
            -translationdir "/usr/share/qt5/translations" \
            -hostdatadir "/usr/lib/arm-linux-gnueabihf/qt5" \
            -sysconfdir "/etc/xdg" \
            -examplesdir "/usr/lib/arm-linux-gnueabihf/qt5/examples" \
            -opensource \
            -plugin-sql-mysql \
            -plugin-sql-odbc \
            -plugin-sql-psql \
            -plugin-sql-sqlite \
            -no-sql-sqlite2 \
            -plugin-sql-tds \
            -system-sqlite \
            -platform linux-g++ \
            -system-harfbuzz \
            -system-zlib \
            -system-libpng \
            -system-libjpeg \
            -system-doubleconversion \
            -openssl \
            -no-rpath \
            -verbose \
            -optimized-qmake \
            -dbus-linked \
            -no-strip \
            -no-separate-debug-info \
            -qpa xcb \
            -xcb \
            -glib \
            -icu \
            -accessibility \
            -compile-examples \
            -no-directfb \
            -gstreamer 1.0 \
            -plugin-sql-ibase -opengl desktop \
…
$ file bin/qmake
bin/qmake: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=ba868730cf34c54d7ddf6df0ab4b6ce5c7d6f2a0, not stripped
$ bin/qmake -v
QMake version 3.0
Using Qt version 5.7.1 in /usr/lib/arm-linux-gnueabihf

./configure in Qt builds qmake, and there is no need to run make afterwards. Good.

DEB_HOST_ARCH_BITS=32 is a hack I added to avoid debian/rules using -platform linux-g++-64 instead of -platform linux-g++.

Let's try to use that in Qt Creator:

sudo cp bin/qmake /usr/local/bin/qmake-arm-linux-gnueabihf

Qt Creator autodetects the new qmake and offers it as one of the available versions of Qt. Nice.

The need for the symlink hack is still there:

11:46:51: Starting: "/usr/local/bin/qmake-arm-linux-gnueabihf" ../project.pro -spec linux-g++-64 CONFIG+=debug CONFIG+=qml_debug
sh: 1: /usr/lib/arm-linux-gnueabihf/qt5/bin/rcc: not found

So is the need for the QMAKE_CXX and QMAKE_LINK overrides in the .pro.

Still, this way I could remove qemu-user-static from my system and the project still builds on my laptop and runs on my Raspberry Pi.

The qemu dependency is not needed anymore, the rest of the problems still stand. I wonder, since I'm rebuilding qmake, if there's a way to tell it to use the compilers I want, and the tools I want, removing the need for the dirty hack and the overrides in the .pro files.

qtwebengine5-dev

How about the other project that depends on qtwebengine5-dev?

# apt install qtwebengine5-dev qtwebengine5-dev:armhf
Reading package lists... Done
Building dependency tree
Reading state information... Done
qtwebengine5-dev is already the newest version (5.7.1+dfsg-6.1).
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 qtwebengine5-dev:armhf : Depends: libqt5webengine5:armhf (= 5.7.1+dfsg-6.1) but it is not going to be installed
                          Depends: libqt5webenginecore5:armhf (= 5.7.1+dfsg-6.1) but it is not going to be installed
                          Depends: libqt5webenginewidgets5:armhf (= 5.7.1+dfsg-6.1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

It turns out that something in the big chain of dependencies of qtwebengine5-dev makes the amd64 and armhf versions not coinstallable.

It seems that I have to abandon the idea of installing armhf build-dependencies in the main development system, and I need to start considering to leave the host system untouched to do the native builds, and use a chroot for the cross-compilers and the cross-build-dependencies.

In the next post, I'll see how that goes.

Pages

Creative Commons License ลิขสิทธิ์ของบทความเป็นของเจ้าของบทความแต่ละชิ้น
ผลงานนี้ ใช้สัญญาอนุญาตของครีเอทีฟคอมมอนส์แบบ แสดงที่มา-อนุญาตแบบเดียวกัน 3.0 ที่ยังไม่ได้ปรับแก้