Planet Debian

Subscribe to Planet Debian feed
Planet Debian - https://planet.debian.org/
Updated: 21 min 13 sec ago

Steve Kemp: Upgraded my first host to buster

9 July, 2019 - 16:01

I upgrade the first of my personal machines to Debian's new stable release, buster, yesterday. So far two minor niggles, but nothing major.

My hosts are controlled, sometimes, by puppet. The puppet-master is running stretch and has puppet 4.8.2 installed. After upgrading my test-host to the new stable I discovered it has puppet 5.5 installed:

root@git ~ # puppet --version 5.5.10

I was not sure if there would be compatibility problems, but after reading the release notes nothing jumped out. Things seemed to work, once I fixed this immediate problem:

     # puppet agent --test
     Warning: Unable to fetch my node definition, but the agent run will continue:
     Warning: SSL_connect returned=1 errno=0 state=error: dh key too small
     Info: Retrieving pluginfacts
     ..

This error-message was repeated multiple times:

SSL_connect returned=1 errno=0 state=error: dh key too small

To fix this comment out the line in /etc/ssl/openssl.cnf which reads:

CipherString = DEFAULT@SECLEVEL=2

The second problem was that I use borg to run backups, once per day on most systems, and twice per day on others. I have an invocation which looks like this:

borg create ${flags} --compression=zlib  --stats ${dest}${self}::$(date +%Y-%m-%d-%H:%M:%S) \
   --exclude=/proc \
   --exclude=/swap.file \
   --exclude=/sys  \
   --exclude=/run  \
   --exclude=/dev  \
   --exclude=/var/log \
   /

That started to fail :

borg: error: unrecognized arguments: /

I fixed this by re-ordering the arguments such that it ended "destination path", and changing --exclude=x to --exclude x:

borg create ${flags} --compression=zlib  --stats \
   --exclude /proc \
   --exclude /swap.file \
   --exclude /sys  \
   --exclude /run  \
   --exclude /dev  \
   --exclude /var/log \
   ${dest}${self}::$(date +%Y-%m-%d-%H:%M:%S)  /

That approach works on my old and new hosts.

I'll leave this single system updated for a few more days to see what else is broken, if anything. Then I'll upgrade them in turn.

Good job!

Daniel Kahn Gillmor: DANE OPENPGPKEY for debian.org

9 July, 2019 - 11:00
DANE OPENPGPKEY for debian.org

I recently announced the publication of Web Key Directory for @debian.org e-mail addresses. This blog post announces another way to fetch OpenPGP certificates for @debian.org e-mail addresses, this time using only the DNS. These two mechanisms are complementary, not in competition. We want to make sure that whatever certificate lookup scheme your OpenPGP client supports, you will be able to find the appropriate certificate.

The additional mechanism we're now supporting (since a few days ago) is DANE OPENPGPKEY, specified in RFC 7929.

How does it work?

DANE OPENPGPKEY works by storing a minimized OpenPGP certificate in the DNS, ideally in a subdomain at label based on a hashed version of the local part of the e-mail address.

With modern GnuPG, if you're interested in retrieving the OpenPGP certificate for dkg as served by the DNS, you can do:

gpg --auto-key-locate clear,nodefault,dane --locate-keys dkg@debian.org

If you're interested in how this DNS zone is populated, take a look at can the code that handles it. Please request improvements if you see ways that this could be improved.

Unfortunately, GnuPG does not currently do DNSSEC validation on these records, so the cryptographic protections offered by this client are not as strong as those provided by WKD (which at least checks the X.509 certificate for a given domain name against the list of trusted root CAs).

Why offer both DANE OPENPGPKEY and WKD?

I'm hoping that the Debian project can ensure that no matter whatever sensible mechanism any OpenPGP client implements for certificate lookup, it will be able to find the appropriate OpenPGP certificate for contacting someone within the @debian.org domain.

A clever OpenPGP client might even consider these two mechanisms -- DANE OPENPGPKEY and WKD -- as corroborative mechanisms, since an attacker who happens to compromise one of them may find it more difficult to compromise both simultaneously.

How to update?

If you are a Debian developer and you want your OpenPGP certificate updated in the DNS, please follow the normal procedures for Debian keyring maintenance like you always have. When a new debian-keyring package is released, we will update these DNS records at the same time.

Thanks

Setting this up would not have been possible without help from weasel on the Debian System Administration team, and Noodles from the keyring-maint team providing guidance.

DANE OPENPGPKEY was documented and shepherded through the IETF by Paul Wouters.

Thanks to all of these people for making it possible.

Rodrigo Siqueira: Status Update, June 2019

9 July, 2019 - 10:00

For a long time, I’m cultivating the desire of getting the habit of writing monthly status update; in some way, Drew DeVault’s Blog posts and Martin Peres advice leverage me toward this direction. So, here I’m am! I decided to embrace the challenge of composing a report per month. I hope this new habit helps me to improve my write, summary, and communication skills; but most importantly, help me to keep track of my work. I want to start this update by describing my work conditions and then focus on the technical stuff.

The last two months, I have to face an infrastructure problem to work. I’m dealing with obstacles such as restricted Internet access and long hours in public transportation from my home to my workplace. Unfortunately, I cannot work in my house due to the lack of space, and the best place to work it is a public library at the University of Brasilia (UnB); go to UnB every day makes me wast around 3h per day in a bus. The library has a great environment, but it also has thousands of internet restrictions, for example, I cannot access websites with ‘.me’ domain and I cannot connect to my IRC bouncer. In summary: It has been hard to work these days. So, let’s stop to talk about non-technical stuff and let’s get to the heart of the matter.

I really like to work on VKMS, I know this isn’t news to anyone, and in June most of my efforts were dedicated to VKMS. One of my paramount endeavors it was found and fixed a bug in vkms that makes kms_cursor_crc, and kms_pipe_crc_basic fails; I was chasing this bug for a long time as can be seen here [1]. After many hours of debugging I sent a patch for handling this issue [2], however, after Daniel’s review, I realize that my patch does not correctly fix the problem. Daniel decided to dig into this issue and find out the root of the problem and later sent a final fix; if you want to see the solution, take a look at [3]. One day, I want to write a post about this fix since it is an interesting subject to discuss.

Daniel also noticed some concurrency problems in the CRC code and sent a patchset composed of 10 patches that tackle the issue. These patches focused on creating better framebuffers manipulation and avoiding race conditions; it took me around 4 days to take a look and test this series. During my review, I asked many things related to concurrency and other clarification about DRM, and Daniel always replied with a very nice and detailed explanation. If you want to learn a little bit more about locks, I recommend you to take a look at [4]; serious, it is really nice!

I also worked for adding the writeback support in vkms; since XDC2018 I could not stop to think about the idea of adding writeback connector in vkms due to the benefits it could bring, such as new test and assist developers with visual output. As a result, I started some clumsy attempts to implement it in January; but I really dove in this issue in the middle of April, and in June I was focused on making it work. It was tough for me to implement these features due to the following reasons:

  1. There isn’t i-g-t test for writeback in the main repository, I had to use a WIP patchset made by Brian and Liviu.
  2. I was not familiar with framebuffer, connectors, and fancy manipulation.

As a result of the above limitations, I had to invest many hours reading the documentation and the DRM/IGT code. In the end, I think that add writeback connectors paid well for me since I feel much more comfortable with many things related to DRM these days. The writeback support was not landed yet, however, at this moment the patch is under review (V3) and changed a lot since the first version; for details about this series take a look at [5]. I’ll write a post about this feature after it gets merged.

After having the writeback connectors working in vkms, I felt so grateful to Brian, Liviu, and Daniel for all the assistance they provided to me. In particular, I was thrilled that Brian and Liviu made kms_writeback test which worked as an implementation guideline for me; as a result, I update their patchset for making it work in the latest version of IGT and made some tiny fixes; my goal was helping them to upstream kms_writeback. I resubmit the series with the hope to see it landed in the IGT [9].

Parallel to my work with writeback, I was trying to figure out how could I expose vkms configurations to the userspace via configfs. After many efforts, I submitted the first version of configfs support; in this patchset I exposed the virtual and writeback connectors. Take a look at [6] for more information about this feature, and definitely, I’ll write a post about this feature after it gets landed.

Finally, I’m still trying to upstream a patch that makes drm_wait_vblank_ioctl return EOPNOTSUPP instead of EINVAL if the driver does not support vblank get landed. Since this change is in the DRM core and also change the userspace, it is not easy to make this patch get landed. For the details about this patch, you can take a look here [7]. I also implemented some changes in the kms_flip to validate the changes that I made in the function drm_wait_vblank_ioctl and it got landed [8].

July Aims

In June I was totally dedicated to vkms, now I want to slow my road a little bit and study more about userspace. I want to take a step back and make some tiny programs using libdrm with the goal to understand the interaction among userspace and kernel space. I also want to take a look at the theoretical part related to computer graphics.

I want to put some effort to improve a tool named kw that help me during my work with Linux Kernel. I also want to take a look at real overlay planes support in vkms. I noted that I have to find a “contribution protocol” (review/write code) that works for me in my current work conditions; otherwise, work will become painful for my relatives and me. Finally, and most importantly, I want to take some days off to enjoy my family.

Info: If you find any problem with this text, please let me know. I will be glad to fix it.

References

[1] “First discussion in the Shayenne’s patch about the CRC problem”. URL: https://lkml.org/lkml/2019/3/10/197

[2] “Patch fix for the CRC issue”. URL: https://patchwork.freedesktop.org/patch/308617/

[3] “Daniel final fix for CRC”. URL: https://patchwork.freedesktop.org/patch/308881/?series=61703&rev=1

[4] “Rework crc worker”. URL: https://patchwork.freedesktop.org/series/61737/

[5] “Introduces writeback support”. URL: https://patchwork.freedesktop.org/series/61738/

[6] “Introduce basic support for configfs”. URL: https://patchwork.freedesktop.org/series/63010/

[7] “Change EINVAL by EOPNOTSUPP when vblank is not supported”. URL: https://patchwork.freedesktop.org/patch/314399/?series=50697&rev=7

[8] “Skip VBlank tests in modules without VBlank”. URL: https://gitlab.freedesktop.org/drm/igt-gpu-tools/commit/2d244aed69165753f3adbbd6468db073dc1acf9a

[9] “Add support for testing writeback connectors”. URL: https://patchwork.freedesktop.org/series/39229/

Jonathan Wiltshire: Too close?

9 July, 2019 - 03:39

At times of stress I’m prone to topical nightmares, but they are usually fairly mundane – last night, for example, I dreamed that I’d mixed up bullseye and bookworm in one of the announcements of future code names.

But Saturday night was a whole different game. Imagine taking a rucksack out of the cupboard under the stairs, and thinking it a bit too heavy for an empty bag. You open the top and it’s full of small packages tied up with brown paper and string. As you take each one out and set it aside you realise, with mounting horror, that these are all packages missing from buster and which should have been in the release. But it’s too late to do anything about that now; you know the press release went out already because you signed it off yourself, so you can’t do anything else but get all the packages out of the bag and see how many were forgotten. And you dig, and count, and dig, and it’s like Mary Poppins’ carpet bag, and still they keep on coming…

Sometimes I wonder if I’ve got too close.

Andy Simpkins: Buster Release Party – Cambridge, UK

8 July, 2019 - 21:21

With the release of Debian GNU/Linux 10.0.0 “Buster” completing in the small hours of yesterday morning (0200hrs UTC or thereabouts) most of the ‘release parties’ had already been and gone…. Not so for the Cambridge contingent who had scheduled a get together for the Sunday [0], knowing that various attendees would have been working on the release until the end.

The Sunday afternoon saw a gathering in the Haymakers pub to celebrate a successful release.   We would publicly like to thank the Raspberry Pi foundation [1], and Mythic Beasts [2] who between them picked up the tab for our bar bill – Cheers and thank you!

 

 

 

 

Ian and Sean also managed to join us, taking time out from the dgit sprint that had been running since Friday [3].

For most of the afternoon we had the inside of the pub all to ourselves.   

This was a friendly and low key event as we decompressed from the previous day’s activities, but we were also joined many other DDs, users and supporters, mostly hanging out in the garden but I confess to staying most of the time indoors in the shade and with a breeze through the pub…

Laptops are still needed even at a party

The start of the next project…

 

[0]    http://www.chiark.greenend.org.uk/pipermail/debian-uk/2019-June/000481.html
       https://wiki.debian.org/ReleasePartyBuster/UK/Cambridge

[1]   https://www.raspberrypi.org/

[2]    https://www.mythic-beasts.com/

[3]    https://wiki.debian.org/Sprints/2019/DgitDebrebase

Matthew Garrett: Creating hardware where no hardware exists

8 July, 2019 - 02:46
The laptop industry was still in its infancy back in 1990, but it still faced a core problem that we do today - power and thermal management are hard, but also critical to a good user experience (and potentially to the lifespan of the hardware). This is in the days where DOS and Windows had no memory protection, so handling these problems at the OS level would have been an invitation for someone to overwrite your management code and potentially kill your laptop. The safe option was pushing all of this out to an external management controller of some sort, but vendors in the 90s were the same as vendors now and would do basically anything to avoid having to drop an extra chip on the board. Thankfully(?), Intel had a solution.

The 386SL was released in October 1990 as a low-powered mobile-optimised version of the 386. Critically, it included a feature that let vendors ensure that their power management code could run without OS interference. A small window of RAM was hidden behind the VGA memory[1] and the CPU configured so that various events would cause the CPU to stop executing the OS and jump to this protected region. It could then do whatever power or thermal management tasks were necessary and return control to the OS, which would be none the wiser. Intel called this System Management Mode, and we've never really recovered.

Step forward to the late 90s. USB is now a thing, but even the operating systems that support USB usually don't in their installers (and plenty of operating systems still didn't have USB drivers). The industry needed a transition path, and System Management Mode was there for them. By configuring the chipset to generate a System Management Interrupt (or SMI) whenever the OS tried to access the PS/2 keyboard controller, the CPU could then trap into some SMM code that knew how to talk to USB, figure out what was going on with the USB keyboard, fake up the results and pass them back to the OS. As far as the OS was concerned, it was talking to a normal keyboard controller - but in reality, the "hardware" it was talking to was entirely implemented in software on the CPU.

Since then we've seen even more stuff get crammed into SMM, which is annoying because in general it's much harder for an OS to do interesting things with hardware if the CPU occasionally stops in order to run invisible code to touch hardware resources you were planning on using, and that's even ignoring the fact that operating systems in general don't really appreciate the entire world stopping and then restarting some time later without any notification. So, overall, SMM is a pain for OS vendors.

Change of topic. When Apple moved to x86 CPUs in the mid 2000s, they faced a problem. Their hardware was basically now just a PC, and that meant people were going to try to run their OS on random PC hardware. For various reasons this was unappealing, and so Apple took advantage of the one significant difference between their platforms and generic PCs. x86 Macs have a component called the System Management Controller that (ironically) seems to do a bunch of the stuff that the 386SL was designed to do on the CPU. It runs the fans, it reports hardware information, it controls the keyboard backlight, it does all kinds of things. So Apple embedded a string in the SMC, and the OS tries to read it on boot. If it fails, so does boot[2]. Qemu has a driver that emulates enough of the SMC that you can provide that string on the command line and boot OS X in qemu, something that's documented further here.

What does this have to do with SMM? It turns out that you can configure x86 chipsets to trap into SMM on arbitrary IO port ranges, and older Macs had SMCs in IO port space[3]. After some fighting with Intel documentation[4] I had Coreboot's SMI handler responding to writes to an arbitrary IO port range. With some more fighting I was able to fake up responses to reads as well. And then I took qemu's SMC emulation driver and merged it into Coreboot's SMM code. Now, accesses to the IO port range that the SMC occupies on real hardware generate SMIs, trap into SMM on the CPU, run the emulation code, handle writes, fake up responses to reads and return control to the OS. From the OS's perspective, this is entirely invisible[5]. We've created hardware where none existed.

The tree where I'm working on this is here, and I'll see if it's possible to clean this up in a reasonable way to get it merged into mainline Coreboot. Note that this only handles the SMC - actually booting OS X involves a lot more, but that's something for another time.

[1] If the OS attempts to access this range, the chipset directs it to the video card instead of to actual RAM.
[2] It's actually more complicated than that - see here for more.
[3] IO port space is a weird x86 feature where there's an entire separate IO bus that isn't part of the memory map and which requires different instructions to access. It's low performance but also extremely simple, so hardware that has no performance requirements is often implemented using it.
[4] Some current Intel hardware has two sets of registers defined for setting up which IO ports should trap into SMM. I can't find anything that documents what the relationship between them is, but if you program the obvious ones nothing happens and if you program the ones that are hidden in the section about LPC decoding ranges things suddenly start working.
[5] Eh technically a sufficiently enthusiastic OS could notice that the time it took for the access to occur didn't match what it should on real hardware, or could look at the CPU's count of the number of SMIs that have occurred and correlate that with accesses, but good enough

comments

Debian GSoC Kotlin project blog: Week 4 & 5 Update

7 July, 2019 - 18:51
Finihsed downgrading the project to be buildable by gradle 4.4.1

I have finished downgrading the project to be buildable using gradle 4.4.1. The project still needed a part of gradle 4.8 that I have successfully patched into sid gradle. here is the link to the changes that I have made.

Now we are officially done with making the project buidl with our gradle; so we can now go on ahead and finally start mapping out and packaging the dependencies.

Packaging dependencies for Kotlin-1.3.30

I split this task into two sub tasks that can be done independently. The 2 subtaks are as follows:
part 1: make the entire project build successfully without :buildSrc:prepare-deps:intellij-sdk:build part1.1:package these dependencies part 2: package the dependencies in :buildSrc:prepare-deps:intellij-sdk:build ; i.e try to recreate whatever is in it. the task has been split into this exact model because this folder has a variety of jars that the project uses and we ll have to minimize it and pacakge the needed jars from it. Also the project uses other plugins and jars other than this one main folder which we can simultaneously map and package out.

I now have successfully mapped out the dependencies need by part 1: all that remains is to package em. I have copied the dependenceis from the original cache(one created when I build the project using ./gradlew -Pteamcity=true dist) to /usr/share/maven-repo so some of these dependencies still need to have their dependencies clearly defined as in which of their dependencies we can ommit and which we need. I have marked such dependencies with a "". So here are the dependencies:
jengeleman:shadow:4.0.3 --> https://github.com/johnrengelman/shadow trove4j 1.x -> https://github.com/JetBrains/intellij-deps-trove4j proguard:6.0.3 in jdk8
io.javaslang:2.0.6 jline 3.0.3 com.jcabi:jcabi-aether:1.0 org.sonatype.aether:aether-api:1.13.1

!!NOTE:Please note that I might have missed out some; I ll add em to the list once I get em mapped out proper!!

So if any of you kind souls wanna help me out please kindly take on any of these and pacakge em.

!!NOTE-ping me if you want to build kotlin in your system and are stuck!!

Here is a link to the work I have done so far. You can find me as m36 or m36[m] on #debian-mobile and #debian-java in OFTC.

I ll try to maintain this blog and post the major updates weekly.

Eriberto Mota: Debian: repository changed its ‘Suite’ value from ‘testing’ to ‘stable’

7 July, 2019 - 10:44

Debian 10 (Buster) was released two hours ago. \o/

When using ‘apt-get update’, I can see the message:

E: Repository 'http://security.debian.org/debian-security buster/updates InRelease' changed its 'Suite' value from 'testing' to 'stable'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.

 

Solution:

# apt-get --allow-releaseinfo-change update

 

Enjoy!

Bits from Debian: Debian 10 "buster" has been released!

7 July, 2019 - 08:25

You've always dreamt of a faithful pet? He is here, and his name is Buster! We're happy to announce the release of Debian 10, codenamed buster.

Want to install it? Choose your favourite installation media and read the installation manual. You can also use an official cloud image directly on your cloud provider, or try Debian prior to installing it using our "live" images.

Already a happy Debian user and you only want to upgrade? You can easily upgrade from your current Debian 9 "stretch" installation; please read the release notes.

Do you want to celebrate the release? We provide some buster artwork that you can share or use as base for your own creations. Follow the conversation about buster in social media via the #ReleasingDebianBuster and #Debian10Buster hashtags or join an in-person or online Release Party!

Andy Simpkins: Debian Buster Release

7 July, 2019 - 07:30

I spent all day smoke testing the install images for yesterday’s (this mornings – gee just after midnight local so we still had 11 hours to spare) Debian GNU/Linux 10.0.0 “Buster” release.

This year we had our “Biglyest test matrix ever”[0], 111 tests were completed at the point the release images were signed and the ftp team pushed the button.  Although more tests were reported to have taken place in IRC we had a total of 9 people completing tests called for in the wiki test matrix.

We also had a large number of people in irc during the image test phase of the release – peeking at 117…

Steve kindly hosted his front room a few of us – using a local mirror[1] of the images so our VM tests have the image available as an NFS share really speeds things up!  Between the 4 of us here in Cambridge we were testing on a total of 14 different machines, mainly AMD64, a couple of “only i386 capable laptops” and 3 entirely different ARM64 machines! (Mustang, Synquacer, & MacchiatoBin).

Room heaters on a hot day – photo RandomBird

In the photo are Sledge, Codehelp, Isy and myself…[2]

A Special thanks should also be given to Schweer who spent time testing the Debian Edu images and to Zdykstra for testing on ppc64el [2].

 

Finally sledge posted a link of the bandwidth utilization on the distribution network servers last week it looked like this:

Debian primary distribution network last week

That is seeing 500MBytes/s daily peek in traffic.

Pushing Buster today the graph looks like:

Guess when Buster release went live?

1.5GByte/Second – that is some network load :-)

 

Anyway – Time to head home.  I have a release party to attend later this afternoon and would like *some* sleep first.

 

 

[0] https://wiki.debian.org/Teams/DebianCD/ReleaseTesting/Buster_r0

[1] Local mirror is another ARM Synquacer ‘Server’ – 24 core Cortex A53  (A53 series is targeted at Mobile Devices)

[2] irc nicks have been used to protect the identity of the guilty :-)

François Marier: SIP Encryption on VoIP.ms

7 July, 2019 - 06:00

My VoIP provider recently added support for TLS/SRTP-based call encryption. Here's what I did to enable this feature on my Asterisk server.

First of all, I changed the registration line in /etc/asterisk/sip.conf to use the "tls" scheme:

[general]
register => tls://mydid:mypassword@servername.voip.ms

then I enabled incoming TCP connections:

tcpenable=yes

and TLS:

tlsenable=yes
tlscapath=/etc/ssl/certs/

Finally, I changed my provider entry in the same file to:

[voipms]
type=friend
host=servername.voip.ms
secret=mypassword
username=mydid
context=from-voipms
allow=ulaw
allow=g729
insecure=port,invite
transport=tls
encryption=yes

(Note the last two lines.)

The dialplan didn't change and so I still have the following in /etc/asterisk/extensions.conf:

[pstn-voipms]
exten => _1NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551234567>)
exten => _1NXXNXXXXXX,n,Dial(SIP/voipms/${EXTEN})
exten => _1NXXNXXXXXX,n,Hangup()
exten => _NXXNXXXXXX,1,Set(CALLERID(all)=Francois Marier <5551234567>)
exten => _NXXNXXXXXX,n,Dial(SIP/voipms/1${EXTEN})
exten => _NXXNXXXXXX,n,Hangup()
exten => _011X.,1,Set(CALLERID(all)=Francois Marier <5551234567>)
exten => _011X.,n,Authenticate(1234) ; require password for international calls
exten => _011X.,n,Dial(SIP/voipms/${EXTEN})
exten => _011X.,n,Hangup(16)
Server certificate

The only thing I still need to fix is to make this error message go away in my logs:

asterisk[8691]: ERROR[8691]: tcptls.c:966 in __ssl_setup: TLS/SSL error loading cert file. <asterisk.pem>

It appears to be related to the fact that I didn't set tlscertfile in /etc/asterisk/sip.conf and that it's using its default value of asterisk.pem, a non-existent file.

Since my Asterisk server is only acting as a TLS client, and not a TLS server, there's probably no harm in not having a certificate. That said, it looks pretty easy to use a Let's Encrypt cert with Asterisk.

Jonathan Wiltshire: Daisy and George Help Debian

7 July, 2019 - 02:33

Daisy and George have decided to get stuck into testing images for #ReleasingDebianBuster.

George is driving the keyboard while Daisy takes notes about their test results.

This test looks like a success. Next!

Jonathan Wiltshire: Testing in Teams

6 July, 2019 - 22:46

The Debian CD images are subjected to a battery of tests before release, even more so when the release is a major new version. Debian has volunteers all over the world testing images as they come off the production line, but it can be a lonely task.

Getting together in a group and having a bit of competitive fun always makes the day go faster:

Debian CD testers in Cambridge, GB (photo: Jo McIntyre)

And, of course, they’re a valuable introduction to the life-cycle of a Debian release for future Debian Developers.

Niels Thykier: A decline in the use of hints in the release team

6 July, 2019 - 17:16

While we working the release, I had a look at how many hints we have deployed during buster like I did for wheezy a few years back.  As it seemed we were using a lot fewer hints than previously and I decided to take a detour into “stats-land”.

When I surfaced from “stats-land”, I confirmed that we have a clear decline in hints in the past two releases[1].

wheezy: 3301
jessie: 3699 (+398)
stretch: 2408 (-1291)
buster: 1478 (-930)

While it is certainly interesting, the number of hints on its own is not really an indicator of how much work we put into the release.  Notably, it says very little about the time spent on evaluating unblock requests before adding the hint.

 

[1]

Disclaimer: This are very rough estimate based on same method as from the previous blog post using entire complete months as smallest time unit. It is apparently not very accurate either.  The observant reader will note that the number for wheezy does not match the number I posted years ago (3254 vs 3301).  I am not entirely sure what causes the difference as I am using the same regex for wheezy and mtime for the files look unchanged.

 

Jonathan Wiltshire: What to expect on buster release day

6 July, 2019 - 14:19

The ‘buster’ release day is today! This is mostly a re-hash of previous checklists, since we’ve done this a few times now and we have a pretty good rhythm.

There have been some preparations going on in advance:

  1. Last week we imposed a “quiet period” on migrations. That’s about as frozen as we can get; it means that even RC bugs need to be exceptional if they aren’t to be deferred to the first point release. Only late-breaking documentation (like the install guide) was accepted.
  2. The security team opened buster-updates for business and carried out a test upload
  3. The debian-installer team made a final release.
  4. Final debtags data was updated.
  5. Yesterday the testing migration script britney and other automatic maintenance scripts that the release team run were disabled for the duration.
  6. We made final preparations of things that can be done in advance, such as drafting the publicity announcements. These have to be done in advance so translators get chance to do their work overnight (translations are starting to arrive right now!).

The following checklist makes the release actually happen:

  1. Once dinstall is completed at 07:52, archive maintenance is suspended – the FTP masters will do manual work for now.
  2. Very large quantities of coffee will be prepared all across Europe.
  3. Release managers carry out consistency checks of the buster index files, and confirm to FTP masters that there are no last-minute changes to be made. RMs get a break to make more coffee.
  4. While they’re away FTP masters begin the process of relabelling stretch as oldstable and buster as stable. If an installer needs to be, er, installed as well, that happens at this point. Old builds of the installer are removed.
  5. A new suite for bullseye (Debian 10) is initialised and populated, and labelled testing.
  6. Release managers check that the newly-generated suite index files look correct and consistent with the checks made earlier in the day. Everything is signed off – both in logistical and cryptographic terms.
  7. FTP masters trigger a push of all the changes to the CD-building mirror so that production of images can begin. As each image is completed, several volunteers download and test it in as many ways as they can dream up (booting and choosing different paths through the installer to check integrity).
  8. Finally a full mirror push is triggered by FTP masters, and the finished CD images are published.
  9. Announcements are sent by the publicity team to various places, and by the release team to the developers at large.
  10. Archive maintenance scripts are re-enabled.
  11. The release team take a break for a couple of weeks before getting back into the next cycle.

During the day much of the coordination happens in the #debian-release, #debian-ftp and #debian-cd IRC channels. You’re welcome to follow along if you’re interested in the process, although we ask that you are read-only while people are still concentrating (during the Squeeze release, a steady stream of people turned up to say “congratulations!” at the most critical junctures; it’s not particularly helpful while the process is still going on). The publicity team will be tweeting and denting progress as it happens, so that makes a good overview too.

If everything goes to plan, enjoy the parties!

(Disclaimer: inaccuracies are possible since so many people are involved and there’s a lot to happen in each step; all errors and omissions are entirely mine.)

Mike Gabriel: My Work on Debian LTS/ELTS (June 2019)

6 July, 2019 - 04:02

In June 2019, I did not at all reach my goal of LTS/ELTS hours, unfortunately. (At this point, I could come up with a long story about our dog'ish family member and the infection diseases he got, the vet visits we did and the daily care and attention he needed, but I won't...).

I have worked on the Debian LTS project for 9,75 hours (of 17 hours planned) and on the Debian ELTS project just for 1 hour (of 12 hours planned) as a paid contributor.

LTS Work
  • LTS: Setup physical box running Debian jessie (for qemu testing)
  • LTS: Bug hunting mupdf regarding my CVE-2018-5686 patch backport
  • LTS: Upload to jessie-security: mupdf (DLA-1838-1), 3 CVEs [1]
  • LTS: Glib2.0: request CVE Id (CVE-2019-13012) + email communication with upstream [2] (minor issue for Glib2.0 << 2.60)
  • LTS: cfengine3: triage CVE-2019-9929, email communication with upstream (helping out security team) [3]
ELTS Work
  • Upload to wheezy-lts: expat (ELA 136-1), 1 CVE [4]
References

Thorsten Alteholz: My Debian Activities in June 2019

6 July, 2019 - 02:08

FTP master

As you might have noticed as well, this month has been a month with the highest average temperature of all June so far. So I spent more time in the lake than in NEW. I only accepted 12 packages and rejected 1 upload. The rest of the team probably did the same because the overall number of packages that got accepted was only 22. Let’s see whether July will be the same …

Debian LTS

This was my sixtieth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

This month my all in all workload has been 17h. During that time I did LTS uploads or prepared security uploads of:

  • [DLA 1830-1] znc security update for one CVE
  • [DLA 1833-1] bzip2 security update for two CVEs
  • [DLA 1841-1] gpac security update for three CVEs

I also prepared bzip2 debdiffs for Buster and Stretch and sent them to the maintainer and security team.
Further I created new packages for testing the patches of bind9 and wpa. I would be more confident to upload those if more people could give it a try. Especially after the python issue, I would really like to have some more people to do smoke tests …

Last but not least I did some days of frontdesk duties.

Debian ELTS

This month was the thirteenth ELTS month.

During my allocated time I uploaded:

  • ELA-132-1 of bzip2 for two CVEs
  • ELA-138-1 of ntfs-3g for one CVE

As like LTS, I am a bit hesitant to upload bind9

I also did some days of frontdesk duties.

Other stuff

As already written above, I did not do much work in front of a computer, so there is nothing to report here.
Ok, maybe I can mention this email here instead of the LTS section above. This is a script to obtain the correct build order of Go packages in case of security patches. As well as mentioned in the other paragraphs I would like more people to have a look at it, but please be kind :-).

Reproducible Builds: Reproducible Builds in June 2019

5 July, 2019 - 20:58

Welcome to the June 2019 report from the Reproducible Builds project! In our reports we outline the most important things that we have been up to over the past month.

In order that everyone knows what this is about, whilst anyone can inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries. The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

In June’s report, we will cover:

  • Media coverageLego bricks, pizza and… Reproducible Builds‽
  • Upstream newsIs Trusting Trust close to a ‘rebuttal’?
  • EventsWhat happened at MiniDebConf Hamburg, the OpenWrt Summit, etc.
  • Software developmentPatches patches patches, etc.
  • Misc newsFrom our mailing list…
  • Getting in touchand how to contribute.
Media coverage
  • The Prototype Fund, an initiative to “aid software developers, hackers and creatives in furthering their ideas from concept to demo” produced a video featuring Holger Levsen explaining Reproducible Builds… using Lego bricks and pizza!

One key motivation for reproducible builds is to enable peak efficiency for the build caches used in modern build systems.

Upstream news


Events

There were a number of events that included or incorporated members of the Reproducible Builds community this month. If you know of any others, please do get in touch. In addition, a number of members of the Reproducible Builds project will be at DebConf 2019 in Curitiba, Brazil and will present on the status of their work.

MiniDebConf Hamburg 2019

Holger Levsen, Jelle van der Waa, kpcyrd and Alexander Couzens attended MiniDebConf Hamburg 2019 and worked on Reproducible Builds. As part of this, Holger gave a status update on the Project with a talk entitled Reproducible Builds aiming for bullseye, referring to the next Debian release name:


Jelle van der Waa kindly gifted Holger with a Reproducible Builds display:

In addition, Lukas Puehringer gave a talk titled Building reproducible builds into apt with in-toto:

As part of various hacking sessions:

  • Jelle van der Waa:

    • Improved the reproducible_json.py script to generate distribution-specific JSON, leading to the availability of an ArchLinux JSON file.
    • Investigated why the Arch Linux kernel package is not reproducible, finding out that KBUILD_BUILD_HOST and KGBUILD_BUILD_TIMESTAMP should be set. The enabling of CONFIG_MODULE_SIG_ALL causes the kernel modules to be signed with a (non-deterministic) build-time key if none is provided, leading to unreproducibility.
    • keyutils was fixed with respect to it embedding the build date in its binary. []
    • nspr was made reproducible in Arch Linux. []
  • kpcyrd:
    • Created various Jenkins jobs to generate Alpine build chroots, schedule new packages and to ultimately build them. [][][]
    • Created an Alpine reproducible testing overview page.
    • Provided a proof of concept SOURCE_DATE_EPOCH patch for abuild to fix timestamp issues in Alpine packages. []
  • Alexander Couzens:
    • Rewrote the database interaction routines for OpenWrt.
    • Migrated the OpenWrt package parser to use Python 3.x as Python 2.x will be reaching end-of-life at the end of this year.
    • Setup a test environment using a new README.development file.

Holger Levsen was on-hand to review and merge all the above commits, providing support and insight into the codebase. He additionally split out a README.development from the regular, more-generic README file.

OpenWrt summit

The OpenWrt project is a Linux operating system targeting embedded devices, particularly wireless network routers. In June, they hosted a summit that took place from 10th to 12th of the month.

Here, Holger participated in the discussions regarding .buildinfo build-attestation documents. As a result of this, Paul Spooren (aparcar) made a pull request to introduce/create a feeds.buildinfo (etc) for reproducibility in OpenWrt.

Software development buildinfo.debian.net

Chris Lamb spent significant time working on buildinfo.debian.net, his experiment into how to process, store and distribute .buildinfo files after the Debian archive software has processed them. This included:

  • Started making the move to Python 3.x (and Django 2.x) [][][][][][][] additionally performing a large number of adjacent cleanups including dropping the authentication framework [], fixing a number of flake8 warnings [], adding a setup.cfg to silence some warnings [], moving to __str__ and str.format(...) over %-style interpolation and u"Unicode" strings [], etc.

  • Added a number of (as-yet unreleased…) features, including caching the expensive landing page queries. []

  • Took the opportunity to start migrating the hosting from its current GitHub home to a more-centralised repository on salsa.debian.org, moving from the Travis to the GitLab continuous integration platform, updating the URL to the source in the footer [] and many other related changes [].

  • Applied the Black “uncompromising code formatter” to the codebase. []

Project website

There was a significant amount of effort on our website this month.

  • Chris Lamb:

    • Moved the remaining site to the newer website design. This was a long-outstanding task (#2) and required a huge number of changes, including moving all the event and documentation pages to the new design [] and migrating/merging the old _layouts/page.html into the new design [] too. This could then allow for many cleanups including moving/deleting files into cleaner directories, dropping a bunch of example layouts [] and dropping the old “home” layout. []

    • Added reports to the homepage. (#16)

    • Re-ordered and merged various top-level sections of the site to make the page easier to parse/navigate [][] and updated the documentation for SOURCE_DATE_EPOCH to clarify that the alternative -r call to date(1) is for compatibility with BSD variants of UNIX [].

    • Made a large number of visual fixups, particularly to accommodate the principles of responsive web design. [][][][][]

    • Updated the lint functionality of the build system to check for URIs that are not using /foo/-style relative URLs. []

  • Jelle van der Waa updated the Events page to correct invalid Markdown [] and fixed a typo of “distribution” on a previous event page [].

  • Thomas Vincent added a huge number of videos and slides to the Resources page [][][][][][] etc. as well as added a button to link to subtitles [] and fixing a bug when displaying metadata links [].

In addition, Atharva Lele added the Buildroot embedded Linux project to the “Who’s involved” page. []

Test framework

We operate a comprehensive Jenkins-based testing framework that powers tests.reproducible-builds.org. The following changes were done in the last month:

  • Alexander Couzens (OpenWrt):
  • Holger Levsen:
    • Show Alpine-related jobs on the job health page. []
    • Alpine needs the jq command-line JSON processor for the new scheduler. []
    • Start a dedicated README.development file. []
    • Add support for some nodes running Debian buster already. []
  • Jelle van der Waa:
    • Change Arch Linux and Alpine BLACKLIST status to blacklist [] and GOOD to reproducible [] respectfully.
    • Add a Jenkins job to generate Arch Linux HTML pages. []
    • Fix the Arch Linux suites in the reproducible.ini file. []
    • Add an Arch JSON export Jenkins job. []
    • Create per-distribution reproducible JSON files. []
  • kpcyrd (Alpine):

    • Start adding an Alpine theme. []
    • Add an Alpine website. [][][][]
    • Add #alpine-reproducible to the KGB chat bot. []
    • Use the apk version instead of vercmp. []
    • Install/configure various parts of the chroot including passing in Git options [], adding the abuild group onto more servers [][], installing GnuPG []
    • Build packages using its own scheduler. [] [][]
    • Misc maintenance and fixups. [][]
  • Mattia Rizzolo:
    • Adjust the setup_pbuilder script to use [check-valid-until=no] instead of Acquire::Check-Valid-Until (re. (#926242)). []
Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Distribution work

In Debian, 39 reviews of packages were added, 3 were updated and 8 were removed this month, adding to our knowledge about identified issues.

Chris Lamb also did more work testing of the reproducibility status of Debian Installer images. In particular, he was working around and patching an issue stemming from us testing builds far into the “future”. (#926242)

In addition, following discussions at MiniDebConf Hamburg, Ivo De Decker reviewed the situation around Debian bug #869184 again (“dpkg: source uploads including _amd64.buildinfo cause problems”) and updated the bug with some recommendations for the next Debian release cycle.

Bernhard M. Wiedemann posted his monthly Reproducible Builds status update for the openSUSE distribution.

Other tools

In diffoscope (our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues) Chris Lamb documented that run_diffoscope should not be considered a stable API [] and adjusted the configuration to build our Docker image from the current Git checkout, not the Debian archive []

Lastly, Chris Lamb added support for the clamping of tIME chunks in .png files [] to strip-nondeterminism, our tool to remove specific non-deterministic results from a completed build.

Misc news

On our mailing list this month Lars Wirzenius continued conversation regarding various questions about reproducible builds and their bearing on building a distributed continuous integration system which received many replies (thread index for May & June). In addition, Sebastian Huber asked whether anyone has attempted a reproducible build of a GCC compiler itself.

If you are interested in contributing the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:


This month’s report was written by Alexander Borkowski, Arnout Engelen, Bernhard M. Wiedemann, Chris Lamb, heinrich5991, Holger Levsen, Jelle van der Waa, kpcyrd & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Patrick Matthäi: Maintainance of GeoIP legacy databases

5 July, 2019 - 20:19

Since 9 months now Maxmind is not providing the CSV sources for their legacy database format, but only for their new GeoLite2 database. That is legitimate in my opinion, because the API is quite old and software projects should move to the new format, but mostly all (IMHO) important software projects still only support the old API.. :-(

So I have decided to spend again some more work in my geoip and geoip-database packages and I can say, that I will upload after the Buster release a new geoip source package, which also provides the converter I took from here:
https://github.com/mschmitt/GeoLite2xtables/

Using this converter (and some more magic etc) I am now able to build the country v4+v6 legacy edition by using the GeoLite2 CSV database source :-)

So testing will be welcome and if everything is fine buster and stretch will get backports from this work in the future.

But I had to drop now the geoip-database-extra package, which includes also the AS and City (v4) database. I didn’t find a way to convert the sources and IMO they are not so important.

Bits from Debian: Upcoming Debian 10 "Buster"!

5 July, 2019 - 13:00

The Debian Release Team in coordination with several other teams are preparing the last bits needed for releasing Debian 10 "Buster" on Saturday 6 July 2019. Please, be patient! Lots of steps are involved and some of them take some time, such as building the images, propagating the release through the mirror network, and rebuilding the Debian website so that "stable" points to Debian 10.

If you are considering create some artwork on the occasion of Buster Release, feel free to send us links to your creations to the (publicly archived) debian-publicity mailing list, so that we can disseminate them throughout our community.

Follow the live coverage of the release on https://micronews.debian.org or the @debian profile in your favorite social network! We'll spread the word about what's new in this version of Debian 10, how the release process is progressing during the weekend and facts about Debian and the wide community of volunteer contributors that make it possible.

If you want to celebrate the release of Debian 10 Buster, join one of the many release parties or consider organizing one in your city! Celebration will also happen online on the Debian Party Line.

Pages

Creative Commons License ลิขสิทธิ์ของบทความเป็นของเจ้าของบทความแต่ละชิ้น
ผลงานนี้ ใช้สัญญาอนุญาตของครีเอทีฟคอมมอนส์แบบ แสดงที่มา-อนุญาตแบบเดียวกัน 3.0 ที่ยังไม่ได้ปรับแก้