Planet Debian

Subscribe to Planet Debian feed
Planet Debian - http://planet.debian.org/
Updated: 11 months 2 weeks ago

Petter Reinholdtsen: First prototype ready making hardware easier to use in Debian

21 January, 2013 - 19:00

Early this month I set out to try to improve the Debian support for pluggable hardware devices. Now my prototype is working, and it is ready for a larger audience. To test it, fetch the source from the Debian Edu subversion repository, build and install the package. You might have to log out and in again activate the autostart script.

The design is simple:

  • Add desktop entry in /usr/share/autostart/ causing a program hw-support-handlerd to start when the user log in.
  • This program listen for kernel events about new hardware (directly from the kernel like udev does), not using HAL dbus events as I initially did.
  • When new hardware is inserted, look up the hardware modalias in the APT database, a database available via HTTP and a database available as part of the package.
  • If a package is mapped to the hardware in question, the package isn't installed yet and this is the first time the hardware was plugged in, show a desktop notification suggesting to install the package or packages.
  • If the user click on the 'install package now' button, ask aptdaemon via the PackageKit API to install the requrired package.
  • aptdaemon ask for root password or sudo password, and install the package while showing progress information in a window.

I still need to come up with a better name for the system. Here are some screen shots showing the prototype in action. First the notification, then the password request, and finally the request to approve all the dependencies. Sorry for the Norwegian Bokmål GUI.





The prototype still need to be improved with longer timeouts, but is already useful. The database of hardware to package mappings also need more work. It is currently compatible with the Ubuntu way of storing such information in the package control file, but could be changed to use other formats instead or in addition to the current method. I've dropped the use of discover for this mapping, as the modalias approach is more flexible and easier to use on Linux as long as the Linux kernel expose its modalias strings directly.

Update 2013-01-21 16:50: Due to popular demand, here is the command required to check out and build the source: Use 'svn checkout svn://svn.debian.org/debian-edu/trunk/src/hw-support-handler/; cd hw-support-handler; debuild'. If you lack debuild, install the devscripts package.

Update 2013-01-23 12:00: The project is now renamed to Isenkram and the source moved from the Debian Edu subversion repository to a Debian collab-maint git repository. See build instructions for details.

Raphael Geissert: January's Debian mirrors update

21 January, 2013 - 15:30
It's been slightly over a month since December's update to http.debian.net. Since then, Debian's mirrors network has grown by 6 more archive mirrors. Many thanks to the Debian sponsors running them!

There are now about 370 archive mirrors serving it over http, an increase of 40 (12%) since April last year. The number of backports mirrors is now at 82, and 25 for archive.debian.org.

On the http.debian.net front there haven't been many changes since last month. Some major changes are in the works, but they didn't make it into January's code update. There were, however, a few issues with one of the hosts during the first couple of days of January. Apologies for the inconveniences it may have caused.

A new version of ftpsync addressing some issues should hopefully be released some time next month. Stay tuned to the debian-mirrors mailing list for a call for testers and probably a survey for mirror administrators.

Russ Allbery: Have some clouds

21 January, 2013 - 14:11

It's been a long time since I've posted any photographs here (about 15 months, it looks like), so have some clouds.

This also serves as a test for whether the scripts I use to post photographs have been properly converted to Git. (And a test to see if the occasional amateur photograph eating up screen real estate will provoke annoyed mutters from the direction of Planet Debian, although so far no complaints over my even-longer book reviews.)

I'm still taking photographs occasionally, although not as much as I was at one point. Too many hobbies and not enough time. Part of why I've been slow in posting them is that I was bad about keeping notes about where I took them and bad about keeping up with sorting through them. So I have large collections of photographs from some days where I don't remember exactly where I was, and where I've not already singled out the ones that are worth sharing.

One of the things my hobbies always seem to generate is yet another pile of stuff that's not organized quite to my liking. Maybe some evening I'll feel inspired to do a bit more sorting.

This is a long weekend in the US for some of us, myself included, so I get another day in this weekend. Which is good, since I've only gotten to a fraction of what I wanted to get to, although at least I found several hours to play video games today.

I have finally converted PGP::Sign to Git, so I have no more personal Subversion repositories, but I forgot how much code I'd written to handle its build-time configuration. All of that desperately needs to be reworked, so it's going to take a bit longer than I expected before I can push out another release.

Hideki Yamane: digging CVEs

21 January, 2013 - 06:40
Sometimes I would check security-tracker to find problems that is already fixed but not tagged as so.

Today, I found problems that not in unstable package but not tagged as fixed for Eucalyptus, because it was removed from unstable once but re-introduced to archive again, then data in security-tracker was not updated as well. So update it (CVE-2010-3905 and CVE-2011-0730).

And also found vulnerablity in firebird2.5 was fixed in upstream svn, so pick it and made it as a patch, just report to BTS (for squeeze, too).

For isc-dhcp, it has a vulnerability from BIND9... I've been surprised but dhcp includes bind9 source since 4.2, so BIND9 vulnerability affects isc-dhcp. Pick a diff from bind9 for squeeze security update and made a patch (Bug#698597) .

Gregor Herrmann: RC bugs 2013/03

21 January, 2013 - 06:37

& again, I was not very active in my RCBW work this week. at least one bug fixed, & another one will be taken care of by the package maintainers.

  • #683312 – uif: "[PATCH] uif uses depricated position of ! to negate rules"
    ask submitter for further info, later upload with improved patch to DELAYED/5
  • #694015 – geda: "geda: copyright file missing after upgrade (policy 12.5)"
    next try to fix the directory -> symlink change, this time with a postinst script, upload to DELAYED/2, later cancelled after feedback from maintainers

Russ Allbery: Review: How I Proposed to My Wife

20 January, 2013 - 12:25

Review: How I Proposed to My Wife, by John Scalzi

Publisher: Subterranean Copyright: 2007 ISBN: 1-59606-480-3 Format: Kindle Pages: 25

How I Proposed to My Wife: An Alien Sex Story is really a novelette, which I would normally give a paragraph review in a larger collection. But it was independently published and is independently purchasable, so it gets its own page. I read this as part of the Subterranean Scalzi Super Bundle, which collects a bunch of Scalzi's independently-published short fiction. It was originally a chapbook that accompanied Subterranean #4.

Those familiar with Scalzi's other work should expect something more like Agent to the Stars than the Old Man's War series. The first-person protagonist is a new writer for New World Man — think the sort of glossy magazine you see in supermarket racks. Every issue, they do an alien story; this issue, they decide to do an article on alien courtship and, of course, assign the new writer to it.

Most of the story is a mildly entertaining series of arranged dates in which Charlie has somewhat awkward conversations with various aliens about how their date protocols work, and then ends up in situations that are sexual to the aliens but mostly just bizarre and embarassing. If you've read any other Scalzi humor, you know what to expect: good-natured, light, mild parody with a touch of slapstick.

There is, of course, a twist that gets the story to the point foreshadowed in the title. I didn't find it particularly believable, but you don't read this story for its complex and realistic world-building. The ending joke, which is back onto the firmer ground of purely human romantic conventions, is probably the funniest moment in the story.

If you like Scalzi's humor, you probably won't regret spending the $1 that the story currently costs on Amazon. (You can also still get it on a shareware basis direct from Scalzi, but given that the minimum useful donation is 30 cents, it's probably easier to just buy from Amazon unless you don't want Amazon's format or DRM.) On the other hand, you probably also won't regret forgetting this story exists and going on with the rest of your life.

Rating: 6 out of 10

Matthew Garrett: Upcoming conferences

20 January, 2013 - 12:02
Unfortunately I'm not going to make it to LCA this year, but there's still plenty of UEFI coverage - Dong Wei, Vice President of the UEFI forum, will be giving an introduction to UEFI, and James Bottomley, who's been working on the Linux Foundation's UEFI bootloader solution, will be talking about Secure Boot.

For those of you on this side of the Pacific (or who just can't get enough of listening to my melodious voice), I'll be keynoting the Southern California Linux Expo next month. This will be a discussion of the more social aspects of our work on Secure Boot over the past year and a half - an epic tail of astonishment, terror, politics, confusion and hard won victories, and the remaining issues that are yet unsolved. Clearly not to be missed. If California's too far for you, then come March I'll be at the North East Linux Fest. I'll be talking about how to make sure that your Linux distribution works with Secure Boot, and, assuming I've got the code finished by then, a little about how to break it.

comments

Steinar H. Gunderson: Movit: A project update

20 January, 2013 - 08:12

It's been two months since I announced Movit, my library for high-quality, high-performance video filters, to the world. Since then, a number of things have happened, so I thought an update might be worthwhile.

First and foremost, Movit now actually has real users interested in using it in real projects, which is what prompted me to start working on it again after a break for over a month. In particular, Dan Dennedy has expressed interest in integrating it with MLT, the library used by (among others) Kdenlive, and the work there is ongoing. (MLT was my primary desired user from the get-go, so this is about as good as it can get.) It's still a while until you can go to Kdenlive and insert Movit effects, but I'm eagerly awaiting the day. :-)

The MLT integration has presented Movit with a set of new challenges, although the main part of the complexity lies on the MLT side. Nevertheless, MLT's needs are now a significant driver for the choices I make for Movit's direction. There are still a bunch of things that need to be done before Movit can handle a significant chunk of an MLT pipeline; the aspect ratio handling is still crap, for instance, there's zero understanding of interlaced material, and there's not really a build system.

On the other hand, I've added comprehensive support for alpha. Even after making some key choices, like that premultiplied alpha is the only format used internally (which was not what I originally anticipated, but I'm pretty convinced by now is the only right choice, just like linear light is), some things have turned out amazingly hard to get right. How do you blur something with alpha? How do you do additive blending with alpha? How do you do unsharp mask with alpha? Heck, how do you even put two layers on top of each other when the result can have alpha? To a certain extent, I suppose I should look more towards what e.g. GIMP does rather than trying to reinvent the wheel, but I've ended up with some what I think is a quite clean, small set of choices that together make things surprisingly simple and look very good at the same time.

So, well, I think the project might actually get somewhere at some point. I still don't have the goal to cater to every possible use case out there (which is a balance I guess every library writer has to struggle with—if you don't count the libraries where you are your own only user, of course), which in particular means that the set of filters will probably always be pretty limited, but the quality is getting better and better every day. The division of labor is interesting, though; this is fundamentally a library about OpenGL filters, and there are over 10,000 lines of C++ (including headers, blank lines, tests, etc.) but only about 350 of GLSL! I hope that as I start adding new filters, this will change a bit, but it's interesting to note.

So, if you have a project that needs a bit more oomph or prettiness in its handling of video, Movit might slowly start getting ready for you. It still doesn't have a homepage, but at least I've added Gitweb to git.sesse.net, so now you can poke around without having to do a git clone yourself. :-)

Dmitrijs Ledkovs: #takethestage

20 January, 2013 - 00:16

#stagetakenI have now permanently settled in the UK.


<iframe allowfullscreen="allowfullscreen" frameborder="0" height="480" src="https://www.youtube-nocookie.com/embed/Ke1Ilm6joO0" width="853"></iframe>

Niels Thykier: Wheezy release progress (January)

19 January, 2013 - 20:14

In December, I wrote a post on the progress of the Wheezy release.  Back then, we had 348 RC bugs in testing and today there are about 249 left.  A bit of simple math put us at 99 RC bugs fixed since last and an average of about 2.4 RC bugs fixed per day (up from 1.8).  Assuming a constant rate, we would be able to release in about 100 days or 3 months (plus 1 week or two) from now.  If you want the release earlier than that, fix RC bugs even faster!

Though, the data we are looking might be inaccurate.  We filed a bug against UDD, which claims that #639407 affects Wheezy while it is in fact fixed as far as the BTS is concerned.  If UDD is only wrong in this direction, we can hope for a positive surprise when the UDD bug is fixed.  On the other hand, we may also be unpleasently surprised if not.

While talking about the UDD, I would like to mention a new feature its bug search script.  It is now possible to filter based on whether or not a package is unblocked.  If you take  this into account (and assume that all unblocked packages will migrate in a timely manner), our RC bug count drops to about 219.  It also gives us a much better view of how many packages that could actually use an unblock. which is now down to about 61.

In case you have been wondering what is up with the “removal of RC buggy leaf packages”.  We have still been looking at those, but for the last couple of times there have been at most one candidate for a given run (obviously not the same package each time). It felt a bit excessive to send an email to debian-devel for just one RC bug.


Petter Reinholdtsen: Thank you Thinkpad X41, for your long and trustworthy service

19 January, 2013 - 16:20

This Christmas my trusty old laptop died. It died quietly and suddenly in bed. With a quiet whimper, it went completely quiet and black. The power button was no longer able to turn it on. It was a IBM Thinkpad X41, and the best laptop I ever had. Better than both Thinkpads X30, X31, X40, X60, X61 and X61S. Far better than the Compaq I had before that. Now I need to find a replacement. To keep going during Christmas, I moved the one year old SSD disk to my old X40 where it fitted (only one I had left that could use it), but it is not a durable solution.

My laptop needs are fairly modest. This is my wishlist from when I got a new one more than 10 years ago. It still holds true.:)

  • Lightweight (around 1 kg) and small volume (preferably smaller than A4).
  • Robust, it will be in my backpack every day.
  • Three button mouse and a mouse pin instead of touch pad.
  • Long battery life time. Preferable a week.
  • Internal WIFI network card.
  • Internal Twisted Pair network card.
  • Some USB slots (2-3 is plenty)
  • Good keyboard - similar to the Thinkpad.
  • Video resolution at least 1024x768, with size around 12" (A4 paper size).
  • Hardware supported by Debian Stable, ie the default kernel and X.org packages.
  • Quiet, preferably fan free (or at least not using the fan most of the time).

You will notice that there are no RAM and CPU requirements in the list. The reason is simply that the specifications on laptops the last 10-15 years have been sufficient for my needs, and I have to look at other features to choose my laptop. But are there still made as robust laptops as my X41? The Thinkpad X60/X61 proved to be less robust, and Thinkpads seem to be heading in the wrong direction since Lenovo took over. But I've been told that X220 and X1 Carbon might still be useful.

Perhaps I should rethink my needs, and look for a pad with an external keyboard? I'll have to check the Linux Laptops site for well-supported laptops, or perhaps just buy one preinstalled from one of the vendors listed on the Linux Pre-loaded site.

Russ Allbery: distribrc 1.0

19 January, 2013 - 12:28

I was just talking to my dad about this script the other day, so now seems like as good of a time as any to finally stick it up with my other publicly released small scripts.

distribrc is a little Perl script I wrote a long time back that copies all my dot-files (everything from .Xresources and .screenrc to more obscure stuff like .gitconfig and .perlcriticrc) to all of my accounts. It supports a simple configuration file that specifies locations to copy files to, commands to do the copying, and which set of files to copy to each host, so it's survived my transition from Kerberos rcp to GSS-API authenticated ssh without needing a single change.

In fact, when I went to tweak the documentation for this public release, I discovered that it was at version 0.7 and the last commit was in May of 1997. So this is a script that I've been using, probably at least once a month, without a single change, for more than 15 years. I suppose you could call that stable. It at least seemed to warrant a 1.0 version number.

The Perl coding style is kind of awful, but hey. At some point I'll probably go through my various scripts and update coding style, and maybe figure out some test suite mechanism, but in the meantime someone else may find it useful. More thorough measures, like using Git to track one's home directory, are probably better, but this script has the advantage of being quite simple and making it very easy to customize exactly what you want to send to each host.

You can get it from my miscellaneous scripts page.

Dirk Eddelbuettel: Sing The Truth at Symphony Center

19 January, 2013 - 12:07
Just back in from an amazing concert at the Chicago Symphony Center. Vocalists Angelique Kidjo, Dianne Reeves and Lizz Wright supported by all-star band of Geri Allen on piano, Romero Lubambo on electric and acoustic guitar, James Genus on electric bass guitar, Munyungo Jackson on percussion and Terri Lyne Carrington on drums (and musical director). The concert alternates between solos and joint pieces with more joy and soul than I heard in a long time. Great evening.

Charles Plessy: Debian-Installer in a Cloud, part 2.

19 January, 2013 - 11:03

I finally manage to create an image of the Debian Installer automatically, with no key or password transiting on the Cloud or the network, and without using the ec2-api-tools, which are non-Free. Thanks to Eucalyptus for implementing the attribute InstanceInitiatedShutdownBehavior in euca2ools 2.1.2.

In brief, I start a micro-instance with an additional volume, and I pass it a script for cloud-init, which will download the Installer on the volume and shut down the instance. Then, I register the volume as a machine image.

Those machine images of the Installer are to be preseeded via instance metadata to install Debian on another volume. I am still struggling with the pre-configuration. More details will follow in the next article. In the meantime, here are the commands I a using.

On the local computer, start an instance on the Cloud. Until #696595 is fixed and cloud-init is installed by default, I use a Ubuntu image.

HELPER_AMI=ami-7609bb77 # Ubuntu 12.04 LTS Precise amd64 EBS (Asia North-East)

Start the instance with an extra volume of 1 GiB, which will persist after termination (/dev/sdb=:1:false). Pass the script debigen-install-installer-cloud (see below).

HELPER_INSTANCE=$( euca-run-instances \
        --instance-initiated-shutdown-behavior terminate \
        --instance-type t1.micro \
        --block-device-mapping /dev/sdb=:1:false \
        --user-data-file debigen-install-installer-cloud \
        $HELPER_AMI |
        tee /dev/stderr | grep INSTANCE | awk '{print $2}')

Wait that the boot finished.

while [ ! $(euca-describe-instances $HELPER_INSTANCE | grep INSTANCE | cut -f 6 | tee /dev/stderr) = "running" ]
    do sleep 30
done

Get the volume's identifier

TARGET_VOLUME=$( euca-describe-volumes |
        grep $HELPER_INSTANCE | grep '/dev/sdb' |
        tee /dev/stderr | awk '{print $2}')

Wait that the scripts shuts down the instance.

while [ ! $(euca-describe-instances $HELPER_INSTANCE | grep INSTANCE | cut -f 6 | tee /dev/stderr) = "terminated" ]
    do sleep 30
done

Snapshot the volume and register it as a machine image.

PV_KERNEL=aki-44992845 # Boot PV-Grub on hd0 (Asia North-East)

TARGET_SNAPSHOT=$( euca-create-snapshot $TARGET_VOLUME |
        tee /dev/stderr | awk '{print $2}')

while euca-describe-snapshots $TARGET_SNAPSHOT | grep -q pending ; do sleep 30 ; done

euca-register \
    --name test_di \
    --description test_of_debian_installer \
    --snapshot $TARGET_SNAPSHOT \
    --kernel $PV_KERNEL \
    --architecture x86_64

Here is the script called debigen-install-installer-cloud, used above.

#!/bin/sh -ex    
mke2fs -L debian-installer /dev/xvdb -F
mount LABEL=debian-installer /mnt/

cd /mnt    
ARCH=amd64
DIST=squeeze
DI_VERSION=20110106+squeeze4+b2
MIRROR=jp
BASEURL=http://ftp.$MIRROR.debian.org/debian/dists/$DIST/main/installer-$ARCH/$DI_VERSION/images/netboot/xen

wget $BASEURL/initrd.gz $BASEURL/vmlinuz    
mkdir -p boot/grub    
cat > boot/grub/menu.lst <<__END__
default 0
timeout 3

title  Debian Installer ($DI_VERSION $ARCH)
root   (hd0)
kernel /vmlinuz root=LABEL=debian-installer ro console=hvc0 auto=true priority=critical url=http://169.254.169.254/latest/user-data DEBIAN_FRONTEND=text
initrd /initrd.gz
__END__

sleep 30     
halt

Richard Hartmann: Release Critical Bug report for Week 03

19 January, 2013 - 04:56

The UDD bugs interface currently knows about the following release critical bugs:

  • In Total: 817
    • Affecting wheezy: 249 That's the number we need to get down to zero before the release. They can be split in two big categories:
      • Affecting wheezy and unstable: 165 Those need someone to find a fix, or to finish the work to upload a fix to unstable:
        • 39 bugs are tagged 'patch'. Please help by reviewing the patches, and (if you are a DD) by uploading them.
        • 11 bugs are marked as done, but still affect unstable. This can happen due to missing builds on some architectures, for example. Help investigate!
        • 117 bugs are neither tagged patch, nor marked done. Help make a first step towards resolution!
      • Affecting wheezy only: 84 Those are already fixed in unstable, but the fix still needs to migrate to wheezy. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.

How do we compare to the Squeeze release cycle?

Week Squeeze Wheezy Diff 43 284 (213+71) 468 (332+136) +184 (+119/+65) 44 261 (201+60) 408 (265+143) +147 (+64/+83) 45 261 (205+56) 425 (291+134) +164 (+86/+78) 46 271 (200+71) 401 (258+143) +130 (+58/+72) 47 283 (209+74) 366 (221+145) +83 (+12/+71) 48 256 (177+79) 378 (230+148) +122 (+53/+69) 49 256 (180+76) 360 (216+155) +104 (+36/+79) 50 204 (148+56) 339 (195+144) +135 (+47/+90) 51 178 (124+54) 323 (190+133) +145 (+66/+79) 52 115 (78+37) 289 (190+99) +174 (+112/+62) 1 93 (60+33) 287 (171+116) +194 (111/+83) 2 82 (46+36) 271 (162+109) +189 (+116/+73) 3 25 (15+10) 249 (165+84) +224 (150/+74) 4 14 (8+6) 5 2 (0+2) 6 release!

Graphical overview of bug stats thanks to azhag:

Russell Coker: Phone Calls and Other Distractions

18 January, 2013 - 20:15

Harald Welte has written about the distraction of phone calls and how it impacts engineering work [1]. He asks why people feel that they are entitled to interrupt him given the cost to his work.

Some years ago while working as a programmer I was discussing such things with a colleague who worked for the consulting part of the same company. He was really surprised when I told him that a phone call at the wrong time would cost me at least 30 minutes work and possibly an hour or more. His work was also quite technical and demanding but the difference between software development (where you need to think about a lot of program state to consider where the problem might be) and consulting (where you have to deal with a smaller number of configuration file options and sometimes getting debugging information to someone like me) is considerable. So the attitudes towards receiving calls also tends to be quite different.

Computer work requires more concentration, thought, and knowledge of system state than many (most?) career choices. If someone finds that an unexpected phone call costs them no more than a few minutes work then it’s quite reasonable of them to phone other people whenever they feel like it – generally by default people think that everyone else is just like them.

In terms of managing interruptions to my work, I generally encourage people to email me and that works reasonably well. So I don’t have too many problems with distracting phone calls. I used Jabber for a while a few years ago but I didn’t reinstall my Jabber server after it became corrupt because of the distraction. I believe that was due to using Jabber in the wrong way. I should have just started a Jabber client when I wasn’t doing anything important and then killed it when I started doing some serious coding. Having a Jabber message interrupt me when I’m watching a TED talk or reading blogs is no big deal. In fact I could tell everyone who has my phone number that if they see me on Jabber then they can just phone me if they wish while knowing that it won’t distract me from anything serious. I wonder if I could configure a Jabber client to only receive messages when a program such as mplayer is running.

I have configured my laptop and workstation to never alert me for new mail. If I’m not concentrating then I’ll be checking my email frequently and if I am concentrating I don’t want a distraction. I have configured my phone to give one brief vibration when it gets mail and not make any sound, I will only notice that if I’m not concentrating on anything. It’s a standard Android feature to associate ring tones with phone numbers, it’s a pity that the K9 MUA doesn’t allow associating email addresses with notifications. There are some people who’s email could usefully trigger an audible alert. There is an K9 feature request from 2009 to allow notifications only when the IMAP flag “Flagged” is set which would allow the mail server to determine which users are important, but there’s no sign that it will be implemented soon.

I’ve started playing with Google+ recently due to Ingress team interaction being managed through it. Google+ seems quite poor in this regard, it defaults to making a ring tone for lots of different events. Turning that off is easy enough but getting notifications only about things that are important to me seems impossible. I would like to get an audible alert when someone makes a Google+ post with an Ingress code (because they expire quickly and because they only seem to be posted at times when I’m not busy) but not get audible alerts about anything else. I’m sure that most people who use Google+ would like to have different notifications for various types of event. But the Android client has options for whether there should be vibration and/or noise and for which events get the notifications. No options for different notifications for different events and for treating some community posts differently from others.

It seems that the default settings for most programs suit people who never need to spend much time concentrating on a task. It also seems that most programs don’t offer configuration options that suit the needs of people who do concentrate a lot but who also sometimes receive important phone calls and email. It’s ironic that so many applications are designed in the least optimal way for the type of people who develop applications. The Google+ developers have an excuse as doing what I desire would be quite complex. But there are other programs which should deal with such things in a user friendly manner.

Related posts:

  1. Globalisation and Phone Calls I just watched an interesting TED talk by Pankaj Ghemawat...
  2. A Mobile Phone for Sysadmin Use My telco Three have just offered me a deal on...
  3. Health and Status Monitoring via Smart Phone Health Monitoring Eric Topol gave an interesting TED talk about...

Michal &#268;iha&#345;: Roadmap for Weblate 1.4 and 1.5

18 January, 2013 - 19:00

As usual, I've changed my plans for Weblate 1.4. Simply before I got to implementing features I wanted to have there, I've already implemented bunch of other things, which are worth releasing.

So my plan is to release Weblate 1.4 next week with current feature set (you can check list of fixed issues) and focus on 1.5 development then.

Some major features which will be available in Weblate 1.4:

  • Added various configuration options to allow more customization.
  • Added sitemaps to allow easier access by crawlers.
  • Improved notification emails (added links and HTML version with colored diff).
  • Provide hints for production setup in admin interface.
  • Indicate failing checks or fuzzy strings in progressbars.
  • Support for per project ACL.

As a clear consequence, some of the big features were moved to 1.5, but I've added also some other things I'd like to implement, see 1.5 milestone on GitHub.

Filed under: English Phpmyadmin Suse Weblate | 0 comments | Flattr this!

Ian Campbell: Using Valgrind to Debug Xen Toolstacks

18 January, 2013 - 18:29
What is Valgrind?

Valgrind is a framework for building dynamic analysis tools. Several useful tools are included in the Valgrind distribution including tools to check dynamic memory usage (memcheck), a cache profiler (cachegrind), heap profiler (massif) and thread debugger (helgrind) among others. Valgrind also provides a framework which can be used to build other tools.

The Valgrind tool which I find most useful and the one which I have most experience with is memcheck. This tool can detect all manner of memory management problems, including use after free, using uninitialized data, memory leaks, double free. Between them these can result in savings of many hours of staring a core dumps and gdb backtraces.

How Does memcheck Work?

At its core Valgrind is a dynamic binary translation engine, which is used to instrument the code at runtime. In the case of memcheck this is used to track properties of the memory in a process, including aspects such as whether each bit (yes, bit) of memory has been initialized since it was allocated. It also tracks this information as data is loaded into registers, so it can know if a given register is currently tainted with uninitialized data or not. As well as instrumentation through binary translation Valgrind also includes instrumented versions of the C dynamic memory allocation functions which are used to track whether a each bit of memory is currently considered free or allocated, as well as tainting registers when they contain a pointer to memory which has been freed.

A large part of memcheck's functionality is built upon Valgrind's ability to determine when memory has been initialized. However although binary translation can be used to instrument when the application itself has initialized some memory it is not currently possible to instrument the behaviour of system calls, in other words Valgrind cannot automatically determine when memory has been initialised e.g. after a read(2) system call. For this reason Valgrind has baked in knowledge about the behaviour of system calls on various platforms.

For example lets consider the read(2) system call. The prototype of read(2) is:

    ssize_t read(int fd, void *buf, size_t count);

This system call reads up to count bytes from the file descriptor fd into the memory pointed to by buf and returns the number of bytes read. Here is the code within Valgrind which handles this system call (found in coregrind/m_syswrap/syswrap-generic.c):

PRE(sys_read)
{
   *flags |= SfMayBlock;
   PRINT("sys_read ( %ld, %#lx, %llu )", ARG1, ARG2, (ULong)ARG3);
   PRE_REG_READ3(ssize_t, "read",
                 unsigned int, fd, char *, buf, vki_size_t, count);

   if (!ML_(fd_allowed)(ARG1, "read", tid, False))
      SET_STATUS_Failure( VKI_EBADF );
   else
      PRE_MEM_WRITE( "read(buf)", ARG2, ARG3 );
}

POST(sys_read)
{
   vg_assert(SUCCESS);
   POST_MEM_WRITE( ARG2, RES );
}

These two functions are called before and after the system call respectively. The main piece of functionality is the calls to PRE_MEM_WRITE and POST_MEM_WRITE. In this case PRE_MEM_WRITE is used to check that the memory starting at ARG2 (recall that this is the supplied buffer) and extending for ARG3 bytes (the system calls count argument) consists of allocated memory. After the system call in complete the post function calls POST_MEM_WRITE to indicate to Valgrind that RES bytes of the buffer have know been initialized.

The above is a relatively simple case, but the above two hooks must be supplied for each system call on each platform which Valgrind wishes to support. One of the most problematic system calls is the ioctl(2) system call. The ioctl call takes a file descriptor, a request number and a pointer to a per-ioctl argument structure and implements per-device control requests. There are literally dozens of devices many with their own specific ioctls. For this reason a sizable portion of Valgrind code is dedicate to decoding the behaviour of the myriad of potential ioctls and providing pre- and post-call instrumentation for them.

Interacting with the Hypervisor from Userspace

Under Xen the toolstack is a normal userspace process running in the control domain (typically domain 0). As part of its operation the toolstack needs to communicate with the hypervisor and to do this it uses hypercalls. However a userspace process cannot simply make hypercalls on its own, instead it must request that the OS kernel make the hypercall for it. This prevents just any userspace process making a hypercall, since the kernel will first check the privilege of the process making the request.

Under Linux a userspace process makes hypercalls by using the ioctl(2) system call on a special "Privileged Command" (privcmd) device. In this case the toolstack uses the IOCTL_PRIVCMD_HYPERCALL ioctl request which takes a hypercall number and the hypercall arguments as its argument. Other OSes which can be used as a control domain posses similar interfaces.

Making memcheck Work With Xen Toolstacks?

As might be expected the majority of the work to allow Valgrind to work on Xen toolstack processes was teaching it about the IOCTL_PRIVCMD_HYPERCALL ioctl. Fortunately it was not necessary to teach Valgrind about every hypercall but only about those which are used by toolstacks. This is a smallish subset of all hypercalls including the domctl interfaces and a handful of others.

The initial batch of patches cover all of the system calls needed to start new domains, shut them down and migrate them using the XL toolstack. New hypercalls will be added as they are discovered to be missing.

Getting and Using This Functionality

The initial set of patches were accepted into Valgrind's Subversion repository trunk revision 13081 in October 2012 and are expected to be part of the 3.9.0 release.

Until 3.9.0 is released it will be necessary to get this functionality from SVN. Fortunately this is relatively simple and follows the usual autofoo dance. Although the usual caveats regarding running unreleased software apply.

  $ svn co svn://svn.valgrind.org/valgrind/trunk valgrind
  $ cd valgrind
  $ ./autogen.sh
  $ ./configure
  $ make
  $ sudo make install

Once Valgrind is built and installed simply call it passing the command to be debugged as an option:

  # valgrind xl list
  ==16186== Memcheck, a memory error detector
  ==16186== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
  ==16186== Using Valgrind-3.9.0.SVN and LibVEX; rerun with -h for copyright info
  ==16186== Command: xl list
  ==16186== 
  Name                                        ID   Mem VCPUs    State   Time(s)
  Domain-0                                     0   512     4     r-----     332.5
  ==16186== 
  ==16186== HEAP SUMMARY:
  ==16186==     in use at exit: 0 bytes in 0 blocks
  ==16186==   total heap usage: 16 allocs, 16 frees, 93,529 bytes allocated
  ==16186== 
  ==16186== All heap blocks were freed -- no leaks are possible
  ==16186== 
  ==16186== For counts of detected and suppressed errors, rerun with: -v
  ==16186== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 39 from 6)

Here we see that xl list has no leaks! However if I comment out the call to libxl_dominfo_list_free in tools/libxl/xl_cmdimpl.c:main_list() then instead Valgrind reports:

[...]
  ==16203== HEAP SUMMARY:
  ==16203==     in use at exit: 90,112 bytes in 1 blocks
  ==16203==   total heap usage: 16 allocs, 15 frees, 93,529 bytes allocated
  ==16203== 
  ==16203== LEAK SUMMARY:
  ==16203==    definitely lost: 0 bytes in 0 blocks
  ==16203==    indirectly lost: 0 bytes in 0 blocks
  ==16203==      possibly lost: 90,112 bytes in 1 blocks
  ==16203==    still reachable: 0 bytes in 0 blocks
  ==16203==         suppressed: 0 bytes in 0 blocks
  ==16203== Rerun with --leak-check=full to see details of leaked memory
  ==16203== 
  ==16203== For counts of detected and suppressed errors, rerun with: -v
  ==16203== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 39 from 6)
[...]

So it has caught the memory leak! Rerunning with the suggests --leak-check=full goes even further and tells me where the leaked memory was allocated:

[...]
  ==16204== 90,112 bytes in 1 blocks are possibly lost in loss record 1 of 1
  ==16204==    at 0x4024480: calloc (vg_replace_malloc.c:593)
  ==16204==    by 0x4050CE4: libxl_list_domain (libxl.c:548)
  ==16204==    by 0x805EEC7: main_list (xl_cmdimpl.c:3931)
  ==16204==    by 0x804D6DD: main (xl.c:285)
[...]

To ease debugging of domain creation I find it useful to use the -F option to xl create to stop xl from forking and daemonising. Most xl subcommands do not daemonize and so need no special treatment.

For more information on running Valgrind see the Valgrind Documentation.

Conclusion

Valgrind is a powerful tool in the arsenal of the C programmer, and can be used to catch a large number of hard to debug and common issues before they happen can save many hours of tedious debugging. The ability to use tools such as this on Xen toolstack processes is of enormous benefit and has already found several bugs in the XL toolstack.

Michael Banck: 18 Jan 2013

18 January, 2013 - 18:26
What I have been up to over the last couple of years

I have now worked at credativ for four years. A sizable part of that time I spent working with the Munich City Council's LiMux project. The focus of our work is helping them with their LDAP/web-based GOsa user and system installation(based on FAI)/configuration/management system. Personally, I mostly did project management and QA besides occasional coding, bug fixing and on-site consulting.

Contrary to a lot of our other customers, the Munich city council openly acknowledges the work by external companies and lets us talk about it as well. I wrote a lengthy blog post (now available in english) last month about our work. It explains how the LiMux project is using GOsa and FAI and work we did for them.

The project took a long time, but managed to pull off the mass migration over the last two years and reached their goal of 12000 migrated workstation last November. It is now widely regarded as a "success story" by german news sites, and I am glad to have had a small part in it. I really think this was (and still is) an important project, especially after several others backpedalled and Munich remained the last high-profile public sector migration in the german speaking countries.

Besides that, the LiMux project members are really nice people I get along with well both on working hours and during the Munich Debconf11 bid and organizing various BSPs at their office. It is a bit of a shame that not more of their work and customizations are available in Debian, but hopefully this well happen one day. At least our changes to GOsa are up on github.

Petter Reinholdtsen: How to find a browser plugin supporting a given MIME type

18 January, 2013 - 17:40

Some times I try to figure out which Iceweasel browser plugin to install to get support for a given MIME type. Thanks to specifications done by Ubuntu and Mozilla, it is possible to do this in Debian. Unfortunately, not very many packages provide the needed meta information, Anyway, here is a small script to look up all browser plugin packages announcing ther MIME support using this specification:

#!/usr/bin/python
import sys
import apt
def pkgs_handling_mimetype(mimetype):
    cache = apt.Cache()
    cache.open(None)
    thepkgs = []
    for pkg in cache:
        version = pkg.candidate
        if version is None:
            version = pkg.installed
        if version is None:
            continue
        record = version.record
        if not record.has_key('Npp-MimeType'):
            continue
        mime_types = record['Npp-MimeType'].split(',')
        for t in mime_types:
            t = t.rstrip().strip()
            if t == mimetype:
                thepkgs.append(pkg.name)
    return thepkgs
mimetype = "audio/ogg"
if 1 < len(sys.argv):
    mimetype = sys.argv[1]
print "Browser plugin packages supporting %s:" % mimetype
for pkg in pkgs_handling_mimetype(mimetype):
    print "  %s" %pkg

It can be used like this to look up a given MIME type:

% ./apt-find-browserplug-for-mimetype 
Browser plugin packages supporting audio/ogg:
  gecko-mediaplayer
% ./apt-find-browserplug-for-mimetype application/x-shockwave-flash
Browser plugin packages supporting application/x-shockwave-flash:
  browser-plugin-gnash
%

In Ubuntu this mechanism is combined with support in the browser itself to query for plugins and propose to install the needed packages. It would be great if Debian supported such feature too. Is anyone working on adding it?

Update 2013-01-18 14:20: The Debian BTS request for icweasel support for this feature is #484010 from 2008 (and #698426 from today). Lack of manpower and wish for a different design is the reason thus feature is not yet in iceweasel from Debian.

Pages

Creative Commons License ลิขสิทธิ์ของบทความเป็นของเจ้าของบทความแต่ละชิ้น
ผลงานนี้ ใช้สัญญาอนุญาตของครีเอทีฟคอมมอนส์แบบ แสดงที่มา-อนุญาตแบบเดียวกัน 3.0 ที่ยังไม่ได้ปรับแก้