Planet Debian

Subscribe to Planet Debian feed
Planet Debian - http://planet.debian.org/
Updated: 1 week 3 days ago

Laura Arjona: Upgrading my computers to Debian Jessie

28 January, 2015 - 05:40

Until now, I usually run Debian stable at work (in my desktop PC) and stable or testing at home in my laptop. I was upgrading to testing during the freeze, and then, stay in testing (future stable) or stable (when it’s published) until the next freeze.

I have changed this ‘conservative’ pattern. I’ve been running Jessie for many months now, and here I’ll document the different experiences in the computers that I use.

Upgrade or clean install?

I decided to upgrade my computers instead of making a clean install (except in the ones  that were not running Debian).

Although the upgrade process have been fine, I’m still not sure which is the best for my needs. Installing from the beginning forces me to re-read the feature list of the different pieces of software and choose the one that fits best (not the one that I was using some years ago). And maybe I just don’t need that non-free driver anymore because there’s free replacement already, the installer is wise. OTOH, upgrading is easier and quicker, and I got all my software and configurations (and my rubbish) there, nothing is lost.

The computers

Here I will link the blog posts of each computer that I upgrade, when I finish writing the corresponding articles:

  • Husband’s laptop (Acer 5250): Clean install – Done, and OK!
  • My laptop (Compaq Mini 110c): Upgrade – Done and OK!
  • Home server (HP Microserver N54L G7): Upgrade – Done and OK!
  • PC at work (motherboard Asus P5KPL-AM-SE): Upgrade – Done, some issues.
  • Mini-laptop Airis Kira N7000 (ARM board, 128MB RAM) – Clean install – Pending

Filed under: My experiences and opinion Tagged: Debian, English, Moving into free software

Matthias Klumpp: AppStream 0.8 released!

27 January, 2015 - 23:48

Yesterday I released version 0.8 of AppStream, the cross-distribution standard for software metadata, that is currently used by GNOME-Software, Muon and Apper in to display rich metadata about applications and other software components.

 What’s new?

The new release contains some tweaks on AppStreams documentation, and extends the specification with a few more tags and refinements. For example we now recommend sizes for screenshots. The recommended sizes are the ones GNOME-Software already uses today, and it is a good idea to ship those to make software-centers look great, as others SCs are planning to use them as well. Normal sizes as well as sizes for HiDPI displays are defined. This change affects only the distribution-generated data, the upstream metadata is unaffected by this (the distro-specific metadata generator will resize the screenshots anyway).

Another addition to the spec is the introduction of an optional <source_pkgname/> tag, which holds the source package name the packages defined in <pkgname/> tags are built from. This is mainly for internal use by the distributor, e.g. it can decide to use this information to link to internal resources (like bugtrackers, package-watch etc.). It may also be used by software-center applications as additional information to group software components.

Furthermore, we introduced a <bundle/> tag for future use with 3rd-party application installation solutions. The tag notifies a software-installer about the presence of a 3rd-party application bundle, and provides the necessary information on how to install it. In order to do that, the software-center needs to support the respective installation solution. Currently, the Limba project and Xdg-App bundles are supported. For software managers, it is a good idea to implement support for 3rd-party app installers, as soon as the solutions are ready. Currently, the projects are worked on heavily. The new tag is currently already used by Limba, which is the reason why it depends on the latest AppStream release.

How do I get it?

All AppStream libraries, libappstream, libappstream-qt and libappstream-glib, are supporting the 0.8 specification in their latest version – so in case you are using one of these, you don’t need to do anything. For Debian, the DEP-11 spec is being updated at time, and the changes will land in the DEP-11 tools soon.

Improve your metadata!

This call goes especilly to many KDE projects! Getting good data is partly a task for the distributor, since packaging issues can result in incorrect or broken data, screenshots need to be properly resized etc. However, flawed upstream data can also prevent software from being shown, since software with broken data or missing data will not be incorporated in the distro XML AppStream data file.

Richard Hughes of Fedora has created a nice overview of software failing to be included. You can see the failed-list here – the data can be filtered by desktop environment etc. For KDE projects, a Comment= field is often missing in their .desktop files (or a <summary/> tag needs to be added to their AppStream upstream XML file). Keep in mind that you are not only helping Fedora by fixing these issues, but also all other distributions cosuming the metadata you ship upstream.

For Debian, we will have a similar overview soon, since it is also a very helpful tool to find packaging issues.

Jingjie Jiang: Yet another post.

27 January, 2015 - 21:21
In the middle of OPW internship

I originally thought taking part in FOSSOPW is a great chance to lift my coding skills and I shouldn’t miss it. As time passes by, I now have a new thought towards it.

# 1 It do improves your coding skill.

Zack, Matthieu and I often have discussions on coding style. For example, once Zack said, “For code like this, you should explicitly use an if/else clause, not if-return.” I was totally unaware of this sort of issue. Actually I even didn’t know how I should call this problem. Matthieu gave me a detailed FYI link on this in no time.

Besides, my completeness of thinking is also trained. Recently, I was fixing a “HTTP GET Method ?suite=suite-name” issue. It’s a trivial task. And you know most trivial tasks require lots of scattered modification on the source code base. I did have fixed most places, such as the pages of “/src/packagename” and “/search/”. Zack did a thorough review, and pointed out that the pages rendered by “/prefix” has some malformed urls in the HTML. Waiiiit, I should have noticed it. But somehow I missed it. Maybe because my mind was wandering at that time? This made me think, I shall have a thorough view of what I should do before getting my hands dirty. Or more preferably, if I could write down what I exactly want to achieve before coding, then silly problems definitely wouldn’t occur. This may sound a little bit like TDD. ;).

# 2 It makes you look like a (not-that-good) ninja.

I use a macbook. It’s not my fault! I’ve tried several times, but I never successfully find a laptop that is not capable of boiling eggs when running Debian. (and especially KDE+Debian). So I have no way but switched to OSX. The development of Debsources happens on a remote Ubuntu LTS (now Debian SID, haha) virtual machine. Of course I have to install all the dependency on my own, e.g., Postgres, set up port-forwarding, e.g., ssh -D, write automate shell scripts, e.g., dash, but more importantly, I am forced to live under the dark terminal with no GUI. You know the feeling when pain hurts? Yes, exactly! But I survived. How shall I call myself now? A dedicated with-a-lot-of-useless-plugin-installed vimmer? A fond-of-fancy-window tmux-er? Yep, both. I finally found a comfort zone under the black-white-blinking screen. I wonder how people feel when they see a girl hanging out in the library, facing a full-screened black console, typing at a speed of 140wpm (Yeah, I am kidding). I don’t know, but please don’t call me a geek. Show me your respect, I am a ninja!

# 3 It tells you communication is the most important.

I bet anyone who has participated in a group-based project would understand what I mean. For one perspective, communication helps to eliminate misunderstanding. So I won’t doing some useless stuff for all day and finally find out that it totally doesn’t meet the requirement. On the other hand, it speeds up your learning process. I often have problems on git. So in the email I will complain if I mess up with the git repo. After a short while, my dear mentors will reply in detail on how to correctly do the git stuff.

My OPW journey is cool! ;).

Thomas Goirand: OpenStack debian image available from cdimage.debian.org

27 January, 2015 - 19:30

About a year and a half after I started writing the openstack-debian-images package, I’m very happy to announce to everyone that, thanks to Steve McIntyre’s help, the official OpenStack Debian image is now generated at the same time as the official Debian CD ISO images. If you are a cloud user, if you use OpenStack on a private cloud, or if you are a public cloud operator, then you may want to download the weekly build of the OpenStack image from here:

http://cdimage.debian.org/cdimage/openstack/testing/

Note that for the moment, there’s only the amd64 arch available, but I don’t think this is a problem: so far, I haven’t found any public cloud provider offering anything else than Intel 64 bits arch. Maybe this will change over the course of this year, and we will need arm64, but this can be added later on.

Now, for later plans: I still have 2 bugs to fix on the openstack-debian-images package (the default 1GB size is now just a bit too small for Jessie, and the script exits with zero in case of error), but nothing that prevents its use right now. I don’t think it will be a problem for the release team to accept these small changes before Jessie is out.

When generating the image, Steve also wants to generate a sources.tar.gz containing all the source packages that we include on the image. He already has the script (which is used as a hook script when running the build-openstack-debian-image script), and I am planning to add it as a documentation in /usr/share/doc/openstack-debian-images.

Last, probably it would be a good idea to install grub-xen, just as Ian Campbell suggested to make it possible for this image to run in AWS or other Xen based clouds. I would need to be able to test this though. If you can contribute with this kind of test, please get in touch.

Feel free to play with all of this, and customize your Jessie images if you need to. The script is (on purpose) very small (around 400 lines of shell script) and easy to understand (no function, it’s mostly linear from top to bottom of the file), so it is also very easy to hack, plus it has a convenient hook script facility where you can do all sorts of things (copying files, apt-get install stuff, running things in the chroot, etc.).

Again, thanks so much to Steve for working on using the script during the CD builds. This feels me with joy that Debian finally has official images for OpenStack.

Steve Kemp: Recording gym-visits on Linux.

27 January, 2015 - 07:00

I go to the gym every couple of days. I lift things up, then put them down, and sometimes I repeat this process another 30 times. When I'm done I write down what I've done, how many times I did the lifty-droppy thing, and so on.

I want to see pretty graphs. I want to have records of different things. I guess I just need some simple text-boxes:

   deadlift  3 x 7 @ 210lbs.

etc. Sometimes I use machines so I'd say instead:

  converging seated-row  3 x 8 @ 150lbs

Anyway that's it. I want a simple GUI, a bit like a spreadsheet where I can easily add rows of each session. (A session might have 10-15 exercises in it, so not many.) I imagine some kind of SQLite database for the back-end. Or CSV. Either works.

Writing GUI software is hard. I guess I should look at GtK or Qt over the next few days and see if it is going to be easier to do it online via a jQuery + CGI system instead. To be honest I expect doing it "online" is liable to be more popular, but I think a desktop toy-application is just as useful.

Daniel Pocock: Get your Nagios issues as an iCalendar feed

27 January, 2015 - 04:37

The other day I demonstrated how to get your Github issues/bugs as an iCalendar feed.

I'm planning to take this concept further and I just whipped up another Python script, exposing Nagios issues as an iCalendar feed.

The script is nagios-icalendar. Usage is explained concisely in the README file, it takes just minutes to get up and running.

One interesting feature is that you can append a contact name to the URL and just get the issues for that contact, e.g.:

http://nagios-server.example.org:5001?contact=daniel
Screenshots

Here I demonstrate using Mozilla Lightning / Iceowl-extension to aggregate issues from Nagios, the Fedora instance of Bugzilla and Lumicall's Github issues into a single to-do list.

Vincent Fourmond: Linux kernels for a macbook pro retina

27 January, 2015 - 03:47
I was unhappy about the recent Linux (3.14-3.16, and I think 3.17 too) kernels on my Macbook Pro Retina (15'), for a few reasons:

  • the nouveau graphics driver was not handling the graphics card very well (hangs when using the DRM after putting the computer to sleep once, garbage screen on various apps, slow 3D rendering), and I could never get the proprietary nvidia drivers to work (would give blank screen at boot time)
  • very unstable wireless (at least with my box home, but not with all the ones I've tried)
  • and the most painful was the need to recompile the kernel by hand, with the following modifications from stock debian kernel:
    -CONFIG_X86_SYSFB=y
    +# CONFIG_X86_SYSFB is not set
    -CONFIG_FB_SIMPLE=y
    +# CONFIG_FB_SIMPLE is not set
    
    Without these modifications, the screen would be garbled some 5-6 seconds after boot (but SSH would still work, as far as I remember).
The latest 3.18-trunk kernel fixes essentially all the above problems, which is just great. Kudos to everyone involved ! Hope it helps...

Tanguy Ortolo: Scale manufacturers…

26 January, 2015 - 20:54

Dear manufacturers of kitchen scales, could you please stop considering your clients as idiots, and start developing useful features?

Liquid measurement: this is one feature that is available on almost every electronic scale available. Except it is completely useless to people that use the metric system, as all it does is replace the usual display in grammes by centilitres and divide the number on display by ten. Thank you, but no person that has been to school in a country that uses the metric system needs electronic assistance to determine the volume corresponding to a given weight of water, and for people that have not, a simple note written on the scale, stating that “for water or milk, divide the weight in grammes by ten to get the volume in centilitres” should be enough.

Now, there is still one thing that an electronic scale could be useful for, which is determining the volume of liquids other than water (density 1 g/ml) or milk (density approx. equal to 1 g/ml), most importantly: oil (density approx. equal to .92 g/ml for edible oils like sunflower, peanut, olive and canola).

Francois Marier: Using unattended-upgrades on Rackspace's Debian and Ubuntu servers

26 January, 2015 - 15:25

I install the unattended-upgrades package on almost all of my Debian and Ubuntu servers in order to ensure that security updates are automatically applied. It works quite well except that I still need to login manually to upgrade my Rackspace servers whenever a new rackspace-monitoring-agent is released because it comes from a separate repository that's not covered by unattended-upgrades.

It turns out that unattended-upgrades can be configured to automatically upgrade packages outside of the standard security repositories but it's not very well documented and the few relevant answers you can find online are still using the old whitelist syntax.

Initial setup

The first thing to do is to install the package if it's not already done:

apt-get install unattended-upgrades

and to answer yes to the automatic stable update question.

If you don't see the question (because your debconf threshold is too low -- change it with dpkg-reconfigure debconf), you can always trigger the question manually:

dpkg-reconfigure -plow unattended-upgrades

Once you've got that installed, the configuration file you need to look at is /etc/apt/apt.conf.d/50unattended-upgrades.

Whitelist matching criteria

Looking at the unattended-upgrades source code, I found the list of things that can be used to match on in the whitelist:

  • origin (shortcut: o)
  • label (shortcut: l)
  • archive (shortcut: a)
  • suite (which is the same as archive)
  • component (shortcut: c)
  • site (no shortcut)

You can find the value for each of these fields in the appropriate _Release file under /var/lib/apt/lists/.

Note that the value of site is the hostname of the package repository, also present in the first part these *_Release filenames (stable.packages.cloudmonitoring.rackspace.com in the example below).

In my case, I was looking at the following inside /var/lib/apt/lists/stable.packages.cloudmonitoring.rackspace.com_debian-wheezy-x86%5f64_dists_cloudmonitoring_Release:

Origin: Rackspace
Codename: cloudmonitoring
Date: Fri, 23 Jan 2015 18:58:49 UTC
Architectures: i386 amd64
Components: main
...

which means that, in addition to site, the only things I could match on were origin and component since there are no Suite or Label fields in the Release file.

This is the line I ended up adding to my /etc/apt/apt.conf.d/50unattended-upgrades:

 Unattended-Upgrade::Origins-Pattern {
         // Archive or Suite based matching:
         // Note that this will silently match a different release after
         // migration to the specified archive (e.g. testing becomes the
         // new stable).
 //      "o=Debian,a=stable";
 //      "o=Debian,a=stable-updates";
 //      "o=Debian,a=proposed-updates";
         "origin=Debian,archive=stable,label=Debian-Security";
         "origin=Debian,archive=oldstable,label=Debian-Security";
+        "origin=Rackspace,component=main";
 };
Testing

To ensure that the config is right and that unattended-upgrades will pick up rackspace-monitoring-agent the next time it runs, I used:

unattended-upgrade --dry-run --debug

which should output something like this:

Initial blacklisted packages: 
Starting unattended upgrades script
Allowed origins are: ['origin=Debian,archive=stable,label=Debian-Security', 'origin=Debian,archive=oldstable,label=Debian-Security', 'origin=Rackspace,component=main']
Checking: rackspace-monitoring-agent (["<Origin component:'main' archive:'' origin:'Rackspace' label:'' site:'stable.packages.cloudmonitoring.rackspace.com' isTrusted:True>"])
pkgs that look like they should be upgraded: rackspace-monitoring-agent
...
Option --dry-run given, *not* performing real actions
Packages that are upgraded: rackspace-monitoring-agent
Making sure that automatic updates are happening

In order to make sure that all of this is working and that updates are actually happening, I always install apticron on all of the servers I maintain. It runs once a day and emails me a list of packages that need to be updated and it keeps doing that until the system is fully up-to-date.

The only thing missing from this is getting a reminder whenever a package update (usually the kernel) requires a reboot to take effect. That's where the update-notifier-common package comes in.

Because that package will add a hook that will create the /var/run/reboot-required file whenever a kernel update has been installed, all you need to do is create a cronjob like this in /etc/cron.daily/reboot-required:

#!/bin/sh
cat /var/run/reboot-required 2> /dev/null || true

assuming of course that you are already receiving emails sent to the root user (if not, add the appropriate alias in /etc/aliases and run newaliases).

NOKUBI Takatsugu: Weak ssh public keys in github

26 January, 2015 - 12:15

A presentation slide, named ”Attacking against 5 millions SSH public keys – 偶然にも500万個のSSH公開鍵を手に入れた俺たちは” is published, it is a lightning talk in “Edomae security seminar” in Jan 24, 2015.

 He grabbed ssh public keys with  GitHub API (https://github.com/${user}.key), the API is obsoleted, but not closed.

He found short (<= 512 bit) DSA/RSA keys and can solve prime decomposition 256bit RSA key in 3 seconds.

And he repoted there are 208 weak ssh keys generated by Debian/Ubuntu (CVS-2008-0166). It was already announced  by GitHub.

On the other hand, such ssh keys couldn’t solve prime decomposition with fastgcd. It means almost ssh keys in GitHub has no bias in almost random number generators implementations, it is a good news.

Richard Hartmann: KDE battery monitor

26 January, 2015 - 04:11

Dear lazyweb,

using a ThinkPad X1 Carbon with Debian unstable and KDE 4.14.2, I have not had battery warnings for a few weeks, now.

The battery status can be read out via acpi -V as well as via the KDE widget. Hibernation via systemctl hibernate works as well.

What does not work is the warning when my battery is low, or automagic hibernation when shutting the lid or when the battery level is critical.

From what I gather, something in the communication between upower and KDE broke down, but I can't find what it is. I have also been told that Cinnamon is affected as well, so this seems to be a more general problem

Sadly, me and anyone else who's affected has been unable to fix this.

So, dear lazyweb, please help.

In loosely related news, this old status is still valid. UMTS is stable-ish now but even though I saved the SIM's PIN, KDE always displays a "SIM PIN unlock request" prompt after booting or hibernating. Once I enter that PIN, systemd tells me that a system policy prevents the change and wants my user password. If anyone knows how to get rid of that, I would also appreciate any pointers.

Chris Lamb: Recent Redis hacking

26 January, 2015 - 03:52

I've done a bunch of hacking on the Redis key/value database server recently:

  • Lua-based maxmemory eviction scripts. (#2319)

    (This changeset was sponsored by an anonymous client.)

    Redis typically stores the entire data set in memory, using the operating system's virtual memory facilities if required. However, one can use Redis more like a cache or ring buffer by enabling a "maxmemory policy" where a RAM limit is set and then data is evicted when required based on a predefined algorithm.

    This change enables entirely custom control over exactly what data to remove from RAM when this maxmemory limit is reached. This is an advantage over the existing policies of, say, removing entire keys based on the existing TTL, Least Recently Used (LRU) or random eviction strategies as it permits bespoke behaviours based on application-specific requirements, crucially without maintaining a private fork of Redis.

    As an example behaviour of what is possible with this change, to remove the lowest ranked member of an arbitrary sorted set, you could load the following eviction policy script:

    local bestkey = nil
    local bestval = 0
    
    for s = 1, 5 do
       local key = redis.call("RANDOMKEY")
       local type_ = redis.call("TYPE", key)
    
       if type_.ok == "zset"
       then
           local tail = redis.call("ZRANGE", key, "0", "0", "WITHSCORES")
           local val = tonumber(tail[2])
           if not bestkey or val < bestval
           then
               bestkey = key
               bestval = val
           end
       end
    end
    
    if not bestkey
    then
        -- We couldn't find anything to remove, so return an error
        return false
    end
    
    redis.call("ZREMRANGEBYRANK", bestkey, "0", "0")
    return true
    
  • TCP_FASTOPEN support. (#2307)

    The aim of TCP_FASTOPEN is to eliminate one roundtrip from a TCP conversation by allowing data to be included as part of the SYN segment that initiates the connection. (More info.)

  • Support infinitely repeating commands in redis-cli. (#2297)

  • Add --failfast option to testsuite runner. (#2290)

  • Add a -q (quiet) argument to redis-cli. (#2305)

  • Making some Redis Sentinel defaults a little saner. (#2292)

I also made the following changes to the Debian packaging:

  • Add run-parts(8) directories to be executed at various points in the daemon's lifecycle. (e427f8)

    This is especially useful for loading Lua scripts as they are not persisted across restarts.

  • Split out Redis Sentinel into its own package. (#775414, 39f642)

    This makes it possible to run Sentinel sanely on Debian systems without bespoke scripts, etc.

  • Ensure /etc/init.d/redis-server start idempotency with --oknodo (60b7dd)

    Idempotency in initscripts is especially important given the rise of configuration managment systems.

  • Uploaded 3.0.0 RC2 to Debian experimental. (37ac55)

  • Re-enabled the testsuite. (7b9ed1)

Dirk Eddelbuettel: RcppArmadillo 0.4.600.4.0

26 January, 2015 - 03:19

Conrad put up a maintenance release 4.600.4 of Armadillo a few days ago. As in the past, we tested this with number of pre-releases and test builds against the now over one hundred CRAN dependents of our RcppArmadillo package. The tests passed fine as usual, and results are as always in the rcpp-logs repository.

Changes are summarized below based on the NEWS.Rd file.

Changes in RcppArmadillo version 0.4.600.4.0 (2015-01-23)
  • Upgraded to Armadillo release Version 4.600.4 (still "Off The Reservation")

    • Speedups in the transpose operation

    • Small bug fixes

Courtesy of CRANberries, there is also a diffstat report for the most recent release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Jonathan Dowland: Frontier: First Encounters

25 January, 2015 - 20:18

Cobra mk. 3

Four years ago, whilst looking for something unrelated, I stumbled across Tom Morton's port of "Frontier: Elite II" for the Atari to i386/OpenGL. This took me right back to playing Frontier on my Amiga in the mid-nineties. I spent a bit of time replaying Frontier and its sequel, First Encounters, for which there exists an interesting family of community-written game engines based on a reverse-engineering of the original DOS release.

I made some scrappy notes about engines, patches etc. at the time, which are on my frontier page.

With the recent release of Elite: Dangerous, I thought I'd pick up where I left in 2010 and see if I could get the Thargoid ship. I'm nowhere near yet, but I've spent some time trying to maximize income during the game's initial Soholian Fever period. My record in a JJFFE-derived engine (and winning the Wiccan Ware race during the same period) is currently £727,800. Can you do better?

Joey Hess: making propellor safer with GADTs and type families

25 January, 2015 - 10:54

Since July, I have been aware of an ugly problem with propellor. Certain propellor configurations could have a bug. I've tried to solve the problem at least a half-dozen times without success; it's eaten several weekends.

Today I finally managed to fix propellor so it's impossible to write code that has the bug, bending the Haskell type checker to my will with the power of GADTs and type-level functions.

the bug

Code with the bug looked innocuous enough. Something like this:

foo :: Property
foo = property "foo" $
    unlessM (liftIO $ doesFileExist "/etc/foo") $ do
        bar <- liftIO $ readFile "/etc/foo.template"
        ensureProperty $ setupFoo bar

The problem comes about because some properties in propellor have Info associated with them. This is used by propellor to introspect over the properties of a host, and do things like set up DNS, or decrypt private data used by the property.

At the same time, it's useful to let a Property internally decide to run some other Property. In the example above, that's the ensureProperty line, and the setupFoo Property is run only sometimes, and is passed data that is read from the filesystem.

This makes it very hard, indeed probably impossible for Propellor to look inside the monad, realize that setupFoo is being used, and add its Info to the host.

Probably, setupFoo doesn't have Info associated with it -- most properties do not. But, it's hard to tell, when writing such a Property if it's safe to use ensureProperty. And worse, setupFoo could later be changed to have Info.

Now, in most languages, once this problem was noticed, the solution would probably be to make ensureProperty notice when it's called on a Property that has Info, and print a warning message. That's Good Enough in a sense.

But it also really stinks as a solution. It means that building propellor isn't good enough to know you have a working system; you have to let it run on each host, and watch out for warnings. Ugh, no!

the solution

This screams for GADTs. (Well, it did once I learned how what GADTs are and what they can do.)

With GADTs, Property NoInfo and Property HasInfo can be separate data types. Most functions will work on either type (Property i) but ensureProperty can be limited to only accept a Property NoInfo.

data Property i where
    IProperty :: Desc -> ... -> Info -> Property HasInfo
    SProperty :: Desc -> ... -> Property NoInfo

data HasInfo
data NoInfo

ensureProperty :: Property NoInfo -> Propellor Result

Then the type checker can detect the bug, and refuse to compile it.

Yay!

Except ...

Property combinators

There are a lot of Property combinators in propellor. These combine two or more properties in various ways. The most basic one is requires, which only runs the first Property after the second one has successfully been met.

So, what's it's type when used with GADT Property?

requires :: Property i1 -> Property i2 -> Property ???

It seemed I needed some kind of type class, to vary the return type.

class Combine x y r where
    requires :: x -> y -> r

Now I was able to write 4 instances of Combines, for each combination of 2 Properties with HasInfo or NoInfo.

It type checked. But, type inference was busted. A simple expression like "foo requires bar" blew up:

   No instance for (Requires (Property HasInfo) (Property HasInfo) r0)
      arising from a use of `requires'
    The type variable `r0' is ambiguous
    Possible fix: add a type signature that fixes these type variable(s)
    Note: there is a potential instance available:
      instance Requires
                 (Property HasInfo) (Property HasInfo) (Property HasInfo)
        -- Defined at Propellor/Types.hs:167:10

To avoid that, it needed "(foo requires bar) :: Property HasInfo" -- I didn't want the user to need to write that.

I got stuck here for an long time, well over a month.

type level programming

Finally today I realized that I could fix this with a little type-level programming.

class Combine x y where
    requires :: x -> y -> CombinedType x y

Here CombinedType is a type-level function, that calculates the type that should be used for a combination of types x and y. This turns out to be really easy to do, once you get your head around type level functions.

type family CInfo x y
type instance CInfo HasInfo HasInfo = HasInfo
type instance CInfo HasInfo NoInfo = HasInfo
type instance CInfo NoInfo HasInfo = HasInfo
type instance CInfo NoInfo NoInfo = NoInfo
type family CombinedType x y
type instance CombinedType (Property x) (Property y) = Property (CInfo x y)

And, with that change, type inference worked again! \o/

(Bonus: I added some more intances of CombinedType for combining things like RevertableProperties, so propellor's property combinators got more powerful too.)

Then I just had to make a massive pass over all of Propellor, fixing the types of each Property to be Property NoInfo or Property HasInfo. I frequently picked the wrong one, but the type checker was able to detect and tell me when I did.

A few of the type signatures got slightly complicated, to provide the type checker with sufficient proof to do its thing...

before :: (IsProp x, Combines y x, IsProp (CombinedType y x)) => x -> y -> CombinedType y x
before x y = (y `requires` x) `describe` (propertyDesc x)

onChange
    :: (Combines (Property x) (Property y))
    => Property x
    => Property y
    => CombinedType (Property x) (Property y)
onChange = -- 6 lines of code omitted

fallback :: (Combines (Property p1) (Property p2)) => Property p1 -> Property p2 -> Property (CInfo p1 p2)
fallback = -- 4 lines of code omitted

.. This mostly happened in property combinators, which is an acceptable tradeoff, when you consider that the type checker is now being used to prove that propellor can't have this bug.

Mostly, things went just fine. The only other annoying thing was that some things use a [Property], and since a haskell list can only contain a single type, while Property Info and Property NoInfo are two different types, that needed to be dealt with. Happily, I was able to extend propellor's existing (&) and (!) operators to work in this situation, so a list can be constructed of properties of several different types:

propertyList "foos" $ props
    & foo
    & foobar
    ! oldfoo    
conclusion

The resulting 4000 lines of changes will be in the next release of propellor. Just as soon as I test that it always generates the same Info as before, and perhaps works when I run it. (eep)

These uses of GADTs and type families are not new; this is merely the first time I used them. It's another Haskell leveling up for me.

Anytime you can identify a class of bugs that can impact a complicated code base, and rework the code base to completely avoid that class of bugs, is a time to celebrate!

Daniel Pocock: Get your Github issues as an iCalendar feed

25 January, 2015 - 06:07

I've just whipped up a Python script that renders Github issue lists from your favourite projects as an iCalendar feed.

The project is called github-icalendar. It uses Python Flask to expose the iCalendar feed over HTTP.

It is really easy to get up and running. All the dependencies are available on a modern Linux distribution, for example:

$ sudo apt-get install python-yaml python-icalendar python-flask python-pygithub

Just create an API token in Github and put it into a configuration file with a list of your repositories like this:

api_token: 6b36b3d7579d06c9f8e88bc6fb33864e4765e5fac4a3c2fd1bc33aad
bind_address: ::0
bind_port: 5000
repositories:
- repository: your-user-name/your-project
- repository: your-user-name/another-project

Run it from the shell:

$ ./github_icalendar/main.py github-ics.cfg

and connect to it with your favourite iCalendar client.

Consolidating issue lists from Bugzilla, Github, Debian BTS and other sources

A single iCalendar client can usually support multiple sources and thereby consolidate lists of issues from multiple bug trackers.

This can be much more powerful than combining RSS bug feeds because iCalendar has built-in support for concepts such as priority and deadline. The client can use these to help you identify the most critical issues across all your projects, no matter which bug tracker they use.

Bugzilla bugtrackers already expose iCalendar feeds directly, just look for the iCalendar link at the bottom of any search results page. Here is an example URL from the Mozilla instance of Bugzilla.

The Ultimate Debian Database consolidates information from the Debian and Ubuntu universe and can already export it as an RSS feed, there is discussion about extrapolating that to an iCalendar feed too.

Further possibilities
  • Prioritizing the issues in Github and mapping these priorities to iCalendar priorities
  • Creating tags in Github that allow issues to be ignored/excluded from the feed (e.g. excluding wishlist items)
  • Creating summary entries instead of listing all the issues, e.g. a single task entry with the title Fix 2 critical bugs for project foo
Screenshots

The screenshots below are based on the issue list of the Lumicall secure SIP phone for Android.

Screenshot - Mozilla Thunderbird/Lightning (Icedove/Iceowl-extension on Debian)

Dirk Eddelbuettel: Rcpp 0.11.4

24 January, 2015 - 22:44

A new release 0.11.4 of Rcpp is now on the CRAN network for GNU R, and an updated Debian package will be uploaded in due course.

Rcpp has become the most popular way of enhancing GNU R with C++ code. As of today, 323 packages on CRAN depend on Rcpp for making analyses go faster and further; BioConductor adds another 41 packages, and casual searches on GitHub suggests dozens mores.

This release once again adds a large number of small bug fixes, polishes and enhancements. And like the last time, these changes were made by a group of seven different contributors (counting code commits) plus three more providing concrete suggestions. This shows that the Rcpp development and maintenance rests a large number of (broad) shoulders.

See below for a detailed list of changes extracted from the NEWS file.

Changes in Rcpp version 0.11.4 (2015-01-20)
  • Changes in Rcpp API:

    • The ListOf<T> class gains the .attr and .names methods common to other Rcpp vectors.

    • The [dpq]nbinom_mu() scalar functions are now available via the R:: namespace when R 3.1.2 or newer is used.

    • Add an additional test for AIX before attempting to include execinfo.h.

    • Rcpp::stop now supports improved printf-like syntax using the small tinyformat header-only library (following a similar implementation in Rcpp11)

    • Pairlist objects are now protected via an additional Shield<> as suggested by Martin Morgan on the rcpp-devel list.

    • Sorting is now prohibited at compile time for objects of type List, RawVector and ExpressionVector.

    • Vectors now have a Vector::const_iterator that is 'const correct' thanks to fix by Romain following a bug report in rcpp-devel by Martyn Plummer.

    • The mean() sugar function now uses a more robust two-pass method, and new unit tests for mean() were added at the same time.

    • The mean() and var() functions now support all core vector types.

    • The setequal() sugar function has been corrected via suggestion by Qiang Kou following a bug report by Søren Højsgaard.

    • The macros major, minor, and makedev no longer leak in from the (Linux) system header sys/sysmacros.h.

    • The push_front() string function was corrected.

  • Changes in Rcpp Attributes:

    • Only look for plugins in the package's namespace (rather than entire search path).

    • Also scan header files for definitions of functions to be considerd by Attributes.

    • Correct the regular expression for source files which are scanned.

  • Changes in Rcpp unit tests

    • Added a new binary test which will load a pre-built package to ensure that the Application Binary Interface (ABI) did not change; this test will (mostly or) only run at Travis where we have reasonable control over the platform running the test and can provide a binary.

    • New unit tests for sugar functions mean, setequal and var were added as noted above.

  • Changes in Rcpp Examples:

    • For the (old) examples ConvolveBenchmarks and OpenMP, the respective Makefile was renamed to GNUmakefile to please R CMD check as well as the CRAN Maintainers.

Thanks to CRANberries, you can also look at a diff to the previous release As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dirk Eddelbuettel: RcppGSL 0.2.4

24 January, 2015 - 22:20

A new version of RcppGSL is now on CRAN. This package provides an interface from R to the GNU GSL using our Rcpp package.

This follows on the heels on the recent RcppGSL 0.2.3 release and extends the excellent point made by Qiang Kou in a contributed section of the vignette: We now not only allow to turn the GSL error handler off (to not abort() on error) but do so on package initialisation.

No other user-facing changes were made.

The NEWS file entries follows below:

Changes in version 0.2.4 (2015-01-24)
  • Two new helper function to turn the default GSL error handler off (and to restore it) were added. The default handler is now turned off when the package is attached so that GSL will no longer abort an R session on error. Users will have to check the error code.

  • The RcppGSL-intro.Rnw vignette was expanded with a short section on the GSL error handler (thanks to Qiang Kou).

Courtesy of CRANberries, a summary of changes to the most recent release is available.

More information is on the RcppGSL page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dirk Eddelbuettel: RcppAnnoy 0.0.5

24 January, 2015 - 21:22

A new version of RcppAnnoy is now on CRAN. RcppAnnoy wraps the small, fast, and lightweight C++ template header library Annoy written by Erik Bernhardsson for use at Spotify. RcppAnnoy uses Rcpp Modules to offer the exact same functionality as the Python module wrapped around Annoy.

This version contains a trivial one-character change requested by CRAN to cleanse the Makevars file of possible GNU Make-isms. Oh well. This release also overcomes an undefined behaviour sanitizer bug noticed by CRAN that took somewhat more effort to deal with. As mentioned recently in another blog post, it took some work to create a proper Docker container with the required compiler and subsequent R setup, but we have one now, and the aforementioned blog post has details on how we replicated the CRAN finding of an UBSAN issue. It also took Erik some extra efforts to set something up for his C++/Python side, but eventually an EC2 instance with Ubuntu 14.10 did the task as my Docker sales skills are seemingly not convincing enough. In any event, he very quickly added the right fix, and I synced RcppAnnoy with his Annoy code.

Courtesy of CRANberries, there is also a diffstat report for this release. More detailed information is on the RcppAnnoy page page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Diego Escalante Urrelo: Link Pack #05

24 January, 2015 - 08:03

Lever Rukhin Photographs Los Angeles From His Car
Lever Rukhin shoots the sketchiest parts of Los Angeles from his car, taking a really unique perspective that helps you perceive what LA looks like, if you were in a car… An experience that is apparently common to all LA people. People drive too much in the US :-).

It’s a very interesting interview that goes well with his full site: Lev Rukhin.

What I love about this, besides the whole premise, is that Lev went the extra mile and actually hacked his car to make the images he wanted:

Phoblographer: It looks like many of these images have artificial lighting in them. What’s your gear setup, and how do you introduce so much light into the scene from your car?

Lever: About 9 months ago, I affixed a Mola beauty dish onto the roof rack of my ’75 Volvo and juice it with a profoto bi-tube. This takes a bit of practice, as making a turn changes the light completely, which I always try to keep balanced. The Canon 5D3 with a 24mm f1.4 is set up on a tripod. The strobe has allowed me to capture more detail as well as creating a somewhat surreal feel to the sets.

Lev Rukhin http://www.levrukhin.com/

The Invisible Woman: A conversation with Björk
Björk is that Icelandic singer we all hear about but never really pay much attention to because her music is too smart for our simple ears. In this interview she goes over how her latest album is a very personal work, and unexpectedly (?) ends talking about how problematic it’s been to be a female auteur in her generation.

I have seen the same problem she denounces about people assuming that the male members of a team did all the work while the women just sticked to making coffee and sandwiches. I’ve worked with exceptional women that don’t get enough credit, but I’ve also worked with potentially exceptional women who don’t give themselves enough credit.

It’s a very interesting read, specially since it comes from someone who couldn’t be higher in the “art” food chain. Björk is god-damn Björk.

Only thing that bugs me is that Pitchfork decided to hold back most of the interview for publishing next month. I’ll try to go back and read it in full, but I wonder if the technique works for them or if perhaps they are missing the opportunity for a bigger impact. But I digress.

Pitchfork: The world has a difficult time with the female auteur.

B: I have nothing against Kanye West. Help me with this—I’m not dissing him—this is about how people talk about him. With the last album he did, he got all the best beatmakers on the planet at the time to make beats for him. A lot of the time, he wasn’t even there. Yet no one would question his authorship for a second. If whatever I’m saying to you now helps women, I’m up for saying it. For example, I did 80% of the beats on Vespertine and it took me three years to work on that album, because it was all microbeats—it was like doing a huge embroidery piece. Matmos came in the last two weeks and added percussion on top of the songs, but they didn’t do any of the main parts, and they are credited everywhere as having done the whole album. [Matmos’] Drew [Daniel] is a close friend of mine, and in every single interview he did, he corrected it. And they don’t even listen to him. It really is strange.

In Defense of the Selfie Stick
Miguel proposes a different take on the consequences of the selfies stick:

When you ask someone to take a picture of you, technically, they are the photographer, and they own the copyright of your picture.

(…)

All of a sudden, your backpacking adventure in Europe requires you to pack a stack of legal contracts.

Now your exchange goes from “Can you take a picture of us?” to “Can you take a picture of us, making sure that the church is on the top right corner, and also, I am going to need you to sign this paper”.

I don’t know what’s with the selfie stick hate. Let people have fun, it doesn’t hurt. If anything, it prevents them from asking you to take their photo, and if we already established you are the kind of people not a big fan of strangers, all the better, right?

Why Top Tech CEOs Want Employees With Liberal Arts Degrees
Here’s a small extra. When I decided to pursue a humanities/art formal training, I got many naysayers telling me that I was screwing up not specializing even more as a formal (titled) engineer. I argued then, and now, that if I was gonna pay for training, I might as well pay for training outside my comfort zone.

The result resonates perfectly with this article. Of course, it’s not like the thing is settled, but I can back the various quotes in there.

Working with purely technical/engineering types can be an echo chamber, and having trained myself in the humanities and arts I have become incredibly much more sensitive to the human factor of things. I used to think I was already good at this (because we hacker types have lots of confidence), but studying humanities like human communication, social conflict and development, film language, etc; it all has made me a much more capable hacker of things.

There’s also a nice argument to be made about joining the arts when you are already highly skilled on technical matters. Like Robert Rodríguez’s teacher (mentioned in his diary/book Rebel Without a Cause, which I also have to review soon) used to say (generous paraphrasing here): the world is of those who can be their own creative and their own technician.

Both Yi and Sheer recognize that the scientific method is valuable, with its emphasis on logic and reason, especially when dealing with data or engineering problems. But they believe this approach can sometimes be limiting. “When I collaborate with people who have a strictly technical background,” says Yi, “the perspective I find most lacking is an understanding of what motivates people and how to balance multiple factors that are at work outside the realm of technology.”

Interesting food for thought, specially if you know an engineer that ditches the arts as of little value for personal growth in their careers/life.

Read more Link Pack, you’ll love it
  • Link Pack #05 - Lever Rukhin Photographs Los Angeles From His Car Lever Rukhin shoots the sketchiest parts of Los Angeles from his car, taking a really unique perspective that helps you perceive what LA looks like, if you were in a car… An experience that is apparently common to all LA people. People drive too much in the…
  • Link Pack #04 - Writing Your Way to Happiness (nytimes.com) Researches believe that the way we think about, and remember, “our story” can be so powerful that it can actually influence our happiness and success. It’s a nice little article summarizing actual research. The main study referred put fresh university students to test: a group received tools to “rewrite”…
  • Link Pack #03 - What’s that? The third edition of Link Pack of course! Playing with Power (7 minutes, Vimeo) A super awesome story about a stop motion animator that turned a Nintendo Power Glove into the perfect animation tool. It’s a fun, inspiring video :-). I love the Power Glove, it’s so bad. The Power Glove – Angry…
  • Link Pack #02 - First sequel to my Link Pack “series” (I’ll remove the quotes when it’s LP#05): Link Pack #01. This time I’m going for fewer articles, to try to keep things less overwhelming. There’s no special theme, and I’m actually leaving out some nice things I read recently. On the plus side, that means I have good…
  • Link pack #01 - Following the lead of my dear friend Daniel and his fantastic and addictive “Summing up” series, here’s a link pack of recent stuff I read around the web. Link pack is definitely a terrible name, but I’m working on it. How to Silence Negative Thinking On how to avoid the pitfall of being a Negatron…

Pages

Creative Commons License ลิขสิทธิ์ของบทความเป็นของเจ้าของบทความแต่ละชิ้น
ผลงานนี้ ใช้สัญญาอนุญาตของครีเอทีฟคอมมอนส์แบบ แสดงที่มา-อนุญาตแบบเดียวกัน 3.0 ที่ยังไม่ได้ปรับแก้